No healthy human being would ever mistake a turtle for a rifle or parking sign for a refrigerator. Gary Marcus, Deep Learning: A Critical Appraisal
A curated list of recent papers exploring the qualitative limits of deep learning tools for NLP and the open-fields beyond them. I limit this selection to constructive papers with interesting insights.
Outside of focus: interpretability, better models of long-term dependencies (memory)
Models
- Controlling Decoding for More Abstractive Summaries with Copy-Based Networks by Weber, N., Shekhar, L., Balasubramanian, N., & Cho, K. (2018)
- Evaluating Compositionality in Sentence Embeddings by Dasgupta, I., Guo, D., Stuhlmüller, A., Gershman, S. J., & Goodman, N. D. (2018).
- Adversarial Examples for Evaluating Reading Comprehension Systems by Jia, R., & Liang, P. (2017)
- Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks by Lake, B. M., & Baroni, M. (2017)
- Tree-structured composition in neural networks without tree-structured architectures by Bowman, S. R., Manning, C. D., & Potts, C. (2015)
- Not NLP: Schema Networks: Zero-shot Transfer with a Generative Causal Model of Intuitive Physics by Kansky, K., Silver, T., Mély, D. A., Eldawy, M., Lázaro-Gredilla, M., Lou, X., … George, D. (2017)
NLP Datasets
- Annotation Artifacts in Natural Language Inference Data by Gururangan, S., Swayamdipta, S., Levy, O., Schwartz, R., Bowman, S. R., & Smith, N. A. (2018)
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge by Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., & Tafjord, O. (2018)
Opinions/review articles
- Pre-Proceedings of the Cognitive Computation Symposium: Thinking Beyond Deep Learning (CoCoSym 2018) by Besold, T. R. (n.d.).
- Building Machines That Learn and Think Like People by Lake, B. M., Ullman, T. D., Tenenbaum, J. B., & Gershman, S. J. (2016).
- DeepMind's response to the previous Lake et al. article Building Machines that Learn and Think for Themselves: Commentary on Lake et al., Behavioral and Brain Sciences by Botvinick, M., Barrett, D. G. T., Battaglia, P., de Freitas, N., Kumaran, D., Leibo, J. Z., … Hassabis, D. (2017)
- Gary Marcus Critical Appraisal trilogy: Deep Learning: A Critical Appraisal, Innateness, AlphaZero, and Artificial Intelligence and In defense of skepticism about deep learning
- UC Berkeley:
- A Berkeley View of Systems Challenges for AI by Stoica, I., Song, D., Popa, R. A., Patterson, D., Mahoney, M. W., Katz, R., … Abbeel, P. (2017)
- Researches at UB Berkeley's Center for Human-Compatible AI
Debats / less formal
- Joel Grus' Fizz Buzz joke
- Yann LeCun and Christopher Manning on February 2 2018, at Stanford University What innate priors should we build into the architecture of deep learning systems?
- Yann LeCun and Gary Marcus at NYU, October 5 2017 Does AI Need More Innate Machinery?