Hackerman Hall B17 @ 3400 N. Charles Street, Baltimore, MD 21218
In defining ?language understanding? for the purposes of natural language processing (NLP), we must inevitably be informed byhuman cognition: the only existing system that has achieved language understanding. Use of human cognition to evaluate NLP systems is nothing new ?nearly any NLP benchmark relies on some form of comparison to human judgments or productions. In this talk I will discuss a series of projects taking this rationale a step further, examining NLP systems’ capturing of information by drawing on our knowledge of information sensitivity at a number of different levels of human cognition. Ideally we want our systems to extract and represent the same information that humans do at the endpoint of language comprehension ? and because we have an idea of what that information is, we can test for it accordingly. However, we find at times thatthe representational patterns observed in our NLP systems show parallels instead with earlier stages of human language processing that reflect coarser information sensitivity. I discuss experiments examining both of thesetypes of parallels: tests probing the extent to which NLP representationsreflect the compositional meaning information to be expected in the finalrepresentation of a sentence, as well as tests examining the correspondence of NLP representations with earlier stages of human comprehension reflected in human predictive brain responses. I discuss the implications of these results for our assessment of these models, and for our targeting ofideal representational capacities as we improve our models moving forward.
Allyson Ettinger is an Assistant Professor in the Department of Linguistics at the University of Chicago. Her interdisciplinary work draws on methods and insights from cognitive science, linguistics, and computerscience to examine the extraction, representation, and deployment of meaning information during language processing in humans and NLP systems. She received her PhD in Linguistics from the University of Maryland, and spent one year as research faculty at the Toyota Technological Institute at Chicago (TTIC) before beginning her appointment at the University of Chicago. She holds an additional courtesy appointment at TTIC.