In this talk, I will mainly discuss a task-based approach to measure the effect of various pretraining objectives (e.g., language modeling, CCG supertagging) of sentence encoders on their understanding of function words. The tasks are created by structurally mutating sentences from existing datasets to target the comprehension of specific types of function words in English (e.g., prepositions, wh-words). Our results show that pretraining on language modeling performs the best on average across our probing tasks, supporting its widespread use for pretrainingstate-of-the-art NLP models, and CCG supertagging and NLI pretraining perform comparably. Overall, no pretraining objective dominates across the board, and our function word probing tasks highlight several intuitive differences between pretraining objectives. In addition to these findings, I will discuss ongoing follow-up works and some promising future directions for probing analyses in NLP.
Najoung Kim – “Probing What Different NLP Tasks Teach Machines About Function Word Comprehension and Where To Go Next”
- Creating A Policy Framework for Self-driving Cars and Autonomous Vehicles March 5, 2021
- Engineering course teaches students to harness power of AI for good January 25, 2021
- Machine learning tool gives early warning of cardiac issues in COVID-19 patients January 13, 2021
- Engineering for Professionals Announces New Online Master’s Degree in Artificial Intelligence May 20, 2020
- Uncategorized (4)