What is NLI?
Natural Language Inference (NLI) is the task of determining, given a context, whether a hypothesis is:
- Definitely correct (Entailment); or
- Definitely incorrect (Contradiction); or
- Neither (Neutral).
NLI is one of the most important tasks in natural language processing and natural language understanding.
What did you do?
We collected a new NLI dataset over multiple rounds, where annotators tried to find examples that current state-of-the-art models get wrong. In other words, we asked annotators to be adversarial, and to try to find weaknesses in current models. We then used the newly collected data to train an improved model, and repeated the process. This principle could be applied for many consecutive rounds, continuously finding and repairing weaknesses until, in the limit, we would achieve full natural language understanding :) This demo shows the BERT model from round 1.
How does it work?
You are presented with a context and a target label. Your goal is to find a hypothesis that fools the system into predicting the wrong label (but another person would correctly assign the target label). Once you've entered your hypothesis, you will get feedback on the model's prediction.
Do you have an example?
Sure! You are given a context and target label, let's say:
- Context: Natural language processing is a subfield of artificial intelligence research.
- Your target (what the correct label should be): Entailing
And now your goal is to fool the model:
- Your hypothesis: AI has multiple subfields
- Model prediction: Neutral
Yay! You fooled the model.