Towards Debiasing Fact Verification Models

• Symmetric evaluation set based on the FEVER (fact verification) dataset
• Regularization-based method

Symmetric dataset

To download the symmetric evaluation dataset from the EMNLP 2019 paper Towards Debiasing Fact Verification Models use this link.

Version 0.2

We release a version that includes new cases. This version is split to dev (708 pairs) and test (712 pairs) to allow models to use the dev set for hyperparameter tuning.

Version 0.1

The version used in "Towards Debiasing Fact Verification Models" paper.

We've implemented the baseline and the reweighted version on the latest version of the pytorch-transformers repository (link). Since the test set is small, there are some random variations across different runs using different servers/GPUs. Therefore, to allow better comparison across methods, we've run the training five times with different random seeds and report the average and std of the runs:

Dataset format

As described in the paper, the cases are based on the FEVER dataset.

Each line in the jsonlines file contains:

• id - matches the FEVER id. For the new pairs, a suffix of 000000{2,3,4} is added.
• label - SUPPORTS or REFUTES.
• claim - the claim.
• evidence_sentence - the evidence.

Training

Our processed FEVER training data is available here. It includes only cases that can be validated with a single evidence sentence. The evidence sentences for the NOT ENOUGH INFORMATION sampled from the NSMN retrieval model.

The processed FEVER evaluation data is available here.

In order to train the baseline model, use the run bash train_baseline.sh.

To use the re-weighted training, add the weighted_loss flag.

Citation

If you find this repo useful, please cite our paper.

@InProceedings{schuster2019towards,
  author = 	"Schuster, Tal and
  			Shah, Darsh J and
  			Yeo, Yun Jie Serene and
  			Filizzola, Daniel and
  			Santus, Enrico and
  			Barzilay, Regina", 			
  title = 	"Towards Debiasing Fact Verification Models",
  booktitle = 	"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP)",
  year = 	"2019",
  publisher = 	"Association for Computational Linguistics",
  url = 	"https://arxiv.org/abs/1908.05267"
}

Related papers

• Automatic Fact-guided Sentence Modification
• Simple but effective techniques to reduce biases
• FEVER: a large-scale dataset for Fact Extraction and VERification
• Adversarial attacks against Fact Extraction and VERification