Summary Explorer
Summary Explorer is a tool to visually inspect the summaries from several state-of-the-art neural summarization models across multiple datasets. It provides a guided assessment of summary quality dimensions such as coverage, faithfulness and position bias. You can inspect summaries from a single model or compare multiple models.
The tool currently hosts the outputs of 55 summarization models across three datasets: CNN DailyMail, XSum, and Webis TL;DR.
To integrate your model in Summary Explorer, please prepare your summaries as described here and contact us.
Accepted at EMNLP 2021 (Demo track). A pre-print version of the paper is available here.
Use cases
1. View Content Coverage of the Summaries
2. Inspect Hallucinations
3. View Named Entity Coverage of the Summaries
4. Inspect Faithfulness via Relation Alignment
5. Compare Agreement among Summaries
6. View Position Bias of a Model
Local Deployment
Download the database dump from here and set up the tool as instructed here. The text processing pipeline and sample data can be found here.
Note: The tool is in active development and we plan to add new features. Please feel free to report any issues and provide suggestions.
Citation
@inproceedings{syed:2021,
title = "Summary Explorer: Visualizing the State of the Art in Text Summarization",
author = {Syed, Shahbaz and
Yousef, Tariq and
Al Khatib, Khalid and
J{\"a}nicke, Stefan and
Potthast, Martin},
booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations",
year = "2021",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2021.emnlp-demo.22"
}
Acknowledgements
We sincerely thank all the authors who made their code and model outputs publicly available, meta evaluations of Fabbri et al., 2020 and Bhandari et al., 2020, and the summarization leaderboard at NLP-Progress.
We hope this encourages more authors to share their models and summaries to help track the qualitative progress in text summarization research.