PAN-WVC-10

Synopsis

The PAN Wikipedia vandalism corpus 2010 (PAN-WVC-10) is a corpus for the evaluation of automatic vandalism detectors for Wikipedia. For research purposes the corpus can be used free of charge.

This corpus is supplemented by the PAN-WVC-11, which features additional edits in English, Spanish and German. Both corpora should be used to get more representative results.

Download

To download the corpus use the following link:

If you use the dataset in your research, please send us a copy of your publication. We kindly ask you to refer to the corpus via [bib].

Research

As part of our research on automatic vandalism detection we have compiled a corpus of vandalism cases found in Wikipedia. The corpus compiles 32452 edits on 28468 Wikipedia articles, among which 2391 vandalism edits have been identified. To annotate the corpus we have used Amazon's Mechanical Turk; 753 workers have been recruited who cast more than 150000 votes on the edits, so that each edit was reviewed by at least 3 annotators. The achieved level of agreement was analyzed in order to label an edit as "regular" or "vandalism."

The corpus has been successfully employed in the 1st International Competition on Wikipedia Vandalism Detection, PAN'10, which was held in conjunction with the CLEF'10 conference. In the competition, 9 vandalism detectors have been evaluated based on the corpus. More details about the competition can be found in the overview paper.

Previous Corpus Versions. The first vandalism corpus was the Webis-WVC-07. However, that corpus is significantly smaller than the PAN-WVC-10 and its class distribution is not representative.

People

Students: Teresa Holfeld

Publications

Martin Potthast and Teresa Holfeld. Overview of the 2nd International Competition on Wikipedia Vandalism Detection. In Vivien Petras, Pamela Forner, and Paul D. Clough, editors, Notebook Papers of CLEF 11 Labs and Workshops, September 2011. ISBN 978-88-904810-1-7. ISSN 2038-4963. [publisher] [paper] [bib]
Benno Stein, Martin Potthast, Alberto Barrón-Cedeño, Paolo Rosso, Efstathios Stamatatos, and Moshe Koppel. 4th International Workshop on Uncovering Plagiarism, Authorship, and Social Software Misuse (PAN 10). SIGIR Forum, 45 (1) : 45-48, June 2011. [doi] [article] [bib]
Martin Potthast, Benno Stein, and Teresa Holfeld. Overview of the 1st International Competition on Wikipedia Vandalism Detection. In Martin Braschler, Donna Harman, and Emanuele Pianta, editors, Working Notes Papers of the CLEF 2010 Evaluation Labs, September 2010. ISBN 978-88-904810-2-4. ISSN 2038-4963. [publisher] [paper] [bib] [slides]
Martin Potthast. Crowdsourcing a Wikipedia Vandalism Corpus. In Fabio Crestani et al, editors, 33rd International ACM Conference on Research and Development in Information Retrieval (SIGIR 10), pages 789-790, July 2010. ACM. ISBN 978-1-4503-0153-4. [doi] [paper] [bib] [poster]
Martin Potthast, Benno Stein, and Robert Gerling. Automatic Vandalism Detection in Wikipedia. In Craig Macdonald et al, editors, Advances in Information Retrieval. 30th European Conference on IR Research (ECIR 08) volume 4956 of Lecture Notes in Computer Science, pages 663-668, Berlin Heidelberg New York, 2008. Springer. ISBN 978-3-540-78645-0. ISSN 0302-9743. [doi] [paper] [bib] [poster]
Martin Potthast and Robert Gerling. Wikipedia Vandalism Corpus Webis-WVC-07. http://www.uni-weimar.de/medien/webis/research/corpora, 2007. [corpus] [bib]