Here you can find the crowdsourced judged data for first experiment in Shallow pooling for sparse labels paper.
For more details please check out section 3 of the paper.
Data is given in the TSV format in shallow_pooling_exp_1.tsv for all 6980 queries in small dev set of msmarco and includes the following:
- query id
- query text
- Category ( A: when first ranked document from the selected run is qrel. B: When the first ranked document is not qrel)
- Prefrence (whether the judge prefered the qrel or the alternative document)
- qrel id
- qrel text
- alternative document id
- alternative document text