Dataset labels: recording disagreements

Agree what is being labelled

Before comparing labels, establish whether reviewers assessed the same thing. One may be judging a whole document while another is classifying a sentence. One may mark a visible defect while another records the reason an item was rejected. Those are different targets, not necessarily inconsistent judgements.

The W3C Web Annotation Data Model distinguishes an annotation body from its target and describes selectors for identifying parts of a resource. That is a structural reference, not a rule for deciding whose judgement is correct.

Our recommendation is to record the item version, the selected region or text span where relevant, and the question presented to the reviewer. If two people saw different versions or instructions, keep that fact visible before deciding whether the labels can be compared.

Keep initial decisions separate from the final label

In an authorised internal review, retain each initial decision with the instruction version used and a reviewer reference appropriate to your access controls. Do not put reviewer names or real labelled records in the first external description. The purpose is traceability, not public attribution.

Record whether a reviewer worked independently or could see an earlier answer. A second person accepting a pre-filled label is a different review process from two people reaching the same answer independently. Describe the process accurately instead of treating every second click as an independent check.

Keep uncertain, not-applicable and insufficient-evidence states available where they fit the task. Forcing a reviewer to select a confident category can conceal missing information. Do not promise that a particular agreement rate establishes suitability for every possible use.

Find the cause before choosing a winner

Use a short internal disagreement log. Suggested fields are the item reference, instruction version, competing labels, reason for review, relevant evidence, decision and unresolved limitation. Keep the source records in their controlled system rather than duplicating them into an informal spreadsheet.

Useful reason categories include:

• The category definitions overlap.
• The target region or text span differs.
• One reviewer had additional evidence.
• The source is unreadable or incomplete.
• An instruction changed between reviews.
• A clerical error occurred.

These categories are editorial suggestions, not a standard or a buyer's acceptance criteria. Their value is in making the next action explicit: clarify an instruction, recheck a source, correct an error or leave the case unresolved.

Record what the final decision does and does not prove

Consider an invented inspection example. One reviewer labels a photograph surface defect; another labels it image unusable because glare obscures the relevant area. A senior reviewer decides the photograph cannot establish the surface condition. The final status is not evidence that the item was defect-free.

The decision record should preserve both initial labels, the glare limitation and the reason for the final status. If the item is later reinspected, treat that as additional evidence with its own date, not a reason to rewrite the earlier image as clear.

For document work, the same discipline applies when a character cannot be read. An adjudicator's guess should not silently become a verified transcription. The OCR archive guide explains how to describe correction coverage separately from automatic output.

Version rule changes and describe the remaining limits

When an instruction changes, record what changed, why and which items may be affected. Decide with the responsible technical team whether earlier annotations need another review. Do not apply a new definition to old labels by changing the handbook date alone.

Before discussing a collection externally, prepare a non-confidential process outline: target type, label categories, instruction history, independence of reviews, how disagreements were handled and what remains unchecked. Say whether the final label is an administrative decision, a specialist judgement or a measured outcome. These support different interpretations.

This is narrower than the general dataset quality checklist: it concerns the evidence behind labels, not a single score for the archive. Ask the relevant owner to check permissions before any controlled review. Fairlode needs only a description initially; no labelled examples, reviewer records or source files. A documented process does not establish a sale or certify the labels as correct.