Document archives for OCR datasets

Separate reading text from understanding a document

Start by naming the possible task. Recognising printed characters, finding the order of paragraphs and extracting fields from a form are different problems. An archive may contain useful evidence for one without supporting the others. These are preparation questions, not evidence that a buyer wants the material.

The Library of Congress ALTO structure documentation separates processing information from page layout, text blocks, lines and strings. That provides a concrete example of how OCR output can retain more structure than a plain text file. It is not a requirement to convert your archive to ALTO or a certification of accuracy.

Write down what you hold now, before planning new processing. If only searchable PDFs remain, say that instead of describing the collection as paired images and verified transcripts.

Count documents, pages and versions separately

A document may span several pages, have attachments and exist in both a scan and a later corrected version. Establish which of these your count includes. A page count should not become a claim about independent documents, and an OCR rerun should not be counted as newly collected material.

Ask an authorised colleague to trace one document inside the existing system. Can they connect its pages in order? Can they distinguish the original image, a processed image, the automatic text layer and a manually amended transcript? Record missing links without exporting the document.

For an invented example, a fictional equipment manual might have a scan, an OCR text layer and a corrected contents page. That is partial checking, not a fully corrected manual. Use this distinction when describing your own review coverage.

What makes a transcript a checked answer?

Record how corrections were produced, who reviewed them by role and which instructions they followed. Ask whether spelling was silently modernised, abbreviations expanded or unreadable text guessed. A faithful transcription and an edited reading copy serve different purposes.

An internal review note can record:

• Whether text was generated automatically, corrected by a person or checked again.
• Whether checking covered whole pages, selected fields or only exceptions.
• How uncertain characters and unreadable regions are marked.
• Whether tables, handwritten notes and page headers were included.
• Which source image and transcript versions were compared.

Do not infer accuracy from a software confidence score alone. If no comparison against the image has been performed, describe the output as unchecked. Preserve disagreements for review rather than turning guesses into reference answers.

Describe the difficult pages as well as the tidy ones

List the document families, languages, approximate periods and capture methods represented. Note changes in scanners or export software if known. Distinguish born-digital pages from scans and photographs; do not imply that a collection covers all three unless it does.

Describe the presence of multi-column text, rotated pages, stamps, faint print, handwriting and tables without reproducing the content. Ask whether blank pages and failed scans were retained or discarded. A collection selected for clean text may not support a review of difficult-document performance.

If the intended task involves extracting fields, establish how each answer links to its page or region. A value copied into a spreadsheet without that connection may be hard to check. Use the annotation disagreement guide when reviewers disagree about boundaries or meanings.

Keep disclosure and technical suitability separate

Before any external review, ask the relevant owner to identify confidential material, third-party documents and personal information. Include images, text layers, filenames and attachments in that discussion. Do not assume that hiding a name on the visible page deals with every version of the document.

Keep permissions questions with the licensing checklist. A technical description is not a decision that documents can be licensed. Do not change retention arrangements, destroy originals or upload the archive to an OCR service to prepare an enquiry.

Your first non-confidential outline can state the broad document type, page and document counts where safe, formats, languages, links between images and text, correction coverage and a known limitation. Fairlode starts with descriptions only. No document, screenshot or real transcript is needed, and a coherent outline is not a buyer offer.