Evaluation preparation
Train/test leakage in business datasets
Separating filenames is not enough to make an independent test. Related cases, later outcomes and preparation steps can let information cross the boundary between training and evaluation.
Last checked
Describe which records belong together and when information became available. Let the intended evaluation question determine the separation.
What must be genuinely unseen?
Start with the proposed evaluation question. Is the system meant to handle a new case for an existing product, a new customer organisation, a new machine or a later time period? The answer changes which relationships matter. Do not choose a random split merely because it is easy to generate.
The scikit-learn cross-validation documentation explains grouped splitting and time-series evaluation. Its GroupKFold approach keeps a group out of both training and test portions of the same split. That supports the principle of defining related groups, not a claim that this specific tool is suitable for every archive.
Our practical recommendation is to document the intended unseen unit before any export. The training versus evaluation guide covers permitted purposes; this article concerns technical independence, which is a separate question.
Map records that share the same underlying event
Identify the relationships available in the source system. A support case may have several messages and a resolution. An inspected component may have multiple photographs and a rework image. A document may appear as a scan, text extraction and corrected transcript. Separate files can still describe the same underlying item.
Prepare a non-confidential relationship note stating which grouping keys exist and whether they are reliable. Do not share the actual customer, equipment or case identifiers. If historical migrations broke the links, mark the affected period as uncertain.
For an invented example, placing a fault report in a test collection while its near-identical resolution summary is in training could make the task less independent than intended. A different filename does not resolve that relationship. Ask the technical reviewer which related material must stay together for the proposed question.
Check what was known at prediction time
A current database row may contain information added after the event being predicted. Ask when each proposed field became available, not just when the record was first created. A final resolution category should not be treated as an input available when the original fault was reported.
Use an internal timeline with three points: the decision being simulated, the information available then and later additions. Record whether historical field values are retained or whether only today's state survives. Do not reconstruct missing historical states by guessing.
Where the intended task concerns future periods, ask the technical team to assess chronology, delayed outcomes and overlapping observation windows. A calendar cut-off is not enough if records on either side still share information that would have been unavailable at the simulated decision time.
Include preparation steps in the review
The scikit-learn guidance on common pitfalls warns against fitting preprocessing steps on test data. It gives feature selection as an example and explains that such leakage can produce overly optimistic evaluation results.
For a business owner, the useful question is not which library to install. Ask who prepares the material and which records influence each preparation step. An internal record can identify the split decision, learned transformations, duplicate checks and who can access held-out answers.
Do not launch a cleaning or model-training job just to answer an initial licensing enquiry. Describe existing processing and known limitations. If an earlier benchmark used the whole collection during preparation, disclose that fact rather than presenting its score as independently validated.
Keep a separation record, not a performance promise
If a controlled technical review is later approved, ask for a versioned record of the input collection, grouping rule, time boundaries, exclusions and resulting partitions. Record how related copies were checked and what was not checked. A reproducible split is useful evidence, but it is not proof that every contamination risk has been removed.
Keep training, development and final evaluation uses explicit. Ask who can see final answers and how changes prompted by test results are recorded. Repeatedly changing a system in response to the same final test raises a different question from running that test once after development.
For the initial Fairlode outline, state only the broad task, related-record structure, available timestamps and unresolved risks. No test answers, benchmark files or real examples are required. This work neither establishes buyer demand nor changes licensing permissions. Use the readiness checklist before any separately agreed disclosure.
General information, not legal advice, a valuation or a buyer offer. Illustrative examples are not customer case studies.
Sources and further reading
Sources checked 4 November 2026. Historical research is background, not evidence of current buyer demand.