Dataset preparation: the stage you cannot skip

February 24, 2026·6 min read

A model cannot be more accurate than the labels it was trained on. If two annotators interpret a borderline case differently, that uncertainty ends up in the weights.

So the work starts with an instruction and a set of reference examples, including contested ones. The instruction keeps growing: every disputed case resolved by hand becomes a new clause.

Inter-annotator agreement is worth measuring numerically on an overlapping subsample. The resulting value is the practical ceiling for the metric you are chasing.

And always version the dataset together with the labeling methodology. Six months later, reproducing the result without it is close to impossible.