Metrics

Detector metrics a client can actually read

Why average precision over a dataset is a poor predictor of satisfaction with the system in operation.

·7 min read

An aggregate metric is convenient for comparing experiments but answers the client question poorly: how many false alarms will the operator get per shift.

It is more useful to translate quality into operational numbers. For example, false positives per camera per day at a fixed recall — that figure can be discussed with the operations team.

Metrics should also be reported by object size and shooting conditions. An average figure often hides a collapse on small objects or night frames, and that is exactly where complaints come from.

Finally, the holdout set must be collected on the client site under the same conditions. Quality on a public dataset says almost nothing about behaviour in a specific plant.

All articles