Inference
Real-time detection: where the time actually goes
Breaking down the latency budget of a video analytics pipeline and why speeding up the model rarely delivers the expected gain.
When a video analytics scenario misses its latency target, the first instinct is usually to optimize the model. That is reasonable only until the full frame-to-event time budget has been measured.
In practice inference is the smaller part of the path. Stream decoding, host-to-device frame copies, preprocessing, postprocessing and tracking logic together often cost more than the network itself.
A useful exercise is to run the pipeline with a stub in place of the model. If latency does not drop dramatically, further architecture optimization will change almost nothing and the work belongs in the surrounding environment.
The second common loss is batching. Collecting frames from several cameras into one batch raises throughput but adds waiting. For scenarios with an operator in the loop that wait is often more expensive than the GPU utilization gain.
All articles