A model that scores well on a public benchmark can still fail on the cameras it will actually run on. Validation has to happen on real footage, with ground truth and failure analysis that reflect how the system will be used.
Pre-trained detectors arrive with published accuracy figures measured on curated datasets. Those numbers tell you the model is competent in general. They tell you very little about how it will behave on a specific camera, mounted at a specific height, looking at a specific loading dock at dusk. Validating on real site footage is the step that separates a demo from a system someone can rely on.
Collect footage that represents the deployment
The validation set should be drawn from the actual cameras, or from cameras with matching placement and optics, across the conditions the system will encounter. That means different times of day, weather, occupancy levels and operational states. If the system has to work at night, a validation set recorded only during the day is not a validation set. A simple coverage matrix, listing each condition and how many clips represent it, makes gaps visible early.
Footage used for validation must be kept apart from footage used for training or fine-tuning. The temptation to peek, especially when a model is underperforming on a particular camera, is strong and it quietly destroys the value of the measurement.
Ground truth that matches the output
Labels should be at the level the system is judged on. If the product is a count of vehicles crossing a line, ground truth is a manual count per interval, not a set of bounding boxes. If the product is an intrusion alert, ground truth is a list of true intrusion events with timestamps. Box-level metrics such as mean average precision remain useful for diagnosing the detector, but they are not the acceptance criterion.
- Define the output the user consumes: count, event, classification or measurement.
- Label validation footage at that level, with clear rules for ambiguous cases.
- Compute user-level metrics first: missed events, false events, count error per interval.
- Use detection-level metrics to explain user-level failures, not to replace them.
Failure analysis over summary metrics
A single accuracy figure hides the information that matters for deployment. The valuable output of validation is a categorized list of failures: missed detections of people carrying large objects, false vehicle detections from headlight reflections on wet ground, identity switches in dense queues. Each category points to a specific remedy, which might be more training data, a change to tracking parameters, a different camera angle or an adjusted rule.
We find it useful to review every failure as a short clip with the model output overlaid. Patterns become obvious in minutes that would take hours to find in a spreadsheet, and the clips become shared evidence when discussing changes with the client.
Validate the whole pipeline, not just the model
Models are validated offline on saved frames at full resolution. Deployed systems decode compressed streams, drop frames under load, resize for inference and run quantized weights on an accelerator. Each step can change results. Final validation should run the actual deployment build, on the target hardware, against the same footage, so that the numbers reported are the numbers the client will experience.
Shadow mode before go-live
Even thorough offline validation misses conditions nobody recorded. A shadow deployment, in which the system runs live and logs its outputs without acting on them, extends validation into the real timeline of the site. Operators compare its events against what they observe; engineers review discrepancies. When the shadow period produces no surprises across the expected range of conditions, the system is ready to act.
Validation done this way takes longer than quoting a benchmark score. It is also the only form of validation that predicts how the system will behave on Monday morning when it matters.