An edge AI device is more than a model file. Here is what a production inference pipeline on NVIDIA Jetson looks like, stage by stage, and where the engineering effort actually goes.
When people picture edge AI they picture a model running on a small board. The model is one component among many, and often not the one that determines whether the device works. A production pipeline on an NVIDIA Jetson module has to decode video, feed an accelerator efficiently, run perception, turn results into events and stay healthy for months without a keyboard attached. This article walks through that pipeline as we build it.
Stage one: ingest and decode
Cameras deliver compressed streams, usually H.264 or H.265 over RTSP. Decoding on the CPU would consume the budget before inference began, so the pipeline uses the module's hardware video decoders. Decoded frames stay in device memory where possible, avoiding costly copies between CPU and GPU. Multiple streams are decoded concurrently, and the pipeline has to tolerate cameras that disconnect, change resolution or deliver corrupt frames without taking everything else down.
Stage two: pre-processing
Models expect a fixed input size and normalization. Resizing, color conversion and normalization are performed on the GPU as part of the same memory pipeline. Frame sampling is also decided here: a counting application may only need a few frames per second per camera, and skipping frames before pre-processing is the cheapest optimization available. Regions of interest can be cropped so the model spends its capacity on the part of the image that matters.
Stage three: the inference engine
Models are trained in PyTorch, exported to ONNX and then compiled for the Jetson's accelerated runtime. Compilation fuses operators, selects kernels for the specific hardware and, with calibration data, quantizes weights to lower precision. The result is a model that runs several times faster than the original framework export. Two practices keep this safe:
- Measure accuracy on the validation set after every export and quantization step; precision changes can shift detection thresholds.
- Batch frames across cameras where latency allows, so the accelerator processes several inputs per call instead of one.
- Pin model versions to a manifest on the device so a running system can report exactly which weights produced each result.
Stage four: post-processing and tracking
Raw detector output is a dense set of candidate boxes. Non-maximum suppression, confidence filtering and class mapping reduce it to a clean list per frame. A multi-object tracker then associates detections across frames into persistent identities, using motion prediction and, where needed, appearance features. Tracking runs on the CPU and is often the second most expensive stage after inference, so it is tuned per use case: a crowded platform needs a different configuration from a quiet car park.
Stage five: analytics and events
Zones, lines, dwell thresholds and schedules turn tracks into events. This stage is where the device becomes useful. It runs entirely locally, so events are produced even when the network is down, and it is the stage most often changed by the customer, so its configuration lives in a file or API rather than in code. Events are written to a local queue with evidence clips cut from a short rolling buffer.
Stage six: delivery and resilience
Events are forwarded to a central platform over an authenticated connection with retries. When connectivity drops, the queue persists to local storage and drains when the link returns. The device also reports its own health: temperatures, decoder utilization, inference latency, dropped frames and disk usage. These metrics are how a fleet operator knows a camera has failed or a module is thermally throttling before any user notices.
What surrounds the pipeline
All of this runs in containers so a device can be rebuilt from a known image. A supervisor restarts failed services. An update agent pulls signed releases and rolls back if the new version fails its health checks. Secrets and camera credentials are stored encrypted. Logs are rotated. None of this is specific to AI, and all of it is necessary for a device that lives in a roadside cabinet for years.
The model is a few megabytes. The system that keeps it useful in the field is most of the engineering.
Understanding the full pipeline changes how edge projects are planned. Latency budgets are allocated across decode, inference and tracking, not just inference. Hardware is chosen for decoder count and memory bandwidth as much as raw compute. And the operational layers are designed from the start, because they are what turn a working prototype into a deployed system.