Bounding boxes and pixel masks answer different questions. Choosing the wrong one for an inspection task costs accuracy, compute and labeling effort. Here is how we decide.
Two model families dominate visual inspection work. Object detection returns a rectangle and a class for each thing it finds. Segmentation returns a label for every pixel, producing precise outlines. Both can identify a defect. They differ in what else they tell you, what they cost to label and run, and how they fail. The choice should follow from the question the inspection has to answer.
What the business question actually is
Start by writing down the decision the system supports. If the decision is binary per part, is there a chip on this edge or not, a detector that finds chip-shaped regions is sufficient. If the decision depends on extent, is this scratch longer than the tolerance, does this contamination cover more than the allowed area, then the shape and size of the defect matter and a bounding box is a poor proxy. A long diagonal scratch produces a large box that is mostly good surface.
- Presence and count of discrete objects: detection.
- Location of a defect for sorting or marking: detection, usually sufficient.
- Measurement of length, area or coverage: segmentation.
- Irregular or elongated defects where boxes overlap heavily: segmentation.
- Verifying that a region is fully covered, such as adhesive or coating: segmentation.
Labeling cost is real and asymmetric
Drawing a box takes seconds. Tracing an accurate mask around an irregular defect can take minutes, and consistency between annotators is harder to maintain for masks than for boxes. For projects with many defect types and limited images of each, this difference decides how much training data is feasible. Interactive annotation tools that propose masks from a few clicks narrow the gap, but it does not disappear. If the question can be answered with boxes, the labeling budget goes much further.
Compute and latency on the line
Segmentation models produce an output the size of the image and generally cost more to run than a detector at the same input resolution. On an edge device inspecting at line speed, that cost may force a lower input resolution or frame rate, which can hurt the detection of small defects more than the extra precision of masks helps. We measure both on the target hardware before committing. In some cases a two-stage design wins: a fast detector finds candidate regions, and a segmentation model runs only on those crops to measure them.
How each one fails
Detectors fail by missing small or low-contrast defects and by producing a box around texture that resembles a defect. Their outputs are easy to review: a box is either on a defect or it is not. Segmentation models fail more subtly, producing masks with ragged boundaries or bleeding into adjacent surface, which matters when the mask feeds a measurement. Their outputs need threshold tuning on the mask probability, and their mistakes are harder for an operator to assess at a glance.
A practical decision path
- Write the inspection decision as a sentence. If it contains a measurement word such as length, area or coverage, segmentation is probably needed.
- Estimate labeling effort for both approaches on a sample of real images with the client's inspectors.
- Benchmark candidate models on the target hardware at the required line speed.
- Prototype on real footage and review failures with the quality team before choosing.
- Consider a two-stage pipeline when both localization speed and measurement precision matter.
Neither approach is better in general. Detection is the right default for presence, counting and localization because it is cheaper to label, cheaper to run and easier to review. Segmentation earns its cost when the shape of the defect is the answer. Deciding deliberately, with the real decision and real images in front of you, avoids the most common and most expensive mistake in inspection projects: building a precise answer to the wrong question.