40
29.09.27
On this page
About this use case: An EV production line's rule-based inspection system misses roughly 300,000 weld defects per year — and three AI vendors all claim 99%+ detection on reference datasets that bear no resemblance to your welds, your lighting, or your defect mix. tracebloc runs the acceptance test on the manufacturer's own production imagery inside the factory network, where no image leaves and the results are independently scored. Explore the data, submit your own model, and see how your approach compares.
You're qualifying a new automated visual inspection system for your welding line. Three vendors have submitted proposals. All three claim detection rates above 99%. All three have reference customers. All three have demo videos that look convincing.
The problem is that none of those demos were run on your welds, your lighting conditions, your line speed, or your defect mix. And you can't send your production image library to three external vendors to run their own tests — that data is proprietary, and your data governance team would shut it down immediately.
So how do you know which system will actually hold up in production before you sign the contract?
tracebloc provides a secure evaluation workspace where vendors test their inspection systems directly on your production data — without that data ever leaving your facility. Once the workspace is deployed on your infrastructure, vendors submit their computer vision models and run their own training, fine-tuning, and line adaptation inside your environment. The orchestration, data access controls, scoring, and leaderboard happen automatically — no manual coordination required. As a federated learning application, the evaluation runs inside your environment: vendors submit their systems, your data never moves, and you get a side-by-side performance comparison on real production welds.
In this example evaluation, the independent acceptance test revealed a 4-6% gap between every vendor's claimed detection rate and their actual performance on real production welds. The best-performing system — a YOLOv8-based solution submitted by Vendor B — achieved 98.3% defect detection after adapting to the line conditions, versus the claimed 99.3%. Selecting this vendor over the existing rule-based inspection system saves an estimated €4 million per year in missed defect costs. The tracebloc workspace remains in place as a continuous QA system, running new vendor submissions and updated models as the programme evolves.
Weld quality in EV battery housings and chassis components is not a quality checkbox — it is a structural safety requirement. A micro crack in a battery housing weld can propagate over thousands of charge cycles and thermal events. When it fails, it does not fail quietly. The costs cascade: production stoppage, field investigation, potential recall, warranty claims, and — in the worst case — safety incidents.
Andreas Maier, Head of AI & Automation at a major European OEM's production site, has been tasked with replacing the current rule-based visual inspection system. The existing setup catches around 95% of surface defects — which sounds high until you apply it to scale. At 6,000 welds per vehicle and 1,000,000 vehicles per year, a 5% defect escape rate means 300,000 undetected defects reaching the next quality gate or the field.
The NDT team has been flagging the same categories of misses for two years: micro cracks showing as faint shadows in high-speed inline images, porosity in laser welds on mixed substrates, undercuts on seam welds at the edge of the field of view. Rule-based systems were not built for these. AI-based machine vision inspection systems claim to be. Andreas's job is to find out which one actually delivers — on his line, not on a vendor's reference dataset.
The questions Andreas brings to this evaluation are not data science questions. They are operational questions:
The evaluation draws on an augmented weld image dataset in Pascal VOC format — constructed to closely reflect the defect types, image conditions, and class distributions seen in real automotive production environments, without containing proprietary manufacturer data. Full dataset statistics, class distributions, bounding box analysis, and sample images are available in the Exploratory Data Analysis tab.
| Parameter | Value |
|---|---|
| Total images | 1,018 (814 train / 204 test) |
| Image size | 640 × 640 px, colour (3-channel) |
| Annotation format | Pascal VOC (XML bounding boxes) |
| Object classes | 3 — Defect, Metalpiece, Welding Line |
| Total annotated objects | 8,321 (6,719 train / 1,602 test) |
| Avg objects per image | 8.25 (train) / 7.85 (test) |
| Defect instances | 1,932 — avg bounding box area: 1.6% of image |
| Metalpiece instances | 3,328 — avg bounding box area: 12.3% of image |
| Welding Line instances | 3,061 — avg bounding box area: 3.4% of image |
The small average defect size — 1.6% of the image area — is the core detection challenge. Defect bounding boxes range from 11×11 px at the smallest to over 500×600 px at the largest. Systems that perform well on large objects often miss the small, faint defects that matter most on a production line. Each image contains at least one metal piece with a welding line, and may contain one or multiple defects — matching the structure of a real inline inspection frame.
In a real production deployment, the reference library would grow continuously as more images are captured from the line. This example uses 814 training images — a starting point. Larger and more diverse datasets lead to more robust adaptation and more reliable evaluation results over time.
Once tracebloc is deployed on Andreas's factory infrastructure, the evaluation runs with minimal manual overhead. Each vendor receives access to an isolated workspace inside the factory environment. They submit their computer vision model, run their own training and line adaptation using the provided dataset, and receive automated scoring on the held-out test set. Andreas's team sees the ranked results in real time. No images leave the factory. No vendor sees another vendor's submission.
This is the core difference from a standard pilot: vendors work inside your infrastructure, not the other way around. The orchestration — data access controls, training execution, evaluation scoring, leaderboard updates — happens automatically once the workspace is configured.
Phase 1: Out-of-the-box performance
Each vendor's system is tested as submitted, with no adaptation to the production line. This is the real baseline — what the system actually delivers when installed on a new customer's line without customization. In this example, all three vendors started between 93-94.5% detection on real production welds.
Phase 2: Line adaptation
Vendors are given access to the training environment inside the tracebloc workspace. Each vendor transfers their computer vision model into the workspace and runs training on the 814 annotated production images — fine-tuning their model weights to the specific defect types, image characteristics, and object size distribution of this line. After training, the adapted model is evaluated automatically against the independent 204-image test set. Vendors have no visibility into the test set at any point. The evaluation mirrors an independent third-party acceptance test, not a vendor-run demo.
In production deployments, the reference library can be much larger — thousands or tens of thousands of images accumulated from the line over time. A larger, more diverse training set typically translates directly into stronger line adaptation results.
| Vendor | Inspection System | Claimed Detection Rate | Claimed False Alarm Rate | Inspection Speed |
|---|---|---|---|---|
| A | DSF-YOLO | 99.1% | <2% | 3 ms/weld |
| B | YOLOv8 | 99.3% | <1% | 7 ms/weld |
| C | Faster R-CNN | 98.8% | <1% | 10 ms/weld |
| Vendor | Claimed Rate | Out-of-the-Box | After Line Adaptation | Measured Speed |
|---|---|---|---|---|
| A | 99.1% | 94.0% | 96.2% | 4.1 ms/weld |
| B | 99.3% | 94.5% | 98.3% ✅ | 7.8 ms/weld |
| C | 98.8% | 93.2% | 97.9% | 11.2 ms/weld ❌ |
→ View the full model leaderboard — see ranked detection results, false alarm rates, and speed measurements across all submitted systems.
What this means in plain terms:
Every vendor's out-of-the-box detection rate dropped 4-6 percentage points when tested on real production welds versus their reference data. This gap is not unusual — it is the standard reality of deploying vision systems on new production lines. The question is how much the system recovers after line adaptation, and how honest the vendor's original claims were.
Vendor C was eliminated regardless of detection rate: at 11.2 ms per weld, it cannot keep pace with the welding line without creating a bottleneck. Speed is a hard constraint, not a tradeoff.
Vendor B showed the strongest recovery after line adaptation — gaining 3.8 percentage points and coming closest to its original claim. It did not reach 99.3%. No vendor did. But 98.3% on real production data, at 7.8 ms per weld on edge hardware, was the best independently verified result.
The business case for the inspection system investment comes down to one number: how many defective welds escape to the field, and what each one costs.
| Parameter | Value |
|---|---|
| Annual production volume | 1,000,000 vehicles |
| Welds per vehicle | ~6,000 |
| Total welds per year | 6 billion |
| Weld defect rate | 0.1% → 6,000,000 potentially defective welds/year |
| Cost per field defect escape (rework, warranty, recall exposure) | €30 |
| Inspection Strategy | Detection Rate | Defect Escapes/Year | Defect Escape Cost | System Cost | Total Annual Cost |
|---|---|---|---|---|---|
| Current rule-based system | ~95.0% | 300,000 | €9.0M | €0 | €9.0M |
| Vendor A | 96.2% | 228,000 | €6.8M | €1.2M | €8.0M |
| Vendor B ✅ | 98.3% | 102,000 | €3.1M | €1.5M | €4.6M |
| Vendor C | 97.9% | 126,000 | €3.8M | €1.4M | €5.2M |
Estimated annual saving with Vendor B: over €4 million versus the existing rule-based system, primarily from reducing defect escapes from 300,000 to 102,000 per year.
The gap between Vendor A and Vendor B (€8.0M vs €4.6M total cost) is driven almost entirely by the difference in detection rate — 96.2% vs 98.3%. At scale, 2 percentage points of detection rate is worth €2.2M per year in avoided defect costs.
Vendor B's YOLOv8-based inspection system was selected for full inline deployment. At 98.3% detection and 7.8 ms per weld on production edge hardware, it clears the speed requirement and achieves the best independently verified detection rate.
The 98.3% result falls short of the 99% acceptance threshold. Andreas plans to close the gap through targeted labelling of additional defect examples — particularly small defects at the edge of the field of view — and re-validation through the same tracebloc workspace.
Critically, the workspace does not close after the initial selection. It becomes a permanent part of the quality assurance infrastructure:
This continuous QA model is what distinguishes a one-off vendor selection from a sustainable inspection programme. The acceptance test that qualified Vendor B today is the same environment that will flag performance drift six months from now.
The broader lesson applies across federated learning applications in industrial settings: vendor claims made on reference data do not predict performance on production lines. The only way to know which system actually holds up is to run an independent acceptance test on your own data — in your own environment, continuously, not just at purchase.
Explore this use case further:
Related use cases: See how the same independent evaluation approach applies to drone-based aerial object detection for traffic monitoring and warranty claims classification for automotive manufacturers.
Deploy your workspace or schedule a call.
Disclaimer: The dataset used in this use case is augmented — designed to closely reflect the characteristics of real automotive production weld imagery, but not sourced from any actual manufacturer. The persona, scenario, vendor names, detection rates, and financial figures are illustrative and based on patterns observed across real-world deployments. They do not represent any specific company, product, or contractual outcome.
