From 60% to 85% pilot evaluation with multimodal retrieval

My role: AI consultant, AI architect, and practice lead; architected both use cases, which the team built.

Executive Context#

A premium German automaker needed to help central engineering teams respond to warranty and insurance claims from service centres and vendors.

The Actual Problem#

Two related decisions required different workflows: whether the claim situation was valid, and which technical remedy could address a rare, complex fault.

For uncommon faults, service-centre staff could struggle to distinguish equipment failure from design failure.

Diagnostic Approach#

• AI Necessity: Engineers had to compare free-text notes, vehicle histories, codes, documents, and visual evidence. Retrieval and drafting assistance addressed that information-search work.
• Data Sufficiency: Claim clarification used vehicle history, previous work, and claim notes. Remedy search added superseded-parts records, similar past problems, short/spec/fault codes with timings, accident and service history, and photographs.
• Methodological Fit: Retrieval needed to be specific to the fault and specification codes. Engineers checked pilot recommendations; a reranker addressed the weak specificity of the initial retrieval.
• Infrastructure Readiness: Several use cases depended on the same vehicle data. A shared car-behaviour API supported reuse. Each service was Dockerised and pushed to AWS ECR, and a QA agent tested edge cases against the image before deploying it.

Strategic Intervention#

We built two workflows on one shared source of vehicle data.

1. Clarify the Claim#

We parsed vehicle history by dimension, including when previous work had been done. Retrieval over past notes and history helped assess the claim situation and catch repeat claims. The workflow passed on the claim value and produced three scores: claim genuineness, service-centre score, and car-behaviour score.

2. Find and Draft a Technical Remedy#

We combined multimodal retrieval with recommendations: search for similar past solutions, assess whether a solution was implementable, surface the documents associated with the relevant codes, and draft a remedy for the engineer.

3. Share Vehicle Data Through One API#

We built a car-behaviour API exposing vehicle behaviour and a car-characteristics metric for reuse across the use cases.

Outcome#

Metric	Before	After	Context
Recommendation evaluation	~60%	~85%	Engineer-checked pilot; improved after reranking
Information retrieval and remedy drafting	~30 minutes per problem	~3 minutes retrieval + ~5 minutes drafting	Approximately 8 minutes for these two steps
Overall engineering time effort	Baseline	Approximately halved	Separately reported overall workflow result

Engineers checked whether each recommendation was right during the pilot. Evaluation started at approximately 60% because retrieval was insufficiently specific about specification codes and document sections. Adding a reranker lifted it to approximately 85%.

Retrieval was also evaluated with a problem-specific variant of Brier scoring: compare the specifications and documents needed for a claim with those supplied by the system, convert that coverage into a probability, and score it. This was an evaluation designed for this problem.

Strategic Takeaway#

The useful improvement was finding the correct technical context before drafting an answer. Reranking improved retrieval specificity, engineer checks grounded the pilot evaluation, and the shared API made central vehicle information reusable across both workflows.