Workflow Incident Investigator

Design

Incident evidence is scattered, and the deployment that happens to land nearby in time is easy to blame. Rather than a free-form agent, the system runs a bounded workflow of narrow roles: validate → triage → runtime analyst ‖ change analyst (in parallel) → verifier → deterministic checks.

Each role is a single model call with a typed JSON output and no tools of its own. Model output is treated as untrusted: application code rejects unknown evidence IDs, uncited facts, out-of-range confidence scores, any recommendation that isn't read-only, and any claim that an action was already taken.

Evaluation & observability

Six fictional scenarios — a load surge, a configuration regression, database saturation, a retry storm, a deliberately under-determined case where the correct answer is "I can't tell," and a trigger-plus-amplifier case — are scored against hidden expected answers on nine rule-based dimensions, so the project measures whether the system identifies the right cause rather than just rewording one diagnosis.

The Java build logs per-step timing, token counts, and correlation IDs; the Python build uses LangSmith traces and experiments. A shared demo UI serves both backends with clickable evidence chips, a timing view, a reveal-expected-answer button, and a side-by-side compare mode.

The whole test suite — 43 Java tests and 26 Python tests — runs against a stubbed model, so it can be verified fully offline without any AWS access.

Challenges & what I learned

A perfect score hid real flaws. The first version of the rubric scored 1.00, but reading the actual generated answers surfaced a wrong claim — a deployment described as happening "3 minutes after onset" when it was actually 3 minutes after the alert fired and 25 minutes after onset. The fix was an application-computed sorted timeline, explicit rules distinguishing alert time from onset time from change time, and a new check that a runner-up hypothesis isn't scored suspiciously close to the winner.

Test data can be the bug. The model correctly flagged that the fictional evidence itself had a one-minute contradiction and a missing "jobs per start" figure — so the evidence got fixed, not the model.

Most of the friction was infrastructure, not AI: Bedrock access-request forms, expired login sessions, API keys, and a shell `NAME= value` trap that silently loaded a key as empty.

Treating evidence as data — never as instructions — and treating model output as untrusted until verified shaped the whole design, not just the guardrail checks.

Implemented

• Two implementations, one contract — Java (Spring Boot 4, Spring AI) and Python (LangGraph, LangSmith) sharing the same prompts, scenarios, and scoring
• Parallel agents — the runtime and change analysts run concurrently (virtual threads in Java; a fan-out/fan-in graph in Python), with tests proving they overlap
• Guardrails enforced in code, not prompts — citation checks, a read-only invariant, and no leaking of provider errors
• Evaluation harness — six scenarios (surge, config regression, DB saturation, retry storm, an under-determined case where the correct answer is "I can't tell," and a trigger-plus-amplifier case) scored on nine rule-based dimensions
• Observability — per-step timing, token counts, and correlation IDs in Java; LangSmith traces and experiments in Python
• Shared demo UI — one page served by both backends, with clickable evidence chips, a timing view, a reveal-expected-answer button, and a side-by-side compare mode
• Tested offline — 43 Java and 26 Python tests against a stubbed model, so no AWS access is needed to run the test suite

Outcomes

Both implementations picked the correct cause in 12 of 12 runs (6 scenarios × 2), using Claude Haiku 4.5. Mean scores on the nine-dimension rubric: Java 0.98, Python 0.93 — the gap is most likely model variance rather than a framework effect, since the prompts were identical and it was one run each. Speed was about 30–40 seconds per investigation (analysts 7–12s in parallel, verifier 18–25s); all six scenarios took about 3 minutes. Cost was a few cents per full evaluation.