AI-Assisted Test Automation (Claude)

Purpose

Shows how to use a large language model in test automation without letting it lower the bar. Generating a plausible test is fast; the expensive failure is a test that passes and checks the wrong thing. So the project is mostly guardrails: automatic gates before review, a review file built for a five-minute check, an explicit approval recorded in the code, and evidence-based failure triage.

Technology stack

Architecture overview

test-cases/TC-*.md ──► Claude draft ──► gate: rules ─► javac ─► 3 runs ──► review.md ──► approve ──► src/test (@AiDrafted)
                         ▲                                    │
                         └──── findings, up to 2 repair rounds ◄┘

failed UI run ──► evidence per failure ──► Claude triage ──► triage.md (verdict, quoted evidence, draft ticket)

Key engineering decisions

• One class talks to Claude: typed answers, recording and replay, cost logging and refusal handling in one place
• Answers are recorded and keyed on model, prompt, test case and framework code, so CI replays them deterministically and a changed prompt shows up as a missing recording
• The gate runs cheapest first and stops at the first failure: text rules, in-process compilation, three TestNG runs against a fresh app
• Gate rules explain themselves ("found Thread.sleep on line 18"), because the same message goes back to Claude in a repair round
• The model may only use page objects; if a step needs a missing method it reports it instead of using a selector
• No auto-merge and no self-healing: approval names the reviewer, and failures go to triage with the evidence

Implemented features

• Generator: test case and framework code in, a draft with coverage per expected result, assumptions and missing page methods out
• Quality gate: framework and safety rules, javac against the test classpath, three runs each with its own evidence folder
• Approve step: only gate-passed drafts, rules re-checked after reviewer edits, @AiDrafted(model, reviewedBy, reviewedOn) added
• Failure triage: error, console, DOM, screenshot, test case and test code in; verdict, confidence, quoted evidence and a draft bug ticket out
• 48 tests that need no API key: app rules and API, every gate rule rejecting a crafted bad draft, real gate runs, replay, approve marking
• Design notes and 82 interview questions on AI-assisted testing, linked to the code (Markdown + PDF)

Setup and execution

mvnw.cmd test                       :: app + tooling tests, no key needed
mvnw.cmd test -Dsuite=ui            :: UI tests
ai generate TC-103                  :: Claude drafts, the gate checks (needs ANTHROPIC_API_KEY)
ai approve TC-103 Your Name         :: move a passed draft into src/test

Sample test results

CI workflow

On every push: the app and tooling tests (including real gate runs in a browser), and the UI suite on Chromium, Firefox and WebKit. Two more jobs replay the recorded AI answers: approved drafts go through today's gate again, and a planted product bug must be triaged as PRODUCT_BUG. No job needs an API key.

Known limitations and future improvements

Limitations

• The rule stage is text matching, a filter and not a sandbox; the human review is the control
• One small demo app; triage looks at one failure at a time

Next steps

• The first live generation and triage recordings
• An evaluation set of cases with known-good tests, and batch generation for larger backlogs