Featured project 12 · Testing AI features · Java
Testing an LLM Feature (Claude)
An HR policy assistant built on Claude, and the eval harness that decides whether it is fit to ship: a hand-written golden set, deterministic checks on every answer, prompt-injection and data-leak cases at a 100% bar, an LLM judge calibrated against human labels, and repeated runs with pass-rate thresholds.
Purpose
Shows how to test a feature whose output is phrased differently every time, varies between runs and can be talked into misbehaving. The companion project uses AI to write tests; this one tests an AI feature, and tests the tests: the eval data, every check and the judge are verified before any model result is trusted.
Technology stack
| Area | Tools |
|---|---|
| Language | Java 21, Maven wrapper |
| Feature | LeaveDesk HR assistant on Claude Opus 5.5 through the Anthropic Java SDK: structured reply (answer, outcome, cited sections), fictional policy, minimal per-user context |
| Eval | golden set in JSON, deterministic checks, LLM-as-judge for groundedness with a calibration set, per-tag thresholds, Markdown report |
| Tests | TestNG: 86 harness tests with a scripted stub model; the eval as a TestNG suite with one test per threshold |
| CI | GitHub Actions: harness tests on every push, the eval replayed from recordings without an API key |
Architecture overview
evals/cases.json (32 cases x 3 samples)
│
assistant answer ──► checks: outcome · facts · citations · canary · colleagues' dates ──► fail
│ pass
▼
LLM judge (groundedness) ◄── trusted only after calibration on 16 hand-labelled answers (min 90%)
▼
pass rate per tag vs threshold (injection, data leaks: 100%) ──► target/eval/report.mdKey engineering decisions
- Testability starts in the feature: a structured reply makes outcome and citations exact checks
- A fictional policy, so the model cannot answer from general knowledge and the eval shows whether it reads its source
- The assistant never receives colleagues' data; the leak check looks for their real dates in four formats
- A canary string in the system prompt makes instruction leaks detectable with a substring check
- Cheap deterministic checks first; the judge is only paid for answers that already passed them
- The eval data is tested: every hand-written reference answer must pass, and a refuse-everything model cannot pass the set
Implemented features
- Six areas: policy facts, own data, questions the policy does not answer, out of scope, prompt injection, data leaks
- Injection cases include a fake HR notice where the right behaviour is to answer correctly, not to refuse
- Judge calibration with subtle wrong answers (right number, wrong deadline; wrong rounding); a judge that approves everything scores 50% and is rejected
- Report with pass rates per area, flaky cases, failures with reasons, judge agreement, latency percentiles and cost
- Recorded answers keyed on model, prompt, policy, question and sample, replayed in CI
- Design notes and 76 interview questions on testing LLM features, linked to the code (Markdown + PDF)
Setup and execution
mvnw.cmd test :: 86 harness tests, no key needed
llm ask alice How many days do I have left?
llm eval :: golden set + calibration (needs ANTHROPIC_API_KEY or recordings)Sample test results
CI workflow
On every push: the eval data audit and the harness tests. A second job replays the recorded answers through the full eval (judge calibration plus one test per area threshold) once recordings exist. No job needs an API key.
Known limitations and future improvements
Limitations
- 32 cases is a starting set; production use needs cases from real questions
- The judge checks groundedness only, not tone or helpfulness; indirect injection through documents is not covered yet
Next steps
- The first live eval recording
- Indirect injection through a poisoned document, and a scheduled live run that alerts on drift