advisory / 02
Agentic Systems Review
A second opinion on the agent you run — or the proposal you are about to sign.
It works in the demo. That sentence is the most expensive one in this field: demonstration success and production reliability are different quantities, and the gap is measurable. Tool-calling agents were under 50% task success in 2024 — and below 25% when the same task was repeated eight times.
A two-week technical second opinion on the agent, LLM feature or retrieval system you already run — or the proposal you are about to sign — measured against the state of the art.
you receive
You receive: a written decision memo — an architecture decision record, an eval-rig specification, and a remediation list ranked by severity, where every finding names the cheapest test that proves it. Plus a 90-minute walk-through with your team.
what is examined — the fixed checklist
- 01
control flow — who decides, the model or the code, and whether that choice is written down anywhere
- 02
eval coverage and consistency — what is measured, on which tasks, and not "did it work once" but "does it work eight times out of eight"; a demo transcript is not evidence
- 03
retrieval quality — chunking, embedding choice, re-index policy, and whether every answer can be traced to the page it came from
- 04
tool and permission boundaries — what the agent may call, under whose identity, and how wide the injection surface is: prompt injection is the first entry on the current top-ten risk list, and excessive agency the one that turns a manipulated output into a real action
- 05
data exposure per route — which calls leave your infrastructure and which stay inside it, per route rather than on average
- 06
cost and latency envelope — agentic patterns consume roughly four times a chat interaction and multi-agent roughly fifteen; the bill is an architecture property, not an operating accident
- 07
fallback and failure behaviour — what happens when a tool is down, an answer is empty, or the model returns something the code does not expect
- 08
handover — who can operate and change this when the person who built it is gone
engagement shape
2–3 weeks, fixed price, one system per review. A second system is a second review — the scope discipline is the point, not an upsell. Where two proposals are in front of you, the review compares both against the same checklist.
2–3 weeks, fixed price, one system per review · one system per review · the checklist is the offer
segment variants
- startups
- bring the repo and the prompts; the shortest path from "it works" to "it works reliably"
- SMB
- the answer to a supplier’s promise, in writing, before the invoice
- enterprise
- run it ahead of the architecture board and file the decision record
- non-tech
- a plain-language verdict: what this system does, what it may touch, and what to ask the provider next
- public administration
- the technical annex to your procurement decision, in German
not for
not a rebuild and not a takeover — the review does not become maintenance.
the artifacts behind this format
the next step
bring the architecture diagram and the eval results; the absence of both is the first finding.
public administration
the technical annex to your procurement decision, in German.
last refreshed 2026-10-17 · reviewed quarterly
Sources: τ-bench (arXiv:2406.12045), ICLR 2025 · OWASP Top 10 for LLM Applications 2025 · Anthropic Engineering, multi-agent research system, 2025.