07 — Metrics & baselines

Capture the 'before' number, or the 'after' number proves nothing.

The most common measurement failure is starting the build before the baseline is recorded — after which every improvement claim is an estimate. Baseline capture belongs in discovery, where it is cheap.
All target values below are illustrative placeholders showing the shape of a threshold to agree, not real benchmarks.

7.1

Four metric families

Executive

Every capability reports in all four. Quality without adoption is a demo; adoption without quality is a liability; both without efficiency is a cost centre; all three without operational health is temporary.

Quality

Correct, grounded, appropriately cautious.

Adoption

Used, re-used, trusted over the old path.

Efficiency

Faster or cheaper, measured vs. baseline.

Operational

Healthy, affordable, supportable.

7.2Metric definitions and instrumentationImplementation

7.3

Baselines to capture during discovery

Each of these takes hours to capture before the build and cannot be reconstructed afterwards. This list is a discovery deliverable in its own right.

Process baselines

· Elapsed and hands-on time per target task, sampled over a full cycle.
· Manual touches, exception rate and rework rate for the same tasks.
· Current turnaround for a typical request, including wait time.
· Volume and seasonality, so improvement can be sized.

System and data baselines

· Existing automations, with failure history and owner.
· Semantic-layer documentation coverage: owner, description, lineage.
· Data-quality assertion pass rates, where they exist.
· Current reporting refresh latency and known reconciliation gaps.
· A verbatim sample of real questions asked today.

Assumption — baseline availability

We assume some of these baselines do not exist in measured form today and would need short sampling exercises rather than a system pull. Where a baseline genuinely cannot be captured, we would say so and report the improvement as unverified rather than present an estimate as a result.

7.4

Reporting cadence

Executive

Different audiences need different frequencies and different framing of the same underlying numbers.

Continuous

Engineering, automatic

· Traces, alerts, cost ceilings, assertion failures

Weekly

Delivery working session

· Evaluation trend
· Incidents
· Negative-feedback triage

Monthly

Capability owner + tech counterpart

· Quality by domain
· Adoption
· Efficiency vs. baseline
· Operational health
· Top user-reported gaps

Quarterly

Programme sponsors

· Programme view: what's delivering
· Marginal cost per vertical
· Risk-signal review
· What we'd design differently

Why — why per-domain, not aggregate

A single average quality score hides the only thing worth knowing: which domain is failing. Per-domain reporting makes remediation targetable.

Design decision — no vanity metrics

Total queries, documents ingested and flows built are activity counts, not outcomes. We would report them only as context alongside repeat-use, quality-by-domain and cycle time vs. baseline.