DocNexus.ai · case study
Answer-consistency audit → knowledge map
Asked the same question five times, the agent gave five different answers. Finding out why.
A read-only audit of production agent traces, and the design that came out of it: a way to make an analytics agent give the same answer to the same question, however it is worded.
Methods and relative results only. Client figures, internal names and scale are not published.
What the traces showed
Definitions were being re-decided on every run. When a user gave no time period, the agent chose one anyway, and in 99% of affected queries the invented period was already in the task before the SQL writer saw it. 21 of 28 entities were queried with more than one code set, and “by state” was split between provider location and patient location. The counting formula itself was stable (98%); the definitions around it were not.
The design
A model at temperature zero turns each question into a canonical frame (subjects, metric, dimensions, filters, shape), chosen from a fixed menu rather than written free-form. The frame is normalised and hashed. A per-organisation graph of trusted definitions decides what happens next: replay verified SQL for a question seen before, bind fixed definitions when every part is known, find the shortest trusted join path when a join is needed, and open a candidate node for anything unknown.
A definition becomes trusted only after repeated successful use across separate chats, explicit approval, or positive feedback, and a conflict never overwrites a trusted definition. Rollout runs off → shadow → on.
Where it falls short
- This is a design with an acceptance bar (≥95% frame stability, zero false merges), not a measured result yet.