DocNexus.ai · case study
FinSignal
Forecasting a drug’s quarterly revenue from claims before the company reports it, then letting the SEC score the forecast.
- 9.42%
- median error, October model
- 11.32%
- August hybrid, same cells
- 228
- brands anchored to SEC revenue
- 0 of 271
- look-ahead leaks
A point-in-time forecast of a branded drug’s quarterly net revenue, produced at each drug’s own tradable window: the last moment before its issuer files, a median of 29 days after the quarter closes. Because the filing eventually publishes the answer, every forecast is scored after the fact. August shipped a deterministic hybrid; October replaced it with a learned model, tiered by its own measured error.
Methods and relative results only. Client figures, internal names and scale are not published.
October: a learned model, held to a frozen protocol
The August hybrid became the baseline to beat. A forecasting protocol was frozen before any candidate was fitted: weekly origins, three horizons reported separately, only disclosures public before each origin, expanding chronological folds, and a promotion bar a model must clear before it may be called qualified.
Nine challengers were compared on identical cells. Shallow gradient-boosted trees predicting growth over the last reported revenue came out ahead: 9.42% median error against 11.32% for the August hybrid on the same 172 drug-quarters, lower on 68% of them. Chronos-2 was benchmarked too and did not beat a seasonal baseline.
An SEC sweep took brands with reviewed US revenue history from 29 to 228, at 11.0% pooled median error. Every drug is then tiered by its own backtest: Forecast where its median error is at most 15% over at least four quarters (92 brands), Indicative where the error or a calibrated band is wide but bounded (527 drugs), and claims data only, with no dollar figure, everywhere else.
Drugs with no revenue history get a separate model, at 47.3% median error, whose bands were recalibrated on later quarters until they covered about ~80% of outcomes. Calibrating on random folds had looked fine and under-covered in use.
The framing that matters (August)
The quarter is already over when the forecast is made. What is uncertain is not demand (the prescriptions happened) but how much of that demand has arrived in the warehouse yet, and what it converts to in dollars.
So the shipped forecast never estimates how complete the quarter is. It reads a ratio of two partial sums taken at the same age: the target quarter as of today, divided by the prior quarter at the same number of ingest weeks after its own close. If both sides are equally incomplete, the incompleteness cancels. The hardest quantity in the problem is designed out instead of modelled.
Making the cancellation actually hold
The two sides are matched on ingest buckets rather than calendar days, because calendar matching left them a whole ingest week apart on 34.3% of rows. Each side is normalised per service week, because thirteen- and fourteen-week quarters inflate an unnormalised ratio by ~7.7% on the calendar alone.
When a vendor backfill breaks the equal-maturity premise, a vintage-weighted blend with zero fitted parameters takes over. It measured 14.5% pooled median error against 19.8% and 18.5% for either leg alone, and beat a hard switch on the same detector (18.2%), because weighting degrades gracefully where switching fails hard.
Five layers of arithmetic, two fitted constants
Anchor on the prior quarter’s reported revenue, refused unless that filing was public at the as-of date. Tilt it by the claims growth ratio. Guard it: if the growth breaches the brand’s own historical swing ceiling, treat it as no change. Route between the claims view and a run-rate view, trusting claims less the larger the move they claim. De-bias using peers in the same quarter that have already reported.
The router finding is the one worth leading with. The original intent was to lean on claims precisely for fast-moving drugs. Measured, that direction was the worst rule tested, worse than the baseline it was meant to improve. A large claimed move is mostly ratio noise, not demand.
Point-in-time integrity, proven adversarially
Two independent checks. A mutation check corrupts every input dated after a cutoff and requires bit-identical forecasts. An equivalence check forecasts every row twice, once against the full filings table and once against a table reduced to what was public at that row’s as-of date, and requires identity. Result: 0 of 271 rows moved.
The gate can also fail. With the filing gate disabled it reports leaks, and with three anchors’ filing dates pushed past their dependents’ as-of dates it flags exactly those three rows. A passing audit means nothing until it can fail.
Three real leaks were found and closed, each of which had made the product look better than it was: a baseline built from matured data, an anchor filed 199 days after its forecast date, and a completeness curve built on a looser boundary than the data it corrected. Closing the first moved the headline the wrong way, from 10.5% to 11.3%, and that is the number that got published.
The bug that looked like model failure
One filer reports on a fifty-two/fifty-three-week fiscal calendar. Two rules broke at once: quarter position was read from the month of the period end, and cumulative-versus-discrete was decided by whether the period started on the first of January. A running year-to-date total was stored as four discrete quarters, so the model was being scored against the wrong target and its errors on that drug looked catastrophic.
The forecasts were fine; the target was wrong. Both rules now derive from the reported period itself, and a structural guard rejects any row whose filing date precedes the end of the quarter it reports. Fixing it moved the pooled median from 11.21% to 10.71%.
Decisions
- In August, no learned model: 271 quarterly observations could not support one, and a gradient-boosted model showed a 57× gap between training and out-of-sample error. The October model only became viable once the SEC sweep multiplied the anchored history, and it is shallow on purpose.
- The worker that refreshes the weekly tracker never bills by itself: every warehouse query is dry-run, partition-filtered, capped, and needs an approval pinned to its exact SQL.
- Both fitted constants (0.24 and 0.3) were set below their in-sample optimum under leave-one-quarter-out.
- The knowledge date of every record is its warehouse ingestion time, not its transaction date, because the transaction date flatters a backtest.
- The actuarial completion machinery (chain-ladder) is a fallback, used on 5 of 273 rows, not the main path.
Where it falls short
- No model is qualified yet under the team’s own protocol: the boosted model was chosen on the same six quarters it is scored on, and has five independent quarters against the eight required. The product labels its estimates accordingly.
- No beat/miss verdict is published. A company-level consensus multiplied by a product’s revenue share implied a decline for 42 of 44 tracked products while the claims showed growth; the “surprise” was an artefact of the consensus construction. A direct drug-level source scores only ~3% of the backtest, so the verdict stays withdrawn, not caveated.
- The August figures (10.71% median error on 271 drug-quarters) come from an earlier build and are not reproduced by current inputs; they stay as the historical record. On current inputs the August hybrid is statistically indistinguishable from repeating the last reported quarter.