The setup
Company: Ledgerline Software, Inc. (fictional), a US vertical-SaaS company selling property management and rent-collection software to mid-size multifamily landlords. Advisor: Harrowgate Capital Advisors (fictional).
The scenario: Ledgerline's shareholders engaged Harrowgate to run a sell-side process. Harrowgate prepared the deal materials — a Confidential Information Memorandum and an accompanying financial model — from Ledgerline's own finance data, and used NarrEx to reconcile the CIM against the model before sending materials to a prospective buyer.
We deliberately chose a company and workflow that shared almost nothing with our first test — a different sector, a different set of metrics, and a different mode of the product.
We planted 10 deliberate inconsistencies on top of that baseline:
- a net revenue retention figure calculated on a different basis than the model supports
- a closing ARR figure that incorrectly folds in non-recurring revenue
- a churn figure that is numerically accurate but mislabeled as a different metric than it actually represents
- a growth rate calculated against an annualized run-rate rather than the model's actual year-over-year figure
- a customer acquisition payback period using a narrower cost base than the model's fully-loaded calculation
- an average revenue per customer figure whose denominator excludes smaller accounts
- a gross margin figure that excludes hosting and infrastructure costs the model correctly includes
- a recurring revenue percentage that treats fixed-term implementation revenue as if it were ongoing subscription revenue
- a lifetime-value-to-acquisition-cost ratio based on a more optimistic retention assumption than the model supports
- a headline "Rule of 40" score that inherits an earlier miscalculated growth figure
Each was designed to test a different part of the system rather than one weak spot.
Test methodology
1 fictional transaction · 1 confidential information memorandum · 1 financial model · 10 deliberately planted inconsistencies · multiple test iterations · independent answer key used for validation
What happened
The first run identified most of the 10 planted inconsistencies correctly. It also surfaced a substantial number of issues we hadn't planted at all, spanning matching logic, taxonomy coverage, and how figures get classified once compared.
We are publishing the failures alongside the successful detections. We did not suppress the false negatives or only report the final pass. The objective here isn't to demonstrate a perfect test. It's to expose failure modes.
One fix, while correcting a separate matching issue, introduced a new regression elsewhere. We caught it in the same test session and fixed it before it went anywhere near a real deal. We're mentioning it because we'd rather show that testing catches this kind of thing than imply every fix lands clean on the first attempt.
We also found and fixed a classification bug affecting how borderline discrepancies get categorized. It could plausibly have been sitting undetected in prior tests, since it only surfaces when a discrepancy happens to land in a specific range. It's fixed now, and we're treating it as a priority finding precisely because bugs like this can hide in plain sight.
We traced each issue, fixed it, and ran the test again, across multiple iterations.
Test outcome:
- 9 of 10 deliberately planted inconsistencies identified
- 14 previously unknown issues identified and fully remediated
- 1 planted inconsistency not caught, discussed below
The planted-inconsistency count and the issue count are separate: a single underlying issue could affect the detection of more than one planted inconsistency, and not every issue we found traces back to something we deliberately planted.
The one planted inconsistency we didn't catch is worth explaining rather than glossing over. The CIM stated a churn figure that was, numerically, a real number pulled correctly from the model. It just wasn't the number the narrative implied it was. NarrEx matched the figure correctly to its source. It didn't catch that the sentence around that figure was using it to describe a different, more favorable claim than what the number actually supports. That's a real, current limitation: NarrEx checks whether a number is supported by the model, not whether the sentence around that number is using it honestly. That's a harder problem, and one we're scoping as deliberate future work rather than something we quietly patched around.
Testing this deal also surfaced a structural gap in how our extraction taxonomy handles different kinds of businesses, enough that we rebuilt part of it in the same session. We'll cover that separately in its own piece.
The honest part
A synthetic test like this is not the same as a messy real-world deal. We know exactly what's wrong because we put it there. A real engagement doesn't come with an answer key. We're not claiming this proves NarrEx works on real transactions. We're showing how rigorously we're trying to find out where it doesn't.
This is test 2. We'll keep running these across different sectors and transaction types, and publish what we find, including when the answer is "this didn't work."