[go: up one dir, main page]

How the numbers are proven

Shared test cases
700
Vector files
37
Calculators
26

How FactLot's numbers are checked, what each check does and does not show, how to repeat every check yourself, and what is not proven. A check that is not listed here has not been done.

Trust pages: Status · Security · Service levels · Assurance. Written 7 November 2026, for release 0.5.0 and the changes the changelog lists as unreleased. Sizes marked "counted at build" are counted from the repository when this site is built; the rest are measured figures with their date.

The checks

Shared test vectors

What it shows. Fixed inputs with fixed expected outputs, in language-neutral JSON, that every implementation must reproduce. Expected values are computed by hand or with exact decimal arithmetic, not by running FactLot (a project rule; spec/README.md).

Size. 700 cases in 37 files, one file for each of the 26 calculators (counted at build).

Repeat it. npm test and npm run test:python.

Two implementations

What it shows. The calculators, IDs and tools exist twice: TypeScript (packages/) and Python (python/src/factlot). Any change goes into both, and both must pass the same vectors.

Size. TypeScript: 328 tests. Python: 785 tests, 2 skipped. Both measured 7 November 2026.

Repeat it. npm test and npm run test:python.

Differential test

What it shows. Random inputs are generated, run through both implementations, and every field of every result is compared. A single difference fails it. It finds disagreements that no hand-written vector anticipated.

Size. 9,808 cases from seed 1 and count 1500, with 0 differences (7 November 2026). CI runs seed 1 and count 3000 on every change.

Repeat it. npm run differential -- 1 1500.

Excel reconciliation

What it shows. Workbooks that rebuild a calculator's answer in live Excel formulas beside FactLot's numbers, with a Difference column. A workbook is not written unless every difference is zero, and a test fails if a committed file is stale. Inputs are synthetic.

Size. 8 workbooks, 341 rows reconciled, all differences 0 (counted at build).

Repeat it. python3 examples/reconciliation/build.py --check, or open a workbook in Excel.

Live suite on the hosted API

What it shows. Replays the vectors against api.factlot.com, calls every MCP tool and checks it equals the REST answer, and does real registry lookups. It shows that the deployed service gives the same answers as the repository.

Size. 4,635 checks passed, 0 failed, in the last full run recorded here (8 November 2026). Run by the maintainer after a deploy, not on a schedule. Shipped after that run: paid gating on the free tier, the optional free key, the usage counters and the nine-region registry release and the registry releases after it.

Repeat it. npm run test:hosted. It is paced to stay under the free 60 requests a minute, and it uses your address's free registry quota.

Continuous integration

What it shows. Lint, formatting, types, the TypeScript and Python tests (which include the vectors and the workbook check), the differential test, the site build, the MCP end-to-end test, OpenAPI validation and a check that generated schemas are current. Required on main. Not in CI: the live suite, and opening the workbooks in Excel.

Size. Every run is public on GitHub.

Bench

What it shows. 14 realistic commercial real estate agent tasks, with answers computed independently of FactLot, each run once with the model alone and once with FactLot's tools; a task passes only if every answer field is right. The repository's offline tests check that each answer is reachable through FactLot's tools and that scoring is strict.

Size. No accuracy number is published.

Repeat it. npm run test:bench (offline tests only).

To start from nothing: git clone https://github.com/9mp96ygnbk-alt/factlot.git && cd factlot && npm ci && npm run build needs Node.js 22.6 or newer and Python 3.10 or newer, then run the commands in the last column. The tests and the differential suite run locally; only the live suite calls the hosted service.

Rules checked against statute text

A rules pack turns law into code. It is called draft until a maintainer has read each cited section in its official source and recorded who checked it and when, and only then does it become a tool. Every result carries the rule's verification status and date.

  • Texas property tax protest deadline (rules.tx.property_tax_protest_deadline@2): verified 2026-11-02 against the official text, by "FactLot maintainers" (the rule records no individual's name). Law: Tex. Tax Code §41.44(a)(1); Tex. Tax Code §1.06; Tex. Gov't Code §662.003.
  • 4 other packs: draft, not verified. NYC Local Law 97, Washington Clean Buildings (WAC 194-50), Boston BERDO 2.0 and Texas protest evidence dates. They are not tools. They run only on an explicit opt-in and put a warning in every result that the pack is a draft not verified against the official text. See rules packs.

The Texas deadline rule is also tested against a full holiday calendar, a differential run over random dates from 2000 to 2100, and an exhaustive check of every notice date in that range (the deadline is always a business day, never before May 15, and the conservative file-by date is never later than the deadline). It is a date calculation, not legal or tax advice.

How facts are cited

  • No uncited facts. A fact must name its source (document, page, section, verbatim quote), except a computed fact, which names its formula and inputs. The schema rejects the rest. Fact model.
  • Every fact says how it was made. The method is one of human_verified, public_record, computed, extracted, reported or inferred, so an agent can tell which ones to send to a person.
  • Rights travel with the fact. Each fact carries the source's licence and its AI-use and redistribution status.
  • Registry facts cite the public file. The citation holds the publisher's URL, the file's content hash and the record locator, so you can fetch the same file from the publisher and check it.
  • Document checks are textual. The document tools confirm that the quote an agent supplied is on the page it names and that the quote states the value. They do not find terms the agent missed, and they do not judge what a clause means.

Formula versions

Every result names its formula, for example calc.net_effective_rent@1. A change that could alter any output increases the version and is listed under "Changed" in the changelog; the rules are in docs/release-process.md. This is used: 10 of the 63 REST tools are on a version above 1 (calc.compare_proposals@2, calc.cam_reconciliation@2, calc.debt_sizing@3, calc.lease_remeasurement_asc842@2, calc.irr@2, calc.lease_vs_own@2, calc.renew_vs_relocate@2, calc.dcf_pro_forma@2, calc.exchange_1031@2, rules.tx.property_tax_protest_deadline@2). See service levels for what that means for the hosted API.

What is not proven

  • ARGUS and Yardi parity is untested. No ARGUS or Yardi export has been run through FactLot, and FactLot does not claim to match either product. The tie-out tools exist and were tested only on synthetic files built from FactLot's own output with one convention changed on purpose. Details.
  • OCR accuracy is not measured. Scanned pages can be read through a pluggable OCR backend on your own machine; facts from OCR pages are flagged so a person can check them, but no accuracy figure exists.
  • Lease extraction accuracy is not measured. There is no hand-labelled set of real leases, so no accuracy number is published (ingest).
  • No bench result is published. The bench exists and its tests pass, but no accuracy figure from running it is published, so this site makes no claim about how often an agent with or without FactLot gets the right number.
  • The Excel check was done in Excel once. On 3 November 2026 the seven workbooks then published were opened in Microsoft Excel for Mac and recalculated: every difference was 0. The Texas unequal-appraisal workbook was added later and is checked by the repository's own formula evaluator, which is what runs on every change, not by Excel.
  • Expected values are independent by rule, not by audit. The project requires vectors to be computed outside FactLot's code, and reference scripts for some are in spec/vectors/calc/reference. No outside party has reviewed the calculators.
  • A calculator is conventions, not judgment. Each calculator follows stated conventions (how rent is prorated, when cash flows are discounted, how rounding works), shown in its steps. Whether they fit your lease or your lender is for you to decide; a result that agrees with another model on the same inputs does not show that either is right for the deal.
  • Registry data are only as good as the public record. Public records are incomplete, out of date or wrong at times. Each fact shows its source and date. Counties marked "audited" on the coverage map had the registry compared with the publisher's file; the others have not been.
  • Only one rule is verified. The other rules packs are drafts. Nothing here is legal, tax, appraisal or accounting advice.

Report a wrong number

A wrong result matters more than any other bug. To report one, email contact@factlot.com with "WRONG NUMBER" at the start of the subject, or open a bug report if the inputs can be public. Please include:

  • the tool, and the formula ID and version it returned;
  • the exact input (use invented figures if the real ones are confidential);
  • the result you got, and the result you expected;
  • how you know: a statute or contract clause, a hand calculation, or another system's figure with its settings.

A confirmed error is fixed in both implementations, gets a new test vector, increases the formula's version, and is recorded in the changelog. A wrong result that comes with no warning is in scope of the security policy too, and you can report it privately the same way. Paid plans' support targets are on service levels (draft).