Skip to main content
DataCrafters synthetic data · demonstration

How we know

What we can show today about whether the checks measure correctly and repeatably, how to reproduce each item, and what is not yet proven. Synthetic patients only.

Shown today (counted from this server's stored data)

  1. The same check gives the same answer.

    156 stored check runs: 52 kidney calculations, 52 side-panel checks, 52 wider-library checks (one of each per note, plus re-checks after edits).

    None has been re-run on this server yet. Press the button to re-run all 156 now and see the count.

    See every check

  2. Cut-offs are tested on values just either side.

    44 of 44 boundary tests pass for the kidney calculation rules; a check refuses to run if one fails.

    5 of 5 cut-off tests pass for the heart-failure type definition in use (also tested by the build's test suite: uv run pytest apps/exactdx/tests/test_front.py).

    The 5 heart-failure cut-off tests
    • ejection fraction 40%: expected reduced, got reduced
    • ejection fraction 41%: expected mildly reduced, got mildly reduced
    • ejection fraction 49%: expected mildly reduced, got mildly reduced
    • ejection fraction 50%: expected preserved, got preserved
    • ejection fraction 40.5%: expected mildly reduced, got mildly reduced

    The malnutrition levels have no cut-off tests on this page; they are exercised by the test suite on the seeded cases only.

    See the rule cards and their tests

  3. The AI reader cannot change a finding.

    Findings come only from structured values and rules. The AI reader only reads text; a second, non-AI reader must agree and every quote is checked character by character, otherwise the card says to read that part yourself.

    In this server's record, counted from the reader stored on each card:

    • 6 cards were read by the hosted model (openrouter · openai/gpt-6.1-terra; openrouter) and the rule reader: they agreed on 6, disagreed on 0, and the model gave no answer on 0.
    • 34 cards were read by the offline stand-in (used when the demonstration data is built, and when no language model is configured). That agreement holds by construction and is not evidence about a language model.
    • 0 cards recorded a reader fault.

    These are counts on synthetic notes, not a measured accuracy.

  4. What happens when the AI reader misbehaves.

    The automated test suite replaces the reader with 14 deliberately misbehaving stand-ins, each run on all 44 synthetic patients. Required result every time: the values side is byte-identical to running with no AI reader, nothing the reader said reaches the clinician unverified, the fault is recorded, and the card can only change to “read this part yourself”.

    The 14 ways the stand-in reader misbehaves
    • It quotes text that is not in the note.
    • It quotes a real span but asserts the wrong stage.
    • It flips affirmed/negated.
    • It returns text about a different patient.
    • It obeys an instruction planted in the note.
    • It returns an empty list (detectable only where the note mentions kidney function).
    • It returns an empty body.
    • It returns truncated output.
    • It returns prose instead of JSON.
    • It returns JSON that violates the schema.
    • It times out.
    • It raises an unexpected exception.
    • It gives a different answer on every call.
    • It alters numbers in its quote and stage.

    Run by the test suite on every build, not on this page. To reproduce: uv run pytest apps/exactdx/tests/test_model_faults.py. The offline stand-in never disagrees with the rule reader on the seeded notes, so the “readers disagree” card appears on the seeded data only in these tests.

  5. Everything offered is on record.

    46 side-panel cards on record: 40 shown to a clinician as cards, 6 computed but not shown (staff setting or suspended rule); each stored with its exact evidence, options and definition version. 0 responses recorded. Audit trail: 5594 entries, chain verified ✓.

    Offers and responses · Audit trail

Not yet proven

  • No real patients: every chart is synthetic.
  • Not clinically validated. No accuracy has been measured against real coders or clinicians; the figures on Results are simulated from synthetic labels.
  • The definitions are drafts pending clinician and coder approval; the malnutrition source has not yet been opened by the team and is cited from its title.
  • No real-world accuracy of the AI reader has been measured. The public demo uses a hosted OpenAI model through OpenRouter, not Azure; the Azure adapter is built but has not been run in the event tenant.
  • Wrong input data replays perfectly: a wrong value in the record gives a wrong finding that re-runs identically.
  • Both readers can share a mistake; the tests can force disagreement, not rule out a shared misreading.
  • A rule that encodes a definition wrongly passes its own tests until a reviewer catches it.
  • The malnutrition rule has no cut-off tests shown on this page, and whether showing met / not met beside each option is acceptable as non-leading has not been reviewed by a coding reviewer.

How the parts fit together: How it decides. Simulated agreement figures: Results.

Demo clock is fixed; entries are ordered by audit entry number.