InvoiceInspect is tested with an evaluation harness: a built-in tool that runs the checker over invoices whose correct values are known, scores every result, and writes a report. These pages publish the harness, the test invoices and every report, including the weak results.
Every test invoice is a PDF with a JSON answer key: each field, each line item and the mistakes deliberately planted in it.
The harness reads each PDF with the real extractor, then runs the same verification rules as the live app.
Fields are compared with the key. A planted mistake counts as found when the tool reports it; anything else it reports counts as a false alarm.
Each report scores the same invoices with the same commands, so a later report shows exactly what improved and what did not.
How the harness works from end to end: the flow, the five extractors it can score, how to read the results and its limits.
The research question behind InvoiceInspect, how the tool is built to answer it, what the measurements show and where it falls short.
The tool as it stands, measured on two sets of invoices, with the issues found. Later reports score the same invoices with the same commands, so they can be compared with this one.
| Measure (rules extractor) | Corpus A: independent invoices | Corpus B: built-in generator |
|---|---|---|
| Field accuracy | 84.5% | 99.9% |
| Planted errors found | 25 of 32 | 20 of 20 |
| False alarms | 30 | 0 |
| Rule engine on perfect input (oracle) | 32 of 32, 0 false alarms | 20 of 20, 0 false alarms |
Raw results
The six reading problems from Report 1 fixed in the extractor, then the same invoices scored again with the same commands. Includes the problems this run found. The independent invoices were used to tune the extractor, so their new score reads higher than a fresh set would give.
| Measure (rules extractor) | Corpus A: independent invoices | Corpus B: built-in generator |
|---|---|---|
| Field accuracy | 98.1% | 99.9% |
| Planted errors found | 31 of 32 | 20 of 20 |
| False alarms | 6 | 0 |
| Correct invoices with no findings | 12 of 15 | 20 of 20 |
Raw results
The checker scored on 40 new multi-page invoices it has never seen, with the problems found. Lower accuracy is expected on invoices like these; the results show where the extractor needs further improvement.
| Measure (rules extractor) | Corpus C: multi-page invoices | Corpus A: single-page (Report 2) |
|---|---|---|
| Field accuracy | 72.2% | 98.1% |
| Planted errors found | 13 of 20 | 31 of 32 |
| False alarms | 50 | 6 |
| Rule engine on perfect input (oracle) | 20 of 20, 0 false alarms | 32 of 32, 0 false alarms |
Raw results
The problems found in Report 3, fixed in the extractor and scored again on the same 40 multi-page invoices, with the older sets scored again to check that nothing got worse. The fixes were made on these invoices, so the new score shows the problems are gone, not how other invoices read.
| Measure (rules extractor) | Corpus C after fixes | Corpus C before (Report 3) |
|---|---|---|
| Field accuracy | 99.98% | 72.2% |
| Planted errors found | 20 of 20 | 13 of 20 |
| False alarms | 0 | 50 |
| Correct invoices with no findings | 20 of 20 | 8 of 20 |
Raw results
A new set of 30 invoices that neither reader has seen, scored with the rules extractor and with a new vision extractor in which an AI model reads a picture of each page. Includes the same-invoice comparison on the older sets and the problems found. The improved instruction was written after seeing mistakes on corpus D, so its corpus D score is not a fresh measure.
| Measure | Rules extractor | Vision extractor |
|---|---|---|
| Field accuracy | 62.0% | 99.4% |
| Planted errors found | 15 of 26 | 26 of 26 |
| False alarms | 30 | 0 |
| Correct invoices with no findings | 5 of 15 | 15 of 15 |
Raw results
Control: oracle (returns the answer key, must score 100%)
Rules extractor
Vision, first instruction (plain)
Vision, improved instruction
Open any invoice to see exactly what the tool was given, and its answer key to see what the tool should find. All invoices are fictitious test data: company names, addresses and tax numbers are invented.
View all test invoices on one page
| Invoice | Page structure | Currency | What the tool should find | Answer key |
|---|---|---|---|---|
| invoice-001 | letter | EUR | Correct invoice | JSON |
| invoice-002 | banner | AUD | tax mismatch | JSON |
| invoice-003 | sidebar | GBP | Correct invoice | JSON |
| invoice-004 | receipt | USD | missing vat id; grand total mismatch; line total mismatch (line 2) | JSON |
| invoice-005 | taxsummary | CHF | Correct invoice | JSON |
| invoice-006 | qtyfirst | EUR | duplicate charge (line 6); line total mismatch (line 4) | JSON |
| invoice-007 | letterhead | INR | Correct invoice | JSON |
| invoice-008 | compact | EUR | duplicate charge (line 6); missing vat id; grand total mismatch | JSON |
| invoice-009 | cards | AUD | Correct invoice | JSON |
| invoice-010 | multipage | EUR | duplicate charge (line 17) | JSON |
| invoice-011 | letter | USD | subtotal mismatch; tax mismatch; line total mismatch (line 4); line total mismatch (line 5) | JSON |
| invoice-012 | banner | GBP | Correct invoice | JSON |
| invoice-013 | sidebar | USD | subtotal mismatch | JSON |
| invoice-014 | receipt | PKR | Correct invoice | JSON |
| invoice-015 | taxsummary | EUR | subtotal mismatch; line total mismatch (line 5); line total mismatch (line 2) | JSON |
| invoice-016 | qtyfirst | CAD | Correct invoice | JSON |
| invoice-017 | letterhead | EUR | grand total mismatch; tax mismatch; subtotal mismatch | JSON |
| invoice-018 | compact | AUD | Correct invoice | JSON |
| invoice-019 | cards | GBP | duplicate charge (line 4); subtotal mismatch | JSON |
| invoice-020 | multipage | USD | Correct invoice | JSON |
| invoice-021 | letter | CHF | Correct invoice | JSON |
| invoice-022 | banner | EUR | tax mismatch; subtotal mismatch; line total mismatch (line 4) | JSON |
| invoice-023 | sidebar | INR | Correct invoice | JSON |
| invoice-024 | receipt | EUR | duplicate charge (line 7) | JSON |
| invoice-025 | taxsummary | AUD | Correct invoice | JSON |
| invoice-026 | qtyfirst | EUR | line total mismatch (line 2) | JSON |
| invoice-027 | letterhead | USD | Correct invoice | JSON |
| invoice-028 | compact | GBP | tax mismatch | JSON |
| invoice-029 | cards | USD | Correct invoice | JSON |
| invoice-030 | multipage | PKR | duplicate charge (line 17); line total mismatch (line 4); subtotal mismatch | JSON |
| Invoice | Design | Currency | What the tool should find | Answer key |
|---|---|---|---|---|
| invoice-001 | corporate-blue | EUR | Correct invoice | JSON |
| invoice-002 | serif-classic | GBP | Correct invoice | JSON |
| invoice-003 | mono-receipt | EUR | Correct invoice | JSON |
| invoice-004 | modern-green | USD | Correct invoice | JSON |
| invoice-005 | bordered-grid | EUR | Correct invoice | JSON |
| invoice-006 | letterhead-logo | EUR | Correct invoice | JSON |
| invoice-007 | compact-orange | EUR | Correct invoice | JSON |
| invoice-008 | large-print | EUR | Correct invoice | JSON |
| invoice-009 | slate-tall-header | EUR | Correct invoice | JSON |
| invoice-010 | teal-minimal | EUR | Correct invoice | JSON |
| invoice-011 | corporate-blue | USD | grand total mismatch | JSON |
| invoice-012 | serif-classic | EUR | grand total mismatch | JSON |
| invoice-013 | mono-receipt | GBP | missing vat id | JSON |
| invoice-014 | modern-green | EUR | tax mismatch | JSON |
| invoice-015 | bordered-grid | EUR | missing vat id | JSON |
| invoice-016 | letterhead-logo | EUR | line total mismatch (line 2) | JSON |
| invoice-017 | compact-orange | USD | missing vat id | JSON |
| invoice-018 | large-print | EUR | subtotal mismatch | JSON |
| invoice-019 | slate-tall-header | GBP | subtotal mismatch | JSON |
| invoice-020 | teal-minimal | EUR | line total mismatch (line 2) | JSON |
| invoice-021 | corporate-blue | GBP | Correct invoice | JSON |
| invoice-022 | serif-classic | EUR | Correct invoice | JSON |
| invoice-023 | mono-receipt | EUR | Correct invoice | JSON |
| invoice-024 | modern-green | EUR | Correct invoice | JSON |
| invoice-025 | bordered-grid | GBP | Correct invoice | JSON |
| invoice-026 | letterhead-logo | GBP | Correct invoice | JSON |
| invoice-027 | compact-orange | EUR | Correct invoice | JSON |
| invoice-028 | large-print | EUR | Correct invoice | JSON |
| invoice-029 | slate-tall-header | USD | Correct invoice | JSON |
| invoice-030 | teal-minimal | USD | Correct invoice | JSON |
| invoice-031 | corporate-blue | GBP | subtotal mismatch | JSON |
| invoice-032 | serif-classic | EUR | subtotal mismatch | JSON |
| invoice-033 | mono-receipt | EUR | tax mismatch | JSON |
| invoice-034 | modern-green | GBP | grand total mismatch | JSON |
| invoice-035 | bordered-grid | EUR | grand total mismatch | JSON |
| invoice-036 | letterhead-logo | EUR | subtotal mismatch | JSON |
| invoice-037 | compact-orange | USD | grand total mismatch | JSON |
| invoice-038 | large-print | EUR | missing vat id | JSON |
| invoice-039 | slate-tall-header | EUR | line total mismatch (line 3) | JSON |
| invoice-040 | teal-minimal | GBP | line total mismatch (line 4) | JSON |
| Invoice | Page structure | Currency | What the tool should find | Answer key |
|---|---|---|---|---|
| invoice-001 | inline | CHF | Correct invoice | JSON |
| invoice-002 | stacked | CHF | Correct invoice | JSON |
| invoice-003 | wide | CAD | Correct invoice | JSON |
| invoice-004 | inline | SEK | Correct invoice | JSON |
| invoice-005 | wide | SEK | Correct invoice | JSON |
| invoice-006 | inline | EUR | Correct invoice | JSON |
| invoice-007 | wide | SEK | Correct invoice | JSON |
| invoice-008 | coded | CHF | Correct invoice | JSON |
| invoice-009 | stacked | CAD | Correct invoice | JSON |
| invoice-010 | stacked | AUD | Correct invoice | JSON |
| invoice-011 | inline | CHF | subtotal mismatch | JSON |
| invoice-012 | coded | CHF | line total mismatch (line 14) | JSON |
| invoice-013 | inline | CAD | subtotal mismatch | JSON |
| invoice-014 | coded | EUR | line total mismatch (line 49) | JSON |
| invoice-015 | inline | SEK | line total mismatch (line 58) | JSON |
| invoice-016 | classic | SEK | line total mismatch (line 40) | JSON |
| invoice-017 | coded | CHF | line total mismatch (line 46) | JSON |
| invoice-018 | inline | CAD | grand total mismatch | JSON |
| invoice-019 | wide | CHF | grand total mismatch | JSON |
| invoice-020 | wide | EUR | line total mismatch (line 73) | JSON |
| invoice-021 | wide | CHF | Correct invoice | JSON |
| invoice-022 | stacked | AUD | Correct invoice | JSON |
| invoice-023 | wide | SEK | Correct invoice | JSON |
| invoice-024 | classic | AUD | Correct invoice | JSON |
| invoice-025 | inline | EUR | Correct invoice | JSON |
| invoice-026 | classic | AUD | Correct invoice | JSON |
| invoice-027 | stacked | CHF | Correct invoice | JSON |
| invoice-028 | wide | AUD | Correct invoice | JSON |
| invoice-029 | stacked | EUR | Correct invoice | JSON |
| invoice-030 | wide | EUR | Correct invoice | JSON |
| invoice-031 | classic | CAD | subtotal mismatch | JSON |
| invoice-032 | inline | EUR | grand total mismatch | JSON |
| invoice-033 | coded | EUR | line total mismatch (line 15) | JSON |
| invoice-034 | wide | AUD | grand total mismatch | JSON |
| invoice-035 | stacked | CHF | tax mismatch | JSON |
| invoice-036 | inline | EUR | grand total mismatch | JSON |
| invoice-037 | classic | AUD | line total mismatch (line 3) | JSON |
| invoice-038 | classic | CAD | subtotal mismatch | JSON |
| invoice-039 | wide | CAD | missing vat id | JSON |
| invoice-040 | coded | EUR | line total mismatch (line 23) | JSON |
| Invoice | Document kind | Currency | What the tool should find | Answer key |
|---|---|---|---|---|
| invoice-001 | receipt | SEK | Correct invoice | JSON |
| invoice-002 | summaryfirst | BRL | subtotal mismatch | JSON |
| invoice-003 | prose | USD | Correct invoice | JSON |
| invoice-004 | qtyfirst | NZD | line total mismatch (line 4) | JSON |
| invoice-005 | sidebar | MXN | Correct invoice | JSON |
| invoice-006 | grouped | EUR | tax mismatch, grand total mismatch | JSON |
| invoice-007 | taxcolumn | SGD | Correct invoice | JSON |
| invoice-008 | cards | EUR | duplicate charge (line 4) | JSON |
| invoice-009 | amountfirst | EUR | Correct invoice | JSON |
| invoice-010 | ZAR | grand total mismatch | JSON | |
| invoice-011 | receipt | GBP | subtotal mismatch, line total mismatch (line 1) | JSON |
| invoice-012 | summaryfirst | EUR | Correct invoice | JSON |
| invoice-013 | prose | AED | line total mismatch (line 7), tax mismatch, subtotal mismatch | JSON |
| invoice-014 | qtyfirst | CAD | Correct invoice | JSON |
| invoice-015 | sidebar | SEK | line total mismatch (line 3), line total mismatch (line 4), subtotal mismatch | JSON |
| invoice-016 | grouped | BRL | Correct invoice | JSON |
| invoice-017 | taxcolumn | USD | subtotal mismatch, tax mismatch | JSON |
| invoice-018 | cards | NZD | Correct invoice | JSON |
| invoice-019 | amountfirst | MXN | line total mismatch (line 3), line total mismatch (line 6), missing vat id, grand total mismatch | JSON |
| invoice-020 | EUR | Correct invoice | JSON | |
| invoice-021 | receipt | SGD | Correct invoice | JSON |
| invoice-022 | summaryfirst | EUR | line total mismatch (line 2) | JSON |
| invoice-023 | prose | EUR | Correct invoice | JSON |
| invoice-024 | qtyfirst | ZAR | subtotal mismatch | JSON |
| invoice-025 | sidebar | GBP | Correct invoice | JSON |
| invoice-026 | grouped | EUR | line total mismatch (line 1) | JSON |
| invoice-027 | taxcolumn | AED | Correct invoice | JSON |
| invoice-028 | cards | CAD | subtotal mismatch | JSON |
| invoice-029 | amountfirst | SEK | Correct invoice | JSON |
| invoice-030 | BRL | line total mismatch (line 5), line total mismatch (line 6) | JSON |
