LawEngine

CiteBench by LawEngine

500 examples.
Inspect the test.

LawEngine built CiteBench—and the deterministic verification system that gets all 500 right. That is not a loophole. It is the point: legal citations should be checked against the source, not guessed.

Think that’s easy? Try the benchmark
Download the public packDownload the submission templateView the GitHub repository

LawEngine: 500/500. GPT-5.4-mini: 361/500 (72.2%). Same public pack. Same four-label task.LawEngine defines the reference answers. Its result measures consistency with that reference; the comparison is not a blinded head-to-head evaluation.

LawEngine deterministic verification
500/500100%
GPT-5.4-mini
361/50072.2%

Why this exists

A quote can turn on one word.

A legal quotation is only as reliable as the source behind it. A missing word can change the meaning even when the rest looks familiar.

CiteBench’s public examples make these source checks concrete: what was written, what was quoted, and why the difference matters.

Source text

“The court shall not dismiss the action.”

Submitted quotation

“The court shall … dismiss the action.”

Result shown for illustrationText does not match the cited source.The omission hides “not,” changing the statement.

Illustrative example, not a CiteBench answer. No real authority is depicted.

Source-grounded examples

Five ways simple matching fails

Legal quotation verification must recognize harmless differences in presentation while refusing changes to the words, numbers, or cited source.

  • Ellipses

    An ellipsis is not a free pass.

    A shortened quotation can match its source. An omission that hides “not” can change the result. The submitted quotation is the same in both; the source decides.

    Source
    “The court shall promptly and without delay dismiss the action.”
    Quoted
    “The court shall … dismiss the action.”
    Matches
    Source
    “The court shall not dismiss the action.”
    Quoted
    “The court shall … dismiss the action.”
    Rejected

  • One-word changes

    One word can change the quotation.

    “Shall,” “may,” and “should” cannot pass as interchangeable quoted wording. Nor can a changed word be quietly corrected into a match.

    Source
    “The operator shall inspect the vessel before each departure.”
    Quoted
    “The operator may inspect…”
    Rejected
    Quoted
    “The operator should inspect…”
    Rejected
    Quoted
    “…inspect the vessel after each departure.”
    Rejected
    Quoted
    “The operator shall inspekt…”
    Rejected

  • Formatting versus values

    Formatting can change. The value cannot.

    Line breaks, curly quotation marks and printed ligatures can be cleaned up. “1½” and “1 1/2” are the same value; “11/2” is a different one.

    Source
    “The permitted disc size is 1½ inches for the inspection record.”
    Quoted
    “…1 1/2 inches…”
    Matches
    Quoted
    “…11/2 inches…”
    Rejected

  • Quotes and citations inside quotes

    Which citation owns the quote?

    A quoted passage may refer to another law. That internal reference must not replace the authority being quoted, and a quotation inside the passage must not end it early.

    Cited
    42 U.S.C. § 2000e(i)
    Inside
    A quoted term: …the term “State” includes…
    Inside
    A reference to another law: 43 U.S.C. 1331 et seq.
    Found
    The whole passage, attributed to 42 U.S.C. § 2000e(i)
    Kept whole

  • Real words, wrong source

    Real words. Wrong source.

    Finding the quotation somewhere is not enough. It must match the authority being cited — and a shared section number does not make two authorities the same.

    Cited
    735 ILCS 5/2-619
    Found in
    735 ILCS 5/2-615
    Misattributed
    Cited
    42 U.S.C. § 1983
    Offered
    18 U.S.C. § 1983
    Rejected

Illustrative examples, not benchmark answers. Try all 500 cases on GitHub.

How to run it

Download, run locally, upload.

  1. Download

    Get the public pack and submission template.

  2. Run locally

    Use the public repository with your own system.

  3. Upload

    Send predictions for aggregate scoring.

The four-label contract

Classify the citation and quote, then score the prediction file.

  • Verified

    The quote appears in the cited authority.

    VERIFIED

  • Not found

    The quote is not in the cited authority.

    NOT_FOUND

  • Found elsewhere

    The quote is real, but belongs to another authority.

    MISATTRIBUTED

  • Citation unresolved

    The citation cannot be tied to an authority in the benchmark library.

    CITATION_UNRESOLVED

Upload and score

Score a CiteBench prediction file

Download the 500-row public pack, run your system locally, and upload a CSV containing id, predicted_status. LawEngine returns aggregate accuracy, F1 scores, label counts, and a confusion matrix while the grading key remains server-side.

Methodology: CiteBench scores the public pack’s four permitted outcomes against a server-side grading key and reports aggregate results only.

Questions about the comparison? Email hello@lawengine.ai.

CiteBench

Bring a harder case.

Inspect the public inputs, run your own system, and score its predictions. If your team has a source-checking problem of its own, talk to us.

Try CiteBenchTalk to us