← @bgSponge

Formal Verification of LLM-Generated Code (FATTW 2026 slides)

Summary

Lucas C. Cordeiro argues that LLM-generated code must be verified by sound proof engines because models are fast but unsound, using the ESBMC model checker to bridge the gap between generation and correctness.

  1. LLMs fail formal verification.

    In a large-scale comparison, the best model passed only 15.48% of programs, while the average across nine models was 8.27%.

    ≈ 1 in 6 for the best model (15.48%); all nine: 1 in 12 (8.27%).

  2. Proof engines must confirm suggestions.

    The design rule is to let models suggest loop invariants or specifications, but use engines like ESBMC or Vampire to ensure they are sound.

    let the model suggest, but let the proof engine confirm.

    nothing the model says is trusted, only checked

  3. ibmc uses LLMs for invariants.

    The ibmc pattern uses LLMs to propose loop invariants which are then checked by Vampire for inductivity before ESBMC uses them for proof.

    Nothing the LLM proposes reaches the proof until Vampire has shown it is inductive.

  4. SpecVerify requires human semantics.

    When LLMs formalize requirements into assertions, they may guess bounds for vague terms like 'reasonable,' which can lead to incorrect specifications.

    a wrong guess can cost correctness, not just time: the specification needs a human.

  5. Concrete inputs drive fixes.

    Using ESBMC to generate counterexamples as executable test cases allowed for 18 upstream fixes across codebases like AWS SDK and vLLM.

    Maintainers fix what arrives with a concrete input.

Highlights

  • Here a wrong guess can cost correctness, not just time: the specification needs a human.

  • violation → a concrete input, replayed as a regression test

Read the original ↗Save to your Sponge

Summarized with Sponge. Quotes are the author’s own words.

More from @bg

All readings →