I accidentally turned LLM memory into program analysis :: pwning.systems
Summary
Jordy Zomer wanted LLM agents to keep a live model of a vulnerability investigation, not a pile of retrieved snippets.
Memory is two problems.
Retrieval answers which past information is relevant; maintained state answers what is currently true given everything learned so far.
cosine vibe similarity and truth are not quite the same thing.
Sometimes we can just maintain the state.
Split fuzzy extraction from deterministic derivation.
The LLM turns source, debugger output, and notes into facts; Datalog derives, retracts, and explains the consequences.
once that information has been converted into structured facts, we no longer need the model to repeatedly determine all of its consequences.
Provenance makes forgetting mechanical.
Tracking derivation support lets dependent conclusions vanish when a fact is disproven, while independently supported conclusions remain.
I wanted it to stop forgetting why we believed things.
The front-end is the bottleneck.
Entity identity, date encoding, retrieval aliases, and reader instructions moved the scores more than a smarter evaluator; missing extraction cannot be derived away.
Competitive, not solved.
Lemmalog leads the cited field on LongMemEval knowledge update, trails PropMem overall, uses far less query context, and still needs a long-running vulnerability investigation as the real test.
Read the original ↗Save to your Sponge
Summarized with Sponge. Quotes are the author’s own words.