Hardening Claude Code’s Memory: Mistake Ledgers, Per-Prompt Context, and Verifying the Dreamer Itself

This is the sixth post in my series on giving Claude Code a long-term memory in Obsidian. The earlier posts covered session archiving and context injection, sub-second vault search, self-curating memory, out-of-band “dreaming” and teaching the agent real Obsidian syntax.

In Hardening Local AI Agents: Procedural Skills, Mistake Ledgers, and Real-Time Verification I described a hardening layer for my Antigravity agent: a ledger of past mistakes, a context packet built for each task, live probes, and a recall benchmark. This post ports that layer to Claude Code.

A straight port would make a short changelog. What makes this post worth reading is what happened during the port:

• one mechanism I deliberately left out
• a failure the original design couldn’t detect
• a check that found a real problem on its first run
• a feature that was noisy until I measured it

1. What I Ported, and What I Didn’t

The Claude Code setup already had two of the original’s pieces:

• Dreaming with a concurrency check. claude_dream.py already hash-checks a file right before replacing it, so a nightly run can’t overwrite a bullet a live session just added.
• Context injection. It already happened at session start, but the same packet went in whatever the task was.

I added four pieces:

Left out: dreaming that writes skills on its own. In the original, the nightly run can create SKILL.md files. A skill is instructions the agent follows. In the previous post I decided to pin third-party skills and read every change before taking it. A model writing new instructions for itself at 3 a.m., unreviewed, contradicts that decision. If I want the feature later, the safe form is drafts in a review folder that I promote by hand.

2. The Mistake Ledger

Semantic memory holds facts (“Omnisearch listens on port 51361”). The ledger holds rules learned the hard way (“do X before Y, otherwise Z”). Each rule is one imperative bullet with its reason, grouped by topic:

## Shell and scripting

- 2026-11-10: Run scripts with set -o pipefail or capture output into a variable and check it:
  'cmd | next' hides cmd's failure, and a build script carried on with an empty registry token.
- 2026-11-10: Give a heredoc a unique delimiter (<<'PYEOF') when its body may contain its own EOF
  line; an inner EOF ends the heredoc early and the rest runs as shell.

## Containers and clusters

- 2026-11-10: When an image's start script is its CMD (no ENTRYPOINT), Kubernetes `args:` replaces
  it; set `command:` to the start script as well, otherwise the first argument runs as a binary.

Seed it with real mistakes only. I seeded five rules, each from a mistake made and then fixed in a single day’s work. An invented “best practice” list would be noise the agent learns to skip.

Dreaming adds to it, with a strict gate. The consolidation schema has a new new_lessons array. Its prompt allows a lesson only when the replay shows a mistake that was made and then corrected: a command failed and a different one worked, or I corrected the agent. Each lesson needs evidence from the replay. The current lessons are sent along with the memory, so the model doesn’t re-propose them. Lessons are written through the same hash-checked write path as facts.

The first real test run replayed a session whose mistakes were already in the ledger. It proposed zero lessons, which is the right answer.

3. Context per Prompt, and Why the First Version Was Noisy

Claude Code’s UserPromptSubmit hook receives the prompt as JSON on stdin, and whatever the hook prints is added to the context. The hook calls obsidian_memory.py prompt-context, which:

1. extracts keywords from the prompt
2. scores the lessons and memory bullets by keyword overlap
3. asks the vault search (Omnisearch) for the top 3 notes, skipping Meta/, which the other two sources already cover
4. prints a packet capped at about 800 tokens

It also keeps a small per-session cache of what it has already injected. Without that, the same lesson would be re-added on every prompt of a long session, and the transcript would fill up with copies.

The first version was noisy, and I saw it immediately. I typed a five-word prompt, roughly “yes add the verify check”, and the hook added two bullets about an unrelated project. One matched on “add”, the other on “verify” against “verifier”. The original post doesn’t mention this problem. With one or two keywords, any match threshold low enough to fire is low enough to fire on noise.

The fix:

def keywords(text):
    words = re.findall(r"[a-z][a-z0-9_.\-]{2,}", text.lower())
    return list(dict.fromkeys(w.strip(".-") for w in words
                              if len(w.strip(".-")) >= 4 and w.strip(".-") not in STOPWORDS))

if len(task.strip()) < 12 or len(kws) < 2:
    return          # "yes", "ok", "add the verify check": too little to match on without noise
need = 2            # a bullet must share at least two keywords with the prompt

Now the noisy prompt gets nothing, and “crc pods ImagePullBackOff after restart” gets exactly the bullet about stale registry tokens after a restart. Short follow-up prompts don’t need fresh context anyway: the session already has it.

The hook takes about 0.14 s, and its whole command ends in || true, so a broken memory script can never block a prompt.

4. Verify: Including the Dreamer Itself

obsidian_memory.py verify checks, in about 10 ms:

• systemd user units: the archive .path and .timer units and the dream timer
• hooks: each expected hook command is present in ~/.claude/settings.json
• files: the memory files and the session archive database exist
• freshness: the last dream run was less than 36 hours ago
• TCP: Omnisearch and my local OpenShift cluster’s API, both marked optional because they are legitimately off at times. Optional probes report down without failing the run.

The most valuable probe came from the original setup’s own failure. The Antigravity dreaming quietly fell back to its heuristic engine instead of the LLM, and nothing reported it. The timer fired, the journal filled up, every “is it running” check was green, and almost nothing was learned.

The Claude Code version can fail the same way. With --engine auto, if the claude CLI is missing from the systemd unit’s PATH, the whole run silently becomes heuristic. So verify checks how the last cycle ran, not just that it ran:

def last_cycle_check():
    """The newest Dream Cycle entry must have used the LLM engine and replayed every session without error."""
    m = re.search(r"(?ms)^## Dream Cycle #(\d+)[^\n]*\n(.*?)(?=^## Dream Cycle #|\Z)", journal_text)
    entry = m.group(2)
    eng = re.search(r"^- engine: (\w+)[^\n]*cost: \$([\d.]+)", entry, re.M)
    errors = entry.count("⚠️")
    ok = bool(eng) and eng.group(1) == "llm" and errors == 0

I tested it against two fake journals. A heuristic cycle fails with engine heuristic, $0.000, and an LLM cycle with a failed session fails with 1 session error(s). My real journal passes with #3: engine llm, $0.295. The non-zero cost is itself the proof that the model ran: the heuristic engine always costs nothing.

Lesson: checking that a background job is alive tells you nothing about whether it did its work. Probe the output for evidence of the real path.

5. Eval: A Recall Benchmark That Earned Its Keep on Day One

obsidian_memory.py eval runs probes from a JSON file in the vault. Each probe names a search query, the file that should hold the answer, tokens that must be present and tokens that must not:

{"id": "omnisearch-port", "category": "infra", "query": "Omnisearch HTTP API",
 "target_file": "Meta/claude-memory.md", "expected_all": ["51361"], "prohibited": []}

Each probe makes three checks: facts (every expected token is in the target file), clean (no prohibited token is there) and rank (the target file is in the top 5 search results for the query). Two additions over the original:

• Prohibited tokens guard against stale facts. When I moved a set of scripts, the probe for them forbids the old path, so a dream that resurrects it fails the benchmark.
• A global secrets probe. It runs every memory file through the same redaction patterns the write paths use. If redaction would change a file, a secret got in.

--save appends one row to a scorecard table. I chose a table over the original’s one file per run, so a year of nightly runs stays one readable note. The function compares each row with the previous one and reports REGRESSED: <probe ids> for any probe that newly fails.

On its first run it failed one probe, and correctly. The probe for a lesson I had added a minute earlier wasn’t in the top 5. The fact was in the file, but Omnisearch indexes a file created outside Obsidian only after a delay of a few minutes. The local fallback scanner found it at rank 1, and a few minutes later Omnisearch did too. It isn’t a bug, but it is real behavior: a note the agent writes is invisible to ranked search for a few minutes. That’s worth knowing when a session writes a note and searches for it right away. The benchmark found it before I would have.

Baseline after that: 12/12.

6. Wiring It into the Night

All of this runs inside the existing nightly dream run, with no new timers:

• Every run executes verify and eval --save, even when there is nothing to dream about.
• On a dream night, both results go into the Dream Cycle entry in the journal.
• On a quiet night, they reach the agent log only when a required probe fails or the eval regresses. A log line every night saying “all fine” is one I’d learn to skip, and then I’d skip the bad one too.
• The dream ledger now records last_run_at separately from last_dream_at, so a quiet weekend doesn’t trip the freshness probe.

Every directive got a mechanism. The global CLAUDE.md tells the agent to use add-lesson when something it did failed and a different approach worked, and the curation skill now merges and prunes lessons and runs eval as part of curating.

7. Lessons

1. Verify the work, not the worker. A green timer and a growing journal hid a dreamer that wasn’t dreaming. Check the output for evidence of the real path: the engine used, a non-zero cost, no errors.
2. Relevance needs a floor. Context per prompt is only worth it if it stays quiet when it has nothing relevant. Two shared keywords, no short words, and no repeats within a session turned noise into signal.
3. Benchmarks find behavior, not just regressions. The first eval run taught me something about the search index that no amount of reading would have.
4. Port the design, not the decisions. Self-written skills fit one setup’s trade-offs and conflict with another’s. Leaving a mechanism out on purpose is part of porting it.
5. Seed with truth. A ledger of five real mistakes beats fifty plausible rules.

The memory now records itself, curates itself, consolidates overnight, writes proper Obsidian, and checks every night that all of this is still working, including the part that runs while I sleep.