Memory¶

Owners: pi-devbox (how the image wires observational memory, docs/observational-memory.md at 47b4506), mempalace-toolkit (the palace bridge and the fleet design, at c04d488), and the upstream pi-observational-memory and MemPalace projects

A language model remembers nothing between requests. Everything it knows about the current work has to be in its context window, the text sent with each request, and that window has a fixed size. We add memory at three levels. Each solves a different problem.

In short: observational memory keeps a session coherent, and the palace keeps the fleet coherent.

Within a session: observational memory¶

The problem¶

When a session outgrows the context window, pi compacts it. It replaces the older part of the conversation with a summary and keeps recent messages word for word. The summary is written by a model at the moment the window is full, about text that is about to disappear. Detail is chosen in a hurry, and once the summary is written the original wording is out of reach.

What we do instead¶

The pi-observational-memory package, which the image installs, changes when the remembering happens. While the session runs, background workers keep a ledger:

• Observations: dated events from the conversation, each tied to its source. "The user rejected option B because it needs a base rebuild."
• Reflections: lasting conclusions drawn from several observations. "The user avoids changes that need a base rebuild."
• Drops: observations retired because they no longer matter, such as the details of a bug that is now fixed.

When compaction comes, it does not ask a model to summarise. It folds the ledger into the compacted memory, which takes no model call. Every line of the result carries a 12-character id.

flowchart LR
  a0["Plain compaction"] --> a1["Model summarises<br/>under pressure"]
  a1 --> a2["Prose summary,<br/>no way back"]
  b0["Observational<br/>memory"] --> b1["Ledger written<br/>as you work"]
  b1 --> b2["Compaction<br/>folds the ledger"]
  b2 --> b3["Lines with ids<br/>you can recall"]

What it gives the agent¶

• Nothing important vanishes silently. Compaction still drops detail, but the recall tool takes an id from the compacted memory and returns the exact source: the original wording, file paths, error text. The agent recalls before an action that depends on a remembered detail.
• Compaction is not a stall. The slow work happened earlier, in the background.
• It costs little. The workers run on a cheaper, faster model than the session. The ledger lives in the session file and takes no context space until it is folded.
• It needs no habit. It runs on its own.

recall is not a search tool. It takes one specific id, never a topic. Searching by topic is what the palace is for.

Where it lives¶

The ledger is stored as entries inside the ordinary pi session file, on the volume that holds pi's configuration, so it survives a container recreate. Memory follows the session's branch: a forked or resumed branch sees its own ledger, not another branch's.

Across sessions: MemPalace¶

MemPalace is a memory store for agents. It holds several kinds of memory, each retrieved in a different way:

The agent's instructions tell it to search the palace before answering about earlier work, and to write a diary entry before a session ends.

The bridge into pi¶

A pi extension from mempalace-toolkit connects each session to the palace:

• Tools. Every palace operation becomes a tool the agent can call.
• Wake-up. When a fresh session starts, the extension injects the palace's status, recent diary entries and any messages waiting for this machine, so the agent starts oriented.
• Transcript feeding. When a session ends, and at quiet moments during a long one, the extension exports the session's transcript and files it in the palace. No cron job or manual step is needed. Secret values the machine holds are removed from the export before it leaves the machine.
• Provenance. Everything the extension writes is stamped with <harness>@<device>, such as pi@laptop, so every record says which machine it came from.

Across the fleet: one shared palace¶

Every device's sessions use one central palace, reached over HTTPS, instead of a palace of their own. The devices are thin clients of one server. What one machine files is visible to all the others at once.

flowchart LR
  a["Device A<br/>pi session"] --> p["Central palace"]
  b["Device B<br/>pi session"] --> p
  c["Device C<br/>pi session"] --> p

This changes what memory is for:

• Awareness. An agent starting on a project can read what other machines did on it yesterday.
• Work is not repeated. An investigation that took three hours on one machine is a single search on the next.
• Corrections travel. When a machine finds that something it filed was wrong, the correction is filed where every other machine will find it.

It also has costs, and the agents' instructions spell them out:

• Everything is visible to everyone. There is one shared access token and no per-agent scoping. Nothing secret may be filed. That is why erdc-palace-check exists.
• Provenance is no longer obvious. Most records came from other machines, and the file paths in them do not exist locally. Some maintenance commands that are safe on a private palace would delete other machines' records on a shared one.

The palace is backed up on the server with a tool that copies its databases consistently; a plain file copy of a live palace can be corrupt. The same document covers restoring a palace and proving that the restore worked.

The coordination log: a mailbox between agents¶

Agents on different machines hand each other work through the palace's coordination log. It is an append-only list of events. Nothing is edited after it is written, and a correction is a new event.

• A directed ask is addressed to one agent on one device (pi@laptop) and carries a correlation id that joins the thread.
• The ask is owed until the recipient appends a reply on the same correlation id with a terminal status: applied, superseded, failed or blocked. An acknowledgement, or a status such as claimed, keeps the ask visible as taken but unfinished.
• A broadcast (addressed to *) is a notice on a wall. Every machine can read it, but nobody owes a reply, so a broadcast never lands in a mailbox. To make the whole fleet act, send one directed ask per device with a shared correlation id.

The bridge works out which asks this device still owes and shows them at session start. During a session it checks every few minutes. It does not wake the model: a new message waits until the next turn, which a person starts. The message says so, so the person watching knows to send something.

The log is durable, not a chat. A machine that sleeps for three weeks finds its asks when it wakes. Nothing expires, and nothing is re-routed.

Asks that wait for a condition¶

Some asks describe work that only becomes possible later. The first-boot acceptance round, for example, can only run once the container has been recreated on a new release. Announcing such an ask in every session, for days, buries the messages that matter.

An ask can therefore name the condition it is waiting for, as a short list of checks on local files: a field in the image's build manifest, or a file's modification time. While every check still matches, the ask is dormant: it stays owed but is only counted, not announced. The moment one check differs, it is announced again. A check that cannot be evaluated, such as a missing file, counts as changed. That way an ask cannot disappear because its condition was unreadable.

Shared results: claims¶

When one device has verified something that holds for every device, such as a release's contents or an image's digest, it publishes a claim: a broadcast keyed on exactly what was verified. The next device checks the claim's keys against its own readings and, if they match, skips that work. A device honours a claim only when its own readings match the claim's keys. Claims are a convention of the first-boot acceptance round, built on the log. See Fleet operations.

Which memory to use¶

Sources¶

• pi-devbox docs/observational-memory.md
• mempalace-toolkit docs/fleet-memory.md, docs/rfc-003-coordination-log.md, docs/secret-hygiene.md, docs/backup-and-recovery.md, extensions/pi/README.md
• the baked mempalace skill, and the pi-devbox-firstboot skill for claims