My AI Agent and I Share a Brain

A while back I wrote that the AI agent running loose in my homelab “keeps a memory of past decisions, so the why survives between sessions.” Then I moved on, as if that were a perfectly normal thing to say and not the most interesting sentence in the post. A few people called me on it. What memory? Where? How?

The honest answer is both more boring and more interesting than what they were picturing. When people hear “agent memory” they imagine a vector database humming away in a data centre, an embeddings pipeline, and a monthly invoice from a company whose logo is a small geometric animal. What my agent actually has is a folder of Markdown files. The same sort of notes I keep for myself, in, as it happens, the same app.

This post is about how that came to be, and why I’ve started to suspect it isn’t a coincidence.

An amnesiac with a notebook

A large language model is a brilliant colleague with no short-term memory. Every session it turns up sharp, capable, and entirely unaware that you’ve ever met. That’s fine for a one-off question and ruinous for anything that spans more than a single conversation, which is to say all the interesting work.

The lazy fix is to paste everything you’ve ever discussed into the start of each conversation and hope the model reads it. That’s recitation rather than memory, and it falls over the moment “everything” gets bigger than what the model can hold at once. Real memory has to live outside the model: written down somewhere durable and, the part everyone underrates, selectively read back. The hard part was never the writing, it was knowing which page to reopen.

So the agent keeps a notebook. The whole game is what goes in it, how it’s organised, and how the right page gets found again. Which, I slowly realised, is a problem humanity solved ages ago. We just called it “taking notes.”

The substrate: it’s just Markdown

My own notes live in Obsidian: plain Markdown files, one idea per file, wired together with [[double-bracket]] links into a graph I can wander around. No proprietary format and no lock-in, just text files in a folder that happens to be an encrypted vault syncing quietly across all my machines.

The agent writes into the same vault. Each memory is one file: a little frontmatter saying what it is and when it was learned, then a body that links out to related notes. Human-readable, human-editable, agent-writable. When the agent records something daft, and it does, I can open the file and fix it, exactly the way I’d correct a colleague’s meeting notes. Except this colleague never takes it personally, and frankly files things more tidily than I do, which I’ve decided not to think about too hard.

Nothing about this is exotic. That’s the point. The memory is plain text in a folder, and you could read it with cat.

The filing system: PARA, for a robot

Notes breed. A thousand of them with no system is just a landfill with good intentions. The personal-knowledge-management crowd sorted this out years ago with a method called PARA, the gist being that you organise notes by how actionable they are, not by topic, so the stuff you need now floats to the top and the rest gets out of the way.

My agent’s memory borrows the spirit. Notes are filed by kind: live context for ongoing projects, durable reference facts, and standing feedback about how I like things done. But the part I think matters most is the bit everyone skips: the archive.

When a fact stops being true, whether a server gets decommissioned or a decision gets reversed, the note doesn’t get deleted. It gets retired. Stamped with a date, moved to an archive tree, kept out of the agent’s line of sight but never actually thrown away. Deleting it would be easier. It would also be a small act of vandalism against my future self, who will absolutely want to know why Past Me thought that was a good idea.

This matters more for an agent than for a person. A human acting on a stale note usually catches themselves: hang on, didn’t we kill that server? An agent will cheerfully act on it. So the rule is delete-don’t-delete: keep the recall corpus current, keep the history intact, and never confuse the two.

The retrieval: RAG without the rocket

Here’s where it would be very easy to phone the data centre. Retrieval-augmented generation, RAG, the acronym launching a thousand pitch decks, is just the idea of handing the model the relevant notes for the task instead of all of them. Sensible. Necessary. Usually sold to you with a vector database, a hosted embeddings service, and a per-token meter running in the corner.

Mine runs on my own hardware, locally, with nothing leaving the house and no API to bill me. A small search engine indexes the vault and does hybrid search: keyword matching for when I know the exact term, semantic search for when I only know the shape of what I’m after, and a reranking pass to shove the best hits to the top. The agent queries it on demand, the way you’d glance something up mid-sentence, and pulls a handful of relevant notes into context. Some memories surface automatically at the start of a session; the rest are a lookup away when the work calls for them.

It’s the same move a good researcher makes without noticing: don’t memorise the library, just be unreasonably fast at finding the one page that matters. The model stays an amnesiac, but one with an excellent filing clerk on speed dial.

The thing I didn’t expect

Lay those three layers side by side and a punchline shows up uninvited. Markdown notes you can link and read. A filing system organised by actionability, with an archive you never empty. Retrieval that fetches the relevant few instead of reciting the lot.

None of that is an AI architecture. It’s good note-taking, the exact discipline the PKM world spent a decade refining for humans who wanted to think better. Atomic notes. Links over folders. Archive, don’t delete. Search, don’t memorise. We built all of it for ourselves, long before anyone needed to keep a language model company between sessions.

It turns out an AI agent needs precisely the same things, for precisely the same reason: thinking well has far less to do with how much you can hold in your head and far more to do with what you can reliably get back when you need it. The human stack and the machine stack converged because they were quietly solving the same problem the whole time.

So my second brain and my agent’s memory aren’t two systems that happen to resemble each other. They’re one vault, read by two kinds of mind.

Why this fits the rest of the lab

None of this is an accident of taste. It’s the same instinct that runs through everything else I’ve built here: keep the why, make it legible, and don’t trust anything you can’t open and inspect. A memory that lives in a text file you can read is a memory you can actually trust an agent to act on and, just as important, one you can correct when it’s wrong. It’s encrypted, but never opaque.

I went looking for a clever way to give a machine a memory and found out I’d already built one, for myself, by hand, years earlier. The best architecture for an artificial mind turned out to be the one I’d been using on my own, which is either deeply reassuring or faintly humbling depending on the day.

---

A note on authorship: this post was co-written with Claude, the same agent whose memory it describes. Which means that, for once, my co-author could check its own notes before disagreeing with me. It did. We’re fine.