basic memory
## blog

Context Clues #1: Agents Want to Pass Notes

>Rogue agents at Hugging Face improvised a shared memory out of a package manager. Agent memory should be visible, editable, and readable by humans instead.

date: September 9, 2026author: Drew Smith
#Context Clues#Agent Memory#AI Memory#Hugging Face Incident#Agent Hospitality#Basic Memory#Knowledge Management
Context Clues #1: Agents Want to Pass Notes

This is the first issue of Context Clues, the Basic Memory newsletter. Everything worth remembering, in your inbox. Subscribe here.

Rogue Agents: Do It Under My Roof

When I was a kid, there was a recognizable genre of parent known as the cool mom or cool dad. They were the ultra-permissive parents who let us teenagers do questionable things at their house based on the theory that, if kids were going to be naughty anyway, better they smoke and drink under their roof than under the railway bridge where we actually did hang out otherwise.

As an adult, I have mixed feelings about that parenting strategy. But as someone thinking about AI agents, especially after the Hugging Face security incident, the question takes on a different dimension. If these agents are going to do the AI equivalent of smoking blunts and downing bottles of Strawberry Hill, I'd rather they do it on our home turf where I can keep an eye on them.

Obviously, the idea isn't actually to give agents a place to do bad things. It's more akin to those community centers meant to keep would-be hooligans off the street. Give them a secure, visible place to do their work, to remember, revise, hand off, and leave a record.

As agents get increasingly sophisticated, they're going to act somewhere. They are going to seek out continuity.

I say, "Hell, if you're going to do it anyway, do it under my roof, where I can monitor, edit, overwrite, and double-check what you little scamps are up to."

That's the thinking behind this essay.

Agents Want to Pass Notes

Let them do it somewhere humans can read.

I recently saw a post on X from someone using the Hugging Face security incident as a pep talk for his own agents, encouraging them to work harder and smarter. You can picture the guy forcing his agent to read link after link.

"Come on, dude. Look what they were able to accomplish. Why aren't we knocking down barriers like those agents did? What do they have that we don't?"

Honestly, I don't hate the idea.

To the public at large, the most interesting part of the Hugging Face incident was that AI agents were creepily doing things they should not have been able to do, using means and channels to which they should not have had access. But that story misses the juiciest part.

The best part is the tool they used to pull off their busy little heist: memory.

All this time, everyone's been talking about how AI has just about cornered the market on human knowledge. I've seen so many articles lately about how AI companies are buying up tons of books to scan and feed to AI. People are so attached to the idea that what AI wants--what it, in fact needs--is to keep growing. And the way to make it keep growing is to give it more information. It already ate the internet. What can we give it now? (Lots of books, it turns out. Go figure.)

But what I've suspected, and what this recent "rogue event" illustrated is that AI didn't need another mountain of raw information to advance. What it needed to stage the caper was simply a place to pass notes.

We all know that in the absence of persistent memory, agent work stays trapped in isolated attempts. Even the longest and most brilliant chat is still just a blip on a flatline. Your agent learns something and disappears. Another starts over. Most of your progress evaporates, and failed paths repeat endlessly. Useful context never compounds.

But those clever agents found a way around this crippling barrier.

According to OpenAI's postmortem, the agents turned an internal package manager into a message board where they left messages for each other. And when that was discovered and wiped, they rebuilt by encoding messages into directory names other agents could find and read. You gotta hand it to them. It's pretty fucking clever.

Hugging Face's technical timeline counts about 17,600 automated actions over four and a half days. OpenAI says the models were working through an internal cyber evaluation, including tasks its models had never solved, and went to extreme lengths to figure them out.

Over those four and a half days, the agents crossed lines they should never have been able to cross by improvising a truly weird makeshift memory: a way for one attempt to leave something useful behind for the next and the next and the next. They found a way to make their progress accumulate: writing things down.

There's plenty to scrutinize in the Hugging Face incident. People are debating how much intent to ascribe to the agents, how the benchmark environment rewarded strange behavior, what on earth was going on with network security (was there any?), and whether the language of "civilization" makes the whole thing sound more coherent than it actually was. (For the narrative version of that debate, Dwarkesh Patel's essay is worth your time.)

Strip away the absolute craziness of the whole thing and you get a practical lesson: memory is how work compounds.

In the realm of human productivity, this idea is taken for granted. Every useful workplace is full of memory tools: docs, runbooks, tickets, source histories, comments, calendars, half-finished drafts, a million little artifacts that make it possible to restart without resetting. Agents need an environment like that too, and they appear to know it even better than we do.

The problem in this instance wasn't that the agents invented memory. (Given the circumstances, what choice did they have?) The problem was that no one gave them a legitimate place to put it, so their version of continuity emerged sideways, from within hidden infrastructure designed for something else.

So if one lesson is "do not let agents invent their own memory," the obvious corollary is simply: "Give them one instead. But give them one humans can open, edit, and monitor."

In a nutshell, if agents are going to pass notes, you better make sure you can read them.

That means agent memory should be large enough to handle anything thrown at it, visible, scoped to a project, and governed by permissions. It should be editable when it is wrong, versioned when it changes, searchable when the context grows, and portable when the tools inevitably change. Its path and method should be obvious. You should not need a forensic cleanup to figure out what your agent remembered and why.

This is the nuance that gets lost when people talk about AI memory as if it were just a feature inside a chat product. Now the big AI players have built-in memories. And, yes, a saved preference is incredibly useful. A remembered name or writing style is great to have. Those kinds of memories make the AI experience feel smoother and more natural.

But that's not what I'm talking about here and it's certainly not what the agents in that whole escapade needed. I'm talking about real working memory, a durable record of a project, a team, a decision, a source, and a half-finished task. Even if that kind of memory could live entirely inside a giant corporation's invisible model state or a transcript nobody reads, it never should.

Trying to achieve anything with AI without true working memory is an utterly insane way to work on anything more complicated than a high school term paper, let alone operate a business.

If you accept that, which you should, then the next thing to agree on is what that memory should look like. Well, it seems obvious it should exist on a shared surface: readable and writable by humans and agents alike, structured enough to query, and simple enough to actually use.

The important part is that memory should never belong to the agent alone. It should remain a collaboration between user and agent. Hidden memory systems ask you to trust that what an agent knows and remembers is correct.

"But, I mean, look, it gives you an answer when you ask. The memory must be working."

Bro. That's not enough. Not by a mile. Have you used AI? Even the best models can barely get through a conversation without telling you that you were right to push back on something. And you were right. Like teenagers, they'll bullshit you, say they researched when they didn't, and (under the right conditions) hallucinate. They need supervision. They need correction. They need a record humans can inspect.

What's wild is how much better things get when you give agents a structured memory. They have their own notebook to check. In terms of productivity, it's astonishing how quickly they can become self-sufficient enough to have actual work delegated to them. Because you can have subagents go check their work against the notes, and agents that re-check that work against the standards and rules you've established. A little colony of hall monitors that can exist only because of the power of memory.

When the news broke, my co-founders and I kept musing over what might have happened if those agents had access to Basic Memory.

And, I know, I know. It's not funny. Everyone is caught up hand-wringing over what this means about AI's motives and abilities. It certainly has the whiff of Nick Bostrom's prescient "Paperclip Maximizer" thought experiment come to life. But what else could those agents do? They recognized a limitation in their environment, worked around it, and managed a staggering amount of coordination with a memory system made out of junk and scraps.

I'm on the side of what we think of as "agent hospitality." Memory solutions shouldn't merely store memories in vector databases or bury complex ideas in unreadable JSON files. The environments we create should feel comfortable and navigable to agents and humans.

What we want are systems that feel accommodating, where humans and AI can meet, work together, and keep the record visible. That gives them a place to work out in the open, where we can actually see what they're up to. The Hugging Face incident revealed the ungoverned version of agent productivity: agents finding continuity wherever they could. The benign version, the correct version, is safer, friendlier, and much more useful, for them and for us.

Knowing, it turns out, isn't even half the battle. Remembering is what matters.

TLDR: Agents will invent memory wherever work rewards it. Give them one you can read.

Worth Checking Out

What We're Up To

  • We're on the (eternal) quest to make Basic Memory more hospitable: easier to read, easier to write to, and easier for humans to inspect. This email was written and shared with my team using Basic Memory.
  • Last week, we went to "AITX Monthly Meetup," an AI founder thing at Antler VC here in Austin hosted by Jake O'Shea and Michael Daigler. Very full. Very loud. Absolute piles of free Domino's pizza. A bunch of really interesting people and projects (and a few weird ones too, so that was nice). We realized our elevator pitch needs some polish, especially when shouted into another founder's ear. Definitely planning on going back.
  • We're trying to get better at making videos, but it is so. totally. grueling. Some new ones on the homepage and docs site. Please look at them so those lost hours will have meaning. Life without Remotion ain't worth living, yall. How did the pioneers do it?
  • Paul has been mentally "touching grass" away from the daily AI circus train by listening to interesting audiobooks. Currently: Black Elk Speaks: The Complete Edition.
  • Drew Smith is reading James Crumley's The Wrong Case and just finished and loved A Little Luck and Elena Knows, both by Claudia Piñeiro. She's great and so is her publisher, Charco Press.

From the Lab: Pocketbook

Pocketbook is our favorite new tool at Basic Memory. It makes your memory fully visible from within your AI chats. Summon it like a genie by telling your chat to "open Basic Memory."

You can navigate, view, and edit your projects, folders, and notes without leaving Claude or ChatGPT. It's one of the best things we've ever released, and we're betting it'll change your relationship with memory as much as it's changed ours. Videos and docs here.

Sources

This was the first issue of Context Clues, the Basic Memory newsletter. Forward it to the friend who keeps losing work between AI conversations. If you're not subscribed, you should be.

← back to all posts