If you think that switching to a better model will solve the problem of Agent forgetfulness, this article will change your perception. arxiv 2603.07670 The empirical findings of the paper are surprising: the performance gap between having and not having memory tends to be larger than the gap between different LLM backbones. Changing models might be better than getting the memory system right first. Today, we deeply dismantle the core principle of the AI Agent memory system, the four types of memory, common failure modes, and the selection ideas of mainstream frameworks.
LLM Born Without Memory: Understanding Problems by Their Nature
Big language models are stateless by nature. Each API call is independent, and the model doesn’t “remember” what you asked yesterday or what the conclusion of the last conversation was. It can only see what was in the context window you passed in that time.
This means that all of Agent’s “memory” is not inherent in the model – it is actively injected into the context by you through engineering. Remember the phrase: Memory is not a feature of the model – it is a feature you build.
This perception is critical. Because it tells us that the quality of a memory system depends entirely on how you design the storage, management, and retrieval of information. Whether the model is changed or not is really a secondary issue.
The limitations of the context window compound the problem: even with very long contexts of 200K tokens, information in long contexts is subject to “attentional dilution” – the model technically “sees” everything, but the key information may be drowned out in the noise, affecting the quality of the output.
Four types of memory: corresponding to human cognitive structures
Cognitive science’s categorization of memory is unexpectedly well suited to understanding the memory architecture of an AI Agent. Modern memory systems are typically divided into four levels:
Working Memory, the current context window. high bandwidth, high speed, but fleeting. At the end of each conversation, this layer of memory is zeroed out. The failure mode is “Attention Dilution” – when the context is too long, Agent behavior degrades in ways that are difficult to debug, because the model technically “has” the information, but is not using it.
Episodic Memory, which records exactly what happened, when, and in what order. Analogous to humans, “I remember last Friday we had a meeting and decided on Option A”. For an Agent, this could be a timestamped task log, a summary of a conversation, a record of a decision. The value is huge: the Agent can look back at yesterday’s work, discover patterns, and avoid repeating mistakes.
Semantic Memory, abstract knowledge, facts, heuristics. Not “what happened”, but “what’s true”. For example, user preferences, domain knowledge, summarized best practices. This layer of memory needs to be actively curated – not all episodes are worth distilling into semantic memory, otherwise it becomes a cluttered room.
Procedural Memory, the “skill memory” that encodes behavioral patterns. For an agent, this is usually reflected in the role settings, behavioral constraints, and tool usage specifications in a system prompt. Many teams will carefully tune a prompt, but there is little to no mechanism to automatically translate user feedback into updated procedural memory – this is a layer that has been sorely neglected.
Highlights:** These four levels of memorization are not multiple choice, but rather hierarchical structures, each with its own division of labor, and each without the other. **
Write-Manage-Read: The Core Cycle of Memory
Most memory systems implement “write” and “read” and then stop. But it’s the second, often overlooked, step of Manage that really fails memory systems.
The complete memory cycle goes like this:
- Write: entry of new information into memory (observations, tool returns, reflections)
- Manage: maintenance, pruning, compression, merging and de-duplication of memories.
- Read: retrieve relevant memories on demand, inject into current context
What happens when you don’t manage memory? Noise builds up, inconsistencies occur, the context becomes more and more bloated, and eventually the Agent’s behavior continues to degrade. Have you ever seen the quality of Claude Code’s answers deteriorate after a long conversation? That’s essentially what happens when you don’t manage your memory – usually the best thing to do is to start a new conversation.
Interestingly, management is also the hardest part to get right. What to keep? What to compress? What to discard? When do you distill episodic memories into semantic memories? All of this requires explicit design, not a “just build it up” mentality.
Long-term memory implementation: vector databases are mainstream, but not unique
The most mainstream implementation of long-term memory across sessions is vector database + semantic retrieval.
The basic process: embedding the information that needs to be stored for a long time into vectors and storing them in vector databases such as Pinecone, ChromaDB, pgvector, etc.; when querying, the current query is also vectorized, and a similarity search (Top-K) is done, so as to inject the most relevant memory fragments into the prompt.
The advantage is obvious: it can find semantically relevant content across lexical differences and is suitable for fuzzy retrieval in open domains.
But vector databases have limitations: poor understanding of causality.” These two things are semantically similar” and “this thing is the cause of that thing” are one thing for vector retrieval, but completely different for Agent decision making. This is why more and more systems in production environments are migrating to vector + graph database hybrid storage.
It’s also interesting to note that on Reddit, there are voices saying “We tried vector databases and graph databases for Agent Memory, and ended up going back to SQL”. Structured queries still have irreplaceable advantages in accurate fact storage and precise condition filtering.
The tool is not the goal, fitting the scenario is.
Mainstream framework selection reference
There are currently several frameworks of interest in the field of Agent memory:
Mem0: open source, memory management framework designed for Agent. The core is a dual LLM architecture – a call is responsible for information extraction , a call is responsible for decision-making (add/update/delete), to achieve intelligent de-duplication and conflict resolution . Support for vector storage + graph storage hybrid program , can be integrated with Amazon Bedrock, Aurora, OpenSearch.
Letta (formerly MemGPT): Agent memory is analogous to the virtual memory of the operating system – context window is RAM, long-term storage is disk, and the Agent can independently decide when to “turn the page”. The concept is very elegant, but the maintenance cost is high, and there are not many cases in production environment.
LangMem: a LangChain product with native support for Semantic, Episodic, and Procedural memories, seamlessly integrated with LangGraph. Suitable for teams already in the LangChain ecosystem.
Amazon Bedrock AgentCore Memory: Fully managed, out-of-the-box, no need to build your own vector database. Built-in multiple memory strategies (semantic extraction, session summarization, user preferences), in the form of a tool that allows the agent to autonomously decide when to read and write memory, but also supports the MCP protocol. It also supports MCP protocol. It is suitable for teams that want to go online quickly and don’t want to operate and maintain the underlying resources.
Different frameworks solve the same problem, which one to choose depends on your technology stack and O&M cost acceptance.
Real-life failures: you’ve probably stepped in these potholes, too
Summarization Drift: Repeatedly compressing the historical context to save tokens, losing details with each compression, and ultimately retaining a “memory” that deviates more and more from what really happened. It’s like the game of pass the word, where the tenth person’s mouth is already completely different.
Self-Reinforcing Errors (SRE): An Agent writes a faulty memory, and every time it makes a decision, it takes it as true, and becomes more and more convinced that it is right. In a real case, the Agent misjudged an integration as faulty, wrote that conclusion to memory, and then all data from that integration was ignored as untrustworthy – the actual cause was a dead battery. It took human intervention to correct it.
Memory Blindness (Memory Blindness): Important information does exist in the memory bank, but when retrieving it, because the vector similarity is not high enough, it is ranked outside the Top-K, and the Agent can never see it. The Agent will never see it. It has clearly memorized it, but it still “forgets” it.
These failures are not small probability events, they are systemic risks. Being good at memorizing systems is not just about being able to deposit and withdraw, it’s also about figuring out ahead of time what to do about these borderline situations.
Practical advice: gradient as needed, don’t go on all layers at once
Don’t implement all four layers of memory on day one – over-engineering is the most common mistake engineers make.
First step: start with episode memorization. At the end of each conversation, have the Agent write a structured log of key events, decisions, and outcomes to a file or database. Read the most recent entries at the start of the next conversation. Low cost and immediate results.
Step 2: Add a curation mechanism for semantic memory. Not all episodes are worth retaining for a long time, design a rule or let Agent judge for itself “is this one worth remembering”. Create a user preference profile, project context knowledge base.
Step 3: When the complexity of the system rises, then consider vector database and framework selection. By this time you know what you really need and won’t be led by the framework’s feature list.
Remember the Write-Manage-Read cycle and focus on investing in management. Clarify write strategy (what’s worth memorizing), manage strategy (when to compress/prune/expire), and read strategy (what to retrieve what in what way). Treat program memory as code – incorporate version control, track changes, review regularly.
Written at the end
The core gap in AI Agents is not in the model, it’s in the memory. This conclusion is counterintuitive, but the deeper you look, the more you realize it’s true. The memory system determines whether an Agent can truly learn from experience, maintain contextual coherence across sessions, and accumulate domain knowledge over time.
Starting today, the next time you make a selection or architectural design, ask yourself: what about memory systems? What is written into it? How will it be managed? How do I retrieve it? Thinking these three questions through is much more important than changing models.
Feel free to leave a comment to discuss if this post was helpful – what memory-related pitfalls have you stepped into when building Agent?
If this article was helpful to you, feel free to leave a comment to discuss.
