This began as a result of my Obsidian assistant saved getting amnesia. I didn’t wish to arise Pinecone or Redis simply so Claude might keep in mind that Alice accredited the Q3 funds final week. Seems, with 200K+ context home windows, you may not want any of that.
I wish to share a brand new mechanism that I’ve began operating. It’s a system constructed on SQLite and direct LLM reasoning, no vector databases, no embedding pipeline. Vector search was largely a workaround for tiny context home windows and retaining prompts from getting messy. With trendy context sizes, you may usually skip that and simply let the mannequin learn your reminiscences immediately.
The Setup
I take detailed notes, each in my private life and at work. I used to scrawl in notebooks that may get misplaced or get caught on a shelf and by no means be referenced once more. A number of years in the past, I moved to Obsidian for the whole lot, and it has been implausible. Within the final yr, I’ve began hooking up genAI to my notes. At the moment I run each Claude Code (for my private notes) and Kiro-CLI (for my work notes). I can ask questions, get them to do roll-ups for management, monitor my targets, and write my reviews. However it’s at all times had one massive Achilles’ heel: reminiscence. After I ask a couple of assembly, it makes use of an Obsidian MCP to go looking my vault. It’s time-consuming, error-prone, and I would like it to be higher.
The apparent repair is a vector database. Embed the reminiscences. Retailer the vectors. Do a similarity search at question time. It really works. However it additionally means a Redis stack, a Pinecone account, or a domestically operating Chroma occasion, plus an embedding API, plus pipeline code to sew all of it collectively. For a private device, that’s so much, and there’s a actual danger that it received’t work precisely like I would like it to. I must ask, what occurred on ‘Feb 1 2026’ or ‘recap the final assembly I had with this particular person’, issues that embeddings and RAG aren’t nice with.
Then I ran throughout Google’s always-on-memory agent https://github.com/GoogleCloudPlatform/generative-ai/tree/predominant/gemini/brokers/always-on-memory-agent. The concept is fairly easy: don’t do a similarity search in any respect; simply give the LLM your current reminiscences immediately and let it cause over them.
I wished to know if that held up on AWS Bedrock with Claude Haiku 4.5. So I constructed it (together with Claude Code, in fact) and added in some additional bells and whistles.
Go to my GitHub repo, however be certain to return again!
https://github.com/ccrngd1/ProtoGensis/tree/predominant/memory-agent-bedrock
An Perception That Modifications the Math
Older fashions topped out at 4K or 8K tokens. You couldn’t match various paperwork in a immediate. Embeddings allow you to retrieve the related paperwork with out loading the whole lot. That was genuinely crucial. Haiku 4.5 gives a context window of 250k, so what can we do with that?
A structured reminiscence (abstract, entities, subjects, significance rating) runs about 300 tokens. Which suggests we are able to get about 650 reminiscences earlier than you hit the ceiling. In observe, it’s a bit much less for the reason that system immediate and question additionally eat tokens, however for a private assistant that tracks conferences, notes, and conversations, that’s months of context.
No embeddings, no vector indexes, no cosine similarity.
The LLM causes immediately over semantics, and it’s higher at that than cosine similarity.
The Structure
The orchestrator isn’t a separate service. It’s a Python class contained in the FastAPI course of that coordinates the three brokers.
The IngestAgent job is straightforward: take uncooked textual content and ask Haiku what’s price remembering. It extracts a abstract, entities (names, locations, issues), subjects, and an significance rating from 0 to 1. That package deal goes into the `reminiscences` desk.
The ConsolidateAgent runs with clever scheduling: at startup if any reminiscences exist, when a threshold is reached (5+ reminiscences by default), and every day as a compelled go. When triggered, it batches unconsolidated reminiscences and asks Haiku to seek out cross-cutting connections and generate insights. Outcomes land in a `consolidations` desk. The system tracks the final consolidation timestamp to make sure common processing even with low reminiscence accumulation.
The QueryAgent reads current reminiscences plus consolidation insights right into a single immediate and returns a synthesized reply with quotation IDs. That’s the entire question path.
What Really Will get Saved
If you ingest textual content like “Met with Alice as we speak. Q3 funds is accredited, $2.4M,” the system doesn’t simply dump that uncooked string into the database. As an alternative, the IngestAgent sends it to Haiku and asks, “What’s essential right here?”
The LLM extracts structured metadata:
{
"id": "a3f1c9d2-...",
"abstract": "Alice confirmed Q3 funds approval of $2.4M",
"entities": ["Alice", "Q3 budget"],
"subjects": ["finance", "meetings"],
"significance": 0.82,
"supply": "notes",
"timestamp": "2026-03-27T14:23:15.123456+00:00",
"consolidated": 0
}The reminiscences desk holds these particular person information. At ~300 tokens per reminiscence when formatted right into a immediate (together with the metadata), the theoretical ceiling is round 650 reminiscences in Haiku’s 200K context window. I deliberately set the default to be 50 current reminiscences, so I’m properly wanting that ceiling.
When the ConsolidateAgent runs, it doesn’t simply summarize reminiscences. It causes over them. It finds patterns, attracts connections, and generates insights about what the reminiscences imply collectively. These insights get saved as separate information within the consolidations desk:
{
"id": "3c765a26-...",
"memory_ids": ["a3f1c9d2-...", "b7e4f8a1-...", "c9d2e5b3-..."],
"connections": "All three conferences with Alice talked about funds issues...",
"insights": "Price range oversight seems to be a recurring precedence...",
"timestamp": "2026-03-27T14:28:00.000000+00:00"
}If you question, the system masses each the uncooked reminiscences *and* the consolidation insights into the identical immediate. The LLM causes over each layers without delay, together with current details plus synthesized patterns. That’s the way you get solutions like “Alice has raised funds issues in three separate conferences [memory:a3f1c9d2, memory:b7e4f8a1] and the sample suggests it is a excessive precedence [consolidation:3c765a26].”
This two-table design is all the persistence layer. A single SQLite file. No Redis. No Pinecone. No embedding pipeline. Simply structured information that an LLM can cause over immediately.
What the Consolidation Agent Really Does
Most reminiscence methods are purely retrieval. They retailer, search, and return related textual content. The consolidation agent works in a different way; It reads a batch of unconsolidated reminiscences and asks, “What connects these?”, “What do these have in widespread?”, “How do these relate?”
These insights get written as a separate consolidations document. If you question, you get each the uncooked reminiscences and the synthesized insights. The agent isn’t simply recalling. It’s reasoning.
The sleeping mind analogy from the unique Google implementation appear fairly correct. Throughout idle time, the system is processing somewhat than simply ready. That is one thing I usually battle with when constructing brokers: how can I make them extra autonomous in order that they’ll work once I don’t, and it is a good use of that “downtime”.
For a private device, this issues. “You’ve had three conferences with Alice this month, and all of them talked about funds issues” is extra helpful than three particular person recall hits.
The unique design used a easy threshold for consolidation: it waited for five reminiscences earlier than consolidating. That works for energetic use. However in the event you’re solely ingesting sporadically, a notice right here, a picture there, you may wait days earlier than hitting the brink. In the meantime, these reminiscences sit unprocessed, and queries don’t profit from the consolidation agent’s sample recognition.
So, I made a decision so as to add two extra triggers. When the server begins, it checks for unconsolidated reminiscences from the earlier session and processes them instantly. No ready. And on a every day timer (configurable), it forces a consolidation go if something is ready, no matter whether or not the 5-memory threshold has been met. So even a single notice per week nonetheless will get consolidated inside 24 hours.
The unique threshold-based mode nonetheless runs for energetic use. However now there’s a security web beneath it. Should you’re actively ingesting, the brink catches it. Should you’re not, the every day go does. And on restart, nothing falls by means of the cracks.
File Watching and Change Detection
I’ve an Obsidian vault with a whole bunch of notes, and I don’t wish to manually ingest each. I wish to level the watcher on the vault and let it deal with the remaining. That’s precisely what this does.
On startup, the watcher scans the listing and ingests the whole lot it hasn’t seen earlier than. It runs two modes within the background: a fast scan each 60 seconds checks for brand spanking new recordsdata (quick, no hash calculation, simply “is that this path within the database?”), and a full scan each half-hour, calculates SHA256 hashes, and compares them to saved values. If a file has modified, the system deletes the outdated reminiscences, cleans up any consolidations that referenced them, re-ingests the brand new model, and updates the monitoring document. No duplicates. No stale knowledge.
For private notice workflows, the watcher covers what you’d anticipate:
- Textual content recordsdata (.txt, .md, .json, .csv, .log, .yaml, .yml)
- Pictures (.png, .jpg, .jpeg, .gif, .webp), analyzed through Claude Haiku’s imaginative and prescient capabilities
- PDFs (.pdf), textual content extracted through PyPDF2
Recursive scanning and listing exclusions are configurable. Edit a notice in Obsidian, and inside half-hour, the agent’s reminiscence displays the change.
Why No Vector DB
Whether or not you want embeddings on your private notes boils down to 2 issues: what number of notes you could have and the way you wish to search them.
Vector search is genuinely crucial when you could have tens of millions of paperwork and might’t match the related ones in context. It’s a retrieval optimization for large-scale issues.
At private scale, you’re working with a whole bunch of reminiscences, not tens of millions. Vector means you’re operating an embedding pipeline, paying for the API calls, managing the index, and implementing similarity search to unravel an issue {that a} 200K context window already solves.
Right here’s how I take into consideration the tradeoffs:
Complexity
Accuracy
Scale
I couldn’t justify having to setup and preserve a vector database, even FAISS for the few notes that I generate.
On high of that, this new methodology offers me higher accuracy for the best way I would like to go looking my notes.
Seeing It In Motion
Right here’s what utilizing it really appears like. Configuration is dealt with through a .env file with wise defaults. You possibly can copy of the instance immediately and begin utilizing it (assuming you could have run aws configure on you’re machine already).
cp .env.instance .envThen, begin the server with the file watcher energetic
./scripts/run-with-watcher.shCURL the /ingest endpoint with to check a pattern ingestion. That is choice, simply to show the way it works. You possibly can skip this in the event you’re establishing in an actual use case.
-H "Content material-Kind: software/json"
-d '{"textual content": "Met with Alice as we speak. Q3 funds is accredited, $2.4M.", "supply": "notes"}'The response will seem like
{
"id": "a3f1c9d2-...",
"abstract": "Alice confirmed Q3 funds approval of $2.4M.",
"entities": ["Alice", "Q3 budget"],
"subjects": ["finance", "meetings"],
"significance": 0.82,
"supply": "notes"
}To question it later CURL the question endpoint with
question?q=What+did+Alice+say+about+the+fundsOr use the CLI:
python cli.py ingest "Paris is the capital of France." --source wikipedia
python cli.py question "What have you learnt about France?"
python cli.py consolidate # set off manually
python cli.py standing # see reminiscence depend, consolidation stateMaking It Helpful Past CURL
curl works, however you’re not going to curve your reminiscence system at 2 am when you could have an concept, so the undertaking has two integration paths.
Claude Code / Kiro-CLI talent. I added a local talent that auto-activates when related. Say “keep in mind that Alice accredited the Q3 funds” and it shops it with out you needing to invoke something. Ask “what did Alice say in regards to the funds?” subsequent week, and it checks reminiscence earlier than answering. It handles ingestion, queries, file uploads, and standing checks by means of pure dialog. That is how I work together with the reminiscence system most frequently, since I are inclined to dwell in CC/Kiro more often than not.
CLI. For terminal customers or scripting
python cli.py ingest "Paris is the capital of France." --source wikipedia
python cli.py question "What have you learnt about France?"
python cli.py consolidate
python cli.py standing
python cli.py listing --limit 10The CLI talks to the identical SQLite database, so you may combine API, CLI, and talent utilization interchangeably. Ingest from a script, question from Claude Code, and examine standing from the terminal. All of it hits the identical retailer.
What’s Subsequent
The excellent news, the system works, and I’m utilizing it as we speak, however listed below are a number of additions it may benefit from.
Significance-weighted question filtering. Proper now, the question agent reads the N most up-to-date reminiscences. Meaning outdated however essential reminiscences can get pushed out by current noise. I wish to filter by significance rating earlier than constructing the context, however I’m unsure but how aggressive to be. I don’t desire a high-importance reminiscence from two months in the past to vanish simply because I ingested a bunch of assembly notes this week.
Metadata filtering. Equally, since every reminiscence has related metadata, I might use that metadata to filter out reminiscences which are clearly unsuitable. If I’m asking questions on Alice, I don’t want any reminiscences that solely contain Bob or Charlie. For my use case, this could possibly be based mostly on my notice hierarchy, since I maintain notes aligned to prospects and/or particular tasks.
Delete and replace endpoints. The shop is append-only proper now. That’s high-quality till you ingest one thing unsuitable and wish to repair it. DELETE /reminiscence/{id} is an apparent hole. I simply haven’t wanted it badly sufficient but to construct it.
MCP integration. Wrapping this as an MCP server would let any Claude-compatible shopper use it as persistent reminiscence. That’s in all probability the highest-lift factor on this listing, nevertheless it’s additionally probably the most work.
Attempt It
The undertaking is up on GitHub as a part of an ongoing sequence I began, the place I implement analysis papers, discover modern concepts, and repurpose helpful instruments for bedrock (https://github.com/ccrngd1/ProtoGensis/tree/predominant/memory-agent-bedrock).
It’s Python with no unique dependencies, simply boto3, FastAPI, and SQLite.
The default mannequin is `us.anthropic.claude-haiku-4-5-20251001-v1:0` (Bedrock cross-region inference profile), configurable through .env.
A notice on safety: the server has no authentication by default; it’s designed for native use. Should you expose it on a community, add auth first. The SQLite database will comprise the whole lot you’ve ever ingested, so deal with it accordingly (chmod 600 reminiscence.db is an effective begin).
Should you’re constructing private AI tooling and stalling on the reminiscence downside, this sample is price a glance. Let me know in the event you resolve to attempt it out, the way it works for you, and which undertaking you’re utilizing it on.
About
Nicholaus Lawson is a Answer Architect with a background in software program engineering and AIML. He has labored throughout many verticals, together with Industrial Automation, Well being Care, Monetary Providers, and Software program firms, from start-ups to giant enterprises.
This text and any opinions expressed by Nicholaus are his personal and never a mirrored image of his present, previous, or future employers or any of his colleagues or associates.
Be at liberty to attach with Nicholaus through LinkedIn at https://www.linkedin.com/in/nicholaus-lawson/
