Teaching a Local AI to Study From My Own Notes

How I debugged a self-hosted LLM agent that kept making things up — and got it to quiz me from my real certification notes instead.

I run a small AI agent stack on a Mac mini in my homelab. One of those agents has a specific job: quiz me on my IT certification material — Security+, Network+, and the others on my roadmap — using the study notes I've written myself. Everything runs locally. No cloud API, no data leaving the machine, no subscription. Just a language model reading my own files and turning them into practice questions.

That was the idea, anyway. In practice, it spent weeks confidently quizzing me on material it had never read.

This is the story of finding out why, because the answer turned out to be a lot more interesting than "the AI is bad." Every quiz looked right — well-formatted questions, plausible answers, the correct exam domains. But the content was generic. It was pulling from whatever the model absorbed during training, not from my notes. For a study tool, that's worse than useless: it feels trustworthy while quietly teaching you the wrong emphasis.

The first wrong assumption: "it's the model"

My initial instinct was that the model wasn't smart enough, so I went shopping. I tried larger models, higher-precision versions, different families, reasoning-tuned variants. Some crashed for lack of memory. Some loaded but still hallucinated. One would confidently announce it was "reading the file" and then produce questions with no relationship to what the file contained.

The turning point was a simple observation: the same model gave different results on different runs. If swapping models didn't reliably fix it, then the model was never the variable. I'd been optimizing the wrong thing.

What was actually going wrong

Digging into the agent's behavior turn by turn, three separate problems were stacked on top of each other — each one enough to break the tool on its own.

Reasoning models struggle to use tools. The most capable-sounding models are often tuned to "think" through a problem in long internal monologues. That's great for math puzzles and terrible for a task that requires stopping to actually run a command and read a file. A reasoning-heavy model would deliberate about the answer, decide it already knew enough, and skip the step where it reads my notes. The models that reliably did the work were the plainer instruction-following ones — less flashy, far more dependable at the boring, critical step of "open the file first."

Accumulated history poisoned the context. Language models have a limited working memory. My agent had been running for months, and every conversation piled onto the same session. Eventually that history grew so large it crowded out everything else — including the fresh file it was supposed to read. The fix was to start each quiz in a clean session so the model's attention was on my notes, not on months of prior chatter.

Given too many options, the agent wandered. When asked a vague question like "quiz me on this topic," the model had access to a whole toolbox — web search, memory lookups, session history — and would try all of them, burn through its working memory, and give up before ever reading the file. The fix was to take the toys away: restrict that agent to exactly the tools it needs (read a file, run a command) and nothing else. With no way to wander, its only path forward was the one that actually worked.

The fixes, in plain terms

None of the solutions were about buying a bigger model. They were about discipline:

• Right tool for the job. A dependable instruction-following model beats an impressive reasoning model when the task is "read this, then respond."
• Start clean. Fresh context every session, so the model reads today's material instead of drowning in yesterday's.
• Constrain the options. An agent with fewer tools makes fewer wrong turns. Removing capabilities made it more reliable, not less.
• Be explicit. A precise instruction — "read this specific file, then build questions only from what you find" — beats a vague one every time.

Put together, the result is an AI tutor that now opens my actual notes, proves it read them, and builds practice questions from my own material. It works from my desk and from my phone. When I get one wrong, it explains why using the source I wrote — not a generic textbook it half-remembers.

Why this matters beyond one study app

The whole project is a small, self-contained example of something I do professionally: a system that "works" on the surface is hiding why it works, and when it breaks, the obvious explanation is usually wrong. The temptation was to throw a bigger, more expensive model at the problem. The actual fixes were free — they came from understanding how the pieces fit together and removing the ones causing trouble.

That's the same discipline behind good infrastructure work: measure before you replace, question the obvious suspect, and prefer the structural fix over the expensive one. A local, private AI that genuinely reads your own knowledge base — instead of paraphrasing the internet — is very achievable. It just rewards patience over horsepower.

Running entirely on local hardware, this setup keeps my notes private and costs nothing per query. If you're weighing self-hosted AI against cloud subscriptions for your own business, that trade-off is worth a serious look.