What It Is
In short, think of RAG (Retrieval-Augmented Generation) as a way for AI to look things up before answering. It is interesting and powerful and I hope to explain why in this blog post!
No Surgery
RAG splits an AI model’s thinking from its knowing. The model deals with language and reasoning. A separate retrieval system deals with facts. And because those are separate, it means people can swap out everything the AI “knows” without retraining a single parameter.
New data can be passed in through various means such as databases, search indexes, and documents. This eliminates the need for training the model on new data and concepts, which is a huge saviour.
The Search Problem
Which means a huge chunk of “AI work” in RAG is actually old-school search and embedding plumbing. You think you’re building something futuristic and then you spend three days tuning how documents get chopped up. Welcome to the job.
If we were to put a smaller “dumber” AI model with great retrieval against a bigger smarter AI model with mediocre retrieval, the “dumber” AI would win. This means a huge amount of work needs to be put into how data is searched. While we have reduced the effort needed for “training” a model, we get this new tradeoff: the search problem.
Meaning Over Words
Let’s say I ask an AI model: "How do I beat the Wall of Flesh?" RAG can pull up a guide that says "strategies for defeating the hardmode-triggering boss" with barely a word in common.
It works because the text gets converted into vectors, mathematical fingerprints of meaning, and similar ideas land near each other in that space.
Document Chopping
One might think slicing a document into pieces would be the boring part. It’s actually one of the deepest rabbit holes in this whole RAG operation.
Cut the chunks too small, and you strip away the surrounding context that gave them meaning. Cut them too big and the relevant sentence drowns in noise. Where you draw the lines (mid-paragraph, by section, with overlap or without) changes your results more than you’d ever expect.
This means there’s real craftsmanship in deciding where to make the cut.
Confidently Wrong
Sometimes we see RAG sold as a cure for AI hallucination. While it does help, it introduces its own failure. A model can become a little too obedient to whatever it retrieved. Feed it a wrong document, and it’ll repeat the wrong thing with total composure. Garbage in, garbage out.
Additionally, did you know that AI models often pay the most attention to the beginning and end of what you give them? This means they can lose track of context, which may be buried in the middle. I guess you could say it’s getting lost in the middle…
This means it can be a good idea to put key information at the edges, because the order in which information is handed to the AI model changes the answer dramatically.
Mutation
The basic “search, then answer” pattern is already starting to look quaint. A few directions worth watching:
- GraphRAG pulls from knowledge graphs, so it captures relationships between things, not just isolated facts.
- Agentic RAG lets the model decide what to look up and when, running several rounds of search and reasoning instead of one.
- Hybrid search mixes the meaning-based approach with plain old keyword matching, because sometimes you really do just need the exact term.