Guides
What is a RAG chatbot? Retrieval-augmented generation explained
A plain-language guide to RAG chatbots: chunking, embeddings, hybrid search, reranking and cited answers, how RAG reduces hallucinations, and its limits.
On this page
A RAG chatbot is an AI chatbot that retrieves relevant passages from your own content — web pages, documents, help articles — and uses them to write its answer. RAG stands for retrieval-augmented generation: retrieve first, then generate. It is the standard way to make a general-purpose language model answer questions about a specific business without retraining the model.
This guide explains each stage in plain language, compares RAG with fine-tuning, states its limits honestly, and describes how Chyt.ai implements it.
Why do chatbots hallucinate in the first place?
A language model generates text that is statistically plausible given its training data. It has no built-in record of your refund policy, your current prices or your product manual. Ask it anyway and it may produce a fluent answer that is simply wrong. That is a hallucination.
RAG changes the task. Instead of asking the model "what is the refund policy?", the system asks "here are three passages from this company's policy pages; answer the question using them". The model is now summarising supplied facts rather than recalling from memory, and it can show which sources it used.
How does a RAG chatbot work, step by step?
1. Ingestion and chunking
Your content is collected — by crawling a website, uploading files or connecting a tool such as Notion or Google Drive — and the text is extracted. Long documents are then split into chunks: passages small enough to be searched and handed to a model precisely, but large enough to keep their meaning. Chunk size is a trade-off. Larger chunks preserve surrounding explanation; smaller chunks isolate focused facts.
2. Embeddings
Each chunk is converted into an embedding: a long list of numbers that represents its meaning. Passages about similar ideas end up with similar numbers, even when they use different words. "How do I get my money back?" and "refund procedure" sit close together in this space, which is what lets a RAG chatbot understand paraphrases.
3. Hybrid search: vector plus keyword
When a visitor asks a question, the question is embedded too, and the system looks for the chunks closest in meaning. This is vector search. It is good at paraphrases but can be weak on exact terms such as a product code, an error number or an unusual name.
Keyword search has the opposite profile: excellent at exact terms, blind to synonyms. Hybrid search runs both and merges the two ranked lists. A common merging method is reciprocal rank fusion, which scores each chunk by its position in each list, so a passage ranked highly by either method — or moderately by both — rises to the top. A keyword-only match still survives even if the vector search missed it.
4. Reranking
The merged list is a shortlist, not a final answer. A reranker — typically a cross-encoder model — reads the question and each candidate passage together and scores how well the passage actually answers the question. It is slower than the first search, so it is applied only to the shortlist, but it is better at pushing the truly relevant passage to the top.
5. Grounded generation with citations
The top passages are placed in the model's prompt along with the question, the conversation so far and the chatbot's instructions. The model writes an answer from that context, and the chatbot shows the sources it drew on. Citations matter for two reasons: visitors can check the original page, and you can diagnose a wrong answer by seeing what was retrieved.
6. Model routing
Not every question needs the most capable model. A RAG system can classify each question and send simple ones to a fast, economical model while reserving a stronger model for questions that need multi-step reasoning. This keeps simple answers quick and spends capacity where it helps.
RAG vs fine-tuning: which should you use?
Fine-tuning adjusts a model's weights by training it further on examples. It is useful for teaching style, format or a specialised task. It is a poor way to give a chatbot facts that change.
| RAG | Fine-tuning | |
|---|---|---|
| What it changes | The information given to the model at answer time | The model itself |
| Updating knowledge | Re-crawl or re-upload the content | Retrain the model |
| Source citations | Natural: the retrieved passages are known | Not available: knowledge is baked into weights |
| Best for | Facts, policies, product and support content | Tone, output format, narrow specialised tasks |
| Skills needed | Content upkeep and testing | Training data preparation and ML expertise |
| Risk when content changes | Stale until the next sync | Stale until the next training run |
For a business chatbot answering questions from a website and documents, RAG is almost always the right starting point. The two approaches are not exclusive, but most teams never need fine-tuning.
What are the limits of RAG?
RAG makes answers more trustworthy. It does not make them perfect, and any vendor claiming otherwise should be treated with caution.
- Missing content. If the fact was never imported, retrieval cannot find it. The model may then say it does not know — or, with weak instructions, fill the gap.
- Conflicting content. If an old and a new policy are both indexed, the chatbot may combine them. Remove or replace obsolete sources.
- Retrieval misses. The right passage exists but is not selected, often because the source is poorly structured or the question is ambiguous.
- Whole-corpus questions. RAG retrieves a handful of passages. Questions that need everything at once — "how many of your products support X?" — are a poor fit unless a page states the answer.
- Extraction quality. Image-only scans, complex tables and content hidden behind scripts or logins may not be extracted well.
- Misreading. Even with the right passage, a model can miss a condition or over-generalise. Citations and a confidence value are diagnostic aids, not proof.
The practical response is a routine: test with real questions, read transcripts, fix the content first and tune settings second.
How does Chyt.ai implement RAG?
Chyt.ai builds the pipeline above into the product, so you work with sources and settings rather than code.
- Sources. Website crawls, PDF, DOCX, TXT, CSV, XLSX and PPTX files, REST APIs, Q&A pairs, and connectors including Notion, Confluence, Jira, Google Drive, SharePoint, Dropbox, GitBook and YouTube.
- Hybrid search and reranking. Vector and keyword search are fused with reciprocal rank fusion, then reordered by a cross-encoder reranker when that service is available.
- Cited answers. Responses are grounded in your content and show their sources. Preview adds diagnostics such as confidence, model, token usage and sources for each test question.
- Retrieval settings you can tune. Settings → Retrieval exposes Top K results, similarity threshold, reranking and chunk size. Change one at a time and retest; a stricter threshold is not automatically better. Chunk size applies to new ingestions only.
- Tiered model routing. With Automatic model selection, simple, standard and complex questions go to different models, which you can override per tier. Pro plans can bring their own keys or a private LLM; see model settings.
If you would rather see it than read about it, the guide to turning your website into an AI chatbot walks through the setup.
When should you add Q&A corrections?
Some answers are too important to leave to whatever the crawler found. Add approved question-and-answer pairs when:
- a fact is missing from your website and documents, such as service hours or an eligibility rule;
- the source states it ambiguously or spreads it across several pages;
- you reviewed a transcript and found a wrong answer. In Chyt.ai, Correct this answer on a conversation saves the right response as input to future answers.
Write answers as complete sentences that include their conditions, then retest the exact question and a paraphrase. Q&A entries inform AI responses; they do not guarantee the answer is repeated word for word.
When is a deterministic flow better than RAG?
RAG is built for open questions with many possible phrasings. It is the wrong tool when the conversation must follow a fixed path or use exact wording:
- collecting details for a quote, booking or support ticket in a set order;
- eligibility checks and triage where each branch must be predictable;
- regulated or legal statements that must appear verbatim;
- high-volume routine journeys where you do not want to spend AI capacity.
A deterministic flow is a scripted conversation of messages, questions, conditions and actions. In Chyt.ai, messages handled entirely by flow nodes use no AI credits, and a flow can hand over to AI answers through a RAG node when the visitor asks something open-ended. Many good chatbots use both. Our comparison of deterministic and AI chatbots goes deeper.
Next steps
The quickest way to understand RAG is to watch it answer from your own pages. Try the demo with your website URL, or start a free trial and use the retrieval and answer quality guide to test and tune your first chatbot.
Frequently asked questions
- What is a RAG chatbot?
- A RAG chatbot is an AI chatbot that uses retrieval-augmented generation: for each question it first retrieves relevant passages from a defined set of content, such as your website and documents, and then has a language model write an answer grounded in those passages, usually with source citations. Chyt.ai is a RAG chatbot platform that builds this from your own content without any model training.
- Does RAG stop chatbot hallucinations?
- RAG reduces hallucinations because the model answers from retrieved facts instead of memory, and citations make answers checkable. It does not eliminate errors. A RAG chatbot can still be wrong when content is missing, outdated or contradictory, when retrieval selects the wrong passage, or when the model misreads a condition. Regular testing, transcript review and a route to a human remain necessary.
- What is hybrid search in a RAG chatbot?
- Hybrid search combines vector search, which matches meaning and handles paraphrases, with keyword search, which matches exact terms such as product codes and names. The two ranked result lists are merged, commonly with reciprocal rank fusion, and often reordered by a reranking model. Chyt.ai uses hybrid vector and keyword search with reciprocal rank fusion and cross-encoder reranking.
- Is RAG better than fine-tuning for a business chatbot?
- For answering questions from business content, RAG is usually the better choice. Knowledge is updated by re-indexing content rather than retraining a model, answers can cite their sources, and no machine-learning expertise is needed. Fine-tuning is better suited to teaching a model a style, format or specialised task than to supplying facts that change.
- How do I improve the accuracy of a RAG chatbot?
- Fix the content before the settings. Make sure the correct, current fact exists in an indexed source, remove conflicting or obsolete versions, and add focused Q&A pairs for important answers. Then test a fixed set of questions and adjust retrieval settings one at a time. In Chyt.ai, Preview diagnostics show the sources, confidence and model used for each answer, which makes it clear whether a problem is retrieval or generation.
- When should I use a scripted flow instead of RAG?
- Use a deterministic flow when the conversation must follow a fixed sequence or use exact wording, such as collecting booking details, running an eligibility check or showing a required statement. Use RAG for open questions with many possible phrasings. Chyt.ai supports both in one chatbot: flow messages use no AI credits, and a RAG node hands over to AI answers when needed.
See it with your own content
Build a demo chatbot from your website or files in a few minutes, or start a free trial and design your first flow.