RAG vs. the Simple Chatbot
A decision playbook for choosing the right knowledge source behind an AI assistant.
Section One
The Core Idea
This is not a choice between "a chatbot" and "RAG." Both are chatbots. The only real difference is where the assistant gets its answers, and that is driven by what the underlying data looks like, not by which approach sounds more advanced.
Pick the knowledge source that matches the data. Anything more is cost and complexity carried for no reason.
The one-line test
If the knowledge fits in a prompt, prompt it. If it is records you filter, query it. If it is a pile of documents you search, that is RAG.
Why this matters
RAG has real moving parts: an embeddings model, a vector table, a reranker, and a live database connection. That machinery earns its keep in exactly one of the three buckets on the next page. Reach for it because the data demands it, not by default.
Section Two
The Three Buckets
01
Step One
Small, fixed facts
A finite, known set of answers — an FAQ, a service list, a set of policies that rarely change. Put the facts directly in the system prompt. No database, no embeddings, nothing to maintain. The answers should come back exact every time; retrieval only adds fuzziness here.
02
Step Two
Structured records you filter
Rows with fields to sort and filter — listings, inventory, suppliers, pricing, member directories. Give the assistant a database lookup tool that queries the table directly. "Show records in region X under this price" is an exact filter; semantic search returns the most similar-sounding rows, not a clean filtered list.
03
Step Three
Large, unstructured documents
Prose too big to paste into a prompt — contracts, manuals, transcripts, policy PDFs, reports. Build Retrieval-Augmented Generation over a vector store: chunk it, embed it, retrieve only the few relevant chunks per question. The knowledge is too large for context and too unstructured to query with fields.
Identify the bucket first. The build follows.
Section Three
Quick Reference
| Data Looks Like | Right Build | Wrong Move |
|---|---|---|
| Known, finite facts | Facts in the system prompt | RAG — overkill, adds fuzziness |
| Structured records to filter | Database lookup tool | RAG — imprecise on fields |
| Big pile of documents | RAG vector store | Prompt stuffing — won't fit |
The buy signal
On a discovery call, listen for this sentence: "The answer is in there somewhere, but nobody can find it." That points at bucket three — a shelf of policy documents, years of contracts, a long program manual. If the client instead describes a set list of answers or a spreadsheet of records, they are in bucket one or two. Do not sell them RAG.
Section Four
Where RAG Pays Off
RAG carries real moving parts: an embeddings model, a vector table sized to match, a reranker, a live database connection, and the discipline to keep embedding dimensions consistent across ingestion and retrieval. A mismatch quietly breaks results. That cost is worth it for bucket three and only bucket three — for buckets one and two it is slower, costlier, and less precise than the simpler build.
▸ Where it fits
- Professional-service firms — years of contracts and playbooks, “have we done a deal like this” questions
- Government & economic development — grant guidelines, zoning rules, RFP requirements
- Healthcare operations — protocols, onboarding manuals, compliance documents (the paperwork, not clinical decisions)
- Nonprofits — program manuals, donor reports, board materials; keeps institutional memory answerable through staff turnover
- Any business with a real knowledge base — product manuals, warranty terms, an internal wiki
When a simple build wins: a compliance FAQ with a fixed set of answers goes in the prompt. A catalog people filter and sort gets a database lookup. Anything that fits comfortably in one prompt does not need infrastructure.
Section Five
How We Actually Build It
This isn't theoretical. It's the exact shape of Soulwire's own bucket-three build — two linked n8n workflows, one that ingests documents and one that answers from them.
Fig. 2 — the real build, two workflows
Watch
Google Drive
Polls a folder every minute for new files
Chunk + Embed
OpenAI embeddings
Recursive text splitter, 200-char overlap
Store
Supabase
Vector table, insert mode
Answer
gpt-4.1-mini agent
Retrieves top 20, reranks with Cohere, replies over a webhook
Two pieces most guides skip. A reranker (Cohere) narrows the top 20 vector matches down to the ones actually worth answering from — retrieval alone returns similar-sounding chunks, not necessarily the right ones. And Postgres-backed session memory (45 messages of context, keyed by a session ID from the incoming request) is what makes it a conversation instead of a one-shot lookup.
The system prompt only allows one move: call the knowledge base tool first, answer only from what it returns, and say so plainly if the answer isn't in the retrieved chunks. Same grounding rule as every build in this library — never invent what the source doesn't say.
“On a discovery call, listen for this sentence: “The answer is in there somewhere, but nobody can find it.””
— RAG vs. the Simple Chatbot, p. 4
§ Fin.
Rather have it built for you?
This guide is one system from the Soulwire Studio workflow library. We build AI automations for businesses that run on process: intake, scheduling, proposals, compliance-heavy operations. If you'd rather spend your time closing deals than debugging automation, book an audit and we'll find the automation hiding in your business.
