What Is RAG: Letting AI Answer From Company Documents, Costs and Common Mistakes

Many businesses want an AI assistant that answers from their policies, price lists and internal documents. General-purpose models have not read them, and pasting the whole archive into every question is costly or does not fit.

This article explains RAG (Retrieval-Augmented Generation). Sources: the original paper by Lewis et al. (2020), documentation from OpenAI, Anthropic and Microsoft, and the OWASP framework, checked as of October 21, 2026.

It covers the steps of a RAG system, when to choose RAG over fine-tuning or long context, why RAG goes wrong, and cost and measurement.

Key points

• RAG adds 1 lookup step before the model answers, within 6 steps from ingesting documents to citing sources. Quality is decided in the early steps, not just the model.
• A base under 200,000 tokens (about 500 pages) can go straight into the prompt (Anthropic, September 19, 2024). RAG fits large, frequently changing or permissioned collections; fine-tuning changes behavior (OpenAI).
• In Anthropic's test, keyword search plus reranking cut missed correct chunks from 5.7% to 1.9%.
• Permissions must be applied at retrieval, not through instructions in the prompt. OWASP lists weaknesses in vector stores as risk LLM08:2025.
• Evaluate with a sample question set (OpenAI suggests 20 or more): measure retrieval, faithfulness and the "I don't know" rate.

What RAG is and why it exists

RAG lets a language model look up outside documents at the moment it answers, instead of relying only on what it learned in training. The original paper by Lewis et al. was posted on arXiv on May 22, 2020 and accepted at NeurIPS 2020.

The paper pairs parametric memory (a seq2seq model) with non-parametric memory (a Wikipedia vector index searched by a retriever). Its motivation: models struggle to access parametric knowledge precisely, to show provenance and to be updated. The results are on Wikipedia in 2020, not Vietnamese business documents.

Which steps does RAG run through

Step 1, ingest documents. Read PDF, Word and web pages into clean text with date, version and the groups allowed to view them. Per Anthropic's documentation, a scanned PDF with no extractable text cannot be cited, so OCR is needed.

Step 2, chunking. Long documents are cut into small chunks. OpenAI defaults to 800 tokens per chunk with 400 overlap; Microsoft suggests starting at 512 tokens with 25% overlap. The numbers differ, so measure.

Step 3, embedding and indexing. An embedding model turns each chunk into numbers encoding meaning, so similar chunks get close vectors. Anthropic has no embedding model of its own and points to Voyage AI; OpenAI and Google do.

Step 4, retrieval. The question is embedded too, to fetch the closest chunks. Anthropic notes that BM25 keyword search helps with identifiers and technical terms; OpenAI blends semantic and keyword scores.

Step 5, reranking. A model re-scores the relevance of each retrieved chunk and keeps only the top few.

Step 6, generate an answer with citations. The model uses only the kept chunks. Anthropic's citations feature returns the passage supporting each claim; the quoted text does not count toward output tokens.

RAG, fine-tuning or long context: when to choose which

Choose RAG when the model lacks knowledge, it is outdated or proprietary data is needed. Per OpenAI's guidance (read October 21, 2026), RAG addresses missing knowledge while fine-tuning addresses consistency of format and tone; the 2 are complementary, and prompt design comes first.

Ovadia et al. (December 10, 2023) found RAG consistently beat unsupervised fine-tuning at injecting knowledge, including new knowledge. Limit: the topics and models of that study.

On long context, Anthropic (September 19, 2024) says a base under 200,000 tokens (about 500 pages) can go in the prompt. Its pricing page (read October 21, 2026) lists models from 4.6 onward, except Haiku 5.5, with a 1 million token window at standard pricing.

Fitting does not mean working well: Liu et al. (July 6, 2023) found accuracy drops when information sits mid-context. Click Star has no equivalent measurement for current models.

5 reasons RAG answers wrongly, and the fixes

When RAG is wrong, find whether the fault is in retrieval or generation before changing the model.

1. Retrieval misses the right chunk. Fix: hybrid queries, reranking, query rewriting.

Anthropic (September 19, 2024) measured how often the correct chunk is missing from the top 20: 5.7% at baseline, 3.7% with contextual embeddings, 2.9% adding BM25, 1.9% with reranking (a 67% reduction). Self-measured, not on Vietnamese.

2. Poor chunking. A chunk cut through a table or clause loses meaning; an oversized one mixes topics. Fix: split by headings, add the document title to mid-text chunks (Microsoft describes this), try 2 to 3 setups and measure.

3. Stale or conflicting documents. RAG answers faithfully from documents even when they are outdated. Fix: attach effective dates, index incrementally, remove superseded versions, show the source date.

4. Wrong access permissions. A chunk sits in the index although the asker may not see it. Fix at retrieval (see the security section).

5. Vietnamese and hard-to-read documents. Embedding models may do worse on Vietnamese; tables and scans break in extraction. Fix: test with real Vietnamese questions, use OCR, keep table structure.

Generation can fail too: with the right chunk, the model can still misread it. Anthropic advises allowing "I don't know", quoting verbatim first for long documents and retracting claims without a citation; these only reduce hallucination, not eliminate it.

RAG in Vietnamese: what is known and not known

Click Star found no public measurement of retrieval accuracy on Vietnamese business documents. VN-MTEB (arXiv, July 29, 2025; Findings of EACL 2026) has 41 datasets and 18 embedding models, but is built by translating MTEB samples, not real business text.

There is also no official table of tokens per Vietnamese page, so count on real documents. Click Star's suggestion (no measured source): test questions with and without diacritics, internal codes, abbreviations, and compare 2 embedding models.

What a RAG system costs: formulas and an illustrative example

Formula 1, embedding documents: source tokens times the overlap factor, divided by 1 million, times the embedding price. With OpenAI's defaults (800-token chunks, 400 overlap), embedded tokens are roughly double the source tokens (Click Star's calculation).

Formula 2, each question: (retrieved chunk tokens + question + instructions) times the input price, plus answer tokens times the output price, each divided by 1 million.

Published prices read on October 21, 2026, per 1 million tokens: OpenAI embeddings cost USD 0.02 (text-embedding-3-small) and USD 0.13 (text-embedding-3-large).

Anthropic's Claude Sonnet 5.5 costs USD 2 input and USD 10 output; a prompt cache read is USD 0.10.

Illustrative example, with assumptions set by Click Star and not measurements: a 10 million token base; each question retrieves 5 chunks of 800 tokens plus 500 tokens of question and instructions (4,500 input), with a 300 token answer.

• Embedding about 20 million tokens: USD 0.40 with the small model, USD 2.60 with the large one.
• Each question on Sonnet 5.5: USD 0.009 input plus USD 0.003 output, USD 0.012 in total; 1,000 questions cost USD 12.
• Long context, a 150,000 token base in every Sonnet 5.5 prompt: the base alone costs USD 0.30 per question, or USD 300 per 1,000 questions. With caching, each read is USD 0.015.

The example excludes cache write fees, vector storage, reranking and labor; use real measurements.

Security and per-user permissions

Permissions must be enforced at retrieval, before the model sees any chunk. Once another department's chunk is in the prompt, "do not reveal" is not a reliable control.

OWASP (LLM08:2025) lists the risks: unauthorized access leading to leakage, cross-context leaks between groups in a shared store, embedding inversion that recovers source information, and data poisoning.

Microsoft describes a security filter pattern: each document carries a field with the identities allowed to view it, and every query adds a filter for the asker.

What to do: store permissions with each chunk, filter by the logged-in user's real identity, and test with accounts of different rights. Click Star already has an article on protecting customer data in AI analysis; this one gives no legal analysis.

How to evaluate a RAG system

OpenAI advises starting with 20 or more questions with reference answers, then reviewing failures to form hypotheses. Ragas (Es et al., September 26, 2023) evaluates RAG on 3 aspects: relevance of retrieved passages, faithfulness to them, and answer quality.

Sample question set. Real questions with answers and the passage holding each answer; include answers in 1 chunk, in several, absent from the documents, conflicting documents and questions needing different permissions.

Retrieval accuracy. The share of questions whose correct chunk is among the k retrieved (recall@k).

Faithfulness to sources. Does each claim have a supporting passage, and does the citation say that. A reviewer reads a sample, or a model grades and a person checks.

The "I don't know" rate. Measure 2 directions: are out-of-scope questions correctly declined, and are in-scope questions wrongly declined.

Safety and operations. Count of out-of-permission leaks (target 0), latency, cost per question and the age of cited documents. No sourced standard thresholds exist; the business sets its own by risk.

Click Star's view. RAG is a search problem first and a model problem second. Most effort should go into document quality, chunking, retrieval, permissions and the measurement set, because that is the part a business controls.

For small and mid-sized businesses, start narrow: 1 document group, 1 group of askers, 20 to 50 real questions and a reviewer on the output.

Click Star offers AI integration and AI automation services and so has an interest here; the article therefore leans toward self-check criteria, even if the answer is that documents in the prompt suffice. Click Star already has articles on website AI chatbots and operational databases.

What to do: a 30-day plan

• Days 1 to 7: pick 1 document group; record owner, effective date and viewing rights; write 20 to 50 real questions, including some outside the documents.
• Days 8 to 14: build a simple pilot; count tokens on real documents; if under 200,000 tokens, also try the prompt-only option.
• Days 15 to 21: try 2 chunking setups and 2 embedding models; add keyword search and reranking; measure recall@k and faithfulness.
• Days 22 to 30: add permission filtering and "I don't know" answers; compute cost per question from measurements; let a small group try it, then decide.

Common mistakes

1. Evaluating with a few hand-tried questions instead of a reference set rerun after every change.
2. Switching to a larger model when the fault is in retrieval or chunking.
3. Loading old versions, drafts and duplicates into the index with no effective date.
4. Relying on prompt instructions to keep secrets instead of filtering permissions at retrieval.
5. Forcing the model to always answer, with no "I don't know" exit.
6. Trusting figures measured in English without testing in Vietnamese.
7. Costing only the model price, ignoring re-embedding, storage and labor.

What is not known

There is no public measurement of Vietnamese retrieval on business documents, nor of how many Vietnamese businesses have deployed RAG. Anthropic's retrieval figures were produced by the company itself.

Click Star read the Anthropic, OpenAI and Microsoft pages through a summarizing tool, so check wording on the original pages. Click Star did not read Voyage AI's price list. Legal duties around personal data in a vector store are not analyzed here.