RAG is the way to let an AI model answer from your organization's own data, with a source. Its success depends almost entirely on indexing: how scanned documents and images are read, how they are split, how permissions are preserved, and how the index stays up to date. A good model will not save a bad index.
RAG in two minutes
RAG (Retrieval-Augmented Generation) is a two-step process. First, retrieval: the system searches a knowledge store for the passages relevant to the question. Then, generation: the model writes an answer using only the passages it found, and cites where they came from.
For retrieval to work, every passage is also stored as a vector, a numeric representation of its meaning. That is what lets the system find "When is yard waste collected?" even when the document says "garden refuse pickup days". The store that holds the vectors is called a vector database.
Four mistakes that turn a good bot into a bot that guesses
1. Scanned documents nobody read
A scanned PDF enters the index as an empty image. Without OCR and Document Intelligence, half of the knowledge simply does not exist as far as the bot is concerned.
2. Splitting by character count
A price table cut in half, or a section whose heading stayed in the previous chunk. We split by document structure.
3. An index without permissions
If each chunk does not carry the permissions of its source, the bot may show an employee a document they are not allowed to see.
4. An index that does not update
The website changed, the document was revised, and the bot still answers from the old version. Indexing must be an ongoing process, not a one-time task.
Images, handwriting and tables
Organizations hold a lot of knowledge in places that are hard to search: photos of license plates, handwritten invoices, diagrams and images on the website. We run them through OCR, handwriting recognition and layout analysis, and add descriptions to images, so that even text inside an image gets indexed. See, for example, Tadiran's Document AI project.
What is special about Hebrew
- Prefixes: in Hebrew, "and", "the" and "in" attach to the start of a word, so a search for "arnona" (municipal property tax) must also match "and in the arnona". We handle prefixes such as ve-, ha- and be-.
- Synonyms and everyday language: "yard waste", "garden refuse" and "branches" are the same thing to a resident.
- Hebrew and English mixed in the same sentence, especially in technical terms.
How to choose a vector database
In our projects, Azure AI Search serves as both a vector and a hybrid search store: semantic search, keyword search and synonyms, together with security trimming and permissions. It runs in the organization's own Azure tenant and serves several bots and agents from the same index, in both Copilot Studio and Azure AI Foundry.
Checklist before you start
- Which sources go in, and who owns each one.
- Which of them are scanned, images or handwriting.
- Who is allowed to see what.
- How often the sources change.
- The 50 questions users ask most often. They are the baseline for quality testing.
Founder & CEO of Zooz-tech, solutions architect, Microsoft 365 Architect Expert.