A bot is only as good as the source it reads. On a legacy website or an older internal system, that source is usually not ready: text inside images and scanned PDFs, menus built in JavaScript, duplicate pages, outdated versions and no metadata. So before you build a bot, you index: extract text and structure, recover dates and owners, remove duplicates, map permissions, build an index in Azure AI Search and test it with real questions. You do not need to replace the old system first.
Why can't you just connect a bot to the old website?
In short: because a bot does not read the system the way a person does. It sees only what made it into the index, and in a legacy system much of the knowledge never gets there, or gets there garbled.
A lot of organizational knowledge lives in websites and systems built ten or fifteen years ago: old content management systems, ASP and WebForms sites, Lotus Notes, SharePoint 2010, file shares and PDF archives. Some of them have no API at all. A long-time employee knows where the current procedure is. A bot does not.
When you connect a bot directly to a source like that, it finds the old version instead of the current one, misses the table inside the scan, and sometimes shows a document the user is not allowed to see. The problem is almost never the model. It is the index. We explain the mechanism itself in the guide What is RAG and how to index enterprise data, and the service page RAG & data indexing for AI gives a short overview of the process. This guide goes into the details.
What problems show up in almost every legacy system?
In short: the same problems come back in almost every project, and each one has a known solution.
| What the bot gets | How to solve it | |
|---|---|---|
| Content inside images and scanned PDFs | A file with no text | OCR and Azure AI Document Intelligence for text, tables and structure, including handwriting |
| Menus and content built in JavaScript | A page with no links and no text | A crawler that renders the page in a browser, or a direct export from the database |
| Session IDs and parameters in URLs | The same page under many URLs, or URLs that expire | URL normalization, removal of unneeded parameters and a stable ID for every document |
| Duplicate pages and outdated versions | Conflicting answers to the same question | Content-based duplicate detection, keeping only the valid version |
| Content behind a login or an internal search | Whole sections it cannot reach | Crawling with a service account, or pulling from the database and export files |
| Mixed Hebrew and English | Search that misses terms and prefixes | A Hebrew language analyzer, synonyms and hybrid search |
How do you prepare a legacy system for a bot, step by step?
In short: nine steps, from mapping to testing with real questions. The old system is not rewritten or replaced.
- Map sources and questions. Which systems and folders are in scope, who owns each one, and which questions users actually ask. The questions decide what gets indexed first.
- Crawl or export. Where there is an API or database access, export directly. Where there is not, a crawler goes through every page and record, including behind old navigation or a login screen, and also saves attachments.
- Extract text and structure. From HTML, remove menus, banners and code, and keep headings, sections and tables. PDFs and scanned documents go through Document Intelligence, including handwriting.
- Recover metadata. For every document: date, owner, document type, topic and validity. Where the system did not store them, extract them from the content, the file path or the title.
- Remove duplicates and outdated versions. Identify identical or near-identical documents and keep only the valid version. A canceled procedure stays out of the index, or goes in with a clear label.
- Map permissions. Translate the old system's permissions into groups in Microsoft Entra ID, and store them on every chunk in the index.
- Chunk and index. Chunking by document structure rather than character count, and an index in Azure AI Search that combines keyword and vector search, with a Hebrew language analyzer and synonyms.
- Keep it updated. An incremental crawl that detects what changed, what was added and what was deleted, and updates only that.
- Test quality. Before go-live, run dozens of real questions, check that the answer and the source are correct, and fix the index based on what failed.
How do you make sure the bot answers only what a user may see?
In short: every chunk in the index carries the permission of the document it came from, and the search filters by the user asking, before the model sees anything.
This method is called security trimming. The user signs in, the system knows which groups they belong to, and retrieval returns only the chunks they are allowed to see. That way an employee in finance and an employee in the field ask the same bot and get different answers, each according to their own permissions.
In legacy systems, permissions are sometimes messy: a folder open to everyone that holds a sensitive document, or a group nobody remembers the members of. Preparing the index is the time to clean them up. For a public-facing bot the rule is simpler: index only content approved for publication.
How does the index stay up to date after go-live?
In short: indexing is a process that runs all the time, not a one-time task.
The old system keeps working, and people keep updating content in it. An incremental crawl compares modification dates or a content hash, updates only what changed, and removes from the index what was deleted at the source. The frequency depends on how fast the source changes: a stable archive is crawled rarely, and content that changes every day is crawled often.
Questions the bot could not answer are collected and passed to the content team. Sometimes a document is missing, and sometimes it exists but is worded far from the way people ask. That is how the index improves over time. If you are considering training a model on the material instead of indexing it, read the comparison of RAG and fine-tuning.
What about the public website? How do search engines and AI engines read it?
In short: the same problems that get in a bot's way also get in the way of search engines and AI engines. A legacy website needs server-rendered text, a clear structure and clear instructions for crawlers.
- Clean HTML and server-rendered text. Content loaded only by JavaScript may not be read. The important text should be in the page as soon as the server sends it.
- One canonical URL per page. No session IDs and no parameters, so duplicate pages do not compete with each other.
- sitemap.xml. A list of all canonical pages, with their last update date.
- robots.txt. An explicit decision about which crawlers are allowed, including AI crawlers such as GPTBot, ClaudeBot and PerplexityBot.
- Schema.org. Structured data that explains what is on the page: organization, article, FAQ.
- llms.txt. A file that summarizes for AI engines what is on the site and where to find it.
This is also how we built the Zooz-Tech website itself.
What does this look like in our projects and products?
Tadiran · Document AI: a system that reads handwritten invoices, photographed receipts and tables, in Hebrew and English, and returns structured data to the ERP. The same kind of extraction turns a scanned archive into text that can be indexed.
KESHER · a bot that answers from approved sources: a bot on the website, on WhatsApp and by voice, that answers only from indexed content and hands over to an agent with the full context when needed.
Frequently asked questions
Do you need to replace the old system before building a bot?
No. Its content is indexed as it is, through export or crawling, and the system keeps working with no change to its code.
What do you do with a system that has no API?
Crawl it the way a user browses it, with a service account, or pull the content directly from the database or from export files.
Will the bot also read scanned documents and handwriting?
Yes, once they go through OCR and Azure AI Document Intelligence. A scanned file that was not extracted stays empty as far as the bot is concerned.
How does the bot avoid exposing restricted documents?
Every chunk in the index keeps the permission of its source, and the search filters by the user asking. This method is called security trimming.
How often is the index updated?
According to how fast the source changes. An incremental crawl updates only what changed, and removes from the index what was deleted at the source.
CTO at Zooz-Tech. Leads development and architecture of AI agents, Generative UI and Azure infrastructure.