Your chatbot's knowledge base is only as good as the crawl that built it

"Point it at your website and it learns your business" is now a standard feature for AI assistants. The tutorial version takes an afternoon: fetch pages, split the text, embed the chunks, store them in a vector database, retrieve the top few at question time.

We build production AI systems, and we run one of our own: ConGreeto, a multi-tenant website assistant that small businesses embed on their sites. Its whole value depends on that "learn the business from its website" step. A large share of its target customers are local service businesses, such as PPF, ceramic coating and detailing shops, whose websites were built on Wix, Squarespace, Framer or a custom React template.

The tutorial pipeline fails quietly on exactly those sites. These are our notes on where it failed, what we changed, and what we'd tell any team building the same thing.

The running example

Imagine a paint protection film shop. Its website has a home page, separate pages for PPF, ceramic coating and paint correction, a pricing page with coverage packages, a gallery, and an FAQ. At 10 PM a visitor asks the assistant:

"I drive 80 miles a day on the highway. Do I need full front PPF, or is ceramic enough?"

A good answer needs the shop's own PPF page (what full front covers), its ceramic page (what a coating does and doesn't protect against), and ideally its price ranges. If any of those pages never reached the knowledge base, the model either answers in generic terms or makes something up. Neither is acceptable for a business that will be judged on the answer.

Failure 1: the crawl found one page

Our first crawler did what most do. It fetched the home page, parsed the <a href> links in the HTML, and followed them within the domain.

On a server-rendered site this works. On a client-rendered site, the server sends an almost empty HTML shell and JavaScript builds the navigation in the browser. The crawler saw no links, so it indexed only the home page and reported success. Nothing errored. The dashboard showed a completed crawl. The bot simply didn't know the business offered paint correction.

We found it while testing on our own site, brillnexsystems.com, which is a client-rendered Next.js app.

The fix: seed the crawl from the sitemap. Nearly every site builder generates a sitemap, and the sitemaps protocol lets robots.txt point to it with a Sitemap: line. The crawler now:

Reads robots.txt for Sitemap: entries, then tries the common locations (/sitemap.xml and similar).
Follows nested sitemap indexes, including gzipped ones.
Seeds the crawl frontier with those URLs before following links.
Treats www and the bare domain as the same site, and de-duplicates on canonical URLs.

On our own site, pages discovered went from 1 to 38, and 39 ended up indexed in production. That part was plain Python, with no new infrastructure.

Lesson: "crawl finished" doesn't mean "crawl found the site". Record pages discovered and pages indexed for every crawl, and show both to the customer.

Failure 2: the pages were found, but empty

Discovery fixed the page count. The content was still wrong. On client-rendered pages, the real copy sits inside <script> payloads, not in the HTML text. A plain HTTP fetch of our /services/ai-consulting page produced 83 characters: the page title and little else.

The fix: render only when needed. Running a headless browser on every page is slow and memory-hungry, so we only do it when the server HTML is thin:

If a page's extracted text is under a threshold (SCRAPPER_WEB_RENDER_MIN_CHARS, default 200 characters), it's re-fetched through a headless renderer.
The renderer sits behind a small interface with two backends: self-hosted Playwright Chromium, or a hosted rendering API. We run the self-hosted one to keep costs at zero, and the hosted option exists for platforms where installing a browser is awkward.
Before capturing, the renderer scrolls down the page in steps, because many templates only mount sections when they scroll into view. Without that, you capture the hero banner and miss the pricing table.
If no renderer is available, the crawl falls back to HTTP-only instead of failing.

After rendering, that same /services/ai-consulting page produced 4,517 characters in 12 chunks.

The SPA edge case

One more class of site broke even this. Some single-page apps are deployed without server rewrites, so their sub-pages exist only as client-side routes. Fetching /pricing directly returns a real 404, even from a browser. The page is only reachable by clicking a link inside the running app.

For those, the renderer loads the app once, clicks each failed link, checks that the router actually reached the target path, and captures the result. It's capped at 15 routes per site and only runs when rendering was used and some URLs failed. Thin "not found" views are dropped.

The security cost of a browser

A headless browser is a server-side request forgery risk. Our HTTP client resolves and checks every destination before connecting, so a customer can't point the crawler at internal addresses. Chromium makes its own network requests and bypasses that check. We added per-URL revalidation in front of the renderer. If you add a browser to an ingestion pipeline, you have reopened your SSRF surface and need to close it again (OWASP SSRF Prevention Cheat Sheet).

Failure 3: every re-crawl re-embedded the whole site

Businesses change their sites. Prices move, services get added, a summer promotion appears. The knowledge base has to follow, so re-crawls are routine, and the naive approach re-embeds everything every time.

We now check for change at three levels before paying for an embedding:

Raw body hash. A SHA-256 of the fetched bytes, plus the ETag where the server provides one. If unchanged, skip the page.
Extracted-content hash. Many pages change bytes on every request because of build IDs, CSRF tokens or rotating markup. We hash the extracted text. If only the markup changed, we update the page's metadata and don't re-index it.
Content-addressed embedding cache. Each chunk's embedding is stored under a key of normalized text + model ID + scope. A chunk that reappears, whether on another page, in a later crawl or in an uploaded document, is a cache hit, not a new API call. The model ID is part of the key, so a future embedding-model upgrade is a controlled backfill and can't happen as an accidental full re-embed.

The cache writes use ON CONFLICT DO NOTHING. Two workers embedding the same new text both compute it, the second insert does nothing, and both end up with a valid vector. That's simpler than locking and harmless at this scale.

Failure 4: a large site took down the worker

A customer site with 291 pages produced about 7,200 chunks. Our first design embedded the whole crawl in one batch. The result came back as a Python list[list[float]], which for 7,200 vectors of 1,536 dimensions (text-embedding-3-small's default size) is about 350 MB of object overhead alone, before the chunk text and page bodies.

The worker ran at 128% of a 512 MB dyno. On Heroku, R14 is a warning, but R15 kills the process. A killed crawl leaves the job marked as running with a half-built knowledge base, which is worse than a clean failure.

The fix was to process the crawl in slices of 40 pages (CRAWL_EMBED_BATCH_PAGES). Each slice is chunked, embedded and committed, and then its vectors go out of scope. Peak memory is now bounded by the slice size, not the site size. Ordering is preserved, because later steps (including billing per crawled page) rely on the results lining up with the page list.

Lesson: Python floats are 24-byte objects, not 4-byte numbers. If a pipeline holds embeddings in plain lists, its memory grows with the largest customer, not the average one.

Failure 5: the crawl overwrote facts a human had fixed

Prose is forgiving. Facts are not. Our assistant also extracts structured items from pages, such as services, packages and prices, so it can answer "how much is full front PPF?" from a field instead of from fuzzy text.

That created a new problem. The shop owner corrects a price in the dashboard. The next nightly crawl reads the old price from a page the owner hasn't updated yet and writes it back. The bot quotes the old price, and nothing in the logs looks wrong.

Our first fix was "a human edit always wins". That was also wrong. When the owner does update the website, the new price should flow through, and a permanent lock stops it.

The rule we use now is recency, not authority. For each field we store what each source last reported, even when that write was refused:

If the crawl reports the same value it reported last time, it's stale. The human's edit stands.
If the crawl reports a changed value, the website has been updated since the edit. The crawl wins.
Availability is exempt from locking. A sold-out or discontinued item must leave the bot's answers as soon as the website says so, whatever anyone edited.

Two guards matter just as much:

Never conclude an item vanished from an empty extraction. A rate-limited model call returns nothing, which looks the same as a deleted catalogue. If you act on it, you wipe a customer's catalogue during a provider outage.
"Vanished" is decided per page URL that disappeared from the site, not per item that wasn't re-extracted. We only re-extract changed pages, so "not seen this run" is the normal state for most items.

Retrieval: smaller than you'd think

With good ingestion, retrieval turned out to be cheap. Measured from real replies:

Chunks are capped at 300 words with a 50-word overlap, split on paragraph boundaries. Most chunks are far below the cap, averaging about 104 tokens.
We retrieve the top 5 knowledge chunks and the top 6 structured items. That whole retrieval block costs about 1.2k tokens per reply.
Across 171 recorded replies, the full prompt averaged 3,757 tokens (median 3,048).

The biggest cost saving didn't come from retrieving less. It came from prompt block order. OpenAI's prompt caching reuses the longest unchanged prefix, and only once that prefix reaches 1,024 tokens. Moving the instruction blocks that never change above the per-conversation summary grew the cached prefix from 1,161 to 2,006 tokens and cut the full-price part of each reply by about a third. No prompt text was removed.

Ingestion also changes what the prompt must say. We found the bot deflecting "what's your phone number?" into its lead-capture routine even when the retrieved chunk contained the number. The retrieval header now tells the model to answer business contact questions directly from the retrieved text. On a website assistant, the business's own phone number is the most important fact in the knowledge base.

The checklist

If you're building "learn the business from its website" for small-business customers:

Seed from the sitemap. Link-following alone misses most modern sites.
Render only thin pages, scroll before capturing, and fall back to HTTP if rendering fails.
Revalidate every browser request for SSRF. A headless browser bypasses your HTTP client's checks.
Record discovered vs indexed pages and show both. A one-page crawl must never look successful.
Detect change at three levels: raw bytes, extracted text, and a content-addressed embedding cache keyed by model.
Bound memory by slice size, not site size.
Keep structured facts separate from prose, resolve conflicts by recency, and exempt availability.
Never act on an empty extraction.
Measure tokens per reply before optimizing. Then fix block order before cutting retrieval.

Frequently asked questions

Why not just use a hosted crawling API? For many teams that's the right call, and our renderer supports one as a drop-in backend. We run Chromium ourselves for cost reasons at our current scale. The other problems in this post (change detection, memory, fact conflicts, empty extractions) remain whichever renderer you use.

Do I need a headless browser at all? Only for client-rendered pages. Rendering just the pages whose server HTML is thin keeps the cost low on sites that don't need it.

How often should a business website be re-crawled? Less often than you'd expect, provided change detection is cheap. With hashes at three levels, a re-crawl of an unchanged site costs fetches and almost no embeddings. The harder problem is deciding what happens when the site and a manual edit disagree.

Should the chatbot quote prices from the website? Ranges and "what affects the price", yes, from structured fields with known provenance. A final quote for a variable job, such as PPF coverage on a specific car, should still come from a person.

Sources & further reading

Sitemaps XML format: sitemaps.org protocol
Playwright for Python: Microsoft
Server-Side Request Forgery Prevention Cheat Sheet: OWASP
Embeddings guide: OpenAI API documentation, accessed October 2026
Prompt caching: OpenAI API documentation, accessed October 2026
Heroku error codes (R14, R15): Heroku Dev Center, accessed October 2026

From us

Multi-Tenant AI SaaS Architecture: Lessons From Production: the queues and connection budgets this pipeline runs inside
What an AI Agent Actually Costs to Run in Production: the token economics of the same system
12 Questions Your PPF and Ceramic Coating Website Should Answer: the customer side of the running example
Talk to our engineering team

Building a Chatbot Knowledge Base From a Website: What the Crawl Gets Wrong