How CrawlBrain Works

CrawlBrain is an AI search engine that crawls and indexes your website content and replaces your existing search bar with one that understands meaning, not just keywords.

Four things happen, and you take part in two of them: entering your URL and adding the widget.

The four parts in 51 seconds:

#It learns from your website content

You enter your website URL. CrawlBrain starts from your sitemap and reads every page it is allowed to read, from product pages and FAQs to documentation and shipping policies. You choose what to include.

Each page is fetched and stored as clean text, with navigation and footers stripped out. Then it is turned into an embedding, a mathematical representation of what the page means.

This is why there is nothing to upload: your site is already the source of truth. You can still add documents such as PDFs or CSV files at any time through Importing Documents.

CrawlBrain checks for new or updated content every night after the first crawl, adding new pages and re-reading changed ones. Pages deleted from your site drop out.

#It understands what visitors mean

Visitors can search in their own words and their own language, misspellings included, and still find the right information.

When a visitor searches, CrawlBrain looks for the content that means the same thing as their question, whether or not it uses their exact words. That one difference fixes the five ways a traditional search box fails:

Search is a hybrid of two methods. Vector search covers meaning while keyword matching covers exact terms like SKUs and error codes, and the two result sets are fused so that a product code still matches exactly and a vague question still finds the right page.

#It upgrades your existing search

You add one script tag, or a plugin, and CrawlBrain works with the search bar you already have. There is no redesign. See Placements.

#It answers, with sources

Relevant pages appear as the visitor types. That is the search layer, and it is fast and light on credit.

When they submit a question, the answer layer runs. CrawlBrain hands the most relevant parts of your content to a language model and gets back a direct answer based on those parts only.

Every answer links to the pages it came from.

Two rules apply to every answer:

• It only answers from your content. It never draws on the open web or general knowledge: if your site does not say it, the answer does not claim it.
• Every answer shows its sources. Your visitor verifies it in one click, and so can you.

When the content is not there, CrawlBrain says so and logs the question as a content gap.

#The parts, named

You will see these words throughout the dashboard.

Website. Each domain that CrawlBrain reads is one website, and subdomains count as separate websites. Every website keeps its own index and widget, along with its own conversation log.

Pages. These are what the crawler found and indexed. See Pages.

Snippets. Snippets are content you add by hand for answers that are not on a public page, such as a shipping exception or an FAQ answer you never wrote up. See Snippets.

Agent. The agent is the search widget for one website, and each website gets exactly one, created automatically. It carries the widget's settings, from placement and colors to texts and model tier, plus its public Agent-Key.

Search. A search is the instant, as-you-type query, and it uses a little credit.

Answer. An answer is a full AI response with source links. It uses much more credit than a search, depending on the AI model, which is why the widget only calls it when someone presses Enter. See Answers and Searches.

#Conversation memory

CrawlBrain understands follow-up questions, because the widget keeps the thread for each visitor's session. Someone who asks about shipping to Europe can follow with "how much for a sofa?" without naming the country again, and a question about a product can be followed by "Does it come in another size?"

You see the same thread on your side, in Conversations.

#Multi-model reliability

CrawlBrain works with Claude, GPT, Gemini, and DeepSeek. If one provider becomes unavailable, requests are routed to another in milliseconds, so your search stays available without any action from your visitors.

You pick a tier, not a model. See AI Model and Tone.

#What CrawlBrain is not

It is not a chatbot. A chatbot asks people to open a bubble in the corner and start a chat. CrawlBrain upgrades the search bar they already use. There is no persona to write and no scripted flow or decision tree to build.

It is your website search, upgraded with AI: a visitor types a question into the search bar you already have, and your own pages answer it with source links.