Pages

The Pages list shows everything the crawler found on your site and which of those pages the AI is allowed to answer from. Open it from Knowledge → Website.

A one-minute tour of the list:

212 pages indexed, last synced 1 hour ago.

Zoomed in: the health line under the title, then Sync now, Filter and Export / llms.txt.

The header line is the whole health check: how many pages are indexed and when they were last read.

#The list

Sort by Last checked to find pages that have not been re-read recently. Use Filter to narrow it down, for example to every page that came back 404.

#Sync now

The button at the top left re-reads the site immediately.

You do not normally need it, because sites re-sync every night on their own. Use it after you publish something you want answered today, or after you change Sync Settings and want to see the effect.

The dropdown arrow next to it holds the stop action for a sync in progress.

#Row actions

The three-dot menu on each row holds these actions:

View content. You see what CrawlBrain stored for that page, after navigation and footers were stripped. This is the fastest way to diagnose a bad answer: if the stored text is empty or full of menu items, the answer was never going to be good, and the fix is in Sync Settings rather than anywhere else.

Open page. The live URL opens in a new tab.

Re-sync page. CrawlBrain fetches this one page again right away. Use it after fixing a single article instead of re-crawling the whole site.

Exclude from AI. The page leaves the answer pool without being deleted. It stays in the index and stays visible in this list, but the AI stops using it.

Delete. This removes the page entirely. It comes back on the next crawl unless you also exclude its path in Sync Settings.

#Excluding pages from answers

Some pages are indexed but should never be the basis of an answer, such as an old pricing page or a duplicated regional variant of a page you already have.

Select them with the checkboxes and use the bulk action to turn AI off for the whole selection. They keep appearing in instant search results, so a visitor who wants them can still find them, but the AI will not write an answer from them.

This is the single most effective fix for "the AI answered from an outdated page".

#Filtering

Filter opens the filter panel, where these are the most useful:

• Status. Active or excluded.
• HTTP code. Find everything that came back 404 or 500 on the last sync.
• Language. On multilingual sites, check each language got indexed.
• Type. Crawled pages versus snippets.

The column selector in the top right adds or removes columns, including word count and HTTP status.

#Exporting

Export downloads your whole indexed content in the format you pick, including llms.txt and llms-full.txt.

Exports have a page of their own: see Exports and llms.txt.

#Page limits

A ceiling caps how many pages one website can index.

When a site hits the cap, the crawler stops adding new pages. Existing pages keep syncing and keep answering.

This is a ceiling, not a bill. Syncing pages costs credit whichever side of the cap you are on, and the cap is simply how far one website may go. See Credits and Limits.

If you are over the cap, exclude what you do not need. Tag archives, author pages, and paginated listings are usually a third of a WordPress site and almost never worth indexing. They also cost credit to sync, so cutting them is worth doing whether or not you are near the cap.

#Why a page might be missing

The causes below are in rough order of how often they happen:

1. It is not in your sitemap and nothing links to it. Add it under Extra pages to include in Sync Settings.
2. It is excluded by path. Check the Exclude tab.
3. It carries a noindex tag. Either remove the tag or turn off Skip noindex pages.
4. It returned an error. Sync History shows the exact status code and the reason.
5. It redirects somewhere else. CrawlBrain follows the redirect and indexes the destination instead, which is correct but can look like a missing page.
6. You hit the page cap.