Pages and Snippets API

These endpoints push content into a website's index without waiting for a crawl.

All endpoints need an Api-Key header and Pro. The base URL is https://api.crawlbrain.com/v1.

#Pages versus snippets

Pages are real URLs on your site. Adding one tells CrawlBrain to fetch and index that URL now, instead of waiting for the nightly crawl.

Snippets are content that has no public page. You supply the text yourself. Use them for anything that exists in your database but not on a URL, such as FAQ answers or policy details.

Both end up in the same answer pool.

#Add a page

POST /page/add

curl -X POST https://api.crawlbrain.com/v1/page/add \
  -H "Api-Key: cb_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{"id":"9f2c1a4e...","url":"https://example.com/new-product"}'

CrawlBrain fetches the URL and indexes its text. Once the website has been synced at least once, the page is available in answers within a minute or two. Before the first sync it is indexed with the rest of the site.

The URL is normalised before it is stored: it is lowercased, and the fragment and tracking parameters such as utm_* and gclid are removed. Use the same form when you later call /page/save or /page/delete.

Fire this from your CMS's publish hook and new content answers immediately instead of tomorrow.

#Update a page

POST /page/save

It takes the same fields as /page/add and re-fetches a URL that is already indexed. Call it from your update hook.

It also brings back a page you removed with /page/delete. A URL that was never added returns 404 with page_not_found.

#Delete a page

DELETE /page/delete

This removes the page from the index and from answers. It stays removed through later syncs until you call /page/save to restore it.

#List pages

GET /pages/list

The response lists crawled pages for that website, most recently updated first. Snippets are not included; use /website/synced?type=article for those.

{
  "success": true,
  "count": 1,
  "pages": [
    {
      "url": "https://example.com/pricing",
      "title": "Pricing",
      "language": "en",
      "http_code": 200,
      "processed": 1,
      "ignore": 0,
      "updated_at": "2026-10-07 09:12:44"
    }
  ]
}

processed is 1 once the page has been fetched. ignore is 1 for removed pages.

#Add a snippet

POST /snippet/add

curl -X POST https://api.crawlbrain.com/v1/snippet/add \
  -H "Api-Key: cb_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "id": "9f2c1a4e...",
    "url": "https://example.com/faq#shipping-canada",
    "title": "Do you ship to Canada?",
    "content": "Yes. Standard delivery is 5 to 8 business days, free over $80.",
    "type": "article",
    "language": "en"
  }'

The url and title decide how useful a snippet is:

The url is the identity. It is unique per website and it is what a visitor clicks in the sources list. Use a real anchor on a real page where you can, and a stable synthetic URL where you cannot.

The title should be the question. Do you ship to Canada? retrieves far better than Shipping, because titles are matched against the visitor's question and visitors phrase things as questions.

A URL that is already taken returns 409 with snippet_already_exists. Use /snippet/save for that snippet instead.

Indexing a snippet costs credit. With an empty balance the call returns 402 with no_credits. A URL that matches an excluded path in Sync Settings returns 422 with excluded_url.

#Update a snippet

POST /snippet/save

This endpoint only updates. It will not create a missing snippet: the call returns 404 with snippet_not_found, and the type of an existing snippet cannot be changed.

#Delete a snippet

DELETE /snippet/delete

The call returns 404 with url_not_found when that website has no snippet at the URL.

#Products

Shops can send exact product data instead of leaving CrawlBrain to read it from the page. The official shop plugins do this for you.

POST /products/sync

Each product:

{ "success": true, "saved": 2, "skipped": [{ "url": "https://other.example/x", "reason": "other_host" }], "total": 148 }

This data drives product cards, sale and price ranking, stock, and the category and attribute filters in search. It takes priority over what the crawler reads from the page for 30 days after each sync, so sync again whenever a product changes. It costs no credit. A website holds at most as many products as its page limit; extra products are skipped with limit_reached.

The product page text is still indexed by the crawler, or by /page/add, for AI answers.

DELETE /products/delete      { "id": "...", "urls": ["https://example.com/product/old"] }

This removes products the shop no longer sells.

#Upsert without guessing

There is no combined endpoint, so try /snippet/add first and fall back to /snippet/save when it returns 409:

async function upsertSnippet(payload) {
  const headers = {
    'Api-Key': process.env.CRAWLBRAIN_KEY,
    'Content-Type': 'application/json',
  };
  const base = 'https://api.crawlbrain.com/v1';

  let res = await fetch(`${base}/snippet/add`, {
    method: 'POST', headers, body: JSON.stringify(payload),
  });

  if (res.status === 409) {
    res = await fetch(`${base}/snippet/save`, {
      method: 'POST', headers, body: JSON.stringify(payload),
    });
  }

  return res.json();
}

#Rate limits

Every write on this page is in the expensive bucket: 20 requests per minute. Each one dispatches an embedding job.

For a bulk import, sleep between calls rather than firing in a loop:

for (const item of items) {
  await upsertSnippet(item);
  await new Promise(r => setTimeout(r, 3200));   // ~19/min
}

For a one-off import of existing material, the document import screen is faster and has no rate limit to respect.

#Credit

Adding a page or snippet is paid from the account's credit balance, at the same price as a page synced by the crawler. With an empty balance, /page/add and /snippet/add return 402 with no_credits. Updates and deletes do not check the balance.

#Patterns worth stealing

Publish hook. Map your CMS's publish event to POST /page/add and its update event to POST /page/save. Wire delete to DELETE /page/delete as well and the index is never stale.

Product catalogue sync. A nightly job reads your product database and upserts one snippet per product, with a question-shaped title such as Is the Acme Widget in stock? and the current answer. Stock and price stay accurate without touching your storefront.

Policy sync from a single source. Keep shipping and returns policy in one system and push it as snippets to every website in your account, so one update corrects every site.