Websites API

These endpoints manage the websites on your account, from adding and syncing one to clearing its index or deleting it. All of them need an Api-Key header and Pro.

Base URL: https://api.crawlbrain.com/v1

#List websites

GET /website/list

This returns every non-deleted website on the account, and it is where you get the id that every other endpoint wants.

curl https://api.crawlbrain.com/v1/website/list \
  -H "Api-Key: cb_your_key_here"

{
  "success": true,
  "websites": [
    { "id": "9f2c1a4e...", "domain": "https://example.com" },
    { "id": "3b7de901...", "domain": "https://docs.example.com" }
  ]
}

The id is opaque and stable. Store it next to the domain in your own system.

#Add a website

POST /website/add

curl -X POST https://api.crawlbrain.com/v1/website/add \
  -H "Api-Key: cb_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com"}'

{ "success": true, "id": "9f2c1a4e..." }

A search widget is created automatically with the website, so there is no separate call for it. The website itself is not synced until you call /website/sync.

The duplicate check compares the URL as sent, so https://example.com and https://www.example.com count as two websites. Send the form your site actually uses.

#Trigger a sync

POST /website/sync

curl -X POST https://api.crawlbrain.com/v1/website/sync \
  -H "Api-Key: cb_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{"id":"9f2c1a4e..."}'

This queues a crawl and returns at once, without waiting for the crawl to finish. Poll /website/synced for status. Calling it again while a sync is running queues another crawl, so check syncing first.

The endpoint is rate limited to 20 calls per minute. Sites already sync nightly on their own, so call this only when you have published something that needs answering today.

#Sync status and content

GET /website/synced

curl "https://api.crawlbrain.com/v1/website/synced?id=9f2c1a4e..." \
  -H "Api-Key: cb_your_key_here"

{
  "success": true,
  "domain": "https://example.com",
  "synced": true,
  "syncing": false,
  "urls": {
    "https://example.com/pricing": "webpage",
    "https://example.com/contact": "webpage"
  }
}

Poll no more than once every 30 seconds, since a crawl takes minutes.

#Update sync settings

POST /website/crawler

These fields mirror the Sync Settings screen. Send only the ones you want to change, and keep in mind that 0 and an empty string are real values: an empty exclude_urls clears the list.

Changes apply from the next sync. Four settings have no API field and are set on the Sync Settings screen only: JavaScript rendering, crawl speed, language and the category, tag and shop page switch.

#Sync log

GET /website/log

{
  "success": true,
  "log": [
    { "type": "ERROR", "http_code": "404", "url": "https://example.com/old-page", "message": "Not Found" },
    { "type": "INFO", "http_code": "200", "url": "https://example.com/pricing", "message": null }
  ]
}

The log holds the same data as Sync History, one row per URL attempted with its status and the reason it failed. http_code is a string, and message is empty on INFO rows. A WARNING row, returned only when type is omitted, records a change the sync made on its own, for example dropping to 1 page at a time after firewall blocks.

Wire this into your own monitoring if you can: a sudden run of 403s means something in front of your site started blocking the crawler.

#Clear a website's index

DELETE /website/clear

This removes all indexed content of that type. The website keeps its widget and its key, and the settings you chose for it are not touched. Pages you removed earlier stay removed, and the rest of the index is rebuilt.

Use it when you have restructured a site and want a clean re-crawl rather than a merge of old and new URLs.

The widget answers nothing until the next sync completes, so run this at a quiet hour.

#Delete a website

DELETE /website/delete

The widget stops answering immediately. Data is recoverable for seven days, then purged.

#Extension sites

GET /extension/sites

This returns your websites paired with their public agent keys, which is what the browser extension uses to mount the right widget on the tab you are looking at.

{
  "success": true,
  "sites": [
    {
      "id": "9f2c1a4e...",
      "domain": "https://example.com",
      "host": "example.com",
      "agent_key": "uuid-..."
    }
  ]
}

Agent keys are not secret, so returning them here leaks nothing. They already ship on every page of the customer's site.

#Errors

Errors about the key itself, such as invalid request or Pro not being active, are listed in Authentication.

#Provisioning a new customer, end to end

KEY="cb_your_key_here"
API="https://api.crawlbrain.com/v1"

# 1. Create the website
curl -sX POST $API/website/add -H "Api-Key: $KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://customer.example.com"}'

# 2. Find its id
ID=$(curl -s "$API/website/list" -H "Api-Key: $KEY" \
  | jq -r '.websites[] | select(.domain=="https://customer.example.com") | .id')

# 3. Get the agent key for the embed snippet
curl -s "$API/agent/get?id=$ID" -H "Api-Key: $KEY" | jq -r '.agent.key'

Step 3 gives you the value for data-key in the script tag you hand the customer.