API Overview

The CrawlBrain API lets your code do what the dashboard does, from adding a website and triggering its sync to pushing content into the index. It can also read the question log, and query the AI directly with the same agent key the widget uses.

#Base URL

https://api.crawlbrain.com/v1

Every endpoint on this page is relative to that.

A legacy base at https://app.crawlbrain.com/api serves the same routes and will keep working forever, because installed integrations depend on it. New code should use /v1.

#Two kinds of key

The API key is for your server and the agent key is for the widget in a browser, and the two do very different jobs.

Never put an API key in front-end code. See Authentication.

#Endpoint map

#Websites, API key

Details in Websites.

#Content, API key

Details in Pages and Snippets.

#Agent, API key

#Search and answers, agent key

Details in Widget and Queries.

#Website ids

Website-scoped endpoints take an id. It is an opaque identifier, not your domain and not a sequential number.

Get it from /website/list:

curl https://api.crawlbrain.com/v1/website/list \
  -H "Api-Key: cb_your_key_here"

{
  "success": true,
  "websites": [
    { "id": "9f2c1a...", "domain": "https://example.com" }
  ]
}

Store the id alongside the domain in your own system. It is stable for the life of the website.

#Responses

Responses are JSON with a success boolean.

{ "success": true }

Errors carry a machine-readable code and a human message:

{
  "success": false,
  "error": "website_not_found",
  "message": "No website matches that id for the authenticated user."
}

Branch on error, never on message. Message text is written for humans and may be reworded.

A failed validation returns 422 with every field that failed:

{
  "success": false,
  "error": "validation_failed",
  "message": "One or more parameters failed validation.",
  "errors": { "url": ["The url field is required."] }
}

Key errors use the same envelope:

A missing or unknown key returns HTTP 200, so check the success field rather than the status code alone. The /agent/settings payload has no success field.

#Rate limits

The expensive bucket covers anything that dispatches work: /website/sync, /page/add, /page/save, /snippet/add, /snippet/save, and /agent/query. Each of those either queues a crawl or runs an embedding or a model call.

Limits are counted per client IP address, or per agent key for widget search. Every request also counts against the default bucket.

On 429, back off and retry after the number of seconds in the Retry-After header. Batch imports should sleep between writes rather than firing in a loop.

#Plan requirement

The API-key endpoints need Pro. It is checked live on every request, so a cancellation takes effect immediately.

Keys created by the official plugins and the browser extension, through their Connect button, work on every account. Free limits still apply: one website and 1,000 pages.

The agent-key endpoints work on every account, because that is what the widget uses. They spend credit like any other answer or search.

#Credit

Work that costs money is paid from the account's credit balance, not from the plan. Indexing a page or snippet costs credit, and so does every search and AI answer. When the balance is too low:

• /page/add and /snippet/add return 402 with no_credits.
• /agent/query returns 402 with credit_balance_empty.
• /agent/search keeps working with keyword matching only, at no cost.

#Interactive reference

The full machine-readable specification, with request and response schemas for every endpoint, is linked from Developer → API Docs in the dashboard.