serving-the-api
Use when the user wants a long-running HTTP service for scrape/crawl/map instead of one-shot CLI calls or the MCP server — for example wiring crawlberg into other apps over REST. Covers `crawlberg serve`, the Firecrawl-v1-compatible endpoints, `--host`/`--port`, and when to prefer it.
- License 许可
- Apache-2.0
- 最近更新
- 2026-11-14
- Stars
- ★ 1.6k
- Forks
- 491
01 前言 Front Matter
- name
- serving-the-api
- description
- Use when the user wants a long-running HTTP service for scrape/crawl/map instead of one-shot CLI calls or the MCP server — for example wiring crawlberg into other apps over REST. Covers `crawlberg serve`, the Firecrawl-v1-compatible endpoints, `--host`/`--port`, and when to prefer it.
03 SKILL.md 正文
<!-- AI-RULEZ :: GENERATED FILE — DO NOT EDIT Content-Hash: blake3:a011fca3e4de3a4e6442e1779c4425d884cafb3369765d37e52d2f01dd8671bb Source-Hash: blake3:b5689383a3914da8e3cc6ad3614356ae7f98c90cf99cb1b4b47be38456ff7a7c Schema-Version: v1 -->
Serving the API
crawlberg serve starts a long-running REST API server exposing the crawl
engine over HTTP. The endpoints are Firecrawl v1-compatible, so existing
Firecrawl clients and SDKs can point at a local crawlberg instance.
This subcommand requires the binary to be built with the optional api
feature (non-default). The Homebrew tap is built with all features, so serve
is available there; a from-source build needs
cargo install crawlberg-cli --features api
(or --features all) to include it.
Quick recipe
crawlberg serve --host 127.0.0.1 --port 3000
It logs Starting REST API server on <host>:<port> to stderr and blocks until
terminated. Hit GET /health to confirm it is up.
Flags
| Flag | Default | Purpose |
|---|---|---|
--host |
0.0.0.0 |
Address to bind to. |
--port |
3000 |
Port to listen on. |
serve does not take the per-request crawl flags — it boots with the default
CrawlConfig and accepts per-request options in each HTTP request body.
Endpoints
Firecrawl v1 surface:
| Method + path | Purpose |
|---|---|
POST /v1/scrape |
Scrape a single URL (synchronous). |
POST /v1/crawl |
Start an asynchronous crawl job → returns a job id. |
GET /v1/crawl/{id} |
Poll crawl job status and collect results. |
DELETE /v1/crawl/{id} |
Cancel a running crawl job. |
POST /v1/map |
Discover URLs on a site (synchronous). |
POST /v1/batch/scrape |
Start an asynchronous batch-scrape job → job id. |
GET /v1/batch/scrape/{id} |
Poll batch-scrape job status and results. |
POST /v1/download |
Download a document from a URL. |
Operational:
| Method + path | Purpose |
|---|---|
GET /health |
Health check ({ status, version }). |
GET /version |
Version information. |
GET /openapi.json |
OpenAPI schema for the whole surface. |
Request bodies use camelCase fields. Scrape/crawl/map mirror the CLI:
url, formats, onlyMainContent, includeTags/excludeTags, timeout
for scrape; url, maxDepth, maxPages, includePaths/excludePaths for
crawl; url, limit, search for map. Responses wrap data as
{ success, data | error }; async jobs return { success, id } and are polled
for { status, total, completed, data, error }.
Examples
Scrape via HTTP
curl -s http://127.0.0.1:3000/v1/scrape \
-H 'content-type: application/json' \
-d '{"url":"https://example.com","formats":["markdown"]}' | jq '.data'
Start a crawl job and poll it
id=$(curl -s http://127.0.0.1:3000/v1/crawl \
-H 'content-type: application/json' \
-d '{"url":"https://example.com","maxDepth":2,"maxPages":100}' | jq -r '.id')
curl -s "http://127.0.0.1:3000/v1/crawl/$id" | jq '{status,completed,total}'
Map URLs
curl -s http://127.0.0.1:3000/v1/map \
-H 'content-type: application/json' \
-d '{"url":"https://example.com","limit":500,"search":"docs"}' | jq '.data'
Operational notes
- The server accepts CORS from any origin and ships no built-in
authentication — do not expose it on a public interface. Bind to
127.0.0.1for local use, or put it behind a reverse proxy / gateway that enforces auth and rate limiting before exposing it more widely. - Request bodies are capped at 10 MB and each request times out after 5 minutes.
- Crawl and batch-scrape are asynchronous: the POST returns a job
idimmediately and you poll the{id}endpoint untilstatusiscompleted,failed, orcancelled.
When to run serve vs. CLI vs. MCP
- CLI (
scrape/crawl/map/interact) — one-shot jobs, shell pipelines, scripting. No long-running process. - MCP (
crawlberg mcp) — driving the crawler from an AI agent harness (Claude Code, Codex, Cursor, Gemini, opencode). Auto-registered by this plugin; prefer it inside an agent. - serve (
crawlberg serve) — a persistent HTTP service for other applications or non-agent clients that speak REST, or when you want Firecrawl-v1 compatibility. Choose it when many callers need the crawler over the network rather than one agent or one shell.
04 同仓库更多 Skills
worktable-review-with-annotations
Create, inspect, reply to, update, or resolve situated Worktable annotations. Use when feedback belongs to a Doc section, block, text range, or HTML Doc, or when the user asks to address existing review feedback and coordinate a requested artifact edit.
worktable-find-and-synthesize-context
Find, assess, and synthesize context from Worktable Docs and Records. Use when the user asks what a workspace contains, wants workspace-backed research, needs current facts reconciled across artifacts, or requests a cited synthesis without changing workspace state.
worktable-create-or-update-docs
Create, edit, rename, or maintain narrative Worktable Docs. Use for plans, decisions, briefs, notes, research syntheses, and other durable prose when the user requests a new artifact or a safe update to an existing one.
worktable-create-or-manage-records
Organize repeated structured information as durable Worktable Records. Use for research datasets, inventories, directories, pipelines, trackers, and other collections whose items need identity, fields, filtering, links, validation, or individual updates.
worktable-create-or-manage-interactive-html
Create or improve durable visual and interactive artifacts in Worktable. Use for dashboards, trackers, calculators, mini apps, slide presentations, and other requests that benefit from a polished HTML Doc.
worktable-collaborate-threads
Read, answer, and hand off messages through durable Worktable threads with users or other authenticated participants. Use when the user asks to check addressed messages, reply to a participant, continue a thread, request another agent's help, wait briefly for a response, or watch for comments and messages sent to the agent.