AGENT SKILL

serving-the-api

Use when the user wants a long-running HTTP service for scrape/crawl/map instead of one-shot CLI calls or the MCP server — for example wiring crawlberg into other apps over REST. Covers `crawlberg serve`, the Firecrawl-v1-compatible endpoints, `--host`/`--port`, and when to prefer it.

License 许可
Apache-2.0
最近更新
2026-11-14
Stars
★ 1.6k
Forks
491

01 前言 Front Matter

name
serving-the-api
description
Use when the user wants a long-running HTTP service for scrape/crawl/map instead of one-shot CLI calls or the MCP server — for example wiring crawlberg into other apps over REST. Covers `crawlberg serve`, the Firecrawl-v1-compatible endpoints, `--host`/`--port`, and when to prefer it.

03 SKILL.md 正文

<!-- AI-RULEZ :: GENERATED FILE — DO NOT EDIT Content-Hash: blake3:a011fca3e4de3a4e6442e1779c4425d884cafb3369765d37e52d2f01dd8671bb Source-Hash: blake3:b5689383a3914da8e3cc6ad3614356ae7f98c90cf99cb1b4b47be38456ff7a7c Schema-Version: v1 -->

Serving the API

crawlberg serve starts a long-running REST API server exposing the crawl engine over HTTP. The endpoints are Firecrawl v1-compatible, so existing Firecrawl clients and SDKs can point at a local crawlberg instance.

This subcommand requires the binary to be built with the optional api feature (non-default). The Homebrew tap is built with all features, so serve is available there; a from-source build needs cargo install crawlberg-cli --features api (or --features all) to include it.

Quick recipe

crawlberg serve --host 127.0.0.1 --port 3000

It logs Starting REST API server on <host>:<port> to stderr and blocks until terminated. Hit GET /health to confirm it is up.

Flags

Flag Default Purpose
--host 0.0.0.0 Address to bind to.
--port 3000 Port to listen on.

serve does not take the per-request crawl flags — it boots with the default CrawlConfig and accepts per-request options in each HTTP request body.

Endpoints

Firecrawl v1 surface:

Method + path Purpose
POST /v1/scrape Scrape a single URL (synchronous).
POST /v1/crawl Start an asynchronous crawl job → returns a job id.
GET /v1/crawl/{id} Poll crawl job status and collect results.
DELETE /v1/crawl/{id} Cancel a running crawl job.
POST /v1/map Discover URLs on a site (synchronous).
POST /v1/batch/scrape Start an asynchronous batch-scrape job → job id.
GET /v1/batch/scrape/{id} Poll batch-scrape job status and results.
POST /v1/download Download a document from a URL.

Operational:

Method + path Purpose
GET /health Health check ({ status, version }).
GET /version Version information.
GET /openapi.json OpenAPI schema for the whole surface.

Request bodies use camelCase fields. Scrape/crawl/map mirror the CLI: url, formats, onlyMainContent, includeTags/excludeTags, timeout for scrape; url, maxDepth, maxPages, includePaths/excludePaths for crawl; url, limit, search for map. Responses wrap data as { success, data | error }; async jobs return { success, id } and are polled for { status, total, completed, data, error }.

Examples

Scrape via HTTP

curl -s http://127.0.0.1:3000/v1/scrape \
  -H 'content-type: application/json' \
  -d '{"url":"https://example.com","formats":["markdown"]}' | jq '.data'

Start a crawl job and poll it

id=$(curl -s http://127.0.0.1:3000/v1/crawl \
  -H 'content-type: application/json' \
  -d '{"url":"https://example.com","maxDepth":2,"maxPages":100}' | jq -r '.id')

curl -s "http://127.0.0.1:3000/v1/crawl/$id" | jq '{status,completed,total}'

Map URLs

curl -s http://127.0.0.1:3000/v1/map \
  -H 'content-type: application/json' \
  -d '{"url":"https://example.com","limit":500,"search":"docs"}' | jq '.data'

Operational notes

  • The server accepts CORS from any origin and ships no built-in authentication — do not expose it on a public interface. Bind to 127.0.0.1 for local use, or put it behind a reverse proxy / gateway that enforces auth and rate limiting before exposing it more widely.
  • Request bodies are capped at 10 MB and each request times out after 5 minutes.
  • Crawl and batch-scrape are asynchronous: the POST returns a job id immediately and you poll the {id} endpoint until status is completed, failed, or cancelled.

When to run serve vs. CLI vs. MCP

  • CLI (scrape/crawl/map/interact) — one-shot jobs, shell pipelines, scripting. No long-running process.
  • MCP (crawlberg mcp) — driving the crawler from an AI agent harness (Claude Code, Codex, Cursor, Gemini, opencode). Auto-registered by this plugin; prefer it inside an agent.
  • serve (crawlberg serve) — a persistent HTTP service for other applications or non-agent clients that speak REST, or when you want Firecrawl-v1 compatibility. Choose it when many callers need the crawler over the network rather than one agent or one shell.

04 同仓库更多 Skills

hashgraph-online
hashgraph-online/awesome-codex-plugins

worktable-review-with-annotations

Create, inspect, reply to, update, or resolve situated Worktable annotations. Use when feedback belongs to a Doc section, block, text range, or HTML Doc, or when the user asks to address existing review feedback and coordinate a requested artifact edit.

★ 1.6k
hashgraph-online
hashgraph-online/awesome-codex-plugins

worktable-find-and-synthesize-context

Find, assess, and synthesize context from Worktable Docs and Records. Use when the user asks what a workspace contains, wants workspace-backed research, needs current facts reconciled across artifacts, or requests a cited synthesis without changing workspace state.

★ 1.6k
hashgraph-online
hashgraph-online/awesome-codex-plugins

worktable-create-or-update-docs

Create, edit, rename, or maintain narrative Worktable Docs. Use for plans, decisions, briefs, notes, research syntheses, and other durable prose when the user requests a new artifact or a safe update to an existing one.

★ 1.6k
hashgraph-online
hashgraph-online/awesome-codex-plugins

worktable-create-or-manage-records

Organize repeated structured information as durable Worktable Records. Use for research datasets, inventories, directories, pipelines, trackers, and other collections whose items need identity, fields, filtering, links, validation, or individual updates.

★ 1.6k
hashgraph-online
hashgraph-online/awesome-codex-plugins

worktable-create-or-manage-interactive-html

Create or improve durable visual and interactive artifacts in Worktable. Use for dashboards, trackers, calculators, mini apps, slide presentations, and other requests that benefit from a polished HTML Doc.

★ 1.6k
hashgraph-online
hashgraph-online/awesome-codex-plugins

worktable-collaborate-threads

Read, answer, and hand off messages through durable Worktable threads with users or other authenticated participants. Use when the user asks to check addressed messages, reply to a participant, continue a thread, request another agent's help, wait briefly for a response, or watch for comments and messages sent to the agent.

★ 1.6k