scraping-html-to-markdown
Use when the user wants a single page rendered as clean Markdown plus structured metadata. Covers `kreuzcrawl scrape <url>`, JSON vs Markdown output, what metadata is returned, and how to handle JS-heavy pages.
- License 许可
- Apache-2.0
- 最近更新
- 2026-11-14
- Stars
- ★ 1.6k
- Forks
- 491
01 前言 Front Matter
- name
- scraping-html-to-markdown
- description
- Use when the user wants a single page rendered as clean Markdown plus structured metadata. Covers `kreuzcrawl scrape <url>`, JSON vs Markdown output, what metadata is returned, and how to handle JS-heavy pages.
03 SKILL.md 正文
Scraping HTML to Markdown
kreuzcrawl scrape <url> is the right tool when the user has a single
page in mind. It returns Markdown plus a full structured payload (metadata,
links, images, JSON-LD, HTTP response info).
Quick recipe
kreuzcrawl scrape https://example.com/article --format markdown
JSON form (default) when downstream needs metadata:
kreuzcrawl scrape https://example.com/article --format json
Flag surface
| Flag | Default | Purpose |
|---|---|---|
--format |
json |
json or markdown. |
--timeout |
30000 |
Per-request timeout in ms. |
--proxy |
— | HTTP, HTTPS, or SOCKS5 proxy URL. |
--user-agent |
— | Override request UA. |
--respect-robots-txt |
off | Honour robots.txt. |
--browser-mode |
auto |
auto, always, never — see headless-fallback skill. |
--browser-endpoint |
— | External CDP ws:// URL. |
--config |
— | Inline JSON or @file.json for full CrawlConfig. |
Output shape
Markdown mode
Prints the rendered Markdown only. Use when piping to a file the user will read, or when the result becomes LLM context downstream.
JSON mode
Top-level PageResult with:
url,final_url(after redirects),status_code.markdown:{ content, fit_content, warnings }—fit_contentis a pruned LLM-optimised variant.metadata: Open Graph, Twitter Card, Dublin Core, article tags, JSON-LD, headings (H1–H6), feeds, favicons, hreflang.links: arrays forInternal,External,Anchor, andDocument.images:<img>,<picture>,srcset,og:image.tables: structured table data preserved separately from Markdown.response: HTTP headers, content type, charset, body size.
Read result.markdown.content for the Markdown string when scripting.
Common pitfalls
Empty or stub content
Static fetch returned a JS shell. Symptoms in JSON output:
markdown.contentis short or only contains nav/footer chrome.markdown.warningsmentions JS-render-required.metadata.headingsis empty when the page clearly has headings.
Re-run with --browser-mode always and see the headless-fallback skill.
WAF block
Auto mode detects 8 WAF vendors and retries through headless Chrome
automatically. If you forced --browser-mode never, the WAF response will
fall through. Check response.status_code — 403/406/503 with WAF headers
(server: cloudflare, x-amz-cf-id, etc.) is the giveaway.
Robots.txt blocking the fetch
If --respect-robots-txt is set and the path is disallowed, the scrape
returns an error rather than partial content. Drop the flag only on hosts
you own or have authorisation for.
Wrong charset
Most pages declare UTF-8. Pages that lie about their charset surface as
mojibake in markdown.content. Override via --config '{"force_encoding":"latin-1"}'
or similar.
Examples
Scrape an article for downstream LLM context
kreuzcrawl scrape https://blog.example.com/post-123 --format markdown \
> /tmp/article.md
Scrape with proxy and custom UA
kreuzcrawl scrape https://example.com \
--proxy http://proxy.internal:3128 \
--user-agent "kreuzcrawl (research@example.com)" \
--format json
Extract just the OG metadata
kreuzcrawl scrape https://example.com --format json \
| jq '.metadata | {title: .og.title, description: .og.description, image: .og.image}'
When to reach for crawl or interact instead
- The user wants the whole site, not one page →
crawling-a-siteskill. - The user needs to click, type, or scroll before extracting → use
kreuzcrawl interactwith the action list. - The user only wants the list of URLs →
kreuzcrawl map.
04 同仓库更多 Skills
worktable-review-with-annotations
Create, inspect, reply to, update, or resolve situated Worktable annotations. Use when feedback belongs to a Doc section, block, text range, or HTML Doc, or when the user asks to address existing review feedback and coordinate a requested artifact edit.
worktable-find-and-synthesize-context
Find, assess, and synthesize context from Worktable Docs and Records. Use when the user asks what a workspace contains, wants workspace-backed research, needs current facts reconciled across artifacts, or requests a cited synthesis without changing workspace state.
worktable-create-or-update-docs
Create, edit, rename, or maintain narrative Worktable Docs. Use for plans, decisions, briefs, notes, research syntheses, and other durable prose when the user requests a new artifact or a safe update to an existing one.
worktable-create-or-manage-records
Organize repeated structured information as durable Worktable Records. Use for research datasets, inventories, directories, pipelines, trackers, and other collections whose items need identity, fields, filtering, links, validation, or individual updates.
worktable-create-or-manage-interactive-html
Create or improve durable visual and interactive artifacts in Worktable. Use for dashboards, trackers, calculators, mini apps, slide presentations, and other requests that benefit from a polished HTML Doc.
worktable-collaborate-threads
Read, answer, and hand off messages through durable Worktable threads with users or other authenticated participants. Use when the user asks to check addressed messages, reply to a participant, continue a thread, request another agent's help, wait briefly for a response, or watch for comments and messages sent to the agent.