[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6MJ2NWRW1NHJGFRDPM1V61F":3,"skill-same:01M6MJ2NWRW1NHJGFRDPM1V61F":79},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":26,"publishedAt":50,"linkOnly":51,"markdownContent":52,"markdownBody":53,"frontMatter":54,"refName":55,"commitSha":56,"skillDirectory":57,"forksCount":58,"repositoryHtmlUrl":59,"repositoryHomepageUrl":60,"ownerHtmlUrl":61,"ownerBlogUrl":62,"ownerDescription":63,"updatedAt":64,"install":65,"resources":78},"01M6MJ2NWRW1NHJGFRDPM1V61F","scraping-html-to-markdown",null,"Use when the user wants a single page rendered as clean Markdown plus structured metadata. Covers `kreuzcrawl scrape \u003Curl>`, JSON vs Markdown output, what metadata is returned, and how to handle JS-heavy pages.","hashgraph-online\u002Fawesome-codex-plugins","hashgraph-online","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F160371962?v=4","01M6CNB6NRJVNNGN14TR01Q6D3","HOL (Hashgraph Online)","01M6JT1HSKKG9V68KQAQGBA7XP",1575,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fkreuzberg-dev\u002Fplugins\u002Fplugins\u002Fkreuzcrawl\u002Fskills\u002Fscraping-html-to-markdown\u002FSKILL.md",[18,23],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000002","文档处理","SCENARIO","使用场景",{"categoryNo":24,"categoryName":25,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000009","前端开发",[27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49],"markdown","scraping html to markdown","scraping","html","html to markdown","clean","when","single","kreuzcrawl","URL","output","how to","as","pages","structured","metadata","use","user","page","json","js","how","scrape","2026-11-10 10:12:30",false,"---\nname: scraping-html-to-markdown\ndescription: >-\n  Use when the user wants a single page rendered as clean Markdown plus\n  structured metadata. Covers `kreuzcrawl scrape \u003Curl>`, JSON vs Markdown\n  output, what metadata is returned, and how to handle JS-heavy pages.\n---\n\n# Scraping HTML to Markdown\n\n`kreuzcrawl scrape \u003Curl>` is the right tool when the user has a single\npage in mind. It returns Markdown plus a full structured payload (metadata,\nlinks, images, JSON-LD, HTTP response info).\n\n## Quick recipe\n\n```bash\nkreuzcrawl scrape https:\u002F\u002Fexample.com\u002Farticle --format markdown\n```\n\nJSON form (default) when downstream needs metadata:\n\n```bash\nkreuzcrawl scrape https:\u002F\u002Fexample.com\u002Farticle --format json\n```\n\n## Flag surface\n\n| Flag                    | Default | Purpose                                                |\n| ----------------------- | ------- | ------------------------------------------------------ |\n| `--format`              | `json`  | `json` or `markdown`.                                  |\n| `--timeout`             | `30000` | Per-request timeout in ms.                             |\n| `--proxy`               | —       | HTTP, HTTPS, or SOCKS5 proxy URL.                      |\n| `--user-agent`          | —       | Override request UA.                                   |\n| `--respect-robots-txt`  | off     | Honour `robots.txt`.                                   |\n| `--browser-mode`        | `auto`  | `auto`, `always`, `never` — see headless-fallback skill. |\n| `--browser-endpoint`    | —       | External CDP `ws:\u002F\u002F` URL.                              |\n| `--config`              | —       | Inline JSON or `@file.json` for full `CrawlConfig`.    |\n\n## Output shape\n\n### Markdown mode\n\nPrints the rendered Markdown only. Use when piping to a file the user will\nread, or when the result becomes LLM context downstream.\n\n### JSON mode\n\nTop-level `PageResult` with:\n\n- `url`, `final_url` (after redirects), `status_code`.\n- `markdown`: `{ content, fit_content, warnings }` — `fit_content` is a\n  pruned LLM-optimised variant.\n- `metadata`: Open Graph, Twitter Card, Dublin Core, article tags, JSON-LD,\n  headings (H1–H6), feeds, favicons, hreflang.\n- `links`: arrays for `Internal`, `External`, `Anchor`, and `Document`.\n- `images`: `\u003Cimg>`, `\u003Cpicture>`, `srcset`, `og:image`.\n- `tables`: structured table data preserved separately from Markdown.\n- `response`: HTTP headers, content type, charset, body size.\n\nRead `result.markdown.content` for the Markdown string when scripting.\n\n## Common pitfalls\n\n### Empty or stub content\n\nStatic fetch returned a JS shell. Symptoms in JSON output:\n\n- `markdown.content` is short or only contains nav\u002Ffooter chrome.\n- `markdown.warnings` mentions JS-render-required.\n- `metadata.headings` is empty when the page clearly has headings.\n\nRe-run with `--browser-mode always` and see the headless-fallback skill.\n\n### WAF block\n\n`Auto` mode detects 8 WAF vendors and retries through headless Chrome\nautomatically. If you forced `--browser-mode never`, the WAF response will\nfall through. Check `response.status_code` — 403\u002F406\u002F503 with WAF headers\n(`server: cloudflare`, `x-amz-cf-id`, etc.) is the giveaway.\n\n### Robots.txt blocking the fetch\n\nIf `--respect-robots-txt` is set and the path is disallowed, the scrape\nreturns an error rather than partial content. Drop the flag only on hosts\nyou own or have authorisation for.\n\n### Wrong charset\n\nMost pages declare UTF-8. Pages that lie about their charset surface as\nmojibake in `markdown.content`. Override via `--config '{\"force_encoding\":\"latin-1\"}'`\nor similar.\n\n## Examples\n\n### Scrape an article for downstream LLM context\n\n```bash\nkreuzcrawl scrape https:\u002F\u002Fblog.example.com\u002Fpost-123 --format markdown \\\n  > \u002Ftmp\u002Farticle.md\n```\n\n### Scrape with proxy and custom UA\n\n```bash\nkreuzcrawl scrape https:\u002F\u002Fexample.com \\\n  --proxy http:\u002F\u002Fproxy.internal:3128 \\\n  --user-agent \"kreuzcrawl (research@example.com)\" \\\n  --format json\n```\n\n### Extract just the OG metadata\n\n```bash\nkreuzcrawl scrape https:\u002F\u002Fexample.com --format json \\\n  | jq '.metadata | {title: .og.title, description: .og.description, image: .og.image}'\n```\n\n## When to reach for crawl or interact instead\n\n- The user wants the whole site, not one page → `crawling-a-site` skill.\n- The user needs to click, type, or scroll before extracting → use\n  `kreuzcrawl interact` with the action list.\n- The user only wants the list of URLs → `kreuzcrawl map`.\n","# Scraping HTML to Markdown\n\n`kreuzcrawl scrape \u003Curl>` is the right tool when the user has a single\npage in mind. It returns Markdown plus a full structured payload (metadata,\nlinks, images, JSON-LD, HTTP response info).\n\n## Quick recipe\n\n```bash\nkreuzcrawl scrape https:\u002F\u002Fexample.com\u002Farticle --format markdown\n```\n\nJSON form (default) when downstream needs metadata:\n\n```bash\nkreuzcrawl scrape https:\u002F\u002Fexample.com\u002Farticle --format json\n```\n\n## Flag surface\n\n| Flag                    | Default | Purpose                                                |\n| ----------------------- | ------- | ------------------------------------------------------ |\n| `--format`              | `json`  | `json` or `markdown`.                                  |\n| `--timeout`             | `30000` | Per-request timeout in ms.                             |\n| `--proxy`               | —       | HTTP, HTTPS, or SOCKS5 proxy URL.                      |\n| `--user-agent`          | —       | Override request UA.                                   |\n| `--respect-robots-txt`  | off     | Honour `robots.txt`.                                   |\n| `--browser-mode`        | `auto`  | `auto`, `always`, `never` — see headless-fallback skill. |\n| `--browser-endpoint`    | —       | External CDP `ws:\u002F\u002F` URL.                              |\n| `--config`              | —       | Inline JSON or `@file.json` for full `CrawlConfig`.    |\n\n## Output shape\n\n### Markdown mode\n\nPrints the rendered Markdown only. Use when piping to a file the user will\nread, or when the result becomes LLM context downstream.\n\n### JSON mode\n\nTop-level `PageResult` with:\n\n- `url`, `final_url` (after redirects), `status_code`.\n- `markdown`: `{ content, fit_content, warnings }` — `fit_content` is a\n  pruned LLM-optimised variant.\n- `metadata`: Open Graph, Twitter Card, Dublin Core, article tags, JSON-LD,\n  headings (H1–H6), feeds, favicons, hreflang.\n- `links`: arrays for `Internal`, `External`, `Anchor`, and `Document`.\n- `images`: `\u003Cimg>`, `\u003Cpicture>`, `srcset`, `og:image`.\n- `tables`: structured table data preserved separately from Markdown.\n- `response`: HTTP headers, content type, charset, body size.\n\nRead `result.markdown.content` for the Markdown string when scripting.\n\n## Common pitfalls\n\n### Empty or stub content\n\nStatic fetch returned a JS shell. Symptoms in JSON output:\n\n- `markdown.content` is short or only contains nav\u002Ffooter chrome.\n- `markdown.warnings` mentions JS-render-required.\n- `metadata.headings` is empty when the page clearly has headings.\n\nRe-run with `--browser-mode always` and see the headless-fallback skill.\n\n### WAF block\n\n`Auto` mode detects 8 WAF vendors and retries through headless Chrome\nautomatically. If you forced `--browser-mode never`, the WAF response will\nfall through. Check `response.status_code` — 403\u002F406\u002F503 with WAF headers\n(`server: cloudflare`, `x-amz-cf-id`, etc.) is the giveaway.\n\n### Robots.txt blocking the fetch\n\nIf `--respect-robots-txt` is set and the path is disallowed, the scrape\nreturns an error rather than partial content. Drop the flag only on hosts\nyou own or have authorisation for.\n\n### Wrong charset\n\nMost pages declare UTF-8. Pages that lie about their charset surface as\nmojibake in `markdown.content`. Override via `--config '{\"force_encoding\":\"latin-1\"}'`\nor similar.\n\n## Examples\n\n### Scrape an article for downstream LLM context\n\n```bash\nkreuzcrawl scrape https:\u002F\u002Fblog.example.com\u002Fpost-123 --format markdown \\\n  > \u002Ftmp\u002Farticle.md\n```\n\n### Scrape with proxy and custom UA\n\n```bash\nkreuzcrawl scrape https:\u002F\u002Fexample.com \\\n  --proxy http:\u002F\u002Fproxy.internal:3128 \\\n  --user-agent \"kreuzcrawl (research@example.com)\" \\\n  --format json\n```\n\n### Extract just the OG metadata\n\n```bash\nkreuzcrawl scrape https:\u002F\u002Fexample.com --format json \\\n  | jq '.metadata | {title: .og.title, description: .og.description, image: .og.image}'\n```\n\n## When to reach for crawl or interact instead\n\n- The user wants the whole site, not one page → `crawling-a-site` skill.\n- The user needs to click, type, or scroll before extracting → use\n  `kreuzcrawl interact` with the action list.\n- The user only wants the list of URLs → `kreuzcrawl map`.\n",{"name":5,"description":7},"main","2b63f856ce19c2a797929cacabd9e0fc936812fa","plugins\u002Fkreuzberg-dev\u002Fplugins\u002Fplugins\u002Fkreuzcrawl\u002Fskills\u002Fscraping-html-to-markdown",491,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins","https:\u002F\u002Fhol.org\u002Fplugins\u002Fbest-codex-plugins","https:\u002F\u002Fgithub.com\u002Fhashgraph-online","https:\u002F\u002Fhol.org","Open-source standards and developer tools for interoperable AI agents: discovery, identity, communication, payments, privacy and security.","2026-11-14 14:06:13",{"source":8,"skillName":5,"commands":66,"prompt":76,"zipAvailable":77},[67,70,73],{"tool":68,"command":69},"npx","npx skills add hashgraph-online\u002Fawesome-codex-plugins --skill scraping-html-to-markdown",{"tool":71,"command":72},"bunx","bunx skills add hashgraph-online\u002Fawesome-codex-plugins --skill scraping-html-to-markdown",{"tool":74,"command":75},"pnpm","pnpm dlx skills add hashgraph-online\u002Fawesome-codex-plugins --skill scraping-html-to-markdown","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fkreuzberg-dev\u002Fplugins\u002Fplugins\u002Fkreuzcrawl\u002Fskills\u002Fscraping-html-to-markdown\u002FSKILL.md",true,[],[80,90,98,106,114,122],{"skillNo":81,"name":82,"nameZh":6,"description":83,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":84,"licenseFileUrl":85,"sourceHtmlUrl":86,"categories":87,"keywords":88,"publishedAt":89},"01M7DV332YQMGD8WTRJ9DZQXTB","worktable-review-with-annotations","Create, inspect, reply to, update, or resolve situated Worktable annotations. Use when feedback belongs to a Doc section, block, text range, or HTML Doc, or when the user asks to address existing review feedback and coordinate a requested artifact edit.","MIT","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-review-with-annotations\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-review-with-annotations\u002FSKILL.md",[],[],"2026-11-15 14:19:17",{"skillNo":91,"name":92,"nameZh":6,"description":93,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":84,"licenseFileUrl":94,"sourceHtmlUrl":95,"categories":96,"keywords":97,"publishedAt":89},"01M7DV2YGHCZVBEB1AHYB6CGHE","worktable-find-and-synthesize-context","Find, assess, and synthesize context from Worktable Docs and Records. Use when the user asks what a workspace contains, wants workspace-backed research, needs current facts reconciled across artifacts, or requests a cited synthesis without changing workspace state.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-find-and-synthesize-context\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-find-and-synthesize-context\u002FSKILL.md",[],[],{"skillNo":99,"name":100,"nameZh":6,"description":101,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":84,"licenseFileUrl":102,"sourceHtmlUrl":103,"categories":104,"keywords":105,"publishedAt":89},"01M7DV2TQN1H11ME2SAF9MXN7V","worktable-create-or-update-docs","Create, edit, rename, or maintain narrative Worktable Docs. Use for plans, decisions, briefs, notes, research syntheses, and other durable prose when the user requests a new artifact or a safe update to an existing one.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-update-docs\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-update-docs\u002FSKILL.md",[],[],{"skillNo":107,"name":108,"nameZh":6,"description":109,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":84,"licenseFileUrl":110,"sourceHtmlUrl":111,"categories":112,"keywords":113,"publishedAt":89},"01M7DV2NE452NEA8TXATKSNAJH","worktable-create-or-manage-records","Organize repeated structured information as durable Worktable Records. Use for research datasets, inventories, directories, pipelines, trackers, and other collections whose items need identity, fields, filtering, links, validation, or individual updates.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-manage-records\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-manage-records\u002FSKILL.md",[],[],{"skillNo":115,"name":116,"nameZh":6,"description":117,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":84,"licenseFileUrl":118,"sourceHtmlUrl":119,"categories":120,"keywords":121,"publishedAt":89},"01M7DV2HENRBR2KSCVM064WH0Y","worktable-create-or-manage-interactive-html","Create or improve durable visual and interactive artifacts in Worktable. Use for dashboards, trackers, calculators, mini apps, slide presentations, and other requests that benefit from a polished HTML Doc.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-manage-interactive-html\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-manage-interactive-html\u002FSKILL.md",[],[],{"skillNo":123,"name":124,"nameZh":6,"description":125,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":84,"licenseFileUrl":126,"sourceHtmlUrl":127,"categories":128,"keywords":129,"publishedAt":89},"01M7DV2CADR5R7RQFFYPNAGR9Y","worktable-collaborate-threads","Read, answer, and hand off messages through durable Worktable threads with users or other authenticated participants. Use when the user asks to check addressed messages, reply to a participant, continue a thread, request another agent's help, wait briefly for a response, or watch for comments and messages sent to the agent.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-collaborate-threads\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-collaborate-threads\u002FSKILL.md",[],[]]