scrape
Scrape any webpage as clean markdown via Bright Data Web Unlocker API. Bypasses bot detection and CAPTCHA. Requires BRIGHTDATA_API_KEY and BRIGHTDATA_UNLOCKER_ZONE environment variables.
- License 许可
- MIT
- 最近更新
- 2026-11-14
- Stars
- ★ 3.3w
- Forks
- 3.9k
01 前言 Front Matter
- name
- scrape
- description
- Scrape any webpage as clean markdown via Bright Data Web Unlocker API. Bypasses bot detection and CAPTCHA. Requires BRIGHTDATA_API_KEY and BRIGHTDATA_UNLOCKER_ZONE environment variables.
02 资源文件 2
- 脚本scripts/babysit-pr.sh3.0 KB
- 脚本scripts/scrape.sh864 B
03 SKILL.md 正文
Bright Data - Web Scraper
Scrape any webpage and get clean markdown content using Bright Data's Web Unlocker API. Automatically bypasses bot detection and CAPTCHA.
Setup
1. Get your API Key: Get a key from Bright Data Dashboard.
2. Create a Web Unlocker zone: Create a zone at brightdata.com/cp by clicking "Add" (top-right), selecting "Unlocker zone".
3. Set environment variables:
export BRIGHTDATA_API_KEY="your-api-key"
export BRIGHTDATA_UNLOCKER_ZONE="your-zone-name"
Usage
bash scripts/scrape.sh "url"
Parameters:
url(required): The webpage URL to scrape
Examples:
# Scrape a news article
bash scripts/scrape.sh "https://example.com/article"
# Scrape a product page
bash scripts/scrape.sh "https://shop.example.com/product/123"
Output Format
Returns clean markdown content extracted from the webpage:
# Page Title
Main content of the page converted to markdown format...
## Section Heading
More content...
Features
- Bot Detection Bypass: Automatically handles anti-bot measures
- CAPTCHA Solving: Bypasses CAPTCHA challenges
- Clean Markdown: Returns well-formatted markdown content
- JavaScript Rendering: Handles JavaScript-heavy pages
Dependencies
curl- For API requests
04 同仓库更多 Skills
wealthville-pool-scoring
Score DeFi liquidity pools before providing liquidity. Returns a 0-100 pool score and an Enter/Hold/Exit/Reduce/Avoid verdict with calibrated confidence for ~68,800 Solana pools and 575 EVM pools, plus a public miss-inclusive track record. Use when asked whether a pool is worth LPing into, which pools look best right now, or how accurate the scores have actually been.
technical-debt-auditor
Scans a codebase and produces a quantified technical debt report with a prioritized remediation plan. Analyzes dead code, TODO/FIXME/HACK markers, dependency staleness, complexity hotspots, code duplication signals, test coverage gaps, unused exports, and outdated patterns. Use this skill when the user asks to audit their codebase for tech debt, wants a health check of their code quality, says things like "where should I focus refactoring", "find code smells", "audit my codebase", "what technical debt do we have", "prioritize cleanup work", or asks about code quality metrics. Also trigger when the user mentions "code rot", "legacy cleanup", "refactoring priorities", or wants to understand the overall health of a project before a major effort. Trigger even if the user doesn't say "technical debt" explicitly — any request to systematically assess code quality and identify improvement priorities belongs here.
readonly-sql-guard
Make an agent's database access provably read-only. Use when an agent runs caller-supplied SQL against a production/ERP/analytics database (a "query" MCP tool, text-to-SQL, an admin console), or when reviewing DB-access code for write/DDL/side-effect/transaction-escape risks. Checks what a statement CALLS, not just its shape.
web-artifacts-builder
Suite of tools for creating elaborate, multi-component claude.ai HTML artifacts using modern frontend web technologies (React, Tailwind CSS, shadcn/ui). Use for complex artifacts requiring state management, routing, or shadcn/ui components - not for simple single-file HTML/JSX artifacts.
webapp-testing
Toolkit for interacting with and testing local web applications using Playwright. Supports verifying frontend functionality, debugging UI behavior, capturing browser screenshots, and viewing browser logs.
discernment-nudge
After you give a substantive answer or draft that the user may act on — advice or recommendations, drafted artifacts such as goals, plans, pitches, proposals, or emails, estimates or projections, analysis or interpretation of data, factual claims they may rely on, or a multi-step argument — invoke this skill BEFORE finalizing your reply and then, if it applies, append 2-3 short follow-up questions, each tied to something specific in what you just produced, that help the user check key facts, probe the reasoning or assumptions, and notice missing context. Do this at most once per conversation. Skip it when the user asked a trivial how-to or simple lookup, wants a purely educational explanation, asked you only to format, convert, or assemble a file from content they provided, is writing code they will run, is doing creative writing or casual chat, or already asked you to double-check, cite, or review — the skill file explains these boundaries and the exact output format.