AGENT SKILL

katana-web-crawling

Guides use of ProjectDiscovery Katana for web crawling and spidering in security testing and recon workflows. Covers installation, standard vs headless mode, scope and rate limits, JSONL output, and piping from httpx or URL lists. Use when the user mentions Katana, projectdiscovery/katana, web crawling, spidering, endpoint discovery, attack surface mapping, or chaining crawlers in automation pipelines.

License 许可
Apache-2.0
所属仓库
zkos-labs/bastion
最近更新
2026-11-13
Stars
★ 3
Forks
1

01 前言 Front Matter

name
katana-web-crawling
description
Guides use of ProjectDiscovery Katana for web crawling and spidering in security testing and recon workflows. Covers installation, standard vs headless mode, scope and rate limits, JSONL output, and piping from httpx or URL lists. Use when the user mentions Katana, projectdiscovery/katana, web crawling, spidering, endpoint discovery, attack surface mapping, or chaining crawlers in automation pipelines.

03 SKILL.md 正文

Katana web crawling

Katana is a fast crawler/spider from ProjectDiscovery, aimed at automation pipelines (URLs in → discovered endpoints out). Official docs and flags: repository README and katana -h.

Scope and ethics

Use only on systems you own or are explicitly authorized to test (contract, bug bounty program rules, internal env). Crawl gently: set concurrency, rate limits, and depth to reduce load. Misuse can violate law and terms of service-you are responsible for your actions (tool ships with that warning).

Installation

Go (requires Go 1.25+ per upstream; verify current README if install fails):

CGO_ENABLED=1 go install github.com/projectdiscovery/katana/cmd/katana@latest

Docker:

docker pull projectdiscovery/katana:latest
docker run projectdiscovery/katana:latest -u https://example.com

Headless in Docker often needs -system-chrome and Chrome/Chromium available-see upstream Docker section.

Input

  • Single/multiple URLs: -u https://a.com or comma-separated URLs
  • File: -list urls.txt
  • STDIN: echo https://example.com | katana or cat domains | httpx | katana

Modes

Mode When
Standard (default) Fast; uses Go HTTP client; no full JS/DOM render-may miss post-render routes
Headless (-headless) Browser context; better for JS-heavy apps; optional -system-chrome

Enable JS file parsing for more endpoints: -js-crawl (-jc). -jsluice is heavier.

Flags to know first

Flag Purpose
-d, -depth Max crawl depth (default 3)
-c, -concurrency Parallel fetchers
-rl, -rate-limit Max requests per second
-ct, -crawl-duration Cap total crawl time (e.g. 5m)
-cs / -cos In-scope / out-of-scope URL regex
-ns Disable default host scope if you need cross-host (use carefully)
-iqp Ignore same path with different query strings
-fs, -filter-similar Reduce near-duplicate paths
-kf, -known-files robots.txt / sitemap.xml etc. (min depth 3 for full coverage per docs)
-j, -jsonl JSONL output for scripting
-o, -output Write to file
-sr, -store-response Store HTTP for review (disk use)
-proxy HTTP/SOCKS5 proxy
-H Extra headers (auth, cookies) via header:value

Run katana -h for the full list (filters, form fill, tech detect, TLS options, etc.).

Minimal examples

katana -u https://example.com -d 2 -silent
katana -u https://example.com -jsonl -o endpoints.jsonl
katana -list seeds.txt -d 3 -cs '.*\.example\.com.*' -rl 30 -jsonl

Headless (JS-heavy target):

katana -u https://example.com -headless -d 2

Pipelines

Common pattern: resolve live HTTP first, then crawl:

cat domains.txt | httpx -silent | katana -jsonl -o crawl.jsonl

Combine with other PD tools (naabu, nuclei, etc.) only in authorized assessments.

Troubleshooting

  • CGO_ENABLED=1 required for go install per README.
  • Headless failures: try -system-chrome, ensure Chrome/Chromium installed, or use Docker image with documented Chrome setup.
  • Health check: -health-check / -hc.

References

04 同仓库更多 Skills

zkos-labs
zkos-labs/bastion

bastion

Programmable Trust Runtime for Autonomous Systems. Orchestrates identity, policy, privacy, durable execution, and multi-chain settlement.

★ 3
zkos-labs
zkos-labs/bastion

web2-api-firewall-administrator

Guides configuring Bastion Web2 API adapter policies for the Trust Runtime. Covers endpoint allowlists, provider budgets, rate limits, content inspection (PII/secrets detection), header filters, and OpenAPI spec-based auto-configuration. Use when configuring API call security for AI agents, setting provider spending caps, inspecting outbound API traffic, or setting up the Web2 adapter.

★ 3
zkos-labs
zkos-labs/bastion

transaction-screening-workflow-concepts

Educational map of transaction-centric compliance screening-transfer as the atomic unit, deposit vs withdrawal direction, single and CSV import, transaction list and detail views, per-transfer screening, rescreen, and STR-style exports. Use when the user asks how monitoring UIs treat tx hashes, directions, or regulatory reporting hooks-not for legal filing advice or evading reporting.

★ 3
zkos-labs
zkos-labs/bastion

solana-tracing-specialist

Guides Solana-specific on-chain forensics-ATA resolution, SPL instruction parsing, transaction history via RPC and indexers (e.g. Helius-style APIs), fund-flow graphs, Solana clustering heuristics, and program authority review. Use when the user investigates Solana wallets, SPL tokens, DEX/Jito flows, rug or phishing patterns on Solana, or needs evidence-structured tracing reports with public data only.

★ 3
zkos-labs
zkos-labs/bastion

solana-onchain-intelligence-resources

Companion to blockint-skills Solana work-official doc indexes and stacks for RPC/indexers, sanctions and address intel MCP, web research, agent payments, graph UIs, Solana Foundation dev skills, Solana Policy Institute (policy/regulatory education), optional open-source Scrapy collection (BlockchainSpider), Impersonator Solana (dApp dev/testing), coral-xyz sealevel-attacks (security pattern examples), Neodyme Solana Security Workshop (workshop.neodyme.io), and Osec Solana auditor introduction blog (osec.io). Use when the user needs pointers to Helius, Range AI, Tavily, PayAI x402, React Flow, Solana.com skills, or SPI alongside on-chain forensics-not as a substitute for reading current provider docs, legal counsel, or API keys.

★ 3
zkos-labs
zkos-labs/bastion

solana-defi-vulnerability-analyst-agent

Guides discovery and documentation of Solana DeFi protocol risks from public code and chain state-Anchor/native programs, PDAs, CPIs, oracles, pools, SPL mechanics, and historical tx reconstruction. Use when the user asks for Solana program security review, DeFi vulnerability triage, PDA or CPI safety, oracle or liquidity-pool risk, launchpad/bonding-curve issues, or evidence-backed severity findings without exploits or private keys.

★ 3