batch-extraction
Use when extracting from many files at once with shared config, bounded parallelism, per-file overrides, and error recovery. Covers the `batch` command, `--file-configs`, `--max-concurrent`, and output layout.
- License 许可
- Apache-2.0
- 最近更新
- 2026-11-14
- Stars
- ★ 1.6k
- Forks
- 491
01 前言 Front Matter
- name
- batch-extraction
- description
- Use when extracting from many files at once with shared config, bounded parallelism, per-file overrides, and error recovery. Covers the `batch` command, `--file-configs`, `--max-concurrent`, and output layout.
03 SKILL.md 正文
Batch extraction
Use this when processing a directory or glob of documents in one pass.
kreuzberg batch shares one extraction config across every file, runs
extractions concurrently, and returns one structured array — failures on
individual files do not abort the run.
Basic usage
# Glob expands to many paths; results come back as a JSON array (default)
kreuzberg batch *.pdf
# Mixed formats, markdown content for LLM ingestion
kreuzberg batch docs/*.docx --content-format markdown
# Recurse with the shell, then extract
kreuzberg batch $(find ./corpus -name '*.pdf')
batch defaults to --format json (vs --format text for single
extract). Each array entry is a full extraction result, so downstream
code can index by position into the input path list.
kreuzberg batch reports/*.pdf \
| jq '.[] | {chars: (.content | length), mime: .mime_type}'
Parallelism
--max-concurrent caps how many files extract at once (default: CPU
count). Lower it on memory-constrained hosts or when OCR/ML models are
active, since each in-flight extraction holds its own buffers:
# Cap at 4 concurrent extractions
kreuzberg batch scans/*.pdf --ocr true --max-concurrent 4
--max-threads additionally caps total internal threads (Rayon, ONNX
intra-op, the batch semaphore) for tightly constrained environments:
kreuzberg batch *.pdf --max-concurrent 2 --max-threads 4
Per-file config overrides
A single shared config does not always fit. --file-configs points at a
JSON file mapping each path to its own override object, merged on top of
the shared config for that file only:
{
"scan.pdf": { "force_ocr": true },
"report.pdf": { "output_format": "markdown" },
"data.xlsx": { "output_format": "json" }
}
kreuzberg batch scan.pdf report.pdf data.xlsx --file-configs overrides.json
Keys are file paths (matching the paths passed on the command line); values are per-file extraction config objects in snake_case, the same shape as a config file.
Output layout
For text/toon output with image extraction, --output-dir controls where
referenced image files (e.g. image_0.png) are written; the directory
must already exist. JSON output embeds image bytes inline and ignores
--output-dir.
mkdir -p out/images
kreuzberg batch slides/*.pptx --extract-images true --output-dir out/images --format text
Error recovery
Batch extraction is fault-tolerant per file: one unreadable or corrupt
document does not stop the rest. Inspect results for partial content and
surfaced errors rather than relying on the process exit code alone. Pair
with --max-concurrent to avoid exhausting memory when a few large files
sit in a big batch.
Shared config
Every extract flag also applies to batch (OCR, chunking, layout,
content format, etc.) and is shared across all files unless a
--file-configs entry overrides it:
kreuzberg batch invoices/*.pdf \
--layout --layout-table-model slanet_wireless \
--content-format markdown --max-concurrent 8
A config file works too and auto-discovers from the cwd upward:
output_format = "markdown"
[ocr]
backend = "tesseract"
language = "eng"
kreuzberg batch corpus/*.pdf --config kreuzberg.toml
Programmatic access
From Python, use the batch helpers (async and sync):
from kreuzberg import batch_extract_files, batch_extract_files_sync, ExtractionConfig
config = ExtractionConfig(output_format="markdown")
# Async
results = await batch_extract_files(["a.pdf", "b.docx", "c.xlsx"], config=config)
# Sync
results = batch_extract_files_sync(["a.pdf", "b.docx"], config=config)
for result in results:
print(len(result.content))
Node.js mirrors this with batchExtractFiles; Rust uses
batch_extract_file (requires the tokio-runtime feature). See
references/python-api.md, references/nodejs-api.md, and
references/rust-api.md in the sibling kreuzberg skill.
MCP
When the kreuzberg MCP server is registered, prefer the
batch_extract_files tool over shelling out — it takes the file list and a
config object and returns structured results directly.
Common pitfalls
- Default format differs —
batchdefaults to--format json,extractto--format text. Set--formatexplicitly if a script depends on one shape. --output-dirmust exist — the CLI does not create it.- Memory blowups — large batches with OCR/layout active need a lower
--max-concurrent; the default is CPU count. --file-configspath keys — must match the paths as passed on the command line, not absolute-resolved variants.
See references/cli-reference.md for the full batch flag set.
04 同仓库更多 Skills
worktable-review-with-annotations
Create, inspect, reply to, update, or resolve situated Worktable annotations. Use when feedback belongs to a Doc section, block, text range, or HTML Doc, or when the user asks to address existing review feedback and coordinate a requested artifact edit.
worktable-find-and-synthesize-context
Find, assess, and synthesize context from Worktable Docs and Records. Use when the user asks what a workspace contains, wants workspace-backed research, needs current facts reconciled across artifacts, or requests a cited synthesis without changing workspace state.
worktable-create-or-update-docs
Create, edit, rename, or maintain narrative Worktable Docs. Use for plans, decisions, briefs, notes, research syntheses, and other durable prose when the user requests a new artifact or a safe update to an existing one.
worktable-create-or-manage-records
Organize repeated structured information as durable Worktable Records. Use for research datasets, inventories, directories, pipelines, trackers, and other collections whose items need identity, fields, filtering, links, validation, or individual updates.
worktable-create-or-manage-interactive-html
Create or improve durable visual and interactive artifacts in Worktable. Use for dashboards, trackers, calculators, mini apps, slide presentations, and other requests that benefit from a polished HTML Doc.
worktable-collaborate-threads
Read, answer, and hand off messages through durable Worktable threads with users or other authenticated participants. Use when the user asks to check addressed messages, reply to a participant, continue a thread, request another agent's help, wait briefly for a response, or watch for comments and messages sent to the agent.