chunking
Use when splitting extracted text into chunks for LLM context windows or RAG ingestion. Covers chunk size, overlap, markdown/yaml/semantic chunkers, tokenizer-based sizing, and the standalone `chunk` command.
- License 许可
- Apache-2.0
- 最近更新
- 2026-11-14
- Stars
- ★ 1.6k
- Forks
- 491
01 前言 Front Matter
- name
- chunking
- description
- Use when splitting extracted text into chunks for LLM context windows or RAG ingestion. Covers chunk size, overlap, markdown/yaml/semantic chunkers, tokenizer-based sizing, and the standalone `chunk` command.
03 SKILL.md 正文
Chunking
Use this when feeding documents into an LLM context window or a vector
store. Kreuzberg chunks two ways: inline during extraction (chunks land on
result.chunks), or standalone via the chunk command for text you
already have. Sizing is character-based by default, or token-based when a
tokenizer model is supplied.
Inline during extraction
Turn on chunking with --chunk and the chunks appear on the structured
result under chunks:
# 1000-char chunks, 200-char overlap (defaults when --chunk is on)
kreuzberg extract report.pdf --chunk --format json | jq '.chunks | length'
# Explicit size + overlap
kreuzberg extract report.pdf --chunk --chunk-size 1500 --chunk-overlap 300 --format json
Overlap must be smaller than chunk size — the CLI rejects
--chunk-overlap >= --chunk-size. When you set only --chunk-overlap
against an existing config, an overlap that exceeds the size is clamped to
chunk_size / 4.
Standalone chunk command
Chunk text you already have, from --text or stdin. Output defaults to
JSON:
# From a flag
kreuzberg chunk --text "long document text ..." --chunk-size 800 --chunk-overlap 100
# From stdin (pipe extracted content straight in)
kreuzberg extract notes.md | kreuzberg chunk --chunk-size 500 --format json
JSON output carries chunks (array of strings), chunk_count, the
resolved config (max_characters, overlap, chunker_type), and
input_size_bytes. Use --format text for a human-readable dump with
--- chunk N --- separators.
Note: in the JSON output,
chunker_typeis rendered capitalized ("Text","Markdown","Yaml","Semantic") because it is emitted via Rust's Debug formatting, whereas the--chunker-typeinput flag is lowercase (text,markdown,yaml,semantic). Lowercase the value before comparing if you parse it back.
Chunker types
--chunker-type selects the splitting strategy (standalone chunk
command):
| Type | Behavior |
|---|---|
text |
Default. Plain character-window splitting with overlap. |
markdown |
Markdown-aware — splits on structure (headings, blocks) where possible. |
yaml |
YAML-aware splitting for structured config/data documents. |
semantic |
Topic-boundary splitting driven by --topic-threshold (0.0–1.0, default 0.75). |
# Markdown-aware chunking keeps headings and blocks intact
kreuzberg chunk --text "$(cat README.md)" --chunker-type markdown
# Semantic chunking — lower threshold = more, smaller topic chunks
kreuzberg chunk --text "$(cat transcript.txt)" --chunker-type semantic --topic-threshold 0.6
Token-based sizing
By default --chunk-size counts characters. To size chunks by tokens for
a specific model, pass --chunking-tokenizer with a HuggingFace tokenizer
id. On the extract command this implicitly enables chunking. Requires the
chunking-tokenizers feature (present in the default CLI build).
# Size chunks by GPT-4o tokens during extraction
kreuzberg extract report.pdf --chunking-tokenizer Xenova/gpt-4o --format json
# Or on the standalone command
kreuzberg chunk --text "$(cat doc.txt)" --chunking-tokenizer Xenova/gpt-4o --chunk-size 512
With a tokenizer set, --chunk-size is interpreted in tokens, not
characters.
Config file alternative
Field names in config files are snake_case under [chunking]:
[chunking]
max_characters = 1000
overlap = 200
chunker_type = "markdown"
kreuzberg extract report.pdf --config kreuzberg.toml --format json
CLI flags map to config fields as
--chunk-size→max_charactersand--chunk-overlap→overlap. In config files use the snake_case names.
Programmatic access
From Python, enable chunking on the config and read result.chunks:
from kreuzberg import extract_file_sync, ExtractionConfig, ChunkingConfig
config = ExtractionConfig(
chunking=ChunkingConfig(max_chars=1000, max_overlap=200),
)
result = extract_file_sync("report.pdf", config=config)
for chunk in result.chunks:
print(len(chunk))
Python
ChunkingConfigusesmax_chars/max_overlap. Rust usesmax_characters/overlap. Seereferences/python-api.mdandreferences/rust-api.mdin the siblingkreuzbergskill.
Picking parameters
- RAG / vector store — 500–1000 chars (or 256–512 tokens) with
10–20% overlap. Use
markdownchunking for docs to keep sections whole. - LLM summarization — larger chunks (1500–4000 chars) with small overlap; size by tokens to stay under the model window.
- Topic segmentation —
semanticchunker; tune--topic-thresholddown for finer splits, up for coarser ones.
Common pitfalls
- Overlap ≥ size — rejected on
extract; clamped tosize / 4when only overlap is changed against an existing config. - Tokenizer without the feature —
--chunking-tokenizererrors if the CLI was built withoutchunking-tokenizers. The default build includes it. - Empty input — the standalone
chunkcommand bails on empty text; provide--textor pipe non-empty stdin.
See references/configuration.md for the full [chunking] schema and
references/cli-reference.md for every chunk flag.
04 同仓库更多 Skills
worktable-review-with-annotations
Create, inspect, reply to, update, or resolve situated Worktable annotations. Use when feedback belongs to a Doc section, block, text range, or HTML Doc, or when the user asks to address existing review feedback and coordinate a requested artifact edit.
worktable-find-and-synthesize-context
Find, assess, and synthesize context from Worktable Docs and Records. Use when the user asks what a workspace contains, wants workspace-backed research, needs current facts reconciled across artifacts, or requests a cited synthesis without changing workspace state.
worktable-create-or-update-docs
Create, edit, rename, or maintain narrative Worktable Docs. Use for plans, decisions, briefs, notes, research syntheses, and other durable prose when the user requests a new artifact or a safe update to an existing one.
worktable-create-or-manage-records
Organize repeated structured information as durable Worktable Records. Use for research datasets, inventories, directories, pipelines, trackers, and other collections whose items need identity, fields, filtering, links, validation, or individual updates.
worktable-create-or-manage-interactive-html
Create or improve durable visual and interactive artifacts in Worktable. Use for dashboards, trackers, calculators, mini apps, slide presentations, and other requests that benefit from a polished HTML Doc.
worktable-collaborate-threads
Read, answer, and hand off messages through durable Worktable threads with users or other authenticated participants. Use when the user asks to check addressed messages, reply to a participant, continue a thread, request another agent's help, wait briefly for a response, or watch for comments and messages sent to the agent.