[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6XG3BV0TEM7X49HFJ69VJYH":3,"skill-same:01M6XG3BV0TEM7X49HFJ69VJYH":92},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":39,"linkOnly":40,"markdownContent":41,"markdownBody":42,"frontMatter":43,"refName":54,"commitSha":55,"skillDirectory":56,"forksCount":14,"repositoryHtmlUrl":57,"repositoryHomepageUrl":6,"ownerHtmlUrl":58,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":59,"install":60,"resources":73},"01M6XG3BV0TEM7X49HFJ69VJYH","huggingface-tokenizers",null,"Fast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in \u003C20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignments, handle padding\u002Ftruncation. Integrates seamlessly with transformers. Use when you need high-performance tokenization or custom tokenizer training.","fadrienne\u002FMyGitHub","fadrienne","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F197874571?v=4","01M6X64V8VWCRSMFZ4EPS2BWXG","Francesca Adrienne","01M6X64VD5MW4F9PHDWQKS7ZAF",0,"MIT","https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fopenscience\u002Fbackend\u002Fcli\u002Fskills\u002Fllm-tools\u002Fhuggingface-tokenizers\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38],"HuggingFace","huggingface tokenizers","Tokenization","when","track","you","Alignment Tracking","Custom Tokenizer","Fast Tokenization","Unigram","WordPiece","BPE","performance","high-performance","synthetic","alignment","Tools","Transformers","training","rust","2026-11-10 10:12:30",false,"---\nname: huggingface-tokenizers\ndescription: Fast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in \u003C20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignments, handle padding\u002Ftruncation. Integrates seamlessly with transformers. Use when you need high-performance tokenization or custom tokenizer training.\ncategory: llm-tools\nversion: 1.0.0\nauthor: Synthetic Sciences\nlicense: MIT\ntags: [Tokenization, HuggingFace, BPE, WordPiece, Unigram, Fast Tokenization, Rust, Custom Tokenizer, Alignment Tracking, Production]\ndependencies: [tokenizers, transformers, datasets]\n---\n\n# HuggingFace Tokenizers - Fast Tokenization for NLP\n\nFast, production-ready tokenizers with Rust performance and Python ease-of-use.\n\n## When to use HuggingFace Tokenizers\n\n**Use HuggingFace Tokenizers when:**\n- Need extremely fast tokenization (\u003C20s per GB of text)\n- Training custom tokenizers from scratch\n- Want alignment tracking (token → original text position)\n- Building production NLP pipelines\n- Need to tokenize large corpora efficiently\n\n**Performance**:\n- **Speed**: \u003C20 seconds to tokenize 1GB on CPU\n- **Implementation**: Rust core with Python\u002FNode.js bindings\n- **Efficiency**: 10-100× faster than pure Python implementations\n\n**Use alternatives instead**:\n- **SentencePiece**: Language-independent, used by T5\u002FALBERT\n- **tiktoken**: OpenAI's BPE tokenizer for GPT models\n- **transformers AutoTokenizer**: Loading pretrained only (uses this library internally)\n\n## Quick start\n\n### Installation\n\n```bash\n# Install tokenizers\npip install tokenizers\n\n# With transformers integration\npip install tokenizers transformers\n```\n\n### Load pretrained tokenizer\n\n```python\nfrom tokenizers import Tokenizer\n\n# Load from HuggingFace Hub\ntokenizer = Tokenizer.from_pretrained(\"bert-base-uncased\")\n\n# Encode text\noutput = tokenizer.encode(\"Hello, how are you?\")\nprint(output.tokens)  # ['hello', ',', 'how', 'are', 'you', '?']\nprint(output.ids)     # [7592, 1010, 2129, 2024, 2017, 1029]\n\n# Decode back\ntext = tokenizer.decode(output.ids)\nprint(text)  # \"hello, how are you?\"\n```\n\n### Train custom BPE tokenizer\n\n```python\nfrom tokenizers import Tokenizer\nfrom tokenizers.models import BPE\nfrom tokenizers.trainers import BpeTrainer\nfrom tokenizers.pre_tokenizers import Whitespace\n\n# Initialize tokenizer with BPE model\ntokenizer = Tokenizer(BPE(unk_token=\"[UNK]\"))\ntokenizer.pre_tokenizer = Whitespace()\n\n# Configure trainer\ntrainer = BpeTrainer(\n    vocab_size=30000,\n    special_tokens=[\"[UNK]\", \"[CLS]\", \"[SEP]\", \"[PAD]\", \"[MASK]\"],\n    min_frequency=2\n)\n\n# Train on files\nfiles = [\"train.txt\", \"validation.txt\"]\ntokenizer.train(files, trainer)\n\n# Save\ntokenizer.save(\"my-tokenizer.json\")\n```\n\n**Training time**: ~1-2 minutes for 100MB corpus, ~10-20 minutes for 1GB\n\n### Batch encoding with padding\n\n```python\n# Enable padding\ntokenizer.enable_padding(pad_id=3, pad_token=\"[PAD]\")\n\n# Encode batch\ntexts = [\"Hello world\", \"This is a longer sentence\"]\nencodings = tokenizer.encode_batch(texts)\n\nfor encoding in encodings:\n    print(encoding.ids)\n# [101, 7592, 2088, 102, 3, 3, 3]\n# [101, 2023, 2003, 1037, 2936, 6251, 102]\n```\n\n## Tokenization algorithms\n\n### BPE (Byte-Pair Encoding)\n\n**How it works**:\n1. Start with character-level vocabulary\n2. Find most frequent character pair\n3. Merge into new token, add to vocabulary\n4. Repeat until vocabulary size reached\n\n**Used by**: GPT-2, GPT-3, RoBERTa, BART, DeBERTa\n\n```python\nfrom tokenizers import Tokenizer\nfrom tokenizers.models import BPE\nfrom tokenizers.trainers import BpeTrainer\nfrom tokenizers.pre_tokenizers import ByteLevel\n\ntokenizer = Tokenizer(BPE(unk_token=\"\u003C|endoftext|>\"))\ntokenizer.pre_tokenizer = ByteLevel()\n\ntrainer = BpeTrainer(\n    vocab_size=50257,\n    special_tokens=[\"\u003C|endoftext|>\"],\n    min_frequency=2\n)\n\ntokenizer.train(files=[\"data.txt\"], trainer=trainer)\n```\n\n**Advantages**:\n- Handles OOV words well (breaks into subwords)\n- Flexible vocabulary size\n- Good for morphologically rich languages\n\n**Trade-offs**:\n- Tokenization depends on merge order\n- May split common words unexpectedly\n\n### WordPiece\n\n**How it works**:\n1. Start with character vocabulary\n2. Score merge pairs: `frequency(pair) \u002F (frequency(first) × frequency(second))`\n3. Merge highest scoring pair\n4. Repeat until vocabulary size reached\n\n**Used by**: BERT, DistilBERT, MobileBERT\n\n```python\nfrom tokenizers import Tokenizer\nfrom tokenizers.models import WordPiece\nfrom tokenizers.trainers import WordPieceTrainer\nfrom tokenizers.pre_tokenizers import Whitespace\nfrom tokenizers.normalizers import BertNormalizer\n\ntokenizer = Tokenizer(WordPiece(unk_token=\"[UNK]\"))\ntokenizer.normalizer = BertNormalizer(lowercase=True)\ntokenizer.pre_tokenizer = Whitespace()\n\ntrainer = WordPieceTrainer(\n    vocab_size=30522,\n    special_tokens=[\"[UNK]\", \"[CLS]\", \"[SEP]\", \"[PAD]\", \"[MASK]\"],\n    continuing_subword_prefix=\"##\"\n)\n\ntokenizer.train(files=[\"corpus.txt\"], trainer=trainer)\n```\n\n**Advantages**:\n- Prioritizes meaningful merges (high score = semantically related)\n- Used successfully in BERT (state-of-the-art results)\n\n**Trade-offs**:\n- Unknown words become `[UNK]` if no subword match\n- Saves vocabulary, not merge rules (larger files)\n\n### Unigram\n\n**How it works**:\n1. Start with large vocabulary (all substrings)\n2. Compute loss for corpus with current vocabulary\n3. Remove tokens with minimal impact on loss\n4. Repeat until vocabulary size reached\n\n**Used by**: ALBERT, T5, mBART, XLNet (via SentencePiece)\n\n```python\nfrom tokenizers import Tokenizer\nfrom tokenizers.models import Unigram\nfrom tokenizers.trainers import UnigramTrainer\n\ntokenizer = Tokenizer(Unigram())\n\ntrainer = UnigramTrainer(\n    vocab_size=8000,\n    special_tokens=[\"\u003Cunk>\", \"\u003Cs>\", \"\u003C\u002Fs>\"],\n    unk_token=\"\u003Cunk>\"\n)\n\ntokenizer.train(files=[\"data.txt\"], trainer=trainer)\n```\n\n**Advantages**:\n- Probabilistic (finds most likely tokenization)\n- Works well for languages without word boundaries\n- Handles diverse linguistic contexts\n\n**Trade-offs**:\n- Computationally expensive to train\n- More hyperparameters to tune\n\n## Tokenization pipeline\n\nComplete pipeline: **Normalization → Pre-tokenization → Model → Post-processing**\n\n### Normalization\n\nClean and standardize text:\n\n```python\nfrom tokenizers.normalizers import NFD, StripAccents, Lowercase, Sequence\n\ntokenizer.normalizer = Sequence([\n    NFD(),           # Unicode normalization (decompose)\n    Lowercase(),     # Convert to lowercase\n    StripAccents()   # Remove accents\n])\n\n# Input: \"Héllo WORLD\"\n# After normalization: \"hello world\"\n```\n\n**Common normalizers**:\n- `NFD`, `NFC`, `NFKD`, `NFKC` - Unicode normalization forms\n- `Lowercase()` - Convert to lowercase\n- `StripAccents()` - Remove accents (é → e)\n- `Strip()` - Remove whitespace\n- `Replace(pattern, content)` - Regex replacement\n\n### Pre-tokenization\n\nSplit text into word-like units:\n\n```python\nfrom tokenizers.pre_tokenizers import Whitespace, Punctuation, Sequence, ByteLevel\n\n# Split on whitespace and punctuation\ntokenizer.pre_tokenizer = Sequence([\n    Whitespace(),\n    Punctuation()\n])\n\n# Input: \"Hello, world!\"\n# After pre-tokenization: [\"Hello\", \",\", \"world\", \"!\"]\n```\n\n**Common pre-tokenizers**:\n- `Whitespace()` - Split on spaces, tabs, newlines\n- `ByteLevel()` - GPT-2 style byte-level splitting\n- `Punctuation()` - Isolate punctuation\n- `Digits(individual_digits=True)` - Split digits individually\n- `Metaspace()` - Replace spaces with ▁ (SentencePiece style)\n\n### Post-processing\n\nAdd special tokens for model input:\n\n```python\nfrom tokenizers.processors import TemplateProcessing\n\n# BERT-style: [CLS] sentence [SEP]\ntokenizer.post_processor = TemplateProcessing(\n    single=\"[CLS] $A [SEP]\",\n    pair=\"[CLS] $A [SEP] $B [SEP]\",\n    special_tokens=[\n        (\"[CLS]\", 1),\n        (\"[SEP]\", 2),\n    ],\n)\n```\n\n**Common patterns**:\n```python\n# GPT-2: sentence \u003C|endoftext|>\nTemplateProcessing(\n    single=\"$A \u003C|endoftext|>\",\n    special_tokens=[(\"\u003C|endoftext|>\", 50256)]\n)\n\n# RoBERTa: \u003Cs> sentence \u003C\u002Fs>\nTemplateProcessing(\n    single=\"\u003Cs> $A \u003C\u002Fs>\",\n    pair=\"\u003Cs> $A \u003C\u002Fs> \u003C\u002Fs> $B \u003C\u002Fs>\",\n    special_tokens=[(\"\u003Cs>\", 0), (\"\u003C\u002Fs>\", 2)]\n)\n```\n\n## Alignment tracking\n\nTrack token positions in original text:\n\n```python\noutput = tokenizer.encode(\"Hello, world!\")\n\n# Get token offsets\nfor token, offset in zip(output.tokens, output.offsets):\n    start, end = offset\n    print(f\"{token:10} → [{start:2}, {end:2}): {text[start:end]!r}\")\n\n# Output:\n# hello      → [ 0,  5): 'Hello'\n# ,          → [ 5,  6): ','\n# world      → [ 7, 12): 'world'\n# !          → [12, 13): '!'\n```\n\n**Use cases**:\n- Named entity recognition (map predictions back to text)\n- Question answering (extract answer spans)\n- Token classification (align labels to original positions)\n\n## Integration with transformers\n\n### Load with AutoTokenizer\n\n```python\nfrom transformers import AutoTokenizer\n\n# AutoTokenizer automatically uses fast tokenizers\ntokenizer = AutoTokenizer.from_pretrained(\"bert-base-uncased\")\n\n# Check if using fast tokenizer\nprint(tokenizer.is_fast)  # True\n\n# Access underlying tokenizers.Tokenizer\nfast_tokenizer = tokenizer.backend_tokenizer\nprint(type(fast_tokenizer))  # \u003Cclass 'tokenizers.Tokenizer'>\n```\n\n### Convert custom tokenizer to transformers\n\n```python\nfrom tokenizers import Tokenizer\nfrom transformers import PreTrainedTokenizerFast\n\n# Train custom tokenizer\ntokenizer = Tokenizer(BPE())\n# ... train tokenizer ...\ntokenizer.save(\"my-tokenizer.json\")\n\n# Wrap for transformers\ntransformers_tokenizer = PreTrainedTokenizerFast(\n    tokenizer_file=\"my-tokenizer.json\",\n    unk_token=\"[UNK]\",\n    pad_token=\"[PAD]\",\n    cls_token=\"[CLS]\",\n    sep_token=\"[SEP]\",\n    mask_token=\"[MASK]\"\n)\n\n# Use like any transformers tokenizer\noutputs = transformers_tokenizer(\n    \"Hello world\",\n    padding=True,\n    truncation=True,\n    max_length=512,\n    return_tensors=\"pt\"\n)\n```\n\n## Common patterns\n\n### Train from iterator (large datasets)\n\n```python\nfrom datasets import load_dataset\n\n# Load dataset\ndataset = load_dataset(\"wikitext\", \"wikitext-103-raw-v1\", split=\"train\")\n\n# Create batch iterator\ndef batch_iterator(batch_size=1000):\n    for i in range(0, len(dataset), batch_size):\n        yield dataset[i:i + batch_size][\"text\"]\n\n# Train tokenizer\ntokenizer.train_from_iterator(\n    batch_iterator(),\n    trainer=trainer,\n    length=len(dataset)  # For progress bar\n)\n```\n\n**Performance**: Processes 1GB in ~10-20 minutes\n\n### Enable truncation and padding\n\n```python\n# Enable truncation\ntokenizer.enable_truncation(max_length=512)\n\n# Enable padding\ntokenizer.enable_padding(\n    pad_id=tokenizer.token_to_id(\"[PAD]\"),\n    pad_token=\"[PAD]\",\n    length=512  # Fixed length, or None for batch max\n)\n\n# Encode with both\noutput = tokenizer.encode(\"This is a long sentence that will be truncated...\")\nprint(len(output.ids))  # 512\n```\n\n### Multi-processing\n\n```python\nfrom tokenizers import Tokenizer\nfrom multiprocessing import Pool\n\n# Load tokenizer\ntokenizer = Tokenizer.from_file(\"tokenizer.json\")\n\ndef encode_batch(texts):\n    return tokenizer.encode_batch(texts)\n\n# Process large corpus in parallel\nwith Pool(8) as pool:\n    # Split corpus into chunks\n    chunk_size = 1000\n    chunks = [corpus[i:i+chunk_size] for i in range(0, len(corpus), chunk_size)]\n\n    # Encode in parallel\n    results = pool.map(encode_batch, chunks)\n```\n\n**Speedup**: 5-8× with 8 cores\n\n## Performance benchmarks\n\n### Training speed\n\n| Corpus Size | BPE (30k vocab) | WordPiece (30k) | Unigram (8k) |\n|-------------|-----------------|-----------------|--------------|\n| 10 MB       | 15 sec          | 18 sec          | 25 sec       |\n| 100 MB      | 1.5 min         | 2 min           | 4 min        |\n| 1 GB        | 15 min          | 20 min          | 40 min       |\n\n**Hardware**: 16-core CPU, tested on English Wikipedia\n\n### Tokenization speed\n\n| Implementation | 1 GB corpus | Throughput    |\n|----------------|-------------|---------------|\n| Pure Python    | ~20 minutes | ~50 MB\u002Fmin    |\n| HF Tokenizers  | ~15 seconds | ~4 GB\u002Fmin     |\n| **Speedup**    | **80×**     | **80×**       |\n\n**Test**: English text, average sentence length 20 words\n\n### Memory usage\n\n| Task                    | Memory  |\n|-------------------------|---------|\n| Load tokenizer          | ~10 MB  |\n| Train BPE (30k vocab)   | ~200 MB |\n| Encode 1M sentences     | ~500 MB |\n\n## Supported models\n\nPre-trained tokenizers available via `from_pretrained()`:\n\n**BERT family**:\n- `bert-base-uncased`, `bert-large-cased`\n- `distilbert-base-uncased`\n- `roberta-base`, `roberta-large`\n\n**GPT family**:\n- `gpt2`, `gpt2-medium`, `gpt2-large`\n- `distilgpt2`\n\n**T5 family**:\n- `t5-small`, `t5-base`, `t5-large`\n- `google\u002Fflan-t5-xxl`\n\n**Other**:\n- `facebook\u002Fbart-base`, `facebook\u002Fmbart-large-cc25`\n- `albert-base-v2`, `albert-xlarge-v2`\n- `xlm-roberta-base`, `xlm-roberta-large`\n\nBrowse all: https:\u002F\u002Fhuggingface.co\u002Fmodels?library=tokenizers\n\n## References\n\n- **[Training Guide](references\u002Ftraining.md)** - Train custom tokenizers, configure trainers, handle large datasets\n- **[Algorithms Deep Dive](references\u002Falgorithms.md)** - BPE, WordPiece, Unigram explained in detail\n- **[Pipeline Components](references\u002Fpipeline.md)** - Normalizers, pre-tokenizers, post-processors, decoders\n- **[Transformers Integration](references\u002Fintegration.md)** - AutoTokenizer, PreTrainedTokenizerFast, special tokens\n\n## Resources\n\n- **Docs**: https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Ftokenizers\n- **GitHub**: https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftokenizers ⭐ 9,000+\n- **Version**: 0.20.0+\n- **Course**: https:\u002F\u002Fhuggingface.co\u002Flearn\u002Fnlp-course\u002Fchapter6\u002F1\n- **Paper**: BPE (Sennrich et al., 2016), WordPiece (Schuster & Nakajima, 2012)\n\n\n","# HuggingFace Tokenizers - Fast Tokenization for NLP\n\nFast, production-ready tokenizers with Rust performance and Python ease-of-use.\n\n## When to use HuggingFace Tokenizers\n\n**Use HuggingFace Tokenizers when:**\n- Need extremely fast tokenization (\u003C20s per GB of text)\n- Training custom tokenizers from scratch\n- Want alignment tracking (token → original text position)\n- Building production NLP pipelines\n- Need to tokenize large corpora efficiently\n\n**Performance**:\n- **Speed**: \u003C20 seconds to tokenize 1GB on CPU\n- **Implementation**: Rust core with Python\u002FNode.js bindings\n- **Efficiency**: 10-100× faster than pure Python implementations\n\n**Use alternatives instead**:\n- **SentencePiece**: Language-independent, used by T5\u002FALBERT\n- **tiktoken**: OpenAI's BPE tokenizer for GPT models\n- **transformers AutoTokenizer**: Loading pretrained only (uses this library internally)\n\n## Quick start\n\n### Installation\n\n```bash\n# Install tokenizers\npip install tokenizers\n\n# With transformers integration\npip install tokenizers transformers\n```\n\n### Load pretrained tokenizer\n\n```python\nfrom tokenizers import Tokenizer\n\n# Load from HuggingFace Hub\ntokenizer = Tokenizer.from_pretrained(\"bert-base-uncased\")\n\n# Encode text\noutput = tokenizer.encode(\"Hello, how are you?\")\nprint(output.tokens)  # ['hello', ',', 'how', 'are', 'you', '?']\nprint(output.ids)     # [7592, 1010, 2129, 2024, 2017, 1029]\n\n# Decode back\ntext = tokenizer.decode(output.ids)\nprint(text)  # \"hello, how are you?\"\n```\n\n### Train custom BPE tokenizer\n\n```python\nfrom tokenizers import Tokenizer\nfrom tokenizers.models import BPE\nfrom tokenizers.trainers import BpeTrainer\nfrom tokenizers.pre_tokenizers import Whitespace\n\n# Initialize tokenizer with BPE model\ntokenizer = Tokenizer(BPE(unk_token=\"[UNK]\"))\ntokenizer.pre_tokenizer = Whitespace()\n\n# Configure trainer\ntrainer = BpeTrainer(\n    vocab_size=30000,\n    special_tokens=[\"[UNK]\", \"[CLS]\", \"[SEP]\", \"[PAD]\", \"[MASK]\"],\n    min_frequency=2\n)\n\n# Train on files\nfiles = [\"train.txt\", \"validation.txt\"]\ntokenizer.train(files, trainer)\n\n# Save\ntokenizer.save(\"my-tokenizer.json\")\n```\n\n**Training time**: ~1-2 minutes for 100MB corpus, ~10-20 minutes for 1GB\n\n### Batch encoding with padding\n\n```python\n# Enable padding\ntokenizer.enable_padding(pad_id=3, pad_token=\"[PAD]\")\n\n# Encode batch\ntexts = [\"Hello world\", \"This is a longer sentence\"]\nencodings = tokenizer.encode_batch(texts)\n\nfor encoding in encodings:\n    print(encoding.ids)\n# [101, 7592, 2088, 102, 3, 3, 3]\n# [101, 2023, 2003, 1037, 2936, 6251, 102]\n```\n\n## Tokenization algorithms\n\n### BPE (Byte-Pair Encoding)\n\n**How it works**:\n1. Start with character-level vocabulary\n2. Find most frequent character pair\n3. Merge into new token, add to vocabulary\n4. Repeat until vocabulary size reached\n\n**Used by**: GPT-2, GPT-3, RoBERTa, BART, DeBERTa\n\n```python\nfrom tokenizers import Tokenizer\nfrom tokenizers.models import BPE\nfrom tokenizers.trainers import BpeTrainer\nfrom tokenizers.pre_tokenizers import ByteLevel\n\ntokenizer = Tokenizer(BPE(unk_token=\"\u003C|endoftext|>\"))\ntokenizer.pre_tokenizer = ByteLevel()\n\ntrainer = BpeTrainer(\n    vocab_size=50257,\n    special_tokens=[\"\u003C|endoftext|>\"],\n    min_frequency=2\n)\n\ntokenizer.train(files=[\"data.txt\"], trainer=trainer)\n```\n\n**Advantages**:\n- Handles OOV words well (breaks into subwords)\n- Flexible vocabulary size\n- Good for morphologically rich languages\n\n**Trade-offs**:\n- Tokenization depends on merge order\n- May split common words unexpectedly\n\n### WordPiece\n\n**How it works**:\n1. Start with character vocabulary\n2. Score merge pairs: `frequency(pair) \u002F (frequency(first) × frequency(second))`\n3. Merge highest scoring pair\n4. Repeat until vocabulary size reached\n\n**Used by**: BERT, DistilBERT, MobileBERT\n\n```python\nfrom tokenizers import Tokenizer\nfrom tokenizers.models import WordPiece\nfrom tokenizers.trainers import WordPieceTrainer\nfrom tokenizers.pre_tokenizers import Whitespace\nfrom tokenizers.normalizers import BertNormalizer\n\ntokenizer = Tokenizer(WordPiece(unk_token=\"[UNK]\"))\ntokenizer.normalizer = BertNormalizer(lowercase=True)\ntokenizer.pre_tokenizer = Whitespace()\n\ntrainer = WordPieceTrainer(\n    vocab_size=30522,\n    special_tokens=[\"[UNK]\", \"[CLS]\", \"[SEP]\", \"[PAD]\", \"[MASK]\"],\n    continuing_subword_prefix=\"##\"\n)\n\ntokenizer.train(files=[\"corpus.txt\"], trainer=trainer)\n```\n\n**Advantages**:\n- Prioritizes meaningful merges (high score = semantically related)\n- Used successfully in BERT (state-of-the-art results)\n\n**Trade-offs**:\n- Unknown words become `[UNK]` if no subword match\n- Saves vocabulary, not merge rules (larger files)\n\n### Unigram\n\n**How it works**:\n1. Start with large vocabulary (all substrings)\n2. Compute loss for corpus with current vocabulary\n3. Remove tokens with minimal impact on loss\n4. Repeat until vocabulary size reached\n\n**Used by**: ALBERT, T5, mBART, XLNet (via SentencePiece)\n\n```python\nfrom tokenizers import Tokenizer\nfrom tokenizers.models import Unigram\nfrom tokenizers.trainers import UnigramTrainer\n\ntokenizer = Tokenizer(Unigram())\n\ntrainer = UnigramTrainer(\n    vocab_size=8000,\n    special_tokens=[\"\u003Cunk>\", \"\u003Cs>\", \"\u003C\u002Fs>\"],\n    unk_token=\"\u003Cunk>\"\n)\n\ntokenizer.train(files=[\"data.txt\"], trainer=trainer)\n```\n\n**Advantages**:\n- Probabilistic (finds most likely tokenization)\n- Works well for languages without word boundaries\n- Handles diverse linguistic contexts\n\n**Trade-offs**:\n- Computationally expensive to train\n- More hyperparameters to tune\n\n## Tokenization pipeline\n\nComplete pipeline: **Normalization → Pre-tokenization → Model → Post-processing**\n\n### Normalization\n\nClean and standardize text:\n\n```python\nfrom tokenizers.normalizers import NFD, StripAccents, Lowercase, Sequence\n\ntokenizer.normalizer = Sequence([\n    NFD(),           # Unicode normalization (decompose)\n    Lowercase(),     # Convert to lowercase\n    StripAccents()   # Remove accents\n])\n\n# Input: \"Héllo WORLD\"\n# After normalization: \"hello world\"\n```\n\n**Common normalizers**:\n- `NFD`, `NFC`, `NFKD`, `NFKC` - Unicode normalization forms\n- `Lowercase()` - Convert to lowercase\n- `StripAccents()` - Remove accents (é → e)\n- `Strip()` - Remove whitespace\n- `Replace(pattern, content)` - Regex replacement\n\n### Pre-tokenization\n\nSplit text into word-like units:\n\n```python\nfrom tokenizers.pre_tokenizers import Whitespace, Punctuation, Sequence, ByteLevel\n\n# Split on whitespace and punctuation\ntokenizer.pre_tokenizer = Sequence([\n    Whitespace(),\n    Punctuation()\n])\n\n# Input: \"Hello, world!\"\n# After pre-tokenization: [\"Hello\", \",\", \"world\", \"!\"]\n```\n\n**Common pre-tokenizers**:\n- `Whitespace()` - Split on spaces, tabs, newlines\n- `ByteLevel()` - GPT-2 style byte-level splitting\n- `Punctuation()` - Isolate punctuation\n- `Digits(individual_digits=True)` - Split digits individually\n- `Metaspace()` - Replace spaces with ▁ (SentencePiece style)\n\n### Post-processing\n\nAdd special tokens for model input:\n\n```python\nfrom tokenizers.processors import TemplateProcessing\n\n# BERT-style: [CLS] sentence [SEP]\ntokenizer.post_processor = TemplateProcessing(\n    single=\"[CLS] $A [SEP]\",\n    pair=\"[CLS] $A [SEP] $B [SEP]\",\n    special_tokens=[\n        (\"[CLS]\", 1),\n        (\"[SEP]\", 2),\n    ],\n)\n```\n\n**Common patterns**:\n```python\n# GPT-2: sentence \u003C|endoftext|>\nTemplateProcessing(\n    single=\"$A \u003C|endoftext|>\",\n    special_tokens=[(\"\u003C|endoftext|>\", 50256)]\n)\n\n# RoBERTa: \u003Cs> sentence \u003C\u002Fs>\nTemplateProcessing(\n    single=\"\u003Cs> $A \u003C\u002Fs>\",\n    pair=\"\u003Cs> $A \u003C\u002Fs> \u003C\u002Fs> $B \u003C\u002Fs>\",\n    special_tokens=[(\"\u003Cs>\", 0), (\"\u003C\u002Fs>\", 2)]\n)\n```\n\n## Alignment tracking\n\nTrack token positions in original text:\n\n```python\noutput = tokenizer.encode(\"Hello, world!\")\n\n# Get token offsets\nfor token, offset in zip(output.tokens, output.offsets):\n    start, end = offset\n    print(f\"{token:10} → [{start:2}, {end:2}): {text[start:end]!r}\")\n\n# Output:\n# hello      → [ 0,  5): 'Hello'\n# ,          → [ 5,  6): ','\n# world      → [ 7, 12): 'world'\n# !          → [12, 13): '!'\n```\n\n**Use cases**:\n- Named entity recognition (map predictions back to text)\n- Question answering (extract answer spans)\n- Token classification (align labels to original positions)\n\n## Integration with transformers\n\n### Load with AutoTokenizer\n\n```python\nfrom transformers import AutoTokenizer\n\n# AutoTokenizer automatically uses fast tokenizers\ntokenizer = AutoTokenizer.from_pretrained(\"bert-base-uncased\")\n\n# Check if using fast tokenizer\nprint(tokenizer.is_fast)  # True\n\n# Access underlying tokenizers.Tokenizer\nfast_tokenizer = tokenizer.backend_tokenizer\nprint(type(fast_tokenizer))  # \u003Cclass 'tokenizers.Tokenizer'>\n```\n\n### Convert custom tokenizer to transformers\n\n```python\nfrom tokenizers import Tokenizer\nfrom transformers import PreTrainedTokenizerFast\n\n# Train custom tokenizer\ntokenizer = Tokenizer(BPE())\n# ... train tokenizer ...\ntokenizer.save(\"my-tokenizer.json\")\n\n# Wrap for transformers\ntransformers_tokenizer = PreTrainedTokenizerFast(\n    tokenizer_file=\"my-tokenizer.json\",\n    unk_token=\"[UNK]\",\n    pad_token=\"[PAD]\",\n    cls_token=\"[CLS]\",\n    sep_token=\"[SEP]\",\n    mask_token=\"[MASK]\"\n)\n\n# Use like any transformers tokenizer\noutputs = transformers_tokenizer(\n    \"Hello world\",\n    padding=True,\n    truncation=True,\n    max_length=512,\n    return_tensors=\"pt\"\n)\n```\n\n## Common patterns\n\n### Train from iterator (large datasets)\n\n```python\nfrom datasets import load_dataset\n\n# Load dataset\ndataset = load_dataset(\"wikitext\", \"wikitext-103-raw-v1\", split=\"train\")\n\n# Create batch iterator\ndef batch_iterator(batch_size=1000):\n    for i in range(0, len(dataset), batch_size):\n        yield dataset[i:i + batch_size][\"text\"]\n\n# Train tokenizer\ntokenizer.train_from_iterator(\n    batch_iterator(),\n    trainer=trainer,\n    length=len(dataset)  # For progress bar\n)\n```\n\n**Performance**: Processes 1GB in ~10-20 minutes\n\n### Enable truncation and padding\n\n```python\n# Enable truncation\ntokenizer.enable_truncation(max_length=512)\n\n# Enable padding\ntokenizer.enable_padding(\n    pad_id=tokenizer.token_to_id(\"[PAD]\"),\n    pad_token=\"[PAD]\",\n    length=512  # Fixed length, or None for batch max\n)\n\n# Encode with both\noutput = tokenizer.encode(\"This is a long sentence that will be truncated...\")\nprint(len(output.ids))  # 512\n```\n\n### Multi-processing\n\n```python\nfrom tokenizers import Tokenizer\nfrom multiprocessing import Pool\n\n# Load tokenizer\ntokenizer = Tokenizer.from_file(\"tokenizer.json\")\n\ndef encode_batch(texts):\n    return tokenizer.encode_batch(texts)\n\n# Process large corpus in parallel\nwith Pool(8) as pool:\n    # Split corpus into chunks\n    chunk_size = 1000\n    chunks = [corpus[i:i+chunk_size] for i in range(0, len(corpus), chunk_size)]\n\n    # Encode in parallel\n    results = pool.map(encode_batch, chunks)\n```\n\n**Speedup**: 5-8× with 8 cores\n\n## Performance benchmarks\n\n### Training speed\n\n| Corpus Size | BPE (30k vocab) | WordPiece (30k) | Unigram (8k) |\n|-------------|-----------------|-----------------|--------------|\n| 10 MB       | 15 sec          | 18 sec          | 25 sec       |\n| 100 MB      | 1.5 min         | 2 min           | 4 min        |\n| 1 GB        | 15 min          | 20 min          | 40 min       |\n\n**Hardware**: 16-core CPU, tested on English Wikipedia\n\n### Tokenization speed\n\n| Implementation | 1 GB corpus | Throughput    |\n|----------------|-------------|---------------|\n| Pure Python    | ~20 minutes | ~50 MB\u002Fmin    |\n| HF Tokenizers  | ~15 seconds | ~4 GB\u002Fmin     |\n| **Speedup**    | **80×**     | **80×**       |\n\n**Test**: English text, average sentence length 20 words\n\n### Memory usage\n\n| Task                    | Memory  |\n|-------------------------|---------|\n| Load tokenizer          | ~10 MB  |\n| Train BPE (30k vocab)   | ~200 MB |\n| Encode 1M sentences     | ~500 MB |\n\n## Supported models\n\nPre-trained tokenizers available via `from_pretrained()`:\n\n**BERT family**:\n- `bert-base-uncased`, `bert-large-cased`\n- `distilbert-base-uncased`\n- `roberta-base`, `roberta-large`\n\n**GPT family**:\n- `gpt2`, `gpt2-medium`, `gpt2-large`\n- `distilgpt2`\n\n**T5 family**:\n- `t5-small`, `t5-base`, `t5-large`\n- `google\u002Fflan-t5-xxl`\n\n**Other**:\n- `facebook\u002Fbart-base`, `facebook\u002Fmbart-large-cc25`\n- `albert-base-v2`, `albert-xlarge-v2`\n- `xlm-roberta-base`, `xlm-roberta-large`\n\nBrowse all: https:\u002F\u002Fhuggingface.co\u002Fmodels?library=tokenizers\n\n## References\n\n- **[Training Guide](references\u002Ftraining.md)** - Train custom tokenizers, configure trainers, handle large datasets\n- **[Algorithms Deep Dive](references\u002Falgorithms.md)** - BPE, WordPiece, Unigram explained in detail\n- **[Pipeline Components](references\u002Fpipeline.md)** - Normalizers, pre-tokenizers, post-processors, decoders\n- **[Transformers Integration](references\u002Fintegration.md)** - AutoTokenizer, PreTrainedTokenizerFast, special tokens\n\n## Resources\n\n- **Docs**: https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Ftokenizers\n- **GitHub**: https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftokenizers ⭐ 9,000+\n- **Version**: 0.20.0+\n- **Course**: https:\u002F\u002Fhuggingface.co\u002Flearn\u002Fnlp-course\u002Fchapter6\u002F1\n- **Paper**: BPE (Sennrich et al., 2016), WordPiece (Schuster & Nakajima, 2012)\n\n\n",{"name":5,"tags":44,"author":47,"license":15,"version":48,"category":49,"description":7,"dependencies":50},[21,19,30,29,28,27,45,26,25,46],"Rust","Production","Synthetic Sciences","1.0.0","llm-tools",[51,52,53],"tokenizers","transformers","datasets","main","4aa39efd1cdab0b4657ecee16c92a92011a5b10f","openscience\u002Fbackend\u002Fcli\u002Fskills\u002Fllm-tools\u002Fhuggingface-tokenizers","https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub","https:\u002F\u002Fgithub.com\u002Ffadrienne","2026-10-28 00:43:09",{"source":8,"skillName":5,"commands":61,"prompt":71,"zipAvailable":72},[62,65,68],{"tool":63,"command":64},"npx","npx skills add fadrienne\u002FMyGitHub --skill huggingface-tokenizers",{"tool":66,"command":67},"bunx","bunx skills add fadrienne\u002FMyGitHub --skill huggingface-tokenizers",{"tool":69,"command":70},"pnpm","pnpm dlx skills add fadrienne\u002FMyGitHub --skill huggingface-tokenizers","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fopenscience\u002Fbackend\u002Fcli\u002Fskills\u002Fllm-tools\u002Fhuggingface-tokenizers\u002FSKILL.md",true,[74,80,84,88],{"path":75,"resourceType":76,"resourceTypeDesc":77,"sizeBytes":78,"sourceHtmlUrl":79},"references\u002Falgorithms.md","REFERENCE","参考文档",15182,"https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fopenscience\u002Fbackend\u002Fcli\u002Fskills\u002Fllm-tools\u002Fhuggingface-tokenizers\u002Freferences\u002Falgorithms.md",{"path":81,"resourceType":76,"resourceTypeDesc":77,"sizeBytes":82,"sourceHtmlUrl":83},"references\u002Fintegration.md",15389,"https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fopenscience\u002Fbackend\u002Fcli\u002Fskills\u002Fllm-tools\u002Fhuggingface-tokenizers\u002Freferences\u002Fintegration.md",{"path":85,"resourceType":76,"resourceTypeDesc":77,"sizeBytes":86,"sourceHtmlUrl":87},"references\u002Fpipeline.md",16746,"https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fopenscience\u002Fbackend\u002Fcli\u002Fskills\u002Fllm-tools\u002Fhuggingface-tokenizers\u002Freferences\u002Fpipeline.md",{"path":89,"resourceType":76,"resourceTypeDesc":77,"sizeBytes":90,"sourceHtmlUrl":91},"references\u002Ftraining.md",14559,"https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fopenscience\u002Fbackend\u002Fcli\u002Fskills\u002Fllm-tools\u002Fhuggingface-tokenizers\u002Freferences\u002Ftraining.md",[93,124,150,184,211,238],{"skillNo":94,"name":95,"nameZh":6,"description":96,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":97,"categories":98,"keywords":104,"publishedAt":39},"01M7167B28PK20HP7R7ZED2DY0","statistical-power","Sample size calculation, power analysis, and effect size conversion for scientific studies. Use BEFORE data collection (a priori) to determine required N, or after (post-hoc) to compute achieved power. Covers parametric tests (t-test, ANOVA, chi-square, proportions, correlation, regression), simulation-based power for complex\u002Fclustered\u002Fnested designs, and power curves. Use this skill when someone asks 'how many samples do I need', 'what is my power', 'what effect size can I detect', or any variant. Pairs with \u002Fexperimental-design (choose design first, then compute power) and \u002Fstatistical-analysis (analyze data after collection).","https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fscientific-agent-skills\u002Fskills\u002Fstatistical-power\u002FSKILL.md",[99],{"categoryNo":100,"categoryName":101,"taxonomyType":102,"taxonomyTypeDesc":103},"01JBSEEDCAT000000000000023","Skill与智能体开发","SCENARIO","使用场景",[105,106,107,108,109,110,111,112,22,113,114,115,116,117,118,119,120,121,122,123],"power","statistical power","effect","scipy","regression","choose","design","uv","Bash","collection","analysis","calculation","ANOVA","conversion","analyze","t-test","chi-square","simulation","sample",{"skillNo":125,"name":126,"nameZh":6,"description":127,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":128,"categories":129,"keywords":133,"publishedAt":39},"01M71677EBCM8EPCE9BZCK8505","statistical-analysis","Guided statistical analysis with test selection and reporting. Use when you need help choosing appropriate tests for your data, assumption checking, power analysis, and APA-formatted results. Best for academic research reporting, test selection guidance. For implementing specific models programmatically use statsmodels.","https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fscientific-agent-skills\u002Fskills\u002Fstatistical-analysis\u002FSKILL.md",[130],{"categoryNo":131,"categoryName":132,"taxonomyType":102,"taxonomyTypeDesc":103},"01JBSEEDCAT000000000000008","测试与质量",[115,134,105,135,136,137,22,24,138,139,140,141,142,143,144,145,146,147,148,149],"statistical analysis","academic research","implementing","selection","reporting","data","for","tests","use","with","test","research","results","statsmodels","models","help",{"skillNo":151,"name":152,"nameZh":6,"description":153,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":154,"categories":155,"keywords":165,"publishedAt":39},"01M71673R93XAHFH81DBNVNG7H","scientific-writing","Core skill for the deep research and writing tool. Write scientific manuscripts in full paragraphs (never bullet points). Use two-stage process with (1) section outlines with key points using research-lookup then (2) convert to flowing prose. IMRAD structure, citations (APA\u002FAMA\u002FVancouver), figures\u002Ftables, reporting guidelines (CONSORT\u002FSTROBE\u002FPRISMA), for research papers and journal submissions.","https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fscientific-agent-skills\u002Fskills\u002Fscientific-writing\u002FSKILL.md",[156,159,160],{"categoryNo":157,"categoryName":158,"taxonomyType":102,"taxonomyTypeDesc":103},"01JBSEEDCAT000000000000018","写作与文案",{"categoryNo":100,"categoryName":101,"taxonomyType":102,"taxonomyTypeDesc":103},{"categoryNo":161,"categoryName":162,"taxonomyType":163,"taxonomyTypeDesc":164},"01JBSEEDCAT000000000000036","内容创作者","PROFESSION","适用职业",[166,167,168,169,170,171,172,173,174,175,113,176,177,178,179,180,181,182,138,183],"writing","scientific","scientific writing","Citations","deep","deep research","never","papers","journal","false","api","prisma","structure","openrouter","Outlines","core skill","read","features",{"skillNo":185,"name":186,"nameZh":6,"description":187,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":188,"categories":189,"keywords":194,"publishedAt":39},"01M7166ZPKXBF4EBPHVTJPQBB1","scientific-visualization","Meta-skill for publication-ready figures. Use when creating journal submission figures requiring multi-panel layouts, significance annotations, error bars, colorblind-safe palettes, and specific journal formatting (Nature, Science, Cell). Orchestrates matplotlib\u002Fseaborn\u002Fplotly with publication styles. For quick exploration use seaborn or plotly directly.","https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fscientific-agent-skills\u002Fskills\u002Fscientific-visualization\u002FSKILL.md",[190,193],{"categoryNo":191,"categoryName":192,"taxonomyType":102,"taxonomyTypeDesc":103},"01JBSEEDCAT000000000000013","分析与可视化",{"categoryNo":100,"categoryName":101,"taxonomyType":102,"taxonomyTypeDesc":103},[195,167,196,197,198,199,200,201,202,174,22,203,204,205,206,207,208,209,140,210],"visualization","scientific visualization","error","ready","quick","exploration","meta-skill","Meta","safe","plotly","annotations","formatting","Publication","figures","panel","submission",{"skillNo":212,"name":213,"nameZh":6,"description":214,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":215,"categories":216,"keywords":223,"publishedAt":39},"01M7166VXJWQSSC3RQZP60WBGR","research-code","Debug, profile, review, and restructure Python research and analysis code. Use when analysis scripts crash, produce wrong results, run too slowly, or need a correctness review. Covers: reading tracebacks and diagnosing root causes, hypothesis-driven debugging, reviewing code for common statistical\u002Fnumerical errors, profiling data pipelines, improving reproducibility (seeds, environments, determinism), and structuring analysis scripts for sharing. Distinct from \u002Fexploratory-data-analysis (which understands data) and \u002Fstatistical-analysis (which chooses and runs tests) — this skill focuses on the code itself.","https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fscientific-agent-skills\u002Fskills\u002Fresearch-code\u002FSKILL.md",[217,220],{"categoryNo":218,"categoryName":219,"taxonomyType":102,"taxonomyTypeDesc":103},"01JBSEEDCAT000000000000021","资料检索与研究",{"categoryNo":221,"categoryName":222,"taxonomyType":163,"taxonomyTypeDesc":164},"01JBSEEDCAT000000000000030","开发工程师",[224,225,145,226,227,228,229,112,230,22,113,115,231,232,233,234,235,236,182,237],"code","research code","which","memory","data-analysis","errors","debugging","common","review","data pipelines","run","scripts","line","write",{"skillNo":239,"name":240,"nameZh":6,"description":241,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":242,"licenseFileUrl":6,"sourceHtmlUrl":243,"categories":244,"keywords":251,"publishedAt":39},"01M7166CF5HSR7SVZYZFWTDX55","markdown-mermaid-writing","Comprehensive markdown and Mermaid diagram writing skill. Use when creating any scientific document, report, analysis, or visualization. Establishes text-based diagrams as the default documentation standard with full style guides (markdown + mermaid), 24 diagram type references, and 9 document templates.","Apache-2.0","https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fscientific-agent-skills\u002Fskills\u002Fmarkdown-mermaid-writing\u002FSKILL.md",[245,248,249,250],{"categoryNo":246,"categoryName":247,"taxonomyType":102,"taxonomyTypeDesc":103},"01JBSEEDCAT000000000000002","文档处理",{"categoryNo":157,"categoryName":158,"taxonomyType":102,"taxonomyTypeDesc":103},{"categoryNo":100,"categoryName":101,"taxonomyType":102,"taxonomyTypeDesc":103},{"categoryNo":161,"categoryName":162,"taxonomyType":163,"taxonomyTypeDesc":164},[166,252,253,254,255,256,195,257,258,22,113,115,259,260,261,262,263,264,167,182],"markdown mermaid writing","mermaid","markdown","community","diagrams","project","default","team","guides","text","documentation","references","diagram"]