[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6XGCVWJPVV09G3EFZ0VH6BV":3,"skill-same:01M6XGCVWJPVV09G3EFZ0VH6BV":100},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":28,"publishedAt":49,"linkOnly":50,"markdownContent":51,"markdownBody":52,"frontMatter":53,"refName":70,"commitSha":71,"skillDirectory":72,"forksCount":14,"repositoryHtmlUrl":73,"repositoryHomepageUrl":6,"ownerHtmlUrl":74,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":75,"install":76,"resources":89},"01M6XGCVWJPVV09G3EFZ0VH6BV","nemo-curator",null,"GPU-accelerated data curation for LLM training. Supports text\u002Fimage\u002Fvideo\u002Faudio. Features fuzzy deduplication (16× faster), quality filtering (30+ heuristics), semantic deduplication, PII redaction, NSFW detection. Scales across GPUs with RAPIDS. Use for preparing high-quality training datasets, cleaning web data, or deduplicating large corpora.","fadrienne\u002FMyGitHub","fadrienne","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F197874571?v=4","01M6X64V8VWCRSMFZ4EPS2BWXG","Francesca Adrienne","01M6X64VD5MW4F9PHDWQKS7ZAF",0,"MIT","https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fopenscience\u002Fbackend\u002Fcli\u002Fskills\u002Fml-training\u002Fnemo-curator\u002FSKILL.md",[18,23],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000016","图像与多媒体","SCENARIO","使用场景",{"categoryNo":24,"categoryName":25,"taxonomyType":26,"taxonomyTypeDesc":27},"01JBSEEDCAT000000000000033","数据分析师","PROFESSION","适用职业",[29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48],"NeMo","nemo curator","Video","features","curation","GPU Acceleration","processing","synthetic","across","text","Audio","image","dask","pii","heuristics","deduplication","data","NVIDIA","multimodal","training","2026-11-10 10:12:30",false,"---\nname: nemo-curator\ndescription: GPU-accelerated data curation for LLM training. Supports text\u002Fimage\u002Fvideo\u002Faudio. Features fuzzy deduplication (16× faster), quality filtering (30+ heuristics), semantic deduplication, PII redaction, NSFW detection. Scales across GPUs with RAPIDS. Use for preparing high-quality training datasets, cleaning web data, or deduplicating large corpora.\ncategory: ml-training\nversion: 1.0.0\nauthor: Synthetic Sciences\nlicense: MIT\ntags: [Data Processing, NeMo Curator, Data Curation, GPU Acceleration, Deduplication, Quality Filtering, NVIDIA, RAPIDS, PII Redaction, Multimodal, LLM Training Data]\ndependencies: [nemo-curator, cudf, dask, rapids]\n---\n\n# NeMo Curator - GPU-Accelerated Data Curation\n\nNVIDIA's toolkit for preparing high-quality training data for LLMs.\n\n## When to use NeMo Curator\n\n**Use NeMo Curator when:**\n- Preparing LLM training data from web scrapes (Common Crawl)\n- Need fast deduplication (16× faster than CPU)\n- Curating multi-modal datasets (text, images, video, audio)\n- Filtering low-quality or toxic content\n- Scaling data processing across GPU cluster\n\n**Performance**:\n- **16× faster** fuzzy deduplication (8TB RedPajama v2)\n- **40% lower TCO** vs CPU alternatives\n- **Near-linear scaling** across GPU nodes\n\n**Use alternatives instead**:\n- **datatrove**: CPU-based, open-source data processing\n- **dolma**: Allen AI's data toolkit\n- **Ray Data**: General ML data processing (no curation focus)\n\n## Quick start\n\n### Installation\n\n```bash\n# Text curation (CUDA 12)\nuv pip install \"nemo-curator[text_cuda12]\"\n\n# All modalities\nuv pip install \"nemo-curator[all_cuda12]\"\n\n# CPU-only (slower)\nuv pip install \"nemo-curator[cpu]\"\n```\n\n### Basic text curation pipeline\n\n```python\nfrom nemo_curator import ScoreFilter, Modify\nfrom nemo_curator.datasets import DocumentDataset\nimport pandas as pd\n\n# Load data\ndf = pd.DataFrame({\"text\": [\"Good document\", \"Bad doc\", \"Excellent text\"]})\ndataset = DocumentDataset(df)\n\n# Quality filtering\ndef quality_score(doc):\n    return len(doc[\"text\"].split()) > 5  # Filter short docs\n\nfiltered = ScoreFilter(quality_score)(dataset)\n\n# Deduplication\nfrom nemo_curator.modules import ExactDuplicates\ndeduped = ExactDuplicates()(filtered)\n\n# Save\ndeduped.to_parquet(\"curated_data\u002F\")\n```\n\n## Data curation pipeline\n\n### Stage 1: Quality filtering\n\n```python\nfrom nemo_curator.filters import (\n    WordCountFilter,\n    RepeatedLinesFilter,\n    UrlRatioFilter,\n    NonAlphaNumericFilter\n)\n\n# Apply 30+ heuristic filters\nfrom nemo_curator import ScoreFilter\n\n# Word count filter\ndataset = dataset.filter(WordCountFilter(min_words=50, max_words=100000))\n\n# Remove repetitive content\ndataset = dataset.filter(RepeatedLinesFilter(max_repeated_line_fraction=0.3))\n\n# URL ratio filter\ndataset = dataset.filter(UrlRatioFilter(max_url_ratio=0.2))\n```\n\n### Stage 2: Deduplication\n\n**Exact deduplication**:\n```python\nfrom nemo_curator.modules import ExactDuplicates\n\n# Remove exact duplicates\ndeduped = ExactDuplicates(id_field=\"id\", text_field=\"text\")(dataset)\n```\n\n**Fuzzy deduplication** (16× faster on GPU):\n```python\nfrom nemo_curator.modules import FuzzyDuplicates\n\n# MinHash + LSH deduplication\nfuzzy_dedup = FuzzyDuplicates(\n    id_field=\"id\",\n    text_field=\"text\",\n    num_hashes=260,      # MinHash parameters\n    num_buckets=20,\n    hash_method=\"md5\"\n)\n\ndeduped = fuzzy_dedup(dataset)\n```\n\n**Semantic deduplication**:\n```python\nfrom nemo_curator.modules import SemanticDuplicates\n\n# Embedding-based deduplication\nsemantic_dedup = SemanticDuplicates(\n    id_field=\"id\",\n    text_field=\"text\",\n    embedding_model=\"sentence-transformers\u002Fall-MiniLM-L6-v2\",\n    threshold=0.8  # Cosine similarity threshold\n)\n\ndeduped = semantic_dedup(dataset)\n```\n\n### Stage 3: PII redaction\n\n```python\nfrom nemo_curator.modules import Modify\nfrom nemo_curator.modifiers import PIIRedactor\n\n# Redact personally identifiable information\npii_redactor = PIIRedactor(\n    supported_entities=[\"EMAIL_ADDRESS\", \"PHONE_NUMBER\", \"PERSON\", \"LOCATION\"],\n    anonymize_action=\"replace\"  # or \"redact\"\n)\n\nredacted = Modify(pii_redactor)(dataset)\n```\n\n### Stage 4: Classifier filtering\n\n```python\nfrom nemo_curator.classifiers import QualityClassifier\n\n# Quality classification\nquality_clf = QualityClassifier(\n    model_path=\"nvidia\u002Fquality-classifier-deberta\",\n    batch_size=256,\n    device=\"cuda\"\n)\n\n# Filter low-quality documents\nhigh_quality = dataset.filter(lambda doc: quality_clf(doc[\"text\"]) > 0.5)\n```\n\n## GPU acceleration\n\n### GPU vs CPU performance\n\n| Operation | CPU (16 cores) | GPU (A100) | Speedup |\n|-----------|----------------|------------|---------|\n| Fuzzy dedup (8TB) | 120 hours | 7.5 hours | 16× |\n| Exact dedup (1TB) | 8 hours | 0.5 hours | 16× |\n| Quality filtering | 2 hours | 0.2 hours | 10× |\n\n### Multi-GPU scaling\n\n```python\nfrom nemo_curator import get_client\nimport dask_cuda\n\n# Initialize GPU cluster\nclient = get_client(cluster_type=\"gpu\", n_workers=8)\n\n# Process with 8 GPUs\ndeduped = FuzzyDuplicates(...)(dataset)\n```\n\n## Multi-modal curation\n\n### Image curation\n\n```python\nfrom nemo_curator.image import (\n    AestheticFilter,\n    NSFWFilter,\n    CLIPEmbedder\n)\n\n# Aesthetic scoring\naesthetic_filter = AestheticFilter(threshold=5.0)\nfiltered_images = aesthetic_filter(image_dataset)\n\n# NSFW detection\nnsfw_filter = NSFWFilter(threshold=0.9)\nsafe_images = nsfw_filter(filtered_images)\n\n# Generate CLIP embeddings\nclip_embedder = CLIPEmbedder(model=\"openai\u002Fclip-vit-base-patch32\")\nimage_embeddings = clip_embedder(safe_images)\n```\n\n### Video curation\n\n```python\nfrom nemo_curator.video import (\n    SceneDetector,\n    ClipExtractor,\n    InternVideo2Embedder\n)\n\n# Detect scenes\nscene_detector = SceneDetector(threshold=27.0)\nscenes = scene_detector(video_dataset)\n\n# Extract clips\nclip_extractor = ClipExtractor(min_duration=2.0, max_duration=10.0)\nclips = clip_extractor(scenes)\n\n# Generate embeddings\nvideo_embedder = InternVideo2Embedder()\nvideo_embeddings = video_embedder(clips)\n```\n\n### Audio curation\n\n```python\nfrom nemo_curator.audio import (\n    ASRInference,\n    WERFilter,\n    DurationFilter\n)\n\n# ASR transcription\nasr = ASRInference(model=\"nvidia\u002Fstt_en_fastconformer_hybrid_large_pc\")\ntranscribed = asr(audio_dataset)\n\n# Filter by WER (word error rate)\nwer_filter = WERFilter(max_wer=0.3)\nhigh_quality_audio = wer_filter(transcribed)\n\n# Duration filtering\nduration_filter = DurationFilter(min_duration=1.0, max_duration=30.0)\nfiltered_audio = duration_filter(high_quality_audio)\n```\n\n## Common patterns\n\n### Web scrape curation (Common Crawl)\n\n```python\nfrom nemo_curator import ScoreFilter, Modify\nfrom nemo_curator.filters import *\nfrom nemo_curator.modules import *\nfrom nemo_curator.datasets import DocumentDataset\n\n# Load Common Crawl data\ndataset = DocumentDataset.read_parquet(\"common_crawl\u002F*.parquet\")\n\n# Pipeline\npipeline = [\n    # 1. Quality filtering\n    WordCountFilter(min_words=100, max_words=50000),\n    RepeatedLinesFilter(max_repeated_line_fraction=0.2),\n    SymbolToWordRatioFilter(max_symbol_to_word_ratio=0.3),\n    UrlRatioFilter(max_url_ratio=0.3),\n\n    # 2. Language filtering\n    LanguageIdentificationFilter(target_languages=[\"en\"]),\n\n    # 3. Deduplication\n    ExactDuplicates(id_field=\"id\", text_field=\"text\"),\n    FuzzyDuplicates(id_field=\"id\", text_field=\"text\", num_hashes=260),\n\n    # 4. PII redaction\n    PIIRedactor(),\n\n    # 5. NSFW filtering\n    NSFWClassifier(threshold=0.8)\n]\n\n# Execute\nfor stage in pipeline:\n    dataset = stage(dataset)\n\n# Save\ndataset.to_parquet(\"curated_common_crawl\u002F\")\n```\n\n### Distributed processing\n\n```python\nfrom nemo_curator import get_client\nfrom dask_cuda import LocalCUDACluster\n\n# Multi-GPU cluster\ncluster = LocalCUDACluster(n_workers=8)\nclient = get_client(cluster=cluster)\n\n# Process large dataset\ndataset = DocumentDataset.read_parquet(\"s3:\u002F\u002Flarge_dataset\u002F*.parquet\")\ndeduped = FuzzyDuplicates(...)(dataset)\n\n# Cleanup\nclient.close()\ncluster.close()\n```\n\n## Performance benchmarks\n\n### Fuzzy deduplication (8TB RedPajama v2)\n\n- **CPU (256 cores)**: 120 hours\n- **GPU (8× A100)**: 7.5 hours\n- **Speedup**: 16×\n\n### Exact deduplication (1TB)\n\n- **CPU (64 cores)**: 8 hours\n- **GPU (4× A100)**: 0.5 hours\n- **Speedup**: 16×\n\n### Quality filtering (100GB)\n\n- **CPU (32 cores)**: 2 hours\n- **GPU (2× A100)**: 0.2 hours\n- **Speedup**: 10×\n\n## Cost comparison\n\n**CPU-based curation** (AWS c5.18xlarge × 10):\n- Cost: $3.60\u002Fhour × 10 = $36\u002Fhour\n- Time for 8TB: 120 hours\n- **Total**: $4,320\n\n**GPU-based curation** (AWS p4d.24xlarge × 2):\n- Cost: $32.77\u002Fhour × 2 = $65.54\u002Fhour\n- Time for 8TB: 7.5 hours\n- **Total**: $491.55\n\n**Savings**: 89% reduction ($3,828 saved)\n\n## Supported data formats\n\n- **Input**: Parquet, JSONL, CSV\n- **Output**: Parquet (recommended), JSONL\n- **WebDataset**: TAR archives for multi-modal\n\n## Use cases\n\n**Production deployments**:\n- NVIDIA used NeMo Curator to prepare Nemotron-4 training data\n- Open-source datasets curated: RedPajama v2, The Pile\n\n## References\n\n- **[Filtering Guide](references\u002Ffiltering.md)** - 30+ quality filters, heuristics\n- **[Deduplication Guide](references\u002Fdeduplication.md)** - Exact, fuzzy, semantic methods\n\n## Resources\n\n- **GitHub**: https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FNeMo-Curator ⭐ 500+\n- **Docs**: https:\u002F\u002Fdocs.nvidia.com\u002Fnemo-framework\u002Fuser-guide\u002Flatest\u002Fdatacuration\u002F\n- **Version**: 0.4.0+\n- **License**: Apache 2.0\n\n\n\n","# NeMo Curator - GPU-Accelerated Data Curation\n\nNVIDIA's toolkit for preparing high-quality training data for LLMs.\n\n## When to use NeMo Curator\n\n**Use NeMo Curator when:**\n- Preparing LLM training data from web scrapes (Common Crawl)\n- Need fast deduplication (16× faster than CPU)\n- Curating multi-modal datasets (text, images, video, audio)\n- Filtering low-quality or toxic content\n- Scaling data processing across GPU cluster\n\n**Performance**:\n- **16× faster** fuzzy deduplication (8TB RedPajama v2)\n- **40% lower TCO** vs CPU alternatives\n- **Near-linear scaling** across GPU nodes\n\n**Use alternatives instead**:\n- **datatrove**: CPU-based, open-source data processing\n- **dolma**: Allen AI's data toolkit\n- **Ray Data**: General ML data processing (no curation focus)\n\n## Quick start\n\n### Installation\n\n```bash\n# Text curation (CUDA 12)\nuv pip install \"nemo-curator[text_cuda12]\"\n\n# All modalities\nuv pip install \"nemo-curator[all_cuda12]\"\n\n# CPU-only (slower)\nuv pip install \"nemo-curator[cpu]\"\n```\n\n### Basic text curation pipeline\n\n```python\nfrom nemo_curator import ScoreFilter, Modify\nfrom nemo_curator.datasets import DocumentDataset\nimport pandas as pd\n\n# Load data\ndf = pd.DataFrame({\"text\": [\"Good document\", \"Bad doc\", \"Excellent text\"]})\ndataset = DocumentDataset(df)\n\n# Quality filtering\ndef quality_score(doc):\n    return len(doc[\"text\"].split()) > 5  # Filter short docs\n\nfiltered = ScoreFilter(quality_score)(dataset)\n\n# Deduplication\nfrom nemo_curator.modules import ExactDuplicates\ndeduped = ExactDuplicates()(filtered)\n\n# Save\ndeduped.to_parquet(\"curated_data\u002F\")\n```\n\n## Data curation pipeline\n\n### Stage 1: Quality filtering\n\n```python\nfrom nemo_curator.filters import (\n    WordCountFilter,\n    RepeatedLinesFilter,\n    UrlRatioFilter,\n    NonAlphaNumericFilter\n)\n\n# Apply 30+ heuristic filters\nfrom nemo_curator import ScoreFilter\n\n# Word count filter\ndataset = dataset.filter(WordCountFilter(min_words=50, max_words=100000))\n\n# Remove repetitive content\ndataset = dataset.filter(RepeatedLinesFilter(max_repeated_line_fraction=0.3))\n\n# URL ratio filter\ndataset = dataset.filter(UrlRatioFilter(max_url_ratio=0.2))\n```\n\n### Stage 2: Deduplication\n\n**Exact deduplication**:\n```python\nfrom nemo_curator.modules import ExactDuplicates\n\n# Remove exact duplicates\ndeduped = ExactDuplicates(id_field=\"id\", text_field=\"text\")(dataset)\n```\n\n**Fuzzy deduplication** (16× faster on GPU):\n```python\nfrom nemo_curator.modules import FuzzyDuplicates\n\n# MinHash + LSH deduplication\nfuzzy_dedup = FuzzyDuplicates(\n    id_field=\"id\",\n    text_field=\"text\",\n    num_hashes=260,      # MinHash parameters\n    num_buckets=20,\n    hash_method=\"md5\"\n)\n\ndeduped = fuzzy_dedup(dataset)\n```\n\n**Semantic deduplication**:\n```python\nfrom nemo_curator.modules import SemanticDuplicates\n\n# Embedding-based deduplication\nsemantic_dedup = SemanticDuplicates(\n    id_field=\"id\",\n    text_field=\"text\",\n    embedding_model=\"sentence-transformers\u002Fall-MiniLM-L6-v2\",\n    threshold=0.8  # Cosine similarity threshold\n)\n\ndeduped = semantic_dedup(dataset)\n```\n\n### Stage 3: PII redaction\n\n```python\nfrom nemo_curator.modules import Modify\nfrom nemo_curator.modifiers import PIIRedactor\n\n# Redact personally identifiable information\npii_redactor = PIIRedactor(\n    supported_entities=[\"EMAIL_ADDRESS\", \"PHONE_NUMBER\", \"PERSON\", \"LOCATION\"],\n    anonymize_action=\"replace\"  # or \"redact\"\n)\n\nredacted = Modify(pii_redactor)(dataset)\n```\n\n### Stage 4: Classifier filtering\n\n```python\nfrom nemo_curator.classifiers import QualityClassifier\n\n# Quality classification\nquality_clf = QualityClassifier(\n    model_path=\"nvidia\u002Fquality-classifier-deberta\",\n    batch_size=256,\n    device=\"cuda\"\n)\n\n# Filter low-quality documents\nhigh_quality = dataset.filter(lambda doc: quality_clf(doc[\"text\"]) > 0.5)\n```\n\n## GPU acceleration\n\n### GPU vs CPU performance\n\n| Operation | CPU (16 cores) | GPU (A100) | Speedup |\n|-----------|----------------|------------|---------|\n| Fuzzy dedup (8TB) | 120 hours | 7.5 hours | 16× |\n| Exact dedup (1TB) | 8 hours | 0.5 hours | 16× |\n| Quality filtering | 2 hours | 0.2 hours | 10× |\n\n### Multi-GPU scaling\n\n```python\nfrom nemo_curator import get_client\nimport dask_cuda\n\n# Initialize GPU cluster\nclient = get_client(cluster_type=\"gpu\", n_workers=8)\n\n# Process with 8 GPUs\ndeduped = FuzzyDuplicates(...)(dataset)\n```\n\n## Multi-modal curation\n\n### Image curation\n\n```python\nfrom nemo_curator.image import (\n    AestheticFilter,\n    NSFWFilter,\n    CLIPEmbedder\n)\n\n# Aesthetic scoring\naesthetic_filter = AestheticFilter(threshold=5.0)\nfiltered_images = aesthetic_filter(image_dataset)\n\n# NSFW detection\nnsfw_filter = NSFWFilter(threshold=0.9)\nsafe_images = nsfw_filter(filtered_images)\n\n# Generate CLIP embeddings\nclip_embedder = CLIPEmbedder(model=\"openai\u002Fclip-vit-base-patch32\")\nimage_embeddings = clip_embedder(safe_images)\n```\n\n### Video curation\n\n```python\nfrom nemo_curator.video import (\n    SceneDetector,\n    ClipExtractor,\n    InternVideo2Embedder\n)\n\n# Detect scenes\nscene_detector = SceneDetector(threshold=27.0)\nscenes = scene_detector(video_dataset)\n\n# Extract clips\nclip_extractor = ClipExtractor(min_duration=2.0, max_duration=10.0)\nclips = clip_extractor(scenes)\n\n# Generate embeddings\nvideo_embedder = InternVideo2Embedder()\nvideo_embeddings = video_embedder(clips)\n```\n\n### Audio curation\n\n```python\nfrom nemo_curator.audio import (\n    ASRInference,\n    WERFilter,\n    DurationFilter\n)\n\n# ASR transcription\nasr = ASRInference(model=\"nvidia\u002Fstt_en_fastconformer_hybrid_large_pc\")\ntranscribed = asr(audio_dataset)\n\n# Filter by WER (word error rate)\nwer_filter = WERFilter(max_wer=0.3)\nhigh_quality_audio = wer_filter(transcribed)\n\n# Duration filtering\nduration_filter = DurationFilter(min_duration=1.0, max_duration=30.0)\nfiltered_audio = duration_filter(high_quality_audio)\n```\n\n## Common patterns\n\n### Web scrape curation (Common Crawl)\n\n```python\nfrom nemo_curator import ScoreFilter, Modify\nfrom nemo_curator.filters import *\nfrom nemo_curator.modules import *\nfrom nemo_curator.datasets import DocumentDataset\n\n# Load Common Crawl data\ndataset = DocumentDataset.read_parquet(\"common_crawl\u002F*.parquet\")\n\n# Pipeline\npipeline = [\n    # 1. Quality filtering\n    WordCountFilter(min_words=100, max_words=50000),\n    RepeatedLinesFilter(max_repeated_line_fraction=0.2),\n    SymbolToWordRatioFilter(max_symbol_to_word_ratio=0.3),\n    UrlRatioFilter(max_url_ratio=0.3),\n\n    # 2. Language filtering\n    LanguageIdentificationFilter(target_languages=[\"en\"]),\n\n    # 3. Deduplication\n    ExactDuplicates(id_field=\"id\", text_field=\"text\"),\n    FuzzyDuplicates(id_field=\"id\", text_field=\"text\", num_hashes=260),\n\n    # 4. PII redaction\n    PIIRedactor(),\n\n    # 5. NSFW filtering\n    NSFWClassifier(threshold=0.8)\n]\n\n# Execute\nfor stage in pipeline:\n    dataset = stage(dataset)\n\n# Save\ndataset.to_parquet(\"curated_common_crawl\u002F\")\n```\n\n### Distributed processing\n\n```python\nfrom nemo_curator import get_client\nfrom dask_cuda import LocalCUDACluster\n\n# Multi-GPU cluster\ncluster = LocalCUDACluster(n_workers=8)\nclient = get_client(cluster=cluster)\n\n# Process large dataset\ndataset = DocumentDataset.read_parquet(\"s3:\u002F\u002Flarge_dataset\u002F*.parquet\")\ndeduped = FuzzyDuplicates(...)(dataset)\n\n# Cleanup\nclient.close()\ncluster.close()\n```\n\n## Performance benchmarks\n\n### Fuzzy deduplication (8TB RedPajama v2)\n\n- **CPU (256 cores)**: 120 hours\n- **GPU (8× A100)**: 7.5 hours\n- **Speedup**: 16×\n\n### Exact deduplication (1TB)\n\n- **CPU (64 cores)**: 8 hours\n- **GPU (4× A100)**: 0.5 hours\n- **Speedup**: 16×\n\n### Quality filtering (100GB)\n\n- **CPU (32 cores)**: 2 hours\n- **GPU (2× A100)**: 0.2 hours\n- **Speedup**: 10×\n\n## Cost comparison\n\n**CPU-based curation** (AWS c5.18xlarge × 10):\n- Cost: $3.60\u002Fhour × 10 = $36\u002Fhour\n- Time for 8TB: 120 hours\n- **Total**: $4,320\n\n**GPU-based curation** (AWS p4d.24xlarge × 2):\n- Cost: $32.77\u002Fhour × 2 = $65.54\u002Fhour\n- Time for 8TB: 7.5 hours\n- **Total**: $491.55\n\n**Savings**: 89% reduction ($3,828 saved)\n\n## Supported data formats\n\n- **Input**: Parquet, JSONL, CSV\n- **Output**: Parquet (recommended), JSONL\n- **WebDataset**: TAR archives for multi-modal\n\n## Use cases\n\n**Production deployments**:\n- NVIDIA used NeMo Curator to prepare Nemotron-4 training data\n- Open-source datasets curated: RedPajama v2, The Pile\n\n## References\n\n- **[Filtering Guide](references\u002Ffiltering.md)** - 30+ quality filters, heuristics\n- **[Deduplication Guide](references\u002Fdeduplication.md)** - Exact, fuzzy, semantic methods\n\n## Resources\n\n- **GitHub**: https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FNeMo-Curator ⭐ 500+\n- **Docs**: https:\u002F\u002Fdocs.nvidia.com\u002Fnemo-framework\u002Fuser-guide\u002Flatest\u002Fdatacuration\u002F\n- **Version**: 0.4.0+\n- **License**: Apache 2.0\n\n\n\n",{"name":5,"tags":54,"author":64,"license":15,"version":65,"category":66,"description":7,"dependencies":67},[55,56,57,34,58,59,46,60,61,62,63],"Data Processing","NeMo Curator","Data Curation","Deduplication","Quality Filtering","RAPIDS","PII Redaction","Multimodal","LLM Training Data","Synthetic Sciences","1.0.0","ml-training",[5,68,41,69],"cudf","rapids","main","4aa39efd1cdab0b4657ecee16c92a92011a5b10f","openscience\u002Fbackend\u002Fcli\u002Fskills\u002Fml-training\u002Fnemo-curator","https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub","https:\u002F\u002Fgithub.com\u002Ffadrienne","2026-10-28 00:43:09",{"source":8,"skillName":5,"commands":77,"prompt":87,"zipAvailable":88},[78,81,84],{"tool":79,"command":80},"npx","npx skills add fadrienne\u002FMyGitHub --skill nemo-curator",{"tool":82,"command":83},"bunx","bunx skills add fadrienne\u002FMyGitHub --skill nemo-curator",{"tool":85,"command":86},"pnpm","pnpm dlx skills add fadrienne\u002FMyGitHub --skill nemo-curator","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fopenscience\u002Fbackend\u002Fcli\u002Fskills\u002Fml-training\u002Fnemo-curator\u002FSKILL.md",true,[90,96],{"path":91,"resourceType":92,"resourceTypeDesc":93,"sizeBytes":94,"sourceHtmlUrl":95},"references\u002Fdeduplication.md","REFERENCE","参考文档",2142,"https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fopenscience\u002Fbackend\u002Fcli\u002Fskills\u002Fml-training\u002Fnemo-curator\u002Freferences\u002Fdeduplication.md",{"path":97,"resourceType":92,"resourceTypeDesc":93,"sizeBytes":98,"sourceHtmlUrl":99},"references\u002Ffiltering.md",2351,"https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fopenscience\u002Fbackend\u002Fcli\u002Fskills\u002Fml-training\u002Fnemo-curator\u002Freferences\u002Ffiltering.md",[101,131,157,188,215,242],{"skillNo":102,"name":103,"nameZh":6,"description":104,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":105,"categories":106,"keywords":110,"publishedAt":49},"01M7167B28PK20HP7R7ZED2DY0","statistical-power","Sample size calculation, power analysis, and effect size conversion for scientific studies. Use BEFORE data collection (a priori) to determine required N, or after (post-hoc) to compute achieved power. Covers parametric tests (t-test, ANOVA, chi-square, proportions, correlation, regression), simulation-based power for complex\u002Fclustered\u002Fnested designs, and power curves. Use this skill when someone asks 'how many samples do I need', 'what is my power', 'what effect size can I detect', or any variant. Pairs with \u002Fexperimental-design (choose design first, then compute power) and \u002Fstatistical-analysis (analyze data after collection).","https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fscientific-agent-skills\u002Fskills\u002Fstatistical-power\u002FSKILL.md",[107],{"categoryNo":108,"categoryName":109,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000023","Skill与智能体开发",[111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130],"power","statistical power","effect","scipy","regression","choose","design","uv","when","Bash","collection","analysis","calculation","ANOVA","conversion","analyze","t-test","chi-square","simulation","sample",{"skillNo":132,"name":133,"nameZh":6,"description":134,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":135,"categories":136,"keywords":140,"publishedAt":49},"01M71677EBCM8EPCE9BZCK8505","statistical-analysis","Guided statistical analysis with test selection and reporting. Use when you need help choosing appropriate tests for your data, assumption checking, power analysis, and APA-formatted results. Best for academic research reporting, test selection guidance. For implementing specific models programmatically use statsmodels.","https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fscientific-agent-skills\u002Fskills\u002Fstatistical-analysis\u002FSKILL.md",[137],{"categoryNo":138,"categoryName":139,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000008","测试与质量",[122,141,111,142,143,144,119,145,146,45,147,148,149,150,151,152,153,154,155,156],"statistical analysis","academic research","implementing","selection","you","reporting","for","tests","use","with","test","research","results","statsmodels","models","help",{"skillNo":158,"name":159,"nameZh":6,"description":160,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":161,"categories":162,"keywords":170,"publishedAt":49},"01M71673R93XAHFH81DBNVNG7H","scientific-writing","Core skill for the deep research and writing tool. Write scientific manuscripts in full paragraphs (never bullet points). Use two-stage process with (1) section outlines with key points using research-lookup then (2) convert to flowing prose. IMRAD structure, citations (APA\u002FAMA\u002FVancouver), figures\u002Ftables, reporting guidelines (CONSORT\u002FSTROBE\u002FPRISMA), for research papers and journal submissions.","https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fscientific-agent-skills\u002Fskills\u002Fscientific-writing\u002FSKILL.md",[163,166,167],{"categoryNo":164,"categoryName":165,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000018","写作与文案",{"categoryNo":108,"categoryName":109,"taxonomyType":21,"taxonomyTypeDesc":22},{"categoryNo":168,"categoryName":169,"taxonomyType":26,"taxonomyTypeDesc":27},"01JBSEEDCAT000000000000036","内容创作者",[171,172,173,174,175,176,177,178,179,180,120,181,182,183,184,185,186,187,146,32],"writing","scientific","scientific writing","Citations","deep","deep research","never","papers","journal","false","api","prisma","structure","openrouter","Outlines","core skill","read",{"skillNo":189,"name":190,"nameZh":6,"description":191,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":192,"categories":193,"keywords":198,"publishedAt":49},"01M7166ZPKXBF4EBPHVTJPQBB1","scientific-visualization","Meta-skill for publication-ready figures. Use when creating journal submission figures requiring multi-panel layouts, significance annotations, error bars, colorblind-safe palettes, and specific journal formatting (Nature, Science, Cell). Orchestrates matplotlib\u002Fseaborn\u002Fplotly with publication styles. For quick exploration use seaborn or plotly directly.","https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fscientific-agent-skills\u002Fskills\u002Fscientific-visualization\u002FSKILL.md",[194,197],{"categoryNo":195,"categoryName":196,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000013","分析与可视化",{"categoryNo":108,"categoryName":109,"taxonomyType":21,"taxonomyTypeDesc":22},[199,172,200,201,202,203,204,205,206,179,119,207,208,209,210,211,212,213,147,214],"visualization","scientific visualization","error","ready","quick","exploration","meta-skill","Meta","safe","plotly","annotations","formatting","Publication","figures","panel","submission",{"skillNo":216,"name":217,"nameZh":6,"description":218,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":219,"categories":220,"keywords":227,"publishedAt":49},"01M7166VXJWQSSC3RQZP60WBGR","research-code","Debug, profile, review, and restructure Python research and analysis code. Use when analysis scripts crash, produce wrong results, run too slowly, or need a correctness review. Covers: reading tracebacks and diagnosing root causes, hypothesis-driven debugging, reviewing code for common statistical\u002Fnumerical errors, profiling data pipelines, improving reproducibility (seeds, environments, determinism), and structuring analysis scripts for sharing. Distinct from \u002Fexploratory-data-analysis (which understands data) and \u002Fstatistical-analysis (which chooses and runs tests) — this skill focuses on the code itself.","https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fscientific-agent-skills\u002Fskills\u002Fresearch-code\u002FSKILL.md",[221,224],{"categoryNo":222,"categoryName":223,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000021","资料检索与研究",{"categoryNo":225,"categoryName":226,"taxonomyType":26,"taxonomyTypeDesc":27},"01JBSEEDCAT000000000000030","开发工程师",[228,229,152,230,231,232,233,118,234,119,120,122,235,236,237,238,239,240,187,241],"code","research code","which","memory","data-analysis","errors","debugging","common","review","data pipelines","run","scripts","line","write",{"skillNo":243,"name":244,"nameZh":6,"description":245,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":246,"licenseFileUrl":6,"sourceHtmlUrl":247,"categories":248,"keywords":255,"publishedAt":49},"01M7166CF5HSR7SVZYZFWTDX55","markdown-mermaid-writing","Comprehensive markdown and Mermaid diagram writing skill. Use when creating any scientific document, report, analysis, or visualization. Establishes text-based diagrams as the default documentation standard with full style guides (markdown + mermaid), 24 diagram type references, and 9 document templates.","Apache-2.0","https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fscientific-agent-skills\u002Fskills\u002Fmarkdown-mermaid-writing\u002FSKILL.md",[249,252,253,254],{"categoryNo":250,"categoryName":251,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000002","文档处理",{"categoryNo":164,"categoryName":165,"taxonomyType":21,"taxonomyTypeDesc":22},{"categoryNo":108,"categoryName":109,"taxonomyType":21,"taxonomyTypeDesc":22},{"categoryNo":168,"categoryName":169,"taxonomyType":26,"taxonomyTypeDesc":27},[171,256,257,258,259,260,199,261,262,119,120,122,263,264,38,265,266,267,172,187],"markdown mermaid writing","mermaid","markdown","community","diagrams","project","default","team","guides","documentation","references","diagram"]