[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6BXHT5YPZDFEMBJD7HRW5TB":3,"skill-same:01M6BXHT5YPZDFEMBJD7HRW5TB":125},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":22,"publishedAt":32,"linkOnly":33,"markdownContent":34,"markdownBody":35,"frontMatter":36,"refName":37,"commitSha":38,"skillDirectory":39,"forksCount":40,"repositoryHtmlUrl":41,"repositoryHomepageUrl":6,"ownerHtmlUrl":42,"ownerBlogUrl":43,"ownerDescription":6,"updatedAt":44,"install":45,"resources":58},"01M6BXHT5YPZDFEMBJD7HRW5TB","getting-started",null,"Set up NeMo Curator for data curation (text, image, video, audio)","NVIDIA-NeMo\u002FCurator","NVIDIA-NeMo","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F213689629?v=4","01M6BXH6HS4CXMTZ8GCBQTKPAP","01M6BXH6PD7DSNJ54NXBQ577TR",1857,"Apache-2.0","https:\u002F\u002Fgithub.com\u002FNVIDIA-NeMo\u002FCurator\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fgetting-started\u002FSKILL.md",[17],{"categoryNo":18,"categoryName":19,"taxonomyType":20,"taxonomyTypeDesc":21},"01JBSEEDCAT000000000000016","图像与多媒体","SCENARIO","使用场景",[23,24,25,26,27,28,29,30,31],"getting started","image","NeMo","data","SET","Video","Audio","text","data curation","2026-11-10 10:12:30",false,"---\nname: getting-started\ndescription: Set up NeMo Curator for data curation (text, image, video, audio)\n---\n\n# Getting Started with NeMo Curator\n\nGuide users through setting up NeMo Curator for their data curation needs. This skill prioritizes using `uv` for fast, reliable dependency management.\n\n## Step 1: Ask Questions\n\nUse `AskUserQuestion` to ask these questions in a single call:\n\n### Q1: Data Modality (multiSelect: true)\n**Question:** \"What type of data will you be working with?\"\n- **Text** - LLM training data curation (filtering, deduplication, PII redaction)\n- **Images** - Vision model training (embeddings, NSFW filtering, deduplication)\n- **Video** - Video processing (splitting, captioning, embeddings)\n- **Audio** - Speech data (ASR transcription, quality filtering)\n\n### Q2: Compute Environment\n**Question:** \"What's your compute environment?\"\n- **Local (no GPU)** - CPU-only, limited to smaller datasets\n- **Local (with NVIDIA GPU)** - GPU-accelerated processing\n- **Docker container** - Containerized deployment (recommended for video)\n- **Cloud\u002FHPC cluster** - Distributed processing with Ray\n\n### Q3: Synthetic Data\n**Question:** \"Will you be generating synthetic data with LLMs?\"\n- **Yes** - Requires NVIDIA API key\n- **No \u002F Not sure**\n\n### Q4: Gated Models\n**Question:** \"Will you use gated HuggingFace models (e.g., Llama tokenizers)?\"\n- **Yes** - Requires HuggingFace token\n- **No \u002F Not sure**\n\n## Step 2: Platform Compatibility Check\n\n**IMPORTANT: Check platform compatibility before proceeding.**\n\n### Platform Support Matrix\n| Platform | Text | Image | Video | Audio | GPU Acceleration |\n|----------|------|-------|-------|-------|------------------|\n| **Linux x86_64** | ✅ | ✅ | ✅ | ✅ | ✅ Full support |\n| **Linux ARM64** | ✅ | ✅ | ❌ | ✅ | ⚠️ Limited |\n| **macOS (Intel)** | ✅ | ✅ | ❌ | ✅ | ❌ No CUDA |\n| **macOS (Apple Silicon)** | ✅ | ⚠️ | ❌ | ✅ | ❌ No CUDA |\n| **Windows** | ❌ | ❌ | ❌ | ❌ | ❌ Not supported |\n\nIf user selected **Video** on macOS or Windows, warn them:\n```\n⚠️ Video curation requires Linux x86_64 with NVIDIA GPU.\nConsider using Docker or a cloud environment instead.\n```\n\nCheck the platform:\n```bash\nuname -s && uname -m\n```\n\n## Step 3: Environment Checks\n\nRun these checks in parallel:\n\n### 3.1 Python Version (REQUIRED: 3.10, 3.11, or 3.12)\n```bash\npython3 --version\n```\n**Python 3.13+ is NOT supported** due to dependency constraints.\n\nIf wrong version, advise:\n```\nNeMo Curator requires Python 3.10, 3.11, or 3.12.\nInstall with pyenv or your system package manager.\n```\n\n### 3.2 Check for existing virtual environment\n```bash\necho $VIRTUAL_ENV\n```\n\n### 3.3 Check if uv is installed\n```bash\nwhich uv && uv --version\n```\n\n### 3.4 GPU Check (if GPU selected)\n```bash\nnvidia-smi --query-gpu=name,memory.total --format=csv,noheader\n```\n\n### GPU VRAM Requirements\n| Modality | Minimum VRAM | Recommended |\n|----------|--------------|-------------|\n| Text (GPU) | 16 GB | 24+ GB |\n| Image | 24 GB | 48 GB |\n| Video | 21 GB (optimized) | 38 GB (full) |\n| Audio | 16 GB | 24+ GB |\n\nWarn if GPU VRAM is insufficient for selected modality.\n\n## Step 4: Install uv (PRIORITY)\n\n**uv is the recommended package manager for NeMo Curator.** It provides faster dependency resolution and better handling of complex extras.\n\nIf uv is not installed, ask permission and install:\n```bash\ncurl -LsSf https:\u002F\u002Fastral.sh\u002Fuv\u002Finstall.sh | sh\n```\n\nThen source it:\n```bash\nsource $HOME\u002F.local\u002Fbin\u002Fenv\n```\n\nVerify installation:\n```bash\nuv --version\n```\n\n## Step 5: Create Virtual Environment\n\nIf not in a virtual environment, create one with uv:\n```bash\nuv venv --python 3.11\nsource .venv\u002Fbin\u002Factivate\n```\n\nOr specify exact Python version if needed:\n```bash\nuv venv --python 3.10\n```\n\n## Step 6: Video Prerequisites (if Video modality selected)\n\n**Video curation has additional requirements:**\n\n### 6.1 FFmpeg with H.264 Encoders\nVideo processing requires FFmpeg compiled with specific encoders. Check if already installed:\n```bash\nffmpeg -encoders 2>\u002Fdev\u002Fnull | grep -E \"h264|264\"\n```\n\nShould show at least one of: `h264_nvenc` (GPU), `libx264`, or `libopenh264`.\n\nIf not available, on Ubuntu\u002FDebian:\n```bash\ncurl -fsSL https:\u002F\u002Fraw.githubusercontent.com\u002FNVIDIA-NeMo\u002FCurator\u002Fmain\u002Fdocker\u002Fcommon\u002Finstall_ffmpeg.sh -o install_ffmpeg.sh\nchmod +x install_ffmpeg.sh\nsudo bash install_ffmpeg.sh\n```\n\n**Note:** This installs ~30 system packages and builds FFmpeg from source. Takes 10-20 minutes.\n\n**Alternative:** Use the Docker container which has FFmpeg pre-built.\n\n### 6.2 System packages for FFmpeg build\nIf building FFmpeg, these are required:\n```bash\nsudo apt-get install -y build-essential autoconf automake cmake pkg-config libtool nasm yasm\n```\n\n## Step 7: Install NeMo Curator\n\n### Build the install command based on selections:\n\n**Single modality (CPU):**\n```bash\nuv pip install \"nemo-curator[text_cpu]\"\nuv pip install \"nemo-curator[image_cpu]\"\nuv pip install \"nemo-curator[video_cpu]\"\nuv pip install \"nemo-curator[audio_cpu]\"\n```\n\n**Single modality (GPU\u002FCUDA 12):**\n```bash\nuv pip install \"nemo-curator[text_cuda12]\"\nuv pip install \"nemo-curator[image_cuda12]\"\nuv pip install \"nemo-curator[video_cuda12]\"\nuv pip install \"nemo-curator[audio_cuda12]\"\n```\n\n**Multiple modalities - combine extras:**\n```bash\nuv pip install \"nemo-curator[text_cuda12,image_cuda12]\"\n```\n\n### Version Constraints to Know\n- PyTorch is limited to ≤2.9.1 for video modality\n- scikit-learn must be \u003C1.8.0 (handled automatically)\n- These are managed by uv, but be aware if you have existing installations\n\n### For Docker users:\nRecommend using the pre-built NGC container (includes FFmpeg and all dependencies):\n```bash\ndocker pull nvcr.io\u002Fnvidia\u002Fnemo-curator:25.09\ndocker run --gpus all -it --rm nvcr.io\u002Fnvidia\u002Fnemo-curator:25.09\n```\nThis container includes CUDA 12.8.1, Python 3.12, Ubuntu 24.04, and all modalities pre-installed.\n\n## Step 8: API Key Guidance\n\nBased on answers to Q3 and Q4, provide relevant guidance:\n\n### If synthetic data generation (Q3 = Yes):\n```\nTo use synthetic data generation, you'll need an NVIDIA API key.\n\n1. Get your key at: https:\u002F\u002Fbuild.nvidia.com\u002Fsettings\u002Fapi-keys\n2. Press CTRL\u002FCMD+Z, then type 'fg' to return to your shell\n3. Export the key:\n   export NVIDIA_API_KEY=\"your-key-here\"\n\nOr add to a .env file in your project.\n```\n\n### If gated HuggingFace models (Q4 = Yes):\n```\nTo use gated models like Llama tokenizers, you'll need a HuggingFace token.\n\n1. Get your token at: https:\u002F\u002Fhuggingface.co\u002Fsettings\u002Ftokens\n2. Accept the model license on HuggingFace (e.g., for Llama models)\n3. Press CTRL\u002FCMD+Z, then type 'fg' to return to your shell\n4. Export the token:\n   export HF_TOKEN=\"your-token-here\"\n```\n\n## Step 9: Verify Installation\n\nRun verification with user permission. Execute the verifications that apply based on the user's selections from Step 1.\n\n**Note:** All verification commands below use paths relative to the repository root. Run them from the Curator repo root, or prefix the script paths with `$(git rev-parse --show-toplevel)\u002F`.\n\n### 9.1 Basic Import Check (ALWAYS run)\n```bash\npython3 -c \"import nemo_curator; print(f'NeMo Curator {nemo_curator.__version__} installed')\"\n```\n\n### 9.2 Ray Cluster Verification (ALWAYS run)\nVerify Ray can initialize and stop correctly:\n```bash\npython3 .claude\u002Fskills\u002Fgetting-started\u002Fscripts\u002Fverify_ray.py\n```\n\n### 9.3 Modality-Specific Import Checks (run based on Q1 selection)\n\n| Modality | Command |\n|----------|---------|\n| Text | `python3 .claude\u002Fskills\u002Fgetting-started\u002Fscripts\u002Fverify_text.py` |\n| Image | `python3 .claude\u002Fskills\u002Fgetting-started\u002Fscripts\u002Fverify_image.py` |\n| Video | `python3 .claude\u002Fskills\u002Fgetting-started\u002Fscripts\u002Fverify_video.py` |\n| Audio | `python3 .claude\u002Fskills\u002Fgetting-started\u002Fscripts\u002Fverify_audio.py` |\n\n### 9.4 GPU Verification (if GPU selected in Q2)\n```bash\npython3 -c \"import torch; print(f'PyTorch {torch.__version__}'); print(f'CUDA available: {torch.cuda.is_available()}'); print(f'GPU: {torch.cuda.get_device_name(0)}' if torch.cuda.is_available() else '')\"\n```\n\n### 9.5 Video Encoder Verification (if Video selected in Q1)\n```bash\nffmpeg -encoders 2>\u002Fdev\u002Fnull | grep -E \"h264_nvenc|libx264|libopenh264\" && echo \"✓ H.264 encoder available\"\n```\n\n### 9.6 Pipeline Execution Verification (run based on Q2 selection)\n\n| Environment | Command |\n|-------------|---------|\n| GPU users | `python tutorials\u002Fquickstart.py` |\n| CPU-only users | `python3 .claude\u002Fskills\u002Fgetting-started\u002Fscripts\u002Fverify_pipeline_cpu.py` |\n\n## Step 10: Data Formats & Storage\n\n**Supported input\u002Foutput formats:**\n| Modality | Formats |\n|----------|---------|\n| Text | JSONL (primary), Parquet |\n| Image | WebDataset tar archives |\n| Video | Local paths, S3, HTTP(S) URLs |\n| Audio | Local files with JSONL manifests |\n\n**Cloud storage (S3):** Configure via `storage_options` dict passed to readers:\n```python\nstorage_options = {\"key\": \"AWS_KEY\", \"secret\": \"AWS_SECRET\"}\n```\n\n## Step 11: Ray & Network (Multi-Node\u002FCluster)\n\nIf running on a cluster or multi-node setup, be aware of Ray ports:\n\n| Port | Service | Purpose |\n|------|---------|---------|\n| 6379 | Ray GCS | Cluster communication |\n| 8265 | Ray Dashboard | Monitoring UI (http:\u002F\u002Flocalhost:8265) |\n| 10001 | Ray Client | Python client connection |\n| 10002-19999 | Workers | Worker communication range |\n\n**Firewall rules:** Ensure these ports are open between head and worker nodes.\n\n**Debugging tip:** Access the Ray dashboard at `http:\u002F\u002Flocalhost:8265` to monitor jobs, view logs, and check resource usage.\n\n**Environment variable:** Set `CURATOR_IGNORE_RAY_HEAD_NODE=1` to override Ray head node detection if needed.\n\n**Execution backends:** NeMo Curator uses `XennaExecutor` by default (recommended). Alternative experimental backends exist (`RayActorPoolExecutor`, `RayDataExecutor`) but stick with the default for getting started.\n\n## Step 12: First-Run Notes\n\n**Important:** Some models download on first use:\n- **Text classifiers** (quality, domain, safety): 100MB - 2GB each\n- **Image CLIP models**: ~1.5GB\n- **Video embedding models**: Cosmos-Embed1 (varies by resolution)\n- **Audio ASR models**: 100MB - 1GB depending on language\n\nThese download to `~\u002F.cache\u002Fhuggingface\u002F` by default. Ensure you have disk space and network access on first run.\n\n## Step 13: Recommend Tutorial\n\nBased on modality selection, recommend the most relevant tutorial from https:\u002F\u002Fgithub.com\u002FNVIDIA-NeMo\u002FCurator\u002Ftree\u002Fmain\u002Ftutorials\u002F:\n\n| Selection | Recommended Tutorial | Description |\n|-----------|---------------------|-------------|\n| **Text (beginner)** | `tutorials\u002Ftext\u002Ftinystories\u002F` | Small dataset, end-to-end pipeline (~22k samples) |\n| **Text (large-scale)** | `tutorials\u002Ftext\u002Fllama-nemotron-data-curation\u002F` | 30M sample pipeline with filtering |\n| **Text (PII)** | `tutorials\u002Ftext\u002Fgliner-pii-redaction\u002F` | Privacy\u002FPII redaction with GLiNER |\n| **Image** | `tutorials\u002Fimage\u002Fgetting-started\u002Fimage_curation_example.py` | CLIP embeddings, aesthetic scoring, NSFW filtering |\n| **Video** | `tutorials\u002Fvideo\u002Fgetting-started\u002Fvideo_split_clip_example.py` | Splitting, scene detection, captioning |\n| **Audio** | `tutorials\u002Faudio\u002Ffleurs\u002F` | Multilingual ASR with FLEURS dataset |\n| **Synthetic** | `tutorials\u002Fsynthetic\u002Fsynthetic_data_generation_example.py` | LLM-based Q&A generation |\n\n**Always also recommend:** `tutorials\u002Fquickstart.py` for understanding the core pipeline architecture (Tasks, Stages, Pipelines).\n\n**Pipeline config YAMLs:** Pre-built configs exist in `nemo_curator\u002Fconfig\u002Ftext\u002F` for common workflows:\n- `exact_deduplication_pipeline.yaml`\n- `semantic_deduplication_pipeline.yaml`\n- `fuzzy_deduplication_pipeline.yaml`\n- `heuristic_filter_english_pipeline.yaml`\n- `heuristic_filter_non_english_pipeline.yaml`\n- `fasttext_filter_pipeline.yaml`\n- `code_filter_pipeline.yaml`\n\n## Step 14: Summary Output\n\nAt the end, provide a summary:\n```\nNeMo Curator Setup Complete!\n\nPlatform: {Linux\u002FmacOS} {x86_64\u002Farm64}\nPython: {version}\nPackage manager: uv {version}\nInstalled: nemo-curator[{extras}]\nModalities: {selected modalities}\nCompute: {CPU\u002FGPU with VRAM}\nAPI Keys needed: {list or \"None\"}\n\nFirst-run note: Models will download on first use (~1-2GB per modality)\n\nNext steps:\n1. {If API keys needed: Set your API keys (CTRL\u002FCMD+Z -> fg)}\n2. Try the quickstart: python tutorials\u002Fquickstart.py\n3. Then try: {recommended tutorial path}\n4. Docs: https:\u002F\u002Fdocs.nvidia.com\u002Fnemo-framework\u002Fuser-guide\u002Flatest\u002Fnemocurator\u002F\n```\n\n## Troubleshooting\n\nIf installation fails, check:\n\n1. **\"No matching distribution\"** - Wrong Python version (need 3.10-3.12)\n2. **CUDA errors** - Ensure CUDA 12.x drivers installed (`nvidia-smi`)\n3. **scikit-learn conflicts** - NeMo Curator requires \u003C1.8.0\n4. **FFmpeg encoder missing** - Run the FFmpeg install script\n5. **Out of memory** - Check GPU VRAM requirements above\n6. **Ray port conflicts** - Check if ports 6379, 8265, 10001 are in use\n7. **Job interrupted** - Resumable: compare input\u002Foutput file counts to find remaining work\n\n**Debugging:**\n- Ray dashboard: `http:\u002F\u002Flocalhost:8265`\n- Logs use `loguru` - check stdout\u002Fstderr for detailed traces\n","# Getting Started with NeMo Curator\n\nGuide users through setting up NeMo Curator for their data curation needs. This skill prioritizes using `uv` for fast, reliable dependency management.\n\n## Step 1: Ask Questions\n\nUse `AskUserQuestion` to ask these questions in a single call:\n\n### Q1: Data Modality (multiSelect: true)\n**Question:** \"What type of data will you be working with?\"\n- **Text** - LLM training data curation (filtering, deduplication, PII redaction)\n- **Images** - Vision model training (embeddings, NSFW filtering, deduplication)\n- **Video** - Video processing (splitting, captioning, embeddings)\n- **Audio** - Speech data (ASR transcription, quality filtering)\n\n### Q2: Compute Environment\n**Question:** \"What's your compute environment?\"\n- **Local (no GPU)** - CPU-only, limited to smaller datasets\n- **Local (with NVIDIA GPU)** - GPU-accelerated processing\n- **Docker container** - Containerized deployment (recommended for video)\n- **Cloud\u002FHPC cluster** - Distributed processing with Ray\n\n### Q3: Synthetic Data\n**Question:** \"Will you be generating synthetic data with LLMs?\"\n- **Yes** - Requires NVIDIA API key\n- **No \u002F Not sure**\n\n### Q4: Gated Models\n**Question:** \"Will you use gated HuggingFace models (e.g., Llama tokenizers)?\"\n- **Yes** - Requires HuggingFace token\n- **No \u002F Not sure**\n\n## Step 2: Platform Compatibility Check\n\n**IMPORTANT: Check platform compatibility before proceeding.**\n\n### Platform Support Matrix\n| Platform | Text | Image | Video | Audio | GPU Acceleration |\n|----------|------|-------|-------|-------|------------------|\n| **Linux x86_64** | ✅ | ✅ | ✅ | ✅ | ✅ Full support |\n| **Linux ARM64** | ✅ | ✅ | ❌ | ✅ | ⚠️ Limited |\n| **macOS (Intel)** | ✅ | ✅ | ❌ | ✅ | ❌ No CUDA |\n| **macOS (Apple Silicon)** | ✅ | ⚠️ | ❌ | ✅ | ❌ No CUDA |\n| **Windows** | ❌ | ❌ | ❌ | ❌ | ❌ Not supported |\n\nIf user selected **Video** on macOS or Windows, warn them:\n```\n⚠️ Video curation requires Linux x86_64 with NVIDIA GPU.\nConsider using Docker or a cloud environment instead.\n```\n\nCheck the platform:\n```bash\nuname -s && uname -m\n```\n\n## Step 3: Environment Checks\n\nRun these checks in parallel:\n\n### 3.1 Python Version (REQUIRED: 3.10, 3.11, or 3.12)\n```bash\npython3 --version\n```\n**Python 3.13+ is NOT supported** due to dependency constraints.\n\nIf wrong version, advise:\n```\nNeMo Curator requires Python 3.10, 3.11, or 3.12.\nInstall with pyenv or your system package manager.\n```\n\n### 3.2 Check for existing virtual environment\n```bash\necho $VIRTUAL_ENV\n```\n\n### 3.3 Check if uv is installed\n```bash\nwhich uv && uv --version\n```\n\n### 3.4 GPU Check (if GPU selected)\n```bash\nnvidia-smi --query-gpu=name,memory.total --format=csv,noheader\n```\n\n### GPU VRAM Requirements\n| Modality | Minimum VRAM | Recommended |\n|----------|--------------|-------------|\n| Text (GPU) | 16 GB | 24+ GB |\n| Image | 24 GB | 48 GB |\n| Video | 21 GB (optimized) | 38 GB (full) |\n| Audio | 16 GB | 24+ GB |\n\nWarn if GPU VRAM is insufficient for selected modality.\n\n## Step 4: Install uv (PRIORITY)\n\n**uv is the recommended package manager for NeMo Curator.** It provides faster dependency resolution and better handling of complex extras.\n\nIf uv is not installed, ask permission and install:\n```bash\ncurl -LsSf https:\u002F\u002Fastral.sh\u002Fuv\u002Finstall.sh | sh\n```\n\nThen source it:\n```bash\nsource $HOME\u002F.local\u002Fbin\u002Fenv\n```\n\nVerify installation:\n```bash\nuv --version\n```\n\n## Step 5: Create Virtual Environment\n\nIf not in a virtual environment, create one with uv:\n```bash\nuv venv --python 3.11\nsource .venv\u002Fbin\u002Factivate\n```\n\nOr specify exact Python version if needed:\n```bash\nuv venv --python 3.10\n```\n\n## Step 6: Video Prerequisites (if Video modality selected)\n\n**Video curation has additional requirements:**\n\n### 6.1 FFmpeg with H.264 Encoders\nVideo processing requires FFmpeg compiled with specific encoders. Check if already installed:\n```bash\nffmpeg -encoders 2>\u002Fdev\u002Fnull | grep -E \"h264|264\"\n```\n\nShould show at least one of: `h264_nvenc` (GPU), `libx264`, or `libopenh264`.\n\nIf not available, on Ubuntu\u002FDebian:\n```bash\ncurl -fsSL https:\u002F\u002Fraw.githubusercontent.com\u002FNVIDIA-NeMo\u002FCurator\u002Fmain\u002Fdocker\u002Fcommon\u002Finstall_ffmpeg.sh -o install_ffmpeg.sh\nchmod +x install_ffmpeg.sh\nsudo bash install_ffmpeg.sh\n```\n\n**Note:** This installs ~30 system packages and builds FFmpeg from source. Takes 10-20 minutes.\n\n**Alternative:** Use the Docker container which has FFmpeg pre-built.\n\n### 6.2 System packages for FFmpeg build\nIf building FFmpeg, these are required:\n```bash\nsudo apt-get install -y build-essential autoconf automake cmake pkg-config libtool nasm yasm\n```\n\n## Step 7: Install NeMo Curator\n\n### Build the install command based on selections:\n\n**Single modality (CPU):**\n```bash\nuv pip install \"nemo-curator[text_cpu]\"\nuv pip install \"nemo-curator[image_cpu]\"\nuv pip install \"nemo-curator[video_cpu]\"\nuv pip install \"nemo-curator[audio_cpu]\"\n```\n\n**Single modality (GPU\u002FCUDA 12):**\n```bash\nuv pip install \"nemo-curator[text_cuda12]\"\nuv pip install \"nemo-curator[image_cuda12]\"\nuv pip install \"nemo-curator[video_cuda12]\"\nuv pip install \"nemo-curator[audio_cuda12]\"\n```\n\n**Multiple modalities - combine extras:**\n```bash\nuv pip install \"nemo-curator[text_cuda12,image_cuda12]\"\n```\n\n### Version Constraints to Know\n- PyTorch is limited to ≤2.9.1 for video modality\n- scikit-learn must be \u003C1.8.0 (handled automatically)\n- These are managed by uv, but be aware if you have existing installations\n\n### For Docker users:\nRecommend using the pre-built NGC container (includes FFmpeg and all dependencies):\n```bash\ndocker pull nvcr.io\u002Fnvidia\u002Fnemo-curator:25.09\ndocker run --gpus all -it --rm nvcr.io\u002Fnvidia\u002Fnemo-curator:25.09\n```\nThis container includes CUDA 12.8.1, Python 3.12, Ubuntu 24.04, and all modalities pre-installed.\n\n## Step 8: API Key Guidance\n\nBased on answers to Q3 and Q4, provide relevant guidance:\n\n### If synthetic data generation (Q3 = Yes):\n```\nTo use synthetic data generation, you'll need an NVIDIA API key.\n\n1. Get your key at: https:\u002F\u002Fbuild.nvidia.com\u002Fsettings\u002Fapi-keys\n2. Press CTRL\u002FCMD+Z, then type 'fg' to return to your shell\n3. Export the key:\n   export NVIDIA_API_KEY=\"your-key-here\"\n\nOr add to a .env file in your project.\n```\n\n### If gated HuggingFace models (Q4 = Yes):\n```\nTo use gated models like Llama tokenizers, you'll need a HuggingFace token.\n\n1. Get your token at: https:\u002F\u002Fhuggingface.co\u002Fsettings\u002Ftokens\n2. Accept the model license on HuggingFace (e.g., for Llama models)\n3. Press CTRL\u002FCMD+Z, then type 'fg' to return to your shell\n4. Export the token:\n   export HF_TOKEN=\"your-token-here\"\n```\n\n## Step 9: Verify Installation\n\nRun verification with user permission. Execute the verifications that apply based on the user's selections from Step 1.\n\n**Note:** All verification commands below use paths relative to the repository root. Run them from the Curator repo root, or prefix the script paths with `$(git rev-parse --show-toplevel)\u002F`.\n\n### 9.1 Basic Import Check (ALWAYS run)\n```bash\npython3 -c \"import nemo_curator; print(f'NeMo Curator {nemo_curator.__version__} installed')\"\n```\n\n### 9.2 Ray Cluster Verification (ALWAYS run)\nVerify Ray can initialize and stop correctly:\n```bash\npython3 .claude\u002Fskills\u002Fgetting-started\u002Fscripts\u002Fverify_ray.py\n```\n\n### 9.3 Modality-Specific Import Checks (run based on Q1 selection)\n\n| Modality | Command |\n|----------|---------|\n| Text | `python3 .claude\u002Fskills\u002Fgetting-started\u002Fscripts\u002Fverify_text.py` |\n| Image | `python3 .claude\u002Fskills\u002Fgetting-started\u002Fscripts\u002Fverify_image.py` |\n| Video | `python3 .claude\u002Fskills\u002Fgetting-started\u002Fscripts\u002Fverify_video.py` |\n| Audio | `python3 .claude\u002Fskills\u002Fgetting-started\u002Fscripts\u002Fverify_audio.py` |\n\n### 9.4 GPU Verification (if GPU selected in Q2)\n```bash\npython3 -c \"import torch; print(f'PyTorch {torch.__version__}'); print(f'CUDA available: {torch.cuda.is_available()}'); print(f'GPU: {torch.cuda.get_device_name(0)}' if torch.cuda.is_available() else '')\"\n```\n\n### 9.5 Video Encoder Verification (if Video selected in Q1)\n```bash\nffmpeg -encoders 2>\u002Fdev\u002Fnull | grep -E \"h264_nvenc|libx264|libopenh264\" && echo \"✓ H.264 encoder available\"\n```\n\n### 9.6 Pipeline Execution Verification (run based on Q2 selection)\n\n| Environment | Command |\n|-------------|---------|\n| GPU users | `python tutorials\u002Fquickstart.py` |\n| CPU-only users | `python3 .claude\u002Fskills\u002Fgetting-started\u002Fscripts\u002Fverify_pipeline_cpu.py` |\n\n## Step 10: Data Formats & Storage\n\n**Supported input\u002Foutput formats:**\n| Modality | Formats |\n|----------|---------|\n| Text | JSONL (primary), Parquet |\n| Image | WebDataset tar archives |\n| Video | Local paths, S3, HTTP(S) URLs |\n| Audio | Local files with JSONL manifests |\n\n**Cloud storage (S3):** Configure via `storage_options` dict passed to readers:\n```python\nstorage_options = {\"key\": \"AWS_KEY\", \"secret\": \"AWS_SECRET\"}\n```\n\n## Step 11: Ray & Network (Multi-Node\u002FCluster)\n\nIf running on a cluster or multi-node setup, be aware of Ray ports:\n\n| Port | Service | Purpose |\n|------|---------|---------|\n| 6379 | Ray GCS | Cluster communication |\n| 8265 | Ray Dashboard | Monitoring UI (http:\u002F\u002Flocalhost:8265) |\n| 10001 | Ray Client | Python client connection |\n| 10002-19999 | Workers | Worker communication range |\n\n**Firewall rules:** Ensure these ports are open between head and worker nodes.\n\n**Debugging tip:** Access the Ray dashboard at `http:\u002F\u002Flocalhost:8265` to monitor jobs, view logs, and check resource usage.\n\n**Environment variable:** Set `CURATOR_IGNORE_RAY_HEAD_NODE=1` to override Ray head node detection if needed.\n\n**Execution backends:** NeMo Curator uses `XennaExecutor` by default (recommended). Alternative experimental backends exist (`RayActorPoolExecutor`, `RayDataExecutor`) but stick with the default for getting started.\n\n## Step 12: First-Run Notes\n\n**Important:** Some models download on first use:\n- **Text classifiers** (quality, domain, safety): 100MB - 2GB each\n- **Image CLIP models**: ~1.5GB\n- **Video embedding models**: Cosmos-Embed1 (varies by resolution)\n- **Audio ASR models**: 100MB - 1GB depending on language\n\nThese download to `~\u002F.cache\u002Fhuggingface\u002F` by default. Ensure you have disk space and network access on first run.\n\n## Step 13: Recommend Tutorial\n\nBased on modality selection, recommend the most relevant tutorial from https:\u002F\u002Fgithub.com\u002FNVIDIA-NeMo\u002FCurator\u002Ftree\u002Fmain\u002Ftutorials\u002F:\n\n| Selection | Recommended Tutorial | Description |\n|-----------|---------------------|-------------|\n| **Text (beginner)** | `tutorials\u002Ftext\u002Ftinystories\u002F` | Small dataset, end-to-end pipeline (~22k samples) |\n| **Text (large-scale)** | `tutorials\u002Ftext\u002Fllama-nemotron-data-curation\u002F` | 30M sample pipeline with filtering |\n| **Text (PII)** | `tutorials\u002Ftext\u002Fgliner-pii-redaction\u002F` | Privacy\u002FPII redaction with GLiNER |\n| **Image** | `tutorials\u002Fimage\u002Fgetting-started\u002Fimage_curation_example.py` | CLIP embeddings, aesthetic scoring, NSFW filtering |\n| **Video** | `tutorials\u002Fvideo\u002Fgetting-started\u002Fvideo_split_clip_example.py` | Splitting, scene detection, captioning |\n| **Audio** | `tutorials\u002Faudio\u002Ffleurs\u002F` | Multilingual ASR with FLEURS dataset |\n| **Synthetic** | `tutorials\u002Fsynthetic\u002Fsynthetic_data_generation_example.py` | LLM-based Q&A generation |\n\n**Always also recommend:** `tutorials\u002Fquickstart.py` for understanding the core pipeline architecture (Tasks, Stages, Pipelines).\n\n**Pipeline config YAMLs:** Pre-built configs exist in `nemo_curator\u002Fconfig\u002Ftext\u002F` for common workflows:\n- `exact_deduplication_pipeline.yaml`\n- `semantic_deduplication_pipeline.yaml`\n- `fuzzy_deduplication_pipeline.yaml`\n- `heuristic_filter_english_pipeline.yaml`\n- `heuristic_filter_non_english_pipeline.yaml`\n- `fasttext_filter_pipeline.yaml`\n- `code_filter_pipeline.yaml`\n\n## Step 14: Summary Output\n\nAt the end, provide a summary:\n```\nNeMo Curator Setup Complete!\n\nPlatform: {Linux\u002FmacOS} {x86_64\u002Farm64}\nPython: {version}\nPackage manager: uv {version}\nInstalled: nemo-curator[{extras}]\nModalities: {selected modalities}\nCompute: {CPU\u002FGPU with VRAM}\nAPI Keys needed: {list or \"None\"}\n\nFirst-run note: Models will download on first use (~1-2GB per modality)\n\nNext steps:\n1. {If API keys needed: Set your API keys (CTRL\u002FCMD+Z -> fg)}\n2. Try the quickstart: python tutorials\u002Fquickstart.py\n3. Then try: {recommended tutorial path}\n4. Docs: https:\u002F\u002Fdocs.nvidia.com\u002Fnemo-framework\u002Fuser-guide\u002Flatest\u002Fnemocurator\u002F\n```\n\n## Troubleshooting\n\nIf installation fails, check:\n\n1. **\"No matching distribution\"** - Wrong Python version (need 3.10-3.12)\n2. **CUDA errors** - Ensure CUDA 12.x drivers installed (`nvidia-smi`)\n3. **scikit-learn conflicts** - NeMo Curator requires \u003C1.8.0\n4. **FFmpeg encoder missing** - Run the FFmpeg install script\n5. **Out of memory** - Check GPU VRAM requirements above\n6. **Ray port conflicts** - Check if ports 6379, 8265, 10001 are in use\n7. **Job interrupted** - Resumable: compare input\u002Foutput file counts to find remaining work\n\n**Debugging:**\n- Ray dashboard: `http:\u002F\u002Flocalhost:8265`\n- Logs use `loguru` - check stdout\u002Fstderr for detailed traces\n",{"name":5,"description":7},"main","8a9a217c90127d424a8f503cb948698d7203002a",".claude\u002Fskills\u002Fgetting-started",371,"https:\u002F\u002Fgithub.com\u002FNVIDIA-NeMo\u002FCurator","https:\u002F\u002Fgithub.com\u002FNVIDIA-NeMo","https:\u002F\u002Fnvidia.com\u002F","2026-11-11 08:35:27",{"source":8,"skillName":5,"commands":46,"prompt":56,"zipAvailable":57},[47,50,53],{"tool":48,"command":49},"npx","npx skills add NVIDIA-NeMo\u002FCurator --skill getting-started",{"tool":51,"command":52},"bunx","bunx skills add NVIDIA-NeMo\u002FCurator --skill getting-started",{"tool":54,"command":55},"pnpm","pnpm dlx skills add NVIDIA-NeMo\u002FCurator --skill getting-started","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FNVIDIA-NeMo\u002FCurator\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fgetting-started\u002FSKILL.md",true,[59,65,71,75,79,83,89,93,97,101,105,109,113,117,121],{"path":60,"resourceType":61,"resourceTypeDesc":62,"sizeBytes":63,"sourceHtmlUrl":64},"LICENSE.txt","LICENSE","许可证",11345,"https:\u002F\u002Fgithub.com\u002Fvox-foundation\u002Fvox\u002Fblob\u002Fmain\u002Fassets\u002Fskills\u002Fmcp-builder\u002FLICENSE.txt",{"path":66,"resourceType":67,"resourceTypeDesc":68,"sizeBytes":69,"sourceHtmlUrl":70},"reference\u002Fevaluation.md","REFERENCE","参考文档",21663,"https:\u002F\u002Fgithub.com\u002Fvox-foundation\u002Fvox\u002Fblob\u002Fmain\u002Fassets\u002Fskills\u002Fmcp-builder\u002Freference\u002Fevaluation.md",{"path":72,"resourceType":67,"resourceTypeDesc":68,"sizeBytes":73,"sourceHtmlUrl":74},"reference\u002Fmcp_best_practices.md",7330,"https:\u002F\u002Fgithub.com\u002Fvox-foundation\u002Fvox\u002Fblob\u002Fmain\u002Fassets\u002Fskills\u002Fmcp-builder\u002Freference\u002Fmcp_best_practices.md",{"path":76,"resourceType":67,"resourceTypeDesc":68,"sizeBytes":77,"sourceHtmlUrl":78},"reference\u002Fnode_mcp_server.md",28550,"https:\u002F\u002Fgithub.com\u002Fvox-foundation\u002Fvox\u002Fblob\u002Fmain\u002Fassets\u002Fskills\u002Fmcp-builder\u002Freference\u002Fnode_mcp_server.md",{"path":80,"resourceType":67,"resourceTypeDesc":68,"sizeBytes":81,"sourceHtmlUrl":82},"reference\u002Fpython_mcp_server.md",25099,"https:\u002F\u002Fgithub.com\u002Fvox-foundation\u002Fvox\u002Fblob\u002Fmain\u002Fassets\u002Fskills\u002Fmcp-builder\u002Freference\u002Fpython_mcp_server.md",{"path":84,"resourceType":85,"resourceTypeDesc":86,"sizeBytes":87,"sourceHtmlUrl":88},"scripts\u002Fconnections.py","SCRIPT","脚本",4875,"https:\u002F\u002Fgithub.com\u002Fvox-foundation\u002Fvox\u002Fblob\u002Fmain\u002Fassets\u002Fskills\u002Fmcp-builder\u002Fscripts\u002Fconnections.py",{"path":90,"resourceType":85,"resourceTypeDesc":86,"sizeBytes":91,"sourceHtmlUrl":92},"scripts\u002Fevaluation.py",12579,"https:\u002F\u002Fgithub.com\u002Fvox-foundation\u002Fvox\u002Fblob\u002Fmain\u002Fassets\u002Fskills\u002Fmcp-builder\u002Fscripts\u002Fevaluation.py",{"path":94,"resourceType":85,"resourceTypeDesc":86,"sizeBytes":95,"sourceHtmlUrl":96},"scripts\u002Fexample_evaluation.xml",1194,"https:\u002F\u002Fgithub.com\u002Fvox-foundation\u002Fvox\u002Fblob\u002Fmain\u002Fassets\u002Fskills\u002Fmcp-builder\u002Fscripts\u002Fexample_evaluation.xml",{"path":98,"resourceType":85,"resourceTypeDesc":86,"sizeBytes":99,"sourceHtmlUrl":100},"scripts\u002Frequirements.txt",29,"https:\u002F\u002Fgithub.com\u002Fvox-foundation\u002Fvox\u002Fblob\u002Fmain\u002Fassets\u002Fskills\u002Fmcp-builder\u002Fscripts\u002Frequirements.txt",{"path":102,"resourceType":85,"resourceTypeDesc":86,"sizeBytes":103,"sourceHtmlUrl":104},"scripts\u002Fverify_audio.py",1252,"https:\u002F\u002Fgithub.com\u002FNVIDIA-NeMo\u002FCurator\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fgetting-started\u002Fscripts\u002Fverify_audio.py",{"path":106,"resourceType":85,"resourceTypeDesc":86,"sizeBytes":107,"sourceHtmlUrl":108},"scripts\u002Fverify_image.py",204,"https:\u002F\u002Fgithub.com\u002FNVIDIA-NeMo\u002FCurator\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fgetting-started\u002Fscripts\u002Fverify_image.py",{"path":110,"resourceType":85,"resourceTypeDesc":86,"sizeBytes":111,"sourceHtmlUrl":112},"scripts\u002Fverify_pipeline_cpu.py",1901,"https:\u002F\u002Fgithub.com\u002FNVIDIA-NeMo\u002FCurator\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fgetting-started\u002Fscripts\u002Fverify_pipeline_cpu.py",{"path":114,"resourceType":85,"resourceTypeDesc":86,"sizeBytes":115,"sourceHtmlUrl":116},"scripts\u002Fverify_ray.py",269,"https:\u002F\u002Fgithub.com\u002FNVIDIA-NeMo\u002FCurator\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fgetting-started\u002Fscripts\u002Fverify_ray.py",{"path":118,"resourceType":85,"resourceTypeDesc":86,"sizeBytes":119,"sourceHtmlUrl":120},"scripts\u002Fverify_text.py",283,"https:\u002F\u002Fgithub.com\u002FNVIDIA-NeMo\u002FCurator\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fgetting-started\u002Fscripts\u002Fverify_text.py",{"path":122,"resourceType":85,"resourceTypeDesc":86,"sizeBytes":123,"sourceHtmlUrl":124},"scripts\u002Fverify_video.py",219,"https:\u002F\u002Fgithub.com\u002FNVIDIA-NeMo\u002FCurator\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fgetting-started\u002Fscripts\u002Fverify_video.py",[126,140,163,177,190,196],{"skillNo":127,"name":128,"nameZh":6,"description":129,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":130,"categories":131,"keywords":132,"publishedAt":32},"01M6BXJ86EP64TND1NAHC9MHSS","checkpoint-placement","Select a core-proven metadata checkpoint for same-dataset parameter feedback. Use after recipe validation and semantic critique, before authoritative smoke, when the user expects to tune a declared downstream decision after seeing results.","https:\u002F\u002Fgithub.com\u002FNVIDIA-NeMo\u002FCurator\u002Fblob\u002Fmain\u002Fnemo_curator\u002Fstages\u002Faudio\u002Faudio_agent\u002Fskills\u002Fcheckpoint-placement\u002FSKILL.md",[],[133,134,135,136,137,138,139],"checkpoint placement","checkpoint","feedback","critique","metadata","validation","core",{"skillNo":141,"name":142,"nameZh":6,"description":143,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":144,"categories":145,"keywords":158,"publishedAt":32},"01M6BXJ4WRPMMBPADDY22KS3RG","audio-stage-authoring","Make a new or existing NeMo Curator audio stage agent-ready so the audio agent can discover, configure and chain it. Use when adding an audio stage, writing or fixing its describe() contract or StageContract, adding *_key fields or semantic roles, writing a capability card, or when card conformance or assert_agent_ready fails. This is for authoring stage code in a Curator checkout, not for curating a dataset - use the audio-curation skill for that.","https:\u002F\u002Fgithub.com\u002FNVIDIA-NeMo\u002FCurator\u002Fblob\u002Fmain\u002Fnemo_curator\u002Fstages\u002Faudio\u002Faudio_agent\u002Fskills\u002Faudio-stage-authoring\u002FSKILL.md",[146,147,150,153],{"categoryNo":18,"categoryName":19,"taxonomyType":20,"taxonomyTypeDesc":21},{"categoryNo":148,"categoryName":149,"taxonomyType":20,"taxonomyTypeDesc":21},"01JBSEEDCAT000000000000018","写作与文案",{"categoryNo":151,"categoryName":152,"taxonomyType":20,"taxonomyTypeDesc":21},"01JBSEEDCAT000000000000023","Skill与智能体开发",{"categoryNo":154,"categoryName":155,"taxonomyType":156,"taxonomyTypeDesc":157},"01JBSEEDCAT000000000000036","内容创作者","PROFESSION","适用职业",[159,29,160,161,25,162],"audio stage authoring","writing","configure","code",{"skillNo":164,"name":165,"nameZh":6,"description":166,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":167,"categories":168,"keywords":171,"publishedAt":32},"01M6BXJ1FTZZ76B6E79ZCJP2Q6","audio-curation","Build and run a NeMo Curator audio curation pipeline from a natural-language goal (host-driven planner over the audio_agent tool core). Use for any audio, speech or audio-dataset task - quality filtering, transcription, WER filtering, VAD, diarization, ALM windowing, resampling, channel conversion, or inspecting an audio corpus. Route such work through this skill rather than writing ad-hoc ffprobe, librosa or soundfile scripts.","https:\u002F\u002Fgithub.com\u002FNVIDIA-NeMo\u002FCurator\u002Fblob\u002Fmain\u002Fnemo_curator\u002Fstages\u002Faudio\u002Faudio_agent\u002Fskills\u002Faudio-curation\u002FSKILL.md",[169,170],{"categoryNo":18,"categoryName":19,"taxonomyType":20,"taxonomyTypeDesc":21},{"categoryNo":151,"categoryName":152,"taxonomyType":20,"taxonomyTypeDesc":21},[172,29,173,174,25,175,176,139,160],"audio curation","tool","pipeline","speech","run",{"skillNo":178,"name":179,"nameZh":6,"description":180,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":181,"categories":182,"keywords":183,"publishedAt":32},"01M6BXHYGV7VCDAMVQXPJ1CMDE","nemo-curator-docs","Maintain the NeMo Curator Fern docs site — add, update, move, or remove pages under fern\u002F. Use for any documentation changes.","https:\u002F\u002Fgithub.com\u002FNVIDIA-NeMo\u002FCurator\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fnemo-curator-docs\u002FSKILL.md",[],[184,25,185,186,187,188,189],"nemo curator docs","docs","site","documentation","update","docs site",{"skillNo":191,"name":165,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":192,"categories":193,"keywords":195,"publishedAt":32},"01M6BXHQ573S6BCYDZ224MEX9K","https:\u002F\u002Fgithub.com\u002FNVIDIA-NeMo\u002FCurator\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Faudio-curation\u002FSKILL.md",[194],{"categoryNo":18,"categoryName":19,"taxonomyType":20,"taxonomyTypeDesc":21},[29,172],{"skillNo":197,"name":128,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":198,"categories":199,"keywords":200,"publishedAt":32},"01M6BXHM5EPNGD9S4ED7P7KEYS","https:\u002F\u002Fgithub.com\u002FNVIDIA-NeMo\u002FCurator\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fcheckpoint-placement\u002FSKILL.md",[],[133,134]]