[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6MA6EVNMM7TWBPYMQF433PJ":3,"skill-same:01M6MA6EVNMM7TWBPYMQF433PJ":85},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":40,"linkOnly":41,"markdownContent":42,"markdownBody":43,"frontMatter":44,"refName":51,"commitSha":52,"skillDirectory":53,"forksCount":54,"repositoryHtmlUrl":55,"repositoryHomepageUrl":56,"ownerHtmlUrl":57,"ownerBlogUrl":58,"ownerDescription":59,"updatedAt":60,"install":61,"resources":74},"01M6MA6EVNMM7TWBPYMQF433PJ","gguf-quantization",null,"GGUF format and llama.cpp quantization for efficient CPU\u002FGPU inference. Use when deploying models on consumer hardware, Apple Silicon, or when needing flexible quantization from 2-8 bit without GPU requirements.","hashgraph-online\u002Fawesome-codex-plugins","hashgraph-online","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F160371962?v=4","01M6CNB6NRJVNNGN14TR01Q6D3","HOL (Hashgraph Online)","01M6JT1HSKKG9V68KQAQGBA7XP",1575,"MIT","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fgguf-quantization\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39],"GGUF","gguf quantization","Quantization","apple","when","for","Model Compression","CPU Inference","llama.cpp","CPU","silicon","bit","python","requirements","use","model","apple silicon","Optimization","cpp","training","ml","2026-11-10 10:12:30",false,"---\nname: gguf-quantization\ndescription: GGUF format and llama.cpp quantization for efficient CPU\u002FGPU inference. Use when deploying models on consumer hardware, Apple Silicon, or when needing flexible quantization from 2-8 bit without GPU requirements.\nlicense: MIT\nmetadata:\n  version: 1.0.0\n  category: ml-training\n  maintainer: Kalaris Labs\n  tags: GGUF, Quantization, llama.cpp, CPU Inference, Apple Silicon, Model Compression, Optimization\n  dependencies: llama-cpp-python>=0.2.0\n---\n\n# GGUF - Quantization Format for llama.cpp\n\nThe GGUF (GPT-Generated Unified Format) is the standard file format for llama.cpp, enabling efficient inference on CPUs, Apple Silicon, and GPUs with flexible quantization options.\n\n## When to use GGUF\n\n**Use GGUF when:**\n- Deploying on consumer hardware (laptops, desktops)\n- Running on Apple Silicon (M1\u002FM2\u002FM3) with Metal acceleration\n- Need CPU inference without GPU requirements\n- Want flexible quantization (Q2_K to Q8_0)\n- Using local AI tools (LM Studio, Ollama, text-generation-webui)\n\n**Key advantages:**\n- **Universal hardware**: CPU, Apple Silicon, NVIDIA, AMD support\n- **No Python runtime**: Pure C\u002FC++ inference\n- **Flexible quantization**: 2-8 bit with various methods (K-quants)\n- **Ecosystem support**: LM Studio, Ollama, koboldcpp, and more\n- **imatrix**: Importance matrix for better low-bit quality\n\n**Use alternatives instead:**\n- **AWQ\u002FGPTQ**: Maximum accuracy with calibration on NVIDIA GPUs\n- **HQQ**: Fast calibration-free quantization for HuggingFace\n- **bitsandbytes**: Simple integration with transformers library\n- **TensorRT-LLM**: Production NVIDIA deployment with maximum speed\n\n## Quick start\n\n### Installation\n\n```bash\n# Clone llama.cpp\ngit clone https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\ncd llama.cpp\n\n# Build (CPU)\nmake\n\n# Build with CUDA (NVIDIA)\nmake GGML_CUDA=1\n\n# Build with Metal (Apple Silicon)\nmake GGML_METAL=1\n\n# Install Python bindings (optional)\npip install llama-cpp-python\n```\n\n### Convert model to GGUF\n\n```bash\n# Install requirements\npip install -r requirements.txt\n\n# Convert HuggingFace model to GGUF (FP16)\npython convert_hf_to_gguf.py .\u002Fpath\u002Fto\u002Fmodel --outfile model-f16.gguf\n\n# Or specify output type\npython convert_hf_to_gguf.py .\u002Fpath\u002Fto\u002Fmodel \\\n    --outfile model-f16.gguf \\\n    --outtype f16\n```\n\n### Quantize model\n\n```bash\n# Basic quantization to Q4_K_M\n.\u002Fllama-quantize model-f16.gguf model-q4_k_m.gguf Q4_K_M\n\n# Quantize with importance matrix (better quality)\n.\u002Fllama-imatrix -m model-f16.gguf -f calibration.txt -o model.imatrix\n.\u002Fllama-quantize --imatrix model.imatrix model-f16.gguf model-q4_k_m.gguf Q4_K_M\n```\n\n### Run inference\n\n```bash\n# CLI inference\n.\u002Fllama-cli -m model-q4_k_m.gguf -p \"Hello, how are you?\"\n\n# Interactive mode\n.\u002Fllama-cli -m model-q4_k_m.gguf --interactive\n\n# With GPU offload\n.\u002Fllama-cli -m model-q4_k_m.gguf -ngl 35 -p \"Hello!\"\n```\n\n## Quantization types\n\n### K-quant methods (recommended)\n\n| Type | Bits | Size (7B) | Quality | Use Case |\n|------|------|-----------|---------|----------|\n| Q2_K | 2.5 | ~2.8 GB | Low | Extreme compression |\n| Q3_K_S | 3.0 | ~3.0 GB | Low-Med | Memory constrained |\n| Q3_K_M | 3.3 | ~3.3 GB | Medium | Balance |\n| Q4_K_S | 4.0 | ~3.8 GB | Med-High | Good balance |\n| Q4_K_M | 4.5 | ~4.1 GB | High | **Recommended default** |\n| Q5_K_S | 5.0 | ~4.6 GB | High | Quality focused |\n| Q5_K_M | 5.5 | ~4.8 GB | Very High | High quality |\n| Q6_K | 6.0 | ~5.5 GB | Excellent | Near-original |\n| Q8_0 | 8.0 | ~7.2 GB | Best | Maximum quality |\n\n### Legacy methods\n\n| Type | Description |\n|------|-------------|\n| Q4_0 | 4-bit, basic |\n| Q4_1 | 4-bit with delta |\n| Q5_0 | 5-bit, basic |\n| Q5_1 | 5-bit with delta |\n\n**Recommendation**: Use K-quant methods (Q4_K_M, Q5_K_M) for best quality\u002Fsize ratio.\n\n## Conversion workflows\n\n### Workflow 1: HuggingFace to GGUF\n\n```bash\n# 1. Download model\nhuggingface-cli download meta-llama\u002FLlama-3.1-8B --local-dir .\u002Fllama-3.1-8b\n\n# 2. Convert to GGUF (FP16)\npython convert_hf_to_gguf.py .\u002Fllama-3.1-8b \\\n    --outfile llama-3.1-8b-f16.gguf \\\n    --outtype f16\n\n# 3. Quantize\n.\u002Fllama-quantize llama-3.1-8b-f16.gguf llama-3.1-8b-q4_k_m.gguf Q4_K_M\n\n# 4. Test\n.\u002Fllama-cli -m llama-3.1-8b-q4_k_m.gguf -p \"Hello!\" -n 50\n```\n\n### Workflow 2: With importance matrix (better quality)\n\n```bash\n# 1. Convert to GGUF\npython convert_hf_to_gguf.py .\u002Fmodel --outfile model-f16.gguf\n\n# 2. Create calibration text (diverse samples)\ncat > calibration.txt \u003C\u003C 'EOF'\nThe quick brown fox jumps over the lazy dog.\nMachine learning is a subset of artificial intelligence.\nPython is a popular programming language.\n# Add more diverse text samples...\nEOF\n\n# 3. Generate importance matrix\n.\u002Fllama-imatrix -m model-f16.gguf \\\n    -f calibration.txt \\\n    --chunk 512 \\\n    -o model.imatrix \\\n    -ngl 35  # GPU layers if available\n\n# 4. Quantize with imatrix\n.\u002Fllama-quantize --imatrix model.imatrix \\\n    model-f16.gguf \\\n    model-q4_k_m.gguf \\\n    Q4_K_M\n```\n\n### Workflow 3: Multiple quantizations\n\n```bash\n#!\u002Fbin\u002Fbash\nMODEL=\"llama-3.1-8b-f16.gguf\"\nIMATRIX=\"llama-3.1-8b.imatrix\"\n\n# Generate imatrix once\n.\u002Fllama-imatrix -m $MODEL -f wiki.txt -o $IMATRIX -ngl 35\n\n# Create multiple quantizations\nfor QUANT in Q4_K_M Q5_K_M Q6_K Q8_0; do\n    OUTPUT=\"llama-3.1-8b-${QUANT,,}.gguf\"\n    .\u002Fllama-quantize --imatrix $IMATRIX $MODEL $OUTPUT $QUANT\n    echo \"Created: $OUTPUT ($(du -h $OUTPUT | cut -f1))\"\ndone\n```\n\n## Python usage\n\n### llama-cpp-python\n\n```python\nfrom llama_cpp import Llama\n\n# Load model\nllm = Llama(\n    model_path=\".\u002Fmodel-q4_k_m.gguf\",\n    n_ctx=4096,          # Context window\n    n_gpu_layers=35,     # GPU offload (0 for CPU only)\n    n_threads=8          # CPU threads\n)\n\n# Generate\noutput = llm(\n    \"What is machine learning?\",\n    max_tokens=256,\n    temperature=0.7,\n    stop=[\"\u003C\u002Fs>\", \"\\n\\n\"]\n)\nprint(output[\"choices\"][0][\"text\"])\n```\n\n### Chat completion\n\n```python\nfrom llama_cpp import Llama\n\nllm = Llama(\n    model_path=\".\u002Fmodel-q4_k_m.gguf\",\n    n_ctx=4096,\n    n_gpu_layers=35,\n    chat_format=\"llama-3\"  # Or \"chatml\", \"mistral\", etc.\n)\n\nmessages = [\n    {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n    {\"role\": \"user\", \"content\": \"What is Python?\"}\n]\n\nresponse = llm.create_chat_completion(\n    messages=messages,\n    max_tokens=256,\n    temperature=0.7\n)\nprint(response[\"choices\"][0][\"message\"][\"content\"])\n```\n\n### Streaming\n\n```python\nfrom llama_cpp import Llama\n\nllm = Llama(model_path=\".\u002Fmodel-q4_k_m.gguf\", n_gpu_layers=35)\n\n# Stream tokens\nfor chunk in llm(\n    \"Explain quantum computing:\",\n    max_tokens=256,\n    stream=True\n):\n    print(chunk[\"choices\"][0][\"text\"], end=\"\", flush=True)\n```\n\n## Server mode\n\n### Start OpenAI-compatible server\n\n```bash\n# Start server\n.\u002Fllama-server -m model-q4_k_m.gguf \\\n    --host 0.0.0.0 \\\n    --port 8080 \\\n    -ngl 35 \\\n    -c 4096\n\n# Or with Python bindings\npython -m llama_cpp.server \\\n    --model model-q4_k_m.gguf \\\n    --n_gpu_layers 35 \\\n    --host 0.0.0.0 \\\n    --port 8080\n```\n\n### Use with OpenAI client\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(\n    base_url=\"http:\u002F\u002Flocalhost:8080\u002Fv1\",\n    api_key=\"not-needed\"\n)\n\nresponse = client.chat.completions.create(\n    model=\"local-model\",\n    messages=[{\"role\": \"user\", \"content\": \"Hello!\"}],\n    max_tokens=256\n)\nprint(response.choices[0].message.content)\n```\n\n## Hardware optimization\n\n### Apple Silicon (Metal)\n\n```bash\n# Build with Metal\nmake clean && make GGML_METAL=1\n\n# Run with Metal acceleration\n.\u002Fllama-cli -m model.gguf -ngl 99 -p \"Hello\"\n\n# Python with Metal\nllm = Llama(\n    model_path=\"model.gguf\",\n    n_gpu_layers=99,     # Offload all layers\n    n_threads=1          # Metal handles parallelism\n)\n```\n\n### NVIDIA CUDA\n\n```bash\n# Build with CUDA\nmake clean && make GGML_CUDA=1\n\n# Run with CUDA\n.\u002Fllama-cli -m model.gguf -ngl 35 -p \"Hello\"\n\n# Specify GPU\nCUDA_VISIBLE_DEVICES=0 .\u002Fllama-cli -m model.gguf -ngl 35\n```\n\n### CPU optimization\n\n```bash\n# Build with AVX2\u002FAVX512\nmake clean && make\n\n# Run with optimal threads\n.\u002Fllama-cli -m model.gguf -t 8 -p \"Hello\"\n\n# Python CPU config\nllm = Llama(\n    model_path=\"model.gguf\",\n    n_gpu_layers=0,      # CPU only\n    n_threads=8,         # Match physical cores\n    n_batch=512          # Batch size for prompt processing\n)\n```\n\n## Integration with tools\n\n### Ollama\n\n```bash\n# Create Modelfile\ncat > Modelfile \u003C\u003C 'EOF'\nFROM .\u002Fmodel-q4_k_m.gguf\nTEMPLATE \"\"\"{{ .System }}\n{{ .Prompt }}\"\"\"\nPARAMETER temperature 0.7\nPARAMETER num_ctx 4096\nEOF\n\n# Create Ollama model\nollama create mymodel -f Modelfile\n\n# Run\nollama run mymodel \"Hello!\"\n```\n\n### LM Studio\n\n1. Place GGUF file in `~\u002F.cache\u002Flm-studio\u002Fmodels\u002F`\n2. Open LM Studio and select the model\n3. Configure context length and GPU offload\n4. Start inference\n\n### text-generation-webui\n\n```bash\n# Place in models folder\ncp model-q4_k_m.gguf text-generation-webui\u002Fmodels\u002F\n\n# Start with llama.cpp loader\npython server.py --model model-q4_k_m.gguf --loader llama.cpp --n-gpu-layers 35\n```\n\n## Best practices\n\n1. **Use K-quants**: Q4_K_M offers best quality\u002Fsize balance\n2. **Use imatrix**: Always use importance matrix for Q4 and below\n3. **GPU offload**: Offload as many layers as VRAM allows\n4. **Context length**: Start with 4096, increase if needed\n5. **Thread count**: Match physical CPU cores, not logical\n6. **Batch size**: Increase n_batch for faster prompt processing\n\n## Common issues\n\n**Model loads slowly:**\n```bash\n# Use mmap for faster loading\n.\u002Fllama-cli -m model.gguf --mmap\n```\n\n**Out of memory:**\n```bash\n# Reduce GPU layers\n.\u002Fllama-cli -m model.gguf -ngl 20  # Reduce from 35\n\n# Or use smaller quantization\n.\u002Fllama-quantize model-f16.gguf model-q3_k_m.gguf Q3_K_M\n```\n\n**Poor quality at low bits:**\n```bash\n# Always use imatrix for Q4 and below\n.\u002Fllama-imatrix -m model-f16.gguf -f calibration.txt -o model.imatrix\n.\u002Fllama-quantize --imatrix model.imatrix model-f16.gguf model-q4_k_m.gguf Q4_K_M\n```\n\n## References\n\n- **[Advanced Usage](references\u002Fadvanced-usage.md)** - Batching, speculative decoding, custom builds\n- **[Troubleshooting](references\u002Ftroubleshooting.md)** - Common issues, debugging, benchmarks\n\n## Resources\n\n- **Repository**: https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\n- **Python Bindings**: https:\u002F\u002Fgithub.com\u002Fabetlen\u002Fllama-cpp-python\n- **Pre-quantized Models**: https:\u002F\u002Fhuggingface.co\u002FTheBloke\n- **GGUF Converter**: https:\u002F\u002Fhuggingface.co\u002Fspaces\u002Fggml-org\u002Fgguf-my-repo\n- **License**: MIT\n\n## Agent operating procedure\n\n1. **Check the environment.** Check GPU type, memory and driver\u002FCUDA versions (`nvidia-smi`), framework versions, and dataset location and size.\n2. **Pin down the inputs.** Confirm formats, identifiers and parameters from the data or the user. Ask rather than guess any value that changes the result.\n3. **Run a small version first.** Do a smoke run: tiny model or subset, few steps, and confirm loss decreases and checkpoints save.\n4. **Execute the full task** using the instructions and references above.\n5. **Validate the result.** Track metrics on held-out data, compare against a baseline, and record seeds, configs and hardware.\n6. **Report.** State what was run (versions, commands, parameters), what was checked, and what is still uncertain.\n\n| If this happens | Do this |\n|---|---|\n| CUDA out-of-memory | Reduce batch size, enable gradient accumulation\u002Fcheckpointing or mixed precision, or shard the model. |\n| Loss is NaN or diverges | Lower the learning rate, check data for invalid values, and enable gradient clipping. |\n| A function, flag or endpoint in these instructions is missing in the installed version | Check the installed version's own documentation (`help()`, `--help`, official docs), adapt, and tell the user. Never invent an API. |\n| A required input, identifier or parameter is ambiguous | Ask the user, or state the assumption explicitly before running. |\n\n**Integrity rules**\n\n- Never fabricate results, parameters, identifiers, citations or statistics. If something cannot be run or verified, say so plainly.\n- Never claim training results without logs; estimate compute cost before launching large jobs.\n- Treat version-specific details here as possibly outdated: confirm them against the official documentation for the installed version.\n- Ask before actions that cost money, consume shared GPUs or cloud quota, touch personal or patient data, or cannot be undone.\n\n## Related skills\n\n- `llama-cpp`: Runs LLM inference on CPU, Apple Silicon, and consumer GPUs without NVIDIA hardware.\n- `hqq-quantization`: Half-Quadratic Quantization for LLMs without calibration data.\n- `awq-quantization`: Activation-aware weight quantization for 4-bit LLM compression with 3x speedup and minimal accuracy loss.\n","# GGUF - Quantization Format for llama.cpp\n\nThe GGUF (GPT-Generated Unified Format) is the standard file format for llama.cpp, enabling efficient inference on CPUs, Apple Silicon, and GPUs with flexible quantization options.\n\n## When to use GGUF\n\n**Use GGUF when:**\n- Deploying on consumer hardware (laptops, desktops)\n- Running on Apple Silicon (M1\u002FM2\u002FM3) with Metal acceleration\n- Need CPU inference without GPU requirements\n- Want flexible quantization (Q2_K to Q8_0)\n- Using local AI tools (LM Studio, Ollama, text-generation-webui)\n\n**Key advantages:**\n- **Universal hardware**: CPU, Apple Silicon, NVIDIA, AMD support\n- **No Python runtime**: Pure C\u002FC++ inference\n- **Flexible quantization**: 2-8 bit with various methods (K-quants)\n- **Ecosystem support**: LM Studio, Ollama, koboldcpp, and more\n- **imatrix**: Importance matrix for better low-bit quality\n\n**Use alternatives instead:**\n- **AWQ\u002FGPTQ**: Maximum accuracy with calibration on NVIDIA GPUs\n- **HQQ**: Fast calibration-free quantization for HuggingFace\n- **bitsandbytes**: Simple integration with transformers library\n- **TensorRT-LLM**: Production NVIDIA deployment with maximum speed\n\n## Quick start\n\n### Installation\n\n```bash\n# Clone llama.cpp\ngit clone https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\ncd llama.cpp\n\n# Build (CPU)\nmake\n\n# Build with CUDA (NVIDIA)\nmake GGML_CUDA=1\n\n# Build with Metal (Apple Silicon)\nmake GGML_METAL=1\n\n# Install Python bindings (optional)\npip install llama-cpp-python\n```\n\n### Convert model to GGUF\n\n```bash\n# Install requirements\npip install -r requirements.txt\n\n# Convert HuggingFace model to GGUF (FP16)\npython convert_hf_to_gguf.py .\u002Fpath\u002Fto\u002Fmodel --outfile model-f16.gguf\n\n# Or specify output type\npython convert_hf_to_gguf.py .\u002Fpath\u002Fto\u002Fmodel \\\n    --outfile model-f16.gguf \\\n    --outtype f16\n```\n\n### Quantize model\n\n```bash\n# Basic quantization to Q4_K_M\n.\u002Fllama-quantize model-f16.gguf model-q4_k_m.gguf Q4_K_M\n\n# Quantize with importance matrix (better quality)\n.\u002Fllama-imatrix -m model-f16.gguf -f calibration.txt -o model.imatrix\n.\u002Fllama-quantize --imatrix model.imatrix model-f16.gguf model-q4_k_m.gguf Q4_K_M\n```\n\n### Run inference\n\n```bash\n# CLI inference\n.\u002Fllama-cli -m model-q4_k_m.gguf -p \"Hello, how are you?\"\n\n# Interactive mode\n.\u002Fllama-cli -m model-q4_k_m.gguf --interactive\n\n# With GPU offload\n.\u002Fllama-cli -m model-q4_k_m.gguf -ngl 35 -p \"Hello!\"\n```\n\n## Quantization types\n\n### K-quant methods (recommended)\n\n| Type | Bits | Size (7B) | Quality | Use Case |\n|------|------|-----------|---------|----------|\n| Q2_K | 2.5 | ~2.8 GB | Low | Extreme compression |\n| Q3_K_S | 3.0 | ~3.0 GB | Low-Med | Memory constrained |\n| Q3_K_M | 3.3 | ~3.3 GB | Medium | Balance |\n| Q4_K_S | 4.0 | ~3.8 GB | Med-High | Good balance |\n| Q4_K_M | 4.5 | ~4.1 GB | High | **Recommended default** |\n| Q5_K_S | 5.0 | ~4.6 GB | High | Quality focused |\n| Q5_K_M | 5.5 | ~4.8 GB | Very High | High quality |\n| Q6_K | 6.0 | ~5.5 GB | Excellent | Near-original |\n| Q8_0 | 8.0 | ~7.2 GB | Best | Maximum quality |\n\n### Legacy methods\n\n| Type | Description |\n|------|-------------|\n| Q4_0 | 4-bit, basic |\n| Q4_1 | 4-bit with delta |\n| Q5_0 | 5-bit, basic |\n| Q5_1 | 5-bit with delta |\n\n**Recommendation**: Use K-quant methods (Q4_K_M, Q5_K_M) for best quality\u002Fsize ratio.\n\n## Conversion workflows\n\n### Workflow 1: HuggingFace to GGUF\n\n```bash\n# 1. Download model\nhuggingface-cli download meta-llama\u002FLlama-3.1-8B --local-dir .\u002Fllama-3.1-8b\n\n# 2. Convert to GGUF (FP16)\npython convert_hf_to_gguf.py .\u002Fllama-3.1-8b \\\n    --outfile llama-3.1-8b-f16.gguf \\\n    --outtype f16\n\n# 3. Quantize\n.\u002Fllama-quantize llama-3.1-8b-f16.gguf llama-3.1-8b-q4_k_m.gguf Q4_K_M\n\n# 4. Test\n.\u002Fllama-cli -m llama-3.1-8b-q4_k_m.gguf -p \"Hello!\" -n 50\n```\n\n### Workflow 2: With importance matrix (better quality)\n\n```bash\n# 1. Convert to GGUF\npython convert_hf_to_gguf.py .\u002Fmodel --outfile model-f16.gguf\n\n# 2. Create calibration text (diverse samples)\ncat > calibration.txt \u003C\u003C 'EOF'\nThe quick brown fox jumps over the lazy dog.\nMachine learning is a subset of artificial intelligence.\nPython is a popular programming language.\n# Add more diverse text samples...\nEOF\n\n# 3. Generate importance matrix\n.\u002Fllama-imatrix -m model-f16.gguf \\\n    -f calibration.txt \\\n    --chunk 512 \\\n    -o model.imatrix \\\n    -ngl 35  # GPU layers if available\n\n# 4. Quantize with imatrix\n.\u002Fllama-quantize --imatrix model.imatrix \\\n    model-f16.gguf \\\n    model-q4_k_m.gguf \\\n    Q4_K_M\n```\n\n### Workflow 3: Multiple quantizations\n\n```bash\n#!\u002Fbin\u002Fbash\nMODEL=\"llama-3.1-8b-f16.gguf\"\nIMATRIX=\"llama-3.1-8b.imatrix\"\n\n# Generate imatrix once\n.\u002Fllama-imatrix -m $MODEL -f wiki.txt -o $IMATRIX -ngl 35\n\n# Create multiple quantizations\nfor QUANT in Q4_K_M Q5_K_M Q6_K Q8_0; do\n    OUTPUT=\"llama-3.1-8b-${QUANT,,}.gguf\"\n    .\u002Fllama-quantize --imatrix $IMATRIX $MODEL $OUTPUT $QUANT\n    echo \"Created: $OUTPUT ($(du -h $OUTPUT | cut -f1))\"\ndone\n```\n\n## Python usage\n\n### llama-cpp-python\n\n```python\nfrom llama_cpp import Llama\n\n# Load model\nllm = Llama(\n    model_path=\".\u002Fmodel-q4_k_m.gguf\",\n    n_ctx=4096,          # Context window\n    n_gpu_layers=35,     # GPU offload (0 for CPU only)\n    n_threads=8          # CPU threads\n)\n\n# Generate\noutput = llm(\n    \"What is machine learning?\",\n    max_tokens=256,\n    temperature=0.7,\n    stop=[\"\u003C\u002Fs>\", \"\\n\\n\"]\n)\nprint(output[\"choices\"][0][\"text\"])\n```\n\n### Chat completion\n\n```python\nfrom llama_cpp import Llama\n\nllm = Llama(\n    model_path=\".\u002Fmodel-q4_k_m.gguf\",\n    n_ctx=4096,\n    n_gpu_layers=35,\n    chat_format=\"llama-3\"  # Or \"chatml\", \"mistral\", etc.\n)\n\nmessages = [\n    {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n    {\"role\": \"user\", \"content\": \"What is Python?\"}\n]\n\nresponse = llm.create_chat_completion(\n    messages=messages,\n    max_tokens=256,\n    temperature=0.7\n)\nprint(response[\"choices\"][0][\"message\"][\"content\"])\n```\n\n### Streaming\n\n```python\nfrom llama_cpp import Llama\n\nllm = Llama(model_path=\".\u002Fmodel-q4_k_m.gguf\", n_gpu_layers=35)\n\n# Stream tokens\nfor chunk in llm(\n    \"Explain quantum computing:\",\n    max_tokens=256,\n    stream=True\n):\n    print(chunk[\"choices\"][0][\"text\"], end=\"\", flush=True)\n```\n\n## Server mode\n\n### Start OpenAI-compatible server\n\n```bash\n# Start server\n.\u002Fllama-server -m model-q4_k_m.gguf \\\n    --host 0.0.0.0 \\\n    --port 8080 \\\n    -ngl 35 \\\n    -c 4096\n\n# Or with Python bindings\npython -m llama_cpp.server \\\n    --model model-q4_k_m.gguf \\\n    --n_gpu_layers 35 \\\n    --host 0.0.0.0 \\\n    --port 8080\n```\n\n### Use with OpenAI client\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(\n    base_url=\"http:\u002F\u002Flocalhost:8080\u002Fv1\",\n    api_key=\"not-needed\"\n)\n\nresponse = client.chat.completions.create(\n    model=\"local-model\",\n    messages=[{\"role\": \"user\", \"content\": \"Hello!\"}],\n    max_tokens=256\n)\nprint(response.choices[0].message.content)\n```\n\n## Hardware optimization\n\n### Apple Silicon (Metal)\n\n```bash\n# Build with Metal\nmake clean && make GGML_METAL=1\n\n# Run with Metal acceleration\n.\u002Fllama-cli -m model.gguf -ngl 99 -p \"Hello\"\n\n# Python with Metal\nllm = Llama(\n    model_path=\"model.gguf\",\n    n_gpu_layers=99,     # Offload all layers\n    n_threads=1          # Metal handles parallelism\n)\n```\n\n### NVIDIA CUDA\n\n```bash\n# Build with CUDA\nmake clean && make GGML_CUDA=1\n\n# Run with CUDA\n.\u002Fllama-cli -m model.gguf -ngl 35 -p \"Hello\"\n\n# Specify GPU\nCUDA_VISIBLE_DEVICES=0 .\u002Fllama-cli -m model.gguf -ngl 35\n```\n\n### CPU optimization\n\n```bash\n# Build with AVX2\u002FAVX512\nmake clean && make\n\n# Run with optimal threads\n.\u002Fllama-cli -m model.gguf -t 8 -p \"Hello\"\n\n# Python CPU config\nllm = Llama(\n    model_path=\"model.gguf\",\n    n_gpu_layers=0,      # CPU only\n    n_threads=8,         # Match physical cores\n    n_batch=512          # Batch size for prompt processing\n)\n```\n\n## Integration with tools\n\n### Ollama\n\n```bash\n# Create Modelfile\ncat > Modelfile \u003C\u003C 'EOF'\nFROM .\u002Fmodel-q4_k_m.gguf\nTEMPLATE \"\"\"{{ .System }}\n{{ .Prompt }}\"\"\"\nPARAMETER temperature 0.7\nPARAMETER num_ctx 4096\nEOF\n\n# Create Ollama model\nollama create mymodel -f Modelfile\n\n# Run\nollama run mymodel \"Hello!\"\n```\n\n### LM Studio\n\n1. Place GGUF file in `~\u002F.cache\u002Flm-studio\u002Fmodels\u002F`\n2. Open LM Studio and select the model\n3. Configure context length and GPU offload\n4. Start inference\n\n### text-generation-webui\n\n```bash\n# Place in models folder\ncp model-q4_k_m.gguf text-generation-webui\u002Fmodels\u002F\n\n# Start with llama.cpp loader\npython server.py --model model-q4_k_m.gguf --loader llama.cpp --n-gpu-layers 35\n```\n\n## Best practices\n\n1. **Use K-quants**: Q4_K_M offers best quality\u002Fsize balance\n2. **Use imatrix**: Always use importance matrix for Q4 and below\n3. **GPU offload**: Offload as many layers as VRAM allows\n4. **Context length**: Start with 4096, increase if needed\n5. **Thread count**: Match physical CPU cores, not logical\n6. **Batch size**: Increase n_batch for faster prompt processing\n\n## Common issues\n\n**Model loads slowly:**\n```bash\n# Use mmap for faster loading\n.\u002Fllama-cli -m model.gguf --mmap\n```\n\n**Out of memory:**\n```bash\n# Reduce GPU layers\n.\u002Fllama-cli -m model.gguf -ngl 20  # Reduce from 35\n\n# Or use smaller quantization\n.\u002Fllama-quantize model-f16.gguf model-q3_k_m.gguf Q3_K_M\n```\n\n**Poor quality at low bits:**\n```bash\n# Always use imatrix for Q4 and below\n.\u002Fllama-imatrix -m model-f16.gguf -f calibration.txt -o model.imatrix\n.\u002Fllama-quantize --imatrix model.imatrix model-f16.gguf model-q4_k_m.gguf Q4_K_M\n```\n\n## References\n\n- **[Advanced Usage](references\u002Fadvanced-usage.md)** - Batching, speculative decoding, custom builds\n- **[Troubleshooting](references\u002Ftroubleshooting.md)** - Common issues, debugging, benchmarks\n\n## Resources\n\n- **Repository**: https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\n- **Python Bindings**: https:\u002F\u002Fgithub.com\u002Fabetlen\u002Fllama-cpp-python\n- **Pre-quantized Models**: https:\u002F\u002Fhuggingface.co\u002FTheBloke\n- **GGUF Converter**: https:\u002F\u002Fhuggingface.co\u002Fspaces\u002Fggml-org\u002Fgguf-my-repo\n- **License**: MIT\n\n## Agent operating procedure\n\n1. **Check the environment.** Check GPU type, memory and driver\u002FCUDA versions (`nvidia-smi`), framework versions, and dataset location and size.\n2. **Pin down the inputs.** Confirm formats, identifiers and parameters from the data or the user. Ask rather than guess any value that changes the result.\n3. **Run a small version first.** Do a smoke run: tiny model or subset, few steps, and confirm loss decreases and checkpoints save.\n4. **Execute the full task** using the instructions and references above.\n5. **Validate the result.** Track metrics on held-out data, compare against a baseline, and record seeds, configs and hardware.\n6. **Report.** State what was run (versions, commands, parameters), what was checked, and what is still uncertain.\n\n| If this happens | Do this |\n|---|---|\n| CUDA out-of-memory | Reduce batch size, enable gradient accumulation\u002Fcheckpointing or mixed precision, or shard the model. |\n| Loss is NaN or diverges | Lower the learning rate, check data for invalid values, and enable gradient clipping. |\n| A function, flag or endpoint in these instructions is missing in the installed version | Check the installed version's own documentation (`help()`, `--help`, official docs), adapt, and tell the user. Never invent an API. |\n| A required input, identifier or parameter is ambiguous | Ask the user, or state the assumption explicitly before running. |\n\n**Integrity rules**\n\n- Never fabricate results, parameters, identifiers, citations or statistics. If something cannot be run or verified, say so plainly.\n- Never claim training results without logs; estimate compute cost before launching large jobs.\n- Treat version-specific details here as possibly outdated: confirm them against the official documentation for the installed version.\n- Ask before actions that cost money, consume shared GPUs or cloud quota, touch personal or patient data, or cannot be undone.\n\n## Related skills\n\n- `llama-cpp`: Runs LLM inference on CPU, Apple Silicon, and consumer GPUs without NVIDIA hardware.\n- `hqq-quantization`: Half-Quadratic Quantization for LLMs without calibration data.\n- `awq-quantization`: Activation-aware weight quantization for 4-bit LLM compression with 3x speedup and minimal accuracy loss.\n",{"name":5,"license":15,"metadata":45,"description":7},{"tags":46,"version":47,"category":48,"maintainer":49,"dependencies":50},"GGUF, Quantization, llama.cpp, CPU Inference, Apple Silicon, Model Compression, Optimization","1.0.0","ml-training","Kalaris Labs","llama-cpp-python>=0.2.0","main","2b63f856ce19c2a797929cacabd9e0fc936812fa","plugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fgguf-quantization",491,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins","https:\u002F\u002Fhol.org\u002Fplugins\u002Fbest-codex-plugins","https:\u002F\u002Fgithub.com\u002Fhashgraph-online","https:\u002F\u002Fhol.org","Open-source standards and developer tools for interoperable AI agents: discovery, identity, communication, payments, privacy and security.","2026-11-14 14:06:13",{"source":8,"skillName":5,"commands":62,"prompt":72,"zipAvailable":73},[63,66,69],{"tool":64,"command":65},"npx","npx skills add hashgraph-online\u002Fawesome-codex-plugins --skill gguf-quantization",{"tool":67,"command":68},"bunx","bunx skills add hashgraph-online\u002Fawesome-codex-plugins --skill gguf-quantization",{"tool":70,"command":71},"pnpm","pnpm dlx skills add hashgraph-online\u002Fawesome-codex-plugins --skill gguf-quantization","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fgguf-quantization\u002FSKILL.md",true,[75,81],{"path":76,"resourceType":77,"resourceTypeDesc":78,"sizeBytes":79,"sourceHtmlUrl":80},"references\u002Fadvanced-usage.md","REFERENCE","参考文档",10887,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fgguf-quantization\u002Freferences\u002Fadvanced-usage.md",{"path":82,"resourceType":77,"resourceTypeDesc":78,"sizeBytes":83,"sourceHtmlUrl":84},"references\u002Ftroubleshooting.md",8904,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fgguf-quantization\u002Freferences\u002Ftroubleshooting.md",[86,95,103,111,119,127],{"skillNo":87,"name":88,"nameZh":6,"description":89,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":90,"sourceHtmlUrl":91,"categories":92,"keywords":93,"publishedAt":94},"01M7DV332YQMGD8WTRJ9DZQXTB","worktable-review-with-annotations","Create, inspect, reply to, update, or resolve situated Worktable annotations. Use when feedback belongs to a Doc section, block, text range, or HTML Doc, or when the user asks to address existing review feedback and coordinate a requested artifact edit.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-review-with-annotations\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-review-with-annotations\u002FSKILL.md",[],[],"2026-11-15 14:19:17",{"skillNo":96,"name":97,"nameZh":6,"description":98,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":99,"sourceHtmlUrl":100,"categories":101,"keywords":102,"publishedAt":94},"01M7DV2YGHCZVBEB1AHYB6CGHE","worktable-find-and-synthesize-context","Find, assess, and synthesize context from Worktable Docs and Records. Use when the user asks what a workspace contains, wants workspace-backed research, needs current facts reconciled across artifacts, or requests a cited synthesis without changing workspace state.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-find-and-synthesize-context\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-find-and-synthesize-context\u002FSKILL.md",[],[],{"skillNo":104,"name":105,"nameZh":6,"description":106,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":107,"sourceHtmlUrl":108,"categories":109,"keywords":110,"publishedAt":94},"01M7DV2TQN1H11ME2SAF9MXN7V","worktable-create-or-update-docs","Create, edit, rename, or maintain narrative Worktable Docs. Use for plans, decisions, briefs, notes, research syntheses, and other durable prose when the user requests a new artifact or a safe update to an existing one.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-update-docs\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-update-docs\u002FSKILL.md",[],[],{"skillNo":112,"name":113,"nameZh":6,"description":114,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":115,"sourceHtmlUrl":116,"categories":117,"keywords":118,"publishedAt":94},"01M7DV2NE452NEA8TXATKSNAJH","worktable-create-or-manage-records","Organize repeated structured information as durable Worktable Records. Use for research datasets, inventories, directories, pipelines, trackers, and other collections whose items need identity, fields, filtering, links, validation, or individual updates.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-manage-records\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-manage-records\u002FSKILL.md",[],[],{"skillNo":120,"name":121,"nameZh":6,"description":122,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":123,"sourceHtmlUrl":124,"categories":125,"keywords":126,"publishedAt":94},"01M7DV2HENRBR2KSCVM064WH0Y","worktable-create-or-manage-interactive-html","Create or improve durable visual and interactive artifacts in Worktable. Use for dashboards, trackers, calculators, mini apps, slide presentations, and other requests that benefit from a polished HTML Doc.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-manage-interactive-html\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-manage-interactive-html\u002FSKILL.md",[],[],{"skillNo":128,"name":129,"nameZh":6,"description":130,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":131,"sourceHtmlUrl":132,"categories":133,"keywords":134,"publishedAt":94},"01M7DV2CADR5R7RQFFYPNAGR9Y","worktable-collaborate-threads","Read, answer, and hand off messages through durable Worktable threads with users or other authenticated participants. Use when the user asks to check addressed messages, reply to a participant, continue a thread, request another agent's help, wait briefly for a response, or watch for comments and messages sent to the agent.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-collaborate-threads\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-collaborate-threads\u002FSKILL.md",[],[]]