[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6QEV26WBCN4JY3KQ2SFXABR":3,"skill-same:01M6QEV26WBCN4JY3KQ2SFXABR":83},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":39,"linkOnly":40,"markdownContent":41,"markdownBody":42,"frontMatter":43,"refName":50,"commitSha":51,"skillDirectory":52,"forksCount":14,"repositoryHtmlUrl":53,"repositoryHomepageUrl":54,"ownerHtmlUrl":55,"ownerBlogUrl":56,"ownerDescription":57,"updatedAt":58,"install":59,"resources":72},"01M6QEV26WBCN4JY3KQ2SFXABR","hqq-quantization",null,"Half-Quadratic Quantization for LLMs without calibration data. Use when quantizing models to 4\u002F3\u002F2-bit precision without needing calibration datasets, for fast quantization workflows, or when deploying with vLLM or HuggingFace Transformers.","Tashima-Tarsh\u002FDisha6.6","Tashima-Tarsh","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F269906498?v=4","01M6PBH98X03XERH0HFT4PFB1P","NITISH KUMAR","01M6PBH9DA0M6KM3Z12P848F0K",0,"MIT","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fkal-hqq-quantization\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38],"hqq quantization","HQQ","Quantization","use","calibration","for","Model Compression","Memory Efficiency","bit","efficiency","llms","model","Transformers","vLLM","data","Optimization","training","ml","workflows","memory","2026-11-10 10:12:30",false,"---\nname: hqq-quantization\ndescription: Half-Quadratic Quantization for LLMs without calibration data. Use when quantizing models to 4\u002F3\u002F2-bit precision without needing calibration datasets, for fast quantization workflows, or when deploying with vLLM or HuggingFace Transformers.\nlicense: MIT\nmetadata:\n  version: 1.0.0\n  category: ml-training\n  maintainer: Kalaris Labs\n  tags: Quantization, HQQ, Optimization, Memory Efficiency, Inference, Model Compression\n  dependencies: hqq>=0.2.0, torch>=2.0.0\n---\n\n# HQQ - Half-Quadratic Quantization\n\nFast, calibration-free weight quantization supporting 8\u002F4\u002F3\u002F2\u002F1-bit precision with multiple optimized backends.\n\n## When to use HQQ\n\n**Use HQQ when:**\n- Quantizing models without calibration data (no dataset needed)\n- Need fast quantization (minutes vs hours for GPTQ\u002FAWQ)\n- Deploying with vLLM or HuggingFace Transformers\n- Fine-tuning quantized models with LoRA\u002FPEFT\n- Experimenting with extreme quantization (2-bit, 1-bit)\n\n**Key advantages:**\n- **No calibration**: Quantize any model instantly without sample data\n- **Multiple backends**: PyTorch, ATEN, TorchAO, Marlin, BitBlas for optimized inference\n- **Flexible precision**: 8\u002F4\u002F3\u002F2\u002F1-bit with configurable group sizes\n- **Framework integration**: Native HuggingFace and vLLM support\n- **PEFT compatible**: Fine-tune quantized models with LoRA\n\n**Use alternatives instead:**\n- **AWQ**: Need calibration-based accuracy, production serving\n- **GPTQ**: Maximum accuracy with calibration data available\n- **bitsandbytes**: Simple 8-bit\u002F4-bit without custom backends\n- **llama.cpp\u002FGGUF**: CPU inference, Apple Silicon deployment\n\n## Quick start\n\n### Installation\n\n```bash\npip install hqq\n\n# With specific backend\npip install hqq[torch]      # PyTorch backend\npip install hqq[torchao]    # TorchAO int4 backend\npip install hqq[bitblas]    # BitBlas backend\npip install hqq[marlin]     # Marlin backend\n```\n\n### Basic quantization\n\n```python\nfrom hqq.core.quantize import BaseQuantizeConfig, HQQLinear\nimport torch.nn as nn\n\n# Configure quantization\nconfig = BaseQuantizeConfig(\n    nbits=4,           # 4-bit quantization\n    group_size=64,     # Group size for quantization\n    axis=1             # Quantize along output dimension\n)\n\n# Quantize a linear layer\nlinear = nn.Linear(4096, 4096)\nhqq_linear = HQQLinear(linear, config)\n\n# Use normally\noutput = hqq_linear(input_tensor)\n```\n\n### Quantize full model with HuggingFace\n\n```python\nfrom transformers import AutoModelForCausalLM, HqqConfig\n\n# Configure HQQ\nquantization_config = HqqConfig(\n    nbits=4,\n    group_size=64,\n    axis=1\n)\n\n# Load and quantize\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=quantization_config,\n    device_map=\"auto\"\n)\n\n# Model is quantized and ready to use\n```\n\n## Core concepts\n\n### Quantization configuration\n\nHQQ uses `BaseQuantizeConfig` to define quantization parameters:\n\n```python\nfrom hqq.core.quantize import BaseQuantizeConfig\n\n# Standard 4-bit config\nconfig_4bit = BaseQuantizeConfig(\n    nbits=4,           # Bits per weight (1-8)\n    group_size=64,     # Weights per quantization group\n    axis=1             # 0=input dim, 1=output dim\n)\n\n# Aggressive 2-bit config\nconfig_2bit = BaseQuantizeConfig(\n    nbits=2,\n    group_size=16,     # Smaller groups for low-bit\n    axis=1\n)\n\n# Mixed precision per layer type\nlayer_configs = {\n    \"self_attn.q_proj\": BaseQuantizeConfig(nbits=4, group_size=64),\n    \"self_attn.k_proj\": BaseQuantizeConfig(nbits=4, group_size=64),\n    \"self_attn.v_proj\": BaseQuantizeConfig(nbits=4, group_size=64),\n    \"mlp.gate_proj\": BaseQuantizeConfig(nbits=2, group_size=32),\n    \"mlp.up_proj\": BaseQuantizeConfig(nbits=2, group_size=32),\n    \"mlp.down_proj\": BaseQuantizeConfig(nbits=4, group_size=64),\n}\n```\n\n### HQQLinear layer\n\nThe core quantized layer that replaces `nn.Linear`:\n\n```python\nfrom hqq.core.quantize import HQQLinear\nimport torch\n\n# Create quantized layer\nlinear = torch.nn.Linear(4096, 4096)\nhqq_layer = HQQLinear(linear, config)\n\n# Access quantized weights\nW_q = hqq_layer.W_q           # Quantized weights\nscale = hqq_layer.scale       # Scale factors\nzero = hqq_layer.zero         # Zero points\n\n# Dequantize for inspection\nW_dequant = hqq_layer.dequantize()\n```\n\n### Backends\n\nHQQ supports multiple inference backends for different hardware:\n\n```python\nfrom hqq.core.quantize import HQQLinear\n\n# Available backends\nbackends = [\n    \"pytorch\",          # Pure PyTorch (default)\n    \"pytorch_compile\",  # torch.compile optimized\n    \"aten\",            # Custom CUDA kernels\n    \"torchao_int4\",    # TorchAO int4 matmul\n    \"gemlite\",         # GemLite CUDA kernels\n    \"bitblas\",         # BitBlas optimized\n    \"marlin\",          # Marlin 4-bit kernels\n]\n\n# Set backend globally\nHQQLinear.set_backend(\"torchao_int4\")\n\n# Or per layer\nhqq_layer.set_backend(\"marlin\")\n```\n\n**Backend selection guide:**\n| Backend | Best For | Requirements |\n|---------|----------|--------------|\n| pytorch | Compatibility | Any GPU |\n| pytorch_compile | Moderate speedup | torch>=2.0 |\n| aten | Good balance | CUDA GPU |\n| torchao_int4 | 4-bit inference | torchao installed |\n| marlin | Maximum 4-bit speed | Ampere+ GPU |\n| bitblas | Flexible bit-widths | bitblas installed |\n\n## HuggingFace integration\n\n### Load pre-quantized models\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n# Load HQQ-quantized model from Hub\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"mobiuslabsgmbh\u002FLlama-3.1-8B-HQQ-4bit\",\n    device_map=\"auto\"\n)\ntokenizer = AutoTokenizer.from_pretrained(\"meta-llama\u002FLlama-3.1-8B\")\n\n# Use normally\ninputs = tokenizer(\"Hello, world!\", return_tensors=\"pt\").to(model.device)\noutputs = model.generate(**inputs, max_new_tokens=50)\n```\n\n### Quantize and save\n\n```python\nfrom transformers import AutoModelForCausalLM, HqqConfig\n\n# Quantize\nconfig = HqqConfig(nbits=4, group_size=64)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=config,\n    device_map=\"auto\"\n)\n\n# Save quantized model\nmodel.save_pretrained(\".\u002Fllama-8b-hqq-4bit\")\n\n# Push to Hub\nmodel.push_to_hub(\"my-org\u002FLlama-3.1-8B-HQQ-4bit\")\n```\n\n### Mixed precision quantization\n\n```python\nfrom transformers import AutoModelForCausalLM, HqqConfig\n\n# Different precision per layer type\nconfig = HqqConfig(\n    nbits=4,\n    group_size=64,\n    # Attention layers: higher precision\n    # MLP layers: lower precision for memory savings\n    dynamic_config={\n        \"attn\": {\"nbits\": 4, \"group_size\": 64},\n        \"mlp\": {\"nbits\": 2, \"group_size\": 32}\n    }\n)\n```\n\n## vLLM integration\n\n### Serve HQQ models with vLLM\n\n```python\nfrom vllm import LLM, SamplingParams\n\n# Load HQQ-quantized model\nllm = LLM(\n    model=\"mobiuslabsgmbh\u002FLlama-3.1-8B-HQQ-4bit\",\n    quantization=\"hqq\",\n    dtype=\"float16\"\n)\n\n# Generate\nsampling_params = SamplingParams(temperature=0.7, max_tokens=100)\noutputs = llm.generate([\"What is machine learning?\"], sampling_params)\n```\n\n### vLLM with custom HQQ config\n\n```python\nfrom vllm import LLM\n\nllm = LLM(\n    model=\"meta-llama\u002FLlama-3.1-8B\",\n    quantization=\"hqq\",\n    quantization_config={\n        \"nbits\": 4,\n        \"group_size\": 64\n    }\n)\n```\n\n## PEFT\u002FLoRA fine-tuning\n\n### Fine-tune quantized models\n\n```python\nfrom transformers import AutoModelForCausalLM, HqqConfig\nfrom peft import LoraConfig, get_peft_model\n\n# Load quantized model\nquant_config = HqqConfig(nbits=4, group_size=64)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=quant_config,\n    device_map=\"auto\"\n)\n\n# Apply LoRA\nlora_config = LoraConfig(\n    r=16,\n    lora_alpha=32,\n    target_modules=[\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\"],\n    lora_dropout=0.05,\n    bias=\"none\",\n    task_type=\"CAUSAL_LM\"\n)\n\nmodel = get_peft_model(model, lora_config)\n\n# Train normally with Trainer or custom loop\n```\n\n### QLoRA-style training\n\n```python\nfrom transformers import TrainingArguments, Trainer\n\ntraining_args = TrainingArguments(\n    output_dir=\".\u002Fhqq-lora-output\",\n    per_device_train_batch_size=4,\n    gradient_accumulation_steps=4,\n    learning_rate=2e-4,\n    num_train_epochs=3,\n    fp16=True,\n    logging_steps=10,\n    save_strategy=\"epoch\"\n)\n\ntrainer = Trainer(\n    model=model,\n    args=training_args,\n    train_dataset=train_dataset,\n    data_collator=data_collator\n)\n\ntrainer.train()\n```\n\n## Quantization workflows\n\n### Workflow 1: Quick model compression\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, HqqConfig\n\n# 1. Configure quantization\nconfig = HqqConfig(nbits=4, group_size=64)\n\n# 2. Load and quantize (no calibration needed!)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=config,\n    device_map=\"auto\"\n)\ntokenizer = AutoTokenizer.from_pretrained(\"meta-llama\u002FLlama-3.1-8B\")\n\n# 3. Verify quality\nprompt = \"The capital of France is\"\ninputs = tokenizer(prompt, return_tensors=\"pt\").to(model.device)\noutputs = model.generate(**inputs, max_new_tokens=20)\nprint(tokenizer.decode(outputs[0]))\n\n# 4. Save\nmodel.save_pretrained(\".\u002Fllama-8b-hqq\")\ntokenizer.save_pretrained(\".\u002Fllama-8b-hqq\")\n```\n\n### Workflow 2: Optimize for inference speed\n\n```python\nfrom hqq.core.quantize import HQQLinear\nfrom transformers import AutoModelForCausalLM, HqqConfig\n\n# 1. Quantize with optimal backend\nconfig = HqqConfig(nbits=4, group_size=64)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=config,\n    device_map=\"auto\"\n)\n\n# 2. Set fast backend\nHQQLinear.set_backend(\"marlin\")  # or \"torchao_int4\"\n\n# 3. Compile for additional speedup\nimport torch\nmodel = torch.compile(model)\n\n# 4. Benchmark\nimport time\ninputs = tokenizer(\"Hello\", return_tensors=\"pt\").to(model.device)\nstart = time.time()\nfor _ in range(10):\n    model.generate(**inputs, max_new_tokens=100)\nprint(f\"Avg time: {(time.time() - start) \u002F 10:.2f}s\")\n```\n\n## Best practices\n\n1. **Start with 4-bit**: Best quality\u002Fsize tradeoff for most models\n2. **Use group_size=64**: Good balance; smaller for extreme quantization\n3. **Choose backend wisely**: Marlin for 4-bit Ampere+, TorchAO for flexibility\n4. **Verify quality**: Always test generation quality after quantization\n5. **Mixed precision**: Keep attention at higher precision, compress MLP more\n6. **PEFT training**: Use LoRA r=16-32 for good fine-tuning results\n\n## Common issues\n\n**Out of memory during quantization:**\n```python\n# Quantize layer-by-layer\nfrom hqq.models.hf.base import AutoHQQHFModel\n\nmodel = AutoHQQHFModel.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=config,\n    device_map=\"sequential\"  # Load layers sequentially\n)\n```\n\n**Slow inference:**\n```python\n# Switch to optimized backend\nfrom hqq.core.quantize import HQQLinear\nHQQLinear.set_backend(\"marlin\")  # Requires Ampere+ GPU\n\n# Or compile\nmodel = torch.compile(model, mode=\"reduce-overhead\")\n```\n\n**Poor quality at 2-bit:**\n```python\n# Use smaller group size\nconfig = BaseQuantizeConfig(\n    nbits=2,\n    group_size=16,  # Smaller groups help at low bits\n    axis=1\n)\n```\n\n## References\n\n- **[Advanced Usage](references\u002Fadvanced-usage.md)** - Custom backends, mixed precision, optimization\n- **[Troubleshooting](references\u002Ftroubleshooting.md)** - Common issues, debugging, benchmarks\n\n## Resources\n\n- **Repository**: https:\u002F\u002Fgithub.com\u002Fmobiusml\u002Fhqq\n- **Paper**: Half-Quadratic Quantization\n- **HuggingFace Models**: https:\u002F\u002Fhuggingface.co\u002Fmobiuslabsgmbh\n- **Version**: 0.2.0+\n- **License**: Apache 2.0\n\n## Related skills\n\n- `gguf-quantization`: GGUF format and llama.cpp quantization for efficient CPU\u002FGPU inference.\n- `gptq`: Quantizes LLMs to 4-bit (also 3-bit) with GPTQ using group-wise quantization (group size 128 by default), via AutoGPTQ and transformers.\n- `quantizing-models-bitsandbytes`: Quantizes LLMs to 8-bit or 4-bit for 50-75% memory reduction with minimal accuracy loss.\n","# HQQ - Half-Quadratic Quantization\n\nFast, calibration-free weight quantization supporting 8\u002F4\u002F3\u002F2\u002F1-bit precision with multiple optimized backends.\n\n## When to use HQQ\n\n**Use HQQ when:**\n- Quantizing models without calibration data (no dataset needed)\n- Need fast quantization (minutes vs hours for GPTQ\u002FAWQ)\n- Deploying with vLLM or HuggingFace Transformers\n- Fine-tuning quantized models with LoRA\u002FPEFT\n- Experimenting with extreme quantization (2-bit, 1-bit)\n\n**Key advantages:**\n- **No calibration**: Quantize any model instantly without sample data\n- **Multiple backends**: PyTorch, ATEN, TorchAO, Marlin, BitBlas for optimized inference\n- **Flexible precision**: 8\u002F4\u002F3\u002F2\u002F1-bit with configurable group sizes\n- **Framework integration**: Native HuggingFace and vLLM support\n- **PEFT compatible**: Fine-tune quantized models with LoRA\n\n**Use alternatives instead:**\n- **AWQ**: Need calibration-based accuracy, production serving\n- **GPTQ**: Maximum accuracy with calibration data available\n- **bitsandbytes**: Simple 8-bit\u002F4-bit without custom backends\n- **llama.cpp\u002FGGUF**: CPU inference, Apple Silicon deployment\n\n## Quick start\n\n### Installation\n\n```bash\npip install hqq\n\n# With specific backend\npip install hqq[torch]      # PyTorch backend\npip install hqq[torchao]    # TorchAO int4 backend\npip install hqq[bitblas]    # BitBlas backend\npip install hqq[marlin]     # Marlin backend\n```\n\n### Basic quantization\n\n```python\nfrom hqq.core.quantize import BaseQuantizeConfig, HQQLinear\nimport torch.nn as nn\n\n# Configure quantization\nconfig = BaseQuantizeConfig(\n    nbits=4,           # 4-bit quantization\n    group_size=64,     # Group size for quantization\n    axis=1             # Quantize along output dimension\n)\n\n# Quantize a linear layer\nlinear = nn.Linear(4096, 4096)\nhqq_linear = HQQLinear(linear, config)\n\n# Use normally\noutput = hqq_linear(input_tensor)\n```\n\n### Quantize full model with HuggingFace\n\n```python\nfrom transformers import AutoModelForCausalLM, HqqConfig\n\n# Configure HQQ\nquantization_config = HqqConfig(\n    nbits=4,\n    group_size=64,\n    axis=1\n)\n\n# Load and quantize\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=quantization_config,\n    device_map=\"auto\"\n)\n\n# Model is quantized and ready to use\n```\n\n## Core concepts\n\n### Quantization configuration\n\nHQQ uses `BaseQuantizeConfig` to define quantization parameters:\n\n```python\nfrom hqq.core.quantize import BaseQuantizeConfig\n\n# Standard 4-bit config\nconfig_4bit = BaseQuantizeConfig(\n    nbits=4,           # Bits per weight (1-8)\n    group_size=64,     # Weights per quantization group\n    axis=1             # 0=input dim, 1=output dim\n)\n\n# Aggressive 2-bit config\nconfig_2bit = BaseQuantizeConfig(\n    nbits=2,\n    group_size=16,     # Smaller groups for low-bit\n    axis=1\n)\n\n# Mixed precision per layer type\nlayer_configs = {\n    \"self_attn.q_proj\": BaseQuantizeConfig(nbits=4, group_size=64),\n    \"self_attn.k_proj\": BaseQuantizeConfig(nbits=4, group_size=64),\n    \"self_attn.v_proj\": BaseQuantizeConfig(nbits=4, group_size=64),\n    \"mlp.gate_proj\": BaseQuantizeConfig(nbits=2, group_size=32),\n    \"mlp.up_proj\": BaseQuantizeConfig(nbits=2, group_size=32),\n    \"mlp.down_proj\": BaseQuantizeConfig(nbits=4, group_size=64),\n}\n```\n\n### HQQLinear layer\n\nThe core quantized layer that replaces `nn.Linear`:\n\n```python\nfrom hqq.core.quantize import HQQLinear\nimport torch\n\n# Create quantized layer\nlinear = torch.nn.Linear(4096, 4096)\nhqq_layer = HQQLinear(linear, config)\n\n# Access quantized weights\nW_q = hqq_layer.W_q           # Quantized weights\nscale = hqq_layer.scale       # Scale factors\nzero = hqq_layer.zero         # Zero points\n\n# Dequantize for inspection\nW_dequant = hqq_layer.dequantize()\n```\n\n### Backends\n\nHQQ supports multiple inference backends for different hardware:\n\n```python\nfrom hqq.core.quantize import HQQLinear\n\n# Available backends\nbackends = [\n    \"pytorch\",          # Pure PyTorch (default)\n    \"pytorch_compile\",  # torch.compile optimized\n    \"aten\",            # Custom CUDA kernels\n    \"torchao_int4\",    # TorchAO int4 matmul\n    \"gemlite\",         # GemLite CUDA kernels\n    \"bitblas\",         # BitBlas optimized\n    \"marlin\",          # Marlin 4-bit kernels\n]\n\n# Set backend globally\nHQQLinear.set_backend(\"torchao_int4\")\n\n# Or per layer\nhqq_layer.set_backend(\"marlin\")\n```\n\n**Backend selection guide:**\n| Backend | Best For | Requirements |\n|---------|----------|--------------|\n| pytorch | Compatibility | Any GPU |\n| pytorch_compile | Moderate speedup | torch>=2.0 |\n| aten | Good balance | CUDA GPU |\n| torchao_int4 | 4-bit inference | torchao installed |\n| marlin | Maximum 4-bit speed | Ampere+ GPU |\n| bitblas | Flexible bit-widths | bitblas installed |\n\n## HuggingFace integration\n\n### Load pre-quantized models\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n# Load HQQ-quantized model from Hub\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"mobiuslabsgmbh\u002FLlama-3.1-8B-HQQ-4bit\",\n    device_map=\"auto\"\n)\ntokenizer = AutoTokenizer.from_pretrained(\"meta-llama\u002FLlama-3.1-8B\")\n\n# Use normally\ninputs = tokenizer(\"Hello, world!\", return_tensors=\"pt\").to(model.device)\noutputs = model.generate(**inputs, max_new_tokens=50)\n```\n\n### Quantize and save\n\n```python\nfrom transformers import AutoModelForCausalLM, HqqConfig\n\n# Quantize\nconfig = HqqConfig(nbits=4, group_size=64)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=config,\n    device_map=\"auto\"\n)\n\n# Save quantized model\nmodel.save_pretrained(\".\u002Fllama-8b-hqq-4bit\")\n\n# Push to Hub\nmodel.push_to_hub(\"my-org\u002FLlama-3.1-8B-HQQ-4bit\")\n```\n\n### Mixed precision quantization\n\n```python\nfrom transformers import AutoModelForCausalLM, HqqConfig\n\n# Different precision per layer type\nconfig = HqqConfig(\n    nbits=4,\n    group_size=64,\n    # Attention layers: higher precision\n    # MLP layers: lower precision for memory savings\n    dynamic_config={\n        \"attn\": {\"nbits\": 4, \"group_size\": 64},\n        \"mlp\": {\"nbits\": 2, \"group_size\": 32}\n    }\n)\n```\n\n## vLLM integration\n\n### Serve HQQ models with vLLM\n\n```python\nfrom vllm import LLM, SamplingParams\n\n# Load HQQ-quantized model\nllm = LLM(\n    model=\"mobiuslabsgmbh\u002FLlama-3.1-8B-HQQ-4bit\",\n    quantization=\"hqq\",\n    dtype=\"float16\"\n)\n\n# Generate\nsampling_params = SamplingParams(temperature=0.7, max_tokens=100)\noutputs = llm.generate([\"What is machine learning?\"], sampling_params)\n```\n\n### vLLM with custom HQQ config\n\n```python\nfrom vllm import LLM\n\nllm = LLM(\n    model=\"meta-llama\u002FLlama-3.1-8B\",\n    quantization=\"hqq\",\n    quantization_config={\n        \"nbits\": 4,\n        \"group_size\": 64\n    }\n)\n```\n\n## PEFT\u002FLoRA fine-tuning\n\n### Fine-tune quantized models\n\n```python\nfrom transformers import AutoModelForCausalLM, HqqConfig\nfrom peft import LoraConfig, get_peft_model\n\n# Load quantized model\nquant_config = HqqConfig(nbits=4, group_size=64)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=quant_config,\n    device_map=\"auto\"\n)\n\n# Apply LoRA\nlora_config = LoraConfig(\n    r=16,\n    lora_alpha=32,\n    target_modules=[\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\"],\n    lora_dropout=0.05,\n    bias=\"none\",\n    task_type=\"CAUSAL_LM\"\n)\n\nmodel = get_peft_model(model, lora_config)\n\n# Train normally with Trainer or custom loop\n```\n\n### QLoRA-style training\n\n```python\nfrom transformers import TrainingArguments, Trainer\n\ntraining_args = TrainingArguments(\n    output_dir=\".\u002Fhqq-lora-output\",\n    per_device_train_batch_size=4,\n    gradient_accumulation_steps=4,\n    learning_rate=2e-4,\n    num_train_epochs=3,\n    fp16=True,\n    logging_steps=10,\n    save_strategy=\"epoch\"\n)\n\ntrainer = Trainer(\n    model=model,\n    args=training_args,\n    train_dataset=train_dataset,\n    data_collator=data_collator\n)\n\ntrainer.train()\n```\n\n## Quantization workflows\n\n### Workflow 1: Quick model compression\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, HqqConfig\n\n# 1. Configure quantization\nconfig = HqqConfig(nbits=4, group_size=64)\n\n# 2. Load and quantize (no calibration needed!)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=config,\n    device_map=\"auto\"\n)\ntokenizer = AutoTokenizer.from_pretrained(\"meta-llama\u002FLlama-3.1-8B\")\n\n# 3. Verify quality\nprompt = \"The capital of France is\"\ninputs = tokenizer(prompt, return_tensors=\"pt\").to(model.device)\noutputs = model.generate(**inputs, max_new_tokens=20)\nprint(tokenizer.decode(outputs[0]))\n\n# 4. Save\nmodel.save_pretrained(\".\u002Fllama-8b-hqq\")\ntokenizer.save_pretrained(\".\u002Fllama-8b-hqq\")\n```\n\n### Workflow 2: Optimize for inference speed\n\n```python\nfrom hqq.core.quantize import HQQLinear\nfrom transformers import AutoModelForCausalLM, HqqConfig\n\n# 1. Quantize with optimal backend\nconfig = HqqConfig(nbits=4, group_size=64)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=config,\n    device_map=\"auto\"\n)\n\n# 2. Set fast backend\nHQQLinear.set_backend(\"marlin\")  # or \"torchao_int4\"\n\n# 3. Compile for additional speedup\nimport torch\nmodel = torch.compile(model)\n\n# 4. Benchmark\nimport time\ninputs = tokenizer(\"Hello\", return_tensors=\"pt\").to(model.device)\nstart = time.time()\nfor _ in range(10):\n    model.generate(**inputs, max_new_tokens=100)\nprint(f\"Avg time: {(time.time() - start) \u002F 10:.2f}s\")\n```\n\n## Best practices\n\n1. **Start with 4-bit**: Best quality\u002Fsize tradeoff for most models\n2. **Use group_size=64**: Good balance; smaller for extreme quantization\n3. **Choose backend wisely**: Marlin for 4-bit Ampere+, TorchAO for flexibility\n4. **Verify quality**: Always test generation quality after quantization\n5. **Mixed precision**: Keep attention at higher precision, compress MLP more\n6. **PEFT training**: Use LoRA r=16-32 for good fine-tuning results\n\n## Common issues\n\n**Out of memory during quantization:**\n```python\n# Quantize layer-by-layer\nfrom hqq.models.hf.base import AutoHQQHFModel\n\nmodel = AutoHQQHFModel.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=config,\n    device_map=\"sequential\"  # Load layers sequentially\n)\n```\n\n**Slow inference:**\n```python\n# Switch to optimized backend\nfrom hqq.core.quantize import HQQLinear\nHQQLinear.set_backend(\"marlin\")  # Requires Ampere+ GPU\n\n# Or compile\nmodel = torch.compile(model, mode=\"reduce-overhead\")\n```\n\n**Poor quality at 2-bit:**\n```python\n# Use smaller group size\nconfig = BaseQuantizeConfig(\n    nbits=2,\n    group_size=16,  # Smaller groups help at low bits\n    axis=1\n)\n```\n\n## References\n\n- **[Advanced Usage](references\u002Fadvanced-usage.md)** - Custom backends, mixed precision, optimization\n- **[Troubleshooting](references\u002Ftroubleshooting.md)** - Common issues, debugging, benchmarks\n\n## Resources\n\n- **Repository**: https:\u002F\u002Fgithub.com\u002Fmobiusml\u002Fhqq\n- **Paper**: Half-Quadratic Quantization\n- **HuggingFace Models**: https:\u002F\u002Fhuggingface.co\u002Fmobiuslabsgmbh\n- **Version**: 0.2.0+\n- **License**: Apache 2.0\n\n## Related skills\n\n- `gguf-quantization`: GGUF format and llama.cpp quantization for efficient CPU\u002FGPU inference.\n- `gptq`: Quantizes LLMs to 4-bit (also 3-bit) with GPTQ using group-wise quantization (group size 128 by default), via AutoGPTQ and transformers.\n- `quantizing-models-bitsandbytes`: Quantizes LLMs to 8-bit or 4-bit for 50-75% memory reduction with minimal accuracy loss.\n",{"name":5,"license":15,"metadata":44,"description":7},{"tags":45,"version":46,"category":47,"maintainer":48,"dependencies":49},"Quantization, HQQ, Optimization, Memory Efficiency, Inference, Model Compression","1.0.0","ml-training","Kalaris Labs","hqq>=0.2.0, torch>=2.0.0","main","0da4485785ee599388900be50db1635489d53056",".agents\u002Fskills\u002Fkal-hqq-quantization","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6","https:\u002F\u002Fdisha6.6.thenitishkr.in\u002F","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh","www.thenitishkr","thenitishkr | AI • Cyber-security • Research\r\n","2026-11-08 03:33:25",{"source":8,"skillName":5,"commands":60,"prompt":70,"zipAvailable":71},[61,64,67],{"tool":62,"command":63},"npx","npx skills add Tashima-Tarsh\u002FDisha6.6 --skill hqq-quantization",{"tool":65,"command":66},"bunx","bunx skills add Tashima-Tarsh\u002FDisha6.6 --skill hqq-quantization",{"tool":68,"command":69},"pnpm","pnpm dlx skills add Tashima-Tarsh\u002FDisha6.6 --skill hqq-quantization","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fkal-hqq-quantization\u002FSKILL.md",true,[73,79],{"path":74,"resourceType":75,"resourceTypeDesc":76,"sizeBytes":77,"sourceHtmlUrl":78},"references\u002Fadvanced-usage.md","REFERENCE","参考文档",14077,"https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fkal-hqq-quantization\u002Freferences\u002Fadvanced-usage.md",{"path":80,"resourceType":75,"resourceTypeDesc":76,"sizeBytes":81,"sourceHtmlUrl":82},"references\u002Ftroubleshooting.md",11086,"https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fkal-hqq-quantization\u002Freferences\u002Ftroubleshooting.md",[84,114,138,165,190,213],{"skillNo":85,"name":86,"nameZh":6,"description":87,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":88,"categories":89,"keywords":95,"publishedAt":39},"01M6QRD07D4TGNDK6VJ4EHF8BV","youtube-full","Use when the user needs YouTube transcripts, video search, channel browsing, playlist extraction, or content monitoring. Trigger phrases: 'get the transcript for', 'search YouTube for', 'what are the latest videos on', 'list this playlist', 'monitor this channel', or any request involving a YouTube URL, video ID, or @handle. Do NOT use for downloading video or audio files, YouTube engagement data (likes, comments), or private\u002Fage-restricted videos.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fyoutube-full\u002FSKILL.md",[90],{"categoryNo":91,"categoryName":92,"taxonomyType":93,"taxonomyTypeDesc":94},"01JBSEEDCAT000000000000016","图像与多媒体","SCENARIO","使用场景",[96,97,98,99,100,101,33,102,103,104,105,106,107,108,109,110,24,111,112,113],"youtube full","youtube","full","transcript","search","monitoring","Video","Audio","list","files","do","get","age","trigger","URL","browsing","request","youtube transcripts",{"skillNo":115,"name":116,"nameZh":6,"description":117,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":118,"categories":119,"keywords":120,"publishedAt":39},"01M6QRCJ3112VYPA97YG7KGMC2","x-twitter-growth","X\u002FTwitter growth engine for building audience, crafting viral content, and analyzing engagement. Use when the user wants to grow on X\u002FTwitter, write tweets or threads, analyze their X profile, research competitors on X, plan a posting strategy, or optimize engagement. Complements social-content (generic multi-platform) with X-specific depth: algorithm mechanics, thread engineering, reply strategy, profile optimization, and competitive intelligence via web search.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fx-twitter-growth\u002FSKILL.md",[],[121,122,123,124,100,125,126,127,128,129,130,131,132,133,22,134,135,136,137,24],"twitter","x twitter growth","growth","research","content","engineering","plan","web search","competitors","analyze","engine","write","strategy","platform","grow","posting","competitive",{"skillNo":139,"name":140,"nameZh":6,"description":141,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":142,"categories":143,"keywords":147,"publishedAt":39},"01M6QRBN3EHQNJY42RERP7A149","write-a-skill","Create new agent skills with proper structure, progressive disclosure, and bundled resources. Use when user wants to create, write, build, or author a new skill.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fwrite-a-skill\u002FSKILL.md",[144],{"categoryNo":145,"categoryName":146,"taxonomyType":93,"taxonomyTypeDesc":94},"01JBSEEDCAT000000000000023","Skill与智能体开发",[148,132,149,150,151,152,153,154,22,155,156,50,157,158,159,160,161,162,163,164],"write a skill","github","create","https","productivity","matt pocock","direct","example","build","new skill","structure","new agent","resources","with","tree","progressive disclosure","author",{"skillNo":166,"name":167,"nameZh":6,"description":168,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":169,"categories":170,"keywords":171,"publishedAt":39},"01M6QRB6G8P33PNGAF9Y69EJ9T","wrap-up","Close out a launched Claude Managed Agent — recap every primitive the founder now owns, regenerate the single-file overview page, and suggest the next 1-2 upgrades. Use when the user says \"wrap up\", \"close this out\", \"what do I own now\", \"give me the summary\", \"recap the agent\", or when the orchestrator routes phase=wrap-up. primitives_inventory.py tables everything owned (agent, environment, session, memory, outcome, deployment); overview_page.py regenerates a self-contained .\u002Fmy-agent\u002Fagent-overview.html; upgrade_suggester.py ranks the next moves from recorded deferrals plus standing hardening steps. Companion to run-without-you; the last stop before phase=done.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fwrap-up\u002FSKILL.md",[],[172,173,174,175,176,177,178,179,180,22,181,106,182,183,184,185,186,187,188,189],"wrap up","up","wrap","gemini","deployment","run","file","hardening","closeout","stop","upgrade","founder","close","companion","you","tables","directions","phase",{"skillNo":191,"name":192,"nameZh":6,"description":193,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":194,"categories":195,"keywords":197,"publishedAt":39},"01M6QRA5AYC1RBFZ39GTWGS3N8","workflow-builder","Design and write deterministic multi-agent workflow scripts (.js files in .claude\u002Fworkflows\u002F) for Claude Code's Workflow tool. Use when a user wants to build, create, author, scaffold, or run a custom Claude Code workflow, orchestrate sub-agents (fan-out, pipeline, loop, judge-panel), or automate a repeatable multi-step task across fresh-context agents.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fworkflow-builder\u002FSKILL.md",[196],{"categoryNo":145,"categoryName":146,"taxonomyType":93,"taxonomyTypeDesc":94},[198,199,200,201,37,150,177,202,203,204,205,206,132,207,105,208,209,210,211,212],"workflow","workflow builder","builder","design","code","orchestrate","pipeline","loop","multi-agent","context","across","fresh","scripts","judge","claude code workflow",{"skillNo":214,"name":215,"nameZh":6,"description":216,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":217,"categories":218,"keywords":219,"publishedAt":39},"01M6QR7QMCZESKQA0ERW1E657M","weekly-review","Use when someone wants to run a weekly review, close open loops, audit stalled projects and commitments, get their system back to trusted, restart a lapsed review habit, or says \"\u002Fcs:weekly-review\". Walks David Allen's three-phase loop — GET CLEAR, GET CURRENT, GET CREATIVE — with deterministic scripts that inventory open loops, gate the checklist with named gaps, and score commitment health 0-100.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fweekly-review\u002FSKILL.md",[],[220,221,222,223,224,225,226,227,228,205,229,230,231,232,233,234,107,235,236,189],"review","weekly review","weekly","deterministic scripts","reflect","feeds","inventory","open","team","projects","project-management","scanner","conversation","back","project","sprint","cadence"]