[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6M9RGCGTYSGF1A5PPP93V70":3,"skill-same:01M6M9RGCGTYSGF1A5PPP93V70":86},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":41,"linkOnly":42,"markdownContent":43,"markdownBody":44,"frontMatter":45,"refName":52,"commitSha":53,"skillDirectory":54,"forksCount":55,"repositoryHtmlUrl":56,"repositoryHomepageUrl":57,"ownerHtmlUrl":58,"ownerBlogUrl":59,"ownerDescription":60,"updatedAt":61,"install":62,"resources":75},"01M6M9RGCGTYSGF1A5PPP93V70","awq-quantization",null,"Activation-aware weight quantization for 4-bit LLM compression with 3x speedup and minimal accuracy loss. Use when deploying large models (7B-70B) on limited GPU memory, when you need faster inference than GPTQ with better accuracy preservation, or for instruction-tuned and multimodal models. MLSys 2024 Best Paper Award winner.","hashgraph-online\u002Fawesome-codex-plugins","hashgraph-online","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F160371962?v=4","01M6CNB6NRJVNNGN14TR01Q6D3","HOL (Hashgraph Online)","01M6JT1HSKKG9V68KQAQGBA7XP",1575,"MIT","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fawq-quantization\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40],"awq quantization","Quantization","AWQ","Activation-Aware","when","2024","accuracy","for","you","GPTQ","Marlin Kernels","vLLM Integration","Memory Optimization","memory","4-Bit","bit","activation","Transformers","Fast Inference","Optimization","multimodal","training","2026-11-10 10:12:30",false,"---\nname: awq-quantization\ndescription: Activation-aware weight quantization for 4-bit LLM compression with 3x speedup and minimal accuracy loss. Use when deploying large models (7B-70B) on limited GPU memory, when you need faster inference than GPTQ with better accuracy preservation, or for instruction-tuned and multimodal models. MLSys 2024 Best Paper Award winner.\nlicense: MIT\nmetadata:\n  version: 1.0.0\n  category: ml-training\n  maintainer: Kalaris Labs\n  tags: Optimization, AWQ, Quantization, 4-Bit, Activation-Aware, Memory Optimization, Fast Inference, vLLM Integration, Marlin Kernels\n  dependencies: autoawq, transformers>=4.45.0, torch>=2.0.0\n---\n\n# AWQ (Activation-aware Weight Quantization)\n\n4-bit quantization that preserves salient weights based on activation patterns, achieving 3x speedup with minimal accuracy loss.\n\n## When to use AWQ\n\n**Use AWQ when:**\n- Need 4-bit quantization with \u003C5% accuracy loss\n- Deploying instruction-tuned or chat models (AWQ generalizes better)\n- Want ~2.5-3x inference speedup over FP16\n- Using vLLM for production serving\n- Have Ampere+ GPUs (A100, H100, RTX 40xx) for Marlin kernel support\n\n**Use GPTQ instead when:**\n- Need maximum ecosystem compatibility (more tools support GPTQ)\n- Working with ExLlamaV2 backend specifically\n- Have older GPUs without Marlin support\n\n**Use bitsandbytes instead when:**\n- Need zero calibration overhead (quantize on-the-fly)\n- Want to fine-tune with QLoRA\n- Prefer simpler integration\n\n## Quick start\n\n### Installation\n\n```bash\n# Default (Triton kernels)\npip install autoawq\n\n# With optimized CUDA kernels + Flash Attention\npip install autoawq[kernels]\n\n# Intel CPU\u002FXPU optimization\npip install autoawq[cpu]\n```\n\n**Requirements**: Python 3.8+, CUDA 11.8+, Compute Capability 7.5+\n\n### Load pre-quantized model\n\n```python\nfrom awq import AutoAWQForCausalLM\nfrom transformers import AutoTokenizer\n\nmodel_name = \"TheBloke\u002FMistral-7B-Instruct-v0.2-AWQ\"\n\nmodel = AutoAWQForCausalLM.from_quantized(\n    model_name,\n    fuse_layers=True  # Enable fused attention for speed\n)\ntokenizer = AutoTokenizer.from_pretrained(model_name)\n\n# Generate\ninputs = tokenizer(\"Explain quantum computing\", return_tensors=\"pt\").to(\"cuda\")\noutputs = model.generate(**inputs, max_new_tokens=200)\nprint(tokenizer.decode(outputs[0], skip_special_tokens=True))\n```\n\n### Quantize your own model\n\n```python\nfrom awq import AutoAWQForCausalLM\nfrom transformers import AutoTokenizer\n\nmodel_path = \"mistralai\u002FMistral-7B-Instruct-v0.2\"\n\n# Load model and tokenizer\nmodel = AutoAWQForCausalLM.from_pretrained(model_path)\ntokenizer = AutoTokenizer.from_pretrained(model_path)\n\n# Quantization config\nquant_config = {\n    \"zero_point\": True,      # Use zero-point quantization\n    \"q_group_size\": 128,     # Group size (128 recommended)\n    \"w_bit\": 4,              # 4-bit weights\n    \"version\": \"GEMM\"        # GEMM for batch, GEMV for single-token\n}\n\n# Quantize (uses pileval dataset by default)\nmodel.quantize(tokenizer, quant_config=quant_config)\n\n# Save\nmodel.save_quantized(\"mistral-7b-awq\")\ntokenizer.save_pretrained(\"mistral-7b-awq\")\n```\n\n**Timing**: ~10-15 min for 7B, ~1 hour for 70B models.\n\n## AWQ vs GPTQ vs bitsandbytes\n\n| Feature | AWQ | GPTQ | bitsandbytes |\n|---------|-----|------|--------------|\n| **Speedup (4-bit)** | ~2.5-3x | ~2x | ~1.5x |\n| **Accuracy loss** | \u003C5% | ~5-10% | ~5-15% |\n| **Calibration** | Minimal (128-1K tokens) | More extensive | None |\n| **Overfitting risk** | Low | Higher | N\u002FA |\n| **Best for** | Production inference | GPU inference | Easy integration |\n| **vLLM support** | Native | Yes | Limited |\n\n**Key insight**: AWQ assumes not all weights are equally important. It protects ~1% of salient weights identified by activation patterns, reducing quantization error without mixed-precision overhead.\n\n## Kernel backends\n\n### GEMM (default, batch inference)\n\n```python\nquant_config = {\n    \"zero_point\": True,\n    \"q_group_size\": 128,\n    \"w_bit\": 4,\n    \"version\": \"GEMM\"  # Best for batch sizes > 1\n}\n```\n\n### GEMV (single-token generation)\n\n```python\nquant_config = {\n    \"version\": \"GEMV\"  # 20% faster for batch_size=1\n}\n```\n\n**Limitation**: Only batch size 1, not good for large context.\n\n### Marlin (Ampere+ GPUs)\n\n```python\nfrom transformers import AwqConfig, AutoModelForCausalLM\n\nconfig = AwqConfig(\n    bits=4,\n    version=\"marlin\"  # 2x faster on A100\u002FH100\n)\n\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"TheBloke\u002FMistral-7B-AWQ\",\n    quantization_config=config\n)\n```\n\n**Requirements**: Compute Capability 8.0+ (A100, H100, RTX 40xx)\n\n### ExLlamaV2 (AMD compatible)\n\n```python\nconfig = AwqConfig(\n    bits=4,\n    version=\"exllama\"  # Faster prefill, AMD GPU support\n)\n```\n\n## HuggingFace Transformers integration\n\n### Direct loading\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"TheBloke\u002Fzephyr-7B-alpha-AWQ\",\n    device_map=\"auto\"\n)\ntokenizer = AutoTokenizer.from_pretrained(\"TheBloke\u002Fzephyr-7B-alpha-AWQ\")\n```\n\n### Fused modules (recommended)\n\n```python\nfrom transformers import AwqConfig, AutoModelForCausalLM\n\nconfig = AwqConfig(\n    bits=4,\n    fuse_max_seq_len=512,  # Max sequence length for fusing\n    do_fuse=True           # Enable fused attention\u002FMLP\n)\n\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"TheBloke\u002FMistral-7B-OpenOrca-AWQ\",\n    quantization_config=config\n)\n```\n\n**Note**: Fused modules cannot combine with FlashAttention2.\n\n## vLLM integration\n\n```python\nfrom vllm import LLM, SamplingParams\n\n# vLLM auto-detects AWQ models\nllm = LLM(\n    model=\"TheBloke\u002FLlama-2-7B-AWQ\",\n    quantization=\"awq\",\n    dtype=\"half\"\n)\n\nsampling = SamplingParams(temperature=0.7, max_tokens=200)\noutputs = llm.generate([\"Explain AI\"], sampling)\n```\n\n## Performance benchmarks\n\n### Memory reduction\n\n| Model | FP16 | AWQ 4-bit | Reduction |\n|-------|------|-----------|-----------|\n| Mistral 7B | 14 GB | 5.5 GB | 2.5x |\n| Llama 2-13B | 26 GB | 10 GB | 2.6x |\n| Llama 2-70B | 140 GB | 35 GB | 4x |\n\n### Inference speed (RTX 4090)\n\n| Model | Prefill (tok\u002Fs) | Decode (tok\u002Fs) | Memory |\n|-------|-----------------|----------------|--------|\n| Mistral 7B GEMM | 3,897 | 114 | 5.55 GB |\n| TinyLlama 1B GEMV | 5,179 | 431 | 2.10 GB |\n| Llama 2-13B GEMM | 2,279 | 74 | 10.28 GB |\n\n### Accuracy (perplexity)\n\n| Model | FP16 | AWQ 4-bit | Degradation |\n|-------|------|-----------|-------------|\n| Llama 3 8B | 8.20 | 8.48 | +3.4% |\n| Mistral 7B | 5.25 | 5.42 | +3.2% |\n| Qwen2 72B | 4.85 | 4.95 | +2.1% |\n\n## Custom calibration data\n\n```python\n# Use custom dataset for domain-specific models\nmodel.quantize(\n    tokenizer,\n    quant_config=quant_config,\n    calib_data=\"wikitext\",       # Or custom list of strings\n    max_calib_samples=256,       # More samples = better accuracy\n    max_calib_seq_len=512        # Sequence length\n)\n\n# Or provide your own samples\ncalib_samples = [\n    \"Your domain-specific text here...\",\n    \"More examples from your use case...\",\n]\nmodel.quantize(tokenizer, quant_config=quant_config, calib_data=calib_samples)\n```\n\n## Multi-GPU deployment\n\n```python\nmodel = AutoAWQForCausalLM.from_quantized(\n    \"TheBloke\u002FLlama-2-70B-AWQ\",\n    device_map=\"auto\",  # Auto-split across GPUs\n    max_memory={0: \"40GB\", 1: \"40GB\"}\n)\n```\n\n## Supported models\n\n35+ architectures including:\n- **Llama family**: Llama 2\u002F3, Code Llama, Mistral, Mixtral\n- **Qwen**: Qwen, Qwen2, Qwen2.5-VL\n- **Others**: Falcon, MPT, Phi, Yi, DeepSeek, Gemma\n- **Multimodal**: LLaVA, LLaVA-Next, Qwen2-VL\n\n## Common issues\n\n**CUDA OOM during quantization**:\n```python\n# Reduce batch size\nmodel.quantize(tokenizer, quant_config=quant_config, max_calib_samples=64)\n```\n\n**Slow inference**:\n```python\n# Enable fused layers\nmodel = AutoAWQForCausalLM.from_quantized(model_name, fuse_layers=True)\n```\n\n**AMD GPU support**:\n```python\n# Use ExLlama backend\nconfig = AwqConfig(bits=4, version=\"exllama\")\n```\n\n## Deprecation notice\n\nAutoAWQ is officially deprecated. For new projects, consider:\n- **vLLM llm-compressor**: https:\u002F\u002Fgithub.com\u002Fvllm-project\u002Fllm-compressor\n- **MLX-LM**: For Mac devices with Apple Silicon\n\nExisting quantized models remain usable.\n\n## References\n\n- **Paper**: AWQ: Activation-aware Weight Quantization (arXiv:2306.00978) - MLSys 2024 Best Paper\n- **GitHub**: https:\u002F\u002Fgithub.com\u002Fcasper-hansen\u002FAutoAWQ\n- **MIT Han Lab**: https:\u002F\u002Fgithub.com\u002Fmit-han-lab\u002Fllm-awq\n- **Models**: https:\u002F\u002Fhuggingface.co\u002Fmodels?library=awq\n\n## Agent operating procedure\n\n1. **Check the environment.** Check GPU type, memory and driver\u002FCUDA versions (`nvidia-smi`), framework versions, and dataset location and size.\n2. **Pin down the inputs.** Confirm formats, identifiers and parameters from the data or the user. Ask rather than guess any value that changes the result.\n3. **Run a small version first.** Do a smoke run: tiny model or subset, few steps, and confirm loss decreases and checkpoints save.\n4. **Execute the full task** using the instructions and references above.\n5. **Validate the result.** Track metrics on held-out data, compare against a baseline, and record seeds, configs and hardware.\n6. **Report.** State what was run (versions, commands, parameters), what was checked, and what is still uncertain.\n\n| If this happens | Do this |\n|---|---|\n| CUDA out-of-memory | Reduce batch size, enable gradient accumulation\u002Fcheckpointing or mixed precision, or shard the model. |\n| Loss is NaN or diverges | Lower the learning rate, check data for invalid values, and enable gradient clipping. |\n| A function, flag or endpoint in these instructions is missing in the installed version | Check the installed version's own documentation (`help()`, `--help`, official docs), adapt, and tell the user. Never invent an API. |\n| A required input, identifier or parameter is ambiguous | Ask the user, or state the assumption explicitly before running. |\n\n**Integrity rules**\n\n- Never fabricate results, parameters, identifiers, citations or statistics. If something cannot be run or verified, say so plainly.\n- Never claim training results without logs; estimate compute cost before launching large jobs.\n- Treat version-specific details here as possibly outdated: confirm them against the official documentation for the installed version.\n- Ask before actions that cost money, consume shared GPUs or cloud quota, touch personal or patient data, or cannot be undone.\n\n## Related skills\n\n- `quantizing-models-bitsandbytes`: Quantizes LLMs to 8-bit or 4-bit for 50-75% memory reduction with minimal accuracy loss.\n- `gptq`: Quantizes LLMs to 4-bit (also 3-bit) with GPTQ using group-wise quantization (group size 128 by default), via AutoGPTQ and transformers.\n- `serving-llms-vllm`: Serves LLMs with high throughput using vLLM's PagedAttention and continuous batching.\n","# AWQ (Activation-aware Weight Quantization)\n\n4-bit quantization that preserves salient weights based on activation patterns, achieving 3x speedup with minimal accuracy loss.\n\n## When to use AWQ\n\n**Use AWQ when:**\n- Need 4-bit quantization with \u003C5% accuracy loss\n- Deploying instruction-tuned or chat models (AWQ generalizes better)\n- Want ~2.5-3x inference speedup over FP16\n- Using vLLM for production serving\n- Have Ampere+ GPUs (A100, H100, RTX 40xx) for Marlin kernel support\n\n**Use GPTQ instead when:**\n- Need maximum ecosystem compatibility (more tools support GPTQ)\n- Working with ExLlamaV2 backend specifically\n- Have older GPUs without Marlin support\n\n**Use bitsandbytes instead when:**\n- Need zero calibration overhead (quantize on-the-fly)\n- Want to fine-tune with QLoRA\n- Prefer simpler integration\n\n## Quick start\n\n### Installation\n\n```bash\n# Default (Triton kernels)\npip install autoawq\n\n# With optimized CUDA kernels + Flash Attention\npip install autoawq[kernels]\n\n# Intel CPU\u002FXPU optimization\npip install autoawq[cpu]\n```\n\n**Requirements**: Python 3.8+, CUDA 11.8+, Compute Capability 7.5+\n\n### Load pre-quantized model\n\n```python\nfrom awq import AutoAWQForCausalLM\nfrom transformers import AutoTokenizer\n\nmodel_name = \"TheBloke\u002FMistral-7B-Instruct-v0.2-AWQ\"\n\nmodel = AutoAWQForCausalLM.from_quantized(\n    model_name,\n    fuse_layers=True  # Enable fused attention for speed\n)\ntokenizer = AutoTokenizer.from_pretrained(model_name)\n\n# Generate\ninputs = tokenizer(\"Explain quantum computing\", return_tensors=\"pt\").to(\"cuda\")\noutputs = model.generate(**inputs, max_new_tokens=200)\nprint(tokenizer.decode(outputs[0], skip_special_tokens=True))\n```\n\n### Quantize your own model\n\n```python\nfrom awq import AutoAWQForCausalLM\nfrom transformers import AutoTokenizer\n\nmodel_path = \"mistralai\u002FMistral-7B-Instruct-v0.2\"\n\n# Load model and tokenizer\nmodel = AutoAWQForCausalLM.from_pretrained(model_path)\ntokenizer = AutoTokenizer.from_pretrained(model_path)\n\n# Quantization config\nquant_config = {\n    \"zero_point\": True,      # Use zero-point quantization\n    \"q_group_size\": 128,     # Group size (128 recommended)\n    \"w_bit\": 4,              # 4-bit weights\n    \"version\": \"GEMM\"        # GEMM for batch, GEMV for single-token\n}\n\n# Quantize (uses pileval dataset by default)\nmodel.quantize(tokenizer, quant_config=quant_config)\n\n# Save\nmodel.save_quantized(\"mistral-7b-awq\")\ntokenizer.save_pretrained(\"mistral-7b-awq\")\n```\n\n**Timing**: ~10-15 min for 7B, ~1 hour for 70B models.\n\n## AWQ vs GPTQ vs bitsandbytes\n\n| Feature | AWQ | GPTQ | bitsandbytes |\n|---------|-----|------|--------------|\n| **Speedup (4-bit)** | ~2.5-3x | ~2x | ~1.5x |\n| **Accuracy loss** | \u003C5% | ~5-10% | ~5-15% |\n| **Calibration** | Minimal (128-1K tokens) | More extensive | None |\n| **Overfitting risk** | Low | Higher | N\u002FA |\n| **Best for** | Production inference | GPU inference | Easy integration |\n| **vLLM support** | Native | Yes | Limited |\n\n**Key insight**: AWQ assumes not all weights are equally important. It protects ~1% of salient weights identified by activation patterns, reducing quantization error without mixed-precision overhead.\n\n## Kernel backends\n\n### GEMM (default, batch inference)\n\n```python\nquant_config = {\n    \"zero_point\": True,\n    \"q_group_size\": 128,\n    \"w_bit\": 4,\n    \"version\": \"GEMM\"  # Best for batch sizes > 1\n}\n```\n\n### GEMV (single-token generation)\n\n```python\nquant_config = {\n    \"version\": \"GEMV\"  # 20% faster for batch_size=1\n}\n```\n\n**Limitation**: Only batch size 1, not good for large context.\n\n### Marlin (Ampere+ GPUs)\n\n```python\nfrom transformers import AwqConfig, AutoModelForCausalLM\n\nconfig = AwqConfig(\n    bits=4,\n    version=\"marlin\"  # 2x faster on A100\u002FH100\n)\n\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"TheBloke\u002FMistral-7B-AWQ\",\n    quantization_config=config\n)\n```\n\n**Requirements**: Compute Capability 8.0+ (A100, H100, RTX 40xx)\n\n### ExLlamaV2 (AMD compatible)\n\n```python\nconfig = AwqConfig(\n    bits=4,\n    version=\"exllama\"  # Faster prefill, AMD GPU support\n)\n```\n\n## HuggingFace Transformers integration\n\n### Direct loading\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"TheBloke\u002Fzephyr-7B-alpha-AWQ\",\n    device_map=\"auto\"\n)\ntokenizer = AutoTokenizer.from_pretrained(\"TheBloke\u002Fzephyr-7B-alpha-AWQ\")\n```\n\n### Fused modules (recommended)\n\n```python\nfrom transformers import AwqConfig, AutoModelForCausalLM\n\nconfig = AwqConfig(\n    bits=4,\n    fuse_max_seq_len=512,  # Max sequence length for fusing\n    do_fuse=True           # Enable fused attention\u002FMLP\n)\n\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"TheBloke\u002FMistral-7B-OpenOrca-AWQ\",\n    quantization_config=config\n)\n```\n\n**Note**: Fused modules cannot combine with FlashAttention2.\n\n## vLLM integration\n\n```python\nfrom vllm import LLM, SamplingParams\n\n# vLLM auto-detects AWQ models\nllm = LLM(\n    model=\"TheBloke\u002FLlama-2-7B-AWQ\",\n    quantization=\"awq\",\n    dtype=\"half\"\n)\n\nsampling = SamplingParams(temperature=0.7, max_tokens=200)\noutputs = llm.generate([\"Explain AI\"], sampling)\n```\n\n## Performance benchmarks\n\n### Memory reduction\n\n| Model | FP16 | AWQ 4-bit | Reduction |\n|-------|------|-----------|-----------|\n| Mistral 7B | 14 GB | 5.5 GB | 2.5x |\n| Llama 2-13B | 26 GB | 10 GB | 2.6x |\n| Llama 2-70B | 140 GB | 35 GB | 4x |\n\n### Inference speed (RTX 4090)\n\n| Model | Prefill (tok\u002Fs) | Decode (tok\u002Fs) | Memory |\n|-------|-----------------|----------------|--------|\n| Mistral 7B GEMM | 3,897 | 114 | 5.55 GB |\n| TinyLlama 1B GEMV | 5,179 | 431 | 2.10 GB |\n| Llama 2-13B GEMM | 2,279 | 74 | 10.28 GB |\n\n### Accuracy (perplexity)\n\n| Model | FP16 | AWQ 4-bit | Degradation |\n|-------|------|-----------|-------------|\n| Llama 3 8B | 8.20 | 8.48 | +3.4% |\n| Mistral 7B | 5.25 | 5.42 | +3.2% |\n| Qwen2 72B | 4.85 | 4.95 | +2.1% |\n\n## Custom calibration data\n\n```python\n# Use custom dataset for domain-specific models\nmodel.quantize(\n    tokenizer,\n    quant_config=quant_config,\n    calib_data=\"wikitext\",       # Or custom list of strings\n    max_calib_samples=256,       # More samples = better accuracy\n    max_calib_seq_len=512        # Sequence length\n)\n\n# Or provide your own samples\ncalib_samples = [\n    \"Your domain-specific text here...\",\n    \"More examples from your use case...\",\n]\nmodel.quantize(tokenizer, quant_config=quant_config, calib_data=calib_samples)\n```\n\n## Multi-GPU deployment\n\n```python\nmodel = AutoAWQForCausalLM.from_quantized(\n    \"TheBloke\u002FLlama-2-70B-AWQ\",\n    device_map=\"auto\",  # Auto-split across GPUs\n    max_memory={0: \"40GB\", 1: \"40GB\"}\n)\n```\n\n## Supported models\n\n35+ architectures including:\n- **Llama family**: Llama 2\u002F3, Code Llama, Mistral, Mixtral\n- **Qwen**: Qwen, Qwen2, Qwen2.5-VL\n- **Others**: Falcon, MPT, Phi, Yi, DeepSeek, Gemma\n- **Multimodal**: LLaVA, LLaVA-Next, Qwen2-VL\n\n## Common issues\n\n**CUDA OOM during quantization**:\n```python\n# Reduce batch size\nmodel.quantize(tokenizer, quant_config=quant_config, max_calib_samples=64)\n```\n\n**Slow inference**:\n```python\n# Enable fused layers\nmodel = AutoAWQForCausalLM.from_quantized(model_name, fuse_layers=True)\n```\n\n**AMD GPU support**:\n```python\n# Use ExLlama backend\nconfig = AwqConfig(bits=4, version=\"exllama\")\n```\n\n## Deprecation notice\n\nAutoAWQ is officially deprecated. For new projects, consider:\n- **vLLM llm-compressor**: https:\u002F\u002Fgithub.com\u002Fvllm-project\u002Fllm-compressor\n- **MLX-LM**: For Mac devices with Apple Silicon\n\nExisting quantized models remain usable.\n\n## References\n\n- **Paper**: AWQ: Activation-aware Weight Quantization (arXiv:2306.00978) - MLSys 2024 Best Paper\n- **GitHub**: https:\u002F\u002Fgithub.com\u002Fcasper-hansen\u002FAutoAWQ\n- **MIT Han Lab**: https:\u002F\u002Fgithub.com\u002Fmit-han-lab\u002Fllm-awq\n- **Models**: https:\u002F\u002Fhuggingface.co\u002Fmodels?library=awq\n\n## Agent operating procedure\n\n1. **Check the environment.** Check GPU type, memory and driver\u002FCUDA versions (`nvidia-smi`), framework versions, and dataset location and size.\n2. **Pin down the inputs.** Confirm formats, identifiers and parameters from the data or the user. Ask rather than guess any value that changes the result.\n3. **Run a small version first.** Do a smoke run: tiny model or subset, few steps, and confirm loss decreases and checkpoints save.\n4. **Execute the full task** using the instructions and references above.\n5. **Validate the result.** Track metrics on held-out data, compare against a baseline, and record seeds, configs and hardware.\n6. **Report.** State what was run (versions, commands, parameters), what was checked, and what is still uncertain.\n\n| If this happens | Do this |\n|---|---|\n| CUDA out-of-memory | Reduce batch size, enable gradient accumulation\u002Fcheckpointing or mixed precision, or shard the model. |\n| Loss is NaN or diverges | Lower the learning rate, check data for invalid values, and enable gradient clipping. |\n| A function, flag or endpoint in these instructions is missing in the installed version | Check the installed version's own documentation (`help()`, `--help`, official docs), adapt, and tell the user. Never invent an API. |\n| A required input, identifier or parameter is ambiguous | Ask the user, or state the assumption explicitly before running. |\n\n**Integrity rules**\n\n- Never fabricate results, parameters, identifiers, citations or statistics. If something cannot be run or verified, say so plainly.\n- Never claim training results without logs; estimate compute cost before launching large jobs.\n- Treat version-specific details here as possibly outdated: confirm them against the official documentation for the installed version.\n- Ask before actions that cost money, consume shared GPUs or cloud quota, touch personal or patient data, or cannot be undone.\n\n## Related skills\n\n- `quantizing-models-bitsandbytes`: Quantizes LLMs to 8-bit or 4-bit for 50-75% memory reduction with minimal accuracy loss.\n- `gptq`: Quantizes LLMs to 4-bit (also 3-bit) with GPTQ using group-wise quantization (group size 128 by default), via AutoGPTQ and transformers.\n- `serving-llms-vllm`: Serves LLMs with high throughput using vLLM's PagedAttention and continuous batching.\n",{"name":5,"license":15,"metadata":46,"description":7},{"tags":47,"version":48,"category":49,"maintainer":50,"dependencies":51},"Optimization, AWQ, Quantization, 4-Bit, Activation-Aware, Memory Optimization, Fast Inference, vLLM Integration, Marlin Kernels","1.0.0","ml-training","Kalaris Labs","autoawq, transformers>=4.45.0, torch>=2.0.0","main","2b63f856ce19c2a797929cacabd9e0fc936812fa","plugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fawq-quantization",491,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins","https:\u002F\u002Fhol.org\u002Fplugins\u002Fbest-codex-plugins","https:\u002F\u002Fgithub.com\u002Fhashgraph-online","https:\u002F\u002Fhol.org","Open-source standards and developer tools for interoperable AI agents: discovery, identity, communication, payments, privacy and security.","2026-11-14 14:06:13",{"source":8,"skillName":5,"commands":63,"prompt":73,"zipAvailable":74},[64,67,70],{"tool":65,"command":66},"npx","npx skills add hashgraph-online\u002Fawesome-codex-plugins --skill awq-quantization",{"tool":68,"command":69},"bunx","bunx skills add hashgraph-online\u002Fawesome-codex-plugins --skill awq-quantization",{"tool":71,"command":72},"pnpm","pnpm dlx skills add hashgraph-online\u002Fawesome-codex-plugins --skill awq-quantization","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fawq-quantization\u002FSKILL.md",true,[76,82],{"path":77,"resourceType":78,"resourceTypeDesc":79,"sizeBytes":80,"sourceHtmlUrl":81},"references\u002Fadvanced-usage.md","REFERENCE","参考文档",7983,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fawq-quantization\u002Freferences\u002Fadvanced-usage.md",{"path":83,"resourceType":78,"resourceTypeDesc":79,"sizeBytes":84,"sourceHtmlUrl":85},"references\u002Ftroubleshooting.md",7733,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fawq-quantization\u002Freferences\u002Ftroubleshooting.md",[87,96,104,112,120,128],{"skillNo":88,"name":89,"nameZh":6,"description":90,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":91,"sourceHtmlUrl":92,"categories":93,"keywords":94,"publishedAt":95},"01M7DV332YQMGD8WTRJ9DZQXTB","worktable-review-with-annotations","Create, inspect, reply to, update, or resolve situated Worktable annotations. Use when feedback belongs to a Doc section, block, text range, or HTML Doc, or when the user asks to address existing review feedback and coordinate a requested artifact edit.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-review-with-annotations\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-review-with-annotations\u002FSKILL.md",[],[],"2026-11-15 14:19:17",{"skillNo":97,"name":98,"nameZh":6,"description":99,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":100,"sourceHtmlUrl":101,"categories":102,"keywords":103,"publishedAt":95},"01M7DV2YGHCZVBEB1AHYB6CGHE","worktable-find-and-synthesize-context","Find, assess, and synthesize context from Worktable Docs and Records. Use when the user asks what a workspace contains, wants workspace-backed research, needs current facts reconciled across artifacts, or requests a cited synthesis without changing workspace state.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-find-and-synthesize-context\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-find-and-synthesize-context\u002FSKILL.md",[],[],{"skillNo":105,"name":106,"nameZh":6,"description":107,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":108,"sourceHtmlUrl":109,"categories":110,"keywords":111,"publishedAt":95},"01M7DV2TQN1H11ME2SAF9MXN7V","worktable-create-or-update-docs","Create, edit, rename, or maintain narrative Worktable Docs. Use for plans, decisions, briefs, notes, research syntheses, and other durable prose when the user requests a new artifact or a safe update to an existing one.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-update-docs\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-update-docs\u002FSKILL.md",[],[],{"skillNo":113,"name":114,"nameZh":6,"description":115,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":116,"sourceHtmlUrl":117,"categories":118,"keywords":119,"publishedAt":95},"01M7DV2NE452NEA8TXATKSNAJH","worktable-create-or-manage-records","Organize repeated structured information as durable Worktable Records. Use for research datasets, inventories, directories, pipelines, trackers, and other collections whose items need identity, fields, filtering, links, validation, or individual updates.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-manage-records\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-manage-records\u002FSKILL.md",[],[],{"skillNo":121,"name":122,"nameZh":6,"description":123,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":124,"sourceHtmlUrl":125,"categories":126,"keywords":127,"publishedAt":95},"01M7DV2HENRBR2KSCVM064WH0Y","worktable-create-or-manage-interactive-html","Create or improve durable visual and interactive artifacts in Worktable. Use for dashboards, trackers, calculators, mini apps, slide presentations, and other requests that benefit from a polished HTML Doc.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-manage-interactive-html\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-manage-interactive-html\u002FSKILL.md",[],[],{"skillNo":129,"name":130,"nameZh":6,"description":131,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":132,"sourceHtmlUrl":133,"categories":134,"keywords":135,"publishedAt":95},"01M7DV2CADR5R7RQFFYPNAGR9Y","worktable-collaborate-threads","Read, answer, and hand off messages through durable Worktable threads with users or other authenticated participants. Use when the user asks to check addressed messages, reply to a participant, continue a thread, request another agent's help, wait briefly for a response, or watch for comments and messages sent to the agent.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-collaborate-threads\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-collaborate-threads\u002FSKILL.md",[],[]]