Awq

Awq is a community skill for quantizing large language models using Activation-aware Weight Quantization, covering model compression to 4-bit precision, calibration data selection, quantization configuration, inference optimization, and deployment of quantized models for efficient LLM serving.

What Is This?

Overview

Awq provides patterns for compressing large language models using AWQ (Activation-aware Weight Quantization) to reduce memory footprint and accelerate inference. It covers model quantization to 4-bit precision that reduces model size by approximately 75 percent while preserving output quality, calibration data selection that chooses representative samples for accurate quantization scaling, quantization configuration that tunes group size, version, and bit width for quality and speed tradeoffs, inference optimization that uses quantized kernels for faster token generation, and deployment integration that serves quantized models through vLLM, TGI, or custom inference servers. The skill enables deployment of large language models on consumer and edge hardware, making previously inaccessible model sizes practical for teams without enterprise-grade GPU infrastructure.

Who Should Use This

This skill serves ML engineers deploying LLMs on resource-constrained hardware, teams optimizing inference costs by reducing GPU memory requirements, and researchers experimenting with large models on consumer GPUs. It is also relevant for platform engineers building cost-efficient serving pipelines where GPU memory is the primary bottleneck.

Why Use It?

Problems It Solves

Full-precision LLMs require expensive multi-GPU setups for deployment. Inference latency increases with model size limiting throughput for serving applications. Consumer GPUs lack sufficient VRAM to load large models at full precision. Serving costs scale linearly with GPU memory requirements, making high-traffic deployments prohibitively expensive without compression.

Core Highlights

4-bit quantization reduces memory by 75 percent with minimal quality degradation. Activation-aware scaling preserves important weight channels for better accuracy. Quantized kernels accelerate matrix multiplication during inference. Compatible with vLLM and TGI for production serving.

How to Use It?

Basic Usage

from awq import AutoAWQFor\
  CausalLM
from transformers\
  import AutoTokenizer

model_path =\
  'meta-llama/Llama-3-8B'
quant_path =\
  'llama-3-8b-awq'

model = AutoAWQForCausalLM\
  .from_pretrained(model_path)
tokenizer = AutoTokenizer\
  .from_pretrained(model_path)

quant_config = {
  'zero_point': True,
  'q_group_size': 128,
  'w_bit': 4,
  'version': 'GEMM',
}

model.quantize(
  tokenizer,
  quant_config=quant_config)

model.save_quantized(
  quant_path)
tokenizer.save_pretrained(
  quant_path)

Real-World Examples

from awq import AutoAWQFor\
  CausalLM
from transformers\
  import AutoTokenizer

quant_path =\
  'llama-3-8b-awq'

model = AutoAWQForCausalLM\
  .from_quantized(
    quant_path,
    fuse_layers=True)
tokenizer = AutoTokenizer\
  .from_pretrained(
    quant_path)

prompt = 'Explain quantum'\
  + ' computing in simple'\
  + ' terms:'
inputs = tokenizer(
  prompt,
  return_tensors='pt')\
  .to('cuda')

output = model.generate(
  **inputs,
  max_new_tokens=256,
  temperature=0.7,
  do_sample=True)

text = tokenizer.decode(
  output[0],
  skip_special_tokens=True)
print(text)

Advanced Tips

Use the GEMM quantization version for GPU inference and GEMV for CPU-bound or batch-size-one scenarios. Provide domain-specific calibration data for better quantization accuracy on specialized tasks. For example, when quantizing a model intended for code generation, use a calibration dataset drawn from source code rather than general web text. Enable fuse_layers during loading to merge quantized operations for faster inference. Benchmark throughput at different batch sizes as quantization benefits vary with serving configuration. A group size of 128 balances accuracy and speed for most models, but reducing it to 64 can recover quality on sensitive architectures at a modest memory cost.

When to Use It?

Use Cases

Quantize a 70B parameter model to fit on a single 24GB GPU for local inference. Deploy a quantized model through vLLM for production serving with reduced costs. Run large models on consumer hardware for development and testing.

Related Topics

Model quantization, AWQ, LLM deployment, inference optimization, and model compression.

Important Notes

Requirements

CUDA-capable GPU with sufficient VRAM for the quantized model. AutoAWQ library with compatible PyTorch version. Calibration dataset with representative text samples for quantization scaling computation. Sufficient disk space for both the original and quantized model files during the quantization process.

Usage Recommendations

Do: evaluate quantized model quality on your target tasks before production deployment. Use representative calibration data from your domain for best accuracy. Compare AWQ against GPTQ and GGUF quantization for your specific model and hardware.

Don't: assume quantization quality is identical across all model architectures. Use random calibration data that does not represent actual inference workloads. Quantize models below 4 bits without thorough quality evaluation.

Limitations

Quantization introduces small accuracy degradation that varies by model and task. 4-bit quantized models may underperform on tasks requiring precise numerical reasoning. AWQ kernels require CUDA GPUs and do not support CPU-only inference natively. Quantization time depends on model size and calibration dataset length. Very small models under 1B parameters may not benefit from quantization as the accuracy cost outweighs memory savings.