high-performance-inference（高性能推理）

High-Performance Inference

Optimize LLM inference for production with vLLM 0.14.x, quantization, and speculative decoding.

vLLM 0.14.0 (Jan): PyTorch 2.9.0, CUDA 12.9, AttentionConfig API, Python 3.12+ recommended.

Overview

• Deploying LLMs with low latency requirements
• Reducing GPU memory for larger models
• Maximizing throughput for batch inference
• Edge/mobile deployment with constrained resources
• Cost optimization through efficient hardware utilization

Quick Reference

# Basic vLLM server
vllm serve meta-llama/Meta-Llama-3.1-70B-Instruct \
    --tensor-parallel-size 4 \
    --max-model-len 8192

# With quantization + speculative decoding
vllm serve meta-llama/Meta-Llama-3.1-70B-Instruct \
    --quantization awq \
    --speculative-config '{"method": "ngram", "num_speculative_tokens": 5}' \
    --tensor-parallel-size 4 \
    --gpu-memory-utilization 0.9

vLLM 0.14.x Key Features

Python vLLM Integration

from vllm import LLM, SamplingParams

# Initialize with optimization flags
llm = LLM(
    model="meta-llama/Meta-Llama-3.1-8B-Instruct",
    quantization="awq",
    tensor_parallel_size=2,
    gpu_memory_utilization=0.9,
    enable_prefix_caching=True,
)

# Sampling parameters
sampling_params = SamplingParams(
    temperature=0.7,
    top_p=0.9,
    max_tokens=1024,
)

# Generate
outputs = llm.generate(prompts, sampling_params)
for output in outputs:
    print(output.outputs[0].text)

Quantization Methods

When to Use Each:

• FP16: Maximum quality, sufficient memory
• INT8/FP8: Balance of quality and efficiency
• AWQ: Best 4-bit quality, activation-aware
• GPTQ: Faster quantization, good quality

Speculative Decoding

Accelerate generation by predicting multiple tokens:

# N-gram based (no extra model)
speculative_config = {
    "method": "ngram",
    "num_speculative_tokens": 5,
    "prompt_lookup_max": 5,
    "prompt_lookup_min": 2,
}

# Draft model (higher quality)
speculative_config = {
    "method": "draft_model",
    "draft_model": "meta-llama/Llama-3.2-1B-Instruct",
    "num_speculative_tokens": 3,
}

Expected Gains: 1.5-2.5x throughput for autoregressive tasks.

Key Decisions

Common Mistakes

• Using GPTQ without calibration data (poor quality)
• Over-allocating GPU memory (OOM on peak loads)
• Ignoring warmup requests (cold start latency)
• Not benchmarking actual workload patterns
• Mixing quantization with incompatible features

Performance Benchmarking

from vllm import LLM, SamplingParams
import time

def benchmark_throughput(llm, prompts, sampling_params, num_runs=3):
    """Benchmark tokens per second."""
    total_tokens = 0
    total_time = 0

    for _ in range(num_runs):
        start = time.perf_counter()
        outputs = llm.generate(prompts, sampling_params)
        elapsed = time.perf_counter() - start

        tokens = sum(len(o.outputs[0].token_ids) for o in outputs)
        total_tokens += tokens
        total_time += elapsed

    return total_tokens / total_time  # tokens/sec

Advanced Patterns

See references/ for:

• vLLM Deployment: PagedAttention, batching, production config
• Quantization Guide: AWQ, GPTQ, INT8, FP8 comparison
• Speculative Decoding: Draft models, n-gram, throughput tuning
• Edge Deployment: Mobile, resource-constrained optimization

Related Skills

• llm-streaming - Streaming token responses
• function-calling - Tool use with inference
• ollama-local - Local inference with Ollama
• prompt-caching - Reduce redundant computation
• semantic-caching - Cache full responses

Capability Details

vllm-deployment

Keywords: vllm, inference server, deploy, serve, production Solves:

• Deploy LLMs with vLLM for production
• Configure tensor parallelism and batching
• Optimize GPU memory utilization

quantization

Keywords: quantize, AWQ, GPTQ, INT8, FP8, compress, reduce memory Solves:

• Reduce model memory footprint
• Choose appropriate quantization method
• Maintain quality with lower precision

speculative-decoding

Keywords: speculative, draft model, faster generation, predict tokens Solves:

• Accelerate autoregressive generation
• Configure draft models or n-gram speculation
• Tune speculative token count

edge-inference

Keywords: edge, mobile, embedded, constrained, optimization Solves:

• Deploy on resource-constrained devices
• Optimize for mobile/edge hardware
• Balance quality and resource usage

throughput-optimization

Keywords: throughput, latency, performance, benchmark, optimize Solves:

• Maximize requests per second
• Reduce time to first token
• Benchmark and tune performance