跳到正文- vLLM
- Test-Time Compute
- Speculative Decoding
- Source: vLLM V1: A Major Upgrade to vLLM's Core Architecture
- Source: vLLM Structured Outputs Docs
- Source: vLLM release notes v0.8.0 and v0.11.0
- Source: vLLM GitHub README
- Source: vLLM Expert Parallel Deployment docs
- Source: vLLM docs: V1 User Guide
- Source: vLLM docs: Speculative Decoding
- Source: vLLM docs: Reproducibility
- Source: vLLM docs: Quantized KV Cache
- Source: vLLM docs: Quantization
- Source: vLLM docs: Parallelism and Scaling
- Source: vLLM docs: Optimization and Tuning
- Source: vLLM docs: OpenAI-Compatible Server
- Source: vLLM docs: Online Serving
- Source: vLLM docs: Engine Arguments
- Source: vLLM docs: Disaggregated Prefilling
- Source: vLLM docs: Data Parallel Deployment
- Source: vLLM docs: Batch Invariance
- Source: vLLM docs: Automatic Prefix Caching design
- Source: vCache, verified semantic prompt caching (Schroeder et al., 2025)
- Source: Transformers v5: Simple model definitions powering the AI ecosystem (Hugging Face blog)
- Source: Transformers docs: Text generation and Chat templates
- Source: Towards Deterministic Inference in SGLang and Reproducible RL Training (LMSYS)
- Source: The early bird catches the leak, timing side channels in LLM serving (Song et al., 2024)
- Source: Text Generation Inference (TGI) documentation
- Source: TensorRT-LLM overview
- Source: SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
- Source: SGLang: Efficient Execution of Structured Language Model Programs
- Source: SGLang documentation (home)
- Source: Sarathi-Serve: Taming Throughput-Latency Tradeoff in LLM Inference
- Source: safetensors docs/README and Hub pickle-scanning docs
- Source: RedisVL SemanticCache user guide
- Source: Redis LangCache docs (overview and API/SDK examples)
- Source: Qwen2.5-72B-Instruct config.json
- Source: Quantization & Pruning: Make Models Small and Fast (Notebook tutorial)
- Source: Performant local mixture-of-experts CPU inference with GPU acceleration in llama.cpp
- Source: Orca: A Distributed Serving System for Transformer-Based Generative Models
- Source: Ollama documentation (FAQ, context length, Modelfile, OpenAI compatibility, import)
- Source: NVIDIA Dynamo (GitHub README)
- Source: MLX and mlx-lm READMEs (Apple ml-explore)
- Source: Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads
- Source: MeanCache, user-centric semantic caching (Gill et al., 2024)
- Source: Mastering LLM Techniques: Inference Optimization
- Source: LM Studio documentation (app and developer docs)
- Source: LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
- Source: LLM Inference Performance Engineering: Best Practices
- Source: llama.cpp README, llama-server and llama-quantize docs
- Source: Kong AI Semantic Cache plugin docs
- Source: huggingface_hub guide: Download files from the Hub
- Source: Hugging Face PEFT docs: overview and Quicktour
- Source: Hugging Face Inference Providers docs
- Source: Hugging Face Inference Endpoints docs
- Source: Hugging Face Hub docs: Model Cards
- Source: Hugging Face Hub docs: GGUF
- Source: Hugging Face Hub docs: Gated models
- Source: Hugging Face Datasets docs: Arrow and Streaming
- Source: How continuous batching enables 23x throughput in LLM inference
- Source: GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
- Source: GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
- Source: GPTCache usage documentation
- Source: Give Me FP32 or Give Me Death? Challenges and Solutions for Reproducible Reasoning (Yuan et al., 2025)
- Source: FP8 Formats for Deep Learning
- Source: Fast Inference from Transformers via Speculative Decoding
- Source: Fast and Expressive LLM Inference with RadixAttention and SGLang
- Source: Efficient Memory Management for LLM Serving with PagedAttention
- Source: EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test
- Source: DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving
- Source: Defeating Nondeterminism in LLM Inference (Thinking Machines Lab)
- Source: DeepSeek-V3/R1 Inference System Overview
- Source: DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
- Source: Category-aware semantic caching for heterogeneous LLM workloads (Wang et al., 2025)
- Source: Beyond Chinchilla-Optimal: Accounting for Inference in Scaling Laws (Sardana et al., 2024)
- Source: Azure API Management llm-semantic-cache-lookup policy reference
- Source: AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
- Source: Advancing semantic caching with domain-specific embeddings and synthetic data (Gill et al., 2025)
- Source: A postmortem of three recent issues (Anthropic, 2025)
- Semantic Caching
- Quantization
- Mixture of Experts (MoE)
- Local LLMs
- LLM Non-Determinism
- LLM Inference Serving
- KV Cache
- Hugging Face Ecosystem
- Constrained Decoding