Source: TensorRT-LLM overview

NVIDIA's official overview of TensorRT-LLM, its PyTorch-based LLM inference library.

Contents
  • What: official documentation (last updated 2026-11-03)
  • Author: NVIDIA
  • Date: 2026-11-03
  • Original: TensorRT-LLM overview

Summary

TensorRT-LLM is NVIDIA’s open-source library for fast LLM inference on NVIDIA GPUs, now architected on PyTorch (the older TensorRT engine backend was removed) and exposing a Python LLM API from one GPU to multi-node.

Key claims

  • In-flight batching with paged KV cache including block reuse.
  • FP8 on H100 (Hopper) and later, FP4 on Blackwell (B200).
  • Speculative decoding: EAGLE, MTP, n-gram.
  • Disaggregated serving (beta), chunked prefill, LoRA, guided decoding, tensor/pipeline/expert parallelism; trtllm-serve CLI.

Feeds

LLM inference serving · Quantization · Speculative decoding

Suggest changes →