KV cache calculator

Cache per token. For each layer that keeps every token, the cache grows by 2 × KV heads × head dimension × bytes per element. Latent attention (MLA) stores one row of latent and position values per layer instead. Sliding-window layers stop growing at their window. Hybrid models add a fixed state per request, which does not grow.

Cache blocks. Tokens round up to whole cache blocks.

Tensor parallelism. KV heads split across the GPUs until there are more GPUs than heads. A latent row never splits, so every GPU holds all of it. Decode context parallelism shards it by token instead, but this page does not model that. Weights divide evenly across the GPUs.

Requests that fit. The memory left after the weights, divided by one request's cache on one GPU. It is an upper bound: activations, CUDA graph buffers and FP8 scale tensors also take memory. Check it against the engine's startup log.

Memory share. It follows the engine. vLLM's gpu_memory_utilization is a share of the GPU's total memory, for weights and cache together. Its default is 0.92 since v0.20 and was 0.9 before; this page starts at the essay's 0.9. SGLang's mem_fraction_static is also a share of total memory, but SGLang sets it at startup from the GPU and its settings, so enter the value its log reports. TensorRT-LLM's free_gpu_memory_fraction (the --kv_cache_free_gpu_memory_fraction flag) is a share of the memory left after the weights, 0.9 by default.

Precision. Bytes per token in BF16 are model math, the same on every engine. FP4 is NVFP4: 4-bit values and one FP8 scale per 16 values, 0.5625 bytes per element. Weights default to the checkpoint as stored, read from its safetensors types.

Latent rows. Latent attention follows vLLM. Plain MLA stores a 576-byte FP8 row and has no FP4 format. For sparse MLA (MLA with an indexer) the row depends on the GPU: in FP8, the 656-byte fp8_ds_mla row on Hopper and a standard 576-byte row on Blackwell; in FP4, the 352-byte nvfp4_ds_mla row, on Blackwell only. Capacity uses the Hopper row. Other engines may store a different row.

Fixed state. It follows the engine. vLLM keeps the recurrent state of Nemotron-H in FP32, and that of Qwen3.5 when its config asks. It raises a hybrid's block size until one block of attention holds one layer's state, and pads each state to that block. Other engines count the state in BF16, at the block size you set.

Sliding windows. On vLLM, window layers count the whole blocks that vLLM keeps around the window.

Speculative decoding. Multi-token-prediction (MTP) layers count when you choose them. Each keeps a cache like a normal layer.

Links. Rates are the vendors' published line rates per direction. NVLink and PCIe rates are per GPU, so the GPUs of a group receive in parallel. The NICs of a node add up to one total that the group shares.

Sources. Engine rules come from the vLLM v0.31.0 source.

The same data, with layer counts and config links, is available as CSV.

The model field searches Hugging Face as you type. For the model you choose, your browser reads its config.json at its current commit. The classifier that builds the site's snapshot names the class and computes the cache. A layout it has not reviewed, such as compressed or chunked attention, shows as unsupported, with the reason. The classifier never guesses. Some terms depend on the engine and the GPU, such as the FP8 row of sparse MLA or the dtype of a hybrid's state. The results and the notes name them.

Before Hugging Face answers, the field offers a snapshot of 76 models, read on 17 October 2026 and chosen from Hugging Face's download counts and recent releases. It is a starting list, not a ranking: any public model is one search away. By class: 35 KV-head attention (MHA, GQA or MQA), 4 sliding window, 2 MLA, 3 MLA sparse, 3 Mamba hybrid, 14 linear hybrid, 15 unsupported. The snapshot, with every number and config link, is available as CSV.

The GPU field lists 120 NVIDIA and AMD GPUs from gpu-specs v0.1.0, an open dataset in which every value cites the vendor's own document. Choosing a GPU fills its memory in GiB. For 6 GPUs that is the figure the driver reports, from a cited nvidia-smi sample; for the others it is the vendor's printed figure, which the driver reports a little under. For a GPU that is not listed, type its memory. The dataset, with every source and quote: gpu-specs v0.1.0 (opens in a new tab).