Debarshi Das

Tool

KV cache calculator

This tool shows how much memory the KV cache takes for one token, one request and one GPU, for grouped-query, latent, sliding-window and hybrid models. It is the companion to the essay KV cache size per token, which explains every number.

Updated . Short link: debarshidas.dev/kvcalc

Calculator

Model and request
The model whose config.json sets the cache shape, read from Hugging Face. Type a name, an organization, a class such as mla or mamba, or paste a repo id or link.

The serving engine whose memory rule sets the cache budget. vLLM and SGLang count a share of total GPU memory; TensorRT-LLM counts a share of the memory left after the weights.
The bytes per element for each stored key and value, set by the engine's KV cache dtype option. FP4 is NVFP4, with one FP8 scale per 16 values; latent rows follow vLLM, whose sparse-MLA rows depend on the GPU generation.
The tokens one request keeps: its prompt and its generated tokens. They are rounded up to whole cache blocks.
The requests decoding at the same time. The results say whether their cache fits beside the weights.
GPUs
The GPU, from the cited gpu-specs dataset. Choosing one fills its memory: the value the driver reports where a public sample exists, or else the vendor's printed figure, which the driver reports a little under.

81 559 MiB as the driver reports it. Source: NVIDIA Multi-Instance GPU User Guide (Supported GPUs page and PDF).

The memory per GPU as the driver reports it, in GiB. An H100 80GB reports about 79.6 GiB; type a value for a GPU that is not listed.
The share of each GPU's memory the serving engine may use, for weights and cache together: vLLM's gpu_memory_utilization. vLLM's default is 0.92 since v0.20 (0.9 before); this page starts at the essay's 0.9.
The GPUs one copy of the model is split across. KV heads split across them until there are more GPUs than heads; a latent row stays whole on every GPU.
How the weights are stored. By default, the checkpoint's own size, from its safetensors types, so FP8 and MXFP4 models count as shipped. A chosen precision applies that many bytes to every parameter.
The tokens per cache block, set by the engine's block size option. Each request's tokens are rounded up to whole blocks; for a hybrid model vLLM raises the block to fit one state page.
Handoff link
The link the cache crosses from prefill to decode, at its vendor's published rate per direction. NVLink and PCIe are per GPU, so the GPUs of a group receive in parallel.
The network interface cards (NICs) per node that carry the handoff. Their rates add up to a node total that the tensor-parallel group shares.
A link rate in GB/s per direction, for a fabric that is not listed. The group shares it, as it shares a node's NICs.

Qwen3-8B, 8192 tokens, BF16 cache

Cache per token
147 456 bytes
One request, cache and state
1.125 GiB
One request, per GPU
1.125 GiB
Weights per GPU
15.3 GiB (as the checkpoint stores them)
Left for the cache per GPU
56.4 GiB
Requests that fit, at most
50
64 requests need, per GPU
72 GiB (does not fit)
Handoff, prefill to decode
24.2 ms (at 50 GB/s, shared by the group)

An upper bound: activations, CUDA graph buffers and FP8 scale tensors also use memory. The budget follows vLLM: gpu_memory_utilization is a share of total GPU memory, for weights and cache together. Handoff times assume ideal line rate.

Architecture of Qwen3-8B

View the model card on Hugging Face

Class
GQA
Layers with attention
36 of 36
KV heads
8
Head dimension
128
Sliding window
none
Fixed state per request
none
Parameters
8 190 735 360

From the site's snapshot of : config.json at b968826.

The config fields this rests on
  • model_type=qwen3
  • num_hidden_layers=36
  • num_attention_heads=32
  • head_dim=128
  • num_key_value_heads=8

How it computes

Cache per token. For each layer that keeps every token, the cache grows by 2 × KV heads × head dimension × bytes per element. Latent attention (MLA) stores one row of latent and position values per layer instead. Sliding-window layers stop growing at their window. Hybrid models add a fixed state per request, which does not grow.

Cache blocks. Tokens round up to whole cache blocks.

Tensor parallelism. KV heads split across the GPUs until there are more GPUs than heads. A latent row never splits, so every GPU holds all of it. Decode context parallelism shards it by token instead, but this page does not model that. Weights divide evenly across the GPUs.

Requests that fit. The memory left after the weights, divided by one request's cache on one GPU. It is an upper bound: activations, CUDA graph buffers and FP8 scale tensors also take memory. Check it against the engine's startup log.

Memory share. It follows the engine. vLLM's gpu_memory_utilization is a share of the GPU's total memory, for weights and cache together. Its default is 0.92 since v0.20 and was 0.9 before; this page starts at the essay's 0.9. SGLang's mem_fraction_static is also a share of total memory, but SGLang sets it at startup from the GPU and its settings, so enter the value its log reports. TensorRT-LLM's free_gpu_memory_fraction (the --kv_cache_free_gpu_memory_fraction flag) is a share of the memory left after the weights, 0.9 by default.

Precision. Bytes per token in BF16 are model math, the same on every engine. FP4 is NVFP4: 4-bit values and one FP8 scale per 16 values, 0.5625 bytes per element. Weights default to the checkpoint as stored, read from its safetensors types.

Latent rows. Latent attention follows vLLM. Plain MLA stores a 576-byte FP8 row and has no FP4 format. For sparse MLA (MLA with an indexer) the row depends on the GPU: in FP8, the 656-byte fp8_ds_mla row on Hopper and a standard 576-byte row on Blackwell; in FP4, the 352-byte nvfp4_ds_mla row, on Blackwell only. Capacity uses the Hopper row. Other engines may store a different row.

Fixed state. It follows the engine. vLLM keeps the recurrent state of Nemotron-H in FP32, and that of Qwen3.5 when its config asks. It raises a hybrid's block size until one block of attention holds one layer's state, and pads each state to that block. Other engines count the state in BF16, at the block size you set.

Sliding windows. On vLLM, window layers count the whole blocks that vLLM keeps around the window.

Speculative decoding. Multi-token-prediction (MTP) layers count when you choose them. Each keeps a cache like a normal layer.

Links. Rates are the vendors' published line rates per direction. NVLink and PCIe rates are per GPU, so the GPUs of a group receive in parallel. The NICs of a node add up to one total that the group shares.

Sources. Engine rules come from the vLLM v0.31.0 source.

Every model at a glance

Bytes per token and one request's cache and state, all in BF16, 16-token blocks. Computed from each model's config.json, not measured.
ModelTypeBytes per token8K tokens32K tokens
Qwen3-8BGQA147 4561.125 GiB4.5 GiB
Llama-2-7BMHA524 2884 GiB16 GiB
Falcon-7BMQA819264 MiB256 MiB
Qwen3-30B-A3BGQA, mixture of experts98 304768 MiB3 GiB
DeepSeek-V3MLA70 272549 MiB2.145 GiB
DeepSeek-V3.2MLA, sparse attention78 324612 MiB2.39 GiB
gpt-oss-120bGQA, sliding window36 864293 MiB1.13 GiB
Qwen3-Next-80B-A3BLinear attention hybrid24 576242 MiB816 MiB
Nemotron-H-8BMamba hybrid16 384228 MiB618 MiB

The same data, with layer counts and config links, is available as CSV.

Any model on Hugging Face

The model field searches Hugging Face as you type. For the model you choose, your browser reads its config.json at its current commit. The classifier that builds the site's snapshot names the class and computes the cache. A layout it has not reviewed, such as compressed or chunked attention, shows as unsupported, with the reason. The classifier never guesses. Some terms depend on the engine and the GPU, such as the FP8 row of sparse MLA or the dtype of a hybrid's state. The results and the notes name them.

Before Hugging Face answers, the field offers a snapshot of 76 models, read on and chosen from Hugging Face's download counts and recent releases. It is a starting list, not a ranking: any public model is one search away. By class: 35 KV-head attention (MHA, GQA or MQA), 4 sliding window, 2 MLA, 3 MLA sparse, 3 Mamba hybrid, 14 linear hybrid, 15 unsupported. The snapshot, with every number and config link, is available as CSV.

GPUs and their memory

The GPU field lists 120 NVIDIA and AMD GPUs from gpu-specs v0.1.0, an open dataset in which every value cites the vendor's own document. Choosing a GPU fills its memory in GiB. For 6 GPUs that is the figure the driver reports, from a cited nvidia-smi sample; for the others it is the vendor's printed figure, which the driver reports a little under. For a GPU that is not listed, type its memory. The dataset, with every source and quote: gpu-specs v0.1.0.

Choose a style to copy