Explainer

GPU Benchmarks for Local LLMs: Tokens per Second

In this article
  1. What the numbers measure
  2. Where the numbers come from
  3. Token generation across tiers
  4. CUDA, Vulkan and ROCm are different roads
  5. Capacity beats speed
  6. What the benchmarks leave out
  7. What this means when buying
  8. Frequently asked questions
  9. Sources & credits

Short answer

For local LLMs, published benchmarks agree on one order: the RTX 5090 first, then the RTX 5080, the RTX 5070 Ti, and the Radeon cards behind them on the same software. Token generation depends mostly on memory bandwidth, and the 5090 led the 5080 by about 54% to 77% across three models. Memory size decides what runs at all.

  • Token generation is how fast words appear; prompt processing is how fast the card reads your input.
  • Puget Systems measured 248.0 tokens per second on the RTX 5090 and 112.5 on the RX 9070 XT in one MLPerf Client test.
  • Results from CUDA, Vulkan and ROCm are not interchangeable.
  • Community scoreboards show direction, not exact rankings.

What the numbers measure#

A language model produces text one token at a time. A token is a short piece of a word. Benchmarks report two main rates.

Token generation, often written tg, is how many new tokens the card produces each second after the first. It decides how fast a reply streams onto your screen. Most of the time it is limited by memory bandwidth, because every new token needs the model's weights read from memory.

Prompt processing, written pp, is how fast the card reads the text you send in before it starts answering. It leans on raw compute and matters for long documents, retrieval and coding assistants that send whole files. Time to first token, the wait before the first word appears, mixes the two.

Two more settings change every figure. Quantisation shrinks a model to 4-bit or 8-bit weights, which cuts the memory it needs and raises speed. Context length is how much text the model holds in mind. Longer context slows generation and uses more memory, so a test at 16K context reads lower than one at a few hundred tokens.

How to read a benchmark

Check five things before you trust a number.

  1. The model and quantisation. A figure for an 8B model says little about a 30B one.
  2. The context length. Compare only at the same depth.
  3. The backend. CUDA, Vulkan, ROCm and DirectML are not comparable with each other.
  4. The card. Reference cards and partner cards can differ in clocks and cooling.
  5. The date and drivers. Early results on a new card often improve. Puget's launch-window numbers came with a warning that they may be out of date.

Where a source does not state a detail, leave it out of your conclusion. No chart here settles which board partner is best.

Where the numbers come from#

No single outlet tests every card. The figures in this article come from four kinds of source, and each answers a different question.

Puget Systems is a workstation builder that publishes lab reviews. Its roundup ran MLPerf Client, an industry benchmark for language-model speed, on the RTX 5090, 5080 and 5070 Ti and on two Radeon cards, with the same processor and test platform for each. It does not name the model on its chart.

Hardware Corner publishes a GPU ranking for local LLMs built on llama.cpp's llama-bench tool, run on Ubuntu with CUDA 12.8 and named open models at 16K context. It covers NVIDIA cards only.

Tom's Hardware ran UL Procyon and an older MLPerf Client build in its Radeon RX 9070 XT review. Back2Gaming ran llama.cpp and ComfyUI on Linux in its review of a 32 GB Radeon workstation card. Both include the RTX 5070 Ti and Radeon cards on the same software.

The llama.cpp community keeps public scoreboards for CUDA, Vulkan and ROCm. Volunteers submit runs from different boards, drivers and builds, so these are community-reported and useful for direction only.

Token generation across tiers#

Hardware Corner's ranking gives the cleanest NVIDIA-only view. It used the same software and the same context length on every card.

gpt-oss 20B 4-bit, token generation at 16K context: token_generation (tokens/s, higher is better)

gpt-oss 20B 4-bit, token generation at 16K context: token_generationINNO3D GeForce RTX 5090 X3 OC 32GB: 249.2 tokens/sINNO3D GeForce RTX 5090 X3 OC 32GB249.2 tokens/sASUS TUF Gaming GeForce RTX 5080 16GB OC: 140.5 tokens/sASUS TUF Gaming GeForce RTX 5080 16GB OC140.5 tokens/sASUS Prime GeForce RTX 5070 Ti 16GB OC: 133.1 tokens/sASUS Prime GeForce RTX 5070 Ti 16GB OC133.1 tokens/sASUS Prime GeForce RTX 5070 Ti 16GB GDDR7 OC: 133.1 tokens/sASUS Prime GeForce RTX 5070 Ti 16GB GDDR7 OC133.1 tokens/s

4 products. Highest: INNO3D GeForce RTX 5090 X3 OC 32GB, 249.19 tokens/s. Lowest: ASUS Prime GeForce RTX 5070 Ti 16GB GDDR7 OC, 133.05 tokens/s.

gpt-oss 20B 4-bit, token generation at 16K context: token_generation
Producttokens/s
INNO3D GeForce RTX 5090 X3 OC 32GB249.19
ASUS TUF Gaming GeForce RTX 5080 16GB OC140.48
ASUS Prime GeForce RTX 5070 Ti 16GB OC133.05
ASUS Prime GeForce RTX 5070 Ti 16GB GDDR7 OC133.05
Source: Hardware Corner (opens in a new tab). llama-bench, Ubuntu 24.04, CUDA 12.8, Q4_K_XL, 16K context depth. Chip-level result mapped to the listed model; Page 'Updated: December 9, 2025'. Hardware Corner does not say whether reference/Founders or partner cards were used. Their own measurements on Ubuntu 24.04, llama.cpp llama-bench, CUDA 12.8, Q4_K_XL quantisation; driver version not stated. Page also lists prices (not recorded). Product label on page: 'RTX Pro 6000 WS'.

About this data: figures are copied from the named source, not measured by us, and the test system and settings are those the source states above. See all benchmark sources and credits.

Taking gpt-oss 20B at 4-bit, the table gives the RTX 5090 249.19 tokens per second, 140.48 for the RTX 5080 and 133.05 for the RTX 5070 Ti.

Qwen3 14B 4-bit, token generation at 16K context: token_generation (tokens/s, higher is better)

Qwen3 14B 4-bit, token generation at 16K context: token_generationINNO3D GeForce RTX 5090 X3 OC 32GB: 102.7 tokens/sINNO3D GeForce RTX 5090 X3 OC 32GB102.7 tokens/sASUS TUF Gaming GeForce RTX 5080 16GB OC: 64 tokens/sASUS TUF Gaming GeForce RTX 5080 16GB OC64 tokens/sASUS Prime GeForce RTX 5070 Ti 16GB OC: 58 tokens/sASUS Prime GeForce RTX 5070 Ti 16GB OC58 tokens/s

3 products. Highest: INNO3D GeForce RTX 5090 X3 OC 32GB, 102.68 tokens/s. Lowest: ASUS Prime GeForce RTX 5070 Ti 16GB OC, 57.98 tokens/s.

Qwen3 14B 4-bit, token generation at 16K context: token_generation
Producttokens/s
INNO3D GeForce RTX 5090 X3 OC 32GB102.68
ASUS TUF Gaming GeForce RTX 5080 16GB OC64.04
ASUS Prime GeForce RTX 5070 Ti 16GB OC57.98
Source: Hardware Corner (opens in a new tab). llama-bench, Ubuntu 24.04, CUDA 12.8, Q4_K_XL, 16K context depth. Chip-level result mapped to the listed model; Page 'Updated: December 9, 2025'. Hardware Corner does not say whether reference/Founders or partner cards were used. Their own measurements on Ubuntu 24.04, llama.cpp llama-bench, CUDA 12.8, Q4_K_XL quantisation; driver version not stated. Page also lists prices (not recorded). Product label on page: 'RTX Pro 6000 WS'.

About this data: figures are copied from the named source, not measured by us, and the test system and settings are those the source states above. See all benchmark sources and credits.

On Qwen3 14B the figures are 102.68, 64.04 and 57.98. On Qwen3 8B they are 145.34, 94.14 and 87.54.

Table
Model (4-bit, 16K context)RTX 5090RTX 5080RTX 5070 Ti
Qwen3 8B145.3494.1487.54
Qwen3 14B102.6864.0457.98
gpt-oss 20B249.19140.48133.05

All figures are tokens per second from Hardware Corner. The pattern is easy to see. The RTX 5090 leads the RTX 5080 by about 54% to 77%, and the RTX 5080 leads the RTX 5070 Ti by about 6% to 10%. The size of the gap varies by model, so no single figure fits every workload.

Puget Systems' MLPerf Client chart gives a cross-vendor view with one lab setup.

MLPerf Client v1.0 LLM tests, 2nd+ token generation rate (model not named on chart): token_generation_2nd_plus (tokens/s, higher is better)

MLPerf Client v1.0 LLM tests, 2nd+ token generation rate (model not named on chart): token_generation_2nd_plusINNO3D GeForce RTX 5090 X3 OC 32GB: 248 tokens/sINNO3D GeForce RTX 5090 X3 OC 32GB248 tokens/sASUS TUF Gaming GeForce RTX 5080 16GB OC: 168.7 tokens/sASUS TUF Gaming GeForce RTX 5080 16GB OC168.7 tokens/sASUS Prime GeForce RTX 5070 Ti 16GB OC: 157.6 tokens/sASUS Prime GeForce RTX 5070 Ti 16GB OC157.6 tokens/sASUS Prime GeForce RTX 5070 Ti 16GB GDDR7 OC: 157.6 tokens/sASUS Prime GeForce RTX 5070 Ti 16GB GDDR7 OC157.6 tokens/sASUS TUF Gaming Radeon RX 9070 XT 16GB OC: 112.5 tokens/sASUS TUF Gaming Radeon RX 9070 XT 16GB OC112.5 tokens/s

5 products. Highest: INNO3D GeForce RTX 5090 X3 OC 32GB, 248 tokens/s. Lowest: ASUS TUF Gaming Radeon RX 9070 XT 16GB OC, 112.5 tokens/s.

MLPerf Client v1.0 LLM tests, 2nd+ token generation rate (model not named on chart): token_generation_2nd_plus
Producttokens/s
INNO3D GeForce RTX 5090 X3 OC 32GB248
ASUS TUF Gaming GeForce RTX 5080 16GB OC168.7
ASUS Prime GeForce RTX 5070 Ti 16GB OC157.6
ASUS Prime GeForce RTX 5070 Ti 16GB GDDR7 OC157.6
ASUS TUF Gaming Radeon RX 9070 XT 16GB OC112.5
Source: Puget Systems (opens in a new tab). MLPerf Client v1.0; NVIDIA/AMD via ONNX (DirectML path); Windows 11 Pro 26100; NVIDIA driver 577.00; AMD Adrenalin 25.8.1; Ryzen 9 9950X3D; Balanced power profile. Chip-level result mapped to the listed model; Published 2025-09-25. Values read from chart bar labels. NVIDIA 5090/5080/4090 listed by Puget only as "NVIDIA GeForce RTX" (reference vs partner not stated); 5070 Ti was ASUS PRIME OC (partner); 9070 XT was Sapphire Nitro+ OC (partner); 7900 XTX listed as "AMD Radeon RX 7900 XTX" (variant not stated).

About this data: figures are copied from the named source, not measured by us, and the test system and settings are those the source states above. See all benchmark sources and credits.

The chart shows 248.0 for the RTX 5090, 168.7 for the RTX 5080, 157.6 for the RTX 5070 Ti, then 112.5 for the RX 9070 XT, all in tokens per second. The RTX 5070 Ti leads the Radeon card by about 40% in this run. That test uses a Windows software path for both brands, which does not suit every Radeon card equally.

Prompt processing and long context

Prompt processing results are tricky for new cards. Puget flagged its own launch-window prompt-processing numbers for the RTX 5090 and RTX 5080 as unreliable and suspected a driver problem. Hardware Corner's table is more consistent.

Qwen3 14B 4-bit, prompt processing at 16K context: prompt_processing (tokens/s, higher is better)

Qwen3 14B 4-bit, prompt processing at 16K context: prompt_processingINNO3D GeForce RTX 5090 X3 OC 32GB: 4473.1 tokens/sINNO3D GeForce RTX 5090 X3 OC 32GB4473.1 tokens/sASUS TUF Gaming GeForce RTX 5080 16GB OC: 2542 tokens/sASUS TUF Gaming GeForce RTX 5080 16GB OC2542 tokens/sASUS Prime GeForce RTX 5070 Ti 16GB OC: 2303.9 tokens/sASUS Prime GeForce RTX 5070 Ti 16GB OC2303.9 tokens/sASUS Prime GeForce RTX 5070 Ti 16GB GDDR7 OC: 2303.9 tokens/sASUS Prime GeForce RTX 5070 Ti 16GB GDDR7 OC2303.9 tokens/s

4 products. Highest: INNO3D GeForce RTX 5090 X3 OC 32GB, 4473.08 tokens/s. Lowest: ASUS Prime GeForce RTX 5070 Ti 16GB GDDR7 OC, 2303.9 tokens/s.

Qwen3 14B 4-bit, prompt processing at 16K context: prompt_processing
Producttokens/s
INNO3D GeForce RTX 5090 X3 OC 32GB4473.08
ASUS TUF Gaming GeForce RTX 5080 16GB OC2542.01
ASUS Prime GeForce RTX 5070 Ti 16GB OC2303.9
ASUS Prime GeForce RTX 5070 Ti 16GB GDDR7 OC2303.9
Source: Hardware Corner (opens in a new tab). llama-bench, Ubuntu 24.04, CUDA 12.8, Q4_K_XL, 16K context depth. Chip-level result mapped to the listed model; Page 'Updated: December 9, 2025'. Hardware Corner does not say whether reference/Founders or partner cards were used. Their own measurements on Ubuntu 24.04, llama.cpp llama-bench, CUDA 12.8, Q4_K_XL quantisation; driver version not stated. Page also lists prices (not recorded). Product label on page: 'RTX Pro 6000 WS'.

About this data: figures are copied from the named source, not measured by us, and the test system and settings are those the source states above. See all benchmark sources and credits.

On Qwen3 14B at 16K context the RTX 5090 reads 4473.08 tokens per second, the RTX 5080 2542.01 and the RTX 5070 Ti 2303.9. Prompt processing speeds are far higher than generation speeds, because the card handles many tokens in parallel. For tasks that read long inputs, such as summarising a report, this number sets how long you wait before the answer begins.

Context length also hits generation. Hardware Corner lists Qwen3 8B on the RTX 5090 at 145.34 tokens per second at 16K context and 111.91 at 32K. Longer context costs speed on every card, so compare numbers at the same depth.

CUDA, Vulkan and ROCm are different roads#

A backend is the software path that talks to the card. llama.cpp offers CUDA for NVIDIA, HIP (AMD's ROCm) and Vulkan, a vendor-neutral option. The same card can post different speeds on different backends, so a chart that mixes them misleads.

The community scoreboards use Llama 2 at 7B size, quantised to 4-bit, with flash attention on. Each backend sits in its own thread with its own volunteers, so the table below is community-reported and shows direction only.

Table
CardCUDAVulkanROCm
RTX 5090300.4273.68n/a
RTX 5080184.68192.17n/a
RTX 5070 Ti182.43176.18n/a
Radeon AI PRO R9700n/a152.797.98
RX 9070 XTn/a142.097.28

All figures are tokens per second on the token-generation test, community-reported. They come from three separate threads, with different volunteers, drivers and builds, so the columns are not a controlled comparison and small gaps are noise.

The firmer evidence comes from Phoronix. In its Lemonade tests on a Radeon AI PRO R9700, Vulkan gave more tokens per second, while ROCm gave lower latency to the first token. The community threads point the same way for Radeon cards, though they cannot prove it. Backend choice can change a result as much as the card does. Our CUDA vs ROCm explainer covers why the software matters beyond speed.

llama-bench, Llama 2 7B Q4_0, CUDA backend, tg128, flash attention on: token_generation_tg128 (tokens/s, higher is better)

llama-bench, Llama 2 7B Q4_0, CUDA backend, tg128, flash attention on: token_generation_tg128INNO3D GeForce RTX 5090 X3 OC 32GB: 300.4 tokens/sINNO3D GeForce RTX 5090 X3 OC 32GB300.4 tokens/sNVIDIA RTX PRO 5000 Blackwell 48GB: 226.3 tokens/sNVIDIA RTX PRO 5000 Blackwell 48GB226.3 tokens/sASUS TUF Gaming GeForce RTX 5080 16GB OC: 184.7 tokens/sASUS TUF Gaming GeForce RTX 5080 16GB OC184.7 tokens/sASUS Prime GeForce RTX 5070 Ti 16GB OC: 182.4 tokens/sASUS Prime GeForce RTX 5070 Ti 16GB OC182.4 tokens/s

4 products. Highest: INNO3D GeForce RTX 5090 X3 OC 32GB, 300.4 tokens/s. Lowest: ASUS Prime GeForce RTX 5070 Ti 16GB OC, 182.43 tokens/s.

llama-bench, Llama 2 7B Q4_0, CUDA backend, tg128, flash attention on: token_generation_tg128
Producttokens/s
INNO3D GeForce RTX 5090 X3 OC 32GB300.4
NVIDIA RTX PRO 5000 Blackwell 48GB226.31
ASUS TUF Gaming GeForce RTX 5080 16GB OC184.68
ASUS Prime GeForce RTX 5070 Ti 16GB OC182.43
Source: llama.cpp GitHub discussion (community-reported) (opens in a new tab). llama-bench -ngl 99/100 -fa 1. Chip-level result mapped to the listed model; commit 8cf6b42; Community-reported (support only). Scoreboard table in the discussion post, last edited 2026-09-06, read 2026-11-17. Each row is a different volunteer with own board maker, driver, OS and commit; not a controlled test; memory per scoreboard: 32 GB / GDDR7 / 512 bit.

About this data: figures are copied from the named source, not measured by us, and the test system and settings are those the source states above. See all benchmark sources and credits.

The CUDA scoreboard shows the same order as the lab tests, with the RTX 5090 first and the other two NVIDIA cards close together.

GPU Benchmarks for Local LLMs (last checked November 2026).

Capacity beats speed#

A fast card that cannot hold the model loses to a slower card that can. Back2Gaming's test shows it.

Qwen3.8-27B (UD-IQ4_XS) chat decode, 512-token answer: token_generation_decode (tokens/s, higher is better)

Qwen3.8-27B (UD-IQ4_XS) chat decode, 512-token answer: token_generation_decodePowerColor Radeon AI PRO R9700 32GB: 31.7 tokens/sPowerColor Radeon AI PRO R9700 32GB31.7 tokens/sASUS Prime GeForce RTX 5070 Ti 16GB OC: 30.3 tokens/sASUS Prime GeForce RTX 5070 Ti 16GB OC30.3 tokens/sASUS TUF Gaming Radeon RX 9070 XT 16GB OC: 27.2 tokens/sASUS TUF Gaming Radeon RX 9070 XT 16GB OC27.2 tokens/s

3 products. Highest: PowerColor Radeon AI PRO R9700 32GB, 31.7 tokens/s. Lowest: ASUS TUF Gaming Radeon RX 9070 XT 16GB OC, 27.2 tokens/s.

Qwen3.8-27B (UD-IQ4_XS) chat decode, 512-token answer: token_generation_decode
Producttokens/s
PowerColor Radeon AI PRO R9700 32GB31.7
ASUS Prime GeForce RTX 5070 Ti 16GB OC30.3
ASUS TUF Gaming Radeon RX 9070 XT 16GB OC27.2
Source: Back2Gaming (opens in a new tab). 32,768-token context, flash attention on, fixed-seed greedy sampling; 10-minute steady-state run. Chip-level result mapped to the listed model; Published 2026-09-23. Single small outlet; Ubuntu 24.04.4, ROCm 7.2.4 on Radeon, NVIDIA driver 595.91.07 CUDA 13.2, llama.cpp build 11046, Ryzen 7 9800X3D. R9700 was a PowerColor AI PRO R9700 (AMD reference-style blower, loaner); brand of the 5070 Ti and 9070 XT not stated. Values read from article text; the 16GB cards had to offload layers to CPU (4 of 66 on 5070 Ti, 2 of 66 on 9070 XT), so this reflects VRAM capacity, not raw speed.

About this data: figures are copied from the named source, not measured by us, and the test system and settings are those the source states above. See all benchmark sources and credits.

With a 27B-parameter model at 4-bit and a 32,768-token context, the Radeon AI PRO R9700 with 32 GB reached 31.7 tokens per second. The RTX 5070 Ti reached 30.3 and the RX 9070 XT 27.2. The two 16 GB cards had to offload layers to system memory, 4 of 66 on the RTX 5070 Ti and 2 of 66 on the RX 9070 XT. The 32 GB card held everything.

On a smaller 9B model that fits on every card, the picture reverses.

Qwen3.5-9B (Q4_K_M) chat decode, 512-token answer: token_generation_decode (tokens/s, higher is better)

Qwen3.5-9B (Q4_K_M) chat decode, 512-token answer: token_generation_decodeASUS Prime GeForce RTX 5070 Ti 16GB OC: 124.1 tokens/sASUS Prime GeForce RTX 5070 Ti 16GB OC124.1 tokens/sASUS TUF Gaming Radeon RX 9070 XT 16GB OC: 84.2 tokens/sASUS TUF Gaming Radeon RX 9070 XT 16GB OC84.2 tokens/sPowerColor Radeon AI PRO R9700 32GB: 83.4 tokens/sPowerColor Radeon AI PRO R9700 32GB83.4 tokens/s

3 products. Highest: ASUS Prime GeForce RTX 5070 Ti 16GB OC, 124.1 tokens/s. Lowest: PowerColor Radeon AI PRO R9700 32GB, 83.4 tokens/s.

Qwen3.5-9B (Q4_K_M) chat decode, 512-token answer: token_generation_decode
Producttokens/s
ASUS Prime GeForce RTX 5070 Ti 16GB OC124.1
ASUS TUF Gaming Radeon RX 9070 XT 16GB OC84.2
PowerColor Radeon AI PRO R9700 32GB83.4
Source: Back2Gaming (opens in a new tab). 32,768-token context, flash attention on, fixed-seed greedy sampling; 10-minute steady-state run. Chip-level result mapped to the listed model; Published 2026-09-23. Single small outlet; Ubuntu 24.04.4, ROCm 7.2.4 on Radeon, NVIDIA driver 595.91.07 CUDA 13.2, llama.cpp build 11046, Ryzen 7 9800X3D. R9700 was a PowerColor AI PRO R9700 (AMD reference-style blower, loaner); brand of the 5070 Ti and 9070 XT not stated. Values read from article text; model fits in VRAM on all three.

About this data: figures are copied from the named source, not measured by us, and the test system and settings are those the source states above. See all benchmark sources and credits.

The RTX 5070 Ti reached 124.1 tokens per second, the RX 9070 XT 84.2 and the R9700 83.4. The same three cards swap places depending on whether the model fits. That is why our VRAM explainer comes before any speed ranking in the buying process.

Image generation benchmarks

Image models have their own tests. Tom's Hardware ran UL Procyon's Stable Diffusion XL test on the RTX 5070 Ti and RX 9070 XT.

Procyon AI Image Generation, Stable Diffusion XL, benchmark score: benchmark_score (score, higher is better)

Procyon AI Image Generation, Stable Diffusion XL, benchmark score: benchmark_scoreASUS Prime GeForce RTX 5070 Ti 16GB OC: 3265 scoreASUS Prime GeForce RTX 5070 Ti 16GB OC3265 scoreASUS TUF Gaming Radeon RX 9070 XT 16GB OC: 1969 scoreASUS TUF Gaming Radeon RX 9070 XT 16GB OC1969 score

2 products. Highest: ASUS Prime GeForce RTX 5070 Ti 16GB OC, 3265 score. Lowest: ASUS TUF Gaming Radeon RX 9070 XT 16GB OC, 1969 score.

Procyon AI Image Generation, Stable Diffusion XL, benchmark score: benchmark_score
Productscore
ASUS Prime GeForce RTX 5070 Ti 16GB OC3265
ASUS TUF Gaming Radeon RX 9070 XT 16GB OC1969
Source: Tom's Hardware (opens in a new tab). Procyon (UL), best available engine per GPU. Chip-level result mapped to the listed model; Published 2025-03-05. Values read from chart bar labels. Cards: Asus RTX 5070 Ti Prime (partner), PowerColor Reaper RX 9070 XT (partner), RX 7900 XTX MBA reference. Windows 11 24H2; AMD preview driver 24.30.31.03 for 9070 cards, 25.2.1 for 7000 series; Nvidia 572-family drivers; Ryzen 7 9800X3D. Tom's used the best available variant (TensorRT/DirectML/OpenVINO, FP32/FP16/INT8) per GPU.

About this data: figures are copied from the named source, not measured by us, and the test system and settings are those the source states above. See all benchmark sources and credits.

The RTX 5070 Ti scored 3265 against 1969 for the Radeon card. Back2Gaming's ComfyUI test, SDXL at 1024 x 1024 and 30 steps, gave 9.84 images per minute for the RTX 5070 Ti, 9.03 for the R9700 and 8.22 for the RX 9070 XT. Image tests are not interchangeable with text tests, and launch-window results can shift with drivers.

GPU Benchmarks for Local LLMs (last checked November 2026).

What the benchmarks leave out#

Speed charts leave out the parts of a machine that decide whether you can use a card at all. Power is the first. NVIDIA puts the RTX 5090 at 575 W total graphics power, the RTX 5080 at 360 W and the RTX 5070 Ti at 300 W. INNO3D gives 1000 W as the minimum for the system around its RTX 5090 card, and ASUS recommends 850 W for its RTX 5080 and 750 W for its RTX 5070 Ti. A card that tops a chart can still need a new power supply and a larger case.

Heat is the second. Long inference or fine-tuning jobs hold a card near its limit for hours. A card that draws more power also puts more heat into the room, so airflow counts as much as the tokens per second in a chart.

Software is the third. A benchmark run on one backend says nothing about the tools you will use. If you plan to run Ollama, ComfyUI or PyTorch on Windows, check that your card is supported on that operating system before you pick on speed.

Matching the numbers to your use

Different jobs lean on different columns of the chart.

Matching the numbers to your use
UseNumber to watchWhy
Chat with a local modelToken generationIt sets how quickly the reply streams
Summarising long filesPrompt processingThe card reads the whole file first
Retrieval over many documentsPrompt processing and VRAMLong inputs need speed and memory
Image generationImage-model scoresText rates do not predict image speed
Running a large modelVRAM firstA model that does not fit runs slowly

Use the table to decide which chart to trust. A buyer who mostly chats can read the generation charts. A buyer who feeds in long documents should weight the prompt-processing charts and the memory size more heavily.

What this means when buying#

Pick the memory size first, then the speed tier. When 16 GB is enough, the RTX 5070 Ti and RTX 5080 are the NVIDIA options, and our RTX 5080 vs RTX 5070 Ti comparison shows the gap is single-digit to low double-digit percent. If you need more than 16 GB, the RTX 5090 is the fastest, and the best high VRAM GPU guide covers the 32 GB and 48 GB options.

For AMD cards, our RTX 5070 Ti vs RX 9070 XT comparison covers the trade-off. Our best GPU for local LLMs and the GPU category page list more choices. Three cards from the benchmarks above, with maker pages: the INNO3D RTX 5090 X3 OC (opens in a new tab), the ASUS TUF RTX 5080 (opens in a new tab) and the ASUS Prime RTX 5070 Ti (opens in a new tab).

Benchmarks come from overseas testers, so check how Indian retailers list the same card, as partner models sold in India can differ in clocks and cooling from the tested units.

Frequently asked questions#

What is a good tokens-per-second rate for a local LLM?

Anything that streams faster than you can read feels smooth. Rates of a few tens of tokens per second already do that for chat. Higher rates matter more for long answers, coding assistants and agents.

Why do two sites show different numbers for the same card?

They use different models, quantisation, context lengths, backends and drivers. Compare figures only inside one chart from one source, and treat a number from another site as a rough guide, not as a like-for-like result.

Is prompt processing or token generation more important?

Generation matters for chat. Prompt processing matters when you feed in long documents or code. Check both, because a card that reads long prompts quickly can still feel slow if its token generation is weak.

How far can I trust the community llama.cpp scoreboards?

They show direction. Each row is a different volunteer with a different board and commit, so small gaps are noise. Use them as a cross-check next to a controlled test, and not as the only basis for a choice.

Do Radeon cards run local LLMs well?

They run them, often best through Vulkan in llama.cpp. They trail NVIDIA cards of the same tier in the tests cited. Software support is the main cost, so check your tools before you pick an AMD card.

Should I compare CUDA and Vulkan results directly?

No. They are different software paths, and community scoreboards use different volunteers, drivers and builds for each. Compare cards on one backend from one source.

Does more VRAM make a model faster?

Not directly. More VRAM lets a model fit entirely on the card, which avoids the slowdown of moving layers to system memory. Once the model fits, speed depends mostly on memory bandwidth and the chip.

Sources & credits#

We have not tested these products ourselves. Specs come from the makers; benchmark figures come from the sources we name.

Facts and figures in this article come from these sources. Articles are compiled from the makers' published specifications and the benchmarks named above, and checked by the NextAIGrid Editorial Team.

Disclaimer

This article is for information only. Benchmark results change with drivers and software, and availability changes, so check the maker's page before you buy. Found an error? Tell us.

How was this explainer put together?

It draws on 4 named sources: maker specification pages and published third-party benchmarks, listed in the article. We have not tested these products ourselves. Specs come from the makers; benchmark figures come from the sources we name. Specifications and sources were last checked on 17 November 2026. Read how we research and our editorial policy.

Spotted an error? Tell us. Confirmed fixes are listed on the corrections page.

What questions do readers ask next?

Back to top