In this article
Short answer
For local LLMs, published benchmarks agree on one order: the RTX 5090 first, then the RTX 5080, the RTX 5070 Ti, and the Radeon cards behind them on the same software. Token generation depends mostly on memory bandwidth, and the 5090 led the 5080 by about 54% to 77% across three models. Memory size decides what runs at all.
- Token generation is how fast words appear; prompt processing is how fast the card reads your input.
- Puget Systems measured 248.0 tokens per second on the RTX 5090 and 112.5 on the RX 9070 XT in one MLPerf Client test.
- Results from CUDA, Vulkan and ROCm are not interchangeable.
- Community scoreboards show direction, not exact rankings.
What the numbers measure#
A language model produces text one token at a time. A token is a short piece of a word. Benchmarks report two main rates.
Token generation, often written tg, is how many new tokens the card produces each second after the first. It decides how fast a reply streams onto your screen. Most of the time it is limited by memory bandwidth, because every new token needs the model's weights read from memory.
Prompt processing, written pp, is how fast the card reads the text you send in before it starts answering. It leans on raw compute and matters for long documents, retrieval and coding assistants that send whole files. Time to first token, the wait before the first word appears, mixes the two.
Two more settings change every figure. Quantisation shrinks a model to 4-bit or 8-bit weights, which cuts the memory it needs and raises speed. Context length is how much text the model holds in mind. Longer context slows generation and uses more memory, so a test at 16K context reads lower than one at a few hundred tokens.
How to read a benchmark
Check five things before you trust a number.
- The model and quantisation. A figure for an 8B model says little about a 30B one.
- The context length. Compare only at the same depth.
- The backend. CUDA, Vulkan, ROCm and DirectML are not comparable with each other.
- The card. Reference cards and partner cards can differ in clocks and cooling.
- The date and drivers. Early results on a new card often improve. Puget's launch-window numbers came with a warning that they may be out of date.
Where a source does not state a detail, leave it out of your conclusion. No chart here settles which board partner is best.
Where the numbers come from#
No single outlet tests every card. The figures in this article come from four kinds of source, and each answers a different question.
Puget Systems is a workstation builder that publishes lab reviews. Its roundup ran MLPerf Client, an industry benchmark for language-model speed, on the RTX 5090, 5080 and 5070 Ti and on two Radeon cards, with the same processor and test platform for each. It does not name the model on its chart.
Hardware Corner publishes a GPU ranking for local LLMs built on llama.cpp's llama-bench tool, run on Ubuntu with CUDA 12.8 and named open models at 16K context. It covers NVIDIA cards only.
Tom's Hardware ran UL Procyon and an older MLPerf Client build in its Radeon RX 9070 XT review. Back2Gaming ran llama.cpp and ComfyUI on Linux in its review of a 32 GB Radeon workstation card. Both include the RTX 5070 Ti and Radeon cards on the same software.
The llama.cpp community keeps public scoreboards for CUDA, Vulkan and ROCm. Volunteers submit runs from different boards, drivers and builds, so these are community-reported and useful for direction only.
Token generation across tiers#
Hardware Corner's ranking gives the cleanest NVIDIA-only view. It used the same software and the same context length on every card.
gpt-oss 20B 4-bit, token generation at 16K context: token_generation (tokens/s, higher is better)
4 products. Highest: INNO3D GeForce RTX 5090 X3 OC 32GB, 249.19 tokens/s. Lowest: ASUS Prime GeForce RTX 5070 Ti 16GB GDDR7 OC, 133.05 tokens/s.
| Product | tokens/s |
|---|---|
| INNO3D GeForce RTX 5090 X3 OC 32GB | 249.19 |
| ASUS TUF Gaming GeForce RTX 5080 16GB OC | 140.48 |
| ASUS Prime GeForce RTX 5070 Ti 16GB OC | 133.05 |
| ASUS Prime GeForce RTX 5070 Ti 16GB GDDR7 OC | 133.05 |
About this data: figures are copied from the named source, not measured by us, and the test system and settings are those the source states above. See all benchmark sources and credits.
Taking gpt-oss 20B at 4-bit, the table gives the RTX 5090 249.19 tokens per second, 140.48 for the RTX 5080 and 133.05 for the RTX 5070 Ti.
Qwen3 14B 4-bit, token generation at 16K context: token_generation (tokens/s, higher is better)
3 products. Highest: INNO3D GeForce RTX 5090 X3 OC 32GB, 102.68 tokens/s. Lowest: ASUS Prime GeForce RTX 5070 Ti 16GB OC, 57.98 tokens/s.
| Product | tokens/s |
|---|---|
| INNO3D GeForce RTX 5090 X3 OC 32GB | 102.68 |
| ASUS TUF Gaming GeForce RTX 5080 16GB OC | 64.04 |
| ASUS Prime GeForce RTX 5070 Ti 16GB OC | 57.98 |
About this data: figures are copied from the named source, not measured by us, and the test system and settings are those the source states above. See all benchmark sources and credits.
On Qwen3 14B the figures are 102.68, 64.04 and 57.98. On Qwen3 8B they are 145.34, 94.14 and 87.54.
| Model (4-bit, 16K context) | RTX 5090 | RTX 5080 | RTX 5070 Ti |
|---|---|---|---|
| Qwen3 8B | 145.34 | 94.14 | 87.54 |
| Qwen3 14B | 102.68 | 64.04 | 57.98 |
| gpt-oss 20B | 249.19 | 140.48 | 133.05 |
All figures are tokens per second from Hardware Corner. The pattern is easy to see. The RTX 5090 leads the RTX 5080 by about 54% to 77%, and the RTX 5080 leads the RTX 5070 Ti by about 6% to 10%. The size of the gap varies by model, so no single figure fits every workload.
Puget Systems' MLPerf Client chart gives a cross-vendor view with one lab setup.
MLPerf Client v1.0 LLM tests, 2nd+ token generation rate (model not named on chart): token_generation_2nd_plus (tokens/s, higher is better)
5 products. Highest: INNO3D GeForce RTX 5090 X3 OC 32GB, 248 tokens/s. Lowest: ASUS TUF Gaming Radeon RX 9070 XT 16GB OC, 112.5 tokens/s.
| Product | tokens/s |
|---|---|
| INNO3D GeForce RTX 5090 X3 OC 32GB | 248 |
| ASUS TUF Gaming GeForce RTX 5080 16GB OC | 168.7 |
| ASUS Prime GeForce RTX 5070 Ti 16GB OC | 157.6 |
| ASUS Prime GeForce RTX 5070 Ti 16GB GDDR7 OC | 157.6 |
| ASUS TUF Gaming Radeon RX 9070 XT 16GB OC | 112.5 |
About this data: figures are copied from the named source, not measured by us, and the test system and settings are those the source states above. See all benchmark sources and credits.
The chart shows 248.0 for the RTX 5090, 168.7 for the RTX 5080, 157.6 for the RTX 5070 Ti, then 112.5 for the RX 9070 XT, all in tokens per second. The RTX 5070 Ti leads the Radeon card by about 40% in this run. That test uses a Windows software path for both brands, which does not suit every Radeon card equally.
Prompt processing and long context
Prompt processing results are tricky for new cards. Puget flagged its own launch-window prompt-processing numbers for the RTX 5090 and RTX 5080 as unreliable and suspected a driver problem. Hardware Corner's table is more consistent.
Qwen3 14B 4-bit, prompt processing at 16K context: prompt_processing (tokens/s, higher is better)
4 products. Highest: INNO3D GeForce RTX 5090 X3 OC 32GB, 4473.08 tokens/s. Lowest: ASUS Prime GeForce RTX 5070 Ti 16GB GDDR7 OC, 2303.9 tokens/s.
| Product | tokens/s |
|---|---|
| INNO3D GeForce RTX 5090 X3 OC 32GB | 4473.08 |
| ASUS TUF Gaming GeForce RTX 5080 16GB OC | 2542.01 |
| ASUS Prime GeForce RTX 5070 Ti 16GB OC | 2303.9 |
| ASUS Prime GeForce RTX 5070 Ti 16GB GDDR7 OC | 2303.9 |
About this data: figures are copied from the named source, not measured by us, and the test system and settings are those the source states above. See all benchmark sources and credits.
On Qwen3 14B at 16K context the RTX 5090 reads 4473.08 tokens per second, the RTX 5080 2542.01 and the RTX 5070 Ti 2303.9. Prompt processing speeds are far higher than generation speeds, because the card handles many tokens in parallel. For tasks that read long inputs, such as summarising a report, this number sets how long you wait before the answer begins.
Context length also hits generation. Hardware Corner lists Qwen3 8B on the RTX 5090 at 145.34 tokens per second at 16K context and 111.91 at 32K. Longer context costs speed on every card, so compare numbers at the same depth.
CUDA, Vulkan and ROCm are different roads#
A backend is the software path that talks to the card. llama.cpp offers CUDA for NVIDIA, HIP (AMD's ROCm) and Vulkan, a vendor-neutral option. The same card can post different speeds on different backends, so a chart that mixes them misleads.
The community scoreboards use Llama 2 at 7B size, quantised to 4-bit, with flash attention on. Each backend sits in its own thread with its own volunteers, so the table below is community-reported and shows direction only.
| Card | CUDA | Vulkan | ROCm |
|---|---|---|---|
| RTX 5090 | 300.4 | 273.68 | n/a |
| RTX 5080 | 184.68 | 192.17 | n/a |
| RTX 5070 Ti | 182.43 | 176.18 | n/a |
| Radeon AI PRO R9700 | n/a | 152.7 | 97.98 |
| RX 9070 XT | n/a | 142.0 | 97.28 |
All figures are tokens per second on the token-generation test, community-reported. They come from three separate threads, with different volunteers, drivers and builds, so the columns are not a controlled comparison and small gaps are noise.
The firmer evidence comes from Phoronix. In its Lemonade tests on a Radeon AI PRO R9700, Vulkan gave more tokens per second, while ROCm gave lower latency to the first token. The community threads point the same way for Radeon cards, though they cannot prove it. Backend choice can change a result as much as the card does. Our CUDA vs ROCm explainer covers why the software matters beyond speed.
llama-bench, Llama 2 7B Q4_0, CUDA backend, tg128, flash attention on: token_generation_tg128 (tokens/s, higher is better)
4 products. Highest: INNO3D GeForce RTX 5090 X3 OC 32GB, 300.4 tokens/s. Lowest: ASUS Prime GeForce RTX 5070 Ti 16GB OC, 182.43 tokens/s.
| Product | tokens/s |
|---|---|
| INNO3D GeForce RTX 5090 X3 OC 32GB | 300.4 |
| NVIDIA RTX PRO 5000 Blackwell 48GB | 226.31 |
| ASUS TUF Gaming GeForce RTX 5080 16GB OC | 184.68 |
| ASUS Prime GeForce RTX 5070 Ti 16GB OC | 182.43 |
About this data: figures are copied from the named source, not measured by us, and the test system and settings are those the source states above. See all benchmark sources and credits.
The CUDA scoreboard shows the same order as the lab tests, with the RTX 5090 first and the other two NVIDIA cards close together.
GPU Benchmarks for Local LLMs (last checked November 2026).
Capacity beats speed#
A fast card that cannot hold the model loses to a slower card that can. Back2Gaming's test shows it.
Qwen3.8-27B (UD-IQ4_XS) chat decode, 512-token answer: token_generation_decode (tokens/s, higher is better)
3 products. Highest: PowerColor Radeon AI PRO R9700 32GB, 31.7 tokens/s. Lowest: ASUS TUF Gaming Radeon RX 9070 XT 16GB OC, 27.2 tokens/s.
| Product | tokens/s |
|---|---|
| PowerColor Radeon AI PRO R9700 32GB | 31.7 |
| ASUS Prime GeForce RTX 5070 Ti 16GB OC | 30.3 |
| ASUS TUF Gaming Radeon RX 9070 XT 16GB OC | 27.2 |
About this data: figures are copied from the named source, not measured by us, and the test system and settings are those the source states above. See all benchmark sources and credits.
With a 27B-parameter model at 4-bit and a 32,768-token context, the Radeon AI PRO R9700 with 32 GB reached 31.7 tokens per second. The RTX 5070 Ti reached 30.3 and the RX 9070 XT 27.2. The two 16 GB cards had to offload layers to system memory, 4 of 66 on the RTX 5070 Ti and 2 of 66 on the RX 9070 XT. The 32 GB card held everything.
On a smaller 9B model that fits on every card, the picture reverses.
Qwen3.5-9B (Q4_K_M) chat decode, 512-token answer: token_generation_decode (tokens/s, higher is better)
3 products. Highest: ASUS Prime GeForce RTX 5070 Ti 16GB OC, 124.1 tokens/s. Lowest: PowerColor Radeon AI PRO R9700 32GB, 83.4 tokens/s.
| Product | tokens/s |
|---|---|
| ASUS Prime GeForce RTX 5070 Ti 16GB OC | 124.1 |
| ASUS TUF Gaming Radeon RX 9070 XT 16GB OC | 84.2 |
| PowerColor Radeon AI PRO R9700 32GB | 83.4 |
About this data: figures are copied from the named source, not measured by us, and the test system and settings are those the source states above. See all benchmark sources and credits.
The RTX 5070 Ti reached 124.1 tokens per second, the RX 9070 XT 84.2 and the R9700 83.4. The same three cards swap places depending on whether the model fits. That is why our VRAM explainer comes before any speed ranking in the buying process.
Image generation benchmarks
Image models have their own tests. Tom's Hardware ran UL Procyon's Stable Diffusion XL test on the RTX 5070 Ti and RX 9070 XT.
Procyon AI Image Generation, Stable Diffusion XL, benchmark score: benchmark_score (score, higher is better)
2 products. Highest: ASUS Prime GeForce RTX 5070 Ti 16GB OC, 3265 score. Lowest: ASUS TUF Gaming Radeon RX 9070 XT 16GB OC, 1969 score.
| Product | score |
|---|---|
| ASUS Prime GeForce RTX 5070 Ti 16GB OC | 3265 |
| ASUS TUF Gaming Radeon RX 9070 XT 16GB OC | 1969 |
About this data: figures are copied from the named source, not measured by us, and the test system and settings are those the source states above. See all benchmark sources and credits.
The RTX 5070 Ti scored 3265 against 1969 for the Radeon card. Back2Gaming's ComfyUI test, SDXL at 1024 x 1024 and 30 steps, gave 9.84 images per minute for the RTX 5070 Ti, 9.03 for the R9700 and 8.22 for the RX 9070 XT. Image tests are not interchangeable with text tests, and launch-window results can shift with drivers.
GPU Benchmarks for Local LLMs (last checked November 2026).
What the benchmarks leave out#
Speed charts leave out the parts of a machine that decide whether you can use a card at all. Power is the first. NVIDIA puts the RTX 5090 at 575 W total graphics power, the RTX 5080 at 360 W and the RTX 5070 Ti at 300 W. INNO3D gives 1000 W as the minimum for the system around its RTX 5090 card, and ASUS recommends 850 W for its RTX 5080 and 750 W for its RTX 5070 Ti. A card that tops a chart can still need a new power supply and a larger case.
Heat is the second. Long inference or fine-tuning jobs hold a card near its limit for hours. A card that draws more power also puts more heat into the room, so airflow counts as much as the tokens per second in a chart.
Software is the third. A benchmark run on one backend says nothing about the tools you will use. If you plan to run Ollama, ComfyUI or PyTorch on Windows, check that your card is supported on that operating system before you pick on speed.
Matching the numbers to your use
Different jobs lean on different columns of the chart.
| Use | Number to watch | Why |
|---|---|---|
| Chat with a local model | Token generation | It sets how quickly the reply streams |
| Summarising long files | Prompt processing | The card reads the whole file first |
| Retrieval over many documents | Prompt processing and VRAM | Long inputs need speed and memory |
| Image generation | Image-model scores | Text rates do not predict image speed |
| Running a large model | VRAM first | A model that does not fit runs slowly |
Use the table to decide which chart to trust. A buyer who mostly chats can read the generation charts. A buyer who feeds in long documents should weight the prompt-processing charts and the memory size more heavily.
What this means when buying#
Pick the memory size first, then the speed tier. When 16 GB is enough, the RTX 5070 Ti and RTX 5080 are the NVIDIA options, and our RTX 5080 vs RTX 5070 Ti comparison shows the gap is single-digit to low double-digit percent. If you need more than 16 GB, the RTX 5090 is the fastest, and the best high VRAM GPU guide covers the 32 GB and 48 GB options.
For AMD cards, our RTX 5070 Ti vs RX 9070 XT comparison covers the trade-off. Our best GPU for local LLMs and the GPU category page list more choices. Three cards from the benchmarks above, with maker pages: the INNO3D RTX 5090 X3 OC (opens in a new tab), the ASUS TUF RTX 5080 (opens in a new tab) and the ASUS Prime RTX 5070 Ti (opens in a new tab).
Benchmarks come from overseas testers, so check how Indian retailers list the same card, as partner models sold in India can differ in clocks and cooling from the tested units.
Frequently asked questions#
What is a good tokens-per-second rate for a local LLM?
Anything that streams faster than you can read feels smooth. Rates of a few tens of tokens per second already do that for chat. Higher rates matter more for long answers, coding assistants and agents.
Why do two sites show different numbers for the same card?
They use different models, quantisation, context lengths, backends and drivers. Compare figures only inside one chart from one source, and treat a number from another site as a rough guide, not as a like-for-like result.
Is prompt processing or token generation more important?
Generation matters for chat. Prompt processing matters when you feed in long documents or code. Check both, because a card that reads long prompts quickly can still feel slow if its token generation is weak.
How far can I trust the community llama.cpp scoreboards?
They show direction. Each row is a different volunteer with a different board and commit, so small gaps are noise. Use them as a cross-check next to a controlled test, and not as the only basis for a choice.
Do Radeon cards run local LLMs well?
They run them, often best through Vulkan in llama.cpp. They trail NVIDIA cards of the same tier in the tests cited. Software support is the main cost, so check your tools before you pick an AMD card.
Should I compare CUDA and Vulkan results directly?
No. They are different software paths, and community scoreboards use different volunteers, drivers and builds for each. Compare cards on one backend from one source.
Does more VRAM make a model faster?
Not directly. More VRAM lets a model fit entirely on the card, which avoids the slowdown of moving layers to system memory. Once the model fits, speed depends mostly on memory bandwidth and the chip.
Sources & credits#
We have not tested these products ourselves. Specs come from the makers; benchmark figures come from the sources we name.
- Puget Systems, 2025 Consumer GPU Content Creation Roundup (opens in a new tab)
- Hardware Corner, The Definitive GPU Ranking for LLMs (opens in a new tab)
- Tom's Hardware, AMD Radeon RX 9070 XT and RX 9070 review (opens in a new tab)
- Back2Gaming, AMD Radeon AI PRO R9700 32GB AI Performance Review (opens in a new tab)
- llama.cpp community, Performance of llama.cpp on Nvidia CUDA (opens in a new tab)
- llama.cpp community, Performance of llama.cpp with Vulkan (opens in a new tab)
- llama.cpp community, Performance of llama.cpp on AMD ROCm (HIP) (opens in a new tab)
- Phoronix, Lemonade Vulkan vs ROCm (opens in a new tab)
- INNO3D, GeForce RTX 5090 X3 OC product page (opens in a new tab)
- ASUS India, TUF Gaming GeForce RTX 5080 16GB OC product page (opens in a new tab)
- ASUS India, Prime GeForce RTX 5070 Ti 16GB OC product page (opens in a new tab)
Facts and figures in this article come from these sources. Articles are compiled from the makers' published specifications and the benchmarks named above, and checked by the NextAIGrid Editorial Team.
Disclaimer
This article is for information only. Benchmark results change with drivers and software, and availability changes, so check the maker's page before you buy. Found an error? Tell us.
How was this explainer put together?
It draws on 4 named sources: maker specification pages and published third-party benchmarks, listed in the article. We have not tested these products ourselves. Specs come from the makers; benchmark figures come from the sources we name. Specifications and sources were last checked on 17 November 2026. Read how we research and our editorial policy.
Spotted an error? Tell us. Confirmed fixes are listed on the corrections page.