In this article
Short answer
Tensor cores are units inside NVIDIA GPUs that multiply small blocks of numbers at low precision, the core arithmetic of neural networks. Each generation, from Volta to Blackwell, has added precisions such as INT8, TF32, FP8 and FP4. The generation decides which precisions run fast; memory size and bandwidth decide which models fit and how quickly they generate text.
- Tensor cores speed up matrix multiplication and convolution, not every operation.
- Newer generations add lower precisions, which help only once software uses them.
- AMD's RX 9070 XT has matrix units with their own peak figures for FP16, FP8, INT8 and INT4.
- For language models, memory bandwidth often matters more than tensor core speed.
What a tensor core does#
A neural network spends most of its time multiplying matrices and adding the results. A conventional GPU core handles one multiply at a time. A tensor core handles a whole small block at once, which is why it is much faster for this one job.
NVIDIA introduced them with the Volta generation. Its first tensor cores each perform a 4x4x4 matrix multiply-accumulate, written D = A x B + C, with FP16 inputs and FP16 or FP32 accumulation. NVIDIA states that 8 tensor cores per streaming multiprocessor do 1,024 floating-point operations per clock.
The idea has not changed since. What changed is the precision each core can use, the size of the blocks it handles and the software around it. A tensor core does nothing for work that is not a matrix operation, so a program that spends its time elsewhere sees a smaller gain than the raw numbers suggest.
A useful picture is a calculator that can only do one trick very well. Give it a large grid of numbers to combine and it finishes quickly. Give it a branching program and it waits for the ordinary cores. Neural networks are mostly grids of numbers, which is why vendors put so much silicon into this one trick. It also explains why a GPU still needs its general-purpose cores, its memory system and its software stack: tensor cores are only one part of the card.
Another consequence is that tensor cores are almost invisible in daily use. You do not select them in a menu. Libraries such as cuDNN and frameworks such as PyTorch send suitable work to them when the data type and shapes allow it. If your code uses an unusual data type or very small matrices, the work may fall back to the ordinary cores and the card will look slower than its rating suggests.
How the generations differ#
Each new generation is mostly a story about lower precision. Lower precision means smaller numbers, so more of them move through the hardware per clock and less memory is needed.
| Generation | Architecture | What NVIDIA says it added |
|---|---|---|
| 1st | Volta | FP16 matrix multiply with FP16 or FP32 accumulation |
| 2nd | Turing | INT8 and INT4 modes for inference |
| 3rd | Ampere | TensorFloat-32 (TF32) for FP32 inputs |
| 4th | Ada Lovelace | FP8 |
| 4th (data centre) | Hopper | Mixed FP8 and FP16 through the Transformer Engine |
| 5th | Blackwell | FP4 and microscaling formats, including NVFP4 |
The Ampere step deserves a closer look. TF32 keeps the 8 exponent bits of FP32, so it covers the same range of values, but uses 10 mantissa bits. NVIDIA says FP32 inputs are rounded to TF32 inside the tensor core and the results accumulate in FP32. Storage and other operations stay in FP32. The gain is that existing FP32 code can use tensor cores with little change.
NVIDIA also says TF32 only affects convolutions and matrix multiplications. Other layers and memory accesses stay in FP32, so real speed-ups are smaller than the raw tensor figure.
The precision ladder on one chip
NVIDIA's own table for the A100 data-centre GPU shows how much the format matters. Take plain FP32 on the ordinary cores as the baseline. TF32 on tensor cores is rated at 8 times, and FP16 or BF16 at 16 times.
Peak math, data-centre GPU: Relative math throughput (FP32 = 1x) (x, higher is better)
3 products. Highest: NVIDIA A100 (FP16/BF16 Tensor Cores), 16 x. Lowest: NVIDIA A100 (FP32 CUDA cores), 1 x.
| Product | x |
|---|---|
| NVIDIA A100 (FP16/BF16 Tensor Cores) | 16 |
| NVIDIA A100 (TF32 Tensor Cores) | 8 |
| NVIDIA A100 (FP32 CUDA cores) | 1 |
About this data: figures are copied from the named source, not measured by us, and the test system and settings are those the source states above. See all benchmark sources and credits.
These are peak figures from NVIDIA, not end-to-end measurements, and the A100 is a data-centre part rather than a card you would buy in India. The pattern still holds on newer chips: each step down in precision raises the peak. NVIDIA adds that mixed-precision FP16 training gives up to about 2 times on top of TF32, because 16-bit tensor cores are twice as fast as TF32 mode and memory traffic is halved.
What it means in PyTorch
Most people never write tensor core code. Frameworks do it for them. PyTorch's documentation says TF32 can be used internally for float32 matrix multiplication and convolution on NVIDIA Ampere and later GPUs. The setting that allows this was on by default in PyTorch 1.7 to 1.11 and has been off since 1.12, with newer precision settings that accept either IEEE or TF32.
The more common route is automatic mixed precision. In PyTorch, autocast runs eligible CUDA operations in float16 or bfloat16 and the rest in float32, and a gradient scaler handles the float16 range problem during training. For inference, formats such as INT8 and FP8 are usually applied through quantisation tools. Our VRAM explainer shows how quantisation changes the memory a model needs.
The practical lesson: a card's tensor cores help only when your software asks for a precision the card handles well. If a model ships in FP16, an FP4 unit sits idle.
Tensor Cores Explained (last checked November 2026).
Consumer cards: what NVIDIA lists#
NVIDIA's product pages list a tensor core generation and an AI TOPS figure for each GeForce card. The pages do not state the precision behind the TOPS figure, so treat it as a ranking inside one generation and not as a number to compare with other makers.
| Card | Tensor core generation | NVIDIA AI TOPS |
|---|---|---|
| GeForce RTX 5090 | 5th | 3352 |
| GeForce RTX 5080 | 5th | 1801 |
| GeForce RTX 5070 Ti | 5th | 1406 |
| GeForce RTX 5070 | 5th | 988 |
| GeForce RTX 5060 Ti | 5th | 759 |
| GeForce RTX 5060 | 5th | 614 |
| GeForce RTX 4090 | 4th | 1321 |
Puget Systems, an AI workstation builder, adds tensor core counts that NVIDIA's pages leave out. It reports 680 fifth-generation tensor cores on the RTX 5090 against 512 fourth-generation cores on the RTX 4090, and 336 on the RTX 5080 against 304 on the RTX 4080. Its FP16 and BF16 figures rise about 27% from the RTX 4090 to the RTX 5090, from 165.2 to 209.5 TFLOPS.
Puget calls the real change FP4 support rather than speed-ups to formats the older cards already handled. It also reports that, at publication, end-user software support for FP4 was almost non-existent, with an FP4 version of Black Forest Labs' FLUX.1 model an early exception. A new generation therefore helps first through its higher counts and clocks, and later through new precisions as software catches up.
One caution on numbers. Puget's own table lists INT8 AI TOPS on a different basis from NVIDIA's marketing figures. The two sets should not be compared with each other. Our TOPS explainer covers why a bare TOPS number is incomplete.
AMD's version: matrix units on RDNA 4
AMD does not use the name tensor core. Its RDNA 4 graphics chips carry AI accelerators that do the same job. The Radeon RX 9070 XT lists 128 of them, and AMD publishes separate peak figures for vector and matrix work.
For FP16, AMD lists 48.7 TFLOPS on the ordinary vector units and 195 TFLOPS on the matrix units. The FP8 matrix figure is 389 TFLOPS, and INT8 matrix is 389 TOPS. AMD adds higher figures with structured sparsity, and says RDNA 4 supports FP8 and INT4 formats.
Peak theoretical: Peak throughput (TFLOPS, higher is better)
3 products. Highest: AMD Radeon RX 9070 XT FP8 matrix, 389 TFLOPS. Lowest: AMD Radeon RX 9070 XT FP16 vector, 48.7 TFLOPS.
| Product | TFLOPS |
|---|---|
| AMD Radeon RX 9070 XT FP8 matrix | 389 |
| AMD Radeon RX 9070 XT FP16 matrix | 195 |
| AMD Radeon RX 9070 XT FP16 vector | 48.7 |
About this data: figures are copied from the named source, not measured by us, and the test system and settings are those the source states above. See all benchmark sources and credits.
The chart shows the same point as the A100 table. Specialised matrix hardware raises the peak several times over the general-purpose cores. These are AMD's peak ratings without sparsity, not benchmarks. Whether you can use them depends on software, which is why the CUDA vs ROCm explainer matters when choosing an AMD card.
Tensor Cores Explained (last checked November 2026).
Memory, sparsity and other limits#
Faster tensor cores do not always make a language model faster. Puget Systems reports that llama.cpp token generation depends more on memory bandwidth than on compute. In its testing the RTX 5090, at 1.79 TB/s, led the RTX 4090 by about 29%. Prompt processing, in contrast, is limited by compute.
That split explains a common surprise. A card with many more AI TOPS can generate text only a little faster than its predecessor because both read the weights at similar rates. Capacity matters even more: a model that does not fit in VRAM will not run at all. Check the memory size and bandwidth before the tensor core headline. The guide to the best GPU for local LLMs and the deep learning GPU guide apply this to real cards.
Sparsity: the second label on every figure
Structured sparsity lets a chip skip multiplications by zero when the weights follow a particular pattern. NVIDIA's Ada Lovelace page lists it among its features, and AMD publishes sparse figures next to its dense ones. For the RX 9070 XT, AMD lists the FP16 matrix figure as 195 TFLOPS dense and 389 with sparsity, and the INT4 figure as 779 dense and 1557 with sparsity.
The catch is that a model must be pruned to the pattern before the hardware can skip anything. Most downloaded models are not. Read a sparse figure as an upper limit that your model may never reach, and compare dense figures with dense ones.
A practical check helps before you rely on any of these figures. Look up which precision the software you plan to use supports, then look up which precision the card handles natively. If the two meet at FP16 or INT8, the card will run the work whatever its generation. If your software moves to FP8 or FP4, the card's generation decides whether it can follow. This keeps the buying decision tied to your workload, not to the newest label on the box.
Tensor cores beyond GeForce
The same hardware idea runs from desktop cards down to small boards. NVIDIA's Jetson Orin Nano Super lists 32 tensor cores and 1024 CUDA cores in an edge board, so the software skills transfer. Our Jetson Orin Nano Super review looks at that board. At the other end, NVIDIA says Hopper data-centre GPUs triple FLOPS for TF32, FP64, FP16 and INT8 over the prior generation. The data-centre accelerators explainer shows what that scale looks like in published inference results.
Common mistakes#
- Buying on AI TOPS alone. NVIDIA does not state the precision behind its consumer TOPS figures, so use them only to rank cards inside one generation.
- Expecting FP4 gains today. Puget Systems reported that software support for FP4 was close to non-existent at publication. Check that your tools use it.
- Ignoring VRAM. A model that does not fit will not run, whatever the tensor core count.
- Treating sparse and dense figures as the same. Sparse ratings assume a pruned model.
- Comparing across makers. NVIDIA's AI TOPS, AMD's matrix TFLOPS and Puget's INT8 figures use different bases.
What this means when buying#
Use the tensor core generation to decide which precisions the card handles well, then match that to the software you plan to run. If your tools use FP16 or INT8, an older generation is still fully useful. If you want to try FP4 models as they appear, a Blackwell card is the one that supports them.
Buy for memory first, bandwidth second and tensor core generation third. The NPU vs GPU explainer shows where laptop AI units fit alongside all of this.
When shopping in India, check the NVIDIA architecture name in Indian listings, since sellers sometimes mix old and new cards, and tensor core generation depends on the chip.
Tensor cores also speed up AI video tools, and the best GPU for AI video upscaling guide shows which cards suit that work.
Frequently asked questions#
Do tensor cores help with gaming?
NVIDIA's pages tie them to DLSS, which uses AI. Their main design purpose is matrix arithmetic for AI work, so gaming gains come from AI upscaling features and not from the cores speeding up ordinary graphics.
Is a higher AI TOPS figure always better?
Within one generation it ranks cards roughly. Across makers it does not, because the precision and sparsity behind each figure differ. Compare published results for your own workload before trusting the headline number.
Which tensor core generation do I need for FP8?
NVIDIA says its fourth-generation tensor cores in the Ada Lovelace architecture added FP8. Fifth-generation cores add FP4. Older cards can still run lower-precision models in software, but without the hardware speed-up.
Does PyTorch use tensor cores automatically?
For many operations, yes. PyTorch's documentation describes TF32 use for float32 matrix multiplication and autocast for float16 or bfloat16. You may need to enable these settings, so check the documentation for your version.
Why does my model not run faster on a newer card?
If the job is limited by memory bandwidth, tensor core gains do not show up. Puget Systems found this with llama.cpp token generation. A newer card helps most when the work is compute-bound.
Does AMD have tensor cores?
AMD calls its units AI accelerators. The RX 9070 XT lists 128 of them with FP16, FP8, INT8 and INT4 matrix figures. They do similar matrix work, though software support differs from NVIDIA.
Sources & credits#
We have not tested these products ourselves. Specs come from the makers; benchmark figures come from the sources we name.
- NVIDIA, Tensor Cores (opens in a new tab)
- NVIDIA Technical Blog, Programming Tensor Cores in CUDA 9 (opens in a new tab)
- NVIDIA Technical Blog, NVIDIA Turing Architecture In-Depth (opens in a new tab)
- NVIDIA Technical Blog, Accelerating AI Training with NVIDIA TF32 Tensor Cores (opens in a new tab)
- NVIDIA, Hopper architecture (opens in a new tab)
- NVIDIA, Ada Lovelace architecture (opens in a new tab)
- NVIDIA, GeForce RTX 50 Series (opens in a new tab)
- NVIDIA, GeForce RTX 5090 (opens in a new tab)
- NVIDIA, GeForce RTX 5080 (opens in a new tab)
- NVIDIA, GeForce RTX 5070 family (opens in a new tab)
- NVIDIA, GeForce RTX 5060 family (opens in a new tab)
- NVIDIA, GeForce RTX 4090 (opens in a new tab)
- AMD, Radeon RX 9070 XT (opens in a new tab)
- AMD, RDNA architecture (opens in a new tab)
- PyTorch Docs, CUDA semantics (opens in a new tab)
- PyTorch Docs, Automatic Mixed Precision (opens in a new tab)
- Puget Systems, NVIDIA GeForce RTX 5090 & 5080 AI Review (opens in a new tab)
Facts and figures in this article come from these sources. Articles are compiled from the makers' published specifications and the benchmarks named above, and checked by the NextAIGrid Editorial Team.
Disclaimer
This explainer is for information only. Specifications and availability change, so check the maker's page. Found an error? Tell us.
How was this explainer put together?
It draws on maker specification pages and published third-party benchmarks, named in the article. We have not tested these products ourselves. Specs come from the makers; benchmark figures come from the sources we name. Specifications and sources were last checked on 17 November 2026. Read how we research and our editorial policy.
Spotted an error? Tell us. Confirmed fixes are listed on the corrections page.