In this article
Short answer
TOPS means trillions of operations per second, a peak rating that makers quote for NPUs, edge boards and accelerators. The number only means something when the precision, such as INT8 or INT4, and any sparsity assumption are stated. It does not tell you how fast a real model will run.
- Read the precision next to the number. INT8 and INT4 figures are not the same thing.
- A sparse rating assumes the model skips zero values, so it is not a plain peak.
- Memory bandwidth, software and the model decide real speed.
- Microsoft uses 40+ TOPS as the NPU level for many Copilot+ PC features.
What TOPS measures#
An AI chip spends most of its time on multiply and add operations. TOPS counts how many such operations it can perform in one second, in trillions. A chip rated at 4 TOPS can do four trillion operations per second at best. Coral states exactly this for its Edge TPU: 4 trillion operations per second.
Think of it as the top speed printed on a speedometer. It tells you what the engine is built for, not how fast any journey will be.
The word "peak" matters here, and it explains most of the confusion around the unit. The rating describes what the hardware could do if every arithmetic unit were busy every cycle. Real models rarely achieve that, because they wait for data from memory, run steps that the chip does not accelerate, or use layers that do not map neatly onto the hardware.
Makers calculate the figure from the number of arithmetic units and the clock speed. You can therefore treat TOPS as a statement about the design of the chip, not a measurement of any program. It is useful for ranking chips in the same family and for setting a minimum bar, such as a requirement for a software feature. It is a poor guide for choosing between chips from different makers.
The related units are TFLOPS and PFLOPS. FLOPS counts floating-point operations, which are used at higher precision. TOPS usually counts integer operations at low precision. A figure in TFLOPS and a figure in TOPS are not directly comparable, because they measure different kinds of arithmetic.
Precision: why INT8 and INT4 change the number#
AI models store their weights as numbers. At full precision each number takes more space and more energy to process. Many edge chips run models at 8-bit integers (INT8) or even 4-bit integers (INT4), which are smaller and faster to process, at some cost in accuracy that is handled by quantisation.
Smaller values mean more of them fit through the arithmetic units each cycle. NVIDIA makes this point for its Blackwell GPUs, saying FP4 offers twice the peak throughput of FP8. That is why a figure quoted at a lower precision looks larger than one at a higher precision for similar hardware. Comparing them without noting the precision overstates the difference.
The table shows how makers state their ratings. The precision and conditions are as each maker writes them.
| Product | Rating | Precision and conditions as stated |
|---|---|---|
| NVIDIA Jetson Orin Nano Super Developer Kit | 67 TOPS | Sparse INT8, with 33 TOPS dense INT8, as stated by NVIDIA |
| NVIDIA Jetson AGX Orin 64GB Developer Kit | 275 TOPS | Sparse INT8, as stated by NVIDIA |
| Coral USB Accelerator | 4 TOPS | int8, 2 TOPS per watt, as stated by Coral |
| Raspberry Pi AI HAT+ | 13 TOPS or 26 TOPS | Hailo-8L or Hailo-8, as stated by Raspberry Pi |
| Raspberry Pi AI HAT+ 2 | 40 TOPS | INT4, Hailo-10H, as stated by Raspberry Pi |
| Copilot+ PC NPU | 40+ TOPS | Requirement set by Microsoft |
Look at the first two rows. NVIDIA's Jetson Orin table gives the Orin Nano Super as 67 sparse INT8 TOPS and 33 dense INT8 TOPS, and the AGX Orin's 275 is also a sparse figure. The same chip therefore has two valid ratings, and the larger one assumes a pruned model. The Raspberry Pi add-ons carry a similar warning: the 40 TOPS board is rated at INT4, so it cannot be set beside an INT8 figure.
Sparsity: the second label to look for
Sparsity means skipping the multiplications that involve zero. Some chips can detect a particular pattern of zeros in a model and skip that work, which raises the effective throughput for models prepared for it. A "sparse" TOPS rating assumes the model was pruned to that pattern.
Most models downloaded from the web are not pruned this way. A buyer who runs an ordinary model should therefore read a sparse rating as a ceiling that may never apply. The H200 data-centre GPU shows the same convention: NVIDIA lists its FP8 tensor performance as 3,958 TFLOPS with sparsity. That figure uses TFLOPS rather than TOPS, and it describes a very different class of product, but the sparse label has the same meaning.
The Jetson pair shows how much this matters: the dense figure is half the headline. When a maker writes only "TOPS" with no precision and no sparsity note, treat the figure as incomplete. Check the datasheet.
Why TOPS does not equal real speed#
Two chips with the same rating can run the same model at very different speeds. The reasons are practical.
Memory bandwidth. In Puget Systems' llama.cpp testing, bandwidth mattered more than compute for generating tokens. The Jetson Orin Nano Super lists 102 GB/s, and NVIDIA raised it from 68 GB/s in the same update that lifted the sparse rating from 40 to 67 TOPS. For a language model the bandwidth figure is as relevant as the TOPS figure.
Software. A model has to be converted into a form the chip can run. Ultralytics' published results make the point on a single board: on the Jetson, its small YOLO model took 15.60 ms per image with plain PyTorch and 4.57 ms with TensorRT FP16. Same board, same rating, a gap of more than three times.
The model itself. Layers that an accelerator cannot handle usually fall back to the CPU, so a model with unsupported layers can run far below the rating.
YOLO26n COCO128 imgsz 640: Inference time (ms/image, lower is better)
4 products. Lowest: Jetson Orin Nano Super, TensorRT INT8, 3.8 ms/image. Highest: Jetson Orin Nano Super, PyTorch, 15.6 ms/image.
| Product | ms/image |
|---|---|
| Jetson Orin Nano Super, TensorRT INT8 | 3.8 |
| Jetson Orin Nano Super, TensorRT FP16 | 4.57 |
| Jetson Orin Nano Super, TensorRT FP32 | 7.53 |
| Jetson Orin Nano Super, PyTorch | 15.6 |
About this data: figures are copied from the named source, not measured by us, and the test system and settings are those the source states above. See all benchmark sources and credits.
The chart uses the times Ultralytics lists for its Jetson guide, from plain PyTorch to TensorRT INT8. The board and its rating stay the same across all four bars, yet the time falls from 15.60 ms to 3.80 ms. The rating alone would not have predicted that spread.
Data-centre results show the same pattern
At the other end of the scale, the MLPerf Inference benchmark from MLCommons measures complete systems on set models. NVIDIA reports that in MLPerf Inference v5.0, an eight-GPU B200 system reached 98,858 tokens per second offline on Llama 2 70B, against 34,988 for an eight-GPU H200 system. NVIDIA says that is 2.8 times higher. Part of that gain comes from running the B200 at FP4, a lower precision, as NVIDIA states.
Llama 2 70B: Throughput (tokens/s, higher is better)
2 products. Highest: NVIDIA DGX B200 (8x B200), 98858 tokens/s. Lowest: NVIDIA 8x H200 server, 34988 tokens/s.
| Product | tokens/s |
|---|---|
| NVIDIA DGX B200 (8x B200) | 98858 |
| NVIDIA 8x H200 server | 34988 |
About this data: figures are copied from the named source, not measured by us, and the test system and settings are those the source states above. See all benchmark sources and credits.
The point for a buyer: the largest speed-ups come from the combination of the chip, the precision, the memory and the software. Our data-centre accelerators explainer goes through these results in more detail.
Where TOPS thresholds do matter#
TOPS works well as a minimum requirement. Microsoft says many new Windows AI features on Copilot+ PCs require an NPU able to run at 40+ TOPS, and lists Qualcomm Snapdragon X, AMD Ryzen AI 300 series and Intel Core Ultra 200V series machines among the qualifying devices. If a laptop must support those features, the rating is a pass or fail test, and a higher number gives no further benefit for that purpose. For how NPUs compare with GPUs, read NPU vs GPU explained and the AI PC processors comparison.
For edge boards, a TOPS figure helps you sort products into tiers. Raspberry Pi's AI HAT+ comes at 13 TOPS or 26 TOPS, Coral's Edge TPU at 4 TOPS, and the Jetson Orin Nano Super at 67 sparse TOPS. Tiers like these are a quick way to shortlist boards, as long as you read the precision first.
How to read a TOPS claim#
Use this checklist whenever a product page leads with a TOPS number.
- Find the precision. INT8, INT4, FP8 and FP4 are different measures. A claim without one is incomplete.
- Look for the word sparse. If it appears, the rating assumes a pruned model.
- Check what the number covers. Some boards state the figure for the AI engine alone. Others add the CPU and GPU together. Read the footnotes.
- Check the power mode. Boards with a power range often reach the rating only in their highest mode.
- Find the memory size and bandwidth. Weights larger than the available memory will not load, whatever the rating.
- Look for a published result on your kind of workload. Third-party figures, such as the Ultralytics inference times or MLPerf entries, tell you more than the headline.
TOPS inside laptops and PCs
On a laptop, TOPS usually describes the NPU, a small chip block built for steady, low-power AI work. Microsoft describes it as a specialised chip for AI-intensive tasks such as real-time translation and image generation, and says Windows 11 assigns each task to the CPU, GPU or NPU that suits it best. The NPU therefore sits beside the GPU and does not replace it.
That is why a laptop with a modest NPU and a strong GPU can still be the better machine for model work. The NPU handles background features, while the GPU does heavy lifting such as training and large local models. The NPU vs GPU explainer goes through the split.
For a buyer, the Copilot+ threshold is the simple rule: if you want Microsoft's AI features, check that the NPU meets 40+ TOPS. If you want to run large models, spend on GPU memory instead.
TOPS per watt
Efficiency often matters more than peak figures on a battery or a small power supply. Coral states 2 TOPS per watt for its Edge TPU, which means 0.5 W per TOPS. NVIDIA lists a 7 W to 25 W range for the Jetson Orin Nano Super, and a 15 W to 60 W range for the AGX Orin.
Dividing the TOPS rating by a power figure is only a rough guide, because the power range covers different modes and the rating may apply to only the highest one. Use it to shortlist products, then read the maker's mode table.
A good habit is to write the figure down with its label, for example "67 sparse INT8 TOPS" or "40 TOPS INT4", before comparing it with anything. If a product page will not supply the label, treat the number as marketing and look for the datasheet.
What this means when buying#
Treat TOPS as a first filter and not as a verdict. Use the rating to sort products into tiers. Then check four things: the precision, whether the figure is sparse, the memory size and bandwidth, and published results for a workload like yours. Pricing moves, so confirm it with a seller before buying.
If you are weighing edge boards, the Jetson Orin Nano Super review and the Jetson vs Raspberry Pi comparison put these ideas into practice. For laptops with NPUs, see the AI workstations and PCs guide. Browse the full AI accelerators hub for more.
One product in this explainer is sold in India as a retail kit:
- NVIDIA Jetson Orin Nano Super Developer Kit 8GB: a 67 sparse INT8 TOPS board with a CUDA software stack and a bundled 19 V supply.
When you compare AI laptops sold in India, ask for the maker's own TOPS page for the exact model number, since regional variants of the same laptop can carry different chips.
Frequently asked questions#
What does one TOPS mean?
One TOPS is one trillion operations per second. Makers use it as a peak rating for an AI chip, usually at low-precision integer arithmetic, so it describes a ceiling and not the speed you will see on your own model.
Is a higher TOPS number always better?
No. Precision, sparsity, memory and software change what the number means. Two chips with the same rating can differ widely in real speed, so compare published results for your own kind of model before choosing.
What is the difference between TOPS and TFLOPS?
TOPS usually counts integer operations at low precision. TFLOPS counts floating-point operations. They measure different arithmetic and should not be compared directly, so check which type of operation a maker is quoting first.
How many TOPS do I need for a Copilot+ PC?
Microsoft says many of its new Windows AI features need an NPU that can run at 40+ TOPS. A laptop below that level can still run Windows, but those features may be missing, so check the NPU rating before buying.
Why do some makers list sparse TOPS?
Sparse ratings assume the model skips zero values in a specific pattern. NVIDIA labels the AGX Orin's 275 TOPS and the Orin Nano Super's 67 TOPS as sparse INT8, for example.
Do TOPS matter for running language models?
Less than memory capacity and bandwidth. A model must fit in memory, and its speed depends on how fast the weights can be read. A chip with a high TOPS rating but little fast memory will still feel slow.
Sources & credits#
We have not tested these products ourselves. Specs come from the makers; benchmark figures come from the sources we name.
- NVIDIA India, Jetson Orin Nano Super Developer Kit (opens in a new tab)
- NVIDIA India, Jetson Orin series (opens in a new tab)
- NVIDIA, H200 product page (opens in a new tab)
- NVIDIA Technical Blog, MLPerf Inference v5.0 results (opens in a new tab)
- Coral, USB Accelerator product page (opens in a new tab)
- Coral, Edge TPU benchmarks (opens in a new tab)
- Puget Systems, NVIDIA GeForce RTX 5090 & 5080 AI Review (opens in a new tab)
- Raspberry Pi, AI HAT+ product page (opens in a new tab)
- Raspberry Pi, AI HAT+ 2 product page (opens in a new tab)
- Microsoft Learn, Copilot+ PCs developer guide (opens in a new tab)
- Ultralytics Docs, NVIDIA Jetson guide (opens in a new tab)
Facts and figures in this article come from these sources. Articles are compiled from the makers' published specifications and the benchmarks named above, and checked by the NextAIGrid Editorial Team.
Disclaimer
This explainer is for information only. Specifications and availability change, so check the maker's page. Found an error? Tell us.
How was this explainer put together?
It draws on 1 named source: maker specification pages and published third-party benchmarks, listed in the article. We have not tested these products ourselves. Specs come from the makers; benchmark figures come from the sources we name. Specifications and sources were last checked on 17 November 2026. Read how we research and our editorial policy.
Spotted an error? Tell us. Confirmed fixes are listed on the corrections page.