Explainer

What Is TOPS? AI Chip Performance Explained

In this article
  1. What TOPS measures
  2. Precision: why INT8 and INT4 change the number
  3. Why TOPS does not equal real speed
  4. Where TOPS thresholds do matter
  5. How to read a TOPS claim
  6. What this means when buying
  7. Frequently asked questions
  8. Sources & credits

Short answer

TOPS means trillions of operations per second, a peak rating that makers quote for NPUs, edge boards and accelerators. The number only means something when the precision, such as INT8 or INT4, and any sparsity assumption are stated. It does not tell you how fast a real model will run.

  • Read the precision next to the number. INT8 and INT4 figures are not the same thing.
  • A sparse rating assumes the model skips zero values, so it is not a plain peak.
  • Memory bandwidth, software and the model decide real speed.
  • Microsoft uses 40+ TOPS as the NPU level for many Copilot+ PC features.

What TOPS measures#

An AI chip spends most of its time on multiply and add operations. TOPS counts how many such operations it can perform in one second, in trillions. A chip rated at 4 TOPS can do four trillion operations per second at best. Coral states exactly this for its Edge TPU: 4 trillion operations per second.

Think of it as the top speed printed on a speedometer. It tells you what the engine is built for, not how fast any journey will be.

The word "peak" matters here, and it explains most of the confusion around the unit. The rating describes what the hardware could do if every arithmetic unit were busy every cycle. Real models rarely achieve that, because they wait for data from memory, run steps that the chip does not accelerate, or use layers that do not map neatly onto the hardware.

Makers calculate the figure from the number of arithmetic units and the clock speed. You can therefore treat TOPS as a statement about the design of the chip, not a measurement of any program. It is useful for ranking chips in the same family and for setting a minimum bar, such as a requirement for a software feature. It is a poor guide for choosing between chips from different makers.

The related units are TFLOPS and PFLOPS. FLOPS counts floating-point operations, which are used at higher precision. TOPS usually counts integer operations at low precision. A figure in TFLOPS and a figure in TOPS are not directly comparable, because they measure different kinds of arithmetic.

Precision: why INT8 and INT4 change the number#

AI models store their weights as numbers. At full precision each number takes more space and more energy to process. Many edge chips run models at 8-bit integers (INT8) or even 4-bit integers (INT4), which are smaller and faster to process, at some cost in accuracy that is handled by quantisation.

Smaller values mean more of them fit through the arithmetic units each cycle. NVIDIA makes this point for its Blackwell GPUs, saying FP4 offers twice the peak throughput of FP8. That is why a figure quoted at a lower precision looks larger than one at a higher precision for similar hardware. Comparing them without noting the precision overstates the difference.

The table shows how makers state their ratings. The precision and conditions are as each maker writes them.

Table
ProductRatingPrecision and conditions as stated
NVIDIA Jetson Orin Nano Super Developer Kit67 TOPSSparse INT8, with 33 TOPS dense INT8, as stated by NVIDIA
NVIDIA Jetson AGX Orin 64GB Developer Kit275 TOPSSparse INT8, as stated by NVIDIA
Coral USB Accelerator4 TOPSint8, 2 TOPS per watt, as stated by Coral
Raspberry Pi AI HAT+13 TOPS or 26 TOPSHailo-8L or Hailo-8, as stated by Raspberry Pi
Raspberry Pi AI HAT+ 240 TOPSINT4, Hailo-10H, as stated by Raspberry Pi
Copilot+ PC NPU40+ TOPSRequirement set by Microsoft

Look at the first two rows. NVIDIA's Jetson Orin table gives the Orin Nano Super as 67 sparse INT8 TOPS and 33 dense INT8 TOPS, and the AGX Orin's 275 is also a sparse figure. The same chip therefore has two valid ratings, and the larger one assumes a pruned model. The Raspberry Pi add-ons carry a similar warning: the 40 TOPS board is rated at INT4, so it cannot be set beside an INT8 figure.

Sparsity: the second label to look for

Sparsity means skipping the multiplications that involve zero. Some chips can detect a particular pattern of zeros in a model and skip that work, which raises the effective throughput for models prepared for it. A "sparse" TOPS rating assumes the model was pruned to that pattern.

Most models downloaded from the web are not pruned this way. A buyer who runs an ordinary model should therefore read a sparse rating as a ceiling that may never apply. The H200 data-centre GPU shows the same convention: NVIDIA lists its FP8 tensor performance as 3,958 TFLOPS with sparsity. That figure uses TFLOPS rather than TOPS, and it describes a very different class of product, but the sparse label has the same meaning.

The Jetson pair shows how much this matters: the dense figure is half the headline. When a maker writes only "TOPS" with no precision and no sparsity note, treat the figure as incomplete. Check the datasheet.

Why TOPS does not equal real speed#

Two chips with the same rating can run the same model at very different speeds. The reasons are practical.

Memory bandwidth. In Puget Systems' llama.cpp testing, bandwidth mattered more than compute for generating tokens. The Jetson Orin Nano Super lists 102 GB/s, and NVIDIA raised it from 68 GB/s in the same update that lifted the sparse rating from 40 to 67 TOPS. For a language model the bandwidth figure is as relevant as the TOPS figure.

Software. A model has to be converted into a form the chip can run. Ultralytics' published results make the point on a single board: on the Jetson, its small YOLO model took 15.60 ms per image with plain PyTorch and 4.57 ms with TensorRT FP16. Same board, same rating, a gap of more than three times.

The model itself. Layers that an accelerator cannot handle usually fall back to the CPU, so a model with unsupported layers can run far below the rating.

YOLO26n COCO128 imgsz 640: Inference time (ms/image, lower is better)

YOLO26n COCO128 imgsz 640: Inference timeJetson Orin Nano Super, TensorRT INT8: 3.8 ms/imageJetson Orin Nano Super, TensorRT INT83.8 ms/imageJetson Orin Nano Super, TensorRT FP16: 4.57 ms/imageJetson Orin Nano Super, TensorRT FP164.57 ms/imageJetson Orin Nano Super, TensorRT FP32: 7.53 ms/imageJetson Orin Nano Super, TensorRT FP327.53 ms/imageJetson Orin Nano Super, PyTorch: 15.6 ms/imageJetson Orin Nano Super, PyTorch15.6 ms/image

4 products. Lowest: Jetson Orin Nano Super, TensorRT INT8, 3.8 ms/image. Highest: Jetson Orin Nano Super, PyTorch, 15.6 ms/image.

YOLO26n COCO128 imgsz 640: Inference time
Productms/image
Jetson Orin Nano Super, TensorRT INT83.8
Jetson Orin Nano Super, TensorRT FP164.57
Jetson Orin Nano Super, TensorRT FP327.53
Jetson Orin Nano Super, PyTorch15.6
Source: Ultralytics Docs (opens in a new tab). Ultralytics 8.4.33, JetPack 6.1, Jetson guide. Page read 2026-11-17; excludes pre/post-processing; route: TensorRT INT8.

About this data: figures are copied from the named source, not measured by us, and the test system and settings are those the source states above. See all benchmark sources and credits.

The chart uses the times Ultralytics lists for its Jetson guide, from plain PyTorch to TensorRT INT8. The board and its rating stay the same across all four bars, yet the time falls from 15.60 ms to 3.80 ms. The rating alone would not have predicted that spread.

Data-centre results show the same pattern

At the other end of the scale, the MLPerf Inference benchmark from MLCommons measures complete systems on set models. NVIDIA reports that in MLPerf Inference v5.0, an eight-GPU B200 system reached 98,858 tokens per second offline on Llama 2 70B, against 34,988 for an eight-GPU H200 system. NVIDIA says that is 2.8 times higher. Part of that gain comes from running the B200 at FP4, a lower precision, as NVIDIA states.

Llama 2 70B: Throughput (tokens/s, higher is better)

Llama 2 70B: ThroughputNVIDIA DGX B200 (8x B200): 98858 tokens/sNVIDIA DGX B200 (8x B200)98858 tokens/sNVIDIA 8x H200 server: 34988 tokens/sNVIDIA 8x H200 server34988 tokens/s

2 products. Highest: NVIDIA DGX B200 (8x B200), 98858 tokens/s. Lowest: NVIDIA 8x H200 server, 34988 tokens/s.

Llama 2 70B: Throughput
Producttokens/s
NVIDIA DGX B200 (8x B200)98858
NVIDIA 8x H200 server34988
Source: NVIDIA Technical Blog (MLPerf Inference v5.0 entries) (opens in a new tab). MLPerf Inference v5.0 Closed Datacenter, Offline. Vendor-published MLCommons Closed data-centre entries; blog dated 2025-04-02; per-system (8 accelerators) not per-GPU; B200 entries used FP4 so not precision-matched to H200.

About this data: figures are copied from the named source, not measured by us, and the test system and settings are those the source states above. See all benchmark sources and credits.

The point for a buyer: the largest speed-ups come from the combination of the chip, the precision, the memory and the software. Our data-centre accelerators explainer goes through these results in more detail.

Where TOPS thresholds do matter#

TOPS works well as a minimum requirement. Microsoft says many new Windows AI features on Copilot+ PCs require an NPU able to run at 40+ TOPS, and lists Qualcomm Snapdragon X, AMD Ryzen AI 300 series and Intel Core Ultra 200V series machines among the qualifying devices. If a laptop must support those features, the rating is a pass or fail test, and a higher number gives no further benefit for that purpose. For how NPUs compare with GPUs, read NPU vs GPU explained and the AI PC processors comparison.

For edge boards, a TOPS figure helps you sort products into tiers. Raspberry Pi's AI HAT+ comes at 13 TOPS or 26 TOPS, Coral's Edge TPU at 4 TOPS, and the Jetson Orin Nano Super at 67 sparse TOPS. Tiers like these are a quick way to shortlist boards, as long as you read the precision first.

How to read a TOPS claim#

Use this checklist whenever a product page leads with a TOPS number.

  1. Find the precision. INT8, INT4, FP8 and FP4 are different measures. A claim without one is incomplete.
  2. Look for the word sparse. If it appears, the rating assumes a pruned model.
  3. Check what the number covers. Some boards state the figure for the AI engine alone. Others add the CPU and GPU together. Read the footnotes.
  4. Check the power mode. Boards with a power range often reach the rating only in their highest mode.
  5. Find the memory size and bandwidth. Weights larger than the available memory will not load, whatever the rating.
  6. Look for a published result on your kind of workload. Third-party figures, such as the Ultralytics inference times or MLPerf entries, tell you more than the headline.

TOPS inside laptops and PCs

On a laptop, TOPS usually describes the NPU, a small chip block built for steady, low-power AI work. Microsoft describes it as a specialised chip for AI-intensive tasks such as real-time translation and image generation, and says Windows 11 assigns each task to the CPU, GPU or NPU that suits it best. The NPU therefore sits beside the GPU and does not replace it.

That is why a laptop with a modest NPU and a strong GPU can still be the better machine for model work. The NPU handles background features, while the GPU does heavy lifting such as training and large local models. The NPU vs GPU explainer goes through the split.

For a buyer, the Copilot+ threshold is the simple rule: if you want Microsoft's AI features, check that the NPU meets 40+ TOPS. If you want to run large models, spend on GPU memory instead.

TOPS per watt

Efficiency often matters more than peak figures on a battery or a small power supply. Coral states 2 TOPS per watt for its Edge TPU, which means 0.5 W per TOPS. NVIDIA lists a 7 W to 25 W range for the Jetson Orin Nano Super, and a 15 W to 60 W range for the AGX Orin.

Dividing the TOPS rating by a power figure is only a rough guide, because the power range covers different modes and the rating may apply to only the highest one. Use it to shortlist products, then read the maker's mode table.

A good habit is to write the figure down with its label, for example "67 sparse INT8 TOPS" or "40 TOPS INT4", before comparing it with anything. If a product page will not supply the label, treat the number as marketing and look for the datasheet.

What this means when buying#

Treat TOPS as a first filter and not as a verdict. Use the rating to sort products into tiers. Then check four things: the precision, whether the figure is sparse, the memory size and bandwidth, and published results for a workload like yours. Pricing moves, so confirm it with a seller before buying.

If you are weighing edge boards, the Jetson Orin Nano Super review and the Jetson vs Raspberry Pi comparison put these ideas into practice. For laptops with NPUs, see the AI workstations and PCs guide. Browse the full AI accelerators hub for more.

One product in this explainer is sold in India as a retail kit:

  • NVIDIA Jetson Orin Nano Super Developer Kit 8GB: a 67 sparse INT8 TOPS board with a CUDA software stack and a bundled 19 V supply.

When you compare AI laptops sold in India, ask for the maker's own TOPS page for the exact model number, since regional variants of the same laptop can carry different chips.

Frequently asked questions#

What does one TOPS mean?

One TOPS is one trillion operations per second. Makers use it as a peak rating for an AI chip, usually at low-precision integer arithmetic, so it describes a ceiling and not the speed you will see on your own model.

Is a higher TOPS number always better?

No. Precision, sparsity, memory and software change what the number means. Two chips with the same rating can differ widely in real speed, so compare published results for your own kind of model before choosing.

What is the difference between TOPS and TFLOPS?

TOPS usually counts integer operations at low precision. TFLOPS counts floating-point operations. They measure different arithmetic and should not be compared directly, so check which type of operation a maker is quoting first.

How many TOPS do I need for a Copilot+ PC?

Microsoft says many of its new Windows AI features need an NPU that can run at 40+ TOPS. A laptop below that level can still run Windows, but those features may be missing, so check the NPU rating before buying.

Why do some makers list sparse TOPS?

Sparse ratings assume the model skips zero values in a specific pattern. NVIDIA labels the AGX Orin's 275 TOPS and the Orin Nano Super's 67 TOPS as sparse INT8, for example.

Do TOPS matter for running language models?

Less than memory capacity and bandwidth. A model must fit in memory, and its speed depends on how fast the weights can be read. A chip with a high TOPS rating but little fast memory will still feel slow.

Sources & credits#

We have not tested these products ourselves. Specs come from the makers; benchmark figures come from the sources we name.

Facts and figures in this article come from these sources. Articles are compiled from the makers' published specifications and the benchmarks named above, and checked by the NextAIGrid Editorial Team.

Disclaimer

This explainer is for information only. Specifications and availability change, so check the maker's page. Found an error? Tell us.

How was this explainer put together?

It draws on 1 named source: maker specification pages and published third-party benchmarks, listed in the article. We have not tested these products ourselves. Specs come from the makers; benchmark figures come from the sources we name. Specifications and sources were last checked on 17 November 2026. Read how we research and our editorial policy.

Spotted an error? Tell us. Confirmed fixes are listed on the corrections page.

What questions do readers ask next?

Back to top