Explainer

Data Centre AI Accelerators Explained

In this article
  1. What a data centre accelerator is
  2. Memory: the first spec to read
  3. Precision: where recent gains come from
  4. How the MLPerf results are produced
  5. Interconnect: how many chips act as one
  6. What about AMD
  7. What this means when buying
  8. Frequently asked questions
  9. Sources & credits

Short answer

Data centre AI accelerators are large GPUs built to train and serve big models in racks. They offer far more memory, higher memory bandwidth and faster chip-to-chip links than consumer hardware. In NVIDIA's MLPerf Inference v5.0 entries, an eight-GPU B200 system reached 98,858 tokens per second on Llama 2 70B against 34,988 for an eight-GPU H200 system.

  • Memory size and bandwidth matter as much as raw arithmetic.
  • Published results are for whole systems, usually eight accelerators, not single chips.
  • Lower precision such as FP4 is a large part of recent gains.
  • These parts are sold to cloud and enterprise buyers, not through consumer shops.

What a data centre accelerator is#

A data centre accelerator is a processor designed to run neural networks at scale. NVIDIA's H200 and B200 are GPUs. AMD's Instinct MI300X and MI325X are accelerators from AMD's GPU line. They are fitted into servers in groups of eight, or into rack-scale systems with many more.

Consumer GPUs share some of the same ideas, but the data centre parts add three things. The first is high-bandwidth memory (HBM), stacked next to the processor to move data faster. The second is a fast interconnect so many chips work as one. The third is support for the lowest-precision number formats that large models can use without losing accuracy.

Most readers will reach this hardware through a cloud provider, a college cluster or a company data centre, not through a shop.

Memory: the first spec to read#

The makers' published figures show a clear ladder in memory size and bandwidth.

Table
AcceleratorMemoryBandwidthPower
NVIDIA H200141 GB HBM3e4.8 TB/sUp to 700 W, configurable
AMD Instinct MI300X192 GB HBM35.3 TB/s750 W peak typical board power
AMD Instinct MI325X256 GB HBM3E6 TB/s1000 W peak typical board power

NVIDIA says the H200 is the first GPU to offer 141 GB of HBM3e at 4.8 TB/s, which it describes as nearly double the capacity of the H100 and 1.4 times the bandwidth. AMD lists the MI325X at 256 GB and 6 TB/s, and the MI300X at 192 GB and 5.3 TB/s.

For comparison, the largest consumer cards in our GPU guides have a small fraction of this memory. Our VRAM explainer shows how model size maps to memory need. In a data centre, one chip with 256 GB can hold a model that would need several consumer cards, which is why memory leads the spec sheet.

Power: the other side of the table

The power column shows why these parts live in data centres. NVIDIA lists up to 700 W for the H200 SXM, and says the NVL version, which suits air-cooled racks, goes up to 600 W. AMD lists 750 W for the MI300X and 1000 W for the MI325X.

Those figures are per accelerator. An eight-chip server multiplies them, before counting the host CPUs and networking. Cooling and electrical capacity are the buyer's first problems, and both are specified by the makers. This is why the parts are rarely bought one at a time.

Precision: where recent gains come from#

NVIDIA's Blackwell chips support FP4 and FP6 formats. NVIDIA lists its HGX B200 board, which carries eight Blackwell SXM GPUs, with 1.4 TB of total memory and 1.8 TB/s of NVLink bandwidth between GPUs. NVIDIA says FP4 gives twice the peak throughput of FP8 on Blackwell, and that its software can use FP4 while meeting the benchmark's accuracy requirements. That is a lot of the reason the B200 pulls so far ahead of the H200 in the results below. It is an architectural change, not only a faster chip.

When you read a chip-to-chip comparison, ask which precision each chip ran at. A result at FP4 against one at FP8 is a comparison of two different jobs. Our TOPS explainer explains why the same warning applies to every rating.

How the MLPerf results are produced#

MLPerf Inference, run by MLCommons, publishes results for fixed models in the Closed division, where rules limit how a submitter may change the model. Systems are tested in scenarios. In Offline, all queries are available at once and the system runs as fast as it can. In Server, queries arrive at random times and the system must meet latency limits.

NVIDIA's v5.0 write-up lists the benchmarks in the round, including ResNet-50, RetinaNet, 3D U-Net, DLRMv2, GPT-J, Stable Diffusion XL, Llama 2 70B and Mixtral 8x7B. It also adds Llama 3.1 405B and a stricter Llama 2 70B Interactive test, which sets latency limits of 450 ms to the first token and 40 ms per output token.

Most results are for whole systems. MLPerf does not treat per-GPU performance as a primary metric. NVIDIA derives it by dividing throughput by the number of accelerators, and says so in its notes.

What the numbers show

The charts below use the figures NVIDIA published in its v5.0 results post, from the Closed data-centre division. Each compares an eight-GPU B200 system with an eight-GPU H200 system. The B200 ran at FP4, which makes it a result of chip and precision together. Higher is better.

Llama 2 70B: Throughput (tokens/s, higher is better)

Llama 2 70B: ThroughputNVIDIA DGX B200 (8x B200): 98858 tokens/sNVIDIA DGX B200 (8x B200)98858 tokens/sNVIDIA 8x H200 server: 34988 tokens/sNVIDIA 8x H200 server34988 tokens/s

2 products. Highest: NVIDIA DGX B200 (8x B200), 98858 tokens/s. Lowest: NVIDIA 8x H200 server, 34988 tokens/s.

Llama 2 70B: Throughput
Producttokens/s
NVIDIA DGX B200 (8x B200)98858
NVIDIA 8x H200 server34988
Source: NVIDIA Technical Blog (MLPerf Inference v5.0 entries) (opens in a new tab). MLPerf Inference v5.0 Closed Datacenter, Offline. Vendor-published MLCommons Closed data-centre entries; blog dated 2025-04-02; per-system (8 accelerators) not per-GPU; B200 entries used FP4 so not precision-matched to H200.

About this data: figures are copied from the named source, not measured by us, and the test system and settings are those the source states above. See all benchmark sources and credits.

On Llama 2 70B in the Offline scenario, the B200 system reached 98,858 tokens per second and the H200 system 34,988. NVIDIA puts the speed-up at 2.8 times.

Llama 2 70B: Throughput (tokens/s, higher is better)

Llama 2 70B: ThroughputNVIDIA DGX B200 (8x B200): 98443 tokens/sNVIDIA DGX B200 (8x B200)98443 tokens/sNVIDIA 8x H200 server: 33072 tokens/sNVIDIA 8x H200 server33072 tokens/s

2 products. Highest: NVIDIA DGX B200 (8x B200), 98443 tokens/s. Lowest: NVIDIA 8x H200 server, 33072 tokens/s.

Llama 2 70B: Throughput
Producttokens/s
NVIDIA DGX B200 (8x B200)98443
NVIDIA 8x H200 server33072
Source: NVIDIA Technical Blog (MLPerf Inference v5.0 entries) (opens in a new tab). MLPerf Inference v5.0 Closed Datacenter, Server. Vendor-published MLCommons Closed data-centre entries; blog dated 2025-04-02; per-system (8 accelerators) not per-GPU; B200 entries used FP4 so not precision-matched to H200.

About this data: figures are copied from the named source, not measured by us, and the test system and settings are those the source states above. See all benchmark sources and credits.

In the Server scenario, the figures are 98,443 and 33,072 tokens per second, which NVIDIA lists as 3 times. Server mode adds latency limits, so the result shows the chips under a more realistic serving load.

Mixtral 8x7B: Throughput (tokens/s, higher is better)

Mixtral 8x7B: ThroughputNVIDIA DGX B200 (8x B200): 128148 tokens/sNVIDIA DGX B200 (8x B200)128148 tokens/sNVIDIA 8x H200 server: 62630 tokens/sNVIDIA 8x H200 server62630 tokens/s

2 products. Highest: NVIDIA DGX B200 (8x B200), 128148 tokens/s. Lowest: NVIDIA 8x H200 server, 62630 tokens/s.

Mixtral 8x7B: Throughput
Producttokens/s
NVIDIA DGX B200 (8x B200)128148
NVIDIA 8x H200 server62630
Source: NVIDIA Technical Blog (MLPerf Inference v5.0 entries) (opens in a new tab). MLPerf Inference v5.0 Closed Datacenter, Offline. Vendor-published MLCommons Closed data-centre entries; blog dated 2025-04-02; per-system (8 accelerators) not per-GPU; B200 entries used FP4 so not precision-matched to H200.

About this data: figures are copied from the named source, not measured by us, and the test system and settings are those the source states above. See all benchmark sources and credits.

Mixtral 8x7B uses a mixture-of-experts design, where only part of the model runs for each token. In the Offline scenario the B200 system reached 128,148 tokens per second against 62,630. NVIDIA lists the Server results as 126,845 and 61,802, a speed-up of 2.1 times in both scenarios.

A few cautions. These are NVIDIA's own submissions, reported on NVIDIA's blog, so they are vendor-published entries to a public benchmark. The systems are not matched on precision. And the figures are for eight-chip systems, so they do not translate into a rating for a single card. NVIDIA also reports that on the Llama 2 70B Interactive test the B200 system had 3.1 times the throughput of an eight-H200 system, and that its GB200 NVL72 rack system reached up to 3.4 times the per-GPU performance of eight H200s on Llama 3.1 405B.

Software moves the results too

The same chip can move up the table through software alone. NVIDIA says that on Llama 2 70B, H100 throughput rose by up to 1.5 times over the previous year through software optimisations in its TensorRT-LLM software. Hopper chips were first introduced years earlier, and they still improved.

This is worth remembering when you read old results. A published number is a snapshot of a chip, a software version and a model. Newer rounds will differ. Check the round and the date before using any figure, and prefer results that name the software versions.

Interconnect: how many chips act as one#

A single accelerator cannot hold the largest models, so systems join many of them. NVIDIA uses NVLink and NVSwitch for this. Its DGX B200 system connects eight Blackwell GPUs that way, and its GB200 NVL72 is a rack-scale system with 36 Grace CPUs and 72 Blackwell GPUs, fully connected with NVLink and NVSwitch. NVIDIA lists 900 GB/s of NVLink bandwidth on the H200, and says the H200 NVL can link up to four GPUs with a bridge.

The link speed matters because the chips must share results many times per second while a big model runs. A fast link lets the group behave like one large accelerator. A slow link leaves chips waiting for each other. That is why a system's result cannot be predicted from a single chip's memory and arithmetic ratings alone.

NVIDIA says the GB200 NVL72 gave up to 30 times the system-level throughput of an eight-GPU H200 system on Llama 3.1 405B. The gain comes from higher per-GPU performance and from nine times as many GPUs sharing a single NVLink domain.

Questions to ask of any result

Use the checklist below when a vendor shares a benchmark figure.

  1. Which round and which software version? Results age quickly, as the H100 example shows.
  2. Which precision did each chip run at? FP4 against FP8 is not a like-for-like test.
  3. Is it a system or a single chip? Divide by the accelerator count only if the source says how.
  4. Which scenario? Offline measures peak throughput. Server adds latency limits and is closer to serving real users.
  5. Which division? The Closed division limits changes to the model, so entries are easier to compare.
  6. Who published it? Vendor blogs report real entries, but they pick the comparisons that suit them.

Reading the vendor claims fairly

Each maker publishes the comparison that suits its product. NVIDIA's post compares Blackwell with Hopper, its own earlier generation, and AMD's post compares its newest part with its own earlier one. Neither is a neutral cross-vendor test. Where MLCommons publishes the full table, the entries from several makers can be read side by side, which is the better way to judge a market. Treat any single vendor's chart as one input and look for the original entries.

What about AMD#

AMD publishes its own MLPerf results. In its blog about MLPerf Inference 6.0, AMD reported 100,282 tokens per second for an MI355X eight-GPU platform on Llama 2 70B in the Server scenario, and said that is 3.1 times its MI325X submission. AMD also gave its own comparison with NVIDIA's B200 submission. Those are AMD's claims from a later round, so this article does not chart them against the NVIDIA figures from an earlier round.

The honest reading is that both vendors are improving quickly. Memory is where AMD leads on paper, with 256 GB on the MI325X against 141 GB on the H200. Software and ecosystem are where buyers weigh NVIDIA's long-standing CUDA advantage. Our CUDA vs ROCm explainer covers that split.

What this means when buying#

You will not buy an H200 or an MI300X from a shop, and the pages for these parts are information only. What you can do with the results is read the market. Memory capacity and bandwidth lead the spec sheet. Precision support decides real speed. Software maturity decides whether the hardware is usable.

For hardware you can buy in India, start with the AI accelerators hub, where the edge boards sit at the opposite end of the scale. The Jetson Orin Nano Super review shows a board that shares NVIDIA's software stack with the data centre parts. For desktop GPUs that run local models, see the GPU buying guides.

For Indian readers, these systems matter mostly through cloud providers and research labs, so most buyers in India rent such hardware by the hour and do not buy it.

Frequently asked questions#

What is HBM and why does it matter?

High-bandwidth memory is stacked beside the processor to move data much faster than standard graphics memory. Large models are limited by how fast they can read their weights, so HBM is central to these chips.

Can I buy an H200 or B200 in India?

They are sold to cloud and enterprise buyers through server makers and distributors, not through consumer shops. Most people use them through a cloud service.

Why are the B200 results so much higher than the H200 results?

NVIDIA attributes it to the Blackwell design, which supports FP4, and to its software. The B200 ran at FP4 in these entries, so the comparison is not matched on precision.

What does MLPerf measure?

MLPerf Inference, run by MLCommons, measures how fast systems run fixed AI models under set rules, in scenarios such as Offline and Server. Because the rules are fixed, results from different vendors can be compared fairly.

Are the results per chip or per system?

They are per system, usually eight accelerators in one server. Per-GPU figures are derived by dividing the system result by the number of chips, so they are estimates and not a figure the benchmark itself reports.

How does an MI325X compare with an H200 on memory?

AMD lists 256 GB of HBM3E at 6 TB/s for the MI325X. NVIDIA lists 141 GB of HBM3e at 4.8 TB/s for the H200. The AMD part has more capacity and bandwidth on paper, though real speed also depends on software.

Do these chips matter for a home AI build?

Indirectly. They train the models you run locally and set the direction for consumer hardware, but they are not part of a home build. Their cost and power needs put them out of reach for home buyers.

Sources & credits#

We have not tested these products ourselves. Specs come from the makers; benchmark figures come from the sources we name.

Facts and figures in this article come from these sources. Articles are compiled from the makers' published specifications and the benchmarks named above, and checked by the NextAIGrid Editorial Team.

Disclaimer

This explainer is for information only. Specifications and availability change, so check the maker's page. Found an error? Tell us.

How was this explainer put together?

It draws on maker specification pages and published third-party benchmarks, named in the article. We have not tested these products ourselves. Specs come from the makers; benchmark figures come from the sources we name. Specifications and sources were last checked on 17 November 2026. Read how we research and our editorial policy.

Spotted an error? Tell us. Confirmed fixes are listed on the corrections page.

What questions do readers ask next?

Back to top