In this article
- What VRAM does for a language model
- Memory needed by model size and quantisation
- Working through the formula
- What changes for image models
- Why context length adds to the total
- Which memory size fits which models
- VRAM is not the only limit
- What this means when buying
- Frequently asked questions
- Sources & credits
Short answer
A model needs about 0.5 byte per parameter at 4-bit and 1 byte at 8-bit, plus room for context. LLM Hardware puts the Q4_K_M version of an 8B model at 6.8 GB and the 27B version at 18.2 GB, so a 16 GB card suits models of about 20B or fewer at 4-bit.
- Weights take roughly 0.5 byte per parameter at 4-bit, 1 byte at 8-bit and 2 bytes at 16-bit.
- Context length adds memory on top of the weights, so leave a 15 to 20 percent buffer.
- A 16 GB card suits models of about 20B or fewer at 4-bit; 24 GB reaches about 36B.
- Memory bandwidth, not only size, decides how fast a model speaks.
- Pick 1ASUS Prime GeForce RTX 5070 12GB GDDR7 OC
- Pick 2ASUS Dual GeForce RTX 5060 Ti 16GB GDDR7 OC
- Pick 3ASUS Prime Radeon RX 9070 XT 16GB GDDR6 OC
What VRAM does for a language model#
VRAM is the memory on the graphics card. A language model is mostly a very large table of numbers called weights, and the card reads those weights again and again to produce each word. If the weights sit in VRAM, the card reads them at full speed. If they spill into ordinary system memory, every read crosses the much slower link between the card and the PC, and speed drops sharply.
That is why VRAM size is the first spec to check when you plan to run a model at home. A card with plenty of compute but too little memory cannot load the model at all. A card with enough memory and modest compute will run it slowly but correctly.
Three things decide how much memory a model uses: the number of parameters, the precision of each weight and the length of the text you give it. Parameters are counted in billions, written as 8B, 14B or 70B. Precision is set by quantisation, which stores each weight in fewer bits. The third factor, context, is covered further down.
Memory needed by model size and quantisation#
Quantisation shrinks a model by storing weights in 8 or 4 bits instead of 16. LLM Hardware gives the raw weight size per parameter as about 2 bytes at 16-bit, about 1 byte at 8-bit (Q8_0) and about 0.5 byte at 4-bit (Q4_K_M). Real files run a little larger. Its estimate adds a 20 percent overhead and 2 GB of reserve to the weights.
The table below shows the estimates that LLM Hardware publishes. They come from a formula, not from measurements, so treat them as planning figures.
| Model | Precision | Estimated memory |
|---|---|---|
| Mistral 7B | Q4_K_M | 6.2 GB |
| Llama 3.1 8B | Q4_K_M | 6.8 GB |
| Llama 3.1 8B | Q8_0 | 11.6 GB |
| Llama 3.1 8B | F16 | 21.2 GB |
| 14B model | Q4_K_M | 10.4 GB |
| 27B dense model | Q4_K_M | 18.2 GB |
| Llama 3.1 70B | Q4_K_M | 44 GB |
A real file gives a useful cross-check. In the llama.cpp community logs, the Llama 2 7B file at Q4_0 is 3.56 GiB for 6.74 billion parameters, which is close to half a byte per parameter plus a small overhead.
Notice how quickly the figures climb. Going from 4-bit to 8-bit nearly doubles the 8B model's memory, and going to 16-bit triples it. This is why most local users run 4-bit or 8-bit files. Pick the precision that leaves the most headroom for context without making answers noticeably worse.
Working through the formula#
You can check the table yourself. LLM Hardware's rule is parameters in billions, times bytes per parameter, times 1.2, plus 2 GB. Take the 14B model in Q4_K_M form: 14 times 0.5 gives 7, times 1.2 gives 8.4, plus 2 gives 10.4 GB. For the 27B model at the same precision: 27 times 0.5 gives 13.5, times 1.2 gives 16.2, plus 2 gives 18.2 GB. For the 70B model, 70 times 0.5 is 35, times 1.2 is 42, plus 2 is 44 GB.
The same steps work for any model you meet. Find the parameter count on the model page, choose the precision of the file you plan to download, and run the arithmetic. If the answer lands within a gigabyte or two of your card's memory, assume it will not fit once you add a long prompt.
Two cautions apply. First, the formula is an estimate by one publisher, and real files differ with the quantisation scheme: Q4_K_M files are slightly larger than a flat half byte per parameter. Second, mixture-of-experts models need memory for all their experts, even though they read only some of them for each token. A model described as 20B with only part of it active still needs room for the full 20B.
What changes for image models#
Image generators follow the same logic with different numbers. ComputingForGeeks says an 8 GB card covers SDXL and distilled models, which is why 8 GB cards remain popular for image work even though they are poor for language models.
Larger image models need more. The same site puts the FLUX.1-dev fp8 checkpoint at 17.2 GB on disk, so even a 16 GB card cannot hold it whole. It measured 63.91 seconds per image on an RTX 3060 12GB and 37.02 seconds on an RTX 5060 Ti 16GB, both with weights streamed from system memory, and recommends 32 GB of system RAM as a floor and 64 GB for comfortable work. The lesson matches the language-model case: when the model does not fit, the system still runs, but speed and system RAM become the cost.
For a fuller look at image generation cards, see our guide to the best GPU for Stable Diffusion.
Why context length adds to the total#
Context is the amount of text the model holds in mind at once: your prompt, any documents and its own reply so far. The model stores working data for all of it in memory, and this grows with the length of the text. A short chat adds little. A long document or a long conversation adds a lot.
LLM Hardware advises a 15 to 20 percent buffer above the estimate, and its own formula already adds a 2 GB reserve. In practice, a model that fits on the card with only a hair of space will work for a short question and then slow down or fail on a long document.
Hardware Corner makes the same point from the other direction. It reports that a 32 GB card holds the same models as a 24 GB card but allows far longer context, which is why extra memory pays off with long documents. If you plan to ask questions over long files, add memory headroom to every row of the table above.
Which memory size fits which models#
LLM Hardware matches card sizes to model sizes at 4-bit, and Hardware Corner gives its own limits for the cards it ranks. The two agree in outline.
| Card memory | Models that fit at 4-bit | Source note |
|---|---|---|
| 8 GB | Up to about 10B | LLM Hardware estimate |
| 12 GB | Up to about 16B | LLM Hardware estimate |
| 16 GB | Up to about 20B | Hardware Corner |
| 24 GB | Up to about 36B, including 27B to 32B models | LLM Hardware and Hardware Corner |
| 48 GB | 70B models | LLM Hardware and Hardware Corner |
So a 16 GB card takes the 8B model even at 8-bit, with context to spare, and the 14B model at 4-bit. The 27B model at 4-bit is out of reach, since LLM Hardware gives 18.2 GB for it. A preprint by Knoop and Holtmann adds a practical data point: a Gemma 3 12B model did not run on a single 16 GB card in their vLLM setup, which shows that the serving software and format also change what fits.
For the image-model side, ComputingForGeeks says the FLUX.1-dev fp8 checkpoint is 17.2 GB on disk, so a 16 GB card cannot hold it whole.
VRAM is not the only limit#
Once a model fits, speed depends on memory bandwidth, which is how fast the card reads its own memory. Token generation reads the whole model for every token, so a wider or faster memory bus produces more tokens per second.
Hardware Corner's llama.cpp table shows the effect between two 16 GB cards. On Qwen3 8B at Q4, the RTX 5070 Ti with a 256-bit bus reached 87.54 tokens per second, and the RTX 5060 Ti 16GB with a 128-bit bus reached 51.41. Both hold the model easily, so the gap is bandwidth and compute, not memory size.
System memory matters too. When a model is larger than VRAM, some software can offload layers to system RAM, but each offloaded layer slows every token. It works as a fallback, and it is a poor plan for daily use. Plan the card's memory around the model you want to run, and treat offloading as an emergency.
Finally, mixture-of-experts models behave differently. They hold many parameters but read only some for each token, so a 20B model of that kind can generate faster than a smaller dense model. On the RX 9060 XT 16GB, Phoronix recorded gpt-oss-20b at 102.70 tokens per second and Qwen3 8B at 36.53, both at Q8_0 with the Vulkan backend of llama.cpp under Linux. The weights still have to fit in memory, so the VRAM requirement follows the full parameter count.
AMD gives that card a 128-bit bus with 16 GB of GDDR6, and quotes memory bandwidth of up to 320 GB/s. Buying by memory size alone is therefore a start, not a finish. Pair the memory figure with a bandwidth figure from the maker, and with a published tokens-per-second result for a model of the size you plan to run.
What this means when buying#
Work backwards from the model. Decide the largest model you want to run, add a 15 to 20 percent buffer, and pick the smallest card that clears the number. For most home users that is an 8B to 14B model, which points to 12 GB or 16 GB.
Our guide to the best GPU for local LLM compares three 16 GB cards on tokens per second. The best 16 GB GPU for AI guide covers the same memory class across vendors, and the best high-VRAM GPU for AI guide covers cards above 16 GB. If your models are small, the best budget GPU for machine learning shows where 8 GB falls short. The GPU category page lists everything.
Three short scenarios show how the arithmetic plays out.
A student who wants a coding assistant picks the 8B model in Q4_K_M form. LLM Hardware gives 6.8 GB for that file, so even a 12 GB card leaves several gigabytes for context, and the choice comes down to speed and software.
A developer who wants better answers moves up to the 14B model in Q4_K_M form, listed at 10.4 GB by the same source. A 12 GB card holds it with little room for long prompts, so 16 GB is the comfortable target. Hardware Corner's results for 16 GB cards were run on this model class, with the RTX 5070 Ti reaching 57.98 tokens per second on Qwen3 14B and the RTX 5060 Ti 32.91.
A researcher who wants the 27B model in Q4_K_M form needs about 18.2 GB, which rules out every 16 GB card. That buyer should look at 24 GB and above, and our best high-VRAM GPU for AI guide covers that tier.
Three cards illustrate the memory steps discussed here.
- ASUS Prime GeForce RTX 5070 12GB GDDR7 OC: 12 GB of GDDR7 on a 192-bit bus, enough for a 14B model at 4-bit by LLM Hardware's estimate.
- ASUS Dual GeForce RTX 5060 Ti 16GB GDDR7 OC: 16 GB of GDDR7, enough for an 8B model at 8-bit with context.
- ASUS Prime Radeon RX 9070 XT 16GB GDDR6 OC: 16 GB of GDDR6 across a 256-bit bus, aimed at AMD buyers.
Since graphics memory cannot be upgraded later, Indian buyers who expect to try larger models should pay for the extra VRAM now, not buy a second card down the line.
For compact systems, the best low-profile GPU for AI guide lists slim cards and how much memory they offer.
For readers in India, the extra rupees for a larger-memory card usually cost less than replacing a card that proves too small within a year.
Frequently asked questions#
How much VRAM do I need to run an 8B model?
LLM Hardware estimates 6.8 GB at Q4_K_M and 11.6 GB at Q8_0, plus room for context. A 12 GB or 16 GB card works well.
Can a 16 GB card run a 27B model?
Not at 4-bit with context. LLM Hardware's figure for the 27B dense model in Q4_K_M form is 18.2 GB, which is above 16 GB before any context is added, so you need a larger card or heavier compression.
Is 8 GB enough for local LLMs?
It suits models of about 10B parameters or fewer at 4-bit, with little context left over. Many users outgrow it quickly, so choose 16 GB if you expect to try larger models or longer prompts.
What is quantisation?
It stores each weight in fewer bits, such as 8 or 4, which shrinks the model and its memory use at a small cost in quality.
Does more bandwidth matter more than more memory?
Memory decides whether a model loads. Once it loads, bandwidth decides how fast it generates. A card with more memory but a narrow bus can run bigger models, while a faster bus speeds up smaller ones.
Can I use system RAM instead of VRAM?
Software can offload layers to system RAM, but speed drops sharply, so it works only as a fallback. System memory is far slower than graphics memory, so plan to fit the whole model on the card where you can.
Sources & credits#
We have not tested these products ourselves. Specs come from the makers; benchmark figures come from the sources we name.
- ASUS India, ASUS Prime GeForce RTX 5070 12GB GDDR7 OC specifications (opens in a new tab)
- NVIDIA, GeForce RTX 5070 product page (opens in a new tab)
- ASUS India, ASUS Dual GeForce RTX 5060 Ti 16GB GDDR7 OC specifications (opens in a new tab)
- NVIDIA, GeForce RTX 5060 Ti product page (opens in a new tab)
- ASUS India, ASUS Prime Radeon RX 9070 XT 16GB GDDR6 OC specifications (opens in a new tab)
- AMD, Radeon RX 9070 XT product page (opens in a new tab)
- LLM Hardware, LLM quantization guide (opens in a new tab)
- Hardware Corner, The Definitive GPU Ranking for LLMs (opens in a new tab)
- llama.cpp GitHub, ROCm scoreboard discussion (community-reported) (opens in a new tab)
- Knoop and Holtmann, Private LLM Inference on Consumer Blackwell GPUs (arXiv preprint) (opens in a new tab)
- ComputingForGeeks, Best GPU for ComfyUI and Stable Diffusion by VRAM (opens in a new tab)
- Phoronix, Intel Arc B580 vs AMD Radeon RX 9000 vs NVIDIA RTX 50 for llama.cpp Vulkan (opens in a new tab)
Facts and figures in this article come from these sources. Articles are compiled from the makers' published specifications and the benchmarks named above, and checked by the NextAIGrid Editorial Team.
Disclaimer
This article is for information only. Specifications and availability change, so check the maker page before you buy. Found an error? Tell us.
How was this explainer put together?
It draws on 8 named sources: maker specification pages and published third-party benchmarks, listed in the article. We have not tested these products ourselves. Specs come from the makers; benchmark figures come from the sources we name. Specifications and sources were last checked on 17 November 2026. Read how we research and our editorial policy.
Spotted an error? Tell us. Confirmed fixes are listed on the corrections page.