LLM Inference Explained: Weights, Quantization, KV Cache and MoE

| 2 Min Read
A practical guide to the model components and inference steps that shape LLM quality, speed and memory use, from quantization to retrieval.

Running a large language model locally is a memory problem as much as a compute problem. Model weights, numerical precision, context length and the KV cache all affect whether a model fits on a machine and how quickly it responds. For anyone comparing hardware or choosing a model to run, those terms describe practical tradeoffs rather than abstract AI theory.

This guide focuses on text LLMs used for chat, programming and agentic tasks. It follows the path from the trained model to a generated answer, including techniques such as mixture-of-experts routing and retrieval-augmented generation that change how models use their capacity and information.

What a model contains

A language model is a system of learned statistical relationships derived from large training datasets. Those datasets may include text from the internet and books, along with transcripts and other material. During use, the model processes tokens—units that can represent whole words, word fragments, punctuation, code or other symbols—and predicts which tokens should come next.

The learned values that shape those predictions are called weights. They determine how strongly different parts of the model respond to patterns in the input. A model’s weights are set through training; changing them requires further training or another adjustment process. The words themselves are not stored as the model’s learned relationships, though tokenization information is packaged with models so that text can be converted into tokens and back.

Model specifications commonly use parameter counts such as 4B, 8B or 20B, where B means billions of parameters. Consumer models are often measured in billions, while some frontier models are described in trillions. A larger count can mean more capacity, but it doesn’t guarantee better results. The quality of training, model design and learned weights also matter.

The distinction between open-weight and closed models affects what users can access. Projects such as Llama, DeepSeek and Kimi have published model weights that users can download and run, subject to each model’s terms. Access to weights can allow customization, but it doesn’t reveal the full training data or how the model was created. Open weights are closer to a customizable application than a complete disclosure of the process behind it.

Precision determines the memory tradeoff

Labels such as FP16, BF16, INT8 and INT4 describe ways to represent numbers. Different formats use different amounts of memory and provide different levels of numerical precision. GPUs are especially sensitive to that choice: memory is limited, and lower-precision formats can make it practical to run models that would otherwise exceed available VRAM.

Training often uses more precise values than a deployed model needs. Keeping every weight at training precision can make an LLM too large or slow for many uses. Quantization reduces the precision of weights, in effect rounding them into a more compact representation. The process can reduce memory demands and improve the chance that a model will run on consumer hardware, with some potential loss in output quality.

That tradeoff depends on the model and the task. Frontier systems often use formats such as BF16 to retain response quality and nuance. Open-weight models may be offered in several quantized versions. For example, Llama 3.3 70B can be run in BF16 or in an INT4 form. The lower-precision version needs less memory, which can make local use possible on a broader range of machines, including systems with a Mac Studio or a graphics card.

There isn’t one best data type for every workload. Developers choose formats based on the available hardware and the accuracy they need. For users, the practical question is whether the memory savings are worth any change in the model’s responses.

Context and the KV cache shape long sessions

A model’s context window sets how many tokens it can process as part of a session. It must hold more than the latest prompt: conversation history, system instructions, uploaded files and retrieved material can all take up space. A million-token window is common among frontier models, but increasing the limit also raises memory requirements.

To a person, a conversation may contain distinct elements such as instructions, source documents and code. For the model, these are all represented as tokens in its context. The distinction between data and code is therefore difficult to enforce perfectly, as illustrated by research showing how AI agents can be tricked through supplied content.

More context can give a model more information to work with, but a larger window doesn’t ensure that every detail will influence the answer. Models can lose track of material buried in the middle of a long input, a behavior known as the “lost in the middle” effect. The useful context length depends on how well a particular model handles information across its window, as well as the memory available to run it.

The KV cache stores intermediate data from a model’s processing so it can be reused later in the same session. Without that cache, the model would need to recalculate information about earlier tokens as it generated each new one. Reuse matters in extended conversations: processing a history of hundreds of thousands of tokens repeatedly would add substantial work and delay.

KV cache data also consumes memory. With long contexts, it can take more space than the model weights themselves. Some systems reduce the cache’s precision to save VRAM, accepting a possible small effect on accuracy in exchange for supporting longer sessions or leaving more memory for other parts of inference.

Mixture-of-experts activates selected model parts

Mixture-of-experts, or MoE, places multiple sub-models inside a larger model. The experts aren’t necessarily specialists in human topics such as programming or art. They learn different patterns among tokens, and their strengths can emerge from training rather than from explicit subject labels.

A router chooses which experts to activate for each token. Because the model uses only selected parts at a time, an MoE system can have a large total parameter count while requiring less computation for each step than a dense model that activates all its weights. That can improve speed, though the full set of experts still has to be stored or otherwise made available, so memory requirements can remain high.

The routing also changes from token to token. One input may activate a different combination of experts than the next, based on the patterns the router detects. For hardware planning, this means total parameter count alone may not describe the model’s compute needs; memory capacity and the model’s active parameter count both matter.

Retrieval brings outside information into a prompt

Retrieval-augmented generation (RAG) pairs a model with information retrieved from another source. A chat application can search the web, company documents, email or spreadsheets, then pass relevant material into the model’s context. The model can use that material to answer questions without relying only on what it learned during training.

Retrieval is valuable when a query depends on current information or private documents. A separate, smaller model may help generate search queries or decide what to look up. Some applications make retrieval available through a web search control; others may connect to a user’s document services.

The cost is additional processing and context. A prompt and answer might account for around 10,000 tokens, while retrieved pages can contribute hundreds of thousands more. That raises the work required for prefill and can increase memory use. Perplexity, for example, was built with an emphasis on RAG; its approach shows how retrieval can make a system more context-aware even when the underlying model is also used elsewhere.

Inference has two distinct phases

Inference is the process of running a trained model to produce output. It begins with prefill, when the system tokenizes and processes the prompt, conversation history and any supplied or retrieved material. This phase builds the KV cache and processes input tokens in parallel.

The time required to receive the first generated token is called time to first token (TTFT). Prompt length, retrieval volume, context size and hardware all affect it. Benchmarks may report prefill speed in tokens per second alongside TTFT, helping distinguish the time spent processing input from the time spent generating the answer.

After prefill comes decode. The model generates its response one token at a time, using the KV cache as it goes. Each new token depends on the preceding sequence, so decode is sequential. It’s often limited by memory bandwidth: the hardware must repeatedly read model weights and cache data to produce each token. Decode speed is commonly reported as tokens per second (TPS).

Speculative decoding can accelerate this stage. A draft mechanism proposes several upcoming tokens, and the main model checks those proposals in a parallel pass. Accepted tokens can reduce the number of costly sequential steps; rejected guesses are discarded, and the main model’s verified output remains authoritative. Multi-token prediction (MTP), in which a model is trained to predict several upcoming tokens, can provide the draft mechanism for speculative decoding.

These concepts explain why the same model can feel very different across machines and applications. Weight precision affects whether it fits; context and retrieval shape prefill; the KV cache grows with the session; and decode speed depends heavily on memory bandwidth. When comparing local LLM setups, those details offer a more useful picture than parameter count alone.

Source: [email protected] (Bruno Ferreira) · www.tomshardware.com

Comments

Please sign in to comment.