Inference guide

Already quantized your LLM? Here’s what to look at next.

Already running FP8 or INT4 and still spending too much on inference? Further savings can come from changing the model’s structure, improving GPU execution, or tuning how requests are served. Start with the cost or capacity problem that remains.

Start with what still limits your deployment

Choose the outcome you need: fewer GPUs for today’s traffic, more capacity before your next hardware purchase, or faster responses. Use your current quantized deployment, with its existing serving settings, as the baseline.

Use a sample of request lengths and traffic levels to choose an experiment:

  • Memory fills up. Check how much goes to weights and how much goes to active requests. Weight compression is one option; large KV caches may call for changes to cache precision or request scheduling.
  • Answers arrive too slowly. Separate prompt processing from token generation, then profile the slow part. Compare kernel and batching options at the same response-time target.
  • GPUs spend time idle. Check request distribution and how capacity follows demand. Test scheduling or scaling changes before adding hardware.

Look at the model and the software running it

Quantization reduces the precision used to store or compute values. Further optimization can change the model’s structure and the way it executes.

  • Smaller layers. Structural compression replaces selected layers with smaller representations. Recovery training helps the compressed model retain performance on the tasks you care about.
  • GPU execution. Kernels suited to the compressed representation can reduce execution overhead. A smaller layer’s speed depends on how efficiently those operations run on your hardware.
  • Serving configuration. Batching, scheduling, caching and memory allocation affect how much traffic a GPU can handle. The useful changes depend on your request patterns and response-time requirements.

These approaches can work together. SVD-LLM studies structural compression combined with quantization. NVIDIA Model Optimizer provides tools for combining optimization techniques and exporting models for serving.

At Less Compute, we compress model layers, retrain the compressed model and optimize how it runs on your GPUs. We test those changes together on your workload.

Compare against the model you serve today

A dense BF16 comparison shows the change from the original model. Comparing with your existing FP8 or INT4 deployment shows the additional benefit for your team.

Keep the workload consistent and compare the GPU capacity needed to meet your quality and response-time targets. Check the tasks, languages and output formats your application depends on. Then test the expected traffic range, including busy periods.

Read throughput and latency together. A configuration that generates more tokens per second may also make individual requests wait longer. The useful result is more work completed within your response-time requirements.

Red Hat’s LLM Compressor example compares quantization approaches across request rates and GPU counts. You can see how each configuration performs as traffic increases.

Our benchmark guide explains which settings and measurements to record so your engineers can repeat the comparison.

Choose a change worth operating

Compare the expected GPU savings with the work to deploy and maintain the change. Who will own the serving-engine integration and repeat the checks when you update or fine-tune the model? Agree that responsibility before a pilot.

Start with one model and one workload. Compare the current deployment with the proposed version, then use the result to decide whether a pilot makes sense. If the immediate problem is model size, see how to assess a larger model on your existing GPUs.

Tell us what you already run.

Share your model, quantization format, GPUs and the problem you want to solve. We’ll discuss the workload and propose a benchmark if there’s a fit. You can review the test plan before deciding on a pilot.