Inference guide
Can you run a larger LLM on the GPUs you have?
You’ve found a model that gives better answers, but it needs more GPU memory than you have. Compression can help make it practical to serve. Start with the tasks it improves, then work out what it would take to run it on your hardware.
Choose the model for the work you want to do
Identify where a better model would help: more reliable document extraction, stronger reasoning, more accurate code, or support for languages your current model handles poorly.
Compare the candidate with your current model on those tasks. Keep examples where the candidate handles work your current model gets wrong, along with the everyday requests it must continue to answer well. Use these to evaluate the compressed version.
Work out how much memory the deployment needs
GPU memory holds the model weights, the key-value (KV) cache for active requests, and the runtime’s working memory. Quantization and structural compression can reduce the space used by weights. The room left for requests affects the context lengths and traffic you can serve.
For an initial estimate, multiply the stored parameter count by the bytes per weight. BF16 uses two bytes per value; FP8 uses one. Low-bit formats also have scales and other data to account for. Use the actual checkpoint size when it’s available.
Next, include the workload: prompt length, generated answer length and simultaneous requests. The serving engine also needs space for its own operations. NVIDIA’s inference memory reference explains these components.
Once a configuration fits, test it on your GPUs and record peak memory, throughput and response times as you increase the load.
Compare the ways to make it fit
- Try a smaller model that already fits. Include it in the task comparison. If it delivers the answers you need, it may be simpler to deploy and maintain.
- Use a supported lower-precision checkpoint. Quantization is often a straightforward first experiment. Check the model’s quality and the serving engine’s support for its format.
- Compress and retrain the candidate. Structural compression can reduce selected layers further. Recovery training and evaluation help retain the capabilities that made you choose the model.
- Change the serving setup. Distributing a model across GPUs or offloading some data to host memory expands the deployment options. Measure the resulting response times and operating cost.
Hugging Face’s quantization overview helps compare formats and hardware support. If you already use quantization, our guide to further serving-cost improvements covers the next questions.
At Less Compute, we compress and retrain the model, then optimize how it runs on your GPUs. We test whether it can deliver the quality and serving capacity your application needs.
Test the model at the traffic you need
Run the compressed candidate on the examples that justified choosing it. Then measure response times and capacity with representative requests, including your longer prompts and busiest periods.
Compare task quality with your current model and a smaller candidate that already fits. Use the uncompressed larger model as the quality reference for its compressed version; you can run that evaluation on temporary larger hardware if needed. Measure the compressed model’s serving capacity and GPU-hours on the hardware you intend to use.
The decision should answer three questions: does it improve the application, does it fit the hardware, and can it serve the traffic within your budget? Our benchmark guide covers how to make that comparison repeatable.
Have a model you’d like to run?
Tell us which model you want to use, the GPUs you have and the work it needs to do. We’ll discuss whether compression could help and propose a benchmark. You can review the test plan before deciding on a pilot.