Inference guide
How to reduce LLM inference costs
At Less Compute, we compress and retrain your open-weight model, then optimize how it runs on your GPUs to lower serving costs. We check the result against your quality and response-time requirements.
Find what drives your cost
Start with the cost you want to reduce: the GPUs serving today’s traffic or the extra capacity you’ll need as demand grows. Set the quality and response-time requirements the optimized deployment must meet.
GPU memory holds model weights, the key-value (KV) cache used for active requests, and runtime tensors. Longer contexts and more simultaneous requests can increase the KV cache. Reducing weight memory creates room, while the amount of extra traffic you can serve depends on the full workload and serving system.
Technical reference: NVIDIA’s breakdown of inference memory.
Measure GPU cost per completed task that meets your quality and latency requirements. For a fixed workload, you can also track cost per million output tokens. Use the same mix of input and output lengths in each run.
If you own the hardware, measure how much more work your fleet can serve and whether that delays the next capacity purchase.
Combine model compression with GPU optimization
Smaller weights reduce memory use. Recovery training addresses changes in model quality. GPU optimization improves how the compressed model executes.
| Change | What changes | Check in your deployment |
|---|---|---|
| Quantization | Stores values at lower precision, reducing the bytes needed for weights. | Quality, supported precision and the speed of the available GPU kernels. |
| Structural compression | Replaces selected layers with a smaller representation, such as a factorized matrix. | Quality and actual execution time with kernels that support the representation. |
| Recovery training | Retrains the compressed model to recover performance on relevant tasks. | The tasks, languages and output formats your application relies on. |
| Runtime optimization | Changes how the model runs: kernels, batching, scheduling and memory management. | Throughput and latency across your expected traffic range. |
Quantization reduces the precision used to store weights. Structural compression changes how some of those weights are represented. These techniques can be combined, with recovery training used to address quality changes. Research such as SVD-LLM studies low-rank compression and its combination with quantization.
A compressed layer can run slower if the runtime has to split it into several operations or convert its data. GPU kernels built for the compressed representation address that execution overhead. Measure the full serving path to see how the change affects throughput and latency.
Technical references: Hugging Face’s quantization overview and compatibility table; TensorRT-LLM runtime tuning options.
Already using a quantized model?
Structural compression and GPU optimization can be combined with quantization. Use your current deployment as the baseline, then compare GPU cost, quality and response times on the same requests.
Measure the improvement on your workload
Compare GPU cost for the same workload before and after optimization. Agree the quality and response-time requirements before the work starts.
- Keep a reproducible baseline.Record the model checkpoint and fine-tune, precision, GPU configuration, serving software and settings. Save the commands and results so another engineer can repeat the run.
- Use representative requests.Include the prompt lengths, output lengths, languages and tasks you actually serve. Test the arrival rates and bursts your system needs to handle.
- Agree quality checks.Choose examples and scores that reflect acceptable answers. Include structured output, specialist knowledge and difficult cases where they matter. Keep the evaluation examples separate from recovery-training data.
- Measure the serving experience.Track time to first token, time between output tokens and end-to-end latency. Report tail latency, such as the 95th percentile, alongside throughput and errors.
- Compare costs at the required service level.Measure throughput as output tokens per second per GPU while checking the agreed latency limits. Also report completed requests that meet those limits. Repeat the runs and show variability.
Report the workload, hardware, baseline, quality scores, latency limits and serving throughput together. Your engineering team should be able to repeat the comparison and connect the results to your GPU bill.
Tool reference: vLLM’s serving benchmark supports percentile latency metrics and goodput targets for time to first token, time per output token and end-to-end latency.
Decide whether the improvement is worth deploying
Compare the recurring GPU savings or additional capacity with the work needed to integrate and maintain the change. Include evaluation, deployment changes and the frequency of model updates. A new fine-tune or model version needs its own checks.
For rented hardware, calculate the GPU-hours you can stop paying for at your expected traffic. For an owned fleet, calculate how much demand it can absorb before you need more hardware.
A practical decision has three parts: the model passes your quality checks, the deployment meets its response-time requirements, and the savings or extra capacity justify the change.
Let’s lower your serving costs.
We compress models like Qwen 27B and Gemma 4 31B to run on half the GPUs while retaining 98%+ of their benchmark scores.
Tell us which model you run. We’ll start with a 30-minute call, then work with your team for about two weeks to optimize and test it on your tasks and languages. If you have a fine-tune, we start from that.
You pay only if we hit the agreed targets. You keep the compressed model and GPU kernels.