# Inference optimisation for self-hosted LLMs

> Serve more users on the same GPUs through quantisation, batching and speculative decoding, without measurable quality loss on your evals.

Source: https://aibyos.com/research/inference

Research · Inference

# Inference optimisation for self-hosted LLMs

Serve more users on the same GPUs through quantisation, batching and speculative decoding, without measurable quality loss on your evals.

**2–5×**more throughput per GPU

**50–75%**lower serving cost

**same**quality on your eval set

## Pipeline

1.  Profile latency, throughput and memory
2.  FP8 / AWQ quantisation with quality checks
3.  Continuous batching and prefix caching
4.  Speculative decoding with a small draft model
5.  Autoscaling on real traffic patterns

Figures show the typical order of magnitude for this approach compared with calling a large general-purpose model. Actual results depend on the task and data; we measure them on your data during the baseline phase.

[Discuss a project like this](https://aibyos.com/contact)

## Typical tooling

-   vLLM
-   TensorRT-LLM
-   SGLang
-   Prometheus / Grafana

## Related

[All research](https://aibyos.com/research)[Sovereign GPU compute](https://aibyos.com/gpu)[Fine-tuning](https://aibyos.com/services/fine-tuning)