The GPU is the new datacenter bottleneck. As enterprises move LLMs from proof-of-concept to production, the infrastructure challenge shifts from "can we get it working?" to "can we afford to keep it running?"
A single H100 GPU costs $30,000+. A production inference cluster for a 70B parameter model requires 2-4 of them. At cloud prices, that is $15-30/hour before a single query is served.
This article covers the engineering decisions that determine whether your GPU infrastructure burns money or operates efficiently.
Hardware Selection: Matching GPU to Workload
Not every workload needs the latest GPU. Choosing correctly can reduce costs by 3-5x.
| GPU | VRAM | FP16 TFLOPS | Cost (Cloud/hr) | Best For | |---|---|---|---|---| | NVIDIA H100 SXM | 80GB | 989 | $8-12 | Frontier model inference (70B+) | | NVIDIA A100 80GB | 80GB | 312 | $3-5 | Mid-size models (13B-70B) | | NVIDIA A10G | 24GB | 125 | $1-1.50 | Small models (7B-13B) | | NVIDIA L4 | 24GB | 121 | $0.70-1 | Quantized models, high-volume | | NVIDIA T4 | 16GB | 65 | $0.35-0.50 | Embedding models, lightweight tasks |
The Key Principle: Right-Size Your GPU
- Embedding models (sentence-transformers, nomic-embed) run perfectly on T4s. Don't waste A100s on embeddings.
- 7B quantized models fit on L4/A10G. These handle classification, extraction, and simple generation at extremely low cost.
- 70B models require A100 80GB (FP16) or H100 (for throughput). But quantization can bring them down to A10G territory.
- 405B models need multi-GPU setups with NVLink interconnects. H100 SXM clusters are the practical minimum.
Quantization: Trading Precision for Efficiency
Quantization reduces model weights from 16-bit to 8-bit, 4-bit, or even 2-bit representations.
| Quantization | VRAM Savings | Quality Impact | Speed Impact | |---|---|---|---| | FP16 (baseline) | — | — | — | | INT8 (GPTQ/AWQ) | ~50% | < 1% degradation | 1.5-2x faster | | INT4 (GPTQ/AWQ) | ~75% | 1-3% degradation | 2-3x faster | | GGUF Q4_K_M | ~75% | 1-3% degradation | CPU-runnable | | INT2 (QuIP#) | ~87% | 5-10% degradation | Experimental |
Practical recommendation: INT4 AWQ or GPTQ is the sweet spot for most enterprise workloads. The quality degradation is negligible for tasks like classification, extraction, and summarization. Reserve FP16 for tasks requiring maximum reasoning fidelity.
Example: 70B Model GPU Requirements
Llama 3.1 70B:
FP16: ~140GB VRAM → 2× A100 80GB or 2× H100
INT8: ~70GB VRAM → 1× A100 80GB or 1× H100
INT4: ~35GB VRAM → 1× A100 40GB or 2× A10G (tensor parallel)
Quantizing from FP16 to INT4 turns a 2-GPU setup into a 1-GPU setup — halving your infrastructure cost.
Inference Engines: vLLM, TensorRT-LLM, and SGLang
The inference engine determines how efficiently the GPU is utilized.
vLLM
The most popular open-source inference engine. Key innovation: PagedAttention — manages KV-cache memory like an operating system manages virtual memory, eliminating waste from variable-length sequences.
Strengths: Easy to deploy, broad model support, continuous batching, speculative decoding.
TensorRT-LLM (NVIDIA)
NVIDIA's optimized inference engine. Compiles models into GPU-optimized kernels.
Strengths: Maximum throughput on NVIDIA hardware, FP8 support on H100, in-flight batching. Trades deployment simplicity for raw performance (15-30% faster than vLLM on equivalent hardware).
SGLang
Emerging framework with innovative features like RadixAttention for efficient prefix caching. Particularly fast for agentic workloads with repeated tool-use patterns.
Batching Strategies: The Single Biggest Optimization
A GPU serving one request at a time is massively underutilized. Batching multiple requests together is the single most impactful optimization.
Continuous Batching (In-Flight Batching)
Instead of waiting for a batch to complete before starting the next, new requests are inserted into the running batch as slots free up.
Time →
GPU: [Req1 Req2 Req3] [Req1 Req4 Req3] [Req5 Req4 Req3] ...
Req2 done, Req1 done,
Req4 starts Req5 starts
Impact: 2-5x throughput improvement over static batching.
Prefix Caching
When multiple requests share a common prefix (system prompt, RAG context), cache the KV values for the shared prefix and reuse them.
Impact: 30-60% latency reduction for requests with shared contexts (common in enterprise RAG applications where the system prompt + retrieved context is similar across requests).
Autoscaling: Matching Capacity to Demand
GPU clusters must scale with demand — enterprise workloads have strong daily and weekly patterns.
Metrics to Scale On
- Queue depth — Number of requests waiting for GPU time.
- Time-to-first-token (TTFT) — If TTFT exceeds SLA, scale up.
- GPU utilization — Below 40% sustained, scale down.
- Tokens per second per GPU — Throughput declining indicates saturation.
Scaling Strategy
Min replicas: 2 (availability)
Scale-up trigger: Queue depth > 10 for 30 seconds
Scale-down trigger: GPU utilization < 30% for 5 minutes
Max replicas: Based on budget cap
Cool-down period: 3 minutes between scaling events
Critical consideration: GPU instances take 2-5 minutes to start and load models. Use predictive scaling (based on historical patterns) in addition to reactive scaling to avoid latency spikes during ramp-up.
Multi-Tenancy: Shared Infrastructure, Isolated Workloads
Serving multiple models or tenants on the same GPU cluster reduces costs through better utilization.
Model Multiplexing
Deploy multiple small models on a single GPU. A T4 with 16GB can simultaneously host:
- An embedding model (2GB)
- A classification model (4GB)
- A 7B quantized generation model (4GB)
- 6GB headroom for KV-cache
Request Routing
Route requests to the least-loaded GPU instance with the required model already loaded. Avoid model swapping (loading a new model onto a GPU) — it takes 30-120 seconds.
Our Infrastructure at ATMA-AI
At ATMA-AI, we build and operate GPU inference clusters for enterprises that need the performance of self-hosted models without the infrastructure overhead. Our platform handles quantization selection, inference engine optimization, autoscaling, and multi-tenancy — delivering production-grade LLM inference at a fraction of raw cloud GPU costs.
Need cost-efficient GPU infrastructure? Talk to our infrastructure team.