Self-Hosting AI Models on GPU Infrastructure
A technical guide for developers moving from per-token APIs to self-hosted LLMs (Llama, DeepSeek, Mistral). Calculate costs, estimate VRAM, and compare GPU cloud providers.
1. The Open-Source LLM Landscape
Context: Before provisioning hardware, you must select a model. This section outlines the major families of open-weights models available for self-hosting in 2026. Review their sizes (parameters) and primary strengths to determine which fits your use case.
🦙 Meta Llama 3.x / 4
Verified 03/2026- Sizes: 8B, 70B, 405B
- License: Llama Community License (Commercial use allowed, limits >700M MAU)
- Quality: 405B rivals GPT-4 class; 70B is the standard for high-end self-hosting.
🐋 DeepSeek
Verified 03/2026- Sizes: V3 (671B MoE), R1 (Reasoning, various sizes 7B-671B)
- License: MIT License (Highly permissive commercial use)
- Quality: Unmatched price-to-performance; coding and math capabilities exceed many proprietary models.
🌪️ Mistral
Verified 03/2026- Sizes: 7B (v0.3), Mixtral 8x7B/8x22B, NeMo 12B
- License: Apache 2.0 (Small models), Custom (Large models)
- Quality: Excellent efficiency. Mixtral 8x22B is a highly capable MoE alternative to Llama 70B.
🐉 Qwen 2.5
Verified 03/2026- Sizes: 0.5B, 1.5B, 7B, 14B, 32B, 72B
- License: Apache 2.0 (Most models)
- Quality: Exceptional multi-lingual support, strong coding, very competitive with Llama across all parameter brackets.
2. Interactive VRAM & Hardware Estimator
Context: The most critical bottleneck in self-hosting is GPU VRAM (Video RAM). Models must fit entirely in VRAM for fast inference. Use this calculator to estimate how much VRAM you need based on the model size and quantization level (compressing the model to save space), and see which GPUs match your needs.
Configure Model Parameters
FP16 offers best quality. 4-Bit uses the least VRAM but slightly degrades reasoning.
Estimated VRAM Required
Includes 20% overhead for context window & KV cache.
Recommended GPU Setup:
2x NVIDIA H100 (80GB) or 4x A100 (40GB)
3. API vs. Self-Hosting Cost Calculator
Context: Does self-hosting actually save you money? It depends on your volume. Use this interactive chart to find your "break-even point". Adjust your daily token usage and infrastructure costs to see when fixed server pricing becomes cheaper than paying per-token API fees (like OpenAI or Anthropic).
Avg mixed input/output for GPT-4o-mini class is ~$0.30. Llama 70B API is ~$0.50. Claude 3.5 Sonnet is ~$15.00.
Example: 1x RTX 4090 on RunPod is ~$0.45/hr. 1x A100 80GB is ~$1.20/hr.
Cumulative 30-Day Cost Projection
4. GPU Cloud Provider Comparison
Context: Once you know your VRAM needs and budget, you must choose where to rent the GPUs. This table compares leading providers for AI workloads, categorized by enterprise reliability vs. budget-friendly unmanaged instances.
| Provider | Type | Flagship GPU | Est. Hourly Rate | Key Advantage |
|---|
5. Inference Software Stack
Context: You have the server and the model. Now you need software to load the model into the GPU and serve an API. The ecosystem has coalesced around a few dominant inference engines. Select a tab to view details.
⚙️ vLLM (Virtual Large Language Model)
The industry standard for high-throughput production serving. Utilizes PagedAttention to manage KV cache efficiently, dramatically increasing the number of concurrent users you can support.
- Best for: Production environments, high concurrency, maximum tokens/second.
- API Compatibility: Drop-in replacement for OpenAI API.
- Setup Difficulty: Medium (Docker recommended).