Home/Private LLM
Private LLM deployment & optimizationYour model, your hardware, your data boundary
Velyrix installs, quantises, benchmarks and operates open-weight frontier models on dedicated GPUs — in Velyrix cloud, in your colocation footprint, or fully air-gapped on your premises. You get an OpenAI-compatible endpoint with contracted throughput, latency and availability.
Public APIs are convenient until the data is yours
Regulated data, proprietary source code, patient records, trade positions and sovereign workloads rarely belong in someone else's inference queue. Open-weight models now close enough to frontier quality make private deployment the default for serious workloads.
- Data never leaves your boundary — no third-party retention, no training on your prompts
- Predictable unit economics — fixed GPU cost instead of per-token billing that scales with success
- Version stability — the model you validated stays the model you serve, until you choose otherwise
- Latency you control — endpoint in the same region, or the same building, as the application
- Deep customisation — LoRA adapters, domain fine-tunes, custom tool schemas and guardrails
- Auditability — full request logging into your own SIEM, with your own retention policy
# Before — public API client = OpenAI(api_key="sk-...") # After — your Velyrix private endpoint client = OpenAI( base_url="https://llm.internal.acme.com/v1", api_key=os.environ["VELYRIX_KEY"], ) resp = client.chat.completions.create( model="qwen3-235b-a22b-fp8", messages=[{"role": "user", "content": prompt}], tools=tools, stream=True, ) # Same SDK. Same schema. Your VPC, your logs, your keys.
Ten open-weight model families, deployed and supported
Velyrix maintains validated serving recipes, quantisation profiles and benchmark baselines for each family, refreshed as new checkpoints are released.
DeepSeek
Sparse MoE flagships and reasoning models, including distilled dense variants for cost-sensitive serving.
Qwen
Alibaba's dense and MoE series with strong multilingual, coding and vision-language variants across 0.6B to 235B+.
Llama
Meta's widely adopted open-weight family — the safest default for ecosystem tooling, adapters and evaluation harnesses.
OpenAI gpt-oss
OpenAI's open-weight release, MoE with configurable reasoning effort and strong tool use on modest GPU counts.
Mistral
European models with permissive licensing on many checkpoints — dense, MoE, code and small edge variants.
GLM
Zhipu's bilingual MoE family with strong agentic and coding performance, plus lighter air-class checkpoints.
Kimi
Moonshot AI's trillion-parameter-class sparse MoE models built for long context and tool-calling agents.
MiniMax
Efficient attention MoE architectures targeting very long context windows at competitive serving cost.
Gemma
Google's lightweight open models — excellent quality per GPU for on-prem, edge and high-volume classification.
NVIDIA Nemotron
NVIDIA's optimised open model family, tuned for throughput on NVIDIA accelerators and enterprise agent workflows.
Model names are trademarks of their respective owners. Velyrix is an independent infrastructure provider and is not affiliated with or endorsed by these model publishers. Each model is deployed under its own licence, which we review with you before deployment — see model licensing.
What it takes to serve each class of model
Indicative single-replica sizing for production serving with room for KV cache at realistic concurrency. Final sizing comes from your context length, concurrency and latency targets.
| Model class | Precision | Minimum GPUs | Recommended | Indicative throughput | Monthly infrastructure from |
|---|---|---|---|---|---|
| Small dense (1B – 9B) | BF16 / FP8 | 1× L40S | 2× L40S / 1× H100 | 2,500 – 6,000 tok/s | $1,640 |
| Mid dense (12B – 32B) | FP8 / AWQ-INT4 | 1× H100 80GB | 2× H100 / 2× H200 | 1,800 – 4,500 tok/s | $3,300 |
| Large dense (70B – 123B) | FP8 | 2× H200 | 4× H200 / 8× H100 | 1,400 – 3,200 tok/s | $10,400 |
| Very large dense (405B) | FP8 | 8× H200 | 8× B200 | 900 – 2,100 tok/s | $23,900 |
| Sparse MoE (100B – 250B total) | FP8 | 4× H200 | 8× H200 | 3,000 – 9,000 tok/s | $18,400 |
| Sparse MoE (400B – 700B total) | FP8 / FP4 | 8× H200 | 8× B200 / 16× H200 | 4,000 – 14,000 tok/s | $23,900 |
| Sparse MoE (1T-class total) | FP8 / FP4 | 16× H200 | 8× B300 / GB300 partition | 6,000 – 20,000 tok/s | $44,800 |
| Vision-language (any size) | BF16 / FP8 | +1 GPU for vision tower | Dedicated encoder replica | Workload dependent | Quoted |
Throughput is aggregate output tokens per second across concurrent requests, measured on Velyrix reference hardware with production-representative prompts. Your numbers are benchmarked on your workload before contract, and the agreed figure becomes a service-level target.
The difference between running a model and running it well
A default container on eight GPUs typically leaves two to five times the throughput on the table. Velyrix optimisation engagements tune the whole path from tokenizer to NIC.
Quantisation
FP8, NVFP4/MXFP4, AWQ, GPTQ and SmoothQuant profiles with accuracy regression testing against your own evaluation set before anything goes live.
Parallelism strategy
Tensor, pipeline, expert and data parallel layouts chosen per model and GPU topology, including NVLink-aware expert placement for MoE.
Continuous batching
Paged attention, chunked prefill and scheduler tuning to hold high GPU occupancy without breaching your time-to-first-token target.
Prefill / decode disaggregation
Separate prefill and decode pools with KV transfer, so long prompts stop blocking interactive decoding on shared hardware.
Speculative decoding
Draft models, EAGLE-style and n-gram speculation tuned for your acceptance rate — commonly 1.5–2.5× on interactive workloads.
KV cache engineering
Prefix and radix caching, cache offload to host memory or NVMe, and quantised KV to extend context without extra GPUs.
Autoscaling & routing
Multi-replica routing with cache-aware load balancing, queue-depth autoscaling and priority classes for interactive versus batch traffic.
Evaluation harness
Your own golden set wired into CI so every model, quantisation or engine upgrade is gated on quality, not just on speed.
Engine selection
vLLM, SGLang, TensorRT-LLM or NVIDIA Dynamo chosen per workload and benchmarked head-to-head on your traffic before we commit.
Three ways to draw your data boundary
On Velyrix GPU Cloud
Single-tenant GPUs in a Velyrix region, private networking to your VPC or on-ramp, Velyrix-operated stack.
- Live in days
- Elastic replica scaling
- Velyrix 24×7 operations
- Regional data residency
In your colocation footprint
Your hardware, your cage, deployed and operated by Velyrix under a managed service agreement.
- You own the assets
- Velyrix runs the stack
- Capex model
- Full physical control
On-premises & air-gapped
Fully disconnected deployment with offline model and container mirrors, and no outbound telemetry.
- No internet dependency
- Offline registry & updates
- Classified / regulated ready
- On-site handover training
Adaptation, retrieval and agents
- Continued pre-training on domain corpora where vocabulary and style matter more than instruction following
- Supervised fine-tuning & LoRA with multi-adapter serving, so one base model can host dozens of task adapters
- Preference optimisation — DPO, GRPO and reward-model workflows against your own rated data
- Distillation from a large teacher to a small student, cutting serving cost by an order of magnitude
- Retrieval augmentation with vector and hybrid search, chunking strategy, reranking and citation enforcement
- Agent infrastructure — tool schemas, structured output, sandboxed execution and MCP-compatible tool servers
- Guardrails — input and output classification, PII redaction, jailbreak detection and full prompt audit logging
Model licensing & governance
Open weights are not the same as unrestricted use. Before deployment, Velyrix reviews the licence of every model you intend to run and documents the obligations that come with it.
- Licence classification — permissive, community, or bespoke terms
- Acceptable-use and field-of-use restrictions
- Attribution, naming and redistribution obligations
- Commercial-use thresholds and derivative-work conditions
- Provenance record for each checkpoint and its source
Velyrix provides infrastructure and engineering services and does not grant rights in any third-party model. The customer remains the licensee of the models it chooses to deploy and is responsible for compliance with those licences and with applicable AI regulation, including the EU AI Act where in scope. Velyrix supports that process with documentation and technical controls.
How a private LLM programme runs
Pilot
2–3 weeks
- Model selection workshop
- Single-replica deployment
- Benchmark vs your evaluation set
- Cost and sizing model
- Go / no-go report
from $27,000
Production deployment
6–10 weeks
- Multi-replica HA architecture
- Quantisation & engine tuning
- Gateway, auth, quotas, logging
- Guardrails & evaluation CI
- Runbooks and team training
from $83,500
Managed LLM service
Ongoing
- 24×7 endpoint operations
- Throughput & latency SLA
- Model and engine upgrades
- Capacity and cost reporting
- Quarterly optimisation review
from $10,500 / month
Indicative professional-services fees, exclusive of GPU infrastructure and tax. Scope is confirmed after a discovery session.
Own the GPUs. Let us run them.
Buy your NVIDIA servers from any OEM or distributor you like, ship them to a Velyrix hall, and we handle the rest — deployment, fabric, cooling, monitoring and support. Or rent ours. Either way, you get a plan in one business day.
Frequently asked questions
Which LLMs can Velyrix deploy for us?
Velyrix maintains validated deployment recipes for ten open-weight families: DeepSeek, Qwen, Llama, OpenAI gpt-oss, Mistral, GLM, Kimi, MiniMax, Gemma and NVIDIA Nemotron, including their vision-language, coding and reasoning variants. Other open-weight models can be onboarded on request.
How many GPUs do we need to run a private LLM?
A 7B-9B model serves comfortably on a single L40S or H100. A 70B dense model typically needs two to four H200s in FP8. Large sparse MoE models in the 400B-700B range need roughly eight H200s or eight B200s, and 1T-class models are usually served on B300 or GB300 partitions. Final sizing depends on context length, concurrency and latency targets.
Will quantisation hurt model quality?
FP8 is generally near-lossless on modern checkpoints, and NVFP4/MXFP4 and 4-bit weight quantisation trade a small amount of quality for large throughput and memory gains. Velyrix always measures the delta against your own evaluation set and reports it before any quantised build goes to production.
Can the deployment be completely air-gapped?
Yes. Air-gapped deployments ship with offline model weights, an internal container registry, offline package mirrors and no outbound telemetry. Updates are delivered as signed, verified media through your change-control process.
Do you support fine-tuning as well as inference?
Yes - continued pre-training, supervised fine-tuning, LoRA adapters with multi-adapter serving, preference optimisation such as DPO and GRPO, and distillation from a large teacher model into a smaller student for cheaper serving.
Who is responsible for the model licence?
The customer is the licensee of any third-party model it chooses to deploy. Velyrix provides infrastructure and engineering services, reviews each model's licence terms with you before deployment, and documents the resulting obligations, but does not grant rights in third-party models.
What performance do you commit to?
After a benchmarking phase on your own traffic, Velyrix contracts specific figures for aggregate throughput in tokens per second, p95 time to first token, p95 inter-token latency and monthly endpoint availability. Those figures are re-verified after every upgrade.