Self-Hosting Decision Models: Serving, Quantisation, Latency and Operations Training Cursus

Deployment Options

• Hosted API versus self-hosted trade-offs
• On-premises, edge and air-gapped patterns
• Single model versus model portfolio
• Cost per decision comparison

Hardware Sizing and Capacity

• Memory footprint by parameter count
• CPU-only versus GPU acceleration
• Concurrency and throughput planning
• Headroom for task length growth

Serving Stacks

• vLLM and SGLang options
• llama.cpp and GGUF execution
• Ollama for local development
• Choosing based on decision workload shape

Quantisation Without Losing Calibration

• Precision formats and memory savings
• Measuring quality and calibration loss
• Per-tensor and importance-aware variants
• Acceptance testing after quantisation

Latency Engineering

• Median versus tail latency
• Latency growth with context length
• Warm start and model caching
• Batching multiple decision questions

Edge and Constrained Environments

• Small decision models on CPU
• Apple Silicon and embedded deployment
• Offline operation and update strategy
• Air-gapped weight distribution

Containerisation and Scaling

• Image design for model serving
• Volume and cache management
• Horizontal scaling and routing
• Resource limits and GPU sharing

Observability

• Request, latency and error metrics
• Confidence distribution monitoring
• Decision outcome and escalation tracking
• Alerting on calibration drift

Versioning and Rollout

• Model and adapter versioning
• Shadow and canary deployment
• Pinned model files for audit
• Rollback procedures

Security and Multi-Tenancy

• Authentication and rate limiting
• Tenant isolation for prompts and logs
• Network isolation and egress control
• Audit logging of decisions