Infrastructure for AI products and LLM workloads.
Self hosted RAG, embedding and reranker services, isolated agent workloads and queues that hold up under real traffic, with usage and cost visible per client. I run this for a production AI platform today.
The prototype worked. Production is harder.
Workers run out of memory
Inference and embedding workers get OOMKilled under load and requests fail without a clear reason.
GPU bills with no owner
GPUs sit idle overnight, and nobody can say which customer or feature is driving spend.
Data leaving your cloud
Customer documents go to third party APIs when an in house pipeline would keep them private.
No visibility into LLM usage
Tokens, latency and failures per model and per client are invisible until the invoice lands.
What the engagement covers.
- 01
Model serving
Embedding and reranker models on Kubernetes with health checks, resource limits and autoscaling.
- 02
RAG pipelines
Self hosted retrieval with Qdrant, ingestion workers and SSO via Keycloak, keeping customer data inside your cloud.
- 03
Dedicated node pools
Separate pools for heavy workloads with taints, tolerations and autoscaling, including GPU pools when you need them.
- 04
Queues and workers
Background jobs for long running tasks, retries and back pressure so the API stays responsive.
- 05
Isolated agent workloads
Per session Kubernetes jobs for browser and code agents, locked down with scoped service accounts.
- 06
LLM observability
Dashboards for tokens, latency, errors and cost per model, feature and client.
What you get at the end
- Model serving and RAG services running on your cluster
- Node pools for heavy workloads with autoscaling and cost limits
- Queue based workers for long running jobs
- Dashboards for LLM usage, latency and cost per client
- Isolation and access model for agent workloads
- Runbooks for scaling, incidents and model updates
Tools I use
- Kubernetes
- Azure AKS
- AWS EKS
- Trieve
- Qdrant
- Keycloak
- KEDA
- cert-manager
- Azure Functions
- PostgreSQL
- Grafana
- Python
AI infrastructure questions.
- Should we self host models or use an API?
- It depends on data sensitivity, volume and latency. Many teams use APIs for generation and self host embeddings and retrieval. We decide on the call.
- Can you fix our OOMKills and latency?
- Yes. I have diagnosed and fixed FastAPI and Gunicorn memory issues in production. It usually starts with profiling and right sizing workers.
- How is the price decided?
- After a short call I send a written scope and a fixed quote. No hourly surprises, and nothing starts until you have agreed the scope.
- Do you build the AI features themselves?
- No. I build and run the infrastructure underneath. Your team or your ML partner owns the models and prompts.
Let us get your AI product production ready.
Tell me about your models, your traffic and where things break under load.