Context
A London based company runs a multi client AI SaaS platform: document retrieval, LLM powered workflows and browser automation agents, all on Azure. I joined as the DevOps and platform engineer responsible for production infrastructure end to end.
The platform runs across four AKS clusters, with PostgreSQL Flexible Server, Azure Functions, Key Vault and Blob Storage around it.
The problem
The platform was growing faster than its operations:
- Monitoring existed, but alerts were noisy and some real incidents were still noticed by users first.
- Logs, metrics and LLM usage lived in different places, so debugging meant jumping between tools.
- The Grafana Cloud bill was climbing every month, with overage charges on top of the plan.
- Production services hit memory pressure and OOMKills under load.
What I did
Observability, end to end. I built and maintain 40+ Grafana Cloud dashboards across Prometheus, Loki, Tempo and Application Insights, covering production logs, metrics, LLM usage, calls and billing.
Alerting people trust. I wrote 30+ Grafana alert rules routed to Slack and email, and built a Python based Loki to Slack pipeline with deduplication, suppression windows and a manual review web UI, so repeated errors become one actionable message.
Cost. I traced the root causes of metric and log growth and applied targeted fixes to cardinality, ingestion and retention.
Reliability. I led incident response on FastAPI and Gunicorn OOMKills and production latency, fixing worker sizing and memory usage at the source.
Secure workloads. I engineered an isolated Kubernetes platform for browser automation agents: namespace scoped service accounts, dedicated node pools with taints and tolerations, and automated per session Jobs and Ingresses with cert-manager TLS. The backend talks to the AKS API with scoped service account tokens instead of an admin kubeconfig.
Self hosted AI services. I run a self hosted retrieval platform on AKS with Qdrant, embedding and reranker models and Keycloak SSO, keeping customer data inside the platform's own cloud.
The result
- Grafana Cloud bill cut by about 71%, from roughly $1,090 a month in overage to roughly $316, without losing visibility.
- One place to see logs, metrics, traces and LLM usage per client.
- Alerts that reach the right channel with context, instead of a muted firehose.
Related reading: Setting up Grafana Cloud alerting that people do not ignore and AKS cost optimization.
Stack
Azure AKS, PostgreSQL Flexible Server, Azure Functions, Key Vault, Blob Storage, Grafana Cloud, Prometheus, Loki, Tempo, Application Insights, cert-manager, Qdrant, Keycloak, Python.