Setting up Grafana Cloud alerting that people do not ignore

An alerting setup fails in one of two ways. Either it is too quiet and customers find the outage first, or it is so noisy that the team mutes the channel. Both end the same way: nobody trusts the alerts.

This is the structure I use for Grafana Cloud alerting. It comes from running 30+ alert rules in production for an AI platform, routed to Slack and email.

Start from symptoms, not causes

Alert on what users feel, then use dashboards to find the cause.

Good: p95 latency on the checkout API is above 800ms for 10 minutes.
Good: error rate on the public API is above 2% for 5 minutes.
Noisy: CPU on one node is above 80%.

High CPU is not a problem by itself. A slow checkout is. Keep cause level signals on dashboards and only page on them when they reliably predict user impact.

One folder and label scheme

Grafana managed alert rules live in folders and evaluation groups. Pick a convention and stick to it:

Folder per service or domain, for example checkout, platform, data.
Evaluation group per evaluation interval, for example 1m for critical, 5m for everything else.
Labels on every rule: severity (critical, warning), team, service, env.

Those labels are what notification policies route on, so they matter more than the rule names.

Write rules that carry context

Every rule should answer three questions in the notification itself: what is wrong, how bad is it, and where do I look first.

Use annotations for that:

summary: one line, for example "Checkout p95 latency 1.2s (threshold 800ms)"
description: what it means for users
runbook_url: a link to the runbook
dashboard_url: a link to the dashboard already filtered to the service

A PromQL example for a latency rule:

histogram_quantile(
  0.95,
  sum by (le, service) (rate(http_request_duration_seconds_bucket{service="checkout-api", env="prod"}[5m]))
)

Set a pending period (the "for" duration) so a single spike does not page anyone. Five to ten minutes is a good default for latency and error rate.

Contact points and notification policies

Create a small number of contact points, for example:

slack-critical: the on call channel, with mentions
slack-warnings: a quieter channel, no mentions
email-owners: service owners, for slow burning issues

Then route with notification policies on labels:

severity=critical and env=prod goes to slack-critical
severity=warning goes to slack-warnings
everything from env=staging goes to the quiet channel only

Group notifications by alertname and service so one incident becomes one message, not twenty.

Kill the noise on purpose

Noise is the default, so remove it deliberately:

Silences for planned maintenance, created before the work starts.
Mute timings for known noisy windows such as nightly batch jobs.
Inhibition style routing: when a whole cluster is down, do not also send an alert for every service on it.
Review weekly: any alert that fired and needed no action gets fixed, downgraded or deleted.

Logs need alerting too

Metrics catch most incidents, but some failures only show up in logs, such as a specific exception from a third party API. Grafana can alert on LogQL queries directly:

sum by (service) (count_over_time({env="prod"} |= "PaymentProviderTimeout" [5m])) > 5

For noisier log signals I have also built a small Loki to Slack pipeline with deduplication, suppression windows and a manual review page, so repeated errors become one actionable message instead of a flood.

Keep the bill in check

Alert rules themselves are cheap. The metrics and logs behind them are not. If a rule needs a high cardinality label to work, consider a recording rule that aggregates first. On one platform, fixing the root causes of metric growth cut the Grafana Cloud bill by about 71% while keeping every alert that mattered. More on that in the AI platform case study.

The short version

Alert on user facing symptoms with a pending period
Label every rule with severity, team, service and env
Put a runbook and a filtered dashboard link in every notification
Route with notification policies, group by service
Silence maintenance, mute known windows, review weekly

If you want this set up for your stack, the observability service covers metrics, logs, dashboards and alerting end to end.