Poorly defined SLIs and SLOs: why they cause false alerts

The hidden cost of monitoring without context#

In many teams the problem is not a lack of metrics, dashboards or observability tools. It is deeper than that: nobody has defined precisely what it means for the service to be “fine”.

When that happens the outcome is predictable: alerts that don't matter, constant noise, an overwhelmed on-call rotation and, worst of all, real incidents that slip through because they look like everything else.

The root cause is almost always the same: poorly defined SLIs and SLOs. Internal telemetry gets mistaken for user experience, and the alerting system ends up reacting to irrelevant symptoms while missing the ones that actually hurt the business.

Alert fatigue does not start with your alerting rules. It starts upstream, at the moment you decide which metrics qualify as SLIs. If every 3 a.m. page ends with “nothing to do”, people learn to ignore pages, and one day they ignore the one that mattered. This guide walks from indicator definition all the way to the alert rule.

Documentation: Google SRE Book · Monitoring Distributed Systems ↗ · Google SRE Book · Service Level Objectives ↗

SLIs, SLOs and error budgets in two minutes#

A Service Level Indicator (SLI) is a quantitative measure of some aspect of the service your users receive. The most useful form is a ratio: good events divided by valid events, such as the share of checkout requests that complete without a server error.

A Service Level Objective (SLO) is the target for that indicator over a time window, for example 99.9% successful checkout requests over a rolling 30 days.

The error budget is simply 1 − SLO. With a 99.9% SLO your budget is 0.1%: over 30 days that is roughly 43 minutes of full unavailability, or the equivalent spread across partial errors. That margin is not a failure. It is room the team can spend on releases, experiments and the failures that will happen anyway.

This reframing is what makes SLOs useful. You stop chasing 100% uptime, which is expensive and unnecessary for almost any product, and start making an explicit trade-off between reliability and delivery speed. Most importantly, the budget tells you when a problem actually matters.

Documentation: Google SRE Book · Service Level Objectives ↗ · Google SRE Workbook · Implementing SLOs ↗

The core mistake: measuring what is easy, not what matters#

The trouble starts when you pick the wrong SLI. Many teams watch high CPU, memory usage, pod counts or internal errors users never see. That is not reliability from the user's point of view; it is internal telemetry. It is valuable for diagnosis, but it should not decide whether someone gets woken up.

A good SLI answers one question: was the user able to do what they came to do, correctly and fast enough? The user does not care about your CPU. They care whether they managed to pay.

User-centric SLIs usually fall into four families: availability (does it work?), latency (is it fast?), quality or correctness (is the answer right?) and completeness or freshness (did the flow finish, is the data current?). For latency, use percentiles such as p95 or p99 rather than averages; averages hide the slow tail, which is exactly where your unhappiest users live. Computing those percentiles in Prometheus requires histograms with buckets placed near your SLO threshold.

A useful sanity check: a service running at 90% CPU with fast p99 latency and no 5xx responses is healthy from the user's perspective. Paging on that CPU spike is noise without signal.

Documentation: Google SRE Book · Service Level Objectives ↗ · Prometheus · Histograms and summaries ↗

Why false alerts happen#

False alerts are not a threshold problem; they are a problem of meaning. They show up when the metric does not represent real impact, when the SLO does not reflect user experience, or when the system alerts on causes instead of consequences.

The classic case: CPU climbs, an alert fires, and the service keeps responding perfectly. Nobody should be woken up for that. Google's SRE book frames it as symptoms (what is broken for the user) versus causes (why it is broken). Page on symptoms; use dashboards to chase causes.

That is why monitoring and alerting need to be treated as different jobs. Not everything you measure should be able to page someone.

Monitoring, to understand and investigate:

• CPU, memory and resource usage.
• Logs and traces.
• Retries, timeouts and internal behavior.
• Queues, throughput and saturation.

Alerting, to trigger human action:

• A real availability drop on a critical journey.
• Sustained latency degradation that users feel.
• Accelerated error-budget consumption.

Without that separation everything blurs together, and when everything pages, nothing is important.

Documentation: Google SRE Book · Monitoring Distributed Systems ↗

The fix: alert on burn rate, not static thresholds#

Traditional rules such as “latency > X”, “errors > Y” or “CPU > Z” have no notion of context or accumulated impact. They fire on one-minute blips, on irrelevant events and on situations that need no action.

Burn rate measures how fast you are consuming the error budget relative to the SLO. A burn rate of 1 means you would spend exactly the whole budget by the end of the window. A burn rate of 14.4 sustained for one hour consumes 2% of a 30-day budget and, if it continues, exhausts it in a little over two days. That deserves a page.

The Google SRE Workbook recommends multiple windows and multiple burn rates: a long window to prove the problem is significant and a short one so the alert stops firing soon after the problem ends. Its starting values for a 30-day SLO are 14.4 over 1 hour (with a 5-minute short window) and 6 over 6 hours (with 30 minutes) as pages, plus 1 over 3 days (with 6 hours) as a ticket. That catches both sharp outages and slow, dangerous degradation.

groups:
  - name: slo-checkout-recording
    rules:
      # Availability SLI expressed as an error ratio: 5xx / total.
      - record: job:slo_errors_per_request:ratio_rate5m
        expr: |
          sum by (job) (rate(http_requests_total{job="checkout-api",code=~"5.."}[5m]))
          /
          sum by (job) (rate(http_requests_total{job="checkout-api"}[5m]))
      # Repeat the same rule for the other windows: 30m, 1h, 6h and 3d.
      - record: job:slo_errors_per_request:ratio_rate1h
        expr: |
          sum by (job) (rate(http_requests_total{job="checkout-api",code=~"5.."}[1h]))
          /
          sum by (job) (rate(http_requests_total{job="checkout-api"}[1h]))

  - name: slo-checkout-alerts
    rules:
      # 99.9% SLO: error budget = 0.001.
      - alert: CheckoutErrorBudgetBurnFast
        expr: |
          (
            job:slo_errors_per_request:ratio_rate1h{job="checkout-api"} > (14.4 * 0.001)
            and
            job:slo_errors_per_request:ratio_rate5m{job="checkout-api"} > (14.4 * 0.001)
          )
          or
          (
            job:slo_errors_per_request:ratio_rate6h{job="checkout-api"} > (6 * 0.001)
            and
            job:slo_errors_per_request:ratio_rate30m{job="checkout-api"} > (6 * 0.001)
          )
        labels:
          severity: page
        annotations:
          summary: "Checkout is burning its error budget too fast"
      - alert: CheckoutErrorBudgetBurnSlow
        expr: |
          job:slo_errors_per_request:ratio_rate3d{job="checkout-api"} > 0.001
          and
          job:slo_errors_per_request:ratio_rate6h{job="checkout-api"} > 0.001
        labels:
          severity: ticket

Two details often break copy-pasted versions: every window referenced by an alert needs its own recording rule (a missing one silently never matches), and the threshold is burn rate times the budget, not a fixed error percentage.

Documentation: Google SRE Workbook · Alerting on SLOs ↗ · Prometheus · Recording rules ↗ · Prometheus · Alerting rules ↗

How to redefine your SLIs and SLOs#

• Audit current alerts against user journeys. List every alert that can page. Trace each one back to a user workflow such as checkout, search, authentication or API ingestion. If the link is indirect or missing, demote it to a dashboard or delete it.
• Define success per journey. Responding is not enough; it must respond correctly and within an acceptable time, for example a successful login in under 500 ms.
• Rewrite SLIs as proportions tied to outcomes. Replace absolute values (latency in ms, raw error counts) with ratios: non-5xx responses over total, or the share of requests under the latency target. Ratios compose cleanly into error-budget math and stay stable as traffic changes.
• Set SLOs from historical data. Do not pick 99.99% because it sounds good. Look at how the service actually behaved over recent months and choose a target that is demanding but achievable, over a rolling 30-day window.
• Move paging to multi-window burn-rate alerts, as in the example above.
• Replay past incidents. Run your recent incidents and benign events through the proposed SLOs. Real degradations should have breached them; routine scaling events and harmless deploys should not have paged. Tune until both hold.
• Agree on an error budget policy. Decide what happens when the budget runs out: freeze non-urgent releases, prioritize reliability work, require a postmortem. Without it the SLO is just a number on a dashboard.
• Review regularly. Journeys change with the product; revisit indicators and targets with product owners, for example every quarter.

Documentation: Google SRE Workbook · Implementing SLOs ↗ · Google SRE Workbook · Error budget policy ↗

Clear signs your SLOs are wrong#

• Constant alerts but few real incidents.
• Nobody trusts the alerting system and on-call ignores pages.
• Users complain while every dashboard is green.
• Postmortems show no automated alert caught the incident; users reported it.
• Too many SLIs and none of them prioritized.

The green dashboard test is the most reliable signal: if everything looks green while support tickets pile up, your SLIs are disconnected from real experience.

It also pays to measure the alerting system itself. Track the share of pages that end with no action, how long on-call needs to decide whether an alert is real, and how many incidents monitoring detects before users do. A high no-action rate, slow triage or users beating your alerts all point to the indicators, not to the people.

Documentation: Google SRE Book · Monitoring Distributed Systems ↗

The problem is not how many alerts you have#

The problem is having alerts that mean nothing. A good reliability practice measures experience, not infrastructure, and good alerting does not flag every anomaly; it flags real risk.

Getting this wrong costs more than noise: on-call burnout, decisions based on the wrong signals, lost trust in monitoring, longer incidents and less time spent improving the product. The most dangerous part is that internal metrics can look fine while users are already having a bad time.

Autoscaling and new tooling will not fix it. It all starts with a clear definition of what it means for your service to work.