Poorly Defined SLIs and SLOs Generate False Alerts

Poorly defined SLIs and SLOs generate false alerts

The hidden cost of monitoring without context

In many teams, the problem is not a lack of metrics, dashboards, or observability tools. The problem is deeper:we don't really know what it means for the service to be "fine".

And when that happens, the inevitable occurs:
👉 alerts that don't matter
👉 constant noise
👉 overwhelmed teams
👉 real incidents that go unnoticed

The root cause is almost always the same:Poorly defined SLIs and SLOs.

The fundamental mistake: measuring what's easy, not what matters

ASLI (Service Level Indicator)is a metric that measures the behavior of the service.
ASLO (Service Level Objective)is the target for that indicator over a period of time.

Example:

• SLI: percentage of successful requests
• SLO: 99.9% success over 30 days
  

So far so good. The problem starts when you choose the wrong SLI.

Many teams monitor things like:

• High CPU
• memory usage
• number of pods
• invisible internal errors

That is not reliability from the user's point of view.
That is internal telemetry.

A good SLI should answer this question:

If it doesn't answer that, you're measuring the wrong thing.

Why false alerts appear

False alerts are not a threshold problem.
They are a problem of meaning.

They happen when:

• the metric does not represent real impact
• the SLO does not reflect user experience
• the system alerts on symptoms, not consequences

Classic example:

• CPU goes up → alert
• but the system keeps responding perfectly → nobody should be woken up

This causes the worst thing that can happen in operations:

👉 alert fatigue

The real problem: disconnect between system and user

A well-defined SLI isobservable by the user.
Not by the system.

Correct example:

“% of checkouts completed in less than 2 seconds”

Incorrect example:

“payment service CPU usage”

Because the user doesn't care about your CPU.
They care if they were able to pay.

SLIs should be based on:

• availability (does it work?)
• latency (is it fast?)
• quality (is it correct?)
• completeness (did the flow complete?)

This aligns with SRE practices: measuring reliability from the user's experience, not from the infrastructure.

The concept that changes everything: error budget

This is where things get serious.

Theerror budgetis the allowed margin of error:

Error Budget = 1 - SLO

Example:

• SLO: 99.9%
• Error budget: 0.1%

That 0.1% is your acceptable margin of error.

This completely changes the approach:

• you don't look for 100% uptime
• you look for a balance between reliability and speed

And more importantly:

👉 they define when a problemreally matters

Why alerting on metrics is a bad idea

Muchas alertas tradicionales funcionan así:

• “latencia > X”
• “errores > Y”
• “CPU > Z”

El problema:
no consideran contexto ni impacto acumulado.

Resultado:

• alertas por picos pequeños
• alerts on irrelevant events
• alerts that do not require action

The solution: alert on error budget consumption

This is where one of the most powerful concepts of SRE comes in:

Burn Rate

Theburn ratemeasures how fast you are consuming the error budget.

It is the rate at which the service “spends” its error budget

Example:

• your SLO allows 0.1% error in 30 days
• if in 1 hour you have 2% errors → you are burning the budget very fast

That is indeed an alert.

Why the burn rate eliminates false alerts

Because it introduces context:

• not just “there are errors”
• but “these errors put the service at risk”

Advantages:

• less noise
• more precision
• actionable alerts

Google even recommends using:

👉 multiple windows + multiple burn rates

to detect both:

• severe (fast) issues
• slow problems (but dangerous)

Clear signs your SLOs are wrong

If you see this, you have a problem:

• constant alerts but few real incidents
• no one trusts the alerting system
• on-call ignores pages
• users complain but “everything is green”
• you have too many SLIs without priority

That means you're not measuring the right thing.

Key difference: monitoring vs alerting

Not everything that is measured should generate an alert. That is one of the most common misconceptions in observability systems.

Monitoring exists to understand the system, to provide visibility and to be able to investigate when something fails. Metrics like CPU, memory usage, logs, retries, or queues fall into this category. They are internal signals that help diagnose issues, but they don't necessarily indicate a real problem for the user.

Alerting, on the other hand, has a completely different purpose: to drive action. An alert should fire only when there is an impact —or a clear risk— to the user experience.

In practice, the difference can be understood as follows:

Monitoring (to investigate):

• CPU, memory, resource usage
• logs and traces
• retries, timeouts, internal behavior
• queues, throughput, saturation

Alerting (to take action):

• actual availability drop
• degradation in critical flows
• sustained latency affecting users
• accelerated error budget consumption

When this separation doesn't exist, everything gets mixed together. And when everything triggers an alert, nothing is truly important. The result is noise, burnout, and a loss of trust in the system.

How to define good SLIs and SLOs (seriously)

Defining SLIs and SLOs correctly isn't just a technical exercise; it's about understanding the product from the user's perspective.

The first step is identifying the most important journeys: login, checkout, search... actions that truly represent value.

Next, you must clearly define what success means in each of those flows. It's not enough for something to simply work; it must work correctly and within an acceptable timeframe.

For example:

• successful login
• payment completed
• response in less than a certain time

With that clear, the SLI is built as a ratio of successful events to total events. Something like:

• percentage of successful logins in under 500 ms

Then the SLO is defined, which is the target for that indicator over a period:

• 99.9% in 30 days

This automatically introduces the error budget (0.1%), which is the allowed margin of failure.

And here is the key point: alerts should not be based on raw metrics, but onhow fast that margin is being consumed. That is, alerting with burn rate, not isolated thresholds.

The true cost of doing it wrong

When SLIs and SLOs are poorly defined, the problem goes far beyond the noise.

Real consequences begin to appear:

• on-call team burnout
• decisions based on incorrect signals
• loss of trust in monitoring
• longer and harder-to-detect incidents
• menos tiempo dedicado a mejorar el producto

But there's something even more dangerous: the system might appear "fine" in internal metrics, while users are already having a poor experience.

Conclusion: The problem isn't having many or few alerts.

The problem is having alerts that mean nothing.
A good reliability system doesn't measure infrastructure, it measures experience, and good alerting doesn't detect just any anomaly, it detects real risk.

Because ultimately, the difference between noise and signal isn't in the tool, but in how you define what it means for your service to work.

Autoscaling, observabilidad o tooling no arreglan esto.
Everything starts with clearly defining what it means for your service to work.

From noisy dashboards to signal you can act on

Alert fatigue does not start with your alerting rules. It starts upstream, at the moment you choose which metrics qualify as SLIs. When you anchor Service Level Indicators to infrastructure telemetry—CPU utilization, memory pressure, pod restart counts—you build a monitoring surface that reacts to internal state changes rather than customer experience. The result is predictable: pages fire during routine scaling events, deployments trigger threshold breaches with zero user impact, and on-call engineers learn to ignore the very system designed to protect reliability.

The fix is not more tooling. It is sharper definitions. Rewrite each SLI as a user-facing proportion—successful requests divided by total requests, or the percentage of responses landing below a latency target at a given percentile. Then attach an SLO with a concrete threshold and a rolling window. Anything that does not map to a measurable customer outcome should remain a dashboard metric, not an alerting trigger.

1. 1Audit your current SLIs against user journeysList every active page-level alert. For each one, trace the underlying SLI back to a specific user workflow—checkout, search, authentication, API ingestion. If the link is indirect or nonexistent, demote the alert to a dashboard or remove it.
2. 2Rewrite SLIs as proportions tied to outcomesReplace absolute metrics (latency in ms, error count) with ratios (percentage of requests completing under 200 ms, ratio of non-5xx responses to total requests). Proportions compose cleanly into error-budget math and remain stable across traffic fluctuations.
3. 3Set multi-window, burn-rate alerting policiesConfigure alerts that trigger only when the error-budget burn rate exceeds a threshold—14.4× over one hour or 6× over six hours for page-level incidents. This suppresses transient spikes while preserving sensitivity to sustained degradation.
4. 4Validate with historical incident dataReplay the last 90 days of production incidents through your proposed SLIs and SLOs. Confirm that genuine degradations would have breached the SLO, and that benign events would not have paged. Adjust thresholds until false-positive coverage falls below 5 %.