Load balancers: what they solve and what they do not

Load balancers: what they solve and what they do not

When an application starts getting slow, unstable, or busier than expected, one of the first infrastructure ideas people reach for is: put a load balancer in front of it.

That instinct is not wrong. A load balancer can distribute requests across multiple healthy targets, stop sending traffic to backends that fail health checks, and give an application a more stable entry point. In AWS, Elastic Load Balancing is built for high availability, automatic scaling, and routing traffic across targets and Availability Zones.

But there is a critical difference between distributing traffic and fixing the actual reason an application is struggling.

A load balancer can decide where a request should go. It cannot optimize a slow query, fix a bug, reduce memory pressure, or make an overloaded database respond faster.

In high-traffic systems, the architecture may eventually need multiple load balancers or DNS-level distribution. The first question is still simpler: is the application itself ready to be scaled horizontally?

The short answer

A load balancer solves traffic distribution, availability, and target failover. It does not solve inefficient code, fragile dependencies, slow databases, stateful sessions, missing observability, or a deployment that breaks every target at the same time.

That distinction matters because a load balancer amplifies the state of the system behind it. If the application is stateless, observable, and healthy, load balancing makes it more resilient. If the application already carries unresolved bottlenecks, the load balancer mostly spreads those bottlenecks across more places.

What a load balancer actually does

The core job of a load balancer is to become the stable entry point for incoming traffic. Instead of clients calling one instance, container, or server directly, they call a shared endpoint. The load balancer receives each request and forwards it to an available target.

That brings three practical benefits. First, it distributes load across multiple backends so one server is not responsible for every request. Second, it improves availability by removing unhealthy targets from rotation. Third, it decouples users from the underlying infrastructure, so instances can be replaced, scaled, or moved without changing the public endpoint.

This is why Elastic Load Balancing is a normal part of production AWS architectures. It is a traffic-control layer that makes horizontal scaling and multi-AZ operation easier to manage.

What a load balancer will not fix

A load balancer does not make a slow application fast by itself. If every request spends most of its time waiting on a database query, an external API, or unnecessary CPU work, routing the request to a different target does not remove that cost.

It also does not repair weak architecture. If every target depends on the same saturated database, adding more application instances can increase the pressure on that shared bottleneck. Traffic becomes better distributed, but the underlying constraint remains.

A load balancer cannot protect you from a bug deployed everywhere either. If all targets have the same defect, the load balancer has no healthy version to route around. It can route around isolated target failures; it cannot rescue a system when the failure mode is shared by every component.

Health checks are signals, not guarantees

Health checks are essential, but they are not a perfect definition of application health. A shallow health check may confirm that the process is running and the endpoint responds. That can still miss real user-facing failures, such as a broken database connection or a downstream service outage.

A deeper health check can validate critical dependencies and catch more meaningful failures. The tradeoff is that it may also mark otherwise healthy targets as unhealthy when a dependency has a temporary incident. In autoscaling environments, that can trigger unnecessary replacement behavior if the health model is too aggressive.

The best health checks are intentionally designed. They should be meaningful enough to protect users, but stable enough to avoid cascading operational noise.

Gray failures: healthy enough to pass, broken enough to hurt

Production failures are not always clean. A target may keep passing active health checks while returning more errors, responding with higher latency, failing only on certain routes, or degrading under load.

AWS describes this pattern as a gray failure: the backend is not completely down, but it is not serving users correctly either. Application Load Balancer Automatic Target Weights can reduce traffic to anomalous targets when error rates deviate from the rest of the fleet.

That mitigation is useful, but it does not remove the root cause. If the issue is a bug, CPU pressure, dependency instability, or a bad cache state, the engineering team still needs to diagnose and fix it.

Traffic distribution is not performance optimization

Load balancing improves performance only when the bottleneck is application-server capacity and the workload can scale horizontally. If one instance can process a certain amount of traffic, several healthy instances behind a load balancer can usually process more.

But if the bottleneck lives somewhere else, adding targets can disappoint. A saturated database may receive more simultaneous queries. A slow endpoint may remain slow on every target. A shared external dependency may keep every server waiting.

The useful question is not 'do we need a load balancer?' It is 'is the bottleneck traffic distribution, or is it the cost of processing each request?'

When adding a load balancer is the right move

A load balancer is the right move when your application has or needs multiple targets, your deployment model supports horizontal scaling, your sessions are not tied to one machine, and you need a stable entry point that can tolerate target replacement.

It is also the right move when you want traffic control as a first-class operational concern: target groups, health checks, routing rules, TLS termination, blue-green patterns, and multi-zone resilience.

It is premature when the application still has unresolved slow queries, local-only session state, poor dependency boundaries, or no baseline metrics for latency and errors.

Pre-flight checklist before adding a load balancer

Before you introduce load balancing, make sure the application is ready to benefit from it. The checklist below separates what the load balancer can handle from what the application team must solve first.

1. 1Make the application statelessMove sessions and local state into shared systems such as Redis, a database, object storage, or another durable service. Requests should be safe no matter which target receives them.
2. 2Optimize frequent queriesReview execution plans, indexes, ORM patterns, and N+1 queries. A slow query remains slow even when requests are distributed across more targets.
3. 3Validate database concurrencyConfirm that the database can handle connections from multiple application instances. Use connection pooling or read/write separation when the connection layer becomes a limit.
4. 4Design meaningful health checksHealth endpoints should represent real readiness without creating noisy false negatives. Validate critical dependencies carefully and document what each check means.
5. 5Measure before and afterCapture P50, P95, P99 latency, error rate, throughput, and saturation before adding the load balancer. Without a baseline, it is hard to know whether the change improved the system or hid the problem.