Cloud Hosting Load Balancing Explained Simply

What Is Cloud Hosting Load Balancing? The Traffic Cop Analogy

Imagine a busy intersection during rush hour without a traffic cop. Cars pile up in one lane while another sits empty. Horns blare. Nobody moves. Now drop a traffic cop into that chaos—she reads the flow, waves one lane forward, holds another back, and keeps everything moving smoothly. That is exactly what cloud hosting load balancing does for your web traffic.

Load balancing is the process of distributing incoming network traffic across multiple backend servers. When a visitor types your domain name into a browser, that request does not land on a single server that either handles it or collapses under pressure. Instead, a load balancer intercepts the request, evaluates which server is best positioned to handle it, and routes it accordingly. The visitor never knows the difference. They just get a fast, responsive website.

In cloud hosting environments, load balancing becomes even more powerful because the servers themselves are virtualized. You can spin up new instances in seconds, attach them to the load balancer pool, and scale horizontally with demand. No forklifts. No rack-and-stack. No waiting on a hardware vendor. This elasticity is what separates cloud load balancing from traditional on-premise solutions.

At Hosting Captain, we have seen businesses transform their uptime from shaky to rock-solid by implementing even a basic load balancing configuration. The reason is straightforward: single points of failure are eliminated. If one cloud instance crashes, the load balancer stops sending traffic to it and redistributes the load to healthy instances. Your users stay online. Your revenue stays protected. Your reputation stays intact.

Understanding load balancing starts with understanding the fundamentals of cloud computing—the on-demand delivery of compute resources over the internet. Once you grasp that your infrastructure no longer lives inside a single box in a single rack, the need for intelligent traffic distribution becomes obvious.

Types of Cloud Load Balancers

Not all load balancers are created equal. They operate at different layers of the networking stack, and each type solves a different problem. Choosing the wrong one can limit your scalability and introduce unnecessary latency. Here are the four primary types you will encounter in cloud hosting environments.

DNS Round-Robin Load Balancing

DNS round-robin is the simplest form of load balancing. You configure multiple A records for the same domain name, each pointing to a different server IP address. When a DNS resolver queries your domain, it receives all the IP addresses and cycles through them—hence "round-robin."

This approach costs nothing beyond your DNS hosting and requires zero infrastructure. However, it comes with significant limitations. DNS resolvers cache records aggressively, meaning traffic distribution is uneven at best. More critically, DNS round-robin has no awareness of server health. If one server goes down, the DNS resolver happily continues directing traffic to it until the TTL expires. For production workloads, DNS round-robin serves best as a supplement to a proper load balancer, not a replacement.

Layer 4 Load Balancing (TCP/UDP)

Layer 4 load balancers operate at the transport layer of the OSI model. They make routing decisions based on IP address and port number, without inspecting the actual content of the packets. Think of it as a postal worker sorting envelopes by address without opening the letters inside.

Because Layer 4 load balancing does not inspect packet payloads, it is extremely fast and can handle massive volumes of traffic with minimal latency. It is ideal for non-HTTP workloads—databases, email servers, gaming servers, and any TCP/UDP-based application. In cloud environments, Layer 4 load balancers excel at distributing raw connection-level traffic where speed outweighs the need for content-aware routing.

Layer 7 Load Balancing (HTTP/HTTPS)

Layer 7 load balancers operate at the application layer. They can inspect the actual content of each request—HTTP headers, cookies, URL paths, and even the request body. This intelligence allows for sophisticated routing decisions that go far beyond "which server is least busy."

A Layer 7 load balancer can route requests for /api/ to one server pool, /images/ to another, and /blog/ to yet another—all from the same domain. It can terminate SSL/TLS, offloading encryption overhead from your backend servers. It can inject security headers, perform rate limiting, and even rewrite URLs on the fly. For web applications, Layer 7 is the gold standard.

Global Server Load Balancing (GSLB)

Global server load balancing extends the concept across geographic regions. Rather than distributing traffic among servers in a single data center, GSLB routes users to the nearest or best-performing data center globally. A visitor from Tokyo might hit your Tokyo cluster while a visitor from Frankfurt hits your Frankfurt cluster—both served from the same domain name.

GSLB typically uses DNS-based routing with geo-awareness, often combined with health checks that remove unhealthy regions from the DNS response. Major cloud providers offer GSLB as part of their load balancing suites, and it is the backbone of any serious multi-region deployment. For businesses with a global audience, GSLB reduces latency dramatically and provides disaster recovery at the regional level.

If you are evaluating whether a dedicated server or cloud infrastructure better suits your growth trajectory, understanding these load balancer types will help you ask the right questions when comparing providers.

Load Balancing Algorithms: How Traffic Gets Distributed

The algorithm inside a load balancer determines how each incoming request finds its target server. This is not a one-algorithm-fits-all decision. The right algorithm depends on your workload characteristics, server homogeneity, and tolerance for uneven distribution. Below are the four most common algorithms found in cloud load balancers.

Round-Robin

Round-robin cycles through the server pool sequentially. Server 1 gets request 1, Server 2 gets request 2, Server 3 gets request 3, then back to Server 1. It is dead simple and works well when all backend servers have identical specifications and all requests impose roughly equal load.

The weakness emerges when servers differ in capacity or when some requests are computationally heavier than others. A server with half the CPU of its peers will receive the same number of requests and will eventually buckle. For homogeneous cloud VM fleets, though, round-robin is a solid starting point.

Least Connections

Least connections sends each new request to the server with the fewest active connections. This algorithm adapts dynamically to varying request durations. If one request triggers a long-running database query, that server's connection count stays elevated, and the load balancer naturally directs new requests elsewhere.

Least connections shines in environments where sessions have unpredictable durations—think WebSocket applications, streaming services, or any workload where a handful of heavy requests can starve lighter ones. Most cloud load balancer services support least connections either as the default or as an easily selectable option.

IP Hash

IP hash generates a consistent hash from the client's IP address and uses that hash to select a backend server. The same client IP always routes to the same server—at least until the server pool changes. This provides a form of session persistence without requiring cookies or sticky sessions.

IP hash works well when backend servers maintain local state that cannot easily be shared. However, it distributes load unevenly when traffic comes from a small set of IP addresses—corporate NAT gateways, for example. It also breaks when servers are added or removed from the pool, causing hash remapping.

Weighted Load Balancing

Weighted algorithms assign a numerical weight to each backend server. A server with weight 3 receives roughly three times the traffic of a server with weight 1. This lets you mix server sizes within the same pool—a practical necessity when you are transitioning between instance types or running a heterogeneous fleet for cost optimization.

Weighted round-robin and weighted least connections combine the base algorithm with server weights. Cloud providers typically expose weighting through target group or backend pool configurations. This flexibility is essential for phased rollouts, A/B testing, and gradual capacity additions.

Cloud Load Balancer Services Compared

Every major cloud provider offers a managed load balancing service. Operating your own load balancer on a VPS is entirely possible—and we will cover that in a later section—but managed services eliminate the operational burden of patching, scaling, and monitoring the load balancer itself. Here is how the major offerings stack up.

AWS Elastic Load Balancing (ALB / NLB / GWLB)

Amazon Web Services splits its load balancing into three products. The Application Load Balancer (ALB) operates at Layer 7 and integrates tightly with AWS services like ECS, EKS, Lambda, and Cognito. It supports host-based and path-based routing, redirects, fixed responses, and native WebSocket support. The Network Load Balancer (NLB) operates at Layer 4 and can handle millions of requests per second with ultra-low latency—ideal for TCP, UDP, and TLS workloads. The Gateway Load Balancer (GWLB) sits in front of virtual appliances like firewalls and intrusion detection systems.

AWS charges for each LCU (Load Balancer Capacity Unit) consumed, which factors in new connections, active connections, processed bytes, and rule evaluations. Costs are predictable but scale with traffic. The ALB also supports AWS WAF for web application firewall capabilities directly at the load balancer tier.

Google Cloud Load Balancing

Google Cloud takes a unified approach. Instead of separate products for different layers, Google Cloud Load Balancing provides a single anycast IP address that handles all your traffic globally. It supports HTTP(S), TCP/UDP, SSL proxy, and internal load balancing through one control plane.

The standout feature is that Google's load balancer is truly global from the start. There is no need to provision regional load balancers and then stitch them together. The anycast IP routes users to the nearest Google edge location, and traffic then travels over Google's private backbone to your backend instances. For globally distributed applications, this architecture reduces complexity significantly.

Azure Load Balancer & Application Gateway

Microsoft Azure splits its offering into Azure Load Balancer (Layer 4) and Application Gateway (Layer 7). Azure Load Balancer handles high-performance, low-latency Layer 4 distribution, while Application Gateway adds HTTP-specific features including SSL termination, cookie-based session affinity, URL-based routing, and a web application firewall.

Azure also offers Traffic Manager (DNS-based global load balancing) and Front Door (global Layer 7 with SSL offloading and WAF). The portfolio mirrors AWS in breadth but uses different terminology, which can trip up teams operating multi-cloud environments.

DigitalOcean Load Balancers

DigitalOcean keeps load balancing refreshingly simple. Their managed load balancer operates at Layer 7, supports SSL termination, sticky sessions, and health checks, and charges a flat hourly rate plus a per-GB data transfer fee. There are no capacity unit calculations or complex pricing formulas to model.

For startups and small-to-medium businesses running on DigitalOcean droplets, this simplicity is a genuine competitive advantage. You can configure forwarding rules, attach droplets by tag, and let the load balancer handle the rest. It is not as feature-rich as AWS ALB, but it covers 90% of what most applications need without the cognitive overhead.

Cloudflare Load Balancing

Cloudflare approaches load balancing from the edge. Their load balancer sits across Cloudflare's global network of data centers and routes traffic based on server health, geographic proximity, and latency. It integrates with Cloudflare's CDN, DDoS protection, and DNS services, making it a compelling option for websites that already use Cloudflare for other purposes.

Cloudflare Load Balancing supports both Layer 7 HTTP(S) and Layer 4 TCP traffic (via Spectrum). It offers sophisticated steering policies including geo-proximity routing, and it can fail over across origins in different cloud providers—a true multi-cloud load balancing solution without managing your own GSLB infrastructure.

Whatever cloud provider you choose, the underlying infrastructure matters. Our dedicated server OS guide covers the operating system layer that powers these servers, whether they run behind a cloud load balancer or stand alone.

When Your Business Needs Load Balancing

Load balancing is not a universal requirement on day one. A single cloud VPS can comfortably handle thousands of daily visitors for a WordPress site, a small e-commerce store, or a SaaS MVP. Adding a load balancer prematurely introduces cost and complexity without proportional benefit. But certain signals tell you the time has come. Recognize them early, and you prevent outages rather than reacting to them.

High Traffic Volume

The most obvious trigger is traffic that outpaces what a single server can handle. This does not always mean millions of monthly visitors. A single unoptimized query, a spike from a social media mention, or a seasonal sales event can saturate CPU and memory on a server that handles normal traffic without breaking a sweat.

When your server consistently runs above 70% CPU utilization during peak hours, or when page load times degrade noticeably under load, a load balancer with additional backend instances will flatten those peaks. Cloud auto-scaling groups combined with load balancing make this dynamic—you spin up instances during peak hours and tear them down when traffic subsides.

High Availability Requirements

If your business loses money for every minute your website is down, you need high availability. A load balancer with at least two backend servers in different availability zones turns a single-server failure from a catastrophe into a non-event. One server can crash, undergo maintenance, or get terminated unexpectedly, and the load balancer simply routes all traffic to the surviving instance.

For businesses processing online transactions, booking appointments, or serving API customers around the clock, high availability is not a luxury. It is the foundational requirement. Load balancing plus redundancy is the minimum viable architecture for HA. Our analysis of power redundancy and uptime explains the infrastructure layer that complements load-balanced architectures.

Zero-Downtime Deployments

Without a load balancer, deploying new code means taking your server offline—even if just for seconds. A load balancer enables rolling deployments and blue-green deployment strategies. You deploy the new version to a subset of backend servers, divert a portion of traffic to them, verify behavior, and gradually shift all traffic over. If something breaks, you roll back instantly by adjusting the load balancer's target pool.

This capability transforms your deployment process from a nerve-wracking late-night operation into a routine, automated pipeline event. CI/CD pipelines integrate with cloud load balancer APIs to register and deregister targets as part of the deployment workflow.

Microservices Architectures

Monolithic applications can survive without load balancers. Microservices cannot. When you decompose an application into independently deployable services, each service needs its own endpoint that the rest of the system can discover and call. A load balancer—or an API gateway with load balancing capabilities—provides that stable endpoint while individual service instances come and go.

This is particularly relevant as more businesses explore AI hosting and next-generation server architectures, where inference workloads might be distributed across GPU-equipped instances behind a load balancer that routes based on model type or request complexity.

How to Set Up Basic Load Balancing on a Cloud VPS

You do not need a managed cloud load balancer to get started. A single cloud VPS running Nginx or HAProxy can serve as an effective load balancer for small to medium workloads. This hands-on approach gives you complete control, costs less at low traffic volumes, and teaches you the fundamentals that managed services abstract away. Here is a practical setup guide.

Option 1: Nginx as a Load Balancer

Nginx is the most popular web server on the planet, and it happens to be an excellent Layer 7 load balancer. Configuring it requires defining an upstream block that lists your backend servers and a server block that proxies requests to that upstream group.

The minimal Nginx load balancer configuration looks like this:

upstream backend_servers {
    server 10.0.0.2:80;
    server 10.0.0.3:80;
    server 10.0.0.4:80;
}

server {
    listen 80;
    server_name yourdomain.com;

    location / {
        proxy_pass http://backend_servers;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

This configuration distributes traffic using round-robin by default. You can change the algorithm by adding a directive—least_conn for least connections, ip_hash for session persistence, or weight=N on individual server lines for weighted distribution.

Nginx also supports passive health checks. If a backend server fails to respond, Nginx marks it as down and stops sending traffic to it for a configurable period. Active health checks require the commercial Nginx Plus subscription, but for most self-managed setups, passive checks combined with aggressive timeouts work adequately.

Option 2: HAProxy as a Load Balancer

HAProxy (High Availability Proxy) is purpose-built for load balancing. It handles extremely high connection counts with minimal resource consumption, supports both Layer 4 and Layer 7 modes, and offers more granular health checking than Nginx's open-source version.

A basic HAProxy configuration for HTTP load balancing looks like this:

frontend http_front
    bind *:80
    default_backend web_servers

backend web_servers
    balance roundrobin
    option httpchk GET /health
    server web1 10.0.0.2:80 check
    server web2 10.0.0.3:80 check
    server web3 10.0.0.4:80 check

The option httpchk directive is the standout feature here. It tells HAProxy to actively probe each backend server's /health endpoint at regular intervals. If a server fails to respond with a 2xx or 3xx status code, HAProxy removes it from the rotation until it recovers. This active health checking catches failures that passive detection would miss—partial application crashes, database connection pool exhaustion, and other gray failures.

SSL Termination

Both Nginx and HAProxy can terminate SSL/TLS connections at the load balancer. This offloads the cryptographic overhead from your backend servers, allowing them to focus on application logic. You will need your SSL certificate and private key on the load balancer server.

For Nginx, add the SSL directives to the server block:

server {
    listen 443 ssl;
    server_name yourdomain.com;

    ssl_certificate     /etc/ssl/certs/yourdomain.crt;
    ssl_certificate_key /etc/ssl/private/yourdomain.key;

    location / {
        proxy_pass http://backend_servers;
        proxy_set_header X-Forwarded-Proto https;
    }
}

Note the X-Forwarded-Proto header. Your backend application needs this to know that the original request came over HTTPS, even though the load-balancer-to-backend connection runs over plain HTTP. For maximum security, you can re-encrypt traffic between the load balancer and backends, though this adds latency and CPU overhead.

Health Checks and Failover Mechanisms

A load balancer without health checks is a traffic cop who does not notice when a lane is blocked. Traffic piles up, drivers get frustrated, and the whole system breaks down. Health checks are the eyes and ears of your load balancer, and understanding how they work separates reliable deployments from fragile ones.

How Health Checks Work

Health checks are periodic probes that the load balancer sends to each backend server. These probes can be as simple as a TCP connection attempt on a specific port or as sophisticated as an HTTP GET request to a dedicated health endpoint with custom headers and expected response codes.

When a health check succeeds, the server is marked as healthy and remains in the rotation. When health checks fail consecutively—typically two or three failures in a row—the server is marked unhealthy and removed from the active pool. Once the server starts passing health checks again, it rejoins the rotation. This hysteresis prevents a single transient failure from flapping a server in and out of service.

Designing Effective Health Check Endpoints

A naive health check simply verifies that the web server process is running—an HTTP 200 from the root path. This tells you nothing useful. A web server that returns 200 while its database connection pool is exhausted will pass a shallow health check while being completely unable to serve real traffic.

An effective health check endpoint should verify that the application can actually do useful work. This typically means:

• Database connectivity: Run a lightweight query like SELECT 1 against the primary database.
• Cache availability: Ping Redis or Memcached to confirm the cache layer is reachable.
• Disk space: Verify that critical mount points have free space above a threshold.
• Queue depth: If your application processes background jobs, check that the queue is not backed up beyond a reasonable limit.

The health check endpoint should return a clear status code—200 for healthy, 503 for degraded or unhealthy—and optionally a JSON payload with component-level status for diagnostic purposes. Keep the response time fast; a health check that triggers a slow operation will itself become a performance problem.

Failover Scenarios

Failover is the automatic redirection of traffic away from failed resources toward healthy ones. At the load balancer level, failover is the natural consequence of health checks. But effective failover architecture extends beyond the load balancer itself.

Consider these failover layers:

• Instance failover: The load balancer detects a failed backend and routes around it. This happens in seconds.
• Availability zone failover: If an entire AZ experiences issues, the load balancer—if configured with cross-zone distribution—shifts traffic to instances in other zones. Cloud providers offer this natively on their managed load balancers.
• Region failover: Global server load balancing (GSLB) or DNS failover directs users away from an entire region that has gone dark. This is the slowest failover because DNS changes take time to propagate.
• Provider failover: Multi-cloud load balancing, where Cloudflare or a self-managed GSLB sits in front of origins in both AWS and GCP, protects against a single cloud provider outage.

Each layer adds cost and complexity. Most businesses should start with instance and AZ failover—which managed cloud load balancers handle automatically—and add regional failover only when their uptime requirements justify the operational overhead.

The Real Cost of Cloud Load Balancers

Cloud load balancer pricing is rarely a single line item. Providers decompose costs into multiple dimensions, and the final bill often surprises teams that only looked at the hourly base rate. Understanding the pricing model before deployment avoids budgeting blind spots.

How Cloud Providers Charge

AWS charges for the Application Load Balancer based on Load Balancer Capacity Units (LCUs), which aggregate new connections, active connections, processed bytes, and rule evaluations per hour. You pay an hourly rate for the load balancer itself plus the LCUs consumed. The Network Load Balancer uses a similar model with NLCUs. At low traffic volumes, the cost is modest—often under $25 per month. At high traffic volumes with millions of requests, costs can climb into hundreds or thousands of dollars monthly.

Google Cloud charges per forwarding rule (the anycast IP), per ingress data processed, and per backend service. Internal load balancing is cheaper than external. The global nature of Google's load balancer means you do not pay separately for regional instances, which can reduce costs in multi-region deployments compared to AWS.

DigitalOcean charges a flat $12 per month per load balancer node, plus $0.01 per GB of data transfer. This predictable pricing is attractive for workloads with stable traffic patterns. Cloudflare Load Balancing starts at $5 per month per origin on the pay-as-you-go plan, with enterprise plans offering custom pricing.

Self-Managed vs. Managed Cost Analysis

Running Nginx or HAProxy on a cloud VPS is the self-managed alternative. A $6/month VPS (like a DigitalOcean basic droplet or a Hetzner CX22) can handle thousands of concurrent connections as a load balancer. Add your time for configuration, monitoring, patching, and incident response.

The break-even calculation depends on your team's hourly cost and the complexity of your setup. For a simple round-robin configuration with passive health checks, self-managed wins on pure cost at nearly any traffic volume under 10,000 concurrent connections. But once you need active health checks, SSL termination with automatic certificate rotation, WAF integration, auto-scaling integration, and multi-region failover, the managed services become cost-competitive when you account for engineering time.

Hidden Costs to Watch For

Data transfer fees are the most common surprise. Cloud providers often charge for data processed by the load balancer—both inbound and outbound. Cross-zone or cross-region data transfer adds another layer. In AWS, enabling cross-zone load balancing on the ALB eliminates zonal affinity, which improves fault tolerance but increases inter-zone data transfer charges.

Idle load balancer costs are another trap. A load balancer provisioned for a staging environment that sits idle 90% of the time still racks up the hourly base charge. Automation that tears down non-production load balancers during off-hours can reduce this substantially.

Finally, monitoring and logging costs accumulate. Load balancer access logs stored in S3 or Cloud Storage, metrics shipped to monitoring platforms, and the storage required for compliance retention all add to the total cost of ownership. Plan for these from the start.

Hosting Captain's Take: Making Load Balancing Work for Your Business

At Hosting Captain, we approach load balancing as a practical engineering decision, not a checkbox on a feature list. The right load balancing architecture for your business depends on your traffic patterns, your tolerance for downtime, your team's operational capacity, and your budget.

We consistently recommend starting with the simplest architecture that meets your high-availability requirements and scaling complexity alongside traffic. A single cloud VPS with automated backups and a documented recovery procedure is perfectly adequate for many small businesses. When traffic grows and downtime becomes costly, adding a managed load balancer with at least two backend instances in different availability zones provides a step change in reliability.

For businesses evaluating dedicated hosting versus cloud, our complete dedicated server guide covers the scenarios where a single powerful server still makes more sense than a fleet of cloud instances behind a load balancer. The answer is rarely universal—it depends on your specific workload.

Load balancing is a tool, not a religion. Use it where it solves a real problem, skip it where it does not, and always instrument your infrastructure so you can measure whether your load balancing investment is actually improving the metrics your users care about: response time, error rate, and availability.

Frequently Asked Questions

What is the difference between a load balancer and a reverse proxy?

A reverse proxy forwards client requests to one or more backend servers and returns the response to the client, but its primary purpose is often security, caching, or SSL termination for a single server. A load balancer is specifically designed to distribute traffic across multiple backend servers for scalability and high availability. In practice, most load balancers—including Nginx and HAProxy—are also reverse proxies, and the terms overlap significantly in cloud hosting contexts.

Can I use a load balancer with a single server?

Yes, and it is more common than you might think. A single-server configuration with a load balancer provides SSL termination, a stable endpoint that does not change when you replace or resize the server, and a foundation for adding more servers later without changing DNS or client configurations. Managed cloud load balancers also provide DDoS protection, WAF integration, and access logging that benefit single-server deployments.

Does load balancing add latency?

Yes, but typically negligible amounts. A Layer 4 load balancer adds microseconds of latency—well below human perception. Layer 7 load balancers add slightly more because they inspect HTTP headers, but this is measured in single-digit milliseconds. The latency added by the load balancer is almost always dwarfed by the latency reduction achieved through distributing load across multiple servers and preventing server overload.

What happens if the load balancer itself fails?

This is the "who watches the watchmen" problem. Managed cloud load balancers solve it through redundancy—AWS ALB, Google Cloud LB, and similar services run across multiple availability zones automatically. Self-managed load balancers require a floating IP (like a Virtual IP or keepalived) with a standby load balancer that takes over if the primary fails. For production deployments on self-managed load balancers, never run a single load balancer instance without a failover plan.

Is DNS round-robin good enough for my website?

For static content sites where occasional downtime is acceptable and traffic is low, DNS round-robin can work. But it provides no health checking, no intelligent traffic distribution, and slow failover due to DNS caching. For any business-critical website, a proper load balancer—whether self-managed or cloud-provided—should be the default choice.

How does load balancing handle WebSocket connections?

WebSockets require sticky sessions (session affinity) because the connection is long-lived and stateful. AWS ALB supports WebSockets natively with configurable stickiness. Nginx requires the ip_hash directive or the sticky module (Nginx Plus) to ensure WebSocket connections from the same client consistently reach the same backend. Without stickiness, each WebSocket upgrade request may land on a different server, breaking the connection state.

Can I load balance across multiple cloud providers?

Yes. This is multi-cloud load balancing, and it typically uses DNS-based global load balancing (like Cloudflare Load Balancing or AWS Route 53 with health checks) to direct traffic to origins in different cloud providers. You can also run your own GSLB using tools like HAProxy or Traefik with health checks across provider boundaries. The main challenge is not the load balancing technology itself but the operational complexity of maintaining consistent infrastructure across multiple cloud platforms.

What is the "thundering herd" problem in load balancing?

The thundering herd problem occurs when many idle connections or threads wake up simultaneously in response to a single event, overwhelming the system. In load balancing, this can happen when a failed server rejoins the pool after an outage—all pending requests suddenly flood the newly recovered server. Most load balancers mitigate this through slow-start mechanisms that gradually ramp up traffic to a newly healthy server rather than dumping the full load on it immediately.