How We Stopped DDoS Attacks from Reaching Our Servers

The alert came in at 2pm on a Tuesday. Request error rates spiking, then the ALB falling over. We scrambled to investigate — and found a volumetric DDoS attack. By the time we identified it and started responding, we’d already been down for 20 minutes. The outage lasted 90 minutes total.

After the post-mortem, one thing was obvious: our WAF was in the wrong place.

The Architecture Problem

Our setup at the time looked like this:

Internet → Application Load Balancer → WAF → Application

Can you spot the flaw? WAF was behind the ALB. Every request, including every attack request, hit the ALB first. By the time WAF could inspect and block traffic, the ALB was already absorbing 100,000+ requests per second. The origin was overwhelmed before WAF could do anything useful.

Blocking at the origin is already too late for volumetric attacks. You need to absorb the traffic before it reaches your infrastructure.

Moving Defense to the Edge

We redesigned around a simple principle: DDoS defense has to happen at the edge, not at the origin.

Internet → CloudFront + WAF → ALB → Application

Three key changes:

1. CloudFront as the entry point

AWS CloudFront distributes incoming traffic across hundreds of edge locations globally. A 200,000 req/sec attack that would overwhelm a single ALB gets absorbed by CloudFront’s edge network — each edge location only sees a fraction of the total volume. The attack is diluted before it reaches the origin.

2. WAF at CloudFront, not the ALB

Moving WAF to CloudFront means rate limiting and rule evaluation happen at the edge. Malicious requests get blocked before they travel to the VPC at all. The latency of inspection is paid at the edge point, not at the origin.

3. AWS Shield Advanced

Shield Advanced adds automatic DDoS detection and mitigation at the network and transport layers. It also provides cost protection — AWS absorbs scaling costs incurred during a verified DDoS event.

The Monitoring Side

Defense without visibility is incomplete. We built Datadog dashboards tracking:

• CloudFront request rate (total and by geography)
• WAF block count and rule hit breakdown
• Origin error rate (what reaches the ALB)
• Ratio of blocked to passed requests

PagerDuty alerts fire when the request rate pattern suggests an active attack — a sudden spike in requests from a narrow IP range, or a geographic distribution that doesn’t match normal traffic patterns. This gives us heads-up before any customer impact.

Two Years Later

Since the redesign:

• 4 DDoS attacks blocked automatically — no engineer action required for any of them
• Largest attack: ~200,000 requests/second — blocked at the edge, zero origin server impact
• Zero production outages from DDoS

The 90-minute outage was expensive. The redesign cost a day of engineering work. That’s a trade I’d make every time.

What We’d Do Differently

If starting from scratch: I’d add CloudFront from the beginning, not as a retrofit after an incident. The architectural change is harder when there’s existing traffic and existing direct-to-ALB integrations to migrate. Starting with CloudFront as the entry point is the right default.

I’d also set up the geographic blocking rules more aggressively upfront. We started with rate limiting only and added geo-blocking later. If your application has a known geographic user base, blocking unexpected geographies from day one reduces the attack surface before you ever face an attack.

---

The architecture changes described here apply broadly to any web application running on AWS. The specific WAF rules and rate limits will vary by application traffic patterns.