Scale from Zero to Millions of Users

Nobody designs a system for millions of users on day one. Real systems grow one bottleneck at a time: something gets slow or fragile, you add the piece that fixes it, and you repeat. This walkthrough follows that path from a single server to a multi-region, sharded architecture. Almost every other design question builds on these steps, so it's the best place to start.

Scale an App to Millions of Users - System Design, by Caleb CurryYouTube

Step 1: One server does everything#

At the start, a single machine runs the web app, the database and maybe a cache. A request goes like this:

1. The user types app.example.com. DNS (usually a managed service) turns that name into the server's IP address.
2. The browser or mobile app sends an HTTP request to that IP.
3. The server returns an HTML page for browsers, or JSON for mobile apps.

This is the right choice for a prototype. It falls over when traffic grows, and if that one machine dies, the whole product is down.

Step 2: Move the database out#

The first split is between the web tier (handling requests) and the data tier (storing data). Each can now be sized and scaled on its own.

SQL or NoSQL?

Step 3: Scale up or scale out#

There are two ways to handle more load:

• Vertical scaling (scale up): a bigger machine with more CPU and RAM. It's easy, but there's a hardware ceiling, it gets expensive fast, and it's still one machine that can fail.
• Horizontal scaling (scale out): more machines sharing the work. There's no ceiling and no single point of failure, but you need a way to spread traffic across them.

Large systems scale out. That brings in the next piece.

Step 4: Put a load balancer in front#

A load balancer receives all traffic on a public IP and forwards each request to one of several web servers over a private network.

• Failover: if web server 1 dies, traffic goes to server 2 and users don't notice.
• Elasticity: when traffic spikes, add servers behind the load balancer and it starts using them.

Step 5: Replicate the database#

The web tier is now redundant, but the database isn't. Replication fixes that with one primary (also called the leader) that takes all writes, and several replicas that copy its data and serve reads.

• Performance: most apps read far more than they write, so spreading reads across replicas multiplies capacity.
• Reliability: data lives on several machines, possibly in different locations.
• Availability: if a replica fails, reads move to the others. If the primary fails, a replica is promoted to primary.

Step 6: Add a cache#

Database reads are still the slowest part of most requests. A cache keeps frequently read data in memory (Redis or Memcached), so repeat reads skip the database entirely.

The usual pattern is cache-aside: check the cache first; on a miss, read the database and store the result in the cache.

Things to decide:

• What to cache: data that's read often and changed rarely. Never treat the cache as the only copy of anything.
• Expiry: give entries a time to live. Too short and you hit the database constantly; too long and users see stale data.
• Consistency: on a write, update the database and delete the cache entry, so the next read fetches fresh data.
• Eviction: when memory fills, drop entries with a policy such as LRU (least recently used).
• No single point of failure: run several cache nodes, and leave memory headroom for traffic spikes.

Step 7: Serve static files from a CDN#

Images, videos, CSS and JavaScript don't change per user. A CDN copies them to edge servers around the world, so a user in Mumbai downloads them from a nearby edge instead of your origin in Virginia.

• Cost: you pay for CDN transfer, so don't push rarely used files through it.
• Expiry: set cache lifetimes per file type; version file names (app.3f9c.js) so new releases are picked up instantly.
• Fallback: if the CDN has an outage, clients should be able to fetch from the origin.
• Invalidation: CDNs offer APIs to purge files early, but versioned names make this rarely necessary.

Step 8: Make the web tier stateless#

If a user's login session lives in one web server's memory, every request from that user must reach that same server ("sticky sessions"). That makes adding and removing servers painful.

Move session data to a shared store (Redis, or a NoSQL table) that every web server can read. Now any server can handle any request, and you can autoscale: add servers when CPU climbs, remove them when traffic falls.

Step 9: Go multi-region#

To serve users worldwide and survive a whole datacenter going down, run the stack in several regions. GeoDNS sends each user to the nearest healthy region.

The hard parts:

• Traffic failover: when a region fails, GeoDNS must route its users elsewhere, and the remaining regions need spare capacity.
• Data sync: each region needs the data. Replicate asynchronously across regions and decide what happens when two regions write the same record.
• Deployments: the same build and configuration must reach every region. Automate it.

Step 10: Decouple with a message queue#

Some work doesn't need to happen while the user waits: resizing photos, sending emails, updating search indexes. A message queue (Kafka, RabbitMQ, SQS) lets web servers publish a job and return immediately, while workers consume jobs at their own pace.

• If workers fall behind, the queue grows instead of requests failing, and you add more workers.
• If workers crash, jobs wait in the queue until they recover.

Step 11: Logging, metrics and automation#

With dozens of servers, you can no longer SSH into a box to see what went wrong.

• Logging: send every server's logs to one place so you can search errors across the fleet.
• Metrics: track host metrics (CPU, memory, disk), service metrics (latency, error rate, database load) and business metrics (sign-ups, revenue, active users), and alert on them.
• Automation: continuous integration runs tests on every change; automated deploys and autoscaling remove slow, error-prone manual steps.

Step 12: Scale the database itself#

Eventually one primary can't take all the writes, or the data no longer fits on one machine. Scaling the database up buys time, but has the same limits as any vertical scaling. The long-term answer is sharding: splitting the data across many databases.

Every shard has the same schema and holds a slice of the rows. The shard key (here user_id) decides where each row lives, so choose one that spreads data and traffic evenly.

Sharding brings new problems:

• Resharding: when shards fill unevenly or the data outgrows them, data must move. Consistent hashing reduces how much moves.
• Hot shards: one celebrity's data can overload one shard. You may need to give very busy keys their own shard or split them further.
• Joins: you can't join tables across shards cheaply. Denormalise so a query can be answered from one shard.

Putting it together#

The final architecture keeps every lesson from the steps above:

1. Keep the web tier stateless.
2. Build redundancy into every tier.
3. Cache as much as you can safely.
4. Run in more than one region.
5. Serve static files from a CDN.
6. Shard the data tier when one database isn't enough.
7. Decouple components with queues.
8. Monitor everything and automate the rest.