Horizontal and vertical scaling, bottlenecks, and load balancing

The coffee app gets famous, and traffic quadruples.

“Scale it,” says the meeting.

What does that physically mean? There are exactly two moves, and the choice between them shapes everything downstream: horizontal and vertical scaling.

What is vertical scaling?

Vertical scalingConcept · lights on your mapvertical scaling“Scaling up”: replacing the machine with a bigger one. Simple, since no software has to change, but it has a ceiling and remains one machine. Its alternative is horizontal scaling., “scaling up,” means replacing the machine with a bigger one.

Module 8 sketched both moves in passing; this lesson is where they take the stage.

Module 2’s worker gets stronger; the desk gets wider. Where the backend ran on 4 CPUs and 16 GB of memory, it now runs on 16 CPUs and 64 GB.

The appeal is real because nothing about the software has to change: the same monolith, the same database, and the same configuration, just running on a larger instance type from Module 8’s catalog. For many systems, especially early ones, scaling up is the simplest correct answer.

But vertical scaling has three limits:

• A ceiling. There is a biggest machine money can rent, and prices climb savagely as you approach it.
• One machine is still one machine. However mighty, it fails as a unit: one power cord, one bad restart, everything down.
• Resizing usually interrupts. Moving to a bigger instance commonly requires a restart, which means planning a maintenance moment.

What is horizontal scaling?

Horizontal scalingConcept · lights on your maphorizontal scaling“Scaling out”: adding more machines rather than a bigger one. No practical ceiling and spares built in, but unlike vertical scaling it only works if the work can be split across them., “scaling out,” means adding more machines.

Instead of one heroic instance, run twelve ordinary ones, and instead of twelve, forty.

Two properties make this the modern default for backend fleets:

• No practical ceiling. Famous systems run on thousands of machines. When one size of fleet isn’t enough, the next size is just… more.
• Redundancy comes built in. Next lesson makes that word formal; the property is already visible. Losing one machine of forty barely registers: Module 9’s load balancer routes around it while a replacement spins up.

The catch is the requirement you already know.

The work must be splittable. Any instance must be able to serve any request. That is exactly why Module 9 ended on statelessness. Stateless middles split perfectly, while the stateful core (the database that cannot simply be photocopied) resists.

Side by side:

The two are not enemies. A common real-world arrangement scales the stateless middle out, and the stubborn stateful core up.

What is a bottleneck?

Before scaling anything, engineers hunt for one thing first.

A bottleneckConcept · lights on your mapbottleneckThe one component limiting the whole system’s throughput; everything else waits on it. Adding capacity anywhere except the bottleneck changes nothing. “Where’s the bottleneck?” is the first question of every performance investigation. is the narrowest point in the system, the one component whose capacity limits everything else.

The arithmetic is merciless:

So scaling anything except the bottleneck is the most expensive way to change nothing, which is why “what’s actually the bottleneck?” is the first adult question of every performance meeting, asked before anyone proposes buying anything.

How do engineers find it? They measure instead of trusting intuition.

Each component reports how busy it is, whether CPU, memory, connections, or queue depth, and the bottleneck is the one pinned near its limit while everything behind it waits. In the Check below, one number tells the whole story.

The usual suspects, in rough order of fame:

• The database, the stateful core that cannot simply be photocopied
• A third-party API’s rate limit (Module 6’s allowance, rediscovered)
• A single queue’s workers, draining slower than messages arrive
• The network path or a connection limit somewhere unglamorous

One more property makes bottlenecks a permanent sport.

Fixing a bottleneck reveals the next one.

Widen the database and the payment provider’s rate limit becomes the narrow door. Widen that, and it is the notification workers. At any moment one constraint dominates, and removing it promotes the next. Performance work becomes an unending rotation of narrowest doors.

Load balancing, revisited as a lever

Two concepts you already own return in this module as design levers.

Load balancingConcept · lights on your mapload balancingRe-encounter from Module 9: the front-door machinery spreading requests across instances. What’s new at scale: it is what makes horizontal scaling work at all, and its health checks are what make machine death boring. is the machinery that makes horizontal scaling work at all.

Forty instances are useless unless something spreads traffic across them, skips the sick ones, and folds in the new ones, and that something is Module 9’s doorman. Seen from this module, it is the enabling half of the horizontal lever.

Autoscaling, revisited as the lever automated

AutoscalingConcept · lights on your mapautoscalingRe-encounter from Module 8: capacity as a rule, not a guess. Seen from this module: the horizontal lever automated, the system reading its own load and pulling the add-machines lever itself. is the horizontal lever pulled automatically.

Module 8 introduced the rules (minimum, maximum, desired) and the signals that trigger them. What this module adds is the framing: the system pulling the add-machines lever itself, before the humans notice the dinner rush.

Two reminders travel with it:

• Autoscaling scales what is scalable, the stateless middle. It cannot simply photocopy the database.
• Scaled-out middles press harder on whatever they call, which is how autoscaling can move the bottleneck downstream (Module 8’s warning, now with this lesson’s vocabulary).

Assemble the pieces and a remarkable sentence becomes possible:

“We 10x’d traffic and changed nothing.”

With stateless instances, a load balancer spreading them, and autoscaling growing them, the design absorbed growth without a redesign, as long as the stateful core still had headroom. That sentence is what scalability (last lesson’s word) looks like when it is true.

The mental model to remember

Vertical scaling makes the machine bigger: simple, no software changes, but it has a ceiling, remains one machine, and resizing usually interrupts.

Horizontal scaling adds machines: no practical ceiling and built-in redundancy, but the work must be splittable, which is why statelessness matters.

The common arrangement: scale the stateless middle out, and the stateful core up.

A bottleneck is the one component limiting everything. Capacity added anywhere else changes nothing, and fixing one bottleneck reveals the next.

Load balancing is what makes horizontal scaling work. Autoscaling is the horizontal lever automated.

You should now be able to hear “scale it” and ask the two questions that make it real: which direction, up or out? And first: where is the bottleneck?