LearnCodeLive

Why Scale?

A single Node.js process on one server has hard limits:

• CPU — one core (event loop is single-threaded)
• Memory — limited by server RAM (e.g., 512MB on a small VPS)
• Connections — limited by file descriptors (default ~1024 on Linux)
• Throughput — limited by event loop capacity (~10K req/s on one core)

When your app exceeds any of these, you need to scale.

Single Node Process              Scaled App
┌────────────────────┐           ┌────────────────────┐
│    1 CPU Core       │           │   Load Balancer     │
│    512MB RAM        │           │         │           │
│    ~1024 sockets    │           │    ┌────┼────┐      │
│    ~5000 req/s      │           │    │    │    │      │
└────────────────────┘           │  ┌─┴┐ ┌─┴┐ ┌─┴┐    │
                                  │  │W1│ │W2│ │W3│    │
                                  │  └──┘ └──┘ └──┘    │
                                  │  3 CPU Cores        │
                                  │  1.5GB RAM total    │
                                  │  ~15,000 req/s      │
                                  └────────────────────┘

Vertical vs Horizontal Scaling

Vertical Scaling:               Horizontal Scaling:
┌────────────────┐              ┌────────────────┐
│ 1 Big Server   │              │ Load Balancer  │
│ 16 cores       │              └───┬───┬───┬────┘
│ 64GB RAM       │              ┌──┴┐ ┌┴┐ ┌┴──┐
└────────────────┘              │S1 │ │S2│ │S3 │
                                └───┘ └──┘ └───┘

1. Vertical Scaling — The Node.js Cluster Module

Before adding more machines, use all the cores on one machine:

const cluster = require('cluster');
const os = require('os');

if (cluster.isMaster) {
  const cpuCount = os.cpus().length;
  console.log(`Master ${process.pid} forking ${cpuCount} workers`);

  for (let i = 0; i < cpuCount; i++) {
    cluster.fork();
  }

  cluster.on('exit', (worker) => {
    console.log(`Worker ${worker.process.pid} died, restarting`);
    cluster.fork();
  });
} else {
  require('./app.js');  // Your app — each worker runs independently
}

PM2 Cluster Mode (Simpler)

# PM2 handles clustering with zero code changes
pm2 start app.js -i max        # One worker per CPU core
pm2 start app.js -i 4          # Exactly 4 workers
pm2 start app.js -i 0          # Auto-detect (max - 1)

// ecosystem.config.js
module.exports = {
  apps: [{
    name: 'my-api',
    script: 'app.js',
    instances: 'max',
    exec_mode: 'cluster',
  }],
};

Cluster Limitations

• In-memory state is NOT shared — each worker has its own heap. Session data, caches, and counters are duplicated. Use Redis or a database for shared state.
• Sticky sessions — if using WebSockets or long-lived sessions, configure the load balancer for sticky sessions (same client → same worker).

2. Horizontal Scaling — Load Balancing

When one machine isn’t enough, distribute traffic across multiple machines:

                          ┌────────────────┐
                          │  Load Balancer  │
                          │  (Nginx / HAProxy / AWS ALB) │
                          └──┬──────┬──────┬──┘
                             │      │      │
                       ┌─────┴┐ ┌───┴──┐ ┌┴─────┐
                       │ App1 │ │ App2 │ │ App3 │
                       │ node │ │ node │ │ node │
                       └──────┘ └──────┘ └──────┘

Nginx as a Load Balancer

# /etc/nginx/nginx.conf
http {
    upstream my_api {
        # Round-robin (default)
        server app1.internal:3000;
        server app2.internal:3000;
        server app3.internal:3000;

        # Optional: sticky sessions via IP hash
        # ip_hash;
    }

    server {
        listen 80;
        server_name api.example.com;

        location / {
            proxy_pass http://my_api;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection 'upgrade';
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }
    }
}

Load Balancing Strategies

3. Statelessness — The Key to Scaling

For horizontal scaling to work, your app must be stateless:

// ❌ BAD — state stored in process memory
const sessions = new Map();

app.post('/login', (req, res) => {
  const token = generateToken();
  sessions.set(token, { user: req.body.username });
  // If this request goes to server 2 on next call, session is lost!
  res.json({ token });
});

// ✅ GOOD — state stored externally
app.post('/login', async (req, res) => {
  const token = generateToken();
  await redis.set(`session:${token}`, JSON.stringify({ user: req.body.username }), 'EX', 3600);
  // Any server can look up the session
  res.json({ token });
});

app.get('/profile', async (req, res) => {
  const session = await redis.get(`session:${req.headers.authorization}`);
  // Works regardless of which server handles the request
});

Stateless Checklist

4. Reverse Proxy

A reverse proxy sits in front of your Node.js app and handles concerns that Node.js shouldn’t:

server {
    listen 443 ssl;
    server_name api.example.com;

    # SSL termination
    ssl_certificate /etc/letsencrypt/live/api.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/api.example.com/privkey.pem;

    # Static files — serve directly, don't hit Node.js
    location /static/ {
        root /var/www/public;
        expires 365d;
        add_header Cache-Control "public, immutable";
    }

    # API requests — proxy to Node.js
    location /api/ {
        proxy_pass http://localhost:3000;
        proxy_set_header X-Forwarded-Proto $scheme;

        # Rate limiting at proxy level
        limit_req zone=api burst=20 nodelay;
    }

    # Compression at proxy level
    gzip on;
    gzip_types text/plain application/json application/javascript text/css;
}

What the reverse proxy handles:

• SSL/TLS termination — Node.js doesn’t handle SSL directly
• Static file serving — Nginx serves static assets 10x faster than Node.js
• Compression — gzip at proxy level, offloads CPU work from Node.js
• Rate limiting — block malicious traffic before it reaches your app
• Request buffering — handles slow clients so Node.js doesn’t wait

5. Microservices Architecture

Instead of one big app, split into small, independent services:

Monolith:                        Microservices:
┌──────────────────────┐         ┌────────┐ ┌────────┐
│ User Management      │         │ Auth   │ │ Users  │
│ Product Catalog      │         │ Service│ │ Service│
│ Order Processing     │         └────────┘ └────────┘
│ Payment              │         ┌────────┐ ┌────────┐
│ Notifications        │         │ Orders │ │Payment │
│ Analytics            │         │ Service│ │Service │
└──────────────────────┘         └────────┘ └────────┘
                                         │
                                    ┌────┴────┐
                                    │  Message │
                                    │  Queue   │
                                    └─────────┘

Communication Between Services

// Synchronous — HTTP API call
const response = await fetch('http://user-service:3001/api/users/42');
const user = await response.json();

// Asynchronous — message queue
const amqp = require('amqplib');
const connection = await amqp.connect('amqp://rabbitmq');
const channel = await connection.createChannel();

// Publish event
channel.publish('orders', 'order.created', Buffer.from(JSON.stringify(order)));

// Consume event (in notification service)
channel.consume('order.created', (msg) => {
  const order = JSON.parse(msg.content.toString());
  await sendEmail(order.userEmail, 'Order confirmed!');
  channel.ack(msg);
});

Microservices Pros & Cons

6. Auto-Scaling

Automatically add or remove servers based on load:

# AWS Auto Scaling config (conceptual)
AutoScalingGroup:
  MinSize: 2
  MaxSize: 10
  ScalingPolicies:
    - PolicyName: ScaleOut
      MetricType: CPUUtilization
      TargetValue: 70
      ScaleInCooldown: 60
    - PolicyName: ScaleIn
      MetricType: CPUUtilization
      TargetValue: 30
      ScaleInCooldown: 120

For simpler setups, use a platform that auto-scales:

• Railway — auto-scales based on CPU/memory
• Fly.io — auto-scales to zero, wakes on request
• Heroku — auto-scaling dynos (paid addon)
• AWS Elastic Beanstalk — auto-scaling with CloudWatch

7. Database Scaling

When your database becomes the bottleneck:

Read Replicas:                    Sharding:
┌────────┐                       ┌────────┐
│ Master │──► Replica 1          │ Shard 1 │ Users A–M
│ (write)│──► Replica 2          ├────────┤
└────────┘──► Replica 3          │ Shard 2 │ Users N–Z
        (reads distributed)       └────────┘

// Read/write splitting
const { Pool } = require('pg');

const writerPool = new Pool({ connectionString: process.env.DB_WRITER_URL });
const readerPool = new Pool({ connectionString: process.env.DB_READER_URL });

async function getUser(id) {
  const { rows } = await readerPool.query('SELECT * FROM users WHERE id = $1', [id]);
  return rows[0];
}

async function updateUser(id, data) {
  const { rows } = await writerPool.query('UPDATE users SET name = $1 WHERE id = $2 RETURNING *', [data.name, id]);
  return rows[0];
}

Scaling Decision Tree

Is your app slow?
    │
    ├── Profile the bottleneck
    │
    ├── Is it CPU? → Cluster mode (PM2 -i max)
    │
    ├── Is it memory? → Bigger server or Redis cache
    │
    ├── Is it database? → Indexes → Read replicas → Sharding
    │
    ├── Is it network? → CDN, compression, keep-alive
    │
    └── Is it one server's limit? → Horizontal scaling with load balancer

Key Takeaways

• Vertical scaling first — use PM2 cluster mode to utilise all CPU cores on one machine
• Cluster mode does NOT share memory — use Redis for sessions, caches, and shared state
• Horizontal scaling requires a load balancer (Nginx, HAProxy, AWS ALB)
• Statelessness is essential for horizontal scaling — sessions go in Redis, files go in S3
• Reverse proxy (Nginx) handles SSL, static files, compression, rate limiting — offloads Node.js
• Microservices enable independent scaling but add operational complexity
• Auto-scaling adjusts capacity based on metrics (CPU, request rate) — use it for variable traffic
• Database scaling (read replicas, sharding) is often the next bottleneck after app servers
• Always measure before scaling — profile to find the real bottleneck; scaling the wrong layer wastes resources