Autoscaling game servers on connections, not CPU

Game infrastructure·Kubernetes·Dmitry Nichiporchik

Why CPU is the wrong signal for a game server

CPU utilisation is the default autoscaling metric because for stateless request handlers it is a good proxy: work arrives, work is done, cores get busy, and any replica can serve any request. A game session server breaks all three assumptions. It holds long-lived connections, it owns state that cannot be moved mid-match, and its tick loop does roughly constant work per session whether the session is exciting or idle.

That produces two failures, and they pull in opposite directions.

Scaling up on CPU is late. CPU is an average over a window; a lobby filling is an event. By the time a rolling average crosses a threshold, the queue has already been waiting, and the players who felt it have already formed an opinion about your matchmaking. The signal you want moves the instant a connection is accepted.

Scaling down on CPU is worse than late; it is wrong. Between matches a pod's CPU falls. A CPU-driven autoscaler reads that as spare capacity and terminates the pod — and if the pod's next match has already started, that is a race ended by an infrastructure decision. This is the failure that turns into a review saying the game disconnects you at random, and it is entirely self-inflicted.

The interesting part is that both failures have the same cause: CPU describes how busy the machine is, and the question the autoscaler is actually asking is how full the tier is and whether this particular pod is safe to remove. Those are different questions, and one metric answers both of them.

Scaling on active connections per pod, with KEDA

Each session pod exports one gauge: the number of live connections (or live sessions — use whichever is the unit your capacity is measured in) that it currently owns. The autoscaler targets an average value of that gauge per pod, not a total.

The per-pod framing matters more than it looks. Capacity planning for a game server is already per-pod: load testing tells you a pod holds N sessions before latency degrades. Targeting an average per pod makes the autoscaler's arithmetic the same arithmetic as the capacity model, so the number in the manifest is a number someone measured rather than a number someone tuned.

# KEDA reads the gauge directly; a plain HPA would need an adapter
# exposing the same value through the custom-metrics API, which is
# one more component to own.
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: session-tier
spec:
  scaleTargetRef:
    name: session-tier
  minReplicaCount: 6        # a floor, not an optimisation
  cooldownPeriod: 300       # seconds of calm before scaling in
  triggers:
    - type: prometheus
      metadata:
        query: sum(game_session_connections)
        threshold: '70'   # target per pod: 70% of the tested ceiling

Three details that are easy to get wrong:

• The floor is load-bearing. minReplicaCount is not an efficiency setting; it is how many pods exist when concurrency is zero at four in the morning and a streamer starts a run. Set it to what an unannounced spike needs while new pods boot.
• Scale-in needs a long cooldown, scale-out does not. Adding a pod too eagerly costs a few cents. Removing one too eagerly costs a match. Asymmetric aggressiveness is correct here.
• Draining pods must not count as capacity. A pod that is finishing its last matches still reports connections. If that value feeds the average, the autoscaler sees a full tier and adds pods you do not need — harmless but noisy. Exclude terminating pods from the query, or accept the noise knowingly.

Why a reactive autoscaler is always late, and what to do about it

Connections per pod is a current load signal. It tells you the tier is filling; it cannot tell you the tier is about to fill. And the gap between those two facts is filled by something no metric changes: how long a pod takes to become useful.

Add it up honestly — image pull, process start, readiness probe passing, and any warm-up your server does before it can accept a session. Call it T. If concurrency can double in less than T, a purely reactive autoscaler cannot keep up, and no threshold tuning will change that. There are only two real levers:

• Carry headroom sized to T. Roughly, headroom should cover the arrival rate multiplied by T. That is what the 70% threshold above is: a permanent 30% of the tier held back so the autoscaler has time to react. It costs money, and it costs less than the alternative.
• Pre-warm on the calendar. The spikes that hurt a live game are mostly known in advance, because the studio schedules them: an event start, a season rollover, a tournament, a marketing beat. A live-ops calendar is a scaling calendar. Scheduled scale-out ahead of a known event is the cheapest capacity you will ever buy, and it is a cron entry.

This is also the honest answer to "why not scale on a leading indicator like queue arrival rate". You can, and it helps at the margin. But a derivative is noisy, and a noisy signal driving scale-in is how you get flapping on a tier where flapping ends matches. We keep the load signal simple and buy the reaction time with headroom instead.

Why we run our own orchestrator instead of Agones

Agones is good, it is the default answer to this problem, and most studios should use it. It gives you game-server lifecycle as a Kubernetes primitive: allocation, fleet management, health, and a well-trodden upgrade path.

We run our own placement for one reason: session placement is coupled to matchmaking. By the time a session needs a home, the matchmaker has already solved constraints — party, region, latency band, rating — and placement has to honour the solution it just produced rather than re-derive it. Putting an allocation API between the matchmaker and the pod means either duplicating those constraints or flattening them, and both cost you the thing the matchmaker exists for.

The trade is real and worth stating plainly: you own the failure modes, the upgrade path and the pager. If you are not already writing a matchmaker with non-trivial constraints, this is not a trade worth making. Use Agones and spend the saved months on the game.

Draining game servers with preStop, so a patch during an event is routine

Autoscaling and deployment are the same problem wearing different hats: both remove pods, and a pod holding a live race must not simply die. This is where most game backends discover their scaling design is incomplete, usually during an event.

What has to be true, in order:

1. The pod stops being a placement target the moment it is marked for termination — before anything else happens. With our own orchestrator that is an explicit signal to the placer; with an allocation API it is the allocation being withdrawn. If new sessions can still land on a draining pod, nothing below matters.
2. Existing sessions finish. Not migrated, not force-ended: finished. A race is short, which is exactly what makes this tractable for a runner and hard for a persistent world.
3. The process exits only then, and the grace period has to be long enough to allow it.

# preStop runs before SIGTERM reaches the process, and the whole
# sequence is bounded by terminationGracePeriodSeconds. If that
# number is smaller than your longest session, Kubernetes ends
# matches for you.
lifecycle:
  preStop:
    exec:
      command: ["/bin/drain", "--stop-accepting", "--wait-for-empty"]
terminationGracePeriodSeconds: 600   # longest match + margin
readinessProbe:                       # false while draining
  httpGet: { path: /ready, port: 8080 }
livenessProbe:                        # still true, or Kubernetes
  httpGet: { path: /healthz, port: 8080 }  # restarts the pod
                                        # you are trying to retire

The reward for getting this right is specific and worth the effort: deploying during a live event stops being a decision. Pods retire as their matches end, new pods take new sessions, and no player sees anything. Without it, every patch waits for a quiet window, and a live game does not have many.

What this looked like at 39,000 concurrent players

The stack above is what runs behind Metarun, a 1v1v1 competitive mobile runner we built end to end: KEDA on a per-pod connection gauge, our own placement coupled to MMR matchmaking, preStop draining. Its measured peak was 39,000 concurrent players, reached during an in-game event, at 60 ms average round-trip to the match service and 99.8% uptime in production. Three players per race at that concurrency is on the order of thirteen thousand simultaneous authoritative sessions, each one a pod's worth of state that cannot be moved.

A separate title, which we cannot name, ran the same design at 52,000 peak concurrent players with 62 ms average latency and 99.98% uptime. Different game, different numbers, same three decisions — and in both cases the event peaks were the ones that mattered, because an average never breaks anything.

What this does not cover

Deliberately out of scope, because each is its own decision: multi-region placement and cross-region matchmaking; spot or preemptible capacity, which interacts badly with long-lived sessions unless draining is already solid; session migration, which is the answer for persistent worlds and mostly unnecessary for match-based ones; and the cost of all of it, which has its own arithmetic where egress rather than compute turns out to be the largest line.

For what the tier this autoscales actually looks like, and what breaks first on the way up, see backend architecture for 50,000 concurrent players.