Calendar Node Pool Autoscaler

A new GKE node takes about ten minutes to join the cluster and pull a user image. A class logging in together waits all of it. The node placeholder scaler avoids that by parking a low-priority pod on a spare node ahead of time. Real users evict the placeholder, and the node is already warm.

Each placeholder nearly fills a node, so one placeholder is one idle node held in reserve. How many to hold comes from a baseline in the support chart and from events on a shared Google Calendar.

Where it lives¶

The scaler is a subchart of the support chart, so it deploys on the support layer with cert-manager, ingress-nginx, Prometheus and Grafana. See the deployment pipeline.

How it works¶

The pod runs python3 -m scaler in a loop, one pass every 60 seconds. Each pass reads the config and placeholder template from a mounted ConfigMap, fetches the calendar, works out a replica count per pool, and applies a <pool>-placeholder deployment with kubectl.

The config is re-read every pass, so a helm upgrade that rewrites the ConfigMap lands within a minute or two. The pod never needs restarting.

Deciding the replica count¶

For each pool under nodePools, in order:

1. If the calendar asks for more than zero and calendarOverrideEnabled is true, that number wins outright.

2. Otherwise, if one of the pool’s placeholder pods is Pending, the count holds at the configured replicas. A real user just evicted that placeholder and it has not landed yet, so scaling down here would throw the spare away.

3. Otherwise the count is the target minus one per node with room to spare, floored at zero. The target is the calendar count if there is one, the configured replicas if not.

A node has room when its free resource ratio is above the threshold for the configured strategy:

Four things keep a node from counting, even when it looks free:

• it already hosts a placeholder pod

• it is cordoned/unschedulable

• the scaler has been watching it for less than nodeGracePeriod seconds

• it was last busy less than nodeGracePeriod seconds ago

Both grace periods exist because a node’s usage swings hard right after it appears and right after a user logs out. Without them the scaler tears down a placeholder it is about to need, and the count oscillates.

The calendar¶

Cal-ICOR Scaling Events is a public Google calendar shared with infrastructure staff. The event description is a YAML mapping of pool name to replica count:

base: 3
workshop: 1

The keys are pool names from nodePools in support/values.yaml, not hub names. Today that means base and workshop. Unknown keys and non-integer counts are dropped silently.

Scope the events tightly. If the surge is a 9am class, run the event from 8:50 to 9:10. When two events overlap and both name the same pool, the larger count wins.

An event asking for 0 skips the override path, which only fires above zero, but it still sets the target to zero, so the count lands there anyway. That is how the standing evening cool-off event works. Only a pending placeholder pod beats it, holding at the configured replicas.

Configuration¶

The config lives in the support helm chart, under node-placeholder-scaler: in support/values.yaml.

A higher memoryThreshold means fewer nodes count as free, so placeholders survive longer and a replacement node starts sooner. 0.40 is deliberately aggressive: our users are memory bound and arrive in bursts.

Sizing a placeholder pod¶

The placeholder has to be big enough that a user pod cannot fit beside it, and small enough to still schedule alongside a node’s system pods. Run the script in the scaler repo against a node from the pool you are sizing:

./tools/determine_placeholder_pod_memory.py <node name>

Node allocatable memory: 61483356160 bytes
Total non-notebook memory used by pods: 466964736 bytes

Recommended placeholder pod size for values.yaml: 60738518784 bytes

Put the recommended number into that pool’s resources.requests.memory.

Changing the configuration¶

Edit support/values.yaml on a feature branch and open a PR against staging in cal-icor-hubs. The path picks up the support-deployment label, and merging redeploys the support chart.

In term the base baseline is normally 1. Over breaks, or for a pool nobody is using, set it to 0.

Changing the scaler itself¶

The scaler lives upstream, in berkeley-dsep-infra/jupyterhub-k8s-node-placeholder. Fork it, branch, and open a PR against main. It wants Python 3.11:

conda create -n scalertest python=3.11
conda activate scalertest
pip install -r dev-requirements.txt
pip install -r node-placeholder-scaler/requirements.txt
pre-commit install

The tests cover scaler.py and calendar_parser.py, and run on every PR through test-scaler.yaml. To run them yourself:

python node-placeholder-scaler/tests/run_tests.py

To add or update a Python dependency, edit requirements.in and recompile from inside node-placeholder-scaler/:

pip-compile --output-file=requirements.txt requirements.in

Merging to main runs build-publish-node-placeholder.yaml, which builds and pushes the image with chartpress, then pushes the packaged chart to GAR. Chartpress derives the version from git history, hence 0.0.1-0.dev.git.424.h1c64a4b.