The instructions are adapted from z2jh documentation for Azure deployment.
Install the azure-cli and login to your account.
curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash
az loginYou’ll need to open a browser and follow the instructions in your terminal to log in.
Consider setting a cloud budget for your Azure account. This can only be done by the account owner. It is not yet applied.
Generate an SSH key pair if you don't have one already.
ssh-keygen -f ssh-key-aiidalab-demo-serverOne procedure for every environment. Set the variables for the one you are building, then run the steps below unchanged — they only refer to those variables.
Names derive from PROJECT and ENV so they cannot drift apart. Production predates that
and states its names literally; the comments show what the scheme would produce, for whenever
it is next rebuilt.
This is needed rarely: production already exists, and staging has no cluster of its own.
production — as built, for reference
Already exists. These are the values it actually has, verified against the live cluster — useful for rebuilding it, not for running now.
ENV=production
PROJECT=aiidalab-demo
LOCATION=eastus
# Literal, not derived: these predate the naming scheme and cannot be changed
# without rebuilding the cluster. What they would be called today is in comments.
RG=aiidalab-demo-server # ${PROJECT}-${ENV}
CLUSTER=demo-server-production # ${PROJECT}-${ENV}
VNET=aiidalab-demo-vnet # ${RG}-vnet
SUBNET=aiidalab-demo-subnet # ${RG}-subnet
VNET_PREFIX=10.0.0.0/8; SUBNET_PREFIX=10.240.0.0/16
SERVICE_CIDR=10.0.0.0/16; DNS_SERVICE_IP=10.0.0.10
SYSTEM_VM=Standard_D2s_v5; SYSTEM_COUNT=1
USER_VM=Standard_D8s_v5; USER_MIN=1; USE
F438
R_MAX=7
OS_DISK_TYPE=Managed; OS_DISK_GB=128
NETWORKING_RG=aiidalab-networking
DNS_ZONE=aiidalab.io; DNS_RECORD=demo
CI_ROLE="Azure Kubernetes Service RBAC Writer"
CI_SCOPE_SUFFIX=""
⚠️ ItsSERVICE_CIDRsits insideVNET_PREFIX. Azure asks that they not overlap; it works today only because the subnet happens to sit elsewhere in that range. Do not copy this layout into a new cluster — use the one in thedevblock.
staging — no cluster to create
Staging runs as a namespace on the production cluster, so there is nothing to build here. Skip to Configuration and give it its own identity, scoped to that namespace:
ENV=staging
RG=aiidalab-demo-server # production's
CLUSTER=demo-server-production # production's
NAMESPACE=staging
CI_ROLE="Azure Kubernetes Service RBAC Writer"
CI_SCOPE_SUFFIX="/namespaces/staging"Its CI identity gets Azure Kubernetes Service RBAC Writer on
<cluster>/namespaces/staging and nothing wider — that scope is what keeps a staging deploy
out of production.
dev — per-pull-request previews
ENV=dev
PROJECT=aiidalab-demo
LOCATION=eastus
# Derived — four names from two variables, so they cannot drift apart.
RG=${PROJECT}-${ENV}
CLUSTER=${PROJECT}-${ENV}
VNET=${RG}-vnet
SUBNET=${RG}-subnet
VNET_PREFIX=10.240.0.0/16; SUBNET_PREFIX=10.240.0.0/20
SERVICE_CIDR=10.0.0.0/16; DNS_SERVICE_IP=10.0.0.10
SYSTEM_VM=Standard_D2ds_v5; SYSTEM_COUNT=1
OS_DISK_TYPE=Ephemeral; OS_DISK_GB=64
NETWORKING_RG=aiidalab-networking
DNS_ZONE=aiidalab.xyz; DNS_RECORD='*.demo'
CI_ROLE="Azure Kubernetes Service RBAC Cluster Admin"
CI_SCOPE_SUFFIX=""The d in D2ds_v5 is load-bearing — only d sizes have the local temp disk that ephemeral
OS disks need, and those are what make a stopped cluster nearly free.
Note the network layout differs from production's: here SERVICE_CIDR is outside
VNET_PREFIX, which is what Azure actually asks for.
az group create --name "$RG" --location "$LOCATION" --output none
az network vnet create \
--resource-group "$RG" --name "$VNET" \
--address-prefixes "$VNET_PREFIX" \
--subnet-name "$SUBNET" --subnet-prefix "$SUBNET_PREFIX" \
--output none
SUBNET_ID=$(az network vnet subnet show \
--resource-group "$RG" --vnet-name "$VNET" --name "$SUBNET" \
--query id --output tsv)az aks create \
--resource-group "$RG" --name "$CLUSTER" --location "$LOCATION" \
--tier free \
--enable-managed-identity \
--enable-aad --enable-azure-rbac \
--node-count "$SYSTEM_COUNT" --node-vm-size "$SYSTEM_VM" \
--node-osdisk-type "$OS_DISK_TYPE" --node-osdisk-size "$OS_DISK_GB" \
--network-plugin azure --network-policy azure \
--service-cidr "$SERVICE_CIDR" --dns-service-ip "$DNS_SERVICE_IP" \
--vnet-subnet-id "$SUBNET_ID" \
--enable-oidc-issuer --enable-workload-identity \
--output none--enable-aad --enable-azure-rbac must be set at creation: enabling them later invalidates
every existing kubeconfig. --enable-oidc-issuer --enable-workload-identity is what later lets
cert-manager hold an Azure identity, since a pod cannot borrow the cluster's own. Both fail
quietly if omitted — nothing breaks until much later.
OS_DISK_TYPE=Ephemeral constrains OS_DISK_GB. An ephemeral OS disk lives on the VM's
local temp disk, so it has to fit: Standard_D2ds_v5 offers 75 GiB, while AKS defaults to
asking for 128 GiB. Leave the default and creation fails outright with
VMCannotFitEphemeralOSDisk. 64 GiB fits comfortably and is ample for a node.
Ephemeral is worth this fuss only where the cluster sleeps — a stopped cluster still bills for
managed OS disks. Production keeps Managed.
Production also has a user node pool — the pool that actually serves users:
az aks nodepool add \
--resource-group "$RG" --cluster-name "$CLUSTER" --name users \
--node-vm-size "$USER_VM" --mode User \
--enable-cluster-autoscaler --min-count "$USER_MIN" --max-count "$USER_MAX" \
--output noneDev does not need one: a single system pool is enough for a handful of testers.
Grant cluster admin through Entra, not the local certificate. Without this, kubectl
returns Forbidden even for subscription Owners, because Azure RBAC governs the Kubernetes
API separately from Azure resource permissions.
Assign the group, not yourself: a recovery path that depends on one person stops being a recovery path the week they are away.
ADMINS_GROUP=$(az ad group list --display-name "AiiDAlab Admins" --query "[0].id" -o tsv)
CLUSTER_ID=$(az aks show -g "$RG" -n "$CLUSTER" --query id -o tsv)
az role assignment create \
--assignee "$ADMINS_GROUP" \
--role "Azure Kubernetes Service RBAC Cluster Admin" \
--scope "$CLUSTER_ID"
az aks get-credentials -g "$RG" -n "$CLUSTER" --overwrite-existing
kubelogin convert-kubeconfig -l azurecli
kubectl get nsThe CI identity for this environment is created separately — see Secrets and variables kept in GitHub.
Each environment deploys as its own app registration, so one environment's pipeline cannot use another's credentials. There is no stored secret: GitHub Actions exchanges an OIDC token for an Azure one, and the trust comes from a federated credential naming this repository and environment.
APP_ID=$(az ad app create --display-name "aiidalab-demo-${ENV}-sp" --query appId -o tsv)
az ad sp create --id "$APP_ID"
az ad app federated-credential create --id "$APP_ID" --parameters "{
\"name\": \"${ENV}\",
\"issuer\": \"https://token.actions.githubusercontent.com\",
\"subject\": \"repo:aiidalab/aiidalab-demo-server:environment:${ENV}\",
\"audiences\": [\"api://AzureADTokenExchange\"]
}"
echo "AZURE_CLIENT_ID = $APP_ID"The subject must equal the GitHub environment name exactly. A mismatch fails with
AADSTS70021: No matching federated identity record found, which names neither side.
Then two roles. The first lets it fetch a kubeconfig; the second decides what it may then do:
CLUSTER_ID=$(az aks show -g "$RG" -n "$CLUSTER" --query id -o tsv)
az role assignment create --assignee "$APP_ID" \
--role "Azure Kubernetes Service Cluster User Role" --scope "$CLUSTER_ID"
az role assignment create --assignee "$APP_ID" \
--role "$CI_ROLE" --scope "${CLUSTER_ID}${CI_SCOPE_SUFFIX}"CI_ROLE and CI_SCOPE_SUFFIX come from the variables block, and differ by environment:
| Environment | CI_ROLE |
CI_SCOPE_SUFFIX |
|---|---|---|
| production | RBAC Writer |
(cluster-wide) |
| staging | RBAC Writer |
/namespaces/staging |
| dev | RBAC Cluster Admin |
(cluster-wide) |
Staging's namespace scope is what keeps a staging deploy out of production — it is the whole reason staging can share production's cluster.
Dev needs Cluster Admin because previews create and delete pr-N namespaces, and neither
RBAC Writer nor RBAC Admin can do that: Writer does not list namespaces among its actions,
and Admin explicitly excludes namespaces/write and namespaces/delete. That is tolerable
only because dev is a cluster of its own, containing nothing but previews.
The same limitation means a namespace-scoped identity cannot recreate its own namespace. If
stagingis ever deleted, CI cannot bring it back — recreate it with an admin credential first.
Dev needs a second federated credential, for teardown.
Required reviewers apply to every job that declares an environment, not to deployments as
such. Teardown needs cluster credentials too — deleting a pr-N namespace, stopping the
cluster overnight — so it must declare an environment as well. Sharing dev would leave every
cleanup job waiting for an approval nobody thinks to give: previews would never be removed and
the cluster would never sleep.
So there are two GitHub environments — dev with required reviewers, dev-cleanup without —
and the credential's subject names the environment, so each needs its own:
az ad app federated-credential create --id "$APP_ID" --parameters "{
\"name\": \"dev-cleanup\",
\"issuer\": \"https://token.actions.githubusercontent.com\",
\"subject\": \"repo:aiidalab/aiidalab-demo-server:environment:dev-cleanup\",
\"audiences\": [\"api://AzureADTokenExchange\"]
}"One identity, two credentials. There is no second app registration and no second set of role assignments: deleting a namespace requires Cluster Admin, so a teardown identity could not be given narrower rights anyway. What the split buys is a different approval gate, not less privilege.
Dev also needs to power the cluster (dev only — nothing else sleeps). The preview job
wakes it and the sweeper puts it back, but start and stop are Azure control-plane
actions and none of the AKS RBAC roles grant them: those govern the Kubernetes API, not
the resource holding it. Cluster User and RBAC Cluster Admin above are both silent on
Microsoft.ContainerService/managedClusters/start|stop/action, so without this the preview
fails at "Wake the cluster" with AuthorizationFailed — after logging in successfully, which
makes it look like something other than a missing role.
Contributor on the cluster would cover it, and a great deal else. Define a custom role with exactly those actions instead:
cat > /tmp/aks-power.json <<EOF
{
"Name": "AKS Cluster Power",
"Description": "Start and stop a managed cluster, nothing else.",
"Actions": [
"Microsoft.ContainerService/managedClusters/read",
"Microsoft.ContainerService/managedClusters/start/action",
"Microsoft.ContainerService/managedClusters/stop/action"
],
"NotActions": [],
"AssignableScopes": ["$(az group show -n "$RG" --query id -o tsv)"]
}
EOF
az role definition create --role-definition /tmp/aks-power.json
az role assignment create --assignee "$APP_ID" \
--role "AKS Cluster Power" --scope "$CLUSTER_ID"Creating a role definition needs Owner or User Access Administrator on the subscription; the AKS roles above will not do it. A new definition takes a few minutes to become assignable and the assignment a few more to propagate, so a preview re-run straight afterwards can still return 403.
One assignment covers cleanup too — dev and dev-cleanup are the same application, so the
sweeper's stop needs nothing further.
Finally, so the cluster can adopt the reserved ingress address:
CLUSTER_IDENTITY=$(az aks show -g "$RG" -n "$CLUSTER" --query identity.principalId -o tsv)
az role assignment create --assignee "$CLUSTER_IDENTITY" \
--role "Network Contributor" \
--scope "$(az group show -n "$NETWORKING_RG" --query id -o tsv)"For dev the address and the record already exist — aiidalab-networking holds
dev-ingress, and *.demo.aiidalab.xyz points at it. Skip the two create commands, but
still run the INGRESS_IP= line: step 7 needs it, and reading it back from Azure beats
copying a value that changes whenever the environment is rebuilt.
Reserve the address before the cluster needs it, in a resource group that is not the
cluster's. A public IP created by AKS lives in its MC_* group and is destroyed with the
cluster, which leaves the DNS record pointing at an address someone else can claim.
az group create --name "$NETWORKING_RG" --location "$LOCATION" --output none
az network public-ip create \
--resource-group "$NETWORKING_RG" --name "${ENV}-ingress" \
--location "$LOCATION" --sku Standard --allocation-method Static \
--output none
INGRESS_IP=$(az network public-ip show -g "$NETWORKING_RG" -n "${ENV}-ingress" --query ipAddress -o tsv)Point DNS at it. Dev's DNS_RECORD is a wildcard, so every pr-N preview resolves without a
new record; production names a single host:
az network dns record-set a add-record \
--resource-group dns-zones --zone-name "$DNS_ZONE" \
--record-set-name "$DNS_RECORD" --ipv4-address "$INGRESS_IP" --ttl 300For the cluster's load balancer to adopt an IP from another resource group, the service must
carry service.beta.kubernetes.io/azure-load-balancer-resource-group: $NETWORKING_RG, and the
cluster identity needs Network Contributor on that group. Without the annotation AKS
silently allocates a different IP and the DNS record quietly points nowhere.
Skip this entirely for production and staging. They each serve a single host through the chart's own
proxy.https, which handles its own certificate. Only dev needs this, because many preview hostnames share one address.
Wildcard DNS resolves every pr-N name to the same IP, so a per-namespace LoadBalancer
cannot work: they would all contend for one address. Instead one ingress controller owns the
address, and each preview gets an Ingress that routes by Host.
The controller, claiming the reserved IP:
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm upgrade --install ingress-nginx ingress-nginx/ingress-nginx \
--namespace ingress-nginx --create-namespace \
--set controller.service.loadBalancerIP="$INGRESS_IP" \
--set controller.service.annotations."service\.beta\.kubernetes\.io/azure-load-balancer-resource-group"="$NETWORKING_RG" \
--set controller.service.annotations."service\.beta\.kubernetes\.io/azure-load-balancer-health-probe-request-path"="/healthz" \
--set controller.extraArgs.default-ssl-certificate=ingress-nginx/wildcard-tlsThe health-probe annotation is not optional on AKS. ingress-nginx sets appProtocol on its
service ports, so the Azure cloud provider builds HTTP and HTTPS probes rather than TCP ones —
and probes /, where nginx answers 404 when no Ingress matches. Azure needs a 200, marks
the node unhealthy, and every connection to the load balancer simply times out. Nothing in the
cluster looks wrong: the pod runs, the service has its external IP, the certificate is valid.
/healthz answers 200.
(--set controller.service.appProtocol=false also works, by falling back to TCP probes. A
/healthz check is worth more than "the port is open".)
Without the resource-group annotation AKS looks for the IP in its own node resource group, does not find it, and silently allocates a different one — the DNS record then points nowhere.
default-ssl-certificate is what lets one wildcard serve every preview. Without it each
pr-N namespace would need its own copy of the TLS secret, which means another component to
replicate it.
cert-manager, and the wildcard itself. DNS-01 is required: HTTP-01 cannot prove a wildcard.
cert-manager runs as a pod, and a pod cannot use the cluster's own identity. Give it a
user-assigned identity federated to its service account — this is what
--enable-workload-identity above was for:
az identity create -g "$RG" -n cert-manager --location "$LOCATION" --output none
CM_CLIENT_ID=$(az identity show -g "$RG" -n cert-manager --query clientId -o tsv)
CM_PRINCIPAL=$(az identity show -g "$RG" -n cert-manager --query principalId -o tsv)
OIDC=$(az aks show -g "$RG" -n "$CLUSTER" --query oidcIssuerProfile.issuerUrl -o tsv)
az identity federated-credential create \
--identity-name cert-manager --resource-group "$RG" --name cert-manager \
--issuer "$OIDC" \
--subject "system:serviceaccount:cert-manager:cert-manager" \
--audiences api://AzureADTokenExchange --output none
az role assignment create --assignee "$CM_PRINCIPAL" --role "DNS Zone Contributor" \
--scope "$(az network dns zone show -g dns-zones -n "$DNS_ZONE" --query id -o tsv)"Install cert-manager itself. jetstack is simply where the chart is published — cert-manager
began life there and the repository kept the name.
helm repo add jetstack https://charts.jetstack.io
helm upgrade --install cert-manager jetstack/cert-manager \
--namespace cert-manager --create-namespace \
--set crds.enabled=true \
--set-string podLabels."azure\.workload\.identity/use"=true \
--set-string serviceAccount.labels."azure\.workload\.identity/use"=true \
--set-string serviceAccount.annotations."azure\.workload\.identity/client-id"="$CM_CLIENT_ID"What those four flags do:
crds.enabled=trueinstalls the custom resource definitions —ClusterIssuer,Certificateand the rest. Without them the next block fails withthe server doesn't have a resource type "clusterissuer".- the two
azure.workload.identity/uselabels tell Azure's workload-identity webhook to inject a projected token into cert-manager's pod. Without them the pod has no way to prove who it is. - the
client-idannotation says which identity to ask for — the user-assigned identity created just above.
Labels and annotations use --set-string; only crds.enabled is a real boolean. A bare
--set …=true makes Helm emit true as a YAML boolean, and Kubernetes rejects the manifest:
cannot unmarshal bool into Go struct field ObjectMeta.metadata.labels of type string.
These flag names have moved between chart versions. If Helm rejects one, helm show values jetstack/cert-manager | grep -A3 serviceAccount shows what the installed version expects.
The issuer and the certificate. SUB and DNS_RG are the subscription and the zone's
resource group:
SUB=$(az account show --query id -o tsv)
kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: aiidalab@materialscloud.org
privateKeySecretRef:
name: letsencrypt-account-key
solvers:
- dns01:
azureDNS:
subscriptionID: ${SUB}
resourceGroupName: dns-zones
hostedZoneName: ${DNS_ZONE}
environment: AzurePublicCloud
managedIdentity:
clientID: ${CM_CLIENT_ID}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: wildcard
namespace: ingress-nginx
spec:
secretName: wildcard-tls
issuerRef:
name: letsencrypt
kind: ClusterIssuer
dnsNames:
- "*.demo.${DNS_ZONE}"
EOFsecretName: wildcard-tls in namespace ingress-nginx is what the controller's
default-ssl-certificate points at. Watch it with:
kubectl -n ingress-nginx get certificate wildcard -wThen restart the controller. It started before this secret existed, logged
Error loading custom default certificate … falling back to generated default, and will keep
serving a self-signed certificate until it reloads. A helm upgrade does not do it — the
annotations live on the Service, so the pod spec never changes:
kubectl -n ingress-nginx rollout restart deploy/ingress-nginx-controller
kubectl -n ingress-nginx rollout status deploy/ingress-nginx-controllerThe ordering is unavoidable: the certificate is issued into the controller's namespace, so the controller has to exist first.
Certificates renew at 60 days. A cluster asleep across that window issues a fresh one on the next wake, which adds a minute to the first preview after a quiet spell.
Finally, the chart's side. values-dev.yaml turns off proxy.https, so the per-PR host
reaches the ingress through jupyterhub.ingress, with the hostname supplied at deploy time
because values files cannot interpolate:
ENVIRONMENT=dev NAMESPACE=pr-42 RELEASE=pr-42 ./deploy.sh \
--set jupyterhub.ingress.hosts[0]=pr-42.demo.aiidalab.xyzvalues-dev.yaml also sets proxy.service.type: ClusterIP. Left at the chart's default,
every preview namespace would ask Azure for a public IP of its own — which is the thing the
shared controller exists to avoid.
The above setup in general is done once.
But make sure the Pre-requisites are done before proceeding, to have az command available.
The following steps are for administrators/maintainers of the cluster to configure in their local machines.
If you’re using the Azure CLI locally, install kubectl, a tool for accessing the Kubernetes API from the commandline:
You may need sudo to install the commands to /usr/local/bin.
az aks install-cliGet credentials from Azure for kubectl to work:
az aks get-credentials \
--name demo-server \
--resource-group aiidalab-demo-server-rg \
--output tableThis will update the ~/.kube/config file with the credentials for the Kubernetes cluster.
Now the nodes are ready to be used. You can check the status of the nodes with the following command:
kubectl get nodesHelm is a package manager for Kubernetes, and it is used to install JupyterHub.
curl https://raw.githubusercontent.com/helm/helm/HEAD/scripts/get-helm-3 | bashPolicy document templates are available at https://github.com/aiidalab/aiidalab-deployment-files.
Follow the instructions there to generate the policy documents for your deployment.
You can then deploy the generated documents in basehub/files/etc/jupyterhub/templates.
Once deployed, set include_policies: true under jupyterhub.hub.config.JupyterHub.template_vars
in that environment's values file. This gates both the /terms-of-use and /privacy-policy
routes and the links to them in the JupyterHub UI.
The generated documents are gitignored, so they must be present in the working tree at deploy time — they are not carried by the repo.
Everything that describes what a deployment looks like is committed to this repo. Only secrets and the Azure coordinates come from GitHub settings.
Set these up before deploying — the order is: create the cluster, set the secrets and variables for the environment, then deploy.
A deployment is always the base file plus exactly one environment file:
basehub/values-base.yaml shared by every environment
basehub/values-<environment>.yaml what makes this environment different
| Environment | File | Where it 1798 runs |
|---|---|---|
local |
values-local.yaml |
kind, on your machine |
dev |
values-dev.yaml |
per-pull-request previews |
staging |
values-staging.yaml |
staging namespace on the production cluster |
production |
values-production.yaml |
production namespace on the production cluster |
Helm merges the two. Dictionaries merge key by key, but lists are replaced whole,
so an environment that needs to change one entry of a list must restate that entire
list. This applies to proxy.https.hosts and to the singleuser volume mounts.
To change the title, the support address, resource limits, hostnames, the session lifetime, or which features are enabled, edit the relevant values file and open a pull request. The change is then reviewed, rendered in CI, and recorded in git.
A short list — everything else lives in the values files above. Each environment has exactly one secret; every other entry is an identifier, and the reasoning for that is below the tables.
Scope follows whether the value differs between environments.
Repository scope (Settings → Secrets and variables → Actions) — the only two that never vary:
| Name | Kind | Value |
|---|---|---|
AZURE_TENANT_ID |
variable | 56b508b4-59b6-40c5-9839-6c4498d10fb9 |
AZURE_SUBSCRIPTION_ID |
variable | a2b00ada-7fb9-4e4f-8648-03f83044441a |
They are repository-scoped because holding one copy beats holding three that can drift apart. GitHub resolves environment over repository, so an environment that ever needed a different subscription can set its own and it wins.
Everything else is environment scope (Settings → Environments → name). Expand the environment you are setting up:
production
| Name | Kind | Value |
|---|---|---|
OAUTH_CLIENT_SECRET |
secret | from the aiidalab-demo-production OAuth app |
OAUTH_CLIENT_ID |
variable | ec9145436c332e45df0a |
OAUTH_CALLBACK_URL |
variable | https://aiidalab-demo.materialscloud.io/hub/oauth_callback |
AZURE_CLIENT_ID |
variable | 930b3bc4-8b2b-4de3-99a7-972b2a2bf7c8 (aiidalab-demo-server-sp) |
AZURE_RESOURCE_GROUP |
variable | aiidalab-demo-server |
AZURE_KUBERNETES_CLUSTER |
variable | demo-server-production |
staging
Staging runs in its own namespace on the production cluster, so the resource group and
cluster are production's. Its identity must therefore be scoped to the staging namespace
rather than the cluster, or a staging deploy could reach production.
| Name | Kind | Value |
|---|---|---|
OAUTH_CLIENT_SECRET |
secret | from the aiidalab-demo-staging OAuth app |
OAUTH_CLIENT_ID |
variable | from the same app |
OAUTH_CALLBACK_URL |
variable | https://staging-demo.aiidalab.io/hub/oauth_callback |
AZURE_CLIENT_ID |
variable | 885f29bd-b0de-4640-8593-32df45a2eb59 (aiidalab-demo-staging-sp) |
AZURE_RESOURCE_GROUP |
variable | aiidalab-demo-server |
AZURE_KUBERNETES_CLUSTER |
variable | demo-server-production |
dev (per-pull-request previews)
Previews use DummyAuthenticator, so there are no OAuth settings at all: a GitHub OAuth app
has one callback URL and no wildcards, so a per-PR hostname could never complet
C36A
e a login.
The OAuth path is exercised on staging instead.
Protect this environment with required reviewers — that approval is what gates each push to a labelled pull request.
| Name | Kind | Value |
|---|---|---|
DUMMY_AUTH_PASSWORD |
secret | choose one — these URLs are public and a login costs a pod, so not demo |
AZURE_CLIENT_ID |
variable | 9b36e574-9818-4699-af04-88e7c9c8508e (aiidalab-demo-dev-sp) |
AZURE_RESOURCE_GROUP |
variable | aiidalab-demo-dev |
AZURE_KUBERNETES_CLUSTER |
variable | aiidalab-demo-dev |
dev-cleanup (teardown, no reviewers)
Required reviewers apply to every job declaring an environment, not to deployments as such.
A teardown job sharing dev would wait for an approval nobody gives, so previews would never
be removed and the cluster would never sleep. Hence a second environment, with no required
reviewers — but a branch policy restricting it to main, so the destructive credential
cannot be reached from a pull request's own workflow files. Cleanup therefore runs from
main, on a schedule; a pull_request-triggered job could not use this environment at all,
because that event's ref is refs/pull/N/merge.
It is the same identity — one app registration with a second federated credential for
…:environment:dev-cleanup. Deleting a namespace requires Cluster Admin, so a teardown
identity could not hold narrower rights anyway; the split buys a different approval gate, not
less privilege.
| Name | Kind | Value |
|---|---|---|
AZURE_CLIENT_ID |
variable | 9b36e574-9818-4699-af04-88e7c9c8508e — same as dev |
AZURE_RESOURCE_GROUP |
variable | aiidalab-demo-dev |
AZURE_KUBERNETES_CLUSTER |
variable | aiidalab-demo-dev |
No DUMMY_AUTH_PASSWORD: teardown deploys nothing.
AKS Cluster Power role to wake the cluster and to
put it back to sleep — see 5. The CI identity. Without it a preview
fails at "Wake the cluster" with AuthorizationFailed whenever the cluster is stopped, which
is its normal state, and someone has to run az aks start by hand before approving the
deployment. One assignment serves both, since they are the same application.
The rest are identifiers, not credentials. An OAuth client ID is visible in the browser's address bar during login; a callback URL is a public address; and the Azure IDs name a tenant, a subscription and an app registration. None of them grants access on its own — CI authenticates to Azure over OIDC, where the trust comes from a federated credential that names this repository and environment, not from knowing an ID.
Storing them as secrets is not more secure, and it costs something real: GitHub will not show you a secret's value again after you save it, so a wrong one can only be replaced, never checked. That is worth paying for a credential, and not worth paying for a GUID.
Set the environment-scoped ones at Settings → Environments → production / staging, and
the two repository-scoped ones at Settings → Secrets and variables → Actions.
Do not put anything else at repository scope. A repository variable applies to every environment at once, which is how a value meant for production silently reaches staging — and how the configuration came to be spread across two pages in the first place.
The OAuth apps live in the aiidalab organisation, named aiidalab-demo-production and
aiidalab-demo-staging. OAUTH_CALLBACK_URL must name the canonical host for that
environment — the first entry of proxy.https.hosts in its values file — because the
oauth_state cookie is host-scoped, and a login begun on a different host comes back
without it and JupyterHub answers 400: Bad Request.
OAUTH_CLIENT_SECRET is passed to Helm on the command line by deploy.sh and is never
written to a file.
The cluster uses Azure RBAC for Kubernetes Authorization, so what a CI identity may do in the cluster is set by Azure role assignments, not Kubernetes RoleBindings. Each identity needs two roles: one to fetch a kubeconfig, and one to act inside the cluster.
| Identity | Role | Scope |
|---|---|---|
production aiidalab-demo-server-sp |
Azure Kubernetes Service Cluster User Role |
cluster |
Azure Kubernetes Service RBAC Admin |
cluster | |
staging aiidalab-demo-staging-sp |
Azure Kubernetes Service Cluster User Role |
cluster |
Azure Kubernetes Service RBAC Admin |
namespaces/staging |
Use RBAC Admin, not RBAC Writer. Writer has no rights on Roles or RoleBindings, and
the chart creates both (for the hub, the TLS proxy and the image pre-puller), so
helm upgrade fails with cannot get resource "roles" or "clusterroles".
Staging's Admin role stops at its namespace, so staging cannot manage the user-scheduler's
ClusterRole. That is why values-staging.yaml disables the user-scheduler. Do not work around
it by granting cluster-scoped rights: an identity that can create ClusterRoleBindings can make
itself cluster-admin and reach production.
CLUSTER=$(az aks show -g aiidalab-demo-server -n demo-server-production --query id -o tsv)
az role assignment create --role "Azure Kubernetes Service RBAC Admin" \
--assignee 930b3bc4-8b2b-4de3-99a7-972b2a2bf7c8 --scope "$CLUSTER"
az role assignment create --role "Azure Kubernetes Service RBAC Admin" \
--assignee 885f29bd-b0de-4640-8593-32df45a2eb59 --scope "$CLUSTER/namespaces/staging"Admins get kubectl access through the AiiDAlab Admins group, which holds
Azure Kubernetes Service RBAC Cluster Admin on the cluster. Subscription Owner does not
grant kubectl access on its own.
Normally you do not run this by hand. Pushing to main deploys staging, and publishing a
release deploys production — see
Releasing to production. The script below is what CI runs, and
what you use for a local cluster or a one-off.
ENVIRONMENT=staging ./deploy.shENVIRONMENT picks the values file, and the namespace and Helm release name default
to it. Override them if needed — pull-request previews use NAMESPACE=pr-42:
ENVIRONMENT=dev NAMESPACE=pr-42 RELEASE=pr-42 ./deploy.shThe namespace is created if it does not exist. In CI this same script runs from
.github/workflows/deploy-to-aks.yml, with the secrets above supplied from the
GitHub environment.
The external IP of the proxy can be retrieved with:
kubectl get svc proxy-public -n <namespace>| Trigger | Deploys | Gate |
|---|---|---|
push to main |
staging | review on the pull request |
| a published GitHub release | production | tag ruleset + required reviewers on the environment |
workflow_dispatch |
staging only | re-runs a staging deploy without an empty commit |
All three run .github/workflows/deploy-to-aks.yml, which authenticates to Azure with
OpenID Connect — no stored Azure credential — and then runs the same ./deploy.sh you
would run by hand.
Each environment has its own app registration, so a staging deploy cannot use production's
identity. The federated credential's subject names the environment exactly
(repo:aiidalab/aiidalab-demo-server:environment:staging), so a workflow that does not
declare that environment cannot obtain a token at all.
Production deploys from published releases, not from a branch.
- Check staging. It is whatever is on
main, which is what you are about to release. - On GitHub, Releases → Draft a new release.
- Create a tag of the form
vYYYY.MM.DD, for examplev2026.06.01, targetingmain. For a second release on the same day, append a counter:v2026.06.01.1, thenv2026.06.01.2. - Generate release notes — this becomes the record of what changed in production.
- Publish. That triggers the deploy; approve it if required reviewers are configured.
Use a dot before the counter, never a hyphen: in SemVer a hyphen means pre-release, and would sort the second release of the day before the first.
Marking a release as a pre-release deliberately does nothing — it will not deploy. Useful for drafting.
Before deploying, the workflow checks that:
- the tag matches
vYYYY.MM.DD[.n], with real month and day ranges; - the tagged commit is an ancestor of
main, so a tag on an unreviewed commit cannot deploy; - the release is not a pre-release.
It then checks out the tag, not a branch.
The
productionenvironment's deployment branch policy must permit tags. A branches-only policy rejects a release deploy before any Azure step runs, with an error that looks unrelated to tags.
Cut a new release from the last good commit. Revert the offending change on main, or
tag the previous good commit, and publish a release for it — v2026.06.02 after a bad
v2026.06.01.
That is deliberately the only route. Nothing can deploy production except a published release, so "what is running" and "the latest release" never drift apart. A mechanism for deploying an older tag directly would break that: the newest release would no longer describe production, and nothing would say so.
For a faster escape hatch that skips CI entirely:
helm -n production rollback productionThat reverts to the previous Helm revision within seconds. It does break the invariant above — the repo now disagrees with the cluster — so treat it as first aid and follow it with a real release once the cause is understood.
JupyterHub uses Let’s Encrypt to automatically create HTTPS certificates for your deployment.
Specify the two bits of information that we need to automatically provision HTTPS certificates - your domain name & a contact email address.
proxy:
https:
enabled: true
hosts:
- <your-domain-name>
letsencrypt:
contactEmail: <your-email-address>The same deploy.sh and the same values files as staging and production, pointed at a
throwaway kind cluster. Four commands, no extra tooling.
- Docker, running
kind,kubectl,helm
kind create cluster --name aiidalab-demo-server-local --config kind-config.yaml
ENVIRONMENT=local NAMESPACE=local ./deploy.sh
kubectl -n local rollout status deploy/hub deploy/proxyThen open http://localhost:8000 and log in with any username and the password demo.
No port-forwarding is needed: kind-config.yaml maps container port 32080 to host port
8000, and values-local.yaml puts the proxy on that NodePort.
No secrets are needed either. values-local.yaml uses DummyAuthenticator, so there is no
GitHub OAuth app to register. The ~2GB singleuser image is pulled lazily, so the hub comes
up in a couple of minutes but the first login is slow.
deploy.sh refuses to deploy ENVIRONMENT=local unless the current kubectl context is a
kind cluster. This is deliberate: a context left pointing at a real cluster once put a
local release into production. Override with ALLOW_ANY_CONTEXT=true if you mean it.
Anything under basehub/files/ is mounted from a ConfigMap, and ConfigMaps do not
hot-reload. After editing a template, a stylesheet or a script:
ENVIRONMENT=local NAMESPACE=local ./deploy.sh
kubectl -n local rollout restart deploy/hub deploy/proxyChanging a values file only needs the first command.
kind delete cluster --name aiidalab-demo-server-local