GeezerVison

Problem

• Our Helm charts do not yet support runtimeClassName, so they cannot run under a sandboxed runtime such as gVisor, Kata Containers, or EKS Nitro Sandboxes

• The k8s executor chart has no value for EXECUTOR_KUBERNETES_RUNTIME_CLASS, so sandboxing job pods means repeating the whole executor.extraEnv list

Solution

• sourcegraph.runtimeClassName in all four charts, with a per-service <serviceName>.runtimeClassName override, modelled on priorityClassName

• Unlike priorityClassName, an empty <serviceName>.runtimeClassName opts that service out of the global value

  • cadvisor (privileged, host paths) and node-exporter (hostPID, host paths) need the host, so a global sandbox runtime needs a way to skip them

  • The charts do not opt any service out by default, since which pods a RuntimeClass can run depends on the runtime

• executor.kubernetesJob.runtimeClassName in the k8s executor chart sets EXECUTOR_KUBERNETES_RUNTIME_CLASS, which the executor has supported since 7.3.0

  • Job pods do not inherit sourcegraph.runtimeClassName, like the other kubernetesJob.* scheduling values

• In the dind chart, dind.gVisor.enabled still defaults executor pods to runtimeClassName: gvisor; executor.runtimeClassName or sourcegraph.runtimeClassName now replaces it, for clusters where the gVisor RuntimeClass has another name

• Example overrides for EKS Nitro Sandboxes, in examples/aws-eks-nitro-sandboxes/ of each of the four charts

  • override.yaml for isolated-sandbox

  • override-confidential.yaml for confidential-sandbox, used on top of override.yaml. It shows where each pod's policy goes, and lists the steps and policy edits that were needed; the policies themselves must be generated per deployment

Chart tests

• New tests/runtimeClassName_test.yaml asserts on all 21 sourcegraph workloads: nothing renders by default, and the global value renders on every one. It also covers the test connection pod, a service override, and the empty-value opt-out. Removing the line from one template fails the suite

• Tests added to the migrator, k8s executor, and dind executor suites for the override, the global fallback, the opt-out, the job pod ConfigMap value, and the gVisor default and its override

• helm unittest on all four charts: 43 suites, 267 tests pass. scripts/ci/lint.sh passes. ./scripts/helm-docs.sh leaves no diff

• helm template with default values is byte-identical to origin/main for sourcegraph and the dind executor, with and without dind.gVisor.enabled. The k8s executor differs only by the new EXECUTOR_KUBERNETES_RUNTIME_CLASS: "" ConfigMap entry, which the executor treats as unset

• kubeconform -strict passes on all four charts rendered with sourcegraph.runtimeClassName set

• The example overrides render with helm template

  • override.yaml: every workload gets isolated-sandbox and the network annotations, and cadvisor and node-exporter are not rendered

  • With override-confidential.yaml on top: every workload gets confidential-sandbox and its own policy annotation, and the job pod annotations parse as JSON

EKS Nitro Sandboxes tests

Every pod of all four charts ran in both sandbox RuntimeClasses, on an EKS Auto Mode cluster (Kubernetes 1.37):

Two test instances:

• Instance 1: the sourcegraph and k8s executor charts, with external PostgreSQL, Redis, and S3. isolated-sandbox only

• Instance 2: all four charts, with the bundled databases, Redis, and blobstore, plus OpenTelemetry, Jaeger, and syntactic-code-intel. Both RuntimeClasses

The cluster needed no change: Auto Mode added sandbox nodes on its own, and no NodePool, StorageClass, or PVC change was needed.

The sections below give, for each RuntimeClass and each chart, what to set and what was tested. The example overrides hold the same values.

isolated-sandbox

Every chart

To deploy:

1. Set sourcegraph.runtimeClassName: isolated-sandbox

2. Raise the network limits. A sandbox defaults to 100 Mbps and 1,500 packets per second each way, read once at pod creation

   • sourcegraph.podAnnotations with the four sandbox.amazonaws.com/{ingress,egress}-{bandwidth,packets-per-second} annotations covers every chart pod

3. Size pods by their requests, because the requests size the microVM and the limits do not

   • Memory is the sum of the containers' memory requests. A container is OOM-killed once the pod uses more than that, even below its limit, so set each memory request to its limit

   • vCPUs are the sum of the containers' CPU requests, rounded up, with a minimum of 1. CPU limits add none: a 100m request gets 1 vCPU whatever its limit

   • Each pod adds 256Mi of memory overhead, which counts against a ResourceQuota

sourcegraph chart

To deploy:

• Disable cadvisor and node-exporter with enabled: false

  • In a sandbox, cadvisor fails to start with open /var/lib/docker: no such file or directory

  • In a sandbox, node-exporter runs, but reports its own microVM (1 CPU, 240 MB) instead of the node (4 CPUs, 16 GB)

  • On the default runtime (runtimeClassName: "") both run, but cadvisor does not see containers that run in sandboxes. On a node with 19 sandbox pods, it reported only itself and node-exporter, so the container metrics on Sourcegraph's dashboards are empty for sandboxed pods. kubectl top and the kubelet's /stats/summary still report them

• With OpenTelemetry enabled, set openTelemetry.nodeSelector to eks.amazonaws.com/nitro-sandbox: "true"

  • The RuntimeClass adds that selector to each pod at admission, which is too late for the DaemonSet controller. Without it, otel-agent gets a pod that stays Pending on every node that has no sandbox support

• With the bundled databases, remove the init container's CPU limit with alpine.resources.limits.cpu: null. With the default 10m limit, the first database init took over 13 minutes

  • pgsql and codeintel-db set shared_buffers to 1GB, so their memory requests must stay above that

Tested:

• Pods

  • Instance 1: all 9 pods are Ready with no restarts on Nitro Sandbox nodes: frontend, worker, gitserver, indexed-search, searcher, syntect-server, precise-code-intel-worker, prometheus, and grafana. Each reports its own guest kernel and an .enclave product name

  • Instance 2: every pod runs in a sandbox, including the bundled databases, Redis, blobstore, otel-agent, the collector, Jaeger, and syntactic-code-intel-worker

• Search: indexed, unindexed, symbol, commit, and diff searches return results, and so does syntax highlighting

• Migrations: the frontend's migrator init container completed. helm test passed

• Code navigation: definitions, references, and hover resolve from precise uploads. Syntactic indexing uploads completed

• Code Insights backfilled a series

• Monitoring: traces from 8 services reach Jaeger through otel-agent and the collector. Prometheus scrapes every target except Jaeger's /metrics, which returns 404, and Grafana serves its dashboards

• Storage: EBS volumes mount with the chart's ReadWriteOnce PVCs, including indexed-search's two containers on one volume. After deleting all 11 pods that have an EBS volume, they moved to other sandbox nodes with their data intact

• Service account tokens work with no extra config: IRSA credentials (frontend to S3 and RDS, Grafana's RDS token sidecar) and the Kubernetes API token

Migrator chart

• To deploy: nothing beyond the steps for every chart

• Tested: the job completed with the chart's default resources

k8s executor chart

To deploy:

• Set executor.kubernetesJob.runtimeClassName: isolated-sandbox for job pods

• Give job pods the network annotations as JSON in KUBERNETES_JOB_POD_ANNOTATIONS, through executor.extraEnv; the chart has no value for it

• With executor.frontendExistingSecret, name the Secret executor-<queueName>, or executor- and the queueNames joined with -: the Deployment reads that name, whatever the value is set to

Tested:

• The executor pod is Ready, and creates Jobs with its Kubernetes API token

• Batch Changes: a 3-repository server-side batch spec (instance 1) and a 2-repository one (instance 2) completed, one sandboxed job pod per repository, with batchChanges.nativeServerSideExecution set in site configuration. Each diff records the job's guest kernel

• Auto-indexing: scip-go jobs completed in sandboxed job pods on both instances, and precise-code-intel-worker processed their uploads

dind executor chart

To deploy:

• Size executor.resources.requests for the jobs too (tested with 2 CPUs and 4G). Jobs run as containers inside the executor pod's microVM, and the chart has no resources value for the dind container

Tested:

• The privileged dind container runs in a sandbox

• Auto-indexing: a scip-go job completed

• Batch Changes: a 2-repository batch spec completed

confidential-sandbox

Everything under isolated-sandbox applies, with the other RuntimeClass name. On top of that, every pod needs a policy.

Every chart

To deploy:

1. Generate a Cedar policy for every pod, with eks-nitro-sandbox-genpolicy. Confidential mode denies everything a policy does not allow, and rejects a pod with no sandbox.amazonaws.com/policy annotation

   • Render the chart, and generate one policy per workload from its manifest: eks-nitro-sandbox-genpolicy --profile confidential --allow-additional-env --output-file-b64 policy.b64 manifest.yaml

   • --allow-additional-env is needed because the charts have no enableServiceLinks value, so the kubelet adds Service env vars that the manifest does not list

   • The generator rejects the charts' octal modes (defaultMode: 0777). Piping the manifest through kubectl create --dry-run=client -o yaml -f - turns them into decimals

   • The generator has no permit for a StatefulSet's volumeClaimTemplates. Add each claim to the manifest's pod volumes as a persistentVolumeClaim volume before generating

   • The generator is a Linux binary, and reads each image from its registry. Docker Hub's anonymous limit of 100 requests per hour ran out part way through the sourcegraph chart

2. Edit the policies that pin a value the sandbox does not present; each chart's section lists them. Decode with base64 -d policy.b64 | gunzip, edit the admission_policy and runtime_policy text, and encode with eks-nitro-sandbox-genpolicy --encode-envelope

3. Set each policy as the sandbox.amazonaws.com/policy annotation in that service's podAnnotations

4. Regenerate the policies on every chart upgrade, and on any values change that alters a pod. A policy pins each container's image digest, command, env, mounts, user, and groups

5. Allow 512Mi of memory overhead per pod, instead of 256Mi

6. Expect kubectl exec to be denied by the generated policy. kubectl logs works

7. Remove the policy annotation when moving a pod back to isolated-sandbox. It is enforced there too: a pod that kept its confidential policy was denied at SetupSandbox

sourcegraph chart

To deploy: the policies from eks-nitro-sandbox-genpolicy 0.1.4274.0 did not admit 15 of the 19 workloads. Each of these needed an edit:

Tested, with the edited policies:

• Pods: every pod became Ready, except 3 of 7 otel-agent pods, which stayed Pending with Insufficient memory on sandbox nodes that were full

• Search: indexed and unindexed searches return results, and so does syntax highlighting

• Code navigation: precise code navigation resolves

• Code Insights: a query returned results

• Not run: helm test. Its pod takes no pod annotations, so it cannot carry a policy

Migrator chart

• To deploy: widen the policy's hostname pin from migrator-<5 characters>-* to migrator-*, in both admission_policy and runtime_policy

  • The chart gives the Job a new random name on every render, and the generated policy pins that name as the pod's hostname. The pod from the next render was denied at SetupSandbox

  • The generator's --no-pin-hostname flag was not tried

• Tested: with the widened pin, the job completed

k8s executor chart

To deploy:

• The executor pod needs no policy edit

• For job pods, set executor.kubernetesJob.runtimeClassName: confidential-sandbox, and add the policy annotation to KUBERNETES_JOB_POD_ANNOTATIONS. Every job pod then gets the same policy

• Build the job policy by hand, because job pods have no manifest to generate from

  1. Save a job pod of each kind of job, and remove the fields the API server added

  2. Generate a policy from each, with --no-pin-args --no-pin-env --no-pin-hostname added, because those three differ on every job

  3. Merge the policies by joining their admission_policy and runtime_policy text. A pod is admitted when any one permit matches

• Generate the job policy before the executor's own policy, which pins the value of that env var

Tested:

• The executor pod became Ready and registered with its generated policy, unedited

• A policy made from a batch job denied an auto-indexing job. A job policy still pins the container names, image digests, working directories, and mounts, so it admits one kind of job

• With the batch and auto-indexing policies merged into one, a 2-repository batch spec and two scip-go jobs completed

• A batch spec with two steps was still denied, at CreateContainer for the second step. Each number of steps, step image, and indexer needs its own permits

dind executor chart

To deploy:

• Before generating the executor's policy, change emptyDir: to emptyDir: {} on the executor-scratch volume in the manifest. When executor.storage.size is unset, the chart renders it with no value, and the generator then expects a pod with no emptyDir volume. With the policy generated as is, the pod was denied at SetupSandbox

• Edit the executor pod's policy before deploying it. As generated, it does not admit the dind container: the pod stayed at 1/2 Ready, with policy denied Agent::Action::"CreateContainer" on container "dind"

  • Cause: the policy has one CreateContainer permit per container, under // Container: executor and // Container: dind. Each permit has a context.mounts == [...] condition, which lists every mount the container must have, with its source path. The dind container mounts daemon.json from the docker-config ConfigMap with subPath, and the source path the generator writes for that mount is not the one the sandbox presents. The jaeger pod in the sourcegraph chart fails the same way

  • Edit: in the admission_policy text, delete the whole context.mounts == [...] && condition from the // Container: dind permit. Leave the executor permit as it is: that container has no subPath mount, and was admitted with its condition in place

  • Result: the policy no longer checks the dind container's mounts. Its other conditions still apply

• private-docker-registry needs no policy edit

• Jobs need no policy: they run as containers inside the executor pod's microVM, not as pods

Tested:

• Both containers of the executor pod are Ready, including the privileged dind container, and so is private-docker-registry

• Auto-indexing: a scip-go job completed

• Batch Changes: a 2-repository batch spec completed

• The jobs ran with the context.mounts conditions of both containers removed. With only the dind container's removed, both containers were Ready; the jobs were not run again

Attestation

Tested with a test pod, not a chart pod: a container image encrypted with a KMS key, which KMS only decrypts for a sandbox that proves it runs the expected policy.

• Setup: a busybox image encrypted with skopeo and nitro-sandbox-keyprovider (encrypted images)

  • The pod's role, through EKS Pod Identity, has no IAM policy

  • The key policy allows kms:Decrypt only when kms:RecipientAttestation:PCR17 equals the value the generator prints for the pod's policy, and kms:RecipientAttestation:PCR8:ServiceName is eks.amazonaws.com

  • The pod's policy was generated with --with-pod-identity

• The pod started and read the files of the decrypted layer. CloudTrail's Decrypt event carries the sandbox's attestation document, and its PCR17 is the predicted value

• The same pod, with one comment line added to its policy, failed to start: KMS answered AccessDeniedException to kms:Decrypt

Cluster issues found

None of these come from the charts.

• A pod with an EBS volume can crash-loop with cannot take exclusive claim on device ... Device or resource busy. The cause was the cloudwatch-agent DaemonSet of the amazon-cloudwatch-observability EKS add-on

  • The agent mounts the node's / without mount propagation. When its pod starts after the volumes are mounted on the node, it keeps its own copies of those mounts, which keeps the devices busy

  • Deleting the agent pod on that node fixed all 11 crash-looping pods; the new agent pod held no EBS mounts. It did not happen again when the agent was already running before the volumes attached

  • The chart's cadvisor mounts / the same way. It was not seen to cause this

• Sandbox pods can get stuck Pending with Insufficient memory. kube-scheduler counts 256Mi more memory for each running sandbox pod than EKS Auto Mode does, so Auto Mode thinks a pending pod fits on an existing node, and does not add one. Two pods hit this after Auto Mode consolidated nodes. Not resolved

  • The same happens with sandbox slots: a node with 40 allocatable eks.amazonaws.com/nitro-sandbox refused new pods with 21 sandbox pods running

• Other limits are in the AWS considerations: no hostNetwork, 100 DNS queries per second and 4,096 outbound connections per pod by default, and growing a PVC or changing CPU or memory needs a pod restart

Not tested

• Other sandboxed runtimes, such as gVisor and Kata Containers

• In confidential-sandbox: helm test, container image signature verification, encrypted Sourcegraph images, and job policies for other indexers or batch specs with more than one step

Amp threads

• https://ampcode.com/threads/T-01a1c75c-2680-718f-950a-b474d79c9e64