Problem

Use sccache with an S3 cache and IRSA. After the sccache server on a compiler replica stops, all Rust builds on that replica fail. Each build fails in 10 to 30 s with:

sccache: error: Timed out waiting for server startup. Maybe the remote service is unreachable?

The compiler reports Rust compilation failed due to an sccache problem. The only workaround that users find is to turn off sccache.

Sequence

Root cause

cargo_command_with_minimal_env() clears the environment. It keeps PATH, RUSTC_WRAPPER, CARGO_INCREMENTAL, AWS_PROFILE and SCCACHE_*. It removes the variables that IRSA needs:

• AWS_ROLE_ARN
• AWS_WEB_IDENTITY_TOKEN_FILE
• AWS_REGION
• AWS_STS_REGIONAL_ENDPOINTS

The image writes role_arn and web_identity_token_file into the [profile sccache] AWS profile. The compiler forwards AWS_PROFILE=sccache. #7102 chose this profile instead of the variables. sccache 0.18 uses reqsign-aws-core 3.1.0, which does not read these profile keys:

Older sccache versions (reqsign 0.16) read role_arn from the profile. But they used the global sts.amazonaws.com endpoint, because the compiler removes AWS_STS_REGIONAL_ENDPOINTS. On a cluster with no route to the global endpoint, the result was the same timeout.

Reproduction

This uses the public.ecr.aws/feldera/compiler-server:0.381.0 image. The S3 and STS host names point to 127.0.0.1, so no request goes out of the machine.

docker run --rm --network bridge \
  --add-host sts.amazonaws.com:127.0.0.1 --add-host sts.us-east-1.amazonaws.com:127.0.0.1 \
  --add-host s3.amazonaws.com:127.0.0.1 --add-host s3.us-east-1.amazonaws.com:127.0.0.1 \
  --add-host repro-bucket.s3.amazonaws.com:127.0.0.1 --add-host repro-bucket.s3.us-east-1.amazonaws.com:127.0.0.1 \
  --entrypoint bash -v "$PWD/repro.sh:/repro.sh:ro" \
  public.ecr.aws/feldera/compiler-server:0.381.0 /repro.sh

#!/bin/bash
# Starts sccache 0.18 two times in the compiler-server image:
#   A: with the environment that cargo_command_with_minimal_env() gives cargo
#   B: with A plus the variables that the EKS IRSA webhook puts in the pod
set -u
cd "$HOME"
tar -xzf dot-cargo.tar.gz -C / home/ubuntu/.cargo/bin/sccache
SCCACHE="$HOME/.cargo/bin/sccache"

# The same files that the Helm chart and the image entrypoint write.
mkdir -p "$HOME/.config/sccache" "$HOME/.aws"
cat > "$HOME/.config/sccache/config" <<'EOF'
[cache.s3]
bucket = "repro-bucket"
region = "us-east-1"
use_ssl = true
key_prefix = "sccache"
no_credentials = false
EOF
echo "dummy.jwt.token" > /tmp/token
cat > "$HOME/.aws/config" <<'EOF'
[profile sccache]
region = us-east-1
role_arn = arn:aws:iam::111122223333:role/repro
web_identity_token_file = /tmp/token
EOF

run() {
  local name="$1"; shift
  echo; echo "=== $name"
  "$SCCACHE" --stop-server >/dev/null 2>&1
  local start=$(date +%s)
  env -i PATH="$HOME/.cargo/bin:/usr/bin:/bin" RUSTC_WRAPPER=sccache CARGO_INCREMENTAL=0 \
    AWS_PROFILE=sccache SCCACHE_LOG=debug SCCACHE_ERROR_LOG="/tmp/$name.log" "$@" \
    "$SCCACHE" --start-server 2>&1 | grep '^sccache: error'
  echo "took $(( $(date +%s) - start )) s"
  grep -E 'No credential found in provider: AssumeRoleWithWebIdentity|starting new connection' \
    "/tmp/$name.log" | sed -E 's/^\[[^]]*\] //' | cut -c1-110
}

run A_compiler_env
run B_with_irsa_env AWS_ROLE_ARN=arn:aws:iam::111122223333:role/repro \
  AWS_WEB_IDENTITY_TOKEN_FILE=/tmp/token AWS_REGION=us-east-1 AWS_STS_REGIONAL_ENDPOINTS=regional

=== A_compiler_env
sccache: error: Timed out waiting for server startup
took 10 s
No credential found in provider: AssumeRoleWithWebIdentityCredentialProvider { role_arn: None, role_session_na
starting new connection 'Some("169.254.169.254")'

=== B_with_irsa_env
sccache: error: Server startup failed: cache storage failed to read: Unexpected (permanent) at read => signing http request
took 0 s
starting new connection 'Some("sts.us-east-1.amazonaws.com")'
starting new connection 'Some("169.254.169.254")'

Proposed fix

1. In cargo_command_with_minimal_env(), forward AWS_ROLE_ARN, AWS_WEB_IDENTITY_TOKEN_FILE, AWS_REGION and AWS_STS_REGIONAL_ENDPOINTS when they are set.
2. Do not forward AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY or AWS_SESSION_TOKEN. Static keys continue to come from the profile's credentials file.
3. Decide if the fix also forwards the EKS Pod Identity variables (AWS_CONTAINER_CREDENTIALS_FULL_URI, AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE). The compiler removes them for the same reason.

Security: the review of #7102 said that UDF build scripts can use these variables to assume the pod's role. Build scripts run as the same user as the compiler. That user can already read ~/.aws/config, which holds the role ARN and the token path. It can also read the token file. Thus the variables give build scripts no new access. The control that works is a pod role that can only use the cache bucket. The docs must tell operators to do this.

Other approach: the compiler starts the sccache server itself, with its own environment, before each build. Then sccache never starts inside cargo. This is more code, and it does not change the security position above.

Done when

• A unit test shows that the four variables go to cargo when they are set, and are absent when they are not set.
• A unit test shows that AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN do not go to cargo.
• With the fix reverted, the first test fails.
• On an IRSA cluster: run sccache --stop-server in a compiler pod, then compile a program. The build succeeds.
• The IRSA section of parallel-compilation.md tells operators to give the pod role access to the cache bucket only.

Related

• Forward the AWS web identity variables to cargo #7102 (closed): forwarded these variables for cargo registries, then changed to the profile approach.
• compiler: report an unreachable sccache server in cluster health #7402 (open): health check for a missing sccache server.
• Don't clear the full environment for compiler server invocation #3825 (open): clear less of the environment.
• sccache UX #6763 (open): sccache UX, which includes a fallback to plain rustc.