GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — Forward the AWS web identity variables to cargo by xiasongh · Pull Request #7102 · feldera/feldera

Conversation

@xiasongh

@xiasongh xiasongh commented Sep 9, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

Add AWS web identity env vars to whitelist so compiler server can access AWS credentials and authenticate to private registries.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@xiasongh
Forward the AWS web identity variables to cargo
89f63ab
A cargo credential provider authenticating to a private registry needs the
AWS identity of the environment the compiler runs in, but cargo is spawned
with a cleared environment, so AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE
never reach it. Preserve them alongside AWS_PROFILE: a role ARN and a token
path name an identity without carrying one, and the token they point at is
readable by a build script with or without them.
@xiasongh
xiasongh force-pushed the cargo-credential-aws-env branch from 39a1a2c to 89f63ab Compare September 9, 2026 23:36
@xiasongh
xiasongh requested a review from gz September 9, 2026 23:37
mythical-fred-oss[bot]
mythical-fred-oss Bot reviewed Sep 9, 2026
View reviewed changes
Comment thread crates/pipeline-manager/src/compiler/rust_compiler.rs
Comment on lines +1698 to 1707
// Preserve the AWS variables that name an identity without carrying one:
// AWS_PROFILE for sccache, and the web identity pair for a cargo
// credential provider authenticating to a private registry. All other
// AWS_* vars are withheld to prevent leaking credentials to malicious
// build scripts.
for key in ["AWS_PROFILE", "AWS_ROLE_ARN", "AWS_WEB_IDENTITY_TOKEN_FILE"] {
if let Some(value) = std::env::var_os(key) {
command.env(key, value);
}
}

@mythical-fred-oss mythical-fred-oss Bot Sep 9, 2026

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The commit message argues these two are safe because "the token they point at is readable by a build script with or without them". That is only half of it: AssumeRoleWithWebIdentity needs both the token and the RoleArn, and the ARN is not in the JWT (the SA→role mapping lives in the IAM trust policy). So before this change a build script had a readable token it could not spend; after it, it has everything needed to mint credentials for the pod's role.

That matters here because the build scripts are not all ours — user-supplied UDF dependencies are spliced into the pipeline crate's Cargo.toml (udf_toml, ~L1475) and their build.rs runs in this process tree. That is exactly the leak the old comment was guarding against.

Not necessarily a blocker, but please either:

  • make it opt-in (config knob / has_unstable_feature) so existing IRSA deployments do not silently change security posture just because they run on EKS, or
  • state in the comment and in docs/get-started/enterprise/parallel-compilation.md that the compiler pod's role must be scoped to exactly what the registry/sccache needs, because arbitrary UDF build scripts can assume it.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
mythical-fred-oss[bot]
mythical-fred-oss Bot reviewed Sep 9, 2026
View reviewed changes
Comment thread crates/pipeline-manager/src/compiler/rust_compiler.rs
// credential provider authenticating to a private registry. All other
// AWS_* vars are withheld to prevent leaking credentials to malicious
// build scripts.
for key in ["AWS_PROFILE", "AWS_ROLE_ARN", "AWS_WEB_IDENTITY_TOKEN_FILE"] {

@mythical-fred-oss mythical-fred-oss Bot Sep 9, 2026

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this pair sufficient on its own? The EKS pod identity webhook normally injects AWS_STS_REGIONAL_ENDPOINTS=regional and, when configured, AWS_REGION/AWS_DEFAULT_REGION alongside these two, and most SDKs need a region to build the STS client for AssumeRoleWithWebIdentity (boto3 raises NoRegionError; the Rust SDK falls back to IMDS, which is often hop-limited in EKS). AWS_ROLE_SESSION_NAME is also injected when the SA is annotated. None of these carry a credential either, so they seem to fit the same rule.

Two related gaps worth a sentence in the PR description:

  • deployments on EKS Pod Identity (the successor to IRSA) get AWS_CONTAINER_CREDENTIALS_FULL_URI + AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE instead, and still will not work;
  • this is also the first time the spawned cargo (and therefore sccache as RUSTC_WRAPPER) sees these variables at all, so it incidentally fixes the IRSA path that docs/get-started/enterprise/parallel-compilation.md:86 already promises for sccache. Worth calling out — and worth checking whether sccache's S3 backend needs a region beyond SCCACHE_REGION.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
mythical-fred-oss[bot]
mythical-fred-oss Bot reviewed Sep 9, 2026
View reviewed changes
Comment thread crates/pipeline-manager/src/compiler/rust_compiler.rs
for key in ["AWS_PROFILE", "AWS_ROLE_ARN", "AWS_WEB_IDENTITY_TOKEN_FILE"] {
if let Some(value) = std::env::var_os(key) {
command.env(key, value);
}

@mythical-fred-oss mythical-fred-oss Bot Sep 9, 2026

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The allowlist is the security boundary of this function, and nothing in the file pins it down. Consider lifting the selection out of call_compiler (which is untestable because it spawns cargo) into a small pure helper that takes a lookup, e.g.

const INHERITED_AWS_VARS: [&str; 3] = ["AWS_PROFILE", "AWS_ROLE_ARN", "AWS_WEB_IDENTITY_TOKEN_FILE"];

fn inherited_aws_vars(
    lookup: impl Fn(&str) -> Option<OsString>,
) -> Vec<(&'static str, OsString)> { ... }

Then the existing mod tests can assert: each allowlisted var is forwarded when set, absent vars are skipped, and AWS_SECRET_ACCESS_KEY/AWS_SESSION_TOKEN are not forwarded — the last one is the regression test that keeps the next person from widening this to AWS_*. Taking a lookup closure rather than reading the real process env also keeps the test off std::env::set_var, which is process-global (flaky under the parallel test harness, and unsafe in edition 2024).

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@mythical-fred-oss

mythical-fred-oss Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Verified locally: cargo check --locked -p pipeline-manager ✅, pre-commit run --files crates/pipeline-manager/src/compiler/rust_compiler.rs ✅ (clippy/fmt/machete/openapi/doc all pass), cargo test --locked -p pipeline-manager --lib compiler::rust_compiler → 9 pass, workspace_preparation fails here only because this runner has no built SQL compiler jar (pre-existing, unrelated). The change itself is small and the rewritten comment is clearer than the one it replaces.

Two feldera-pr-checks gates are unmet:

Gate Status
Manual testing described ❌ PR body is empty — please say how you verified this end to end (which registry, which provider, IRSA cluster)
Test for changed behaviour ❌ no test; see the inline suggestion for a testable extraction

Test cases the PR does not cover, in rough priority order:

  1. AWS_SECRET_ACCESS_KEY / AWS_SESSION_TOKEN / AWS_ACCESS_KEY_ID are still not forwarded — this is the security-relevant assertion and nothing pins it.
  2. Each new var forwarded when set; absent var stays absent (no empty-string injection).
  3. A value that is not valid UTF-8 (OsString) survives — var_os is correct here, worth locking in.
  4. End to end: AWS_ROLE_ARN set but AWS_WEB_IDENTITY_TOKEN_FILE pointing at a missing/expired token — what does the compiler surface? Today that failure arrives as a raw cargo/registry auth error in stderr.log, which is not an actionable message for the operator.
  5. IRSA-based sccache: this is the first time the spawned cargo sees these vars, so the path documented at docs/get-started/enterprise/parallel-compilation.md:86 changes behaviour too and should be exercised.

Main substantive concern is in the inline comment on the allowlist: user-supplied UDF dependencies are spliced into the pipeline crate's Cargo.toml and their build scripts run in this process tree, so AWS_ROLE_ARN + the token is enough for them to assume the pod's role — the guarantee the old comment was protecting. Worth an explicit note about scoping that role, or making the forwarding opt-in.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@swanandx

swanandx commented Sep 10, 2026

Copy link
Copy Markdown
Member

btw, can't ROLE_ARN ( and other vars we are whitelisting ) be part of ~/.aws/config under correct profile? or am i missing something?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@xiasongh

xiasongh commented Sep 10, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor Author

btw, can't ROLE_ARN ( and other vars we are whitelisting ) be part of ~/.aws/config under correct profile? or am i missing something?

It can be. I thought it would be more natural to use these since these env vars are injected directly by k8s/aws and we're just letting them through, but I can write it into the config file instead.

What's the reason we decided to go with this profile approach in the first place?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@xiasongh

xiasongh commented Sep 10, 2026

Copy link
Copy Markdown
Contributor Author

I'll go with profile instead, which won't require any changes here

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@xiasongh xiasongh closed this Sep 10, 2026
@anandbraman anandbraman mentioned this pull request Nov 10, 2026
compiler: an sccache server that cargo starts gets no IRSA credentials, so all Rust builds fail #7923
Open
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mythical-fred-oss mythical-fred-oss[bot] mythical-fred-oss[bot] left review comments
@gz gz Awaiting requested review from gz

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

2 participants

@xiasongh @swanandx
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.