GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — [pipeline-manager] Serve prebuilt pipeline binaries from the compiler by gz · Pull Request #7885 · feldera/feldera

Conversation

@gz

@gz gz commented Nov 7, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

The compiler server can deliver a pipeline binary built ahead of time instead of running cargo. Environments that cannot compile when a pipeline starts, such as the Antithesis deterministic simulation (one slow CPU, no crate registry), still run a fixed set of SQL programs.

Flag Effect
--prebuilt-binaries-directory <dir> Before a Rust compilation, the compiler looks for a binary of the same program in <dir> and delivers it if present.
--prebuild-sql-program <file> (repeatable, with --precompile) --precompile also compiles each file as the nil pipeline and stores its binary in the directory under its lookup key.
Topic Behaviour
Lookup key The source checksum, with the generated code hashed by file paths and contents and under the crate names of the nil pipeline. The plain source checksum cannot find a prebuilt binary: it hashes encoded generated code that carries modification times and pipeline-specific crate names.
Key normalization Rust identifiers and Cargo crate references are normalized structurally; SQL literals are never rewritten, so two programs that differ only in a literal keep distinct keys. Duplicate normalized archive paths are refused.
Cargo target The compiler passes CARGO_BUILD_TARGET and reads artifacts from target/<triple>/, so RUSTFLAGS apply to target artifacts only and coverage instrumentation no longer breaks build scripts and proc macros.
Default Off. Without the flag the compiler behaves as before.

The four commits come from the branch antithesis-prebuilt-binaries, rebased on main with the lock file regenerated; authorship is kept.

Verification

Check Result
git range-diff against the original branch Identical apart from toml crate versions in Cargo.lock
cargo test -p pipeline-manager --no-default-features --features feldera-enterprise --lib prebuilt_binary 4 passed
cargo check -p pipeline-manager (default features) ok
pre-commit run --files <changed files> all hooks passed (clippy, fmt, machete, OpenAPI, docs)

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
gz and others added 4 commits November 6, 2026 17:23
@gz
[pipeline-manager] Serve prebuilt pipeline binaries from the compiler
d8fb20e
Some environments cannot run cargo when a pipeline starts. A
deterministic simulation test runs on a single slow CPU with no crate
registry, yet still needs to start pipelines for a known set of SQL
programs.

The compiler server gains --prebuilt-binaries-directory. Before running
cargo, it looks there for a binary of the same program and, if one
exists, delivers it instead. A binary is valid for any pipeline with the
same SQL, but the source checksum cannot find it: it hashes the encoded
generated code, which carries file modification times, and the
generated crate names embed the pipeline id. The lookup key is
therefore the source checksum with the generated code hashed by file
paths and contents, under the crate names of the nil pipeline.

--precompile fills the directory: each --prebuild-sql-program file is
compiled as the nil pipeline, like the dummy program, and its binary is
copied there under its lookup key.

The compiler now also passes CARGO_BUILD_TARGET to cargo and reads
artifacts from target/<triple>/. With an explicit target, RUSTFLAGS
apply only to target artifacts, so coverage instrumentation flags no
longer break the linking of build scripts and proc macros.

Signed-off-by: Gerd Zellweger <[email protected]>
@gz
[pipeline-manager] Name the prebuilt binary lookup argument after its…
1ab6e75
… key

find_prebuilt_binary receives the prebuilt binary key, which differs
from the source checksum its parameter and test were named after.

Signed-off-by: Gerd Zellweger <[email protected]>
@bestouff @gz
Fix non-embedded compiler test compilation
0049257
Update the test-only PostgreSQL connector to the current TLS API and
use the Rust 2024-compatible random helper. This lets prebuilt cache
unit tests compile without installing an embedded database.
@bestouff @gz
Preserve literals in prebuilt binary cache keys
6d28525
Blind namespace replacement can change SQL literals and conflate distinct
programs. Normalize generated Rust identifiers and Cargo crate references
structurally, including workspace members, while preserving literal data.
Reject duplicate normalized archive paths instead of overwriting them.

Cover namespace independence, literal collisions, Cargo references and
malformed generated sources. The four prebuilt cache tests pass, and real
SQL compiler output normalizes identically across pipeline IDs.
@gz
gz requested a review from bestouff November 7, 2026 01:31
mythical-fred-oss[bot]
mythical-fred-oss Bot reviewed Nov 7, 2026
View reviewed changes
Comment thread crates/pipeline-manager/src/compiler/rust_compiler.rs

let prebuilt_binary_key = match config.prebuilt_binaries_directory {
None => None,
Some(_) => Some(prebuilt_binary_key(

@mythical-fred-oss mythical-fred-oss Bot Nov 7, 2026

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pitfall (medium): the cache lookup is optional, but the ? makes any failure here fatal. If proc_macro2 can't tokenize a generated file, if an archive entry isn't UTF-8 (read_to_string), or if two archive paths normalize to the same path, the whole compilation fails with a SystemError about the "prebuilt cache". Without the cache, cargo would just have run. Suggest logging a warning and treating it as a miss (None), plus a test with an untokenizable main_rust that checks the compilation still goes through cargo.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
mythical-fred-oss[bot]
mythical-fred-oss Bot reviewed Nov 7, 2026
View reviewed changes
Comment thread crates/pipeline-manager/src/config.rs
#[arg(long)]
pub precompile: bool,

/// Directory of pipeline binaries built ahead of time, each named by the

@mythical-fred-oss mythical-fred-oss Bot Nov 7, 2026

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug (low-medium, user-visible --help): this doc string is out of date after the second commit. Files are named by the prebuilt binary key, which is the source checksum with the generated code normalized. The plain source checksum never matches (the PR description says so too). Anyone who populates the directory by hand from this text will never get a hit. Please describe the key, or say that the directory should only be filled with --precompile --prebuild-sql-program. The same goes for "A Rust compilation whose source checksum names a file…" below.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
mythical-fred-oss[bot]
mythical-fred-oss Bot reviewed Nov 7, 2026
View reviewed changes
Comment thread crates/pipeline-manager/src/compiler/rust_compiler.rs
@@ -2922,6 +3057,151 @@ mod test {
}

/// Tests the binary delivery mode configuration.

@mythical-fred-oss mythical-fred-oss Bot Nov 7, 2026

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: the new tests were inserted under this doc line, so /// Tests the binary delivery mode configuration. now documents prebuilt_binary_key_depends_only_on_program. Please move it back down to binary_delivery_mode_from_config.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@mythical-fred-oss

mythical-fred-oss Bot commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

What I ran: cargo test --locked -p pipeline-manager --lib prebuilt (debug, default features). All 4 tests pass. I also checked that toml 1.2 is MIT/Apache-2.0 and pulls in no crypto. It is pinned in the crate, not in the workspace deps (fine).

Gaps in the tests (missing, medium):

Case Covered?
Key is stable across pipeline IDs and mtimes; literals are preserved ✅
find_prebuilt_binary hit, miss, and directory entry ✅
perform_rust_compilation hit path: delivers the prebuilt binary and program info, skips cargo, returns rustc_result ❌
prebuild_program: copies from cargo_profile_dir of the nil pipeline, skips when the file already exists ❌
Key computation fails (bad tokens, non-UTF-8, duplicate path) → should fall back to cargo (see inline) ❌
CARGO_BUILD_TARGET set → cargo_profile_dir and cleanup use target/<triple>/<profile> ❌

Pitfall (low-medium): forwarding CARGO_BUILD_TARGET is not behind the new flag. It changes where every compiler reads artifacts from whenever that variable is set in its environment, so "Default: off" applies only to the binary cache. Please mention it as a behaviour change, or gate it.

No unsafe code, no CI workflow changes, nothing breaking for users. The new flags are internal and test-oriented, so I'm not asking for docs beyond fixing the --help text flagged inline.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

bestouff
bestouff approved these changes Nov 7, 2026
View reviewed changes
@bestouff
bestouff added this pull request to the merge queue Nov 7, 2026
Merged via the queue into main with commit a9104c1 Nov 7, 2026
3 checks passed

Uh oh!

There was an error while loading. Please reload this page.

@bestouff
bestouff deleted the gz/prebuilt-pipeline-binaries branch November 7, 2026 12:10
@bestouff bestouff mentioned this pull request Nov 9, 2026
[manager] Fix review nits from the prebuilt-binaries change #7897
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mythical-fred-oss mythical-fred-oss[bot] mythical-fred-oss[bot] left review comments
@bestouff bestouff bestouff approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

2 participants

@gz @bestouff
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.