GeezerVisonThe World Wide Web, restored to 800 pixels of glory. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Home|About|Gaming cabinet|Archives|How it works|Starter sites | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
GeezerVison is showing github.com — [pipeline-manager] Serve prebuilt pipeline binaries from the compiler by gz · Pull Request #7885 · feldera/feldera Conversation
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Flag | Effect |
|---|---|
--prebuilt-binaries-directory <dir> |
Before a Rust compilation, the compiler looks for a binary of the same program in <dir> and delivers it if present. |
--prebuild-sql-program <file> (repeatable, with --precompile) |
--precompile also compiles each file as the nil pipeline and stores its binary in the directory under its lookup key. |
| Topic | Behaviour |
|---|---|
| Lookup key | The source checksum, with the generated code hashed by file paths and contents and under the crate names of the nil pipeline. The plain source checksum cannot find a prebuilt binary: it hashes encoded generated code that carries modification times and pipeline-specific crate names. |
| Key normalization | Rust identifiers and Cargo crate references are normalized structurally; SQL literals are never rewritten, so two programs that differ only in a literal keep distinct keys. Duplicate normalized archive paths are refused. |
| Cargo target | The compiler passes CARGO_BUILD_TARGET and reads artifacts from target/<triple>/, so RUSTFLAGS apply to target artifacts only and coverage instrumentation no longer breaks build scripts and proc macros. |
| Default | Off. Without the flag the compiler behaves as before. |
The four commits come from the branch antithesis-prebuilt-binaries, rebased on main with the lock file regenerated; authorship is kept.
| Check | Result |
|---|---|
git range-diff against the original branch |
Identical apart from toml crate versions in Cargo.lock |
cargo test -p pipeline-manager --no-default-features --features feldera-enterprise --lib prebuilt_binary |
4 passed |
cargo check -p pipeline-manager (default features) |
ok |
pre-commit run --files <changed files> |
all hooks passed (clippy, fmt, machete, OpenAPI, docs) |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
Some environments cannot run cargo when a pipeline starts. A deterministic simulation test runs on a single slow CPU with no crate registry, yet still needs to start pipelines for a known set of SQL programs. The compiler server gains --prebuilt-binaries-directory. Before running cargo, it looks there for a binary of the same program and, if one exists, delivers it instead. A binary is valid for any pipeline with the same SQL, but the source checksum cannot find it: it hashes the encoded generated code, which carries file modification times, and the generated crate names embed the pipeline id. The lookup key is therefore the source checksum with the generated code hashed by file paths and contents, under the crate names of the nil pipeline. --precompile fills the directory: each --prebuild-sql-program file is compiled as the nil pipeline, like the dummy program, and its binary is copied there under its lookup key. The compiler now also passes CARGO_BUILD_TARGET to cargo and reads artifacts from target/<triple>/. With an explicit target, RUSTFLAGS apply only to target artifacts, so coverage instrumentation flags no longer break the linking of build scripts and proc macros. Signed-off-by: Gerd Zellweger <[email protected]>
… key find_prebuilt_binary receives the prebuilt binary key, which differs from the source checksum its parameter and test were named after. Signed-off-by: Gerd Zellweger <[email protected]>
Update the test-only PostgreSQL connector to the current TLS API and use the Rust 2024-compatible random helper. This lets prebuilt cache unit tests compile without installing an embedded database.
Blind namespace replacement can change SQL literals and conflate distinct programs. Normalize generated Rust identifiers and Cargo crate references structurally, including workspace members, while preserving literal data. Reject duplicate normalized archive paths instead of overwriting them. Cover namespace independence, literal collisions, Cargo references and malformed generated sources. The four prebuilt cache tests pass, and real SQL compiler output normalizes identically across pipeline IDs.
|
|
||
| let prebuilt_binary_key = match config.prebuilt_binaries_directory { | ||
| None => None, | ||
| Some(_) => Some(prebuilt_binary_key( |
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
pitfall (medium): the cache lookup is optional, but the ? makes any failure here fatal. If proc_macro2 can't tokenize a generated file, if an archive entry isn't UTF-8 (read_to_string), or if two archive paths normalize to the same path, the whole compilation fails with a SystemError about the "prebuilt cache". Without the cache, cargo would just have run. Suggest logging a warning and treating it as a miss (None), plus a test with an untokenizable main_rust that checks the compilation still goes through cargo.
Sorry, something went wrong.
There was an error while loading. Please reload this page.
| #[arg(long)] | ||
| pub precompile: bool, | ||
|
|
||
| /// Directory of pipeline binaries built ahead of time, each named by the |
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
bug (low-medium, user-visible --help): this doc string is out of date after the second commit. Files are named by the prebuilt binary key, which is the source checksum with the generated code normalized. The plain source checksum never matches (the PR description says so too). Anyone who populates the directory by hand from this text will never get a hit. Please describe the key, or say that the directory should only be filled with --precompile --prebuild-sql-program. The same goes for "A Rust compilation whose source checksum names a file…" below.
Sorry, something went wrong.
There was an error while loading. Please reload this page.
| @@ -2922,6 +3057,151 @@ mod test { | |||
| } | |||
|
|
|||
| /// Tests the binary delivery mode configuration. | |||
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
nit: the new tests were inserted under this doc line, so /// Tests the binary delivery mode configuration. now documents prebuilt_binary_key_depends_only_on_program. Please move it back down to binary_delivery_mode_from_config.
Sorry, something went wrong.
There was an error while loading. Please reload this page.
|
What I ran: Gaps in the tests (missing, medium):
Pitfall (low-medium): forwarding No unsafe code, no CI workflow changes, nothing breaking for users. The new flags are internal and test-oriented, so I'm not asking for docs beyond fixing the |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
Successfully merging this pull request may close these issues.
There was an error while loading. Please reload this page.