GeezerVisonThe World Wide Web, restored to 800 pixels of glory. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Home|About|Gaming cabinet|Archives|How it works|Starter sites | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
GeezerVison is showing github.com — Support bundle: stream while collecting, limit each step, no blind retry by blp · Pull Request #7818 · feldera/feldera Conversation
|
| Commit | Before | After |
|---|---|---|
| Limit each support data step | The 120 s limit stopped when the headers arrived. A multihost JSON profile then streamed for 434 s. | The limit covers the request and the whole body (within, support_data_collector.rs). The proxy endpoints keep their old limits. |
| Python: no blind retry | support_bundle() retried on 502/503/504, a read timeout, or a dropped connection. Each retry started a new slow collection. |
A bundle that collects new data does not retry after a failure that can come late. collect=false still retries. |
| Stream the bundle | No byte until the whole ZIP was built | The response starts when the request is valid. A new collection_log.txt entry gets a line every 10 s while the collection runs. Then each collection is sent when it is written. |
collection_log.txt example:
2026-11-03T22:20:48Z Collecting data from the pipeline.
2026-11-03T22:20:58Z Still collecting after 10 s.
2026-11-03T22:21:08Z Still collecting after 20 s.
2026-11-03T22:21:18Z Still collecting after 30 s.
2026-11-03T22:21:23Z Collected in 35 s.
ZipWriter::new_stream). A streamed bundle was read correctly by but-unzip 0.1.7 (the profiler UI), unzip -t, Python zipfile, and the zip crate (fda).SupportBundleData::collect already reports each failure inside the bundle, so this happens only if writing the ZIP fails.Content-Length, so the web console's progress bar is indeterminate for the whole download. It still shows the bytes received (DownloadProgressDisplay.svelte). Before, the bar showed nothing until the whole bundle was built (about 8 minutes in the reported case), and then a percentage over a fast download. A bar for the collection phase could be a separate web console change.LockTookTooLong, DeadlockDetected). The changelog and the operations guide describe the changes.Each test was checked against the old behavior:
| Test | Old behavior put back | Result |
|---|---|---|
within_stops_a_body_that_never_ends |
no limit around the fetch | hangs until its 10 s guard |
bundle_streams_while_a_collection_runs |
no compressor flush | empty chunks |
bundle_streams_while_a_collection_runs |
no log lines during the collection | 1 early chunk, not 4 |
test_support_bundle_retries_only_without_collection |
Pipeline.support_bundle() does not pass retry, or retry is True by default |
assertion fails |
limit_counts_the_new_collection |
new collection added last | a stored collection is kept instead |
bundle_without_collection_has_no_collection_log |
log started without a collection | log entry present |
failed_collection_ends_the_stream_with_an_error |
failed collection replaced by data | stream ends with Ok |
cargo test -p pipeline-manager --lib support: 25 passed. 3 sql_compiler tests fail locally because the SQL compiler JAR is not built in this worktree. They do not touch this code.
Sorry, something went wrong.
There was an error while loading. Please reload this page.
| /// | ||
| /// The runner's limit stops when the response headers arrive, so it does not | ||
| /// stop a pipeline that sends its headers and then a slow body. For example, | ||
| /// a multihost profile streams one host at a time. |
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
does it still?
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
I'll update the comment.
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
Not as worded: since feldera/cloud#2164 the coordinator asks all hosts at once, so "one host at a time" was misleading. It still sends the headers with the first host's profile and the rest after that, so the limit is still needed. 9252f08 rewords the example to say that.
Sorry, something went wrong.
There was an error while loading. Please reload this page.
| # A bundle that collects new data is slow and loads the pipeline, so do | ||
| # not repeat it after a failure that can come late, such as a gateway | ||
| # timeout or a dropped connection. | ||
| collect = str((params or {}).get("collect", "true")).lower() != "false" |
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
this doesn't look very robust to me.
what is "true"?
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
Agreed, the "true" default and the string matching were guessing at the server's parsing. 74ddc87 replaces them with a retry: bool = False argument to get_pipeline_support_bundle(). Pipeline.support_bundle() passes retry=not collect from its own bool.
Sorry, something went wrong.
There was an error while loading. Please reload this page.
003c045 to
e5f94df
Compare
| )) | ||
| .insert_header(actix_http::ContentEncoding::Identity) | ||
| .body(bundle.buffer)) | ||
| .streaming(body)) |
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
pitfall (medium, UX): Before, .body(Vec) set Content-Length. A streamed response has none. In the web console, streamToDownload (pipelineManager.ts) now passes totalBytes = 0, so useDownloadProgress.percent is always null. The support bundle download progress bar is then indeterminate for the whole download. Is that intended? If so, say so in the PR description. If not, the UI could show the bytes received, or the collection-log phase.
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
Intended. Without a Content-Length the bar is indeterminate, but DownloadProgressDisplay.svelte still shows the bytes received. Before, the bar showed nothing until the whole bundle was built (about 8 minutes in the reported case), and only then a percentage over a fast download. The PR description now says this. A bar for the collection phase could be a separate web console change.
Sorry, something went wrong.
There was an error while loading. Please reload this page.
|
What I ran:
Missing (medium, docs gate): The Missing (medium, tests): The tests don't cover Test diffdiff --git a/crates/pipeline-manager/src/api/endpoints/pipeline_interaction/support_bundle.rs b/crates/pipeline-manager/src/api/endpoints/pipeline_interaction/support_bundle.rs index 18e4af5..cf5c927 100644 --- a/crates/pipeline-manager/src/api/endpoints/pipeline_interaction/support_bundle.rs +++ b/crates/pipeline-manager/src/api/endpoints/pipeline_interaction/support_bundle.rs @@ -649,6 +649,71 @@ mod tests { assert!(dirs.contains(format!("{base}-1").as_str())); } + /// `limit` counts the new collection, and a failed collection ends the + /// stream with an error after the first chunk. + #[tokio::test(start_paused = true)] + async fn bundle_stream_limit_and_failed_collection() { + use futures_util::StreamExt; + + let params = SupportBundleParameters { + limit: Some(1), + ..Default::default() + }; + let collection = async { Ok(SupportBundleData::test_data()) }; + let chunks: Vec<_> = support_bundle_stream( + fake_pipeline("p"), + vec![SupportBundleData::test_data(), SupportBundleData::test_data()], + Some(collection), + params, + Duration::from_secs(10), + ) + .collect() + .await; + let bundle: Vec<u8> = chunks + .into_iter() + .flat_map(|c| c.unwrap().to_vec()) + .collect(); + let mut archive = read_zip(&bundle); + let mut metadata = String::new(); + archive + .by_name("metadata.json") + .unwrap() + .read_to_string(&mut metadata) + .unwrap(); + let metadata: serde_json::Value = serde_json::from_str(&metadata).unwrap(); + assert_eq!(metadata["collections"].as_array().unwrap().len(), 1); + + // No collection: no log entry. + let chunks: Vec<_> = support_bundle_stream( + fake_pipeline("p"), + vec![SupportBundleData::test_data()], + None::<std::future::Ready<Result<SupportBundleData, ManagerError>>>, + SupportBundleParameters::default(), + Duration::from_secs(10), + ) + .collect() + .await; + let bundle: Vec<u8> = chunks + .into_iter() + .flat_map(|c| c.unwrap().to_vec()) + .collect(); + assert!(read_zip(&bundle).by_name(COLLECTION_LOG).is_err()); + + // Failed collection. + let failing = async { Err(bundle_error("boom", "x")) }; + let results: Vec<_> = support_bundle_stream( + fake_pipeline("p"), + vec![], + Some(failing), + SupportBundleParameters::default(), + Duration::from_secs(10), + ) + .collect() + .await; + assert!(results[0].is_ok()); + assert!(results.last().unwrap().is_err()); + } + /// The bundle sends bytes while a slow collection runs, not only at its /// end, and the result is still one valid ZIP file. #[tokio::test(start_paused = true)] There is no handler-level test that checks the streamed response itself: no |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
e5f94df to
6c62f81
Compare
0613e4f to
4df17c5
Compare
4df17c5 to
1aa5aba
Compare
There was an error while loading. Please reload this page.
1a87426 to
fe698dc
Compare
There was an error while loading. Please reload this page.
|
Verdict: PR bug (high confidence). The merge queue run's test failure is caused by this PR, specifically a gap left by its own last commit. What failed
Why Fix Take it with: Or compare: support-bundle-2163...fred/observe-7818-42285488013 |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
9b48c6f to
b07139d
Compare
Each support data step had a 120 s limit, but awc's limit ends when the response headers arrive. A multihost pipeline sends the headers of its JSON profile with the first host's profile and the other hosts' profiles after that, so the rest of the body had no limit. In a 50-host pipeline in a long transaction commit, the profile step took 434 s, and the bundle response was silent for all that time. Now the 120 s limit also covers the body. A step that is not complete after 120 s fails and is reported in the bundle, and the other steps are still delivered. Part of feldera/cloud#2163. Signed-off-by: Ben Pfaff <[email protected]>
A GET retries on 502, 503, 504, a read timeout, or a dropped connection. For a support bundle that collects new data, each retry starts a new slow collection that loads the pipeline. Such a failure can come after the collection ran for minutes, so a retry seldom helps. A bundle with collect=false only reads stored data, so it still retries. get_pipeline_support_bundle() takes `retry`, which is False by default, and Pipeline.support_bundle() passes `retry=not collect`. Part of feldera/cloud#2163. Signed-off-by: Ben Pfaff <[email protected]>
The support bundle handler sent nothing until it had collected all data and built the whole ZIP file. A slow collection, such as the profile of a large multihost pipeline in a long commit, kept the response silent for minutes, so a proxy or the client closed it first. Now the response starts when the request is valid. When the bundle collects new data, its first entry is collection_log.txt, which gets a line every 10 s until the collection is complete. Then the collections and the metadata follow, each sent when it is written. The ZIP file uses data descriptors, which the profiler UI, unzip, Python, and the zip crate read. A failure after the response starts now cuts the response short instead of setting an error status. The collection itself reports failures inside the bundle, as before. The API docs no longer list 503 examples for a failed pipeline request, because the endpoint reports such failures inside the bundle. They describe collection_log.txt and the missing Content-Length. The operations guide and the changelog describe the change too. Part of feldera/cloud#2163. Signed-off-by: Ben Pfaff <[email protected]>
b07139d to
f719398
Compare
There was an error while loading. Please reload this page.
A UDF that sleeps 45 s keeps one circuit step busy. A support bundle requested during that step must send its first bytes within 15 s, and must still hold the circuit profile when the step completes. The test also checks that the bundle took more than half of the step time, so that it fails if the bundle started before the step did. Before #7818, the bundle sent nothing until it had all of its data, so a proxy could close the idle connection (feldera/cloud#2163). On v0.375.0 this test fails: the first byte arrived after 43.1 s. The CI multihost suite runs this test with 2 hosts too. Signed-off-by: Ben Pfaff <[email protected]>
A UDF that sleeps 45 s keeps one circuit step busy. A support bundle requested during that step must send its first bytes within 15 s, and must still hold the circuit profile when the step completes. Before feldera#7818, the bundle sent nothing until it had all of its data, so a proxy could close the idle connection (feldera/cloud#2163). On v0.375.0 this test fails: the first byte arrived after 43.1 s. The CI multihost suite runs this test with 2 hosts too. Signed-off-by: Ben Pfaff <[email protected]>
There was an error while loading. Please reload this page.
Successfully merging this pull request may close these issues.
There was an error while loading. Please reload this page.