GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — Support bundle: stream while collecting, limit each step, no blind retry by blp · Pull Request #7818 · feldera/feldera

Conversation

@blp

@blp blp commented Nov 3, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

Part of feldera/cloud#2163. The coordinator part is feldera/cloud#2164.

Problem

A support bundle for a 50-host pipeline in a long transaction commit failed in the web console and in the API. The response was silent for about 8 minutes, so a proxy or the client closed it first.

Changes

Commit Before After
Limit each support data step The 120 s limit stopped when the headers arrived. A multihost JSON profile then streamed for 434 s. The limit covers the request and the whole body (within, support_data_collector.rs). The proxy endpoints keep their old limits.
Python: no blind retry support_bundle() retried on 502/503/504, a read timeout, or a dropped connection. Each retry started a new slow collection. A bundle that collects new data does not retry after a failure that can come late. collect=false still retries.
Stream the bundle No byte until the whole ZIP was built The response starts when the request is valid. A new collection_log.txt entry gets a line every 10 s while the collection runs. Then each collection is sent when it is written.

collection_log.txt example:

2026-11-03T22:20:48Z Collecting data from the pipeline.
2026-11-03T22:20:58Z Still collecting after 10 s.
2026-11-03T22:21:08Z Still collecting after 20 s.
2026-11-03T22:21:18Z Still collecting after 30 s.
2026-11-03T22:21:23Z Collected in 35 s.

Behavior changes

  • The ZIP is written with data descriptors (ZipWriter::new_stream). A streamed bundle was read correctly by but-unzip 0.1.7 (the profiler UI), unzip -t, Python zipfile, and the zip crate (fda).
  • A failure after the response starts cuts the response short. Before, it set an error status. SupportBundleData::collect already reports each failure inside the bundle, so this happens only if writing the ZIP fails.
  • The manager does not hold the whole ZIP in memory anymore.
  • The response has no Content-Length, so the web console's progress bar is indeterminate for the whole download. It still shows the bytes received (DownloadProgressDisplay.svelte). Before, the bar showed nothing until the whole bundle was built (about 8 minutes in the reported case), and then a percentage over a fast download. A bar for the collection phase could be a separate web console change.
  • The API docs no longer list the 503 "Disconnected" and "Response timeout" examples: the endpoint reports pipeline failures inside the bundle, so it never returned them. The only 503 left is a busy database (LockTookTooLong, DeadlockDetected). The changelog and the operations guide describe the changes.

Tests

Each test was checked against the old behavior:

Test Old behavior put back Result
within_stops_a_body_that_never_ends no limit around the fetch hangs until its 10 s guard
bundle_streams_while_a_collection_runs no compressor flush empty chunks
bundle_streams_while_a_collection_runs no log lines during the collection 1 early chunk, not 4
test_support_bundle_retries_only_without_collection Pipeline.support_bundle() does not pass retry, or retry is True by default assertion fails
limit_counts_the_new_collection new collection added last a stored collection is kept instead
bundle_without_collection_has_no_collection_log log started without a collection log entry present
failed_collection_ends_the_stream_with_an_error failed collection replaced by data stream ends with Ok

cargo test -p pipeline-manager --lib support: 25 passed. 3 sql_compiler tests fail locally because the SQL compiler JAR is not built in this worktree. They do not touch this code.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
mihaibudiu
mihaibudiu approved these changes Nov 3, 2026
View reviewed changes
Comment thread crates/pipeline-manager/src/api/support_data_collector.rs Outdated
///
/// The runner's limit stops when the response headers arrive, so it does not
/// stop a pipeline that sends its headers and then a slow body. For example,
/// a multihost profile streams one host at a time.

@mihaibudiu mihaibudiu Nov 3, 2026

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does it still?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@blp blp Nov 3, 2026

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll update the comment.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@blp blp Nov 3, 2026

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not as worded: since feldera/cloud#2164 the coordinator asks all hosts at once, so "one host at a time" was misleading. It still sends the headers with the first host's profile and the rest after that, so the limit is still needed. 9252f08 rewords the example to say that.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread python/feldera/rest/feldera_client.py Outdated
# A bundle that collects new data is slow and loads the pipeline, so do
# not repeat it after a failure that can come late, such as a gateway
# timeout or a dropped connection.
collect = str((params or {}).get("collect", "true")).lower() != "false"

@mihaibudiu mihaibudiu Nov 3, 2026

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this doesn't look very robust to me.
what is "true"?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@blp blp Nov 3, 2026

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, the "true" default and the string matching were guessing at the server's parsing. 74ddc87 replaces them with a retry: bool = False argument to get_pipeline_support_bundle(). Pipeline.support_bundle() passes retry=not collect from its own bool.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@blp
blp force-pushed the support-bundle-2163 branch from 003c045 to e5f94df Compare November 3, 2026 22:46
mythical-fred-oss[bot]
mythical-fred-oss Bot reviewed Nov 3, 2026
View reviewed changes
Comment thread crates/pipeline-manager/src/api/endpoints/pipeline_interaction/support_bundle.rs
))
.insert_header(actix_http::ContentEncoding::Identity)
.body(bundle.buffer))
.streaming(body))

@mythical-fred-oss mythical-fred-oss Bot Nov 3, 2026

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pitfall (medium, UX): Before, .body(Vec) set Content-Length. A streamed response has none. In the web console, streamToDownload (pipelineManager.ts) now passes totalBytes = 0, so useDownloadProgress.percent is always null. The support bundle download progress bar is then indeterminate for the whole download. Is that intended? If so, say so in the PR description. If not, the UI could show the bytes received, or the collection-log phase.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@blp blp Nov 3, 2026

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Intended. Without a Content-Length the bar is indeterminate, but DownloadProgressDisplay.svelte still shows the bytes received. Before, the bar showed nothing until the whole bundle was built (about 8 minutes in the reported case), and only then a percentage over a fast download. The PR description now says this. A bar for the collection phase could be a separate web console change.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@mythical-fred-oss

mythical-fred-oss Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

What I ran:

Check Result
cargo test --locked -p pipeline-manager --lib support 22 passed
New streaming tests, run 3 times (paused clock) stable, no flakiness
uv run pytest tests/unit/test_httprequests_retry.py 74 passed
pre-commit run --files <changed> (clippy, fmt, machete, OpenAPI, ruff) all passed

Missing (medium, docs gate): The #[utoipa::path] doc on get_pipeline_support_bundle is now out of date. Its 503 "Response timeout" and "Disconnected" examples can't be returned as a status anymore, because a late failure now cuts a 200 response short. It also doesn't mention the new collection_log.txt entry. No docs.feldera.com or changelog update covers these user-visible changes.

Missing (medium, tests): The tests don't cover limit=1 combined with collect=true (the limit should count the new collection), collect=false (no collection_log.txt entry), or a failed collection (the stream should yield Ok and then Err). I wrote a test for all three and it passes:

Test diff
diff --git a/crates/pipeline-manager/src/api/endpoints/pipeline_interaction/support_bundle.rs b/crates/pipeline-manager/src/api/endpoints/pipeline_interaction/support_bundle.rs
index 18e4af5..cf5c927 100644
--- a/crates/pipeline-manager/src/api/endpoints/pipeline_interaction/support_bundle.rs
+++ b/crates/pipeline-manager/src/api/endpoints/pipeline_interaction/support_bundle.rs
@@ -649,6 +649,71 @@ mod tests {
         assert!(dirs.contains(format!("{base}-1").as_str()));
     }
 
+    /// `limit` counts the new collection, and a failed collection ends the
+    /// stream with an error after the first chunk.
+    #[tokio::test(start_paused = true)]
+    async fn bundle_stream_limit_and_failed_collection() {
+        use futures_util::StreamExt;
+
+        let params = SupportBundleParameters {
+            limit: Some(1),
+            ..Default::default()
+        };
+        let collection = async { Ok(SupportBundleData::test_data()) };
+        let chunks: Vec<_> = support_bundle_stream(
+            fake_pipeline("p"),
+            vec![SupportBundleData::test_data(), SupportBundleData::test_data()],
+            Some(collection),
+            params,
+            Duration::from_secs(10),
+        )
+        .collect()
+        .await;
+        let bundle: Vec<u8> = chunks
+            .into_iter()
+            .flat_map(|c| c.unwrap().to_vec())
+            .collect();
+        let mut archive = read_zip(&bundle);
+        let mut metadata = String::new();
+        archive
+            .by_name("metadata.json")
+            .unwrap()
+            .read_to_string(&mut metadata)
+            .unwrap();
+        let metadata: serde_json::Value = serde_json::from_str(&metadata).unwrap();
+        assert_eq!(metadata["collections"].as_array().unwrap().len(), 1);
+
+        // No collection: no log entry.
+        let chunks: Vec<_> = support_bundle_stream(
+            fake_pipeline("p"),
+            vec![SupportBundleData::test_data()],
+            None::<std::future::Ready<Result<SupportBundleData, ManagerError>>>,
+            SupportBundleParameters::default(),
+            Duration::from_secs(10),
+        )
+        .collect()
+        .await;
+        let bundle: Vec<u8> = chunks
+            .into_iter()
+            .flat_map(|c| c.unwrap().to_vec())
+            .collect();
+        assert!(read_zip(&bundle).by_name(COLLECTION_LOG).is_err());
+
+        // Failed collection.
+        let failing = async { Err(bundle_error("boom", "x")) };
+        let results: Vec<_> = support_bundle_stream(
+            fake_pipeline("p"),
+            vec![],
+            Some(failing),
+            SupportBundleParameters::default(),
+            Duration::from_secs(10),
+        )
+        .collect()
+        .await;
+        assert!(results[0].is_ok());
+        assert!(results.last().unwrap().is_err());
+    }
+
     /// The bundle sends bytes while a slow collection runs, not only at its
     /// end, and the result is still one valid ZIP file.
     #[tokio::test(start_paused = true)]

There is no handler-level test that checks the streamed response itself: no Content-Length header, chunked framing, and a client disconnect dropping the collection future. That case is uncovered.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@blp
blp force-pushed the support-bundle-2163 branch from e5f94df to 6c62f81 Compare November 3, 2026 22:49
@blp
blp added this pull request to the merge queue Nov 3, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Nov 4, 2026
@blp
blp force-pushed the support-bundle-2163 branch from 0613e4f to 4df17c5 Compare November 4, 2026 15:59
@blp
blp enabled auto-merge November 4, 2026 15:59
@blp
blp disabled auto-merge November 4, 2026 21:23
@blp
blp added this pull request to the merge queue Nov 4, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Nov 5, 2026
@blp
blp force-pushed the support-bundle-2163 branch from 4df17c5 to 1aa5aba Compare November 5, 2026 18:36
@blp
blp enabled auto-merge November 5, 2026 18:36
@blp
blp disabled auto-merge November 5, 2026 19:02
@blp
blp added this pull request to the merge queue Nov 5, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 5, 2026

Uh oh!

There was an error while loading. Please reload this page.

@blp
blp added this pull request to the merge queue Nov 5, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Nov 6, 2026
@blp
blp force-pushed the support-bundle-2163 branch from 1a87426 to fe698dc Compare November 6, 2026 17:13
@blp
blp enabled auto-merge November 6, 2026 17:13
@blp
blp disabled auto-merge November 6, 2026 18:39
@blp
blp added this pull request to the merge queue Nov 6, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 6, 2026

Uh oh!

There was an error while loading. Please reload this page.

@mythical-fred-oss

mythical-fred-oss Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Verdict: PR bug (high confidence). The merge queue run's test failure is caused by this PR, specifically a gap left by its own last commit.

What failed

  • Job: Integration Tests / Platform Integration Tests (OSS Docker Image)
  • crates/fda/test.bash line 274 (pre-fix): asserts a support bundle with every collection disabled contains exactly 2 files (metadata.txt, metadata.json), counted as 3 rows via unzip -l.
  • Log shows file_count=4 and lists collection_log.txt, metadata.txt, metadata.json — i.e. 3 real files, not 2 (log lines ~2818-2825 in the job log).

Why
Commit fe698dc (this PR's HEAD) updated python/tests/platform/test_shared_pipeline.py::test_support_bundle_with_selectors to expect collection_log.txt alongside the metadata files, because a support bundle that collects new data now always starts with that file — even with every selector disabled. The equivalent check in crates/fda/test.bash wasn't updated to match, so it still expects the old count and fails.

Fix
Pushed to fred/observe-7818-42285488013 (commit 7fb003c6a): update the test.bash assertion to expect 3 files (4 counted rows), matching the Python test's fix.

Take it with:

git fetch origin fred/observe-7818-42285488013
git cherry-pick 7fb003c6a

Or compare: support-bundle-2163...fred/observe-7818-42285488013

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@blp
blp force-pushed the support-bundle-2163 branch from 9b48c6f to b07139d Compare November 6, 2026 22:23
blp added 3 commits November 6, 2026 14:25
@blp
[pipeline-manager] Limit each support data step to the whole response.
0cd26c8
Each support data step had a 120 s limit, but awc's limit ends when the
response headers arrive. A multihost pipeline sends the headers of its
JSON profile with the first host's profile and the other hosts' profiles
after that, so the rest of the body had no limit. In a 50-host pipeline
in a long transaction commit, the profile step took 434 s, and the
bundle response was silent for all that time.

Now the 120 s limit also covers the body. A step that is not complete
after 120 s fails and is reported in the bundle, and the other steps
are still delivered.

Part of feldera/cloud#2163.

Signed-off-by: Ben Pfaff <[email protected]>
@blp
[python] Do not retry a support bundle that collects new data.
973190d
A GET retries on 502, 503, 504, a read timeout, or a dropped connection.
For a support bundle that collects new data, each retry starts a new
slow collection that loads the pipeline. Such a failure can come after
the collection ran for minutes, so a retry seldom helps.

A bundle with collect=false only reads stored data, so it still retries.
get_pipeline_support_bundle() takes `retry`, which is False by default,
and Pipeline.support_bundle() passes `retry=not collect`.

Part of feldera/cloud#2163.

Signed-off-by: Ben Pfaff <[email protected]>
@blp
[pipeline-manager] Stream the support bundle while it is collected.
f719398
The support bundle handler sent nothing until it had collected all data
and built the whole ZIP file. A slow collection, such as the profile of
a large multihost pipeline in a long commit, kept the response silent
for minutes, so a proxy or the client closed it first.

Now the response starts when the request is valid. When the bundle
collects new data, its first entry is collection_log.txt, which gets a
line every 10 s until the collection is complete. Then the collections
and the metadata follow, each sent when it is written. The ZIP file
uses data descriptors, which the profiler UI, unzip, Python, and the
zip crate read.

A failure after the response starts now cuts the response short
instead of setting an error status. The collection itself reports
failures inside the bundle, as before.

The API docs no longer list 503 examples for a failed pipeline request,
because the endpoint reports such failures inside the bundle. They
describe collection_log.txt and the missing Content-Length. The
operations guide and the changelog describe the change too.

Part of feldera/cloud#2163.

Signed-off-by: Ben Pfaff <[email protected]>
@blp
blp force-pushed the support-bundle-2163 branch from b07139d to f719398 Compare November 6, 2026 22:26
@blp
blp enabled auto-merge November 6, 2026 22:27
@blp
blp disabled auto-merge November 6, 2026 22:45
@blp
blp added this pull request to the merge queue Nov 6, 2026
Merged via the queue into main with commit 3469804 Nov 7, 2026
1 check passed

Uh oh!

There was an error while loading. Please reload this page.

@blp
blp deleted the support-bundle-2163 branch November 7, 2026 00:00
@blp blp mentioned this pull request Nov 11, 2026
[python] Test that a support bundle starts during a slow circuit step #7935
Merged
blp added a commit that referenced this pull request Nov 11, 2026
@blp
[python] Test that a support bundle starts during a slow circuit step.
7f5ac0c
A UDF that sleeps 45 s keeps one circuit step busy. A support bundle
requested during that step must send its first bytes within 15 s, and
must still hold the circuit profile when the step completes. The test
also checks that the bundle took more than half of the step time, so
that it fails if the bundle started before the step did.

Before #7818, the bundle sent nothing until it had all
of its data, so a proxy could close the idle connection
(feldera/cloud#2163). On v0.375.0 this test fails: the first byte
arrived after 43.1 s.

The CI multihost suite runs this test with 2 hosts too.

Signed-off-by: Ben Pfaff <[email protected]>
yermakoffivan pushed a commit to yermakoffivan/feldera that referenced this pull request Nov 11, 2026
@blp
[python] Test that a support bundle starts during a slow circuit step.
9872687
A UDF that sleeps 45 s keeps one circuit step busy. A support bundle
requested during that step must send its first bytes within 15 s, and
must still hold the circuit profile when the step completes.

Before feldera#7818, the bundle sent nothing until it had all
of its data, so a proxy could close the idle connection
(feldera/cloud#2163). On v0.375.0 this test fails: the first byte
arrived after 43.1 s.

The CI multihost suite runs this test with 2 hosts too.

Signed-off-by: Ben Pfaff <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mythical-fred-oss mythical-fred-oss[bot] mythical-fred-oss[bot] left review comments
@mihaibudiu mihaibudiu mihaibudiu approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

2 participants

@blp @mihaibudiu
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.