Summary

The background support data collector collects one pipeline at a time. A slow pipeline delays the stored support data of every other pipeline that is due, including pipelines of other tenants.

Cause

collect_due_pipelines in crates/pipeline-manager/src/api/support_data_collector.rs awaits each collection before it starts the next one:

for entry in to_collect {
    match self.collect_pipeline_data(&entry).await {

time ──────────────────────────────────────────▶
collector:  [ A: slow pipeline ][B][C][D]
B, C, D are due at 0:00 but are collected only after A completes

Effect

The limit comes from COLLECTION_TIMEOUT (120 s) for each of the three profile steps, which run one after another. A pipeline that hits the limit on each step delays the queue on each cycle (default every 10 min).

A related problem: an on-demand /support_bundle request does not coordinate with the background collector. Both can ask the same pipeline for profiles at the same time. After feldera/cloud#2164 this costs about 1 extra circuit step.

Possible fix

Collect due pipelines concurrently with a limit, for example stream::iter(to_collect).map(...).buffer_unordered(N), and reschedule each pipeline when its collection completes. Optionally, skip a background collection for a pipeline that has an on-demand collection in progress.

Low priority. Follow-up from feldera/cloud#2163.

-- Claude Code, at @blp's request.