Description

jyotshnayaparla-00

opened on Oct 30, 2026

Issue body actions

With in-cluster HTTPS on (httpsSecretRef), force-stopping a pipeline breaks an open egress stream. The pipeline closes its TLS connection without sending close_notify. The API server treats that as an error and aborts the chunked response, so the client gets ChunkedEncodingError: Response ended prematurely instead of a clean end of stream. Over plain HTTP the same close ends the stream cleanly.

Repro: tests/runtime/test_spreadsheet.py::TestUDF::test_local against an instance with HTTPS and mTLS on. It listens to a view, waits for the records, calls pipeline.stop(force=True), then reads the buffered output. It failed 3 times out of 3 on both arches on the CI instances after we turned HTTPS on (feldera merge queue run 41134745157), and passed before.

API server log at the same time:
ERROR actix_http::h1::dispatcher: Response payload stream error: ... UnexpectedEof, "peer closed connection without sending TLS close_notify"

Possible fixes:

• The pipeline sends close_notify on shutdown.
• The API server treats an unexpected EOF from the pipeline as the end of the stream.

Impact:

• HTTPS and mTLS are rolled back on the CI instances (cloud-deployments#491) until this is fixed.
• try.feldera.com runs with the same HTTPS setup, so a listener there probably breaks the same way when its pipeline stops.
  Related: feldera/cloud#1449 (ChunkedEncodingError during platform upgrades, not TLS).

Activity

1. jyotshnayaparla-00

   self-assigned this

   on Oct 30, 2026
2. jyotshnayaparla-00

   added the

   Bug

   issue type on Oct 30, 2026
3. blp commented on Oct 30, 2026

   

   blp

   on Oct 30, 2026

   Member

   More actions

   The pipeline sends close_notify on shutdown.

   Is that possible? I think that force-stop kills the pipeline process abruptly.
4. gz commented on Oct 31, 2026

   

   gz

   on Oct 31, 2026

   Contributor

   More actions

   I think this most likely is caused by an ingress being replaced while the test is running or a control plane service being replaced/upgraded, thus something dropping the connection on the path
   if it were really caused by a force stop it should be easy to verify because it's deterministic (& you can reproduce it right now), in this case the test should handle it

   If it turns out to be deterministic then either the test is wrong (and listens too long) and/or needs to be adjusted to handle this failure
5. jyotshnayaparla-00 commented on Nov 2, 2026

   

   jyotshnayaparla-00

   on Nov 2, 2026

   ContributorAuthor

   More actions

   Is that possible? I think that force-stop kills the pipeline process abruptly.

   you're right. Force stop kills the process, so the pipeline can't send close_notify.

   That leaves:

   • The API server treats an unexpected EOF from the pipeline as the end of the stream, at least after a stop was requested. Over plain HTTP the same kill ends the stream, so this would make HTTPS behave the same way.
   • Or the test stops expecting a clean end after stop(force=True).

   @gz on the ingress idea: it failed 3 out of 3 on both arches, in two separate merge queue runs (https://github.com/feldera/feldera/actions/runs/41134745157 and https://github.com/feldera/feldera/actions/runs/41137819254). Nothing was upgraded while those runs were going. It passed again right after we turned HTTPS off (https://github.com/feldera/feldera/actions/runs/41139880352). The error is also logged on the API server's connection to the pipeline, not on the client side. So it looks deterministic and tied to TLS. I can turn HTTPS on for a test instance and repro it again if that helps.

Issue actions