GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — [types,connectors] Show resolved secrets in diagnostics as references by bestouff · Pull Request #7787 · feldera/feldera

Conversation

@bestouff

@bestouff bestouff commented Nov 2, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

Part of feldera/cloud#2136. Stacked on #7786 (cloud-2136-diagnostics).

Register every resolved Secret value with its original reference and replace matching diagnostic text at the production sinks: text/JSON logs, error responses, connector error lists and panic payloads. This supplements the connector-specific source fixes. SQL and operational configuration are unchanged.

The registry matches trimmed values of at least eight bytes at token boundaries and covers percent-encoded, decoded, debug-escaped and JSON-escaped forms. Public-field references remain protected; common words, short values and further transformations retain the documented limitations.

Panic filtering uses a nonblocking registry read. If registration holds the write lock, the payload is omitted while its location and backtrace remain, avoiding a deadlock in the panic hook.

Validation covers reference resolution, both response implementations, production text/JSON filtering, restored connector errors, and engine connection/format failures. Concurrent registration is tested; the real panic hook runs in an isolated child process. Removing the nonblocking read reproduces the deadlock, and disconnecting the hook filter fails its regression. Formatting and Clippy with warnings denied pass on the full stack.

Enterprise/Fremont validation passed; results are recorded in feldera/cloud#2160. Historical checkpoint provenance and inline SQL/bundle policy remain separate follow-up work.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@bestouff bestouff mentioned this pull request Nov 2, 2026
[pipeline-manager] Redact connection errors and bundle failures #7788
Open
mythical-fred-oss[bot]
mythical-fred-oss Bot reviewed Nov 2, 2026
View reviewed changes
Comment thread crates/feldera-types/src/secret_redaction.rs Resolved

Uh oh!

There was an error while loading. Please reload this page.

mythical-fred-oss[bot]
mythical-fred-oss Bot reviewed Nov 2, 2026
View reviewed changes
Comment thread crates/connectors/src/panic.rs Resolved

Uh oh!

There was an error while loading. Please reload this page.

mythical-fred-oss[bot]
mythical-fred-oss Bot reviewed Nov 2, 2026
View reviewed changes
Comment thread crates/feldera-observability/src/json_logging.rs Resolved

Uh oh!

There was an error while loading. Please reload this page.

@mythical-fred-oss

mythical-fred-oss Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

What I ran

Command Result
cargo test -p feldera-types secret (29 tests) ✅
cargo test -p feldera-observability json_logging ✅
cargo test -p dbsp_connectors --lib (the 4 new redaction tests) ✅
cargo test -p static-connectors --test connector_suites (status_error_shows_the_secret…, status_format_error_omits_short_secret) ✅
pre-commit run --files <changed> ✅

Gates: tests, docs, changelog, license (aho-corasick is MIT/Unlicense) and FIPS all pass. There is no unsafe code, and I found no flaky patterns. The env-var test re-executes itself as a child process rather than mutating the environment, which is a nice touch.

Missing tests (medium):

  • Over-redaction. Nothing checks that a secret which is a common word (≥8 bytes) leaves unrelated text alone (see the inline comment).
  • Text log path end to end. Nothing covers the non-JSON pipeline log through init_pipeline_logging_with_redaction. Only RedactingMakeWriter is tested, with a fake redactor.
  • Concurrency. Nothing covers register racing with redact, or a panic raised inside register while the write lock is held (the panic hook then calls redact and needs the read lock).
  • Checkpoint restore. No test covers a connector error list restored from a checkpoint that still holds the raw value.

Observation (low): the "JSON-escaped" form in the description is really the Rust {:?} form. For control characters the two differ: serde_json writes \u0001, while Debug writes \u{1}. A value embedded with serde_json::to_string and containing such a character therefore slips through.

Otherwise this looks solid as a second layer.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bestouff
bestouff marked this pull request as draft November 2, 2026 19:47
@bestouff
bestouff added this pull request to stack #7791 November 2, 2026 20:09
@bestouff
bestouff force-pushed the cloud-2136-redaction branch 3 times, most recently from 96233ea to 291eb8b Compare November 3, 2026 13:22
@bestouff
bestouff force-pushed the cloud-2136-redaction branch from 46d747e to c5bae41 Compare November 3, 2026 17:49
@bestouff
bestouff marked this pull request as ready for review November 3, 2026 18:44
@bestouff
bestouff requested a review from mihaibudiu November 3, 2026 18:44
mythical-fred-oss[bot]
mythical-fred-oss Bot reviewed Nov 3, 2026
View reviewed changes
Comment thread crates/feldera-types/src/secret_redaction.rs Resolved

Uh oh!

There was an error while loading. Please reload this page.

mythical-fred-oss[bot]
mythical-fred-oss Bot reviewed Nov 3, 2026
View reviewed changes
Comment thread crates/feldera-types/src/secret_redaction.rs Resolved

Uh oh!

There was an error while loading. Please reload this page.

mythical-fred-oss[bot]
mythical-fred-oss Bot reviewed Nov 3, 2026
View reviewed changes
Comment thread docs.feldera.com/docs/changelog.md Resolved

Uh oh!

There was an error while loading. Please reload this page.

mythical-fred-oss[bot]
mythical-fred-oss Bot reviewed Nov 3, 2026
View reviewed changes
Comment thread docs.feldera.com/docs/connectors/secret-references.md Resolved

Uh oh!

There was an error while loading. Please reload this page.

@mythical-fred-oss

mythical-fred-oss Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

What I ran (debug build)

Command Result
cargo test -p feldera-types secret 34 passed
cargo test -p feldera-observability passed
dbsp_connectors lib: panic::, stats stored/restored, server::error tests passed
static-connectors --test connector_suites status_error_shows… status_format_error_omits… 2 passed
scratch QA tests on SecretRedactor 2 leaks found, see inline comments

The design looks sound: the panic hook's try_read avoids the deadlock, env refs are excluded, and there are no unsafe blocks. aho-corasick is MIT/Unlicense, which is fine.

Missing (medium): uncovered cases.

  • A secret preceded by an escape sequence in Debug-formatted text-log fields (\n, \t) or by percent-encoded text (%3D). This currently leaks (inline comment).
  • An overlapping shorter form after a longer match is rejected at a token boundary, e.g. the URL password inside a URI followed by a word character. This currently leaks (inline comment).
  • Text (non-JSON) logging through init_pipeline_logging_with_redaction. It is tested only with a fake Replace redactor, never with the real registry and the {value:?} field formatter.
  • status_error_shows_the_secret_reference_not_the_value would likely pass without this PR, because [connectors,types,manager] Keep connector credentials out of diagnostics #7786 already omits the URI at the source. To prove the registry is the layer under test, assert that ${secret:kubernetes:pg/uri} appears in the body.

Pitfall (low): a panic while register holds the write lock still reaches the DBSP hook and the logging layer, which take a blocking read(). The expect is the only panic site, so this is unlikely.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bestouff
bestouff force-pushed the cloud-2136-redaction branch from c5bae41 to f248b57 Compare November 3, 2026 20:27
@bestouff
bestouff force-pushed the cloud-2136-redaction branch from f248b57 to ad3e1ca Compare November 5, 2026 15:54
@bestouff
bestouff force-pushed the cloud-2136-redaction branch from ad3e1ca to ef43d4f Compare November 5, 2026 17:00
mihaibudiu
mihaibudiu reviewed Nov 6, 2026
View reviewed changes
Comment thread crates/connectors/src/panic.rs Outdated
N_PANICS.fetch_add(1, Ordering::Relaxed);
prev(info);
// A payload that copied a resolved secret is printed as its
// reference; any other payload keeps the default printer.

@mihaibudiu mihaibudiu Nov 6, 2026

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is a payload "reference"?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@bestouff bestouff Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a sort-of "path" to find the secret, e.g. ${secret:kubernetes:pg/uri}; I'll rewrite that.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread crates/feldera-observability/src/json_logging.rs Outdated
}

/// Rewrites a formatted log line before it is written.
pub trait LineRedactor: Send + Sync + 'static {

@mihaibudiu mihaibudiu Nov 6, 2026

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know if this belongs to json_logging.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@bestouff bestouff Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, in its own module it goes.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread crates/connectors/src/server.rs Outdated
secret_redactor().redact_bytes(line)
}

/// The registry's own walk: nothing to do while no secret is resolved.

@mihaibudiu mihaibudiu Nov 6, 2026

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

je ne comprends rien

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@bestouff bestouff Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Use the registry's JSON walk, not the default one of the trait: it does nothing when no secret is registered." should be better

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread docs.feldera.com/docs/connectors/secret-references.md Outdated
| Output | What is shown instead of a resolved value |
|---|---|
| Pipeline log | the reference, `${secret:...}`, for the value as resolved; `[REDACTED ${secret:...}]` for a derived form of it, such as a percent-encoded or escaped copy or the password of a URL value |
| `deployment_error`, pipeline events, `/status` | the same two forms, from a pipeline on a runtime with this fix |

@mihaibudiu mihaibudiu Nov 6, 2026

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what are the "same two forms"? each row in this table should be self-contained. Someone may not read all rows.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@bestouff bestouff Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/status and /stats (no potential for confusion at all here). I'll write both.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread docs.feldera.com/docs/connectors/secret-references.md Outdated

The following limits apply:

- The pipeline replaces values of 8 bytes or more, where they stand as a

@mihaibudiu mihaibudiu Nov 6, 2026

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the pipeline? the logging system perhaps.
replaces with what?
This is a rather crude heuristic.
Can't we do better?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@bestouff bestouff Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is the pipeline's log and status output, and the value is replaced by its reference.
Yes, it's crude but it's only the last layer. It runs after #7786 (removes the values from the error messages), #7801 (registers the fields that each connector declares as credentials), and the LibraryError PR that sanitizes the errors of the client libraries.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@bestouff bestouff Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

... and once all of this lands I'll see for something more type-based as a followup

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@bestouff
bestouff force-pushed the cloud-2136-redaction branch from ef43d4f to 2da2fb3 Compare November 12, 2026 12:51
bestouff added 2 commits November 12, 2026 14:44
@bestouff
[types,connectors] Show resolved secrets in diagnostics as references
8ae3cd2
A user who may reference a Kubernetes Secret need not be allowed to
read its value. Register resolved secret values and their common derived
forms in a process-wide Aho-Corasick registry as a second layer behind
the source fixes. Values shorter than 8 trimmed bytes and environment
references are excluded to avoid rewriting unrelated diagnostics.

Apply redaction to both ErrorResponse types, connector error history,
panic payloads, and text logs. Redact JSON fields before serialization
adds another escaping layer. Include object keys and escaped URL
password components without changing operational configuration.

Test the real JSON formatter with quoted and backslash-containing
secrets. Pipeline tests cover a failed referenced Postgres connection
and a format error with a short referenced secret. The environment
reference test configures a child process instead of mutating the
environment of the parallel test process.

Related: feldera/cloud#2136

Signed-off-by: Xavier Bestel <[email protected]>
@bestouff
[connectors] Avoid locking the secret registry in panic hooks
ba2fbee
A panic during registration can run while the registry write lock is
held. Use a nonblocking read for panic payloads and omit the payload when
the registry is unavailable, preserving the location and backtrace.

Exercise concurrent registration and replace the ignored panic-hook test
with an isolated subprocess that checks both counting and redaction.
Removing the nonblocking read reproduces the deadlock; disconnecting the
hook filter fails the subprocess regression.

Signed-off-by: Xavier Bestel <[email protected]>
@bestouff
bestouff force-pushed the cloud-2136-redaction branch from 2da2fb3 to ba2fbee Compare November 12, 2026 13:46
@bestouff
bestouff requested a review from mihaibudiu November 12, 2026 14:35

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mythical-fred-oss mythical-fred-oss[bot] mythical-fred-oss[bot] left review comments
@mihaibudiu mihaibudiu Awaiting requested review from mihaibudiu

At least 1 approving review is required to merge this pull request.

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

2 participants

@bestouff @mihaibudiu
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.