GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — [connectors] Log pipeline configuration settings that have no effect by blp · Pull Request #7659 · feldera/feldera

Conversation

@blp

@blp blp commented Oct 27, 2026

Copy link
Copy Markdown
Member

At startup, the pipeline now prints each key in its configuration file that has no effect, such as a misspelled setting:

Ignoring unknown pipeline configuration setting 'inputs.t.unnamed-0.transport.config.bogus'.

Why not serde_ignored

serde_ignored cannot see keys inside #[serde(flatten)] structs. PipelineConfig flattens RuntimeConfig, and connector configs flatten too, so it would miss most settings that users write.

How it works

crates/feldera-types/src/config/ignored.rs:

  1. Deserialize the config, serialize it again, and list each input key that is missing from the output. A key can be missing because it is unknown, an alias (max_rss_mb), or a default that serialization skips (distributed: false).
  2. For each such key, replace its value with a probe string. If the result deserializes to the same config, nothing reads the key. A typed setting rejects the probe. A string setting accepts it, but then the output changes.

Limits

  • Does not look inside format config (opaque JSON) or the other_options catch-all maps. Unknown dev_tweaks are already logged in crates/dbsp/src/circuit/runtime.rs.
  • Structs with deny_unknown_fields (for example datagen) already fail on unknown keys.

Testing

New unit tests in ignored.rs. With the probe filter disabled, the alias and explicit-default tests fail as expected.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@blp blp self-assigned this Oct 27, 2026
@blp blp added connectors Issues related to the adapters/connectors crate QA Testing and quality assurance rust Pull requests that update Rust code labels Oct 27, 2026
@blp
blp requested a review from bestouff October 27, 2026 18:19
bestouff
bestouff approved these changes Oct 27, 2026
View reviewed changes

@bestouff bestouff left a comment

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, I have a smallish comment on a log.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread crates/connectors/src/server.rs Outdated
if let Ok(value) = serde_json::from_str(&string) {
for setting in ignored_settings::<PipelineConfig>(&value) {
// The logger is not running yet.
eprintln!("Ignoring unknown pipeline configuration setting '{setting}'.");

@bestouff bestouff Oct 27, 2026

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not the first eprintln!() in the code, but I'm wondering if it's not possible to collect and display those logs with warn!() once the logger is initialized ?

Moreover I'd add something like "remove it or fix it".

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@blp blp Oct 28, 2026

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done: messages from before logger initialization now go to a queue and come out through the logger (warn! for this one) after it starts, and the message now says to remove the setting or correct its name.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

bestouff reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

@blp blp Oct 28, 2026

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update: since the coordinator's version can differ from the pipeline's, a setting may be valid but newer than this pipeline, so the message now says "which this version of the pipeline does not recognize (it may be misspelled, or newer than this pipeline)" instead of telling the user to remove it.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

bestouff reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
@blp
blp force-pushed the log-ignored-config-settings branch 2 times, most recently from 733cf15 to d509e01 Compare October 28, 2026 01:20
@blp
[connectors] Log pipeline configuration settings that have no effect
8e374d1
At startup, the pipeline now prints each key in its configuration file
that deserialization ignores, such as a misspelled setting.

serde_ignored cannot see keys inside #[serde(flatten)] structs, and
PipelineConfig flattens RuntimeConfig, so this uses a round trip: keys
that disappear on deserialize/serialize are probed with a value that no
setting accepts, which tells unknown keys apart from aliases and
skipped default values.

Signed-off-by: Ben Pfaff <[email protected]>
@blp
blp force-pushed the log-ignored-config-settings branch from d509e01 to 6d09017 Compare October 28, 2026 01:27
@blp
[connectors] Log startup messages through the logger
1c6f22b
Messages from before logger initialization, such as unknown
configuration settings, went to stderr with eprintln!. Now they wait in
a queue and go to the logger, with a log level, once it is initialized.
If run_server returns before that, the queued messages, and the error
that stopped startup, go to stderr as before.

Signed-off-by: Ben Pfaff <[email protected]>
@blp
blp force-pushed the log-ignored-config-settings branch from 6d09017 to 1c6f22b Compare October 28, 2026 17:11
@blp
blp enabled auto-merge October 28, 2026 17:11
@bestouff
bestouff disabled auto-merge October 28, 2026 18:08
@bestouff
bestouff added this pull request to the merge queue Oct 28, 2026
@bestouff

bestouff commented Oct 28, 2026

Copy link
Copy Markdown
Contributor

(auto-merge still broken)

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@blp

blp commented Oct 28, 2026

Copy link
Copy Markdown
Member Author

(auto-merge still broken)

I filed yet another support ticket with GitHub about that. I expect they'll fail to fix it yet again.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bestouff

bestouff commented Oct 28, 2026

Copy link
Copy Markdown
Contributor

They will reliably fail to address it again.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 29, 2026

Uh oh!

There was an error while loading. Please reload this page.

@blp
[feldera-types] Sort ignored settings in tests.
4e393c2
The order of ignored settings follows serde_json::Map.  Without the
preserve_order feature, keys are sorted.  With it, keys keep input
order.  The pipeline-manager crate enables preserve_order, so in a
workspace build the unknown_settings_are_ignored test got the keys in
input order and failed.  Now the test sorts the result, so it passes
either way.

Signed-off-by: Ben Pfaff <[email protected]>
@blp
blp added this pull request to the merge queue Oct 29, 2026
Merged via the queue into feldera:main with commit 9a8f1ac Oct 29, 2026
1 check passed

Uh oh!

There was an error while loading. Please reload this page.

@blp
blp deleted the log-ignored-config-settings branch October 29, 2026 17:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@bestouff bestouff bestouff approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

@blp blp

Labels

connectors Issues related to the adapters/connectors crate QA Testing and quality assurance rust Pull requests that update Rust code

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

2 participants

@blp @bestouff
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.