GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — Add GCS sink destination support to define_sink_pipe by sandshoes · Pull Request #30 · tinybirdco/tinybird-sdk-python

Conversation

@sandshoes

@sandshoes sandshoes commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

Summary

  • define_sink_pipe currently only supports Kafka and S3 destinations. Adds GCS as a third sink destination, matching the Forward CLI's gcs_hmac export service, for https://linear.app/tinybird/issue/PROTM-2262/python-sdk-add-gcs-sink-destination-support
  • New GCSSinkConfig mirrors S3SinkConfig's field set (bucket_uri, file_template, format, schedule, strategy, compression) but references a GCSConnectionDefinition instead of S3ConnectionDefinition.
  • Since GCS and S3 sinks share an identical EXPORT_* directive shape in .pipe datafiles, the generator emits an explicit EXPORT_SERVICE gcs_hmac directive for GCS sinks so the migration parser can disambiguate them from S3 sinks on reverse-parse (S3 remains the default when EXPORT_SERVICE is omitted, preserving existing behavior/datafiles).
  • Updated the migrate runner's sink/connection compatibility check, which previously compared pipe.sink.service directly against the connection's _connectionType — this would have incorrectly rejected a valid gcs_hmac sink pointed at a gcs connection. It now maps gcs_hmac → gcs before comparing.
  • Also wired SinkGCSModel through migrate/emit_ts.py for TypeScript re-emission parity with S3.

Changes

  • src/tinybird_sdk/schema/pipe.py: GCSSinkConfig dataclass, SinkConfig union, _normalize_sink_config GCS branch.
  • src/tinybird_sdk/generator/pipe.py: emit EXPORT_SERVICE gcs_hmac + shared export directives for GCSSinkConfig.
  • src/tinybird_sdk/migrate/types.py: SinkGCSModel.
  • src/tinybird_sdk/migrate/parse_pipe.py: accept gcs_hmac as an EXPORT_SERVICE value; build SinkGCSModel when explicitly set, default to s3 otherwise (unchanged behavior for existing datafiles).
  • src/tinybird_sdk/migrate/runner.py: fix sink/connection type compatibility check for gcs_hmac.
  • src/tinybird_sdk/migrate/emit_ts.py, schema/__init__.py, migrate/__init__.py: wiring/exports.
  • README.md, CHANGELOG.md: documented the new sink option.
  • tests/test_gcs_sink_destination.py (new, 8 tests): schema validation, datafile generation, parse round-trip (explicit service + default-to-s3 fallback + unsupported service rejection), runner compatibility (accept/reject), and TS emission.

Checklist

  • CI is green (lint, typecheck, test)
  • pre-commit run --all-files passes locally (gitleaks step fails in this sandbox on an unrelated SSL/network error fetching its pre-commit environment, not a code issue)
  • Tests were added or updated when behavior changed
  • Public API / typing changes were reviewed
  • Documentation was updated (README.md)
  • Breaking changes are clearly documented (none — purely additive; default inference for ambiguous existing sinks is unchanged)
  • CHANGELOG.md was updated when user-facing behavior changed

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Tommy Healy @claude
Add GCS sink destination support to define_sink_pipe
ea62b36
Matches the Forward CLI's gcs_hmac export service: GCSSinkConfig
mirrors S3SinkConfig's fields (bucket_uri/file_template/format/
schedule/strategy/compression) but references a GCSConnectionDefinition.
Since GCS and S3 sinks share identical EXPORT_* directives, the
generator emits an explicit EXPORT_SERVICE gcs_hmac line to
disambiguate on reverse parse, and the migrate runner's sink/connection
compatibility check now maps gcs_hmac sinks to "gcs" connections.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
@sandshoes
sandshoes requested a review from jalbertsr October 2, 2026 09:28
jalbertsr
jalbertsr approved these changes Oct 7, 2026
View reviewed changes

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@jalbertsr jalbertsr jalbertsr approved these changes
Reviewers whose approvals may not affect merge requirements

At least 1 approving review is required to merge this pull request.

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

2 participants

@sandshoes @jalbertsr
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.