GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — fix(gcs): close writer on write errors and handle empty write channel by Tyagiquamar · Pull Request #23706 · cloudquery/cloudquery

Conversation

@Tyagiquamar

@Tyagiquamar Tyagiquamar commented Nov 2, 2026

Copy link
Copy Markdown

Summary

Fixes the GCS destination write path so the *storage.Writer is always released, including when WriteHeader, WriteContent or WriteFooter fails, and so a sync that produces no messages no longer touches a nil handle.

Root Cause

WriteTable in plugins/destination/gcs/client/write.go created the object writer lazily on the first message and returned straight from three error paths without closing it:

h, err = c.Client.WriteHeader(w, table)
if err != nil {
    return err // w.Close() never called
}
if err := h.WriteContent(...); err != nil {
    return err // w.Close() never called
}
if err := h.WriteFooter(); err != nil {
    return err // w.Close() never called
}

Without Close, the in-progress upload and its HTTP connection stay alive until garbage collection, so repeated failing syncs accumulate leaked resources.

Separately, when the message channel was empty the loop body never ran, h stayed nil, and h.WriteFooter() panicked on the nil interface value.

Fix

The write loop moved into an internal writeTable function that takes a writer factory, which keeps the production path identical while making the logic testable with stubs:

  • Every error path now returns through closeOnError, which closes the writer and joins any close error with the original write error.
  • The writer is wrapped in a small onceCloser so Close is idempotent and the underlying *storage.Writer can only be closed once.
  • An empty channel returns nil before touching the handle. The successful path is unchanged: header, content per message, footer, then Close.

No public API or configuration changes.

Test Plan

New unit tests in plugins/destination/gcs/client/write_test.go use a stub types.FileType, stub handle and stub writer, so no GCS credentials or network are needed:

  • TestWriteTableClosesWriterOnError covers failing WriteHeader, WriteContent and WriteFooter, asserting each returns the write error and closes the writer exactly once.
  • TestWriteTableJoinsCloseError asserts a close failure is joined with the write error.
  • TestWriteTableEmptyChannel asserts an empty channel returns nil without creating a writer (this panicked before the fix).
  • TestWriteTableSuccess asserts multiple messages produce one content write each, one footer and one close.
  • TestOnceCloserClosesUnderlyingWriterOnce asserts repeated Close calls only reach the underlying writer once.

Commands run from plugins/destination/gcs with Go 1.27.3:

  • go build ./... : pass
  • go vet ./client/ : pass
  • go test ./client/ -run 'TestWriteTable|TestOnceCloser' -v -count=1 : all 6 tests pass (the three error subtests, close-error join, empty channel, success, and once-closer idempotency)
  • gofmt check on both changed files : clean

Fixes #23173

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@Tyagiquamar
fix(gcs): close writer on write errors and handle empty write channel
4fc1b0e
WriteTable leaked the *storage.Writer when WriteHeader, WriteContent or
WriteFooter failed, and closed a nil handle when no messages arrived.
Close the writer on every error path through a small idempotent wrapper and
return early when the channel is empty.
@Tyagiquamar
Tyagiquamar requested a review from a team as a code owner November 2, 2026 09:22
@Tyagiquamar
Tyagiquamar requested a review from murarustefaan November 2, 2026 09:22
@github-actions github-actions Bot added the area/plugin/destination/gcs label Nov 2, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@murarustefaan murarustefaan Awaiting requested review from murarustefaan murarustefaan is a code owner automatically assigned from cloudquery/backend

At least 1 approving review is required to merge this pull request.

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

area/plugin/destination/gcs

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

plugins/destination/gcs/client/write.go: *storage.Writer not closed on WriteHeader/WriteContent/WriteFooter errors + nil writer on empty channel

Uh oh!

There was an error while loading. Please reload this page.

1 participant

@Tyagiquamar
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.