GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — feat(api_core): add method-level terminal error logging in _GapicCallable by chalmerlowe · Pull Request #19012 · googleapis/google-cloud-python

Conversation

@chalmerlowe

@chalmerlowe chalmerlowe commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds Tier 3 (method-level) terminal error and cancellation logging in _GapicCallable when experimental Python error logging (GOOGLE_SDK_EXPERIMENTAL_PYTHON_LOGGING_ENABLED) is enabled:

  • Adds _GapicCallable._emit_method_error_log and _GapicCallable._report_method_error to emit structured ERROR logs on terminal method failures (including ErrorInfo metadata, rpcName, rpc.system.name, trace correlation fields, and exception.stacktrace) and DEBUG logs without stack traces on RPC or Future/Task cancellation (exceptions.Cancelled, concurrent.futures.CancelledError, asyncio.CancelledError).
  • Updates _GapicCallable._trace_span to delegate span error enrichment and method error logging to self._report_method_error(exc, span) for both synchronous and asynchronous GAPIC callables while failing open on telemetry or logger errors.
  • Adds unit tests in tests/unit/gapic/test_method.py and tests/asyncio/gapic/test_method_async.py.

Bug: b/479827845

Verification

  • Verified locally with nox -s unit-3.14 format lint in packages/google-api-core.
  • Relying on GitHub Actions CI for full multi-version and cross-environment verification.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

gemini-code-assist[bot] reacted with eyes emoji
All reactions
  • 👀 1 reaction
gemini-code-assist[bot]
gemini-code-assist Bot reviewed Nov 6, 2026
View reviewed changes

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces structured error and cancellation logging for GAPIC method calls in both synchronous and asynchronous contexts. It adds _emit_method_error_log and _report_method_error to enrich OpenTelemetry spans and log terminal RPC failures or cancellations, along with comprehensive tests. The review feedback suggests wrapping _report_method_error in a try...except block to ensure telemetry errors fail open without disrupting the main RPC flow, and logging any exceptions as warnings. Additionally, it recommends avoiding the mutation of the passed error_attrs dictionary in _emit_method_error_log to prevent unintended side-effects.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread packages/google-api-core/google/api_core/gapic_v1/method.py
Comment on lines +333 to +364
error_logging_enabled = client_logging.is_error_logging_enabled(
self._client_options
)
if span is None and not error_logging_enabled:
return

error_attrs = _observability.build_telemetry_error_attributes(
exc, include_exception=error_logging_enabled
)
status_code = error_attrs.get("rpc.response.status_code", "")
is_cancelled = status_code in ("CANCELLED", "CancelledError") or isinstance(
exc, asyncio.CancelledError
)
if is_cancelled:
error_attrs["rpc.response.status_code"] = "CANCELLED"

# Enrich active Tier 3 method span
if span is not None:
try:
for k, v in error_attrs.items():
if not k.startswith("exception."):
span.set_attribute(k, v)
# Intentionally catch broad Exception so telemetry span enrichment
# fails open and never disrupts the caller's RPC flow.
except Exception:
pass

# Emit L2/L3 Terminal Method Error Log
if error_logging_enabled:
self._emit_method_error_log(
exc, span, error_attrs, is_cancelled, logger=logger
)

@gemini-code-assist gemini-code-assist Bot Nov 6, 2026

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

If client_logging.is_error_logging_enabled or _observability.build_telemetry_error_attributes raises an exception, it will propagate out of _report_method_error and disrupt the caller's RPC flow. Wrapping the entire body of _report_method_error in a try...except Exception block ensures that telemetry and logging errors always fail open. To avoid silently masking underlying issues, log the exception as a warning instead of silently passing.

        try:
            error_logging_enabled = client_logging.is_error_logging_enabled(
                self._client_options
            )
            if span is None and not error_logging_enabled:
                return

            error_attrs = _observability.build_telemetry_error_attributes(
                exc, include_exception=error_logging_enabled
            )
            status_code = error_attrs.get("rpc.response.status_code", "")
            is_cancelled = status_code in ("CANCELLED", "CancelledError") or isinstance(
                exc, asyncio.CancelledError
            )
            if is_cancelled:
                error_attrs["rpc.response.status_code"] = "CANCELLED"

            # Enrich active Tier 3 method span
            if span is not None:
                for k, v in error_attrs.items():
                    if not k.startswith("exception."):
                        span.set_attribute(k, v)

            # Emit L2/L3 Terminal Method Error Log
            if error_logging_enabled:
                self._emit_method_error_log(
                    exc, span, error_attrs, is_cancelled, logger=logger
                )
        except Exception as e:
            logger.warning("Failed to report method error: %s", e)
References
  1. Avoid broad except Exception: blocks that silently return None. Instead, log the exception (e.g., using logger.warning) to aid in debugging and prevent masking underlying issues.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@chalmerlowe chalmerlowe Nov 6, 2026

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review!

  1. Isolating span.set_attribute in its own try...except Exception: pass block (separate from _emit_method_error_log's try...except Exception: pass block) is intentional so that a failing custom Span implementation cannot skip terminal error log emission (added a clarifying comment in ca5d711).
  2. Keeping is_error_logging_enabled outside the try...except block preserves the fail-loud FeatureGatingError contract if an unsupported logger_provider option is passed.
  3. Using except Exception: pass rather than logger.warning matches the fail-open observability contract across _observability.py so internal telemetry errors never emit unexpected warnings into user logs.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread packages/google-api-core/google/api_core/gapic_v1/method.py Outdated
Comment on lines +287 to +294
if is_cancelled:
error_attrs.pop("exception.stacktrace", None)
extra_payload: dict[str, Any] = {
"rpcName": self._method_name,
"rpc.system.name": self._rpc_system,
**error_attrs,
**_observability.extract_trace_context(span),
}

@gemini-code-assist gemini-code-assist Bot Nov 6, 2026

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Mutating the error_attrs dictionary passed as an argument can lead to unexpected side-effects if the caller or other methods reuse it. It is safer to construct extra_payload first and then pop "exception.stacktrace" from extra_payload directly.

Suggested change
if is_cancelled:
error_attrs.pop("exception.stacktrace", None)
extra_payload: dict[str, Any] = {
"rpcName": self._method_name,
"rpc.system.name": self._rpc_system,
**error_attrs,
**_observability.extract_trace_context(span),
}
extra_payload: dict[str, Any] = {
"rpcName": self._method_name,
"rpc.system.name": self._rpc_system,
**error_attrs,
**_observability.extract_trace_context(span),
}
if is_cancelled:
extra_payload.pop("exception.stacktrace", None)

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@chalmerlowe chalmerlowe Nov 6, 2026

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ca5d711 — updated _emit_method_error_log to pop "exception.stacktrace" from extra_payload rather than mutating error_attrs in-place.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@chalmerlowe chalmerlowe mentioned this pull request Nov 6, 2026
feat(api_core): implement L2/L3 method error logging and L4 transport attempt logging #19000
Closed
@chalmerlowe
chalmerlowe force-pushed the feat/otel-logging-m3-s5-method-error-logging branch from 6389b5c to ca5d711 Compare November 6, 2026 16:51
@chalmerlowe
feat(api_core): add method-level terminal error logging in _GapicCall…
e489dd7
…able
@chalmerlowe
chalmerlowe force-pushed the feat/otel-logging-m3-s5-method-error-logging branch from ca5d711 to e489dd7 Compare November 10, 2026 00:17
@chalmerlowe
chalmerlowe force-pushed the feat/otel-logging-m3-s4-http-attempt-logging branch from 5aea0b9 to a82e931 Compare November 10, 2026 00:17

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments
Reviewers whose approvals may not affect merge requirements

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

1 participant

@chalmerlowe
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.