GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — Add worker resource and queue metrics to OTel instrumentation by Mohammed0tarek · Pull Request #613 · taskiq-python/taskiq

Conversation

@Mohammed0tarek

@Mohammed0tarek Mohammed0tarek commented May 2, 2026

Copy link
Copy Markdown
Contributor

What this adds

This extends the existing OpenTelemetryMiddleware with four new metrics that give
visibility into what a worker process is doing at runtime, not just what tasks it has
finished.

Worker resource utilization (observable gauges, worker process only):

  • worker_cpu_utilization — CPU usage percentage of the worker process
  • worker_memory_utilization — RSS memory in bytes

Both gauges are gaurded by broker.is_worker_process so they stay silent in
client/producer processes.

Worker queue depth (UpDownCounters):

  • worker_active_tasks — number of tasks currently executing, incremented in
    pre_execute and decremented in post_execute. Carries a task_name attribute so
    you can see per-task concurrency.
  • worker_prefetched_tasks — number of tasks sitting in the internal prefetch queue
    (fetched from the broker, waiting for a worker slot). Driven by two new hooks,
    on_prefetch_queue_add and on_prefetch_queue_remove, called from the Receiver at
    the right points — after a real message is enqueued, and after it is dequeued but
    only once the QUEUE_DONE sentinel is handled so it never fires on shutdown signals.

Why UpDownCounter for the queue metrics

I didn't want to mess with the queue definition (they are local variables) so gauges where not the right thing
since they require polling and they are not an attribute in the receiver class.

Tests

Added tests for all four metrics to tests/opentelemetry/test_metrics.py:

  • test_active_tasks_counter — verifies net-zero value after task completion and
    task_name attribute correctness
  • test_prefetch_queue_counter — calls the hooks directly (the Receiver is not
    exercised by InMemoryBroker.kiq()) and asserts the counter value
  • test_worker_resource_metrics_when_worker_process — sets is_worker_process = True and
    triggers an OTel collection cycle to verify the gauges yield observations.
    If feel like this is a slight workaround but I hope it is fine.
  • test_metrics_exist updated to include worker_active_tasks

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
s3rius
s3rius approved these changes May 4, 2026
View reviewed changes
Mohammed0tarek added 3 commits May 4, 2026 15:21
@Mohammed0tarek @s3rius
feat: add OTel metrics to OpenTelemetryMiddleware
65223b7
Add counters and histograms to OpenTelemetryMiddleware:
- tasks_sent: producer-side counter per task name
- task_success / task_errors: consumer-side counters with retry_error attribute
- task_execution_time: histogram using result.execution_time
- task_wait_time: histogram measuring queue time from send to receive via UTC timestamps in labels

Add tests covering all instruments, retry_error attribute paths, and queue time correctness.
@Mohammed0tarek @s3rius
Added cpu and memory utilization guages
0670feb
@Mohammed0tarek @s3rius
feat: add worker queue and active task OTel metrics
4188421
- Add worker_active_tasks UpDownCounter driven by pre/post_execute hooks
- Add worker_prefetched_tasks UpDownCounter via on_prefetch_queue_add/remove hooks in receiver
- Add worker_cpu_utilization and worker_memory_utilization observable gauges (worker process only)
- Add tests for all new metrics
@s3rius
s3rius force-pushed the observability/extra_open_telemetry_metrics branch from 8bf420d to 4188421 Compare May 4, 2026 13:21
@codecov

codecov Bot commented May 4, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 92.45283% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 79.76%. Comparing base (eaed387) to head (4188421).
⚠️ Report is 3 commits behind head on master.

Files with missing lines Patch % Lines
taskiq/receiver/receiver.py 33.33% 4 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #613      +/-   ##
==========================================
+ Coverage   79.37%   79.76%   +0.38%     
==========================================
  Files          69       69              
  Lines        2473     2525      +52     
==========================================
+ Hits         1963     2014      +51     
- Misses        510      511       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@s3rius
s3rius merged commit 3f1d0d1 into taskiq-python:master May 4, 2026
35 checks passed

Uh oh!

There was an error while loading. Please reload this page.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@s3rius s3rius s3rius approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

2 participants

@Mohammed0tarek @s3rius
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.