GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — Optimize ListRedisScheduleSource: replace SCAN with sorted set time index by tmkarthi · Pull Request #121 · taskiq-python/taskiq-redis

Conversation

tmkarthi commented Feb 25, 2026 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

Summary

Replace the SCAN-based discovery of past time keys with a ZRANGEBYSCORE lookup on a new {prefix}:time_index sorted set, eliminating full keyspace scans that strain Redis when there are many keys
Track time keys in the sorted set automatically on add_schedule, and clean up stale entries (older than 5 minutes with empty lists) lazily from delete_schedule, rate-limited to once per minute to avoid excess Redis calls
Fetch past time schedules on every get_schedules call (not just the first run) so that schedules added in a past minute after the previous call are never missed
Pass current_time from get_schedules into _get_previous_time_schedules to prevent duplicate fetches when the minute boundary crosses between the two calls
Add populate_time_index constructor parameter for users upgrading from older versions: set to True once to backfill the index via a one-time SCAN, then False for all subsequent runs
Update README with new Redis key documentation and populate_time_index migration guide

Test plan

Existing tests for cron, interval, timed schedules, removal, and migration all pass
test_time_index_populated_on_add — verifies add_schedule writes to the sorted set
test_time_index_not_eagerly_cleaned_on_delete — verifies no race-prone eager cleanup
test_cleanup_removes_old_empty_entries — stale empty entries older than 5 min are removed
test_cleanup_keeps_non_empty_entries — entries with remaining schedules are preserved
test_cleanup_keeps_recent_empty_entries — entries within 5-min window are not removed
test_past_schedules_found_via_time_index — past schedules discovered via sorted set
test_populate_time_index_from_existing_keys — populate_time_index=True backfills index
test_post_send_triggers_cleanup — full lifecycle: add → get → post_send → cleanup
test_cleanup_rate_limited — cleanup runs at most once per minute
test_cron_and_interval_not_in_time_index — non-time schedules don't pollute the index

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

vvanglro reacted with thumbs up emoji

All reactions

👍 1 reaction

tmkarthi added 5 commits

February 25, 2026 13:20


          feat: add time index functionality to ListRedisScheduleSource

fad3046

- Introduced `populate_time_index` parameter to backfill the time index from existing keys.
- Updated `startup` method to populate the time index if `populate_time_index` is set to True.
- Modified schedule addition and deletion to manage the time index sorted set.
- Added tests to verify time index population and cleanup behavior.


          feat: implement lazy cleanup for time index in ListRedisScheduleSource

da282d3

- Added `_maybe_cleanup_time_index` method to manage time index cleanup at most once per minute.
- Introduced `_cleanup_time_index` method to remove stale entries older than one hour with empty time key lists.
- Updated `delete_schedule` to call `_maybe_cleanup_time_index` for efficient cleanup.
- Enhanced tests to verify the behavior of the new cleanup methods, ensuring proper handling of stale and recent entries.


          feat: enhance _get_previous_time_schedules to accept current_time par…

…ameter

- Updated the _get_previous_time_schedules method to take current_time as an argument, allowing for more precise cutoff calculations.
- Adjusted the logic to use the provided current_time for determining previous schedules, ensuring no overlap with the current window.
- Modified the call to _get_previous_time_schedules in the first run logic to pass the current_time parameter.


          refactor: remove first run logic and update schedule fetching behavior

384c862

- Removed the `_is_first_run` flag to simplify the schedule fetching logic.
- Updated `get_schedules` to fetch past time schedules on every call, ensuring no schedules are missed within the current minute and previous minute.
- Enhanced documentation to reflect the new behavior of schedule retrieval.


          docs: update README and tests for time index cleanup logic

0ed6d06

- Expanded README to include details on interval tasks and the new `{prefix}:time_index` sorted set for tracking schedules.
- Updated cleanup logic in `ListRedisScheduleSource` to remove stale entries older than 5 minutes instead of 1 hour.
- Modified tests to reflect the new 5-minute threshold for cleanup, ensuring accurate verification of the time index behavior.

tmkarthi changed the title ~~feat: replace SCAN with sorted set index for time-based schedule lookups~~ Optimize ListRedisScheduleSource: replace SCAN with sorted set time index

Feb 27, 2026

tmkarthi mentioned this pull request

Feb 27, 2026

Variable self._is_first_run in ListRedisScheduleSource doesn't work #89

Open

tmkarthi mentioned this pull request

Mar 6, 2026

ListRedisScheduleSource runs a full SCAN across the entire Redis keyspace on every update_interval #122

Open

tmkarthi commented Apr 6, 2026

Copy link

Copy Markdown

Author

@s3rius Any update on this?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

s3rius approved these changes

Apr 20, 2026

View reviewed changes

s3rius left a comment •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First of all. Thanks for you patience. It took me a while to get to your PR.
I really like the solution. It's elegant, simple and yet robust.

I only have a single comment about simplifying migration for people after upgrading their taskiq-redis. However, it's a complete minor thing.

Also, I noticed that you use floats for scores. I'm not sure if turning scores into ints would make a big of a change. But could be, since floats are known to worse at comparison and take a bit more memory than ints. Of course it all depends on how redis implements score type.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

Comment thread

taskiq_redis/list_schedule_source.py

		await self._previous_schedule_source.shutdown()
		logger.info("Migration complete")

		if self._populate_time_index:

s3rius Apr 20, 2026

Copy link

Copy Markdown

Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that this can be calculated automatically.

if you were using previous version of this schedule source, you are 100% don't have this key set.

We can just check if the key for time_index is present and in case if it's empty, build the initial index. This will allow us to remove the new argument and make a seamless transition.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

s3rius requested a review from Copilot

April 20, 2026 17:33

Copilot started reviewing on behalf of s3rius

April 20, 2026 17:33

View session

Copilot AI reviewed

Apr 20, 2026

View reviewed changes

Copilot AI left a comment

Copy link

Copy Markdown

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR optimizes ListRedisScheduleSource’s handling of time-based schedules by introducing a Redis sorted-set “time index” to avoid keyspace-wide SCANs when discovering past-due time buckets, and adds a migration/backfill option plus documentation and tests.

Changes:

Add {prefix}:time_index sorted set and switch past schedule discovery from SCAN to ZRANGEBYSCORE.
Maintain the time index on add_schedule, and lazily clean stale empty entries from delete_schedule with rate limiting.
Add populate_time_index to backfill the index on startup for upgrades; update README and expand test coverage.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File	Description
`taskiq_redis/list_schedule_source.py`	Implements the time index, backfill-on-startup option, and lazy cleanup; adjusts past schedule fetching behavior.
`tests/test_list_schedule_source.py`	Adds tests for time index maintenance, lazy cleanup semantics, backfill behavior, and rate limiting.
`README.md`	Documents the new Redis keys and provides a `populate_time_index` migration guide.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

Comment thread

taskiq_redis/list_schedule_source.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread

taskiq_redis/list_schedule_source.py

Comment on lines +150 to +151

		self._last_cleanup_time = now
		await self._cleanup_time_index(redis)

Copilot AI Apr 20, 2026

Copy link

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_maybe_cleanup_time_index updates _last_cleanup_time before running _cleanup_time_index. If cleanup raises (network hiccup, Redis error), the rate limiter will suppress cleanup retries for 60s even though no cleanup actually happened. Consider only updating _last_cleanup_time after a successful cleanup (or resetting it on exception).

Suggested change

	self._last_cleanup_time = now
	await self._cleanup_time_index(redis)
	await self._cleanup_time_index(redis)
	self._last_cleanup_time = now

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

Copilot uses AI. Check for mistakes.

Comment thread

taskiq_redis/list_schedule_source.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

This was referenced Nov 12, 2026

Concurrent schedule deletion empties a Cluster source refresh #147

Open

fix(schedule): skip Cluster schedules removed after scanning #148

Draft

Reviewers

s3rius

s3rius approved these changes

Copilot code review Copilot

Copilot review effort

Applies to this pull request for everyone.Learn more about Copilot code review.

Copilot left review comments

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

3 participants

Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.

GeezerVison

Conversation

tmkarthi commented Feb 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

tmkarthi commented Apr 6, 2026

Uh oh!

s3rius left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

s3rius Apr 20, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI Apr 20, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

tmkarthi commented Feb 25, 2026 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

s3rius left a comment •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.