GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — [qa] Replicate Kafka test topics so that they survive a broker restart by ryzhyk · Pull Request #7894 · feldera/feldera

[qa] Replicate Kafka test topics so that they survive a broker restart #7894

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

ryzhyk merged 3 commits into main from issue-7550

Nov 9, 2026

Uh oh!

There was an error while loading. Please reload this page.

Merged

[qa] Replicate Kafka test topics so that they survive a broker restart #7894

Changes from 1 commit

Commits

Show all changes

3 commits

Select commit Hold shift + click to select a range

b392921

[qa] Replicate Kafka test topics so that they survive a broker restart

ryzhyk Nov 8, 2026

d8aa4e8

[qa] Replicate the topics that the Kafka Avro test creates

ryzhyk Nov 8, 2026

e144940

[qa] Delete only stale topics in the Kafka Avro test's setup

ryzhyk Nov 8, 2026

File filter

Filter by extension .py (1) All 1 file type selected

Viewed files

Clear filters

Conversations

Failed to load comments.

Uh oh!

There was an error while loading. Please reload this page.

Jump to

Jump to file

Failed to load files.

Uh oh!

There was an error while loading. Please reload this page.

Diff view

Unified

Split

Hide whitespace

Diff view

Unified

Split

Hide whitespace

Previous commit

[qa] Delete only stale topics in the Kafka Avro test's setup

The setup read the variant id where it expected a topic's creation date, so
it deleted every kafka_avro topic on the cluster, including those of runs
still in progress against it, such as the other architecture's job. It now
reads the date that each topic name carries.

Signed-off-by: Leonid Ryzhyk <[email protected]>

Loading branch information

Uh oh!

There was an error while loading. Please reload this page.

ryzhyk committed Nov 8, 2026

commit e144940c7af177888376d63a0bff5310639fce43

There are no files selected for viewing

53 changes: 42 additions & 11 deletions 53 python/tests/workloads/test_kafka_avro.py

Show comments

View file

Open in desktop

Original file line number	Diff line number	Diff line change
Expand Up		@@ -13,6 +13,7 @@
		import socket
		import uuid
		from datetime import datetime, timedelta
		from typing import Iterable

		_KAFKA_BOOTSTRAP = None
		_SCHEMA_REGISTRY = None
Expand Down Expand Up		@@ -106,6 +107,32 @@ def delete_kafka_topics(admin: AdminClient, topics: list[str]):
		print(f"Failed to delete {topic}: {e}")


		# The creation date that `Variant` writes into a topic name, as in
		# `<pipeline name>_topic2_20261108_ab12`.
		_TOPIC_DATE = re.compile(r"_topic2?_(\d{8})_[0-9a-f]{4}$")


		def stale_topics(topics: Iterable[str], cutoff: str) -> list[str]:
		"""Select the topics of this test that were created before a cutoff date.

		Only the creation date in the name counts, so the topics of a recent run,
		which another job may still be using, stay.

		Args:
		topics: Names of the topics in the cluster.
		cutoff: A date as `%Y%m%d`; topics created before it are stale.

		Returns:
		The names of the stale topics, in the order of `topics`.
		"""
		stale = []
		for topic in topics:
		match = _TOPIC_DATE.search(topic)
		if "kafka_avro" in topic and match and match.group(1) < cutoff:
		stale.append(topic)
		return stale


		def delete_schema_subjects(registry_url: str, subjects: list[str]):
		for subject in subjects:
		r = requests.delete(f"{registry_url}/subjects/{subject}")
Expand Down Expand Up		@@ -407,17 +434,7 @@ def setUpClass(cls):

		admin = get_kafka_admin()
		cutoff = (datetime.now() - timedelta(days=3)).strftime("%Y%m%d")

		topics_to_delete = []
		for topic in admin.list_topics().topics:
		if "kafka_avro" in topic:
		try:
		if topic.split("_")[4] < cutoff:
		topics_to_delete.append(topic)
		except IndexError:
		# Skip if topic doesn't have expected format
		pass

		topics_to_delete = stale_topics(admin.list_topics().topics, cutoff)
		if topics_to_delete:
		print(f"Deleting {len(topics_to_delete)} stale topics: {topics_to_delete}")
		delete_kafka_topics(admin, topics_to_delete)
Expand All		@@ -436,6 +453,20 @@ def test_kafka_avro_variants(self):
		create_and_run_pipeline_variant(cfg)


		def test_stale_topics_spares_recent_runs():
		"""Only this test's topics from before the cutoff are stale, so a run
		leaves the topics of a recent run alone."""
		variant = Variant(
		TestKafkaAvro.TEST_CONFIGS[1], unique_pipeline_name("test_kafka_avro_1")
		)
		recent = [variant.topic1, variant.topic2]
		old = [re.sub(r"_\d{8}_", "_20000101_", topic) for topic in recent]
		other_test = "connector-errors-input-0a34f7998e7a"
		cutoff = (datetime.now() - timedelta(days=3)).strftime("%Y%m%d")

		assert stale_topics(recent + old + [other_test], cutoff) == old


		# To run all pipelines in this test file:
		# python -m pytest ./tests/workloads/test_kafka_avro.py

Expand Down

Oops, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.

You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.