GeezerVison

What type of PR is this?

• Other (performance)

Description

Every statement exports a telemetry event, and the thread that fills a batch (every 100th statement) serializes all of its events inline. JsonSerializableMixin.to_json used dataclasses.asdict, which recursively deep-copies the whole event, about 65 nodes with the system configuration and connection parameters. It took ~64 us per event, and flushing 100 events held the calling thread for ~6.8 ms.

With 64 threads running SELECT 1 against a mock warehouse, py-spy --gil put telemetry export at ~15% of worker GIL time, ~12% of it in asdict. Turning telemetry off raised throughput 16% (983 → 1,137 statements/s), so most of that overhead is serialization.

to_json now walks dataclass fields directly, using cached field names, and drops None fields the way the old dict_factory did. It no longer deep-copies leaves and reuses one EnumEncoder instead of building one per call. The JSON is byte-identical: ~19 us per event (3.4x faster), and a 100-event flush takes ~2.3 ms.

How is this tested?

• Unit tests
• E2E Tests
• Manually

tests/unit/test_telemetry_serialization.py keeps the previous serializer (json.dumps(asdict(..., dict_factory=...), cls=EnumEncoder)) as the oracle and asserts byte-for-byte equality over:

• every event the client exports: connection, failure, and latency events for each statement type and result format;
• the full flush payload;
• every telemetry model, fully populated and with all optional fields None;
• nested structs and containers holding None, enums and dataclasses;
• non-ASCII strings, control characters and lone surrogates;
• floats including -0.0, subnormals, inf and nan;
• 300 seeded random events.

It also checks that unserializable values raise TypeError like before. Six deliberately broken serializers (dropping falsy values, dropping None in lists, ensure_ascii=False, sorted keys, compact separators, enum names instead of values) each fail the suite. The full unit suite passes. The live telemetry e2e tests (test_telemetry_e2e.py, test_concurrent_telemetry.py) pass against a warehouse, so the endpoint accepts the payloads.

Performance: main vs this branch in fresh, interleaved processes, median of 5 runs. Mock Thrift server replaying a captured SELECT 1 reply with 50 ms RTT, one connection per thread, cursor per statement, telemetry on. One thread is RTT-bound and unchanged; a re-run at 64 threads with fetchall gave +5.0%.

Related Tickets & Documents

None.

This pull request and its description were written by Isaac.