GeezerVison is showing github.com — Add MySQL protocol compression with zlib and zstd by methane · Pull Request #1279 · PyMySQL/PyMySQL
Conversation
methane
commented
Oct 1, 2026
•
edited
Loading
Uh oh!
There was an error while loading. Please reload this page.
PyMySQL currently rejects compress=True. This change enables MySQL protocol compression with zlib and zstd, including zstd on Python 3.9–3.13 through the optional PyMySQL[zstd] extra.
compress=True prefers zstd when available and supported by the server, then falls back to zlib or an uncompressed connection. compress="zlib" and compress="zstd" require the selected algorithm. Outgoing payloads below 400 bytes, or those that do not shrink, remain uncompressed within the compressed protocol.
The framing follows the MySQL protocol and the approach used by go-sql-driver/mysql: compression starts after authentication, supports packets split across frames and sequence wraparound, and handles large requests and responses. Decompression is bounded by the declared length and rejects corrupt or incomplete streams. Python 3.14+ uses compression.zstd; earlier versions use backports.zstd>=1.3.0, installed only when python_version < "3.14". CI installs the extra, and the README and changelog document the API.
Compressed-frame lengths use int.to_bytes(3, "little") so overflow is rejected, and the complete header and payload are assembled in one join. Decoder contexts remain independent per frame: the supported stateful decoder APIs stop at EOF and provide no reset operation. Sharing one global decoder would therefore fail on later frames even if calls were serialized. The standard-library one-shot zstd helper also constructs a new decoder internally and does not provide the bounded-output check used here. The code documents this constraint and the concurrent-stream tests verify isolation.
Validation:
- Compression tests against MySQL 8.4.11: 90 passed on each of Python 3.9, 3.13, and 3.14, including TLS/non-TLS connections, reconnects, malformed frames, and request/response payloads above 16 MiB. Added deterministic 1/2/5/17-byte transport reads, MySQL headers and bodies split across compressed frames, normal continuation packets (including an empty terminator), the actual 24-bit frame limit with compressible and incompressible input, and independent decoding across eight concurrent threads.
- Full suite with Python 3.13 and MySQL zstd connections: 272 passed, 12 skipped, 1 expected failure, plus 4 passing subtests; converter doctests: 4 passed.
- Verified extra installation selects backports.zstd 1.3.0 on Python 3.9 and 1.7.0 on Python 3.13; Python 3.14 installs no backport and uses the standard library. The optional-backend tests also cover missing dependencies and zlib fallback.
- Ruff formatting, Ruff lint, and
git diff --check passed.
Summary by CodeRabbit
- New Features
- Added optional MySQL protocol compression using zlib or zstd. Automatic selection prefers zstd when available and supported, then zlib; connections remain uncompressed if the server supports neither.
- Compression skips outgoing payloads smaller than 400 bytes and payloads that would not become smaller.
- Documentation
- Updated the installation guide and changelog with compression options, automatic selection behavior, and zstd setup details.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
coderabbitai
Bot
commented
Oct 1, 2026
•
edited
Loading
Uh oh!
There was an error while loading. Please reload this page.
|
Navigate logical layers of code changes, visualize relationships, and explore their blast radius.
No actionable comments were generated in the recent review. 🎉
ℹ️ Recent review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Advanced
Run ID: 5d737124-0bf4-4532-a638-17ea84e3a008
📥 Commits
Reviewing files that changed from the base of the PR and between e303e22 and 2300c6b.
📒 Files selected for processing (2)
pymysql/_compression.pypymysql/tests/test_compression.py
Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 7 remain after this review.
📝 Walkthrough
Walkthrough
PyMySQL now supports MySQL protocol compression with zlib and optional zstd. The connection negotiates compression during authentication and uses compressed framing for later traffic. The change adds codec and connection tests, an optional zstd dependency, and installation and changelog documentation.
Changes
MySQL protocol compression
| Layer / File(s) |
Summary |
Compressed frame codec and backend support
pymysql/_compression.py, pymysql/tests/test_compression.py, pyproject.toml, .github/workflows/test.yaml |
Adds compressed frame encoding and decoding with zlib or optional zstd. Tests cover frame limits, sequence handling, compression thresholds, malformed frames, and backend selection. |
Compression options and handshake negotiation
pymysql/connections.py, pymysql/constants/CLIENT.py, pymysql/tests/test_compression.py, README.md, CHANGELOG.md |
Adds compression option validation and zlib or zstd capability negotiation. Documents supported modes, fallback behavior, dependencies, and when payloads remain uncompressed. |
Compressed connection I/O and lifecycle
pymysql/connections.py, pymysql/tests/test_compression.py |
Starts compressed framing after authentication and routes later reads and writes through the compressed stream. Resets compression state for connection and command operations. Tests cover authentication, sequence resets, TLS, reconnects, and large packets. |
Priority: ➖ Normal
Estimated code review effort: 4 (Complex) | ~45 minutes
Change: Feature
Sequence Diagram(s)
sequenceDiagram
participant Connection
participant MySQLServer
participant CompressedStream
Connection->>MySQLServer: Negotiate zlib or zstd capability
Connection->>MySQLServer: Complete authentication with normal framing
Connection->>CompressedStream: Create stream after authentication
Connection->>CompressedStream: Pass subsequent writes for framing
CompressedStream->>MySQLServer: Send compressed protocol frames
MySQLServer->>CompressedStream: Return compressed protocol frames
CompressedStream->>Connection: Provide decoded packet bytes
Loading
Merge Risk: ⚪ Minimal · up to 2300c
No concrete merge-blocking issue was identified. Compression support appears ready to merge subject to normal checks.
Security Architecture Review
Security architecture risk: 🔵 Low · up to 2300c
Compression preserves the inspected authentication and TLS ordering and rejects malformed streams with bounded output. The main remaining concern is connection reuse after an unexpected interruption during compressed decoding. Exposure is limited to connections using compression; no credential bypass or broader privilege gain was established.
Retained concerns
- Low · reliability · inferred: Unexpected interruption during compressed-frame processing can escape without closing the connection. The codec consumes frame bytes before decoding and committing its buffer, but Connection._read_bytes closes only for InternalError. If a caller catches an interruption or unexpected decoding/allocation exception and reuses that connection, framing or response-consumption state may be incomplete. Expected malformed-stream errors and raw transport failures already close correctly; the concern is the newly added processing interval outside those safeguards.
Security review details
Security Blast Radius
- inferred — The new decoding surface is reachable by the selected MySQL peer on connections negotiating compression. A peer controlling that stream can supply frame headers and compressed payloads to the client process. The inspected changes introduce no additional database privilege, tenant identity, or infrastructure authority; process-wide resource consequences were not fully characterized.
Security Findings and Attack Paths
- inferred — Unexpected interruption after frame consumption creates a possible response-state desynchronization path if the application subsequently reuses the open connection. This is a failure-containment concern, not a demonstrated credential compromise or attacker-controlled privilege escalation.
Trust Boundaries and Controls
- observed — TLS wrapping, when selected, precedes authentication traffic. Built-in authentication exchanges use normal framing, and compression activates after their terminal OK packet. Custom handlers retain the historical contract that a None return can indicate a completed exchange.
- observed — Frame payload and declared output lengths are 24-bit fields. Decoding limits returned output to the declared length plus one and rejects incorrect lengths, incomplete streams, trailing data, and recognized decoder errors. These are output controls, not proof of a total native-memory or CPU bound.
Resilience and Maintainability Implications
- observed — Recognized decoding errors, short raw reads, and transport failures close the connection and clear its compression state. This prevents normal malformed-frame rejection from leaving a reusable poisoned stream. Connection-level closure tests cover zlib; standalone malformed-frame tests are parameterized for both available algorithms.
Hardening Proposals
- proposed — Make interruption or unexpected failure anywhere in a compressed-read transition terminal for that connection, matching the raw reader’s cleanup policy while preserving the original exception.
🚥 Pre-merge checks | ✅ 4 | ❌ 1
❌ Failed checks (1 warning)
| Check name |
Status |
Explanation |
Resolution |
| Docstring Coverage |
⚠️ Warning |
Docstring coverage is 6.35% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 63 functions across 4 files. |
Write docstrings for the functions missing them to satisfy the coverage threshold. |
✅ Passed checks (4 passed)
| Check name |
Status |
Explanation |
| Linked Issues check |
✅ Passed |
Check skipped because no linked issues were found for this pull request. |
| Out of Scope Changes check |
✅ Passed |
Check skipped because no linked issues were found for this pull request. |
| Description Check |
✅ Passed |
Check skipped - CodeRabbit’s high-level summary is enabled. |
| Title check |
✅ Passed |
The title clearly and concisely identifies the main change: adding MySQL protocol compression with zlib and zstd. |
- Fix all pre-merge checks with AI
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
- Commit to this branch
- Create a new PR
🧪 Generate unit tests (beta)
- Commit to this branch
- Create a new PR
- Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Autopilot is currently an internal CodeRabbit preview.
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
❤️ Share
Comment @coderabbitai help to get the list of available commands.
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
|
|
|
|
|
|
|
def _pack_int24(value):
|
|
|
return struct.pack("<I", value)[:3]
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
value.to_bytes(3, "little") の方が速いしoverflow errorになって安全だからそっちにして。
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Comment on lines
+44
to
+49
|
|
header = (
|
|
|
_pack_int24(len(payload))
|
|
|
+ bytes([self.sequence])
|
|
|
+ _pack_int24(original_length)
|
|
|
)
|
|
|
write_bytes(header + payload)
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
header の一時オブジェクトが無駄だから一気に header + payload まで全部作って write_bytes して。
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
This branch has not been deployed
No deployments
Uh oh!
There was an error while loading. Please reload this page.
Assignees
No one assigned
Development
Successfully merging this pull request may close these issues.
Compression
Uh oh!
There was an error while loading. Please reload this page.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
|