GeezerVisonThe World Wide Web, restored to 800 pixels of glory. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Home|About|Gaming cabinet|Archives|How it works|Starter sites | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
GeezerVison is showing github.com — gh-150449: Fix sqlite3.Blob crash with negative-step slices by ever0de · Pull Request #150450 · python/cpython Conversation
|
| // For positive step: read the contiguous range [start, stop) and pick | ||
| // every step-th byte. For negative step: read the contiguous range | ||
| // [start+(len-1)*step, start] and pick bytes in reverse stride order. | ||
| Py_ssize_t read_offset, read_length; |
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
We have an adjust slice function somewhere in the codebase, use the latter instead.
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
sqlite3.Blob has no direct memory pointer — data can only be fetched via sqlite3_blob_read(blob, buf, nbytes, offset), which reads a single contiguous byte range. To keep that to one call, we must pre-compute the minimal covering range with Py_MIN/Py_ABS, then index into the local buffer as blob_buf[cur - read_offset].
The size_t cur += step loop itself is the same pattern as bytearrayobject.c; only the offset correction differs because our local buffer starts at read_offset, not 0.
Py_ssize_t start, stop, step, slicelength, i; size_t cur; PySlice_Unpack(index, &start, &stop, &step); slicelength = PySlice_AdjustIndices(PyByteArray_GET_SIZE(self), &start, &stop, step); char *source_buf = PyByteArray_AS_STRING(self); char *result_buf = PyByteArray_AS_STRING(result); for (cur = start, i = 0; i < slicelength; cur += step, i++) { result_buf[i] = source_buf[cur]; }
Two approaches are possible:
Current approach (one read, Py_MIN/Py_ABS): Pre-compute the minimal covering range, read it once, then index with blob_buf[cur - read_offset]. The size_t cur += step loop is the same as bytearrayobject.c; only the offset correction differs.
N individual reads: Call sqlite3_blob_read once per element inside the loop — no Py_MIN/Py_ABS needed, but O(n) SQLite API calls instead of one.
I went with 1 to keep the number of SQLite calls minimal, but happy to switch to 2 if you prefer simpler code over fewer API calls.
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
I wanted to know whether you could use PySlice_AdjustIndices() but I guess not?
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
Please also exercise negative stepsize when start <= end (it doesn't make sense but still it needs to be checked).
Sorry, something went wrong.
There was an error while loading. Please reload this page.
| } | ||
| else { | ||
| PyObject *blob_bytes = read_multiple(self, stop - start, start); | ||
| // For positive step: read the contiguous range [start, stop) and |
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
It's not a contiguous range if step != 1.
Sorry, something went wrong.
There was an error while loading. Please reload this page.
|
A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated. Once you have made the requested changes, please leave a comment on this pull request containing the phrase |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
Reading or writing a sqlite3.Blob with a negative-step slice such as blob[9:0:-2] caused a crash (SystemError on older versions, ValueError on current main) because the C code computed stop - start as the read length, which is negative for negative steps. Fix subscript_slice() and ass_subscript_slice() in Modules/_sqlite/blob.c to handle both positive and negative steps correctly: - For step > 1: keep reading [start, stop) and indexing forward as before. - For step < -1: read the contiguous range [start+(len-1)*step, start] and index backward with stride -step.
24ade8a to
af0eb65
Compare
|
Please do NOT force-push; read https://devguide.python.org/getting-started/pull-request-lifecycle/ first. |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
|
Sorry.. I executed the wrong command. |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
When a negative-step slice has start <= stop (e.g. blob[3:8:-1]), the slice is empty (slicelength == 0). Previously ass_subscript_slice() returned 0 immediately without acquiring the value buffer, so assigning a non-empty bytes object to such a slice silently succeeded instead of raising IndexError. Move PyObject_GetBuffer() and the size check before the len == 0 early return so that the element-count mismatch is always detected.
Exercise the case where step is negative but start <= stop, which produces an empty slice. Verify that: - blob[3:8:-1] returns b"" (no crash) - blob[3:8:-1] = b"" is a no-op - blob[3:8:-1] = b"abc" raises IndexError (wrong size) Also add blob[5:5:-1] == b"" to cover the start == stop edge case.
|
I have made the requested changes; please review again |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
|
Thanks for making the requested changes! @picnixz: please review the changes made to this pull request. |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
I think this should be classified as a new feature. Please add a versionchanged directive for blob documentation and the What's New entry.
Sorry, something went wrong.
There was an error while loading. Please reload this page.
| return -1; | ||
| } | ||
|
|
||
| if (len == 0) { |
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
Was there a different bug? Like blob[5:5] = None and blob[5:5] = b'123'? We need tests for such cases.
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
Yes, these are bugs
the original code silently ignores both type errors and size mismatches when the slice length is zero, while the same violations raise errors for non-empty slices:
| Operation | Result (Python 3.13.4) |
|---|---|
blob[0:3] = b'12345' |
IndexError: Blob slice assignment is wrong size |
blob[0:3] = None |
TypeError: a bytes-like object is required |
blob[5:5] = b'123' |
no error (blob unchanged) |
blob[5:5] = None |
no error |
The same invariant (correct type, matching size) should hold regardless of whether the slice is empty.
I've added tests to cover both cases.
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
Classify negative-step slice support as a new feature per reviewer feedback (serhiy-storchaka): - Add versionchanged:: 3.16 directive to sqlite3.Blob documentation noting that negative-step slices are now supported. - Add What's New entry under Doc/whatsnew/3.16.rst. - Update NEWS entry wording to describe the change as a feature. - Add tests for empty-slice assignment edge cases (blob[5:5] = None and blob[5:5] = b'123') to verify TypeError and IndexError are raised correctly. - Rename read_offset to write_offset in ass_subscript_slice() since that variable is used as the offset for inner_write(), not a read.
Per devguide markup guidelines, the version argument for versionchanged should be 'next' during development; the release manager replaces it with the actual version number at release time.
…ystemError or ValueError'
|
@ever0de, as this PR combines the bug fix (for several bugs) and a new feature, it may be better to split it on two -- one for the bug fix, it will be backported, and other on the top of it only for a new feature. |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
|
@serhiy-storchaka I opened issue #150913 for the empty-slice validation bug. My plan is to submit a separate bug-fix PR for #150913 and remove the bug-fix changes from #150450, leaving #150450 focused on the negative-step slice feature. Does that separation look good to you? backlink: #150915 |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
…ythongh-150913 The empty-slice assignment validation change (moving the len==0 early return to after buffer/type/size checks) has been split out to a separate bug-fix PR targeting issue python#150913. This branch retains only the negative-step slice feature.
|
Yes, that's it. Thank you. |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
…-sqlite-blob-negative-step-slice
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
I would like you to either 1) explain why we need the casts 2) or remove them
Sorry, something went wrong.
There was an error while loading. Please reload this page.
| size_t cur; | ||
| Py_ssize_t i; | ||
| for (cur = (size_t)start, i = 0; i < len; cur += (size_t)step, i++) { | ||
| res_buf[i] = blob_buf[(Py_ssize_t)cur - read_offset]; |
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
OOC why all those casts? start and atep are already ssize values, and then we recast cur to ssize_t. Why not having ssize_t everywhere? there should not be any issues with signed comparisons as both operands are signed.
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
SSIZE_MAX + SSIZE_MAX = integer overflow.
The code is more complicated with signed integers.
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
Oh right. If we can add a test case for that it would be good but if it is too complex to do it (by mocking some stuff), let's forget it. I am keeping the comment unresolved so that future readers will remember
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
Would it be worth adding an inline comment here to explain why size_t is used instead of Py_ssize_t? Something like:
// Use size_t for 'cur' so that negative steps work via well-defined // unsigned wrap-around arithmetic. A signed accumulator would risk // undefined behaviour when start or step is near SSIZE_MAX.
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
No need. We use this technique in other places too. And this is not about negative steps, you will never get integer overflow in cur += step for negative step. But you get it for large positive step.
So please add a test case for something like b[5::sys.maxsize]. You can also add a test case for step=-sys.maxsize-1.
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
…ice operations Add test cases for sys.maxsize and -sys.maxsize-1 step values in both read (test_blob_get_slice_with_negative_step) and write (test_blob_set_slice_with_negative_step) to cover the unsigned wrap-around arithmetic path that avoids signed integer overflow.
…in sqlite3.Blob Extend the existing extreme-step test cases (sys.maxsize / -sys.maxsize-1) in test_blob_set_slice_with_negative_step to verify the blob contents were actually written correctly, not just that no crash occurred.
|
This PR exposed existing bug: #155702. It should be fixed first. |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
…p-slice Resolve the conflict in ass_subscript_slice(): keep the region computation which handles negative and extreme steps, and read it into a plain buffer as in pythongh-155702, instead of patching the bytes object read from the blob.
There was an error while loading. Please reload this page.
…nGH-150450) Reading or writing a slice with a negative step computed a negative length for sqlite3_blob_read() and sqlite3_blob_write(), so it failed instead of returning or storing the selected bytes. Compute the contiguous region which covers all selected bytes, and index it with a size_t cursor, so that a step of any sign and magnitude works. Co-authored-by: Serhiy Storchaka <[email protected]>
There was an error while loading. Please reload this page.
Successfully merging this pull request may close these issues.
There was an error while loading. Please reload this page.