GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-150449: Fix sqlite3.Blob crash with negative-step slices by ever0de · Pull Request #150450 · python/cpython

Conversation

@ever0de

@ever0de ever0de commented May 26, 2026 •
edited by bedevere-app Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

Reading or writing a sqlite3.Blob with a negative-step slice such as blob[9:0:-2] caused a crash (SystemError on older versions, ValueError on current main) because the C code computed stop - start as the read length, which is negative for negative steps.

Fix subscript_slice() and ass_subscript_slice() in Modules/_sqlite/blob.c to handle both positive and negative steps correctly:

  • For step > 1: keep reading [start, stop) and indexing forward as before.
  • For step < -1: read the contiguous range [start+(len-1)*step, start] and index backward with stride -step.
  • Issue: sqlite3: Blob crashes when using a negative-step slice #150449

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@bedevere-app bedevere-app Bot mentioned this pull request May 26, 2026
sqlite3: Blob crashes when using a negative-step slice #150449
Closed
@ever0de
ever0de force-pushed the gh-150449-fix-sqlite-blob-negative-step-slice branch from 561ce7f to 24ade8a Compare May 26, 2026 06:58
@ever0de
ever0de marked this pull request as ready for review May 26, 2026 06:58
@ever0de
ever0de requested a review from erlend-aasland as a code owner May 26, 2026 06:58
@bedevere-app bedevere-app Bot added the awaiting review label May 26, 2026
picnixz
picnixz reviewed May 26, 2026
View reviewed changes
Comment thread Modules/_sqlite/blob.c Outdated
// For positive step: read the contiguous range [start, stop) and pick
// every step-th byte. For negative step: read the contiguous range
// [start+(len-1)*step, start] and pick bytes in reverse stride order.
Py_ssize_t read_offset, read_length;

@picnixz picnixz May 26, 2026

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have an adjust slice function somewhere in the codebase, use the latter instead.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

ever0de reacted with eyes emoji
All reactions
  • 👀 1 reaction

@ever0de ever0de May 26, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sqlite3.Blob has no direct memory pointer — data can only be fetched via sqlite3_blob_read(blob, buf, nbytes, offset), which reads a single contiguous byte range. To keep that to one call, we must pre-compute the minimal covering range with Py_MIN/Py_ABS, then index into the local buffer as blob_buf[cur - read_offset].
The size_t cur += step loop itself is the same pattern as bytearrayobject.c; only the offset correction differs because our local buffer starts at read_offset, not 0.

  • bytearrayobject.c
Py_ssize_t start, stop, step, slicelength, i;
size_t cur;
PySlice_Unpack(index, &start, &stop, &step);
slicelength = PySlice_AdjustIndices(PyByteArray_GET_SIZE(self),
                                    &start, &stop, step);

char *source_buf = PyByteArray_AS_STRING(self); 
char *result_buf = PyByteArray_AS_STRING(result);

for (cur = start, i = 0; i < slicelength; cur += step, i++) {
    result_buf[i] = source_buf[cur];
}

Two approaches are possible:

  1. Current approach (one read, Py_MIN/Py_ABS): Pre-compute the minimal covering range, read it once, then index with blob_buf[cur - read_offset]. The size_t cur += step loop is the same as bytearrayobject.c; only the offset correction differs.

  2. N individual reads: Call sqlite3_blob_read once per element inside the loop — no Py_MIN/Py_ABS needed, but O(n) SQLite API calls instead of one.

I went with 1 to keep the number of SQLite calls minimal, but happy to switch to 2 if you prefer simpler code over fewer API calls.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@picnixz picnixz Jun 4, 2026

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wanted to know whether you could use PySlice_AdjustIndices() but I guess not?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
picnixz
picnixz requested changes May 26, 2026
View reviewed changes

@picnixz picnixz left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please also exercise negative stepsize when start <= end (it doesn't make sense but still it needs to be checked).

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Modules/_sqlite/blob.c Outdated
}
else {
PyObject *blob_bytes = read_multiple(self, stop - start, start);
// For positive step: read the contiguous range [start, stop) and

@picnixz picnixz May 26, 2026

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not a contiguous range if step != 1.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@bedevere-app bedevere-app Bot added awaiting changes and removed awaiting review labels May 26, 2026
@bedevere-app

bedevere-app Bot commented May 26, 2026

Copy link
Copy Markdown

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ever0de
pythongh-150449: Fix sqlite3.Blob crash with negative-step slices
af0eb65
Reading or writing a sqlite3.Blob with a negative-step slice such as
blob[9:0:-2] caused a crash (SystemError on older versions, ValueError
on current main) because the C code computed stop - start as the read
length, which is negative for negative steps.

Fix subscript_slice() and ass_subscript_slice() in Modules/_sqlite/blob.c
to handle both positive and negative steps correctly:
- For step > 1: keep reading [start, stop) and indexing forward as before.
- For step < -1: read the contiguous range [start+(len-1)*step, start]
  and index backward with stride -step.
@ever0de
ever0de force-pushed the gh-150449-fix-sqlite-blob-negative-step-slice branch from 24ade8a to af0eb65 Compare May 26, 2026 08:32
@picnixz

picnixz commented May 26, 2026

Copy link
Copy Markdown
Member

Please do NOT force-push; read https://devguide.python.org/getting-started/pull-request-lifecycle/ first.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ever0de

ever0de commented May 26, 2026

Copy link
Copy Markdown
Contributor Author

Sorry.. I executed the wrong command.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

ever0de added 3 commits May 26, 2026 17:42
@ever0de
pythongh-150449: validate slice assignment size even for empty slices
85312b4
When a negative-step slice has start <= stop (e.g. blob[3:8:-1]), the
slice is empty (slicelength == 0).  Previously ass_subscript_slice()
returned 0 immediately without acquiring the value buffer, so assigning
a non-empty bytes object to such a slice silently succeeded instead of
raising IndexError.

Move PyObject_GetBuffer() and the size check before the len == 0 early
return so that the element-count mismatch is always detected.
@ever0de
pythongh-150449: add tests for negative-step slices with start <= stop
c0c2018
Exercise the case where step is negative but start <= stop, which
produces an empty slice.  Verify that:
  - blob[3:8:-1] returns b"" (no crash)
  - blob[3:8:-1] = b"" is a no-op
  - blob[3:8:-1] = b"abc" raises IndexError (wrong size)

Also add blob[5:5:-1] == b"" to cover the start == stop edge case.
@ever0de
Fix PEP 7 line-length violations in blob.c (ass_subscript_slice)
83d3a3c
@ever0de

ever0de commented May 26, 2026

Copy link
Copy Markdown
Contributor Author

I have made the requested changes; please review again

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot added awaiting change review and removed awaiting changes labels May 26, 2026
@bedevere-app

bedevere-app Bot commented May 26, 2026

Copy link
Copy Markdown

Thanks for making the requested changes!

@picnixz: please review the changes made to this pull request.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app
bedevere-app Bot requested a review from picnixz May 26, 2026 09:40
@ever0de ever0de mentioned this pull request May 26, 2026
sqlite3: fix Blob.__setitem__ value range validation RustPython/RustPython#7981
Merged
serhiy-storchaka
serhiy-storchaka reviewed May 29, 2026
View reviewed changes

@serhiy-storchaka serhiy-storchaka left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should be classified as a new feature. Please add a versionchanged directive for blob documentation and the What's New entry.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
serhiy-storchaka
serhiy-storchaka reviewed May 29, 2026
View reviewed changes
Comment thread Modules/_sqlite/blob.c Outdated
return -1;
}

if (len == 0) {

@serhiy-storchaka serhiy-storchaka May 29, 2026

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was there a different bug? Like blob[5:5] = None and blob[5:5] = b'123'? We need tests for such cases.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@ever0de ever0de May 30, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, these are bugs

the original code silently ignores both type errors and size mismatches when the slice length is zero, while the same violations raise errors for non-empty slices:

Operation Result (Python 3.13.4)
blob[0:3] = b'12345' IndexError: Blob slice assignment is wrong size
blob[0:3] = None TypeError: a bytes-like object is required
blob[5:5] = b'123' no error (blob unchanged)
blob[5:5] = None no error

The same invariant (correct type, matching size) should hold regardless of whether the slice is empty.
I've added tests to cover both cases.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Modules/_sqlite/blob.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

ever0de added 3 commits May 30, 2026 13:42
@ever0de
pythongh-150449: Address review: docs, tests, and rename write_offset
f7c9f53
Classify negative-step slice support as a new feature per reviewer
feedback (serhiy-storchaka):

- Add versionchanged:: 3.16 directive to sqlite3.Blob documentation
  noting that negative-step slices are now supported.
- Add What's New entry under Doc/whatsnew/3.16.rst.
- Update NEWS entry wording to describe the change as a feature.
- Add tests for empty-slice assignment edge cases (blob[5:5] = None
  and blob[5:5] = b'123') to verify TypeError and IndexError are
  raised correctly.
- Rename read_offset to write_offset in ass_subscript_slice() since
  that variable is used as the offset for inner_write(), not a read.
@ever0de
pythongh-150449: Fix versionchanged version: use 'next' not '3.16'
5087b42
Per devguide markup guidelines, the version argument for versionchanged
should be 'next' during development; the release manager replaces it
with the actual version number at release time.
@ever0de
pythongh-150449: Fix wording in NEWS and whatsnew: 'crash' → 'raise S…
d774ab7
…ystemError or ValueError'
@ever0de
ever0de requested a review from AA-Turner as a code owner May 30, 2026 05:51
@ever0de
pythongh-150449: Remove redundant issue references from test comments
5bf02de
@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting change review labels May 30, 2026
@ever0de
pythongh-150449: Rename read_length to write_length in ass_subscript_…
0936d6b
…slice
@serhiy-storchaka

serhiy-storchaka commented Jun 3, 2026

Copy link
Copy Markdown
Member

@ever0de, as this PR combines the bug fix (for several bugs) and a new feature, it may be better to split it on two -- one for the bug fix, it will be backported, and other on the top of it only for a new feature.

ever0de reacted with eyes emoji
All reactions
  • 👀 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ever0de ever0de mentioned this pull request Jun 4, 2026
sqlite3: Blob slice assignment skips validation for empty slices #150913
Closed
@ever0de

ever0de commented Jun 4, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor Author

@serhiy-storchaka I opened issue #150913 for the empty-slice validation bug.

My plan is to submit a separate bug-fix PR for #150913 and remove the bug-fix changes from #150450, leaving #150450 focused on the negative-step slice feature.

Does that separation look good to you?


backlink: #150915

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ever0de
pythongh-150449: Move empty-slice validation fix to dedicated PR for p…
5f2f916
…ythongh-150913

The empty-slice assignment validation change (moving the len==0 early
return to after buffer/type/size checks) has been split out to a
separate bug-fix PR targeting issue python#150913.

This branch retains only the negative-step slice feature.
@serhiy-storchaka

serhiy-storchaka commented Jun 4, 2026

Copy link
Copy Markdown
Member

Yes, that's it. Thank you.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ever0de
Merge remote-tracking branch 'upstream/main' into pythongh-150449-fix…
d47abf8
…-sqlite-blob-negative-step-slice
serhiy-storchaka
serhiy-storchaka approved these changes Jun 4, 2026
View reviewed changes
picnixz
picnixz reviewed Jun 4, 2026
View reviewed changes

@picnixz picnixz left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would like you to either 1) explain why we need the casts 2) or remove them

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Modules/_sqlite/blob.c
Comment on lines +468 to +471
size_t cur;
Py_ssize_t i;
for (cur = (size_t)start, i = 0; i < len; cur += (size_t)step, i++) {
res_buf[i] = blob_buf[(Py_ssize_t)cur - read_offset];

@picnixz picnixz Jun 4, 2026

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OOC why all those casts? start and atep are already ssize values, and then we recast cur to ssize_t. Why not having ssize_t everywhere? there should not be any issues with signed comparisons as both operands are signed.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@serhiy-storchaka serhiy-storchaka Jun 4, 2026

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SSIZE_MAX + SSIZE_MAX = integer overflow.

The code is more complicated with signed integers.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

picnixz and ever0de reacted with thumbs up emoji
All reactions
  • 👍 2 reactions

@picnixz picnixz Jun 4, 2026

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh right. If we can add a test case for that it would be good but if it is too complex to do it (by mocking some stuff), let's forget it. I am keeping the comment unresolved so that future readers will remember

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@ever0de ever0de Jun 5, 2026

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be worth adding an inline comment here to explain why size_t is used instead of Py_ssize_t? Something like:

// Use size_t for 'cur' so that negative steps work via well-defined
// unsigned wrap-around arithmetic. A signed accumulator would risk
// undefined behaviour when start or step is near SSIZE_MAX.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@serhiy-storchaka serhiy-storchaka Jun 5, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need. We use this technique in other places too. And this is not about negative steps, you will never get integer overflow in cur += step for negative step. But you get it for large positive step.

So please add a test case for something like b[5::sys.maxsize]. You can also add a test case for step=-sys.maxsize-1.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

ever0de reacted with eyes emoji
All reactions
  • 👀 1 reaction
Comment thread Modules/_sqlite/blob.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

picnixz
picnixz approved these changes Jun 4, 2026
View reviewed changes
ever0de and others added 5 commits June 5, 2026 16:58
@ever0de
pythongh-150449: Add tests for extreme step values in sqlite3.Blob sl…
be1aa18
…ice operations

Add test cases for sys.maxsize and -sys.maxsize-1 step values in both
read (test_blob_get_slice_with_negative_step) and write
(test_blob_set_slice_with_negative_step) to cover the unsigned wrap-around
arithmetic path that avoids signed integer overflow.
@ever0de
pythongh-150449: Assert correctness of extreme-step slice assignment …
f76ebf7
…in sqlite3.Blob

Extend the existing extreme-step test cases (sys.maxsize / -sys.maxsize-1)
in test_blob_set_slice_with_negative_step to verify the blob contents
were actually written correctly, not just that no crash occurred.
@ever0de
pythongh-150449: Isolate extreme-step Blob assignment tests
52084f9
@serhiy-storchaka
Merge branch 'main' into pythongh-150449-fix-sqlite-blob-negative-ste…
a462e70
…p-slice
@serhiy-storchaka
Merge branch 'main' into pythongh-150449-fix-sqlite-blob-negative-ste…
d6e07f5
…p-slice
@serhiy-storchaka
serhiy-storchaka enabled auto-merge (squash) August 11, 2026 14:59
@serhiy-storchaka
Merge branch 'main' into pythongh-150449-fix-sqlite-blob-negative-ste…
98fc681
…p-slice
@serhiy-storchaka
serhiy-storchaka disabled auto-merge August 13, 2026 12:30
@serhiy-storchaka serhiy-storchaka added the DO-NOT-MERGE label Aug 13, 2026
@serhiy-storchaka serhiy-storchaka mentioned this pull request Aug 13, 2026
gh-155702: Fix sqlite3.Blob slice assignment with a step #155703
Merged
@serhiy-storchaka

serhiy-storchaka commented Aug 13, 2026

Copy link
Copy Markdown
Member

This PR exposed existing bug: #155702. It should be fixed first.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@serhiy-storchaka
Merge branch 'main' into pythongh-150449-fix-sqlite-blob-negative-ste…
c587f21
…p-slice

Resolve the conflict in ass_subscript_slice(): keep the region computation
which handles negative and extreme steps, and read it into a plain buffer
as in pythongh-155702, instead of patching the bytes object read from the blob.
@serhiy-storchaka serhiy-storchaka removed the DO-NOT-MERGE label Aug 14, 2026
@serhiy-storchaka
serhiy-storchaka merged commit c72ea53 into python:main Aug 14, 2026
55 of 56 checks passed

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot removed the awaiting merge label Aug 14, 2026
@serhiy-storchaka serhiy-storchaka mentioned this pull request Aug 14, 2026
[3.15] gh-150449: Raise ValueError for sqlite3.Blob slices with a negative step #155773
Merged
mbeijen pushed a commit to mbeijen/cpython that referenced this pull request Aug 14, 2026
@ever0de @serhiy-storchaka @mbeijen
pythongh-150449: Support negative steps in sqlite3.Blob slices (pytho…
b940ed2
…nGH-150450)

Reading or writing a slice with a negative step computed a negative length
for sqlite3_blob_read() and sqlite3_blob_write(), so it failed instead of
returning or storing the selected bytes.

Compute the contiguous region which covers all selected bytes, and index it
with a size_t cursor, so that a step of any sign and magnitude works.

Co-authored-by: Serhiy Storchaka <[email protected]>
@serhiy-storchaka serhiy-storchaka mentioned this pull request Oct 10, 2026
[3.14] gh-150449: Raise ValueError for sqlite3.Blob slices with a negative step (GH-155773) #159110
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@serhiy-storchaka serhiy-storchaka serhiy-storchaka approved these changes
@picnixz picnixz picnixz approved these changes
@erlend-aasland erlend-aasland Awaiting requested review from erlend-aasland erlend-aasland is a code owner
@AA-Turner AA-Turner Awaiting requested review from AA-Turner AA-Turner is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

3 participants

@ever0de @picnixz @serhiy-storchaka
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.