GeezerVison is showing github.com — SQLite rowcount is corrupted when combining UPDATE RETURNING w/ table that is dropped and recreated · Issue #93421 · python/cpython
SQLite rowcount is corrupted when combining UPDATE RETURNING w/ table that is dropped and recreated #93421Descriptionversion info:
$ python
Python 3.10.0 (default, Nov 5 2021, 17:23:47) [GCC 11.2.1 20210728 (Red Hat 11.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqlite3
>>> sqlite3.sqlite_version
'3.36.0'
we have a test suite that creates a table, runs some SQL, then drops it. if multiple tests run that each perform this task, if the same SQLite connection is used, rowcount starts returning "0". Seems to also require RETURNING to be used. Full demonstration:
import os
import sqlite3
def go():
"""function creates a new table, runs INSERT/UPDATE, drops table,
commits connection.
"""
# create table
cursor = conn.cursor()
cursor.execute(
"""CREATE TABLE some_table (
id INTEGER NOT NULL,
value VARCHAR(40) NOT NULL,
PRIMARY KEY (id)
)
"""
)
cursor.close()
conn.commit()
# run operation
cursor = conn.cursor()
cursor.execute(
"INSERT INTO some_table (id, value) VALUES (1, 'v1')"
)
ident = 1
cursor.execute(
"UPDATE some_table SET value='v2' "
"WHERE id=? RETURNING id",
(ident,),
)
new_ident = cursor.fetchone()[0]
assert ident == new_ident
assert cursor.rowcount == 1, cursor.rowcount
cursor.close()
# drop table
cursor = conn.cursor()
cursor.execute("DROP TABLE some_table")
cursor.close()
conn.commit()
if os.path.exists("file.db"):
os.unlink("file.db")
# passes
conn = sqlite3.connect("file.db")
go()
# run again w/ new connection (same DB), passes
conn = sqlite3.connect("file.db")
go()
print("FAILURE NOW OCCURS")
# run again w/ same connection, fails
go()on the third run, where we ran the "test" on the same connection twice, it fails:
$ python test3.py
FAILURE NOW OCCURS
Traceback (most recent call last):
File "/home/classic/dev/sqlalchemy/test3.py", line 62, in <module>
go()
File "/home/classic/dev/sqlalchemy/test3.py", line 39, in go
assert cursor.rowcount == 1, cursor.rowcount
AssertionError: 0
it would appear there's some internal caching of table state that needs to be cleared. ActivityAA-Turner commented on Jun 1, 2022 Reacted by Erlend E. Aasland erlend-aasland commented on Jun 2, 2022 Thanks for the report!
Internally, we simply use the function sqlite3_changes to set rowcount1, implying that we do not calculate rowcount ourself; the value comes straight from the underlying SQLite library. If sqlite3_changes returns zero, rowcount will be zero.
I've verified that this only happens with UPDATE ... RETURNING statements, which also implies that the behaviour we are observing comes from the underlying SQLite library. I suggest to report this on the SQLite forum (they do not have a bug tracker).
Note: I did some tests where I used sqlite3_total_changes() in addition to sqlite3_changes(), and they revealed that the number of total changes on the database connection did not increase whith UPDATE ... RETURNING statements, only with normal UPDATE statements.
Closing this as not-a-bug.
See also:
Footnotes
-
Assuming that you've supplied a valid statement, and that no database errors were raised by SQLite. In case of errors, rowcount will normally be set to -1. ↩
erlend-aasland commented on Jun 8, 2022 erlend-aasland commented on Jun 8, 2022 Thanks for the report, and for pushing this through, Mike. Thanks for reviewing, Ma Lin! erlend-aasland commented on Jun 8, 2022
Just thought, if the next row is cached (like pre 3df0fc8), the .rowcount will be more correct?
We'd still only update .rowcount after SQLITE_DONE, so when the statement is stepped through, you'll have the correct row count, caching or not.
For example, in these conditions: [...]
That is a separate issue (and it has always been). It is not very large or complex fix; we just need to step through a statement for each loop iteration in _pysqlite_query_execute if multiple is true, kinda like we do in executescript. It is a bug, though, so it would be quite ok to backport it through 3.10.
However, as you point out, such code is probably rare; it does not make sense with UPDATE ... RETURNING in executemany. rt121212121 commented on Dec 1, 2022 As a data point, I have a MacOS user who is using 3.10.8, and getting the rowcount 0 problem. From what I understand the 3.10 backport was merged in June and 3.10.8 was tagged as of October, so should have the fix from what I understand.
This is an UPDATE ... RETURNING execute that asserts rowcount == 1, which fails because it is unexpectedly 0. erlend-aasland commented on Dec 2, 2022
As a data point, I have a MacOS user who is using 3.10.8, and getting the rowcount 0 problem. From what I understand the 3.10 backport was merged in June and 3.10.8 was tagged as of October, so should have the fix from what I understand.
This is an UPDATE ... RETURNING execute that asserts rowcount == 1, which fails because it is unexpectedly 0.
Would you mind posting a minimal reproducer?
Python 3.10.8 should have the fix, yes. rt121212121 commented on Dec 5, 2022 The reproducer from the top does not work for him. It'll happen in the course of operation of the larger application repeatedly however. No idea how to reproduce it outside of that. rt121212121 commented on Jan 18, 2023 Finally had the problem happen myself. It appears to be a different problem than tables being dropped or recreated. Repro on the above linked issue.
MetadataMetadataStatusShow more project fields
DevelopmentNo branches or pull requestsIssue actions
|