GeezerVison

Copy link

Copy Markdown

Member

iritkatriel previously requested changes Nov 27, 2022

View reviewed changes

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

serhiy-storchaka commented Jun 3, 2024

View reviewed changes

serhiy-storchaka deleted the long-from-dev_t branch June 4, 2024 16:36

)

Fix os.major(), os.minor() and os.makedev().
Support device numbers larger than 2**63-1.
Support non-existent device number (NODEV).
(cherry picked from commit 7111d96)

Co-authored-by: Serhiy Storchaka <[email protected]>

cherry_picker 7111d9605f9db7aa0b095bb8ece7ccc0b8115c3f 3.12

…onGH-31794)

Fix os.major(), os.minor() and os.makedev().
Support device numbers larger than 2**63-1.
Support non-existent device number (NODEV).
(cherry picked from commit 7111d96)

Co-authored-by: Serhiy Storchaka <[email protected]>

…H-120054)

Fix os.major(), os.minor() and os.makedev().
Support device numbers larger than 2**63-1.
Support non-existent device number (NODEV).
(cherry picked from commit 7111d96)

…H-120053)

Fix os.major(), os.minor() and os.makedev().
Support device numbers larger than 2**63-1.
Support non-existent device number (NODEV).
(cherry picked from commit 7111d96)

Co-authored-by: Serhiy Storchaka <[email protected]>

)

Fix os.major(), os.minor() and os.makedev().
Support device numbers larger than 2**63-1.
Support non-existent device number (NODEV).

)

Fix os.major(), os.minor() and os.makedev().
Support device numbers larger than 2**63-1.
Support non-existent device number (NODEV).

Tests on freebsd with zfs failed with OverflowErrors because zfs was
creating a device (dev_t) with the high bit set (which is is known to
to), and it turns out that python hasn't handled that consistently.
Recent python versions handle it as unsigned, ignoring the platform's
definition (e.g. on macos it's signed), in nearly all cases.  Though
it looks like os.makedev() is still broken.

Python improved the "high bit set" dev_t handling here (switching, as
mentioned, by trying to shift to treat them as unsigned):

[1] 7111d9605f9db7aa0b095bb8ece7ccc0b8115c3f
    gh-89928: Fix integer conversion of device numbers (GH-31794)
    python/cpython#31794

And those changes have been backported to an increasing number of
python minor releases.

I believe we were exposed to these errors when I removed our custom
stat function:

  commit ae032d6
  Drop our custom stat functions in favor of python's

To date, bup has also required dev_t (and some other stat fields) to
be signed and no more than eight bytes via use of Q in the INDEX_SIG
pack format for index entries.  Whether or not python also attempts to
treat all of these consistently as unsigned has not been investigated.

To address the dev_t issue, and to make sure we handle all the other
stat fields consistently over time and across python versions, bring
back and enhance our custom stat functions, provide compatible
versions of major() and minor(), and wrap os.mknod() to make sure it
only sees the unsigned device values it expects (we don't just wrap
mknod too because it's a bit more complex, and it doesn't appear to
have changed behavior over the perids we investigated).

Have all of our stat operations work with the actual platform
ranges (i.e. preserve unsigned/signed values), and make sure that the
index doesn't lose the stat field sign information by providing the
platform specific entry type signature to the index from _helpers.

So that we don't mishandle the index (even across hosts, say via
moving a bup repo from one machine to another), add the entry
signature to the index's header and require reindexing if the current
platform's entry signature doesn't match the one in the index.
Continue to support the previous index format (trivially) so that if
the entry signature hasn't actually changed, you don't have to reindex
just for the new format.

Rename INDEX_SIG to ENTRY_SIG since that's what it is, and rename
INDEX_HDR to INDEX_SIG since that's what *it* is.

(Given these changes, we could also make the index signature even more
 dynamic, i.e. shrink all the entries to the actual platform sizes,
 e.g. on linux, uid and gid could be 4 bytes.)

Thanks to Johannes Berg for a good bit of help figuring all this out.

Signed-off-by: Rob Browning <[email protected]>
Tested-by: Rob Browning <[email protected]>

Tests on freebsd with zfs failed with OverflowErrors because zfs was
creating a device (dev_t) with the high bit set (which is is known to
to), and it turns out that python hasn't handled that consistently.
Recent python versions handle it as unsigned, ignoring the platform's
definition in nearly all cases (e.g. on macos it's signed).  Though it
looks like os.makedev() is still broken.

Python improved the "high bit set" dev_t handling in this
commit (switching, as mentioned, to try to treat them as unsigned):

[1] 7111d9605f9db7aa0b095bb8ece7ccc0b8115c3f
    gh-89928: Fix integer conversion of device numbers (GH-31794)
    python/cpython#31794

And those changes have been backported to an increasing number of
python minor releases.  (Whether or not python attempts to treat stat
fields other than dev_t as unsigned consistently has not been
investigated.)

I believe we were exposed to these errors when I removed our custom
stat function in main:

  commit ae032d6
  Drop our custom stat functions in favor of python's

To date, bup has required dev_t (and some other stat fields) to be
unsigned too, and no more than eight bytes, via use of Q in the
INDEX_SIG pack format for index entries.

To address the dev_t issue, and to make sure we handle all the other
stat fields consistently over time and across python versions, bring
back and enhance our custom stat functions, provide compatible
versions of major() and minor(), and wrap python's os.mknod() to make
sure it only sees the unsigned device values it expects (we don't wrap
mknod too because it's a bit more complex, and it doesn't appear to
have changed behavior over the periods we investigated).

Have all of our stat operations work with the actual platform
ranges (i.e. preserve unsigned/signed values), and make sure that the
index doesn't lose the stat field sign information by providing the
platform specific entry type signature to the index from _helpers.

Finally, so that we don't mishandle the index (even across hosts, say
via moving a bup repo from one machine to another), add the entry
signature to the index's header and require reindexing if the current
platform's signature doesn't match the one in the index.  Continue to
support the previous index format (trivially) so that if the entry
signature hasn't actually changed, you don't have to reindex just for
the new format.

Rename INDEX_SIG to ENTRY_SIG since that's what it is, and rename
INDEX_HDR to INDEX_SIG since that's what *it* is.

Thanks to Johannes Berg for a good bit of help figuring all this out.

Signed-off-by: Rob Browning <[email protected]>
Tested-by: Rob Browning <[email protected]>

Tests on freebsd with zfs failed with OverflowErrors because zfs was
creating a device (dev_t) with the high bit set (which is is known to
to), and it turns out that python hasn't handled that consistently.
Recent python versions handle it as unsigned, ignoring the platform's
definition in nearly all cases (e.g. on macos it's signed).  Though it
looks like os.makedev() is still broken.

Python improved the "high bit set" dev_t handling in this
commit (switching, as mentioned, to try to treat them as unsigned):

[1] 7111d9605f9db7aa0b095bb8ece7ccc0b8115c3f
    gh-89928: Fix integer conversion of device numbers (GH-31794)
    python/cpython#31794

And those changes have been backported to an increasing number of
python minor releases.  (Whether or not python attempts to treat stat
fields other than dev_t as unsigned consistently has not been
investigated.)

I believe we were exposed to these errors when I removed our custom
stat function in main:

  commit ae032d6
  Drop our custom stat functions in favor of python's

To date, bup has required dev_t (and some other stat fields) to be
unsigned too, and no more than eight bytes, via use of Q in the
INDEX_SIG pack format for index entries.

To address the dev_t issue, and to make sure we handle all the other
stat fields consistently over time and across python versions, bring
back and enhance our custom stat functions, provide compatible
versions of major() and minor(), and wrap python's os.mknod() to make
sure it only sees the unsigned device values it expects (we don't wrap
mknod too because it's a bit more complex, and it doesn't appear to
have changed behavior over the periods we investigated).

Have all of our stat operations work with the actual platform
ranges (i.e. preserve unsigned/signed values), and make sure that the
index doesn't lose the stat field sign information by providing the
platform specific entry type signature to the index from _helpers.

Finally, so that we don't mishandle the index (even across hosts, say
via moving a bup repo from one machine to another), add the entry
signature to the index's header and require reindexing if the current
platform's signature doesn't match the one in the index.  Continue to
support the previous index format (trivially) so that if the entry
signature hasn't actually changed, you don't have to reindex just for
the new format.

Rename INDEX_SIG to ENTRY_SIG since that's what it is, and rename
INDEX_HDR to INDEX_SIG since that's what *it* is.

Thanks to Johannes Berg for a good bit of help figuring all this out.

Signed-off-by: Rob Browning <[email protected]>
Tested-by: Rob Browning <[email protected]>

Tests on freebsd with zfs failed with OverflowErrors because zfs was
creating a device (dev_t) with the high bit set (which is is known to
to), and it turns out that python hasn't handled that consistently.
Recent python versions handle it as unsigned, ignoring the platform's
definition in nearly all cases (e.g. on macos it's signed).  Though it
looks like os.makedev() is still broken.

Python improved the "high bit set" dev_t handling in this
commit (switching, as mentioned, to try to treat them as unsigned):

[1] 7111d9605f9db7aa0b095bb8ece7ccc0b8115c3f
    gh-89928: Fix integer conversion of device numbers (GH-31794)
    python/cpython#31794

And those changes have been backported to an increasing number of
python minor releases.  (Whether or not python attempts to treat stat
fields other than dev_t as unsigned consistently has not been
investigated.)

I believe we were exposed to these errors when I removed our custom
stat function in main:

  commit ae032d6
  Drop our custom stat functions in favor of python's

To date, bup has required dev_t (and some other stat fields) to be
unsigned too, and no more than eight bytes, via use of Q in the
INDEX_SIG pack format for index entries.

To address the dev_t issue, and to make sure we handle all the other
stat fields consistently over time and across python versions, bring
back and enhance our custom stat functions, provide compatible
versions of major() and minor(), and wrap python's os.mknod() to make
sure it only sees the unsigned device values it expects (we don't wrap
mknod too because it's a bit more complex, and it doesn't appear to
have changed behavior over the periods we investigated).

Have all of our stat operations work with the actual platform
ranges (i.e. preserve unsigned/signed values), and make sure that the
index doesn't lose the stat field sign information by providing the
platform specific entry type signature to the index from _helpers.

Finally, so that we don't mishandle the index (even across hosts, say
via moving a bup repo from one machine to another), add the entry
signature to the index's header and require reindexing if the current
platform's signature doesn't match the one in the index.  Continue to
support the previous index format (trivially) so that if the entry
signature hasn't actually changed, you don't have to reindex just for
the new format.

Rename INDEX_SIG to ENTRY_SIG since that's what it is, and rename
INDEX_HDR to INDEX_SIG since that's what *it* is.

Thanks to Johannes Berg for a good bit of help figuring all this out.

Signed-off-by: Rob Browning <[email protected]>
Tested-by: Rob Browning <[email protected]>

Tests on freebsd with zfs failed with OverflowErrors because zfs was
creating a device (dev_t) with the high bit set (which it is known
to), and it turns out that python hasn't handled that consistently.
Recent python versions handle it as unsigned, ignoring the platform's
definition in nearly all cases (e.g. on macos it's signed).  Though it
looks like os.makedev() is still broken.

Python improved the "high bit set" dev_t handling in this
commit (switching, as mentioned, to try to treat them as unsigned):

  7111d9605f9db7aa0b095bb8ece7ccc0b8115c3f
  gh-89928: Fix integer conversion of device numbers (GH-31794)
  python/cpython#31794

And those changes have been backported to an increasing number of
python minor releases.  (Whether or not python attempts to treat stat
fields other than dev_t as unsigned consistently has not been
investigated.)

I believe we were exposed to these errors when I removed our custom
stat function in main:

  ae032d6
  Drop our custom stat functions in favor of python's

To date, bup has also required dev_t (and some other stat fields) to
be unsigned and no more than eight bytes, via use of Q in the
INDEX_SIG pack format for index entries.

To address the dev_t issue, and to make sure we handle all the other
stat fields consistently over time and across python versions, bring
back and enhance our custom stat functions, and wrap python's
os.mknod() to make sure it only sees the unsigned device values it
expects (for now we don't add our own mknod because it's a bit more
complex, and python's doesn't appear to have changed behavior over the
periods we investigated).

Have all of our stat operations work with the actual platform
ranges (i.e. preserve unsigned/signed values), and make sure that the
index doesn't lose the stat field sign information by providing the
platform size and sign via _helpers.c_type_signed_size.

Finally, so that we don't mishandle the index (even across hosts, say
via moving a bup repo from one machine to another), add the entry
signature to the index's header and require reindexing if the current
platform's signature doesn't match the one in the index.  Continue to
support the previous index format (trivially) so that if the entry
signature hasn't actually changed, you don't have to reindex just for
the new format.

Rename INDEX_SIG to ENTRY_SIG since that's what it is, and rename
INDEX_HDR to INDEX_SIG since that's what *it* is.

Add bup_int_from_py to pyutil and move the (u)int functions above
bup_ulong_from_py.

Thanks to Johannes Berg for a good bit of help figuring all this out.

Signed-off-by: Rob Browning <[email protected]>
Tested-by: Rob Browning <[email protected]>

Tests on freebsd with zfs failed with OverflowErrors because zfs was
creating a device (dev_t) with the high bit set (which it is known
to), and it turns out that python hasn't handled that consistently.
Recent python versions handle it as unsigned, ignoring the platform's
definition in nearly all cases (e.g. on macos it's signed).  Though it
looks like os.makedev() is still broken.

Python improved the "high bit set" dev_t handling in this
commit (switching, as mentioned, to try to treat them as unsigned):

  7111d9605f9db7aa0b095bb8ece7ccc0b8115c3f
  gh-89928: Fix integer conversion of device numbers (GH-31794)
  python/cpython#31794

And those changes have been backported to an increasing number of
python minor releases.  (Whether or not python attempts to treat stat
fields other than dev_t as unsigned consistently has not been
investigated.)

I believe we were exposed to these errors when I removed our custom
stat function in main:

  ae032d6
  Drop our custom stat functions in favor of python's

To date, bup has also required dev_t (and some other stat fields) to
be unsigned and no more than eight bytes, via use of Q in the
INDEX_SIG pack format for index entries.

To address the dev_t issue, and to make sure we handle all the other
stat fields consistently over time and across python versions, bring
back and enhance our custom stat functions, and wrap python's
os.mknod() to make sure it only sees the unsigned device values it
expects (for now we don't add our own mknod because it's a bit more
complex, and python's doesn't appear to have changed behavior over the
periods we investigated).

Have all of our stat operations work with the actual platform
ranges (i.e. preserve unsigned/signed values), and make sure that the
index doesn't lose the stat field sign information by providing the
platform size and sign via _helpers.c_type_signed_size.

Finally, so that we don't mishandle the index (even across hosts, say
via moving a bup repo from one machine to another), add the entry
signature to the index's header and require reindexing if the current
platform's signature doesn't match the one in the index.  Continue to
support the previous index format (trivially) so that if the entry
signature hasn't actually changed, you don't have to reindex just for
the new format.

Rename INDEX_SIG to ENTRY_SIG since that's what it is, and rename
INDEX_HDR to INDEX_SIG since that's what *it* is.

Add bup_int_from_py to pyutil and move the (u)int functions above
bup_ulong_from_py.

Thanks to Johannes Berg for a good bit of help figuring all this out.

Signed-off-by: Rob Browning <[email protected]>
Tested-by: Rob Browning <[email protected]>