GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — perf: reduce arb pickle size by ~50% by deftasparagusanaconda · Pull Request #447 · flintlib/python-flint

Conversation

@deftasparagusanaconda

@deftasparagusanaconda deftasparagusanaconda commented Nov 2, 2026

Copy link
Copy Markdown

previously, we stored the 4 integers of a pickled arb in hexadecimal. thats 2 bytes per byte. instead, we use python's native int pickling, achieving near-optimal packing

old:

>>> import math, pickle
>>> from flint import arb, ctx
>>>
>>> ctx.dps = 1_000_000
>>>
>>> original = arb(2).sqrt().mid()
>>> pickled = pickle.dumps(original)
>>> limit = math.ceil(ctx.dps * math.log2(10) / 8)
>>>
>>> print('pickle size      :', len(pickled))
>>> print('theoretical limit:', limit)
>>> print('difference       :', len(pickled) - limit)
>>>
pickle size      : 830583
theoretical limit: 415242
difference       : 415341

new:

>>> import math, pickle
>>> from flint import arb, ctx
>>>
>>> ctx.dps = 1_000_000
>>>
>>> original = arb(2).sqrt().mid()
>>> pickled = pickle.dumps(original)
>>> limit = math.ceil(ctx.dps * math.log2(10) / 8)
>>>
>>> print('pickle size      :', len(pickled))
>>> print('theoretical limit:', limit)
>>> print('difference       :', len(pickled) - limit)
>>>
pickle size      : 415311
theoretical limit: 415242
difference       : 69

69 haha nice

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@deftasparagusanaconda
perf: reduce arb pickle size by ~50%
4339714 
use python's native int pickling instead of hexadecimal
@deftasparagusanaconda

deftasparagusanaconda commented Nov 2, 2026

Copy link
Copy Markdown
Author

we can also reduce pickled acbs if we pickle them as

(acb, ((real_mid, real_rad), (imag_mid, imag_rad)) )

instead of

(acb, (self.real, self.imag))

so we avoid including the arb constructor twice. this may bloat the code for acb.__reduce__ though
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@oscarbenjamin

oscarbenjamin commented Nov 2, 2026

Copy link
Copy Markdown
Collaborator

Seems to have failed all the tests.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@deftasparagusanaconda

deftasparagusanaconda commented Nov 3, 2026

Copy link
Copy Markdown
Author

test_arb.py/test_pickle() and test_acb.py/test_acb_pickle() test all pickle protocols from 0 to (currently the highest) 5. protocols 0 and 1 cannot pickle a large int properly:

import pickle

n = 10 ** 10000

for protocol in range(pickle.HIGHEST_PROTOCOL + 1):
    try: 
        pickle.dumps(n, protocol=protocol)
        print('protocol', protocol, 'passed')
    except Exception as e:
        print('protocol', protocol, 'failed')
        print(e)
protocol 0 failed
Exceeds the limit (4300 digits) for integer string conversion; use sys.set_int_max_str_digits() to increase the limit
protocol 1 failed
Exceeds the limit (4300 digits) for integer string conversion; use sys.set_int_max_str_digits() to increase the limit
protocol 2 passed
protocol 3 passed
protocol 4 passed
protocol 5 passed

the fix is to replace __reduce__ with __reduce_ex__(protocol) and use a custom algorithm for protocol = 0 and 1.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@deftasparagusanaconda

deftasparagusanaconda commented Nov 3, 2026

Copy link
Copy Markdown
Author

@oscarbenjamin i see two options moving forward. which do you prefer?

  1. drop support for protocol 0 and 1, and state in the docs that only pickling protocols ≥2 (i.e. python ≥2.3) are supported. also change tests to match accordingly

  2. implement support for protocol 0 and 1 using int.to_bytes() and int.from_bytes(), and reintroduce _arb_from_pickle()

i prefer the first option, because python has used DEFAULT_PROTOCOL = 3 since python 3.0, and = 2 since python 2.3. its better to have simple code instead of special-casing for paleolithically old versions. we can always add support if someone asks for it.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@oscarbenjamin

oscarbenjamin commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator

We can also just not have the change here and use hex. This is why hex as used in the first place.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@deftasparagusanaconda

deftasparagusanaconda commented Nov 3, 2026

Copy link
Copy Markdown
Author

i dont favour hex because it takes up twice the storage needed. its not uncommon to store thousands of arbs with millions of digits that take up gigabytes of storage.

i am currently computing a constant, where i'd like to store 284228 arbs at 100100dps. using hex, that should be roughly 2 * 100100 * log2(10)/8 * 284228 / 1024**3 ≈ 22 GiB. using bit-packing halves that, saving me 11GiB. its the main reason im introducing this PR, actually.

(im just trying to prove that the storage savings are significant)
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@oscarbenjamin

oscarbenjamin commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator

Can it not just use to_bytes for all protocol versions? I don't want to overcomplicate anything here.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@deftasparagusanaconda

deftasparagusanaconda commented Nov 3, 2026

Copy link
Copy Markdown
Author

👍 sure ill check and see in an hour
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@deftasparagusanaconda

deftasparagusanaconda commented Nov 3, 2026

Copy link
Copy Markdown
Author

i also found that fmpz has the same issue, because it also uses native int pickling. the tests did not cover this.

import pickle
from flint import fmpz

n = fmpz(10 ** 10000)

for protocol in range(pickle.HIGHEST_PROTOCOL + 1):
    try: 
        pickle.dumps(n, protocol=protocol)
        print('protocol', protocol, 'passed')
    except Exception as e:
        print('protocol', protocol, 'failed')
        print(e)
protocol 0 failed
Exceeds the limit (4300 digits) for integer string conversion; use sys.set_int_max_str_digits() to increase the limit
protocol 1 failed
Exceeds the limit (4300 digits) for integer string conversion; use sys.set_int_max_str_digits() to increase the limit
protocol 2 passed
protocol 3 passed
protocol 4 passed
protocol 5 passed

we will have to fix any type that pickles ints
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@deftasparagusanaconda
fix: arb supports pickling protocols 0 and 1
58f028f 
using int.to_bytes() and int.from_bytes() instead of native int pickling

fmpz still does not support it. fix later.
@deftasparagusanaconda

deftasparagusanaconda commented Nov 3, 2026

Copy link
Copy Markdown
Author

this new version may be harder to maintain
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
deftasparagusanaconda added 2 commits November 3, 2026 20:08
@deftasparagusanaconda
fix: explicit big-endian int unpacking
fa18c77 
since tests didnt pass for 3.10
@deftasparagusanaconda
perf(pickle): protocols ≥2 pickle ints natively
337ffc0
@deftasparagusanaconda

deftasparagusanaconda commented Nov 4, 2026

Copy link
Copy Markdown
Author

protocols 0 & 1 are still inefficient, because they cant pickle bytes efficiently. i wont improve this

import math, pickle
from flint import arb, ctx

ctx.dps = 1_000_000

for protocol in range(pickle.HIGHEST_PROTOCOL + 1):
    original = arb(2).sqrt().mid()
    pickled = pickle.dumps(original, protocol=protocol)
    roundtrip = pickle.loads(pickled)
    limit = math.ceil(ctx.dps * math.log2(10) / 8)

    assert original == roundtrip

    print('protocol         :', protocol)
    print('pickle size      :', len(pickled))
    print('theoretical limit:', limit)
    print('difference       :', len(pickled) - limit)
    print()

# protocol         : 0
# pickle size      : 455727
# theoretical limit: 415242
# difference       : 40485
#
# protocol         : 1
# pickle size      : 622855
# theoretical limit: 415242
# difference       : 207613
#
# protocol         : 2
# pickle size      : 415294
# theoretical limit: 415242
# difference       : 52
#
# protocol         : 3
# pickle size      : 415294
# theoretical limit: 415242
# difference       : 52
#
# protocol         : 4
# pickle size      : 415311
# theoretical limit: 415242
# difference       : 69
#
# protocol         : 5
# pickle size      : 415311
# theoretical limit: 415242
# difference       : 69
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
oscarbenjamin
oscarbenjamin reviewed Nov 4, 2026
View reviewed changes
Comment thread src/flint/types/arb.pyx
if rad.is_finite():
rad = tuple(hex(x) for x in rad.man_exp())

if protocol < 2: # protocols 0 and 1 pickle ints inefficiently

@oscarbenjamin oscarbenjamin Nov 4, 2026

Copy link
Copy Markdown
Collaborator

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't overcomplicate things. It doesn't matter if protocols 0 and 1 are inefficient.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
deftasparagusanaconda reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

@deftasparagusanaconda deftasparagusanaconda Nov 4, 2026

Copy link
Copy Markdown
Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shall i remove the comment?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@deftasparagusanaconda

deftasparagusanaconda commented Nov 4, 2026

Copy link
Copy Markdown
Author

im happy with this. but i dont know enough cython to make it pass the cython lint tests. i need a little help
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@oscarbenjamin

oscarbenjamin commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator

What does the error from the linter say?
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@deftasparagusanaconda

deftasparagusanaconda commented Nov 7, 2026

Copy link
Copy Markdown
Author

i cant remember without running them again via CI (i think)?
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@oscarbenjamin oscarbenjamin oscarbenjamin left review comments

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

2 participants

@deftasparagusanaconda @oscarbenjamin
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.