GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-93820: Pickle enum.Flag by name by serhiy-storchaka · Pull Request #93891 · python/cpython

Conversation

@serhiy-storchaka

@serhiy-storchaka serhiy-storchaka commented Jun 16, 2022

Copy link
Copy Markdown
Member

#93820.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@serhiy-storchaka
pythongh-93820: Pickle enum.Flag by name
0814335
@serhiy-storchaka serhiy-storchaka added the needs backport to 3.11 only security fixes label Jun 16, 2022
@serhiy-storchaka
serhiy-storchaka requested a review from tiran June 16, 2022 08:19
@serhiy-storchaka
serhiy-storchaka requested a review from ethanfurman as a code owner June 16, 2022 08:19
@bedevere-bot bedevere-bot added the awaiting core review label Jun 16, 2022
@tiran

tiran commented Jun 16, 2022

Copy link
Copy Markdown
Member

A Flag instance with multiple members is serialized like this:

>>> (re.A | re.I | re.DOTALL | 0x1000).__reduce_ex__(1)
(<built-in function or_>, (re.ASCII, re.IGNORECASE|re.DOTALL|0x1000))

It looks like only the first member is reduced by name.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@serhiy-storchaka

serhiy-storchaka commented Jun 16, 2022

Copy link
Copy Markdown
Member Author

The rest is reduced recursively.

>>> pickletools.dis(pickletools.optimize(pickle.dumps(re.A | re.I | re.DOTALL | 0x1000, protocol=3)))
    0: \x80 PROTO      3
    2: c    GLOBAL     '_operator or_'
   17: q    BINPUT     0
   19: c    GLOBAL     'builtins getattr'
   37: q    BINPUT     1
   39: c    GLOBAL     're RegexFlag'
   53: q    BINPUT     2
   55: X    BINUNICODE 'ASCII'
   65: \x86 TUPLE2
   66: R    REDUCE
   67: h    BINGET     0
   69: h    BINGET     1
   71: h    BINGET     2
   73: X    BINUNICODE 'IGNORECASE'
   88: \x86 TUPLE2
   89: R    REDUCE
   90: h    BINGET     0
   92: h    BINGET     1
   94: h    BINGET     2
   96: X    BINUNICODE 'DOTALL'
  107: \x86 TUPLE2
  108: R    REDUCE
  109: h    BINGET     2
  111: M    BININT2    4096
  114: \x85 TUPLE1
  115: R    REDUCE
  116: \x86 TUPLE2
  117: R    REDUCE
  118: \x86 TUPLE2
  119: R    REDUCE
  120: \x86 TUPLE2
  121: R    REDUCE
  122: .    STOP
highest protocol among opcodes = 2
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@tiran

tiran commented Jun 16, 2022

Copy link
Copy Markdown
Member

copy.copy does not recurse. Is that ok?

>>> copy.copy(re.A | re.I | re.DOTALL | 0x1000)
re.ASCII 4114
re.ASCII|re.IGNORECASE|re.DOTALL|0x1000
>>> copy.deepcopy(re.A | re.I | re.DOTALL | 0x1000)
re.ASCII 4114
re.ASCII 0
re.IGNORECASE 4112
re.IGNORECASE 0
re.DOTALL 4096
re.DOTALL 0
re.ASCII|re.IGNORECASE|re.DOTALL|0x1000

output with print(m, rest) in __reduce_ex__.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@serhiy-storchaka

serhiy-storchaka commented Jun 16, 2022

Copy link
Copy Markdown
Member Author

I think that it is okay. Enums are considered singletons (it allows to pickle them by name). It would be even better to add the __copy__ method which returns self.

This PR is not ideal:

  1. The residual member is reduced as re.RegexFlag(0x1000) instead of just 0x1000.
  2. Named flags are reduced as getattr(re.RegexFlag, 'ASCII') instead of global re.RegexFlag.ASCII or even re.ASCII.

It is a matter of the following optimizations.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

serhiy-storchaka added 3 commits June 16, 2022 16:10
@serhiy-storchaka
Optimize.
79f7f7e
@serhiy-storchaka
Merge branch 'main' into enum-flags-pickle
ce9d147
@serhiy-storchaka
Merge branch 'main' into enum-flags-pickle
f1ba34c
@serhiy-storchaka
serhiy-storchaka merged commit 5369858 into python:main Jun 26, 2022
@miss-islington

miss-islington commented Jun 26, 2022

Copy link
Copy Markdown
Contributor

Thanks @serhiy-storchaka for the PR 🌮🎉.. I'm working now to backport this PR to: 3.11.
🐍🍒⛏🤖 I'm not a witch! I'm not a witch!

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the awaiting core review label Jun 26, 2022
@serhiy-storchaka
serhiy-storchaka deleted the enum-flags-pickle branch June 26, 2022 07:54
@miss-islington

miss-islington commented Jun 26, 2022

Copy link
Copy Markdown
Contributor

Sorry, @serhiy-storchaka, I could not cleanly backport this to 3.11 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker 536985814a7116f14c9bc90aa1b3e3d36d5b2367 3.11

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@miss-islington miss-islington assigned serhiy-storchaka Jun 26, 2022
serhiy-storchaka added a commit to serhiy-storchaka/cpython that referenced this pull request Jun 26, 2022
@serhiy-storchaka
[3.11] pythongh-93820: Pickle enum.Flag by name (pythonGH-93891).
632a1eb
(cherry picked from commit 5369858)

Co-authored-by: Serhiy Storchaka <[email protected]>
@bedevere-bot

bedevere-bot commented Jun 26, 2022

Copy link
Copy Markdown

GH-94288 is a backport of this pull request to the 3.11 branch.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the needs backport to 3.11 only security fixes label Jun 26, 2022
serhiy-storchaka added a commit that referenced this pull request Jun 26, 2022
@serhiy-storchaka
[3.11] gh-93820: Pickle enum.Flag by name (GH-93891). (GH-94288)
1b27ec5
(cherry picked from commit 5369858)

Co-authored-by: Serhiy Storchaka <[email protected]>
@serhiy-storchaka serhiy-storchaka removed their assignment Sep 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tiran tiran Awaiting requested review from tiran
@ethanfurman ethanfurman Awaiting requested review from ethanfurman ethanfurman is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

4 participants

@serhiy-storchaka @tiran @miss-islington @bedevere-bot
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.