GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — [mypyc] Use bitsets in data-flow analyses by Jahongir-Qurbonov · Pull Request #22426 · python/mypy

Conversation

@Jahongir-Qurbonov

@Jahongir-Qurbonov Jahongir-Qurbonov commented Nov 10, 2026

Copy link
Copy Markdown
Contributor

run_analysis keeps a Python set for every basic block, and a new one for every op whose gen/kill sets aren't empty. In large functions these sets have thousands of items, so the data-flow analyses take O(ops * values) time and memory. The must-defined and borrowed-arguments analyses are the worst, since a must analysis starts every block from a copy of the full universe. This makes the refcount transform very slow for large functions, such as generated code or long module top levels.

This PR represents the sets as int bit masks:

  • BitIndex assigns a bit position to each value an analysis sees.
  • The worklist algorithm works on masks, so union, intersection and difference are integer operations. Since ints are immutable, an op whose gen/kill sets are empty shares the previous result, as before.
  • The gen/kill masks of each op are computed once, instead of running the visitor again in the op-level pass.
  • The results are BitSet objects that implement collections.abc.Set, so the code using the results only needed annotation changes (set[Value] -> AbstractSet[Value] in refcount.py).
  • analyze_live_regs, analyze_borrowed_arguments and analyze_must_defined_regs take an optional index. The refcount transform shares one index between the three, because it combines their results (e.g. live - live - borrowed after a branch); with a shared index these operations are integer operations too. With different indexes, BitSet still works and, like set, iterates over the smaller operand.
  • Reflected operations (set - bitset, set & bitset, set | bitset) return a real set, since typeshed says the result is a set and compiled code checks this. find_always_defined_attributes relies on it.

Benchmark

A function with n statements of this form:

    a{i} = xs[{i % 7}] + {i}
    if a{i}:
        t = a{i}

Each analysis timed separately, then insert_ref_count_opcodes (interpreted). Max RSS is for the whole process:

n ops / blocks live borrowed must-defined refcount transform max RSS
1000 9k / 2k master 0.02s 0.13s 0.61s 0.83s 1.2 GiB
this PR 0.02s 0.00s 0.01s 0.09s 73 MiB
2000 18k / 4k master 0.03s 0.67s 2.2s 3.5s 4.7 GiB
this PR 0.06s 0.01s 0.02s 0.23s 109 MiB
4000 36k / 8k master 0.09s 2.2s 93s 28–84s (swapping) 16 GiB
this PR 0.19s 0.02s 0.05s 0.63s 225 MiB

Liveness is a bit slower, because live sets are small and a mask grows with the number of values. It is still a small part of the total.

On ordinary code, compiling mypy/checker.py, mypy/checkexpr.py, mypy/semanal.py and mypy/messages.py with the analysis and transform modules compiled by mypyc:

master this PR
insert_ref_count_opcodes 1.59–1.73s 0.94–0.95s
insert_uninit_checks 0.28–0.29s 0.21s

Testing

  • The generated C for those four modules is identical to master, both with interpreted and compiled analysis modules, and the IR for the benchmark functions is identical.
  • The full mypyc test suite passes interpreted. The IR, refcount, exceptions, always-defined, spill and run tests also pass with dataflow.py and its users compiled by mypyc.
  • New mypyc/test/test_bitset.py covers BitSet operations: same index, different index, mixed with set, and the types the reflected operations return.

This is independent of #22425 (which removes temporaries from the must-defined universe). The two merge without conflicts and help in different ways: #22425 makes the sets smaller, this PR makes each item cost a bit instead of a hash table entry.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@Jahongir-Qurbonov
[mypyc] Use bitsets in data-flow analyses
a4b17a2
run_analysis stored a Python set for every basic block and for every op
whose gen/kill sets were non-empty. For large functions these sets have
thousands of items, so the analyses used O(ops * values) memory and time.
The must-defined and borrowed-arguments analyses were the worst, since
they start from the full universe in every block.

Represent the sets as int bit masks instead. The results are BitSet
objects, which implement collections.abc.Set, so the users of the
analyses only needed annotation changes. The refcount transform shares a
BitIndex between its three analyses, so the set operations it does
between their results are integer operations.

The generated IR and C are unchanged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

1 participant

@Jahongir-Qurbonov
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.