GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — bpo-37207: Use PEP 590 vectorcall to speed up range(), list() and dict() by about 30% by markshannon · Pull Request #13930 · python/cpython

Conversation

@markshannon

@markshannon markshannon commented Jun 9, 2019 •
edited by bedevere-bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

https://bugs.python.org/issue37207

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

ZackerySpytz reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
@markshannon
markshannon requested a review from methane as a code owner June 9, 2019 09:40
@the-knights-who-say-ni the-knights-who-say-ni added the CLA signed label Jun 9, 2019
@bedevere-bot bedevere-bot added the awaiting core review label Jun 9, 2019
ulfalizer
ulfalizer reviewed Jun 9, 2019
View reviewed changes
Comment thread Objects/rangeobject.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

tirkarthi
tirkarthi reviewed Jun 9, 2019
View reviewed changes
Comment thread Objects/listobject.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

@tirkarthi

tirkarthi commented Jun 9, 2019

Copy link
Copy Markdown
Member

test_range seems to leak references. Using test.bisect_cmd gives me test.test_range.RangeTest.test_invalid_invocation as the relevant test.

➜  cpython git:(pr_13930) ./python.exe -m test -R 3:3 test_range
Run tests sequentially
0:00:00 load avg: 1.75 [1/1] test_range
beginning 6 repetitions
123456
......
test_range leaked [11, 11, 11] references, sum=33
test_range failed

== Tests result: FAILURE ==

1 test failed:
    test_range

Total duration: 19 sec 722 ms
Tests result: FAILURE
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

methane
methane reviewed Jun 10, 2019
View reviewed changes
Comment thread Objects/rangeobject.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

ZackerySpytz
ZackerySpytz reviewed Jun 11, 2019
View reviewed changes
Comment thread Objects/dictobject.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Objects/dictobject.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Objects/dictobject.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Objects/dictobject.c Outdated

@ZackerySpytz ZackerySpytz Jun 11, 2019

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think there should be a newline right after the (.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@jdemeyer

jdemeyer commented Jun 11, 2019

Copy link
Copy Markdown
Contributor

CC @vstinner because this PR actually affects the stable ABI for PyTypeObject (unlike #13858 where this was erroneously claimed)

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

jdemeyer
jdemeyer reviewed Jun 14, 2019
View reviewed changes
Comment thread Objects/call.c Outdated

@jdemeyer jdemeyer Jun 14, 2019 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change may or may not make sense, but in any case I think that it needs to be justified individually and a test added, preferably on a separate PR.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@jdemeyer jdemeyer Jun 19, 2019

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment in Include/cpython/abstract.h says that _PyStack_UnpackDict does not check the type of keyword arguments. So this change is inconsistent with that. If you think that this should be changed, please open a separate issue explaining the rationale for this change.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@markshannon markshannon Jun 23, 2019

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The additional validation should have been in PyVectorcall_Call which does need to do validation. I've moved it there.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@jdemeyer jdemeyer Jun 23, 2019

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still think that it's better to move that change to a different PR with rationale explained for this change.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@jdemeyer

jdemeyer commented Jun 14, 2019

Copy link
Copy Markdown
Contributor

Do you have an idea why this speeds up those calls that much? I'm just wondering if we could instead try to optimize the existing code path instead of implementing a mostly duplicate code path.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@jdemeyer

jdemeyer commented Jun 17, 2019

Copy link
Copy Markdown
Contributor

What are your plans for inheritance of tp_vectorcall? That should be clarified and documented.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

jdemeyer
jdemeyer requested changes Jul 4, 2019
View reviewed changes

@jdemeyer jdemeyer left a comment

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with the overall approach here. However I have various minor comments (some already mentioned):

  1. I'm not convinced that vectorcall should be used for dict(). There was a discussion about using METH_FASTCALL for dict.update() in bpo-29312 and the conclusion was that it was a bad idea. So either we need to argue that the conclusion on bpo-29312 was wrong or that it doesn't apply to the dict() constructor.
  2. You should try to minimize duplication of functionality. For example range_new and range_vectorcall serve essentially the same purpose, so why are there two functions? You could implement both in terms of a common function or implement range_new as
static PyObject *
range_new(PyTypeObject *type, PyObject *args, PyObject *kw)
{
    if (kw && PyDict_GET_SIZE(kw) != 0) {
        PyErr_Format(PyExc_TypeError, "range() takes no keyword arguments");
        return NULL;
    }
    return range_vectorcall(type, _PyTuple_ITEMS(args),
                            PyTuple_GET_SIZE(args), NULL);
}
  1. Use Py_ssize_t for the nargs variable.
  2. The change to PyVectorcall_Call does not belong here. Please open a separate PR for that.
  3. I'm getting a compiler warning
Objects/rangeobject.c:765:26: warning: initialization from incompatible pointer type [-Wincompatible-pointer-types]
         .tp_vectorcall = range_vectorcall
                          ^~~~~~~~~~~~~~~~

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@jdemeyer

jdemeyer commented Jul 4, 2019

Copy link
Copy Markdown
Contributor

Since dict() and dict.update() are very similar (basically, dict(*args, **kwargs) starts from an empty dict and then calls update(*args, **kwargs) on it), we should handle them together. I suggest to take that change out of this PR.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@methane

methane commented Jul 4, 2019

Copy link
Copy Markdown
Member

dict and dict.update are different.
dict(d1, **d2) is well known idiom to merge two dicts. While there is {**d1, **d2} now, many people are familiar with old idiom.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@jdemeyer

jdemeyer commented Jul 4, 2019

Copy link
Copy Markdown
Contributor

@methane, I don't understand which point you are trying to make.

Right now, there is a common implementation of dict() and dict.update() using VARARGS. I would like to change that to a common implementation of dict() and dict.update() using FASTCALL. Do you agree?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@jdemeyer jdemeyer mentioned this pull request Jul 5, 2019
bpo-37207: enable vectorcall for type.__call__ #14588
Merged
@jdemeyer

jdemeyer commented Jul 10, 2019

Copy link
Copy Markdown
Contributor

I opened https://bugs.python.org/issue37540 for the issue of checking that keyword names are strings.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@jdemeyer

jdemeyer commented Aug 5, 2019

Copy link
Copy Markdown
Contributor

I created #14588 and #14682 to separate two pieces from this PR. May I suggest again to merge those first? That would allow simplifying this PR and allowing to move forward with other PRs depending on #14588 and #14682.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

miss-islington pushed a commit that referenced this pull request Aug 15, 2019
@jdemeyer @miss-islington
bpo-37207: enable vectorcall for type.__call__ (GH-14588)
37806f4
Base PR for other PRs that want to play with `type.__call__` such as #13930 and #14589.

The author is really @markshannon I just made the PR.


https://bugs.python.org/issue37207



Automerge-Triggered-By: @encukou
lisroach pushed a commit to lisroach/cpython that referenced this pull request Sep 10, 2019
@jdemeyer @lisroach
bpo-37207: enable vectorcall for type.__call__ (pythonGH-14588)
3c51383
Base PR for other PRs that want to play with `type.__call__` such as python#13930 and python#14589.

The author is really @markshannon I just made the PR.


https://bugs.python.org/issue37207



Automerge-Triggered-By: @encukou
markshannon added 8 commits September 11, 2019 10:18
@markshannon
Use vector call for range().
c01d5b5
@markshannon
Use vector call for list()
6075fc1
@markshannon
Use vector call for dict()
9d86ea5
@markshannon
Add news entry.
20eea92
@markshannon
Fix up error messages in list/dict/range vectorcall implements to mat…
821af67
…ch previous error messages.
@markshannon
Fix refcount leak in range_vectorcall.
0049db8
@markshannon
Reformat vectorcall functions to conform to standard coding style.
8d4a250
@markshannon
Minor formatting changes.
989a3d7
@markshannon
markshannon force-pushed the vectorcall-range branch from 4671080 to fec6969 Compare September 11, 2019 09:23
@encukou

encukou commented Sep 11, 2019

Copy link
Copy Markdown
Member

It would make sense to delegate tp_call for range() and list() to vectorcall, rather than duplicating the implementations.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

markshannon added 2 commits September 12, 2019 10:08
@markshannon
Inherit tp_vectorcall when safe to do so.
525ff6c
@markshannon
Fix compiler warning.
c319141
@markshannon
markshannon force-pushed the vectorcall-range branch from fec6969 to c319141 Compare September 12, 2019 09:11
methane
methane reviewed Sep 12, 2019
View reviewed changes
Comment thread Objects/typeobject.c
{
type->tp_flags |= _Py_TPFLAGS_HAVE_VECTORCALL;
/* Also inherit tp_vectorcall if tp_call of the
* metaclass is type.__call__ */

@methane methane Sep 12, 2019

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* metaclass is type.__call__ */
* metaclass is type.__call__ */

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
methane
methane reviewed Sep 12, 2019
View reviewed changes
Comment thread Objects/dictobject.c
if (nargs == 1) {
int err = 0;
PyObject *arg = args[0];
_Py_IDENTIFIER(keys);

@methane methane Sep 12, 2019

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't like two same identifiers in single source file.
Please make _Py_IDENTIFIER(keys); global static variable.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
DinoV pushed a commit to DinoV/cpython that referenced this pull request Jan 14, 2020
@jdemeyer @DinoV
bpo-37207: enable vectorcall for type.__call__ (pythonGH-14588)
c784435
Base PR for other PRs that want to play with `type.__call__` such as python#13930 and python#14589.

The author is really @markshannon I just made the PR.


https://bugs.python.org/issue37207



Automerge-Triggered-By: @encukou
@encukou encukou mentioned this pull request Feb 11, 2020
bpo-37207: Use vectorcall for range() #18464
Merged
@encukou

encukou commented Feb 11, 2020

Copy link
Copy Markdown
Member

This PR has 4 relatively independent parts, and all of it seems stuck on one of them. I opened #18464 to get range() out of the way.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

miss-islington pushed a commit that referenced this pull request Feb 18, 2020
@encukou
bpo-37207: Use vectorcall for range() (GH-18464)
6e35da9
This continues the `range()` part of #13930. The complete pull request is stalled on discussions around dicts, but `range()` should not be controversial. (And I plan to open PRs for other parts if this is merged.)
On top of Mark's change, I unified `range_new` and `range_vectorcall`, which had a lot of duplicate code.


https://bugs.python.org/issue37207
@encukou encukou mentioned this pull request Mar 11, 2020
bpo-37207: Use vertorcall for list() #18928
Merged
@vstinner

vstinner commented Mar 30, 2020

Copy link
Copy Markdown
Member

This PR was splitted into multiple smaller PRs which all are now all merged: see https://bugs.python.org/issue37207

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner vstinner closed this Mar 30, 2020
websurfer5 pushed a commit to websurfer5/cpython that referenced this pull request Jul 20, 2020
@jdemeyer @websurfer5
bpo-37207: enable vectorcall for type.__call__ (pythonGH-14588)
9cea68a
Base PR for other PRs that want to play with `type.__call__` such as python#13930 and python#14589.

The author is really @markshannon I just made the PR.


https://bugs.python.org/issue37207



Automerge-Triggered-By: @encukou
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@methane methane methane left review comments
@tirkarthi tirkarthi tirkarthi left review comments
@jdemeyer jdemeyer jdemeyer requested changes
+2 more reviewers
@ulfalizer ulfalizer ulfalizer left review comments
@ZackerySpytz ZackerySpytz ZackerySpytz left review comments
Reviewers whose approvals may not affect merge requirements

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

awaiting core review

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

10 participants

@markshannon @tirkarthi @jdemeyer @methane @encukou @vstinner @ulfalizer @ZackerySpytz @the-knights-who-say-ni @bedevere-bot
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.