GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-111489: Add PyList_FromArrayMoveRef() function by vstinner · Pull Request #112975 · python/cpython

Conversation

@vstinner

@vstinner vstinner commented Dec 11, 2023 •
edited by github-actions Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

Add new functions to create tuple and list from arrays:

  • PyTuple_FromArray().
  • PyTuple_FromArrayMoveRef().
  • PyList_FromArrayMoveRef().

Add tests on new functions.

  • Issue: [C API] Make _PyTuple_FromArray() function public #111489

📚 Documentation preview 📚: https://cpython-previews--112975.org.readthedocs.build/

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@bedevere-app bedevere-app Bot added the awaiting core review label Dec 11, 2023
@bedevere-app bedevere-app Bot mentioned this pull request Dec 11, 2023
[C API] Make _PyTuple_FromArray() function public #111489
Closed
@vstinner

vstinner commented Dec 11, 2023

Copy link
Copy Markdown
Member Author

I tried to clear array items (or set bytes to a pattern, such as PYMEM_DEADBYTE=0xDD) in PyTuple_FromArrayMoveRef() and PyList_FromArrayMoveRef() in debug mode, but it broke current code which expect the array to remain usable. Pseudo-code:

PyObject *const * args = &PyTuple_GET_ITEM(callargs, 0);
for (size_t i = 0; i < argcount; i++) {
    Py_INCREF(args[i]);
}

PyObject *u = _PyTuple_FromArraySteal(args, argcount);
// ... check if u == NULL ...

// ... code using callargs ...

So I chose to leave array unchanged.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner
pythongh-111489: Add PyList_FromArrayMoveRef() function
b12186e
Add new functions to create tuple and list from arrays:

* PyTuple_FromArray().
* PyTuple_FromArrayMoveRef().
* PyList_FromArrayMoveRef().

Add tests on new functions.
@vstinner
vstinner force-pushed the pylist_fromarray branch from 7b20ab0 to b12186e Compare December 11, 2023 18:09
@vstinner

vstinner commented Dec 11, 2023

Copy link
Copy Markdown
Member Author

PyObject* PyList_FromArrayMoveRef(PyObject *const *array, Py_ssize_t size)

I would prefer to declare array type as PyObject **, but since the function does not modify the array, I chose to declare the type using const.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner vstinner mentioned this pull request Dec 11, 2023
Terminology for managing ownership capi-workgroup/api-evolution#25
Open
@sweeneyde

sweeneyde commented Dec 11, 2023

Copy link
Copy Markdown
Member

I think it's confusing that the function consumes references when it succeeds but does not consume them when it fails. I would think that it should always consume references or never consume references.

If it always consumes references, then one can write

    PyObject *pair[2];
    pair[0] = pair[1] = something;
    return _PyList_FromArraySteal(pair);

I think the only-sometimes-consuming was why there used to be a lot of error-handling mistakes in module initialization code with PyModule_AddObject. The docs note that

Unlike other functions that steal references, PyModule_AddObject() only releases the reference to value on success.

I would think that we would want to replicate the always-consuming "other functions" (e.g. PyTuple_SetItem) style, not the only-sometimes-consuming PyModule_AddObject style.

Is there a reason to prefer this style?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

serhiy-storchaka
serhiy-storchaka reviewed Dec 12, 2023
View reviewed changes

@serhiy-storchaka serhiy-storchaka left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Ref suffix is used in many new functions, would not it cause confusion?

What if use PyTuple_FromArrayMove() or PyTuple_FromArrayMoveOwnership()? Or maybe PyTuple_CopyFromArray() and PyTuple_MoveFromArray()?

Why there is no copying API for list?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Doc/c-api/tuple.rst
tuple.

* Return a new reference on success.
* Set an exception and return NULL on error.

@serhiy-storchaka serhiy-storchaka Dec 12, 2023

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* Set an exception and return NULL on error.
* Set an exception and return ``NULL`` on error.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Lib/test/test_capi/test_list.py
# Test PyList_FromArrayMoveRef()
list_fromarraymoveref = _testcapi.list_fromarraymoveref

lst = [object() for _ in range(5)]

@serhiy-storchaka serhiy-storchaka Dec 12, 2023

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would use the following example:

Suggested change
lst = [object() for _ in range(5)]
lst = [[i] for _ in range(5)]

Better repr.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@vstinner vstinner Dec 12, 2023

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First, I used list("abdef"), but all items are immortal and so tests didn't detect refcount bugs in my C code. I used object() to make sure that each object is unique.

I don't think that repr is important here, the test is not supposed to fail :-)

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Lib/test/test_capi/test_list.py
Comment on lines +351 to +352
copy = list_fromarraymoveref(lst)
self.assertEqual(copy, lst)

@serhiy-storchaka serhiy-storchaka Dec 12, 2023

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test also with zero and negative size. with too large size (causing MemoryError or OverflowError), with array=NULL, and with NULLs in the array. We should know what cases raise exceptions, what cases crash and what cases are valid, and how errors are handled.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Objects/tupleobject.c
Comment on lines 401 to 406
if (tuple == NULL) {
for (Py_ssize_t i = 0; i < n; i++) {
Py_DECREF(src[i]);
for (Py_ssize_t i = 0; i < size; i++) {
Py_DECREF(array[i]);
}
return NULL;
}

@serhiy-storchaka serhiy-storchaka Dec 12, 2023

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does not match the documentation.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@vstinner

vstinner commented Dec 12, 2023

Copy link
Copy Markdown
Member Author

What if use PyTuple_FromArrayMove() or PyTuple_FromArrayMoveOwnership()? Or maybe PyTuple_CopyFromArray() and PyTuple_MoveFromArray()?

I suggest discussing the name at: capi-workgroup/api-evolution#25

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner

vstinner commented Dec 12, 2023

Copy link
Copy Markdown
Member Author

Why there is no copying API for list?

I only exposed what already existed. But yeah, when I wrote the PR, I also noticed that it seems like there is "a hole" in the API :-) I can add a "copy" API for list as well.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner
vstinner marked this pull request as draft December 14, 2023 23:59
@bedevere-app bedevere-app Bot removed the awaiting core review label Dec 14, 2023
@vstinner

vstinner commented Dec 15, 2023

Copy link
Copy Markdown
Member Author

I convert this PR to a draft until we agree on naming: capi-workgroup/api-evolution#25

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner

vstinner commented Dec 20, 2023

Copy link
Copy Markdown
Member Author

I convert this PR to a draft until we agree on naming: capi-workgroup/api-evolution#25

I created this PR to have a more concrete API to discuss the naming.

I prefer to close the PR for now, until we agree on naming.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner vstinner closed this Dec 20, 2023
@vstinner
vstinner deleted the pylist_fromarray branch December 20, 2023 11:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@serhiy-storchaka serhiy-storchaka serhiy-storchaka left review comments
@AA-Turner AA-Turner Awaiting requested review from AA-Turner AA-Turner will be requested when the pull request is marked ready for review AA-Turner is a code owner
@ZeroIntensity ZeroIntensity Awaiting requested review from ZeroIntensity ZeroIntensity will be requested when the pull request is marked ready for review ZeroIntensity is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

3 participants

@vstinner @sweeneyde @serhiy-storchaka
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.