GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — bpo-43503: Make limited API objects effectively immutable. by ericsnowcurrently · Pull Request #24828 · python/cpython

bpo-43503: Make limited API objects effectively immutable. #24828

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed

ericsnowcurrently wants to merge 16 commits into python:master from ericsnowcurrently:globals-immortal-objects-flags

Uh oh!

There was an error while loading. Please reload this page.

Closed

bpo-43503: Make limited API objects effectively immutable. #24828

Show file tree

Hide file tree

Changes from all commits

Commits

Show all changes

16 commits

Select commit Hold shift + click to select a range

4951703

Add support for immortal objects (large neg int means "immortal").

ericsnowcurrently Dec 10, 2020

4b1eea3

Switch to a positive number.

ericsnowcurrently Dec 10, 2020

a34f285

Add Py_IMMORTAL_CONST_REFCOUNTS.

ericsnowcurrently Dec 10, 2020

3966551

Add a NEWS entry.

ericsnowcurrently Mar 11, 2021

1a3b4da

Add docs.

ericsnowcurrently Mar 11, 2021

728f24b

Drop special handling of negative refcounts.

ericsnowcurrently Mar 11, 2021

2536e8b

Do not reset the refcount in _PyObject_IsImmortal().

ericsnowcurrently Mar 11, 2021

518e9e8

Do not allow immortal objects in the limited API.

ericsnowcurrently Mar 11, 2021

277e1ea

Do not re-declare _PyObject_IsImmortal.

ericsnowcurrently Mar 11, 2021

edef342

Go back to using Py_SET_REFCNT() in _Py_NewReference().

ericsnowcurrently Mar 12, 2021

4d22e00

Fix an accidental line removal.

ericsnowcurrently Mar 12, 2021

838f88f

Add more comments and drop the internal-only macros from the docs.

ericsnowcurrently Mar 12, 2021

00948a0

Drop a holdover from immortal objects with negative refcounts.

ericsnowcurrently Mar 15, 2021

00504d9

Move _Py_SET_REFCNT() out of the header file.

ericsnowcurrently Mar 15, 2021

e6fc84f

Define _py_is_immortal() only for Py_IMMORTAL_CONST_REFCOUNTS.

ericsnowcurrently Mar 15, 2021

f103804

Drop Py_IMMORTAL_CONST_REFCOUNTS and make immortal objects internal.

ericsnowcurrently Mar 12, 2021

File filter

Filter by extension .c (1) .h (2) .rst (1) All 3 file types selected

Viewed files

Clear filters

Conversations

Failed to load comments.

Uh oh!

There was an error while loading. Please reload this page.

Jump to

Jump to file

Failed to load files.

Uh oh!

There was an error while loading. Please reload this page.

Diff view

Unified

Split

Hide whitespace

Diff view

Unified

Split

Hide whitespace

There are no files selected for viewing

49 changes: 49 additions & 0 deletions 49 Include/internal/pycore_object.h

Show comments

View file

Open in desktop

Original file line number	Diff line number	Diff line change
Expand Up		@@ -30,6 +30,55 @@ _PyType_HasFeature(PyTypeObject *type, unsigned long feature) {
		extern void _PyType_InitCache(PyInterpreterState *interp);


		/* Immortal Objects
		*
		* An "immortal" object is one for which Py_DECREF() will never try
		* to deallocate it.
		*
		* At the moment this API is strictly internal. However, if it proves
		* helpful for extension authors we may move it to the public API. */

		#define _Py_IMMORTAL_OBJECTS 1
Comment thread Outdated vstinner Mar 13, 2021 Copy link Copy Markdown Member There was a problem hiding this comment. Choose a reason for hiding this comment The reason will be displayed to describe this comment to others. Learn more. I don't get the purpose of this macro since it's always defined. Sorry, something went wrong. Uh oh! There was an error while loading. Please reload this page. All reactions ericsnowcurrently Mar 15, 2021 Copy link Copy Markdown Member Author There was a problem hiding this comment. Choose a reason for hiding this comment The reason will be displayed to describe this comment to others. Learn more. It's there only as a marker relative to the public API. However, it is not necessary at this point and I'll drop it. Sorry, something went wrong. Uh oh! There was an error while loading. Please reload this page. All reactions

		/* The implementation-independent API is only the following functions: */
		PyAPI_FUNC(int) _PyObject_IsImmortal(PyObject *);
		PyAPI_FUNC(void) _PyObject_SetImmortal(PyObject *);

		/* In the actual implementation we set the refcount to some positive
		* value that we would never expect to be reachable through use of
		* Py_INCREF() in a program.
		*
		* The only parts that should be used directly are the two
		* _Py*Object_HEAD_IMMORTAL_INIT() macros.
		*/

		/* _PyObject_IMMORTAL_BIT is the bit in the refcount value (Py_ssize_t)
		* that we use to mark an object as immortal. It shouldn't ever be
		* part of the public API.
		*
		* The GC bit-shifts refcounts left by two, and after that shift we still
		* need this to be >> 0, so leave three high zero bits (the sign bit and
		* room for a shift of two.) */
		#define _PyObject_IMMORTAL_BIT (1LL << (8 * sizeof(Py_ssize_t) - 4))

		/* _PyObject_IMMORTAL_INIT_REFCNT is the initial value we use for
		* immortal objects. It shouldn't ever be part of the public API.
		*
		* We leave plenty of room to preserve _PyObject_IMMORTAL_BIT. */
		#define _PyObject_IMMORTAL_INIT_REFCNT \
		(_PyObject_IMMORTAL_BIT + (_PyObject_IMMORTAL_BIT / 2))

		/* These macros are drop-in replacements for the corresponding
		* Py*Object_HEAD_INIT() macros. They will probably become
		* part of the public API. */
		#define _PyObject_HEAD_IMMORTAL_INIT(type) \
		{ _PyObject_EXTRA_INIT _PyObject_IMMORTAL_INIT_REFCNT, type },
		#define _PyVarObject_HEAD_IMMORTAL_INIT(type, size) \
		{ PyObject_HEAD_IMMORTAL_INIT(type) size },

		/* end Immortal Objects */


		/* Inline functions trading binary compatibility for speed:
		_PyObject_Init() is the fast version of PyObject_Init(), and
		_PyObject_InitVar() is the fast version of PyObject_InitVar().
Expand Down

8 changes: 5 additions & 3 deletions 8 Include/object.h

Show comments

View file

Open in desktop

Original file line number	Diff line number	Diff line change
Expand Up		@@ -141,9 +141,7 @@ static inline int _Py_IS_TYPE(const PyObject ob, const PyTypeObject type) {
		#define Py_IS_TYPE(ob, type) _Py_IS_TYPE(_PyObject_CAST_CONST(ob), type)


		static inline void _Py_SET_REFCNT(PyObject *ob, Py_ssize_t refcnt) {
		ob->ob_refcnt = refcnt;
		}
		PyAPI_FUNC(void) _Py_SET_REFCNT(PyObject *ob, Py_ssize_t refcnt);
		#define Py_SET_REFCNT(ob, refcnt) _Py_SET_REFCNT(_PyObject_CAST(ob), refcnt)


Expand Down Expand Up		@@ -427,6 +425,10 @@ PyAPI_FUNC(void) _Py_NegativeRefcount(const char *filename, int lineno,

		PyAPI_FUNC(void) _Py_Dealloc(PyObject *);

		#ifdef Py_IMMORTAL_CONST_REFCOUNTS
		static inline int _py_is_immortal(PyObject *); // forward
		#endif

		static inline void _Py_INCREF(PyObject *op)
		{
		#ifdef Py_REF_DEBUG
Expand Down

5 changes: 5 additions & 0 deletions 5 Misc/NEWS.d/next/Core and Builtins/2021-03-11-13-03-15.bpo-40255._1LfiG.rst

Show comments

View file

Open in desktop

Original file line number	Diff line number	Diff line change
		@@ -0,0 +1,5 @@
		There is now internal support for "immortal" objects (with a "private"
		C-API). Those are objects that will never be deleted, like the
		singletons and static types. This will benefit the C-API and
		subinterpreters. For now the API is currently intended only
		for internal use.

36 changes: 36 additions & 0 deletions 36 Objects/object.c

Show comments

View file

Open in desktop

Original file line number	Diff line number	Diff line change
Expand Up		@@ -136,6 +136,30 @@ Py_DecRef(PyObject *o)
		Py_XDECREF(o);
		}

		void
		_Py_SET_REFCNT(PyObject *ob, Py_ssize_t refcnt) {
		if (_PyObject_IsImmortal(ob)) {
		// XXX It may be worth emitting a warning here.
		return;
		}
		ob->ob_refcnt = refcnt;
		}

		int
		_PyObject_IsImmortal(PyObject *ob)
		{
		if ((ob->ob_refcnt & _PyObject_IMMORTAL_BIT) == 0) {
		return 0;
		}
		return 1;
		}

		void
		_PyObject_SetImmortal(PyObject *ob)
		{
		ob->ob_refcnt = _PyObject_IMMORTAL_INIT_REFCNT;
		}

		PyObject *
		PyObject_Init(PyObject op, PyTypeObject tp)
		{
Expand Down Expand Up		@@ -1728,11 +1752,14 @@ _PyTypes_Init(void)
		return status;
		}

		// XXX We can stop calling _PyObject_SetImmortal() once we change
		// all the static types to use PyVarObject_HEAD_IMMORTAL_INIT.
		#define INIT_TYPE(TYPE, NAME) \
		do { \
		if (PyType_Ready(TYPE) < 0) { \
		return _PyStatus_ERR("Can't initialize " NAME " type"); \
		} \
		_PyObject_SetImmortal((PyObject *)TYPE); \
Comment thread Outdated vstinner Mar 13, 2021 Copy link Copy Markdown Member There was a problem hiding this comment. Choose a reason for hiding this comment The reason will be displayed to describe this comment to others. Learn more. Don't do that: using static types from multiple interpreters in parallel is unsafe. See for example the PyTypeObject.tp_subclasses issue. Sorry, something went wrong. Uh oh! There was an error while loading. Please reload this page. gpshead reacted with thumbs up emoji All reactions 👍 1 reaction gpshead Mar 14, 2021 Copy link Copy Markdown Member There was a problem hiding this comment. Choose a reason for hiding this comment The reason will be displayed to describe this comment to others. Learn more. Yuck, but yes. This detail seems to be a particular problem for the lofty idea of sub-interpreters one day eventually having their own GIL... Sorry, something went wrong. Uh oh! There was an error while loading. Please reload this page. All reactions ericsnowcurrently Mar 15, 2021 Copy link Copy Markdown Member Author There was a problem hiding this comment. Choose a reason for hiding this comment The reason will be displayed to describe this comment to others. Learn more. Right. This is something that has to be addressed before this PR can proceed. Sorry, something went wrong. Uh oh! There was an error while loading. Please reload this page. All reactions
		} while (0)

		INIT_TYPE(&PyBaseObject_Type, "object");
Expand Down Expand Up		@@ -1807,10 +1834,19 @@ _PyTypes_Init(void)
		#undef INIT_TYPE
		}

		/* _Py_NewReference() is called in the following situations:
		* - for newly allocatoed objects, via _PyObject_Init()
		* - when resizing immutable objects (bytes, unicode, tuple)
		* - when "allocating" from a freelist
		* - when resurrecting an object
		*/

		void
		_Py_NewReference(PyObject *op)
		{
		// None of the cases above should ever apply to immortal objects.
		assert(!_PyObject_IsImmortal(op));

		if (_Py_tracemalloc_config.tracing) {
		_PyTraceMalloc_NewReference(op);
		}
Expand Down

Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.

GeezerVison

bpo-43503: Make limited API objects effectively immutable. #24828

Uh oh!

bpo-43503: Make limited API objects effectively immutable. #24828

Uh oh!

Uh oh!

Diff view

Diff view

There are no files selected for viewing

vstinner Mar 13, 2021

Choose a reason for hiding this comment

Uh oh!

ericsnowcurrently Mar 15, 2021

Choose a reason for hiding this comment

Uh oh!

vstinner Mar 13, 2021

Choose a reason for hiding this comment

Uh oh!

gpshead Mar 14, 2021

Choose a reason for hiding this comment

Uh oh!

ericsnowcurrently Mar 15, 2021

Choose a reason for hiding this comment

Uh oh!