Description

colesbury

opened on Nov 9, 2023

Issue body actions

Feature or enhancement

The Modules/_json.c parser is mostly stateless (or the state is immutable). The one exception is the "memo" dictionary, which is used to avoid duplicate PyUnicodeObject instances for the same JSON C strings.

cpython/Modules/_json.c

Lines 696 to 700 in 289af86

	memokey = PyDict_SetDefault(s->memo, key, key);
	if (memokey == NULL) {
	goto bail;
	}
	Py_SETREF(key, Py_NewRef(memokey));

The memo dictionary is already cleared after each call scan_once:

cpython/Modules/_json.c

Line 1118 in 289af86

PyDict_Clear(self->memo);

We should move the creation and destruction of the memo dict to the invocation of scan_once instead of having it as part of the module state. This will avoid contention on the dictionary locks in --disable-gil builds if multiple threads are concurrently parsing JSON strings.

For an example modification, see colesbury/nogil-3.12@964bb33962.

Linked PRs

• gh-111928: make "memo" dict local to scan_once call #112005

Activity

1. colesbury

   added

   type-featureA feature request or enhancementA feature request or enhancement

   3.13only security fixesonly security fixes

   topic-free-threading

   on Nov 9, 2023
2. AlexWaygood

   added

   extension-modulesC modules in the Modules dirC modules in the Modules dir

   on Nov 10, 2023
3. aisk commented on Nov 12, 2023

   

   aisk

   on Nov 12, 2023

   Member

   More actions

   I just took a look at the example modification, and run the test on it locally and it succeded. I think there is no more works to do, so just cherry-picked it and made a PR.

   If anyone think there need some improves, I want to investigate on it.
4. corona10

   added a commit that references this issue on Nov 13, 2023

   gh-111928: make "memo" dict local to scan_once call (gh-112005)

   d0058cb
5. colesbury

   assigned

   aisk

   on Nov 13, 2023
6. colesbury commented on Nov 13, 2023

   

   colesbury

   on Nov 13, 2023

   ContributorAuthor

   More actions

   Thanks @aisk!
7. colesbury

   closed this as completedon Nov 13, 2023
8. aisk

   added a commit that references this issue on Feb 11, 2024

   pythongh-111928: make "memo" dict local to scan_once call (pythongh-1…

   1758976
9. Glyphack

   added a commit that references this issue on Sep 2, 2024

   pythongh-111928: make "memo" dict local to scan_once call (pythongh-1…

   68ce15f

Issue actions