GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — bpo-36876: Avoid static locals. by ericsnowcurrently · Pull Request #13372 · python/cpython

Conversation

@ericsnowcurrently

@ericsnowcurrently ericsnowcurrently commented May 17, 2019 •
edited by bedevere-bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

This patch touches a lot of files but does only two very specific things:

  • move static locals to global scope (if they aren't truly global)
  • move all locally scoped _Py_IDENTIFIER() to global scope

Moving these to global scope makes it easier to identify problematic usage via Tools/c-globals/check-c-globals.py. It also has the side effect of dropping a number of duplicate _Py_IDENTIFIER(), thus saving a little bit of space. :)

The overall objective is to eliminate any global state that isn't truly process-global (and to ensure the remaining global state is thread-safe). This is only one of the early steps.

https://bugs.python.org/issue36876

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@ericsnowcurrently
ericsnowcurrently requested review from 1st1, asvetlov, berkerpeksag, brettcannon, encukou, gpshead, methane, ncoghlan, rhettinger, skrah and warsaw as code owners May 17, 2019 00:46
@the-knights-who-say-ni the-knights-who-say-ni added the CLA signed label May 17, 2019
@bedevere-bot bedevere-bot added the awaiting core review label May 17, 2019
ericsnowcurrently added 11 commits May 16, 2019 18:51
@ericsnowcurrently
Fix static locals in helper modules.
5c7ad09
@ericsnowcurrently
Fix static locals in testing modules.
df8c0d4
@ericsnowcurrently
Fix static locals in builtin modules.
a9cfc15
@ericsnowcurrently
Fix static locals in stdlib extension modules.
c4689e0
@ericsnowcurrently
Verify static locals in the parser code.
6e9289c
@ericsnowcurrently
Fix static locals for objects.
6dc09fa
@ericsnowcurrently
Fix static locals for core runtime code.
e21b68f
@ericsnowcurrently
Move local _Py_IDENTIFER to global for modules.
b1afab1
@ericsnowcurrently
Move local _Py_IDENTIFER to global for objects.
6902106
@ericsnowcurrently
Move local _Py_IDENTIFER to global for core files.
055fa11
@ericsnowcurrently
Add a NEWS entry.
f9aca0c
@ericsnowcurrently
ericsnowcurrently force-pushed the avoid-static-locals branch from 702d6cb to f9aca0c Compare May 17, 2019 00:57
asvetlov
asvetlov approved these changes May 17, 2019
View reviewed changes

@asvetlov asvetlov left a comment

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

asyncio part looks good

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@bedevere-bot bedevere-bot added awaiting merge and removed awaiting core review labels May 17, 2019
skrah
skrah reviewed May 17, 2019
View reviewed changes
Comment thread Modules/_decimal/libmpdec/context.c
mpd_setminalloc(mpd_ssize_t n)
{
static int minalloc_is_set = 0;
static int minalloc_is_set = 0; // Static is okay here (process-global).

@skrah skrah May 17, 2019

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please don't change libmpdec, even if it's just a comment. :)

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
vstinner
vstinner requested changes May 17, 2019
View reviewed changes

@vstinner vstinner left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

move all locally scoped _Py_IDENTIFIER() to global scope

I dislike this change. I prefer to keep "local" static variables in functions. I don't see how moving them as globals is solving any issue, apart your practical issue of detecting them with your tool.

I would prefer to see an API to support static variable per interpreter.

I would suggest to develop an hash table similar to thread TLS: each static variable would have a global unique identifier, and its value would stored in an hash table in the PyInterpreterState. The problem is how to initialize the global unique identifier in a safe way (thread safety / atomicity). But that could be implemented using "local static" variables.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@bedevere-bot bedevere-bot removed the awaiting merge label May 17, 2019
@bedevere-bot

bedevere-bot commented May 17, 2019

Copy link
Copy Markdown

When you're done making the requested changes, leave the comment: I have made the requested changes; please review again.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot added the awaiting changes label May 17, 2019
vstinner
vstinner reviewed May 17, 2019
View reviewed changes
Comment thread Modules/_datetimemodule.c
format_ctime(PyDateTime_Date *date, int hours, int minutes, int seconds)
{
static const char * const DayNames[] = {
static const char * const DayNames[] = { // Static is okay here (immutable data).

@vstinner vstinner May 17, 2019

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are these comments really useful? To me, it seems obvious that static constants are safe. I'm not sure of the value of such comment.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@tiran tiran May 17, 2019

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A comment sounds useful to flag all local statics as "good" and for a linter that checks for unverified local static vars. I'm ok with the comments, but maybe a shorter one like // static ok.?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@berkerpeksag berkerpeksag May 17, 2019

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with @vstinner. If you really want to keep these comments, please move them to separate lines to avoid noise when using git blame.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@vstinner

vstinner commented May 17, 2019

Copy link
Copy Markdown
Member

(...) The problem is how to initialize the global unique identifier in a safe way (thread safety / atomicity). But that could be implemented using "local static" variables.

I forgot to mention that "local static variables" would be easier to initialize properly, since they cannot be accessed before the function is called: before Python is initialized.

The case of global static variables is worse: we cannot statically initialize them, we need something to dynamically initialize them. Maybe at the first access?

About the global unique identifier, one solution would be to use the memory address of the variable: it should be unique. I don't see why a compiler would merge two variables.

Draft pseudo API:

void get_interpreter_specific(uintptr_t key, size_t size, void *value);
int set_interpreter_specific(uintptr_t key, size_t size, void *value);

#define PyStaticVariable(TYPE) static TYPE
// FIXME: add assertion to ensure that VAR is a PyStaticVariable
#define PyStaticVariable_get(VAR, VALUE) \
   get_interpreter_specific((uintptr_t)&(VAR), sizeof(VAR), &(VALUE))
#define PyStaticVariable_set(VAR, VALUE) \
   set_interpreter_specific((uintptr_t)&(VAR), sizeof(VAR), &(VALUE))

PyObject* func(void)
{
PyStaticVariable(int) var;

// if var doesn't exist in the interpreter local storage,
// its value is always 0 by default, for any type,
// as global static variables in C
int value;
PyStaticVariable_get(var, value);
value += 1;
if (PyStaticVariable_set(var, value) < 0) { return NULL; }

Py_RETURN_VALUE;
}

Problem: we cannot pre-allocate the storage of "var" value into the interpreter "local storage" hash table, so set_interpreter_specific() can fail with a memory allocation failure :-(

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

tiran
tiran approved these changes May 17, 2019
View reviewed changes
@bedevere-bot bedevere-bot added awaiting merge and removed awaiting changes labels May 17, 2019
@tiran

tiran commented May 17, 2019

Copy link
Copy Markdown
Member

@ericsnowcurrently Please re-generate the ast module:

Generated files not up to date
 M Python/Python-ast.c
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

skrah
skrah suggested changes May 17, 2019
View reviewed changes

@skrah skrah left a comment

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As Victor has also requested, I'd remove all comments that mention that variables are process local.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@bedevere-bot bedevere-bot removed the awaiting merge label May 17, 2019
@bedevere-bot

bedevere-bot commented May 17, 2019

Copy link
Copy Markdown

When you're done making the requested changes, leave the comment: I have made the requested changes; please review again.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot added the awaiting changes label May 17, 2019
gpshead
gpshead requested changes May 17, 2019
View reviewed changes

@gpshead gpshead left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

overall i'm a little concerned about how fragile this is. it won't be obvious in all code why something static lives outside of the only place it is used. it feels like we need a way to prevent future C code authors from making this mistake.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Modules/_datetimemodule.c
date_fromisocalendar(PyObject *cls, PyObject *args, PyObject *kw)
{
static char *keywords[] = {
static char *kwlist[] = {

@gpshead gpshead May 17, 2019

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there a reason all of the kwlist's everywhere cannot become static const? (lemme guess.. our C API complains about the type?)

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Objects/classobject.c
return NULL;
}
return PyObject_GetAttr(im->im_func, docstr);
// if (cached_str___doc__ == NULL) {

@gpshead gpshead May 17, 2019

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

checking in a block of commented out code looks wrong... what's up here?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Python/pythonrun.c
_Py_IDENTIFIER(last_type);
_Py_IDENTIFIER(last_value);
_Py_IDENTIFIER(lineno);
_Py_IDENTIFIER(msg);

@gpshead gpshead May 17, 2019

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could the _Py_IDENTIFIER macro be modified such that it only works from the global scope so that these don't creep back into function bodies?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@ericsnowcurrently

ericsnowcurrently commented May 18, 2019

Copy link
Copy Markdown
Member Author

Thanks for all the feedback, everyone! I'm going to re-think this as you've all made good points. :) I may re-open the PR if it later makes sense to re-purpose it, but I'll probably instead tackle the statics in groups by Modules/Objects/Python.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ericsnowcurrently ericsnowcurrently closed this May 18, 2019
@ericsnowcurrently ericsnowcurrently mentioned this pull request Jun 22, 2019
Non-critical deficiencies in Tools/c-globals/c-globals.py. ericsnowcurrently/multi-core-python#49
Open
17 tasks
@ericsnowcurrently ericsnowcurrently mentioned this pull request Apr 10, 2022
[subinterpreters] Global C variables are a problem #81057
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vstinner vstinner vstinner requested changes
@gpshead gpshead gpshead requested changes
@asvetlov asvetlov asvetlov approved these changes
@1st1 1st1 Awaiting requested review from 1st1
@berkerpeksag berkerpeksag Awaiting requested review from berkerpeksag
@brettcannon brettcannon Awaiting requested review from brettcannon
@encukou encukou Awaiting requested review from encukou
@methane methane Awaiting requested review from methane
@ncoghlan ncoghlan Awaiting requested review from ncoghlan
@rhettinger rhettinger Awaiting requested review from rhettinger
@warsaw warsaw Awaiting requested review from warsaw
+2 more reviewers
@tiran tiran tiran approved these changes
@skrah skrah skrah requested changes
Reviewers whose approvals may not affect merge requirements

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

awaiting changes

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

9 participants

@ericsnowcurrently @bedevere-bot @vstinner @tiran @berkerpeksag @gpshead @asvetlov @skrah @the-knights-who-say-ni
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.