GeezerVison is showing github.com — IPythonShellEmbed fails to recognize local variables · Issue #136 · ipython/ipython
IPythonShellEmbed fails to recognize local variables #136DescriptionIssue
Embedded IPython shells can lose track of local variables.
Test Case
Minimal test case:
class Foo(object):
""" Container-like object """
def __setattr__(self, obj, val):
self.__dict__[obj] = val
def __getattr__(self, obj, val):
return self.__dict__[obj]
f = Foo()
f.indices = set([1,2,3,4,5])
f.values = {}
for x in f.indices:
f.values[x] = x
def bar(foo):
import IPython
IPython.Shell.IPShellEmbed()()
return sum(foo.values[x] for x in foo.indices)
print bar(f)
To see the error, first run the code in Python (or IPython) and exit from the spawned shell; the final print statement correctly displays '15'. Run the code again, but this time type sum(foo.values[x] for x in foo.indices) in the spawned shell, and we receive the error
" NameError: global name 'foo' is not defined".
Reacted by Victor Maryama, Guiomar Valderrama, Evan, Henry E. Miller and Maciej Olko Activitytomspur commented on Aug 1, 2010 Hmm, where did you define 'foo'?
If you just run call foo outside of the function bar, foo should not exist (and obvously it doesn't).
When you instead run "sum(f.values[x] for x in f.indices)" you get 15 again... prsteel commented on Aug 3, 2010 Quite correct. However, I'm refering to the use of foo inside the spawned IPython shell, which is in turn spawned inside the function definition where foo is a local variable. takluyver commented on Mar 23, 2011 Is this likely to be the same issue as #62? fperez commented on Nov 28, 2011 No, @takluyver: this is a separate issue and indeed a real bug in our embedding code. I was hoping your recent work with namespaces would have fixed it, but it didn't. For reference, here's the example code to run with the current embedding api:
class Foo(object):
""" Container-like object """
def __setattr__(self, obj, val):
self.__dict__[obj] = val
def __getattr__(self, obj, val):
return self.__dict__[obj]
f = Foo()
f.indices = set([1,2,3,4,5])
f.values = {}
for x in f.indices:
f.values[x] = x
def bar(foo):
import IPython
IPython.embed()
return sum(foo.values[x] for x in foo.indices)
print bar(f)Then, in the spawned, embedded IPython, this fails:
In [1]: sum(foo.values[x] for x in foo.indices)
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
/home/fperez/tmp/junk/ipython/foo.py in <module>()
----> 1 sum(foo.values[x] for x in foo.indices)
/home/fperez/tmp/junk/ipython/foo.py in <genexpr>((x,))
----> 1 sum(foo.values[x] for x in foo.indices)
NameError: global name 'foo' is not defined
And it doesn't work even if we pass to the embed call user_ns=locals() explicitly, but in that case we get in addition a crash on exit:
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
func(*targs, **kargs)
File "/home/fperez/usr/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2702, in atexit_operations
self.reset(new_session=False)
File "/home/fperez/usr/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 1100, in reset
self.displayhook.flush()
File "/home/fperez/usr/lib/python2.7/site-packages/IPython/core/displayhook.py", line 319, in flush
self.shell.user_ns['_oh'].clear()
KeyError: '_oh'
It looks like our embedding machinery is in pretty bad shape... takluyver commented on Nov 28, 2011 Assigned this to myself to look at. fperez commented on Nov 29, 2011 takluyver commented on Dec 9, 2011 Unfortunately, I think this is a limitation of Python itself. It appears code compiled dynamically cannot define a closure, which is essentially what we're doing here with a generator expression. Here is a minimal test case:
def f():
x = 1
exec "def g(): print x\ng()"
f()
Which gives:
Traceback (most recent call last):
File "scopetest.py", line 5, in <module>
f()
File "scopetest.py", line 3, in f
exec "def g(): print x\ng()"
File "<string>", line 2, in <module>
File "<string>", line 1, in g
NameError: global name 'x' is not defined
Note that you can still see local variables in IPython - in the example given, plain print foo works. But you can't close a new scope over them. takluyver commented on Jul 18, 2013 I think it may be possible to make this work using collections.ChainMap from Python 3.3 so that IPython sees both local and global variables where it is embedded as globals. However, from the lack of noise about this over the last two years, I don't think it is high priority, so I'm retagging accordingly, and hopefully getting to this some time after 1.0. clebio commented on Nov 13, 2014 Is it acceptable to up-vote, 👍 ? This affects me as well. I can add my use-case if requested. minrk commented on Nov 13, 2014 I would be 100% okay if the fix for this only worked on Python 3. amelio-dx commented on Nov 13, 2014 I am also having the same problem under Python 3. Thanks for re-openning. amelio-vazquez-reina commented on Nov 13, 2014 Same problem here 👍 both in Python 2 and 3. Erotemic commented on Aug 24, 2015 I have this issue in both Python 2 and 3. It affects me on a daily basis. ivanov commented on May 18, 2017 IPython 6.0 and later versions only work in Python 3, so if @nikitakit or anyone else wants to open a pulll request with a test case and a fix for this, that would be welcome. nikitakit commented on Jul 6, 2017 It's been a year since my last comment here, and during this time my understanding of the issue has changed a bit.
Interactively inspecting local scope remains an important feature to me, but there are actually a number of inter-related issues regarding local variables and the embedding machinery. For example, modifying local variables inside an embedded shell doesn't work:
>>> import IPython
>>> def test():
... x = 5
... IPython.embed()
... print('x is', x)
...
>>> test()
Python 3.5.1 |Continuum Analytics, Inc.| (default, Dec 7 2015, 11:24:55)
Type 'copyright', 'credits' or 'license' for more information
IPython 6.1.0 -- An enhanced Interactive Python. Type '?' for help.
In [1]: x
Out[1]: 5
In [2]: x = 6
In [3]:
Do you really want to exit ([y]/n)? y
x is 5A ChainMap-based approach wouldn't help in this situation.
Another question is what happens when closures defined inside an embedded shell are leaked out into the global scope. Consider running the same code as above, but inputting something along the lines of IPython.get_my_x = lambda: x in the embedded IPython shell. A ChainMap-based solution will avoid causing a NameError in this situation, at the expense of potentially introducing two simultaneous copies of x that exist independently of each other (one being the ChainMap, and the other the local variable/closure cell used by the python interpreter).
Given the complexity of the situation, I've decided to focus my efforts on a more comprehensive approach to the problem (which also better aligns with my own usage of IPython). This has led to the development of xdbg, which is essentially a debugger that integrates with IPython. The key idea is to extend the IPython shell by offering debugger commands via magics (e.g. %break to set breakpoints). The fact that breakpoints are set externally, rather than by calling the function embed in-place, has allowed an implementation that addresses many of these problems with local variables.
I don't currently plan on pull-requesting a narrowly-targeted bugfix to core IPython. However, I'm very interested to know what IPython users and devs think about using a debugger-inspired interface for local code inspection. IPython (and now also Jupyter) have had a huge impact on the ability to do interactive coding in Python, but there are still many improvements to be made in how it interacts with heavy encapsulation using classes/functions/modules. rmcgibbo commented on Jul 10, 2020 I've been burned a bunch of times by this bug. E.g.
[mcgibbon@xps13:~]$ cat test.py
x = 1
def func():
x = 2
import IPython
IPython.embed()
if __name__ == "__main__":
func()
[mcgibbon@xps13:~]$ python test.py
Python 3.7.6 (default, Dec 18 2019, 19:23:55)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.12.0 -- An enhanced Interactive Python. Type '?' for help.
In [1]: [x for _ in range(2)]
Out[1]: [1, 1] # lol whoops. looked up in wrong scope.
I'm not sure what can be done, but just adding my voice to the choir. rmcgibbo commented on Jul 11, 2020 Based on https://bugs.python.org/issue13557 and the fact that explicitly passing locals() and globals() to an exec fixes @takluyver's reproduction of the bug, it seems like this ought to be possible to fix in IPython by passing the correct local and global namespaces.
x = 1
def func():
x = 2
exec("def g():\n print(x)\ng()") # prints 1 :(
exec("def g():\n print(x)\ng()", locals(), globals()) # prints 2 yay!
if __name__ == "__main__":
func()
Reacted by Luis Chaves Rodriguez and Jeremiah England kflu commented on Oct 23, 2021 Reacted by Andreas Kaiser and Jeremiah England Reacted by Marcel Rieger krassowski commented on Jun 12, 2025 Closely related, now solved issue: #12199 krassowski commented on Jun 12, 2025 The execution machinery changed a lot since this issue was discussed. Currently the execution happens in:
|
await eval(code_obj, self.user_global_ns, self.user_ns) |
|
else: |
|
exec(code_obj, self.user_global_ns, self.user_ns) |
The user_ns is controlled during embed, so it can be passed locals(), which makes the reproducer from the first issue pass:
# (initialization as in the first comment)
def bar(foo):
import IPython
IPython.embed(user_ns=locals())
bar(f) # once shell shows, type `sum(foo.values[x] for x in foo.indices)` and press enter
added 2 commits that reference this issue on Jul 7, 2026This issue will close once commit 0adf9d6 is merged into the 'main' branch.0adf9d6 This issue will close once commit d8ab325 is merged into the 'main' branch.d8ab325 added a commit that references this issue on Jul 7, 2026This issue will close once commit 07933ea is merged into the 'main' branch.07933ea added a commit that references this issue on Aug 3, 2026This issue will close once commit 083261a is merged into the 'main' branch.083261a
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|
