GeezerVison is showing github.com — Global variables undefined in interactive use of embedded ipython shell · Issue #62 · ipython/ipython
Global variables undefined in interactive use of embedded ipython shell #62DescriptionOriginal Launchpad bug 399627: https://bugs.launchpad.net/ipython/+bug/399627
Reported by: h-fangohr (Hans Fangohr).
The error can be reproduced as follows:
- Start Python, and start embedded ipython session. Following ipython's manual, we do
fangohr@eta:~$ python
Python 2.4.3 (#1, Jun 8 2009, 14:09:06)
[GCC 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from IPython.Shell import IPShellEmbed
>>> ipshell=IPShellEmbed()
>>> ipshell()
Within the just started ipython session, global variables are sometimes not visible. Two examples are:
Example 1:
In [1]: a=1
In [2]: def f(x):
...: return a*x
...:
In [3]: f(2)
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
/home/fangohr/<ipython console>
/home/fangohr/<ipython console> in f(x)
NameError: global name 'a' is not defined
Example 2:
In [4]: b=1
In [5]: (lambda :b)()
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
/home/fangohr/<ipython console>
/home/fangohr/<ipython console> in <lambda>()
NameError: global name 'b' is not defined
There is no error if "b=1; (lambda :b)()" is put in a script, and this script is executed using ipython's "run" command.
There is no error if "b=1; (lambda :b)()" is executed interactively after having started ipython.
The only way the error seems comes up is if an embedded IPython shell is started from a Python prompt AND the commands are executed interactively at the prompt.
I find the same error when trying ipython-0.9.1 (with python2.4).
The bug was reported by Olivier Klein to the nmag team (http://nmag.soton.ac.uk). Reacted by Joelino, kflu, Guiomar Valderrama, Risa, Victor Schmidt, zzl, Henry E. Miller and Fabian Keller Activityfperez commented on Dec 4, 2010 Over at launchpad...
chairmanK wrote 20 hours ago:
Me Too. In addition to functions, this bug also appears in generator expressions. takluyver commented on Mar 23, 2011 The equivalent component in trunk seems to be IPython.frontend.terminal.InteractiveShellEmbed. But that's broken in other ways, and evidently hasn't been tested much. Does anyone know what its future is? takluyver commented on Mar 23, 2011 Could this be the same issue as #136? fperez commented on Nov 28, 2011 This has been now fixed:
amirbar[ipython]> python
Python 2.7.2+ (default, Oct 4 2011, 20:06:09)
[GCC 4.6.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from IPython import embed
>>> embed()
Python 2.7.2+ (default, Oct 4 2011, 20:06:09)
Type "copyright", "credits" or "license" for more information.
IPython 0.12.dev -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.
In [1]: a = 1
In [2]: def f(x):
...: return a*x
...:
In [3]: f(3)
Out[3]: 3
In [4]: b = 1
In [5]: (lambda : b)()
Out[5]: 1
In [6]:
daf commented on Feb 7, 2012 Can someone explain a little more what went into the fix of this? I still encounter the issue, but only when I'm one layer removed via a method call, and only if I run from a script, not interactively.
python 2.7.2 on OSX 10.6.8, ipython 0.11 and 0.12 both exhibit similar behavior (using 0.12 for this comment)
This is a problem with our project which features (prominently) an embedded IPython shell.
testembed.py
from IPython import embed
def hi():
embed()
if __name__ == '__main__':
#embed()
hi()Run this on the command line with python testembed.py and see this session:
Python 2.7.2 (default, Aug 29 2011, 12:33:18)
Type "copyright", "credits" or "license" for more information.
IPython 0.12 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.
In [1]: import time
In [2]: def tim():
...: print time.time()
...:
In [3]: tim()
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
[snip] in <module>()
----> 1 tim()
[snip] in tim()
1 def tim():
----> 2 print time.time()
3
NameError: global name 'time' is not defined
In [4]:
However, comment out the call to hi() and replace it with the embed() call:
Python 2.7.2 (default, Aug 29 2011, 12:33:18)
Type "copyright", "credits" or "license" for more information.
IPython 0.12 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.
In [1]: import time
In [2]: def tim():
print time.time()
...:
In [3]: tim()
1328639444.29
In [4]:
After poking around, I think it has to do with the stack_depth parameter used here and this block: https://github.com/ipython/ipython/blob/master/IPython/frontend/terminal/embed.py#L185
Thoughts? takluyver commented on Feb 7, 2012 This is somewhat complicated, but I believe that's a restriction in Python itself.
In the failing case you show, time is not actually put into the global namespace: because you called embed inside the hi function, new variables you create interactively are local to that function. Ideally, tim() should work as a closure, closing over the reference to the time module. However, closures only appear to work when the containing function is compiled in one go. As far as I can tell, there is no way to define a closure dynamically. This simple example fails:
def outer():
import time
exec("def inner(): return time.time()")
return inner
outer()()
This is probably because nested scopes were added to Python relatively late (they were a future import in 2.1, and always on in 2.2). daf commented on Feb 7, 2012 Ok, I think I understand. Looks like we can't really do anything about this then, other than dumping what I would write interactively to a file and then reading that file back in. Probably too complex for what we want to be able to do interactively.
Thanks, btw. daf commented on Feb 7, 2012 Sorry to keep harping on here, but it just makes interactive sessions feel very clunky:
Regular python:
>>> d={'one':1, 'two':2}
>>> getkeys=lambda: d.keys()
>>> getkeys()
['two', 'one']
Regular IPython:
In [1]: d={'one':1, 'two':2}
In [2]: getkeys=lambda: d.keys()
In [3]: getkeys()
Out[3]: ['two', 'one']
Embedded IPython:
>>> from IPython import embed
>>> embed()
Python 2.7.2 (default, Aug 29 2011, 12:33:18)
Type "copyright", "credits" or "license" for more information.
IPython 0.12.dev -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.
In [1]: d={'one':1, 'two':2}
In [2]: getkeys=lambda: d.keys()
In [3]: getkeys()
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
/Users/asadeveloper/Documents/Dev/code/pyon/bin/python in <module>()
----> 1 getkeys()
/Users/asadeveloper/Documents/Dev/code/pyon/bin/python in <lambda>()
----> 1 getkeys=lambda: d.keys()
NameError: global name 'd' is not defined
I guess there isn't much that can be done, but I don't get why we can create a dynamic closure in regular python/ipython but not the embedded version. takluyver commented on Feb 7, 2012 By the oddities of Python scoping, in standard Python/IPython, we're not actually creating a closure. d is a global variable, and the rules for accessing those work differently from closures. Each function keeps a reference to the global namespace where it was defined (getkeys.func_globals), so it can access any variables defined there.
In contrast, when you make a closure, Python attaches references to each variable over which it has closed, as determined by the compiler - but that only works when the inner and outer functions are compiled at the same time. It looks like this:
In [8]: def outer():
a = 1
def inner():
return a
return inner
...:
In [9]: f = outer()
In [10]: f
Out[10]: <function __main__.inner>
In [11]: f.func_closure
Out[11]: (<cell at 0x9f5e344: int object at 0x9a830b0>,)
This is possibly done to save memory - if the closure carried a reference to the local scope where it was defined, none of the variables from that scope could be freed while the closure was live. liokm commented on Apr 24, 2012 I'm using Python 2.7.2 and IPython 0.12.1 on OSX 10.7.3 and having this issue still. When I run ./manage.py shell of Django which invokes IPython.embed(), the issue occurs. However, manually invoking the embed() in Python shell or from simple script file, there is no problem. takluyver commented on Apr 24, 2012 There's not a lot we can directly do about that, but we should probably encourage third parties away from embed for non-trivial uses. liokm commented on Apr 24, 2012 @takluyver Do you mean that better to use ipython directly in this case? takluyver commented on Apr 24, 2012 It's possible for Django to start IPython in a way that won't cause this problem, but that's not the way we currently make easy. The problem occurs when IPython starts with separate local and global namespaces. There's no reason that Django requires separate local and global namespaces, but that's what calling embed() inside a function implies.
For reference, here's the code in Django:
https://code.djangoproject.com/browser/django/trunk/django/core/management/commands/shell.py limwz01 commented on Jun 14, 2024 I have a workaround posted at https://gist.github.com/limwz01/7e06cacf31d4ed83aa74134971030967 which is not as limited as the current approaches, but it has several caveats.
Allowing closures over local variables is done by using a transformer added
to shell.ast_transformers that creates an outer function scope wrapping the
cell code with parameters from local variables and which finally updates the
locals() and returns the last expression. For illustration, we have the
following generated when the code has statement0; statement1; last_expression and the local variables are x, y, z:
def _ipy_magic(_ipy_magic, x, y, z):
statement0
statement1
return (last_expression, _ipy_magic.update(locals()))[0]
_ipy_magic(locals(), x, y, z)
- exceptions raised in the shell further trigger a weird exception
executing.executing.NotOneValueFound during traceback exception formatting
and this disables pretty tracebacks
- direct changes to local variables are only visible to code within the
shell session
- outer scope variables must already have had a closure referencing it to be
visible (nonlocal does not work) and they become local variables (and
hence above restriction applies)
- local variables named
_ipy_magic will be overwritten
return statements without an enclosing function does not give an error and
results in unpredictable behaviour
What this means is that you can just paste in code as if you are really inside the function you had embedded in, and it should work most of the time. This is useful to test small snippets of code as it is or in the event a program throws an exception, check if a modified piece of code works without re-running the whole program.
Borrowing some of the above examples, these work:
In [1]: import time
In [2]: def tim():
...: print(time.time())
...:
In [3]: tim()
1718358164.5522225
In [4]: a = 1
In [5]: (lambda: a)()
Out[5]: 1
Erotemic commented on Oct 13, 2024 limwz01 commented on Oct 13, 2024 Carreau commented on Oct 14, 2024 Yeah, I'm in the process of watching @3b1b video, and I know this is an ipython limitation. I'd love a PR, I unfortunately dont have the time to dive into it.
My guess is that it's the same issue that affects ipdb where list comprehension don't work.
@limwz01, if you think your utils can be ported to be default in IPython, please send a PR.
Worse case we'll find some issues and that can be reverted, but at least we'll move forward on our comprehension of what it happening. limwz01 commented on Oct 27, 2024 @Carreau I've packaged it as a separate pip installable package ipython_utils as I really need more people to test it in real world usage first. It does a lot of complicated stuff behind the scenes and right now it messes up the stack trace a little and I've had some minor problems with restoring the excepthook. You're welcome to try it out and give me some feedback limwz01 commented on Nov 6, 2024 @gaogaotiantian recently pushed a similar solution to pdb in cpython itself: python/cpython#111094
His solution is a little too simplistic and did not handle all the edge cases, so it may not be suitable here. However, it shows that such a solution is actually acceptable. gaogaotiantian commented on Nov 6, 2024
His solution is a little too simplistic and did not handle all the edge cases, so it may not be suitable here.
Feel free to submit an issue about the edge cases that pdb can't handle. We might not be able to fix all of those because we need to make the pdb code as maintainable as possible (so playing with ast might not be an option), which means we need to balance the code complexity and the capability, but it would be good for us to at least know what are the cases that we can't handle. limwz01 commented on Nov 18, 2024 @gaogaotiantian I use the following to illustrate the problems:
#!/usr/bin/env python3.13
g0 = 0
def main():
import pdb
pdb.Pdb().set_trace()
if __name__ == "__main__":
main()The following are the problems:
- not re-using the same Cells for each local (this is really the biggest issue)
(Pdb) t = 1
(Pdb) f = lambda: t
(Pdb) t = 2
(Pdb) f()
1
- not obtaining the last expression (a pdb test suggests otherwise, but it is actually wrong)
(Pdb) t=10; (lambda: t)()
<no output>
(it does return a value for global g; g=1; g. but I'm at a loss as to where that value comes from)
- giving a weird result with shadowed variables (actually because it bails out):
(Pdb) g0 = 1
(Pdb) global g0; t=g0; t=(lambda: t)()
*** NameError: name 't' is not defined
(Pdb) t:int = 1; (lambda: t)()
*** NameError: name 't' is not defined
* using indentation to set up block scope
(Pdb) print((lambda:len("""
... """))())
7
* not hoisting globals (this is arguable):
(Pdb) global g; g=1
(Pdb) t=2
(Pdb) g=(lambda: t)()
(Pdb) global g; g
1
limwz01 commented on Nov 19, 2024 I can't seem to edit my post to correct some typos and cross-reference the issue, but anyway the issue has been submitted here: python/cpython#126958 ur001 commented on Dec 20, 2024 Use user_ns. It may be some dict, or locals() | globals(), for example.
from IPython import start_ipython
start_ipython(user_ns={})or
from IPython.terminal.embed import InteractiveShellEmbed
InteractiveShellEmbed(user_ns={})()or
import IPython
IPython.embed(user_ns={})Result:
In [1]: a = 1
In [2]: def f(x):
...: return a * x
...:
In [3]: f(2)
Out[3]: 2 limwz01 commented on Dec 21, 2024 This issue is about the problems of embedding a shell in a function scope,
something you totally missed in your examples, @ur001
Edit: okay that wasn't exactly the initial poster's problem, but in a few posts down it becomes obvious that embedding in a function is the problem that is remaining after the "This has been now fixed" post. krassowski commented on Jul 4, 2025 @limwz01 it appears that you problem was a different issue, related to pdb, fixed in Python 3.13 and backported to IPython in #14933.
Passing locals() scope explicitly is now required in Python 3.13 as per:
It looks like we could just ensure that embed defaults to pass a locals() in. Is there any example where
import IPython
IPython.embed(user_ns={})or
import IPython
IPython.embed(user_ns=locals()) would not work? limwz01 commented on Jul 5, 2025 Yes, definitely. What you are doing is running the shell with its globals set to the locals() at that point in question. That is bad because even global variables will now be "out of scope" in the new shell. Here's a piece of code that has all the types of variables:
import IPython
x0 = 100
def main():
y0 = 200
y1 = 300
def f(z0, z1):
nonlocal y1
def g():
nonlocal z1
z1 += 1
y1 += 1
print(x0, y0, y1, z0, z1)
IPython.embed(user_ns=locals())
print(x0, y0, y1, z0, z1)
IPython.embed(user_ns=locals())
f(400, 500)
main()Now, let's see what you can do in the first shell:
x0 cannot be accessed- no changes to the local variables are retained, i.e. changing any of the variables in the first shell wouldn't have any effect on the second
print, or their values in the second shell.
g(); print(z1) changes only the real z1 and the shell's z1 is unchanged
Quoting from my readme at https://github.com/limwz01/ipython_utils?tab=readme-ov-file#limited-functionality-in-default-ipython-embedded-shell :
There are a few workarounds for this, when embedding into the scope of a function f:
- Use a mocked-up
globals() including what was in locals()
- Use the real
globals() and copy over locals() into the real globals()
Both of the above require us to patch all the nonlocal statements referring to locals in the scope of f (and parent functions) to become global. In the first workaround, we use IPython with a globals() namespace that have been updated with both the original frame's globals() and locals(), either by running IPython.start_ipython with user_ns as the mocked-up globals() or patching IPython.terminal.embed.InteractiveShellEmbed (instead of this, one could possibly keep calling globals().update(locals()) for every statement executed but that is really inefficient and only works if the created child functions do not modify the referenced variables). In this workaround, references to real globals in created functions will no longer access/modify the real global variable, but a copy, hence diverging from functions created by other means which use the real globals. In the second workaround, the real global namespace gets polluted and globals might have their values incorrectly overwritten by the locals, causing some existing functions to behave incorrectly.
Instead, one could use the (almost) complete solution that I developed (after pip install ipython_utils), and everything just works fine, you can even create a closure that works in the second shell:
import ipython_utils
x0 = 100
def main():
y0 = 200
y1 = 300
def f(z0, z1):
nonlocal y1
def g():
nonlocal z1
z1 += 1
y1 += 1
print(x0, y0, y1, z0, z1)
ipython_utils.embed([f, lambda: z1])
print(x0, y0, y1, z0, z1)
ipython_utils.embed([f, lambda: z1])
f(400, 500)
main()In the first shell:
def h():
nonlocal z1
z1 += 2In the second shell:
print(z1)
g()
print(z1)
h()
print(z1) And this prints:
If anyone can break my code, feel free to post an issue! EDIT: I've updated my code due to a backward-incompatible change in Python3.13.
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|
