GeezerVison is showing github.com — PythonEngine.Shutdown() / Runtime.Shutdown() slowness · Issue #2008 · pythonnet/pythonnet
PythonEngine.Shutdown() / Runtime.Shutdown() slowness #2008DescriptionEnvironment
- Pythonnet version: 3.0.1
- Python version: 3.9.12
- Operating System: Windows or Linux
- .NET Runtime: net6.0
Details
- Describe what you were trying to get done.
We are embedding Python in a .NET app using pythonnet.
The call to PythonEngine.Shutdown() at the exit of this app lasts for more than 2 minutes.
Looking at the shutdown with a profiler shows that most of the time is spent in Runtime.TryCollectingGarbage().
Looks like the more memory is allocated on the C# side the more the shutdown (i.e. calls to GC.Collect()) is slow.
- Minimal, Complete, and Verifiable example
using Python.Runtime;
namespace Test;
static class Program
{
static void Main()
{
var mem = new List<Array>();
Runtime.PythonDLL = @$"{Environment.GetFolderPath(Environment.SpecialFolder.UserProfile)}\AppData\Local\Programs\Python\Python39\python39.dll";
PythonEngine.Initialize();
var threadState = PythonEngine.BeginAllowThreads();
using (Py.GIL())
{
using (var scope = Py.CreateScope())
{
scope.Set("a", 1);
scope.Set("b", 2);
scope.Exec("result = a + b");
var result = scope.Get("result").As<int>();
Console.WriteLine($"Sum = {result}");
}
}
// allocate dotnet-managed memory unrelated to pythonnet
var rnd = new Random();
for (var i = 0; i < 100_000_000; ++i)
{
mem.Add(new byte[rnd.Next(1, 10)]);
}
PythonEngine.EndAllowThreads(threadState);
Console.WriteLine("Shutdown start");
var start = DateTime.Now;
PythonEngine.Shutdown();
var diff = DateTime.Now - start;
Console.WriteLine($"Shutdown stop {diff.TotalSeconds}");
}
}Typical output:
Sum = 3
Shutdown start
Shutdown stop 61.6796983 Reacted by Anders Edqvist, Bojan Vukasovic, fischja, e16384 and Maciej Sobkowiak Activityfilmor commented on Nov 22, 2022 This would probably be helped by handling #1781.
If we are finally shutting down, we don't actually have to stash anything, so we also shouldn't try to clean up garbage, right @lostmsu @BadSingleton ? fischja commented on Feb 16, 2023 BadSingleton commented on Feb 17, 2023 Apologies for the late reply. Yes, if we're shutting down for the last time, then there is no need to stash anything. If I recall correctly, it used to be that way before when there was still multiple modes. But I'm not sure about garbage collection, I've had some issues in the past (crashing on shutdown?) if it wasn't run correctly. filmor commented on Feb 20, 2023 @lostmsu @BadSingleton Now that issues like #2107 show up because BinaryFormatter.Serialize has been hard-deprecated in .NET 7, we need to do somethhing. How about making Shutdown() behave like it used to (no stashing) and adding a new ShutdownAllowRestart or something like that? lostmsu commented on Feb 20, 2023 BadSingleton commented on Feb 20, 2023
[...] do we even need to support restart after shutdown on .NET 7?
I've been out of the .NET loop for a few months, why wouldn't we need to support it?
Otherwise, I don't think it has to be BinaryFormatter, as long as the behaviour is the same(?) benoithudson commented on Feb 20, 2023 Unity is moving to .NET 7 next year supposedly (in large part to support Span). So, it would be best if it were still supported.
https://blog.unity.com/technology/unity-and-net-whats-next
For the original comment: do we actually need to call Shutdown on shutdown? If not, that would save a lot of time. lostmsu commented on Feb 20, 2023 @benoithudson what is Unity on now? I thought it relies on AppDomain, and AppDomain is not supported in .NET Core/6+. Without AppDomain, I do not think one needs to call Shutdown at all. What should be done instead perhaps is ability to reload individual types. nlogozzo commented on Feb 23, 2023
Now that issues like #2107 show up because BinaryFormatter.Serialize has been hard-deprecated in .NET 7, we need to do somethhing.
Hi, anyway I can work around this issue for now? AlexGilmor commented on Feb 23, 2023 I faced the same problem (Pythonnet 3.0.1, .NET Framework 4.7.2). In my app I have a window with the python code editor. I call PythonEngine.Initialize() when the user opens it, and the PythonEngine.Shotdown() when closing the window.
After executing simple scripts, the window closes with a small but noticeable delay. When executing scripts that create more complex objects, for example, matplayotlib figures, the delay increases to tens or even hundreds of seconds. I can call PythonEngine.Shotdown() not when closing the editor window, but when closing the application. But for now this only shifts the problem and does not solve it.
Unfortunately, I wasn’t able to find the official position on this subject in the documentation. The fact that many people use Pythonnet suggest that there is a way of bypass, because without it the issue looks serious. @dclaudeone11 Did you manage to find a temporary solution?
Without AppDomain, I do not think one needs to call Shutdown at all
@lostmsu First of all, thank you and other contributors for such a useful library.
Could you please explain what exactly is considered as a right way to stop Pythonnet (or app that uses it) for today? If I do not call PythonEngine.Shotdown() at all, after closing the main window of the program, a working process remains, which is obviously unacceptable. Given the popularity of this thread in the last days, it seems that many more people would be grateful for your official commentary, not only me. filmor commented on Feb 23, 2023 @AlexGilmor If you want to shut down Python, use Shutdown. The slowness is due to to multiple garbage collection rounds and (potentially contributing, but not the only factor) the runtime stashing that was introduced to allow for AppDomain switching.
@nlogozzo Since you are not actually using the stashing, you can try setting a no-op formatter:
public class NoopFormatter : IFormatter {
public object Deserialize(Stream s) => throw new NotImplementedException();
public void Serialize(Stream s, object o) {}
public SerializationBinder Binder { get; set; }
public StreamingContext Context { get; set; }
public ISurrogateSelector SurrogateSelector { get; set; }
}
Python.Runtime.RuntimeData.FormatterType = typeof(NoopFormatter);nlogozzo commented on Feb 23, 2023
@nlogozzo Since you are not actually using the stashing, you can try setting a no-op formatter:
That worked...thanks! kikaragyozov commented on Mar 30, 2023
Environment
- Pythonnet version: 3.0.1
- Python version: 3.9.12
- Operating System: Windows or Linux
- .NET Runtime: net6.0
Details
- Describe what you were trying to get done.
We are embedding Python in a .NET app using pythonnet.
The call to PythonEngine.Shutdown() at the exit of this app lasts for more than 2 minutes.
Looking at the shutdown with a profiler shows that most of the time is spent in Runtime.TryCollectingGarbage().
Looks like the more memory is allocated on the C# side the more the shutdown (i.e. calls to GC.Collect()) is slow.
- Minimal, Complete, and Verifiable example
using Python.Runtime;
namespace Test;
static class Program
{
static void Main()
{
var mem = new List<Array>();
Runtime.PythonDLL = @$"{Environment.GetFolderPath(Environment.SpecialFolder.UserProfile)}\AppData\Local\Programs\Python\Python39\python39.dll";
PythonEngine.Initialize();
var threadState = PythonEngine.BeginAllowThreads();
using (Py.GIL())
{
using (var scope = Py.CreateScope())
{
scope.Set("a", 1);
scope.Set("b", 2);
scope.Exec("result = a + b");
var result = scope.Get("result").As<int>();
Console.WriteLine($"Sum = {result}");
}
}
// allocate dotnet-managed memory unrelated to pythonnet
var rnd = new Random();
for (var i = 0; i < 100_000_000; ++i)
{
mem.Add(new byte[rnd.Next(1, 10)]);
}
PythonEngine.EndAllowThreads(threadState);
Console.WriteLine("Shutdown start");
var start = DateTime.Now;
PythonEngine.Shutdown();
var diff = DateTime.Now - start;
Console.WriteLine($"Shutdown stop {diff.TotalSeconds}");
}
}Typical output:
Sum = 3
Shutdown start
Shutdown stop 61.6796983
Your issue does not occur when the project is ran in .NET 7 using a Release configuration. When ran in Debug configuration, the shutdown does indeed slow down significantly - from 0.08 seconds up to 63 seconds.
public class NoopFormatter : IFormatter {
public object Deserialize(Stream s) => throw new NotImplementedException();
public void Serialize(Stream s, object o) {}
public SerializationBinder Binder { get; set; }
public StreamingContext Context { get; set; }
public ISurrogateSelector SurrogateSelector { get; set; }
}
Python.Runtime.RuntimeData.FormatterType = typeof(NoopFormatter);
This does not fix the issue when the code is ran in Debug configuration under .NET 7. gitlsl commented on Aug 30, 2023 v3.0.2 same question, NoopFormatter has no effact
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|