GeezerVison is showing github.com — Best effort serialization by BadSingleton · Pull Request #1892 · pythonnet/pythonnet
Conversation
What does this implement/fix? Explain your changes.
This PR adds a "last line of defense, best effort serialization" to serialize types not marked as Serializable. Such objects, when derivable, are deserialized as derived classes with all (overridable) methods and properties overriden to throw a "Not Serialized" Exception. Fields are not initialized and may be null. Instances of classes that are not derivable are null.
Does this close any currently open issues?
No, but it is an issue we've encountered with Unity.
Any other comments?
This is a "best effort" to ensure no SerializationExceptions are thrown on domain unloads/reloads. This should help users deal with serialization issues as sometimes unserializable classes may be in compiled libraries and therefore unmodifiable.
Checklist
Check all those that are applicable and complete.
- Make sure to include one or more tests for your change
- If an enhancement PR, please create docs and at best an example
- Ensure you have signed the .NET Foundation CLA
- Add yourself to
AUTHORS
- Updated the
CHANGELOG
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
This commit adds a "last line of defense, best effort serialization"
to serialize types not marked as Serializable. Such objects are
deserialized as derived classes with all methods and properties
overriden to throw a "Not Serialized" Exception.
Fields are not initialized and may be null.
Sealed classes and implemented interface methods are still a problem to
be solved.
plus guard rails for private classes
and review fixes
don't try to derive from sealed types
|
Also just a note that I'll soon be away for vacations until october, I just want to get that ball rolling and gather feedback in the meantime.
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
lostmsu
commented
Jul 29, 2022
•
edited
Loading
Uh oh!
There was an error while loading. Please reload this page.
|
@BadSingleton is there a way to provide that functionality without bringing the NonSerializedTypeBuilder into Python.NET? E.g. can we allow user to override the serializer somehow, and you guys would be able to keep this code inside Unity or as a new subproject like pythonnet/unity-serializer?
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
Also, can you describe the approach in more details?
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
@lostmsu I think the serializer was meant to be overridable in the first implementations of the serialization. I think I recall @amos402 saying something about the need for users to implement their own. If it becomes overridable we probably can keep it outside of this repo (just need to make sure with some order of initialization things)
The idea was to ensure that the presence of a non-serializable object in the graph wouldn't throw an exception, therefore corrupting the whole shutdown process and causing crashes on domain (un)load. As sometimes those objects are outside of our control, we can't modify them and/or avoid easily to serialize them (and/or the users of our APIs may be passing a Unity object that is not serializable). After a few google searches I found out that serialization surrogates could be used to serialize objects not marked [Serializable]. Since these objects are not meant to be serialized, but they're in the object graph that is being serialized, we serialize the type, not the object. When deserializing, there are two possibilities:
- The object can be derived from, and we add
_NotSerialized suffix to the class name. If it is, every method and property (get and set) are overridden by a simple function that throws a NotSerializedException (the inspiration for the exception message comes from PyQt. Those who knows, know the pain.). Fields are left null.
- The object cannot be derived from (private, internal, ...). Not much to do here, except returning a null reference.
When serializing a derived _NotSerialized class, we serialize the base class to avoid having _NotSerialized_NotSerialized_... looping.
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
Ignore the part of the previous comment about the serializer not being overridable, I misremembered the code, it is overridable.
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
lostmsu
commented
Jul 29, 2022
•
edited
Loading
Uh oh!
There was an error while loading. Please reload this page.
|
It seems that this code can be abstracted out of this repo with:
- Having a public post stash and pre restore hooks
- Exposing a wrapper for PyCapsule and maybe PyMem (though I think it can be replaced with
Marshal.AllocHGlobal)
- Replacing
FormatterType property with FormatterFactory
- Exposing
MaybeType
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
Worst case, MaybeType is easily re-implementable. I'm not sure about exposing it.
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
I am also experiencing this issue when trying to use this library with Unity3D. Here is the specific serialization error I get on shutdown. @BadSingleton, does your fix prevent this and allow you to initialize and shutdown the python engine multiple times without error?
Got exception during shutdown:
Message:Type '__System_Action`1\[\[System_String\, mscorlib\, Version=4_0_0_0\, Culture=neutral\, PublicKeyToken=b77a5c561934e089\]\]Dispatcher' in Assembly '__Python_Runtime_Generated_Assembly0, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null' is not marked as serializable.
at System.Runtime.Serialization.FormatterServices.InternalGetSerializableMembers (System.RuntimeType type) [0x00045] in <4c36328e5e3c40ecaf35733ee9461b26>:0
at System.Runtime.Serialization.FormatterServices+<>c__DisplayClass9_0.<GetSerializableMembers>b__0 (System.Runtime.Serialization.MemberHolder _) [0x00000] in <4c36328e5e3c40ecaf35733ee9461b26>:0
at System.Collections.Concurrent.ConcurrentDictionary`2[TKey,TValue].GetOrAdd (TKey key, System.Func`2[T,TResult] valueFactory) [0x00034] in <4c36328e5e3c40ecaf35733ee9461b26>:0
at System.Runtime.Serialization.FormatterServices.GetSerializableMembers (System.Type type, System.Runtime.Serialization.StreamingContext context) [0x0005e] in <4c36328e5e3c40ecaf35733ee9461b26>:0
at System.Runtime.Serialization.Formatters.Binary.WriteObjectInfo.InitMemberInfo () [0x0003d] in <4c36328e5e3c40ecaf35733ee9461b26>:0
at System.Runtime.Serialization.Formatters.Binary.WriteObjectInfo.InitSerialize (System.Object obj, System.Runtime.Serialization.ISurrogateSelector surrogateSelector, System.Runtime.Serialization.StreamingContext context, System.Runtime.Serialization.Formatters.Binary.SerObjectInfoInit serObjectInfoInit, System.Runtime.Serialization.IFormatterConverter converter, System.Runtime.Serialization.Formatters.Binary.ObjectWriter objectWriter, System.Runtime.Serialization.SerializationBinder binder) [0x00158] in <4c36328e5e3c40ecaf35733ee9461b26>:0
at System.Runtime.Serialization.Formatters.Binary.WriteObjectInfo.Serialize (System.Object obj, System.Runtime.Serialization.ISurrogateSelector surrogateSelector, System.Runtime.Serialization.StreamingContext context, System.Runtime.Serialization.Formatters.Binary.SerObjectInfoInit serObjectInfoInit, System.Runtime.Serialization.IFormatterConverter converter, System.Runtime.Serialization.Formatters.Binary.ObjectWriter objectWriter, System.Runtime.Serialization.SerializationBinder binder) [0x00006] in <4c36328e5e3c40ecaf35733ee9461b26>:0
at System.Runtime.Serialization.Formatters.Binary.ObjectWriter.Write (System.Runtime.Serialization.Formatters.Binary.WriteObjectInfo objectInfo, System.Runtime.Serialization.Formatters.Binary.NameInfo memberNameInfo, System.Runtime.Serialization.Formatters.Binary.NameInfo typeNameInfo) [0x00101] in <4c36328e5e3c40ecaf35733ee9461b26>:0
at System.Runtime.Serialization.Formatters.Binary.ObjectWriter.Serialize (System.Object graph, System.Runtime.Remoting.Messaging.Header[] inHeaders, System.Runtime.Serialization.Formatters.Binary.__BinaryWriter serWriter, System.Boolean fCheck) [0x001d3] in <4c36328e5e3c40ecaf35733ee9461b26>:0
at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Serialize (System.IO.Stream serializationStream, System.Object graph, System.Runtime.Remoting.Messaging.Header[] headers, System.Boolean fCheck) [0x0006e] in <4c36328e5e3c40ecaf35733ee9461b26>:0
at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Serialize (System.IO.Stream serializationStream, System.Object graph, System.Runtime.Remoting.Messaging.Header[] headers) [0x00000] in <4c36328e5e3c40ecaf35733ee9461b26>:0
at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Serialize (System.IO.Stream serializationStream, System.Object graph) [0x00000] in <4c36328e5e3c40ecaf35733ee9461b26>:0
at Python.Runtime.RuntimeData.Stash () [0x00049] in <17d3e183cc45426f82a1a9207d7443a1>:0
at Python.Runtime.Runtime.Shutdown () [0x00030] in <17d3e183cc45426f82a1a9207d7443a1>:0
at Python.Runtime.PythonEngine.Shutdown () [0x00064] in <17d3e183cc45426f82a1a9207d7443a1>:0
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
Hello, @larryPlayabl. It has been a while, but if my memory serves right, this looks like the exact error this fix was made for. For added convenience, Unity's Python Scripting package has already that fix since version 6.0.0
|
larryPlayabl reacted with thumbs up emoji
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
Closing, superseded by #2336
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Assignees
No one assigned
Milestone
No milestone
Development
Successfully merging this pull request may close these issues.
Uh oh!
There was an error while loading. Please reload this page.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
|