GeezerVison

System.AccessViolationException occurs when iteration quits in the middle of list before reaching the end.

Environment

• Pythonnet version: 3.0.3
• Python version: 3.12.0
• Operating System: Windows 11
• .NET Runtime: .NET Framework 4.8

Reproduce the problem

I'm trying to pass a list from python to .NET method via List Codec.

mylist = ["a","b","c"]
dotnet.method( mylist )

Conversion(decode) is successful and mylist is wrapped by ListWrapper.
.NET code receives the list like this.

void method( IList<string> mylist )
{
    mylist.First( s => s == "b" ); // <- here the crash with System.AccessViolationException
}

Using Enumerator of mylist, if enumerated to the end of the list, nothing bad happens.
However if iteration didn't go to the end of the list, AccessViolationException is thrown.
This case happens in the following code:

mylist.First( s => s == "b" ); // iteration stops at the 2nd item

similarly

foreach( var item in mylist )
{
    if( item == "b" ) break; // iteration stops at the 2nd item
}
mylist.Contains("b"); // iteration stops at the 2nd item
mylist.Any( s => s == "b"); // iteration stops at the 2nd item

The following code doesn't throw exception becuase it always iterates to the end of the list.

mylist.ToList();
mylist.ToArray();

Exception occurs because PyIter(ator) object is Dispose(d) outside of GIL protection only when iteration ends before reaching the end ( before MoveNext() returns null ).
In such a case, code process just slips out of GIL enclosure as the nature of C# scope and yield return and break.

solution for now

original code is like this:

        public IEnumerator<T> GetEnumerator()
        {
            PyIter iterObject;
            using (Py.GIL())
            {
                iterObject = PyIter.GetIter(pyObject);
            }

            using var _ = iterObject;
            while (true)
            {
                using var GIL = Py.GIL();

                if (!iterObject.MoveNext())
                {
                    iterObject.Dispose();
                    break;
                }
                yield return iterObject.Current.As<T>()!;
            }
        }

using var _ = iterObject; above the while loop is the problematic line.
Before iteration reaches the end, the executing process is somewhere out of the using scope.
When c# caller process stops iteration(Enumeration) for some reason, c# suddely calls
using var _ = iterObject; 's closing process which is iterObject.Dispose().
But here iterObject is not inside of GIL() protection. This is why exception occurs.
Solution is to make sure iterObject.Dispose() is always inside of GIL protection and
always executed whatever the caller process abandons the enumeration in the middle.

My suggestion is to use try catch pattern. Without catching any exception it may look odd. But this is one solution to make sure the execution of Dispose and GIL scope surrounding always.

        public IEnumerator<T> GetEnumerator()
        {
            PyIter iterObject;
            using (Py.GIL())
            {
                iterObject = PyIter.GetIter(pyObject);
            }
            try
            {
                while (true)
                {
                    using var _ = Py.GIL();
                    if (!iterObject.MoveNext())
                    {
                        break;
                    }
                    yield return iterObject.Current.As<T>()!;
                }
            }
            finally
            {
                using var _ = Py.GIL();
                iterObject.Dispose();
            }
        }