GeezerVison

Member

More actions

I got another crash from sympy but not sure it's CPython issue or Sympy issue at this moment.
(Buf AFAIK, sympy is pure-python code..)


sympy/core/tests/test_relational.py[64] ...............f.....................f..
...Modules/gcmodule.c:115: gc_decref: Assertion "gc_get_refs(g) > 0" failed: refcount is too small
Enable tracemalloc to get the memory block allocation traceback

object address  : 0x1064b25e0
object refcount : 6
object type     : 0x145c63630
object type name: floor
object repr     : floor(x)

Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
Python runtime state: initialized

Current thread 0x000000010197c580 (most recent call first):
  Garbage-collecting
  File "/Users/user/oss/cpython/.venv/lib/python3.12/site-packages/sympy/core/facts.py", line 533 in <genexpr>
  File "/Users/user/oss/cpython/.venv/lib/python3.12/site-packages/sympy/core/facts.py", line 533 in deduce_all_facts
  File "/Users/user/oss/cpython/.venv/lib/python3.12/site-packages/sympy/core/assumptions.py", line 546 in _ask
  File "/Users/user/oss/cpython/.venv/lib/python3.12/site-packages/sympy/core/assumptions.py", line 483 in getit
  File "/Users/user/oss/cpython/.venv/lib/python3.12/site-packages/sympy/core/add.py", line 703 in _eval_is_zero
  File "/Users/user/oss/cpython/.venv/lib/python3.12/site-packages/sympy/core/assumptions.py", line 539 in _ask
  File "/Users/user/oss/cpython/.venv/lib/python3.12/site-packages/sympy/core/assumptions.py", line 483 in getit
  File "/Users/user/oss/cpython/.venv/lib/python3.12/site-packages/sympy/core/relational.py", line 1561 in is_eq
  File "/Users/user/oss/cpython/.venv/lib/python3.12/site-packages/sympy/core/relational.py", line 1555 in is_eq
  File "/Users/user/oss/cpython/.venv/lib/python3.12/site-packages/sympy/core/relational.py", line 629 in __new__
  File "/Users/user/oss/cpython/.venv/lib/python3.12/site-packages/sympy/core/tests/test_relational.py", line 967 in test_issue_10927
  File "/Users/user/oss/cpython/.venv/lib/python3.12/site-packages/sympy/testing/runtests.py", line 1281 in test_file
  File "/Users/user/oss/cpython/.venv/lib/python3.12/site-packages/sympy/testing/runtests.py", line 1146 in test
  File "/Users/user/oss/cpython/.venv/lib/python3.12/site-packages/sympy/testing/runtests.py", line 568 in _test
  File "/Users/user/oss/cpython/.venv/lib/python3.12/site-packages/sympy/testing/runtests.py", line 498 in test
  File "<stdin>", line 1 in <module>
[1]    21140 abort      python

Reacted by Owain Davies

OTheDev commented on Feb 26, 2023

OTheDev

Contributor

More actions

main (sympy.__version__ == '1.11.1'):

Python 3.12.0a5+ (heads/main:a35fd38b57, Feb 25 2023, 23:27:09) [Clang 13.1.6 (clang-1316.0.21.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sympy
>>> sympy.test(subprocess=False, seed=96737803)
============================= test process starts ==============================
executable:         /...omitted.../python.exe  (3.12.0-alpha-5) [CPython]
architecture:       64-bit
cache:              yes
ground types:       python 
numpy:              None
random seed:        96737803
hash randomization: off
...
sympy/functions/combinatorial/tests/test_comb_numbers.py[24] ....zsh: segmentation fault  ./python.exe

Second run

sympy/functions/special/tests/test_bsplines.py[11] .........wzsh: segmentation fault  ./python.exe

Not on 3.11.2 (Feb 8 Gzipped source tarball, sympy.__version__ == '1.11.1') though:

Python 3.11.2 (main, Feb 26 2023, 15:08:59) [Clang 13.1.6 (clang-1316.0.21.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sympy
>>> sympy.test(subprocess=False, seed=96737803)
============================= test process starts ==============================
executable:         /...omitted.../python.exe  (3.11.2-final-0) [CPython]
architecture:       64-bit
cache:              yes
ground types:       python 
numpy:              None
random seed:        96737803
hash randomization: off
...
True

Moreover

Python 3.11.2+ (heads/3.11:7a0dc8a802, Feb 26 2023, 16:22:23) [Clang 13.1.6 (clang-1316.0.21.2.5)] on darwin
...
True

It looks like if CPython is the culprit, it probably should be bisectable in 3.12. Unfortunately, the test suite is quite long! :)

oscarbenjamin commented on Feb 26, 2023
oscarbenjamin
on Feb 26, 2023
ContributorAuthor
More actions

I got another crash from sympy but not sure it's CPython issue or Sympy issue at this moment. (Buf AFAIK, sympy is pure-python code..)

The crash you showed suggests this is a segfault arising from a refcounting bug. Yes, SymPy is pure Python. It can be used with other optional dependencies that are not pure Python like numpy, gmpy2 etc but this bug is seen without any of those. A refcounting bug here is definitely a CPython bug.

oscarbenjamin commented on Feb 26, 2023

ContributorAuthor

More actions

do you have time to debug it to help us? Which test exacty fails? Or is it a combination / sequence of tests?

I am trying to bisect it currently but it's a slow process because of the time taken to build cpython and then run the sympy test suite. Also it is difficult because the failure itself is not completely deterministic. As I said in the OP running individual tests does not reproduce the failure so it is not possible to narrow it down to less than running the whole test suite.

Actually I have narrowed it down a little bit. It seems that the first 1/6 of the test suite is enough to reproduce this:

import sympy
sympy.test(subprocess=False, split="1/6")

That does not reduce the time taken for the crash to show up but it does reduce the time taken to reproduce the crash not happening which is useful for bisecting.

I haven't quite finished bisecting but I'm currently down to this range:

commit df10571a130c46d67756239e76ad69a3d730a990 (refs/bisect/bad)
Author: Steve Dower <[email protected]>
Date:   Mon Jan 16 16:05:39 2023 +0000

    gh-100320: Fix path calculations on Windows when python.exe is moved outside of the normal location (GH-100947)

commit 7b14c2ef194b6eed79670aa9d7e29ab8e2256a56
Author: Mark Shannon <[email protected]>
Date:   Mon Jan 16 12:35:21 2023 +0000

    GH-100982: Add `COMPARE_AND_BRANCH` instruction (GH-100983)

commit b1a74a182d8762bda51838401ac92b6ebad9632a
Author: Nikita Sobolev <[email protected]>
Date:   Mon Jan 16 13:46:07 2023 +0300

    gh-101056: Fix memory leak in `formatfloat()` in `bytesobject.c` (#101057)

commit b82049993f74185da71adf2eb8d6c8f15db063e1
Author: scrazzz <[email protected]>
Date:   Mon Jan 16 13:44:50 2023 +0300

    Docs: Remove duplicate `TraversableResources` reference (#101068)

commit 74c20b6ecee125e76c187c3f78c55f39501702ae (HEAD)
Author: Kumar Aditya <[email protected]>
Date:   Mon Jan 16 16:11:57 2023 +0530

    GH-100997: fix refleak in `_testinternalcapi.c` (#101058)

commit 8e9d08b062bbabfe439bc73f82e3d7bb3800189e
Author: Oleg Iarygin <[email protected]>
Date:   Mon Jan 16 00:36:31 2023 +0400

    gh-101041: Fix a misspelled name of `utctimetuple` in a doc warning (GH-101042)

commit 5e9df888dd5ab0e59d1cebc30c998a17aa65a3e2
Author: Cristián Maureira-Fredes <[email protected]>
Date:   Sun Jan 15 16:12:37 2023 +0100

    Fix `ctypes` typo in `set_exception` (#101040)

commit f6307d44167e9c110a1db442f7a33deb365e89a8 (refs/bisect/good-f6307d44167e9c110a1db442f7a33deb365e89a8)
Author: Kumar Aditya <[email protected]>
Date:   Sun Jan 15 20:39:26 2023 +0530

    GH-100892: consolidate `HEAD_LOCK/HEAD_UNLOCK` macros (#100953)

Reacted by Owain Davies

OTheDev commented on Feb 26, 2023
OTheDev
on Feb 26, 2023
Contributor
More actions
Tests pass in 3.12.0a4 (Head 3d5d3f7, Jan 10)

oscarbenjamin commented on Feb 26, 2023

ContributorAuthor

More actions

Bisect complete:

sympy/functions/elementary/tests/test_miscellaneous.py[18] ......s...........                                                                             [OK]
sympy/functions/elementary/tests/test_piecewise.py[63] ....ww...w..............................................w......                                    [OK]
sympy/functions/elementary/tests/test_trigonometric.py[86] ./bug.sh: line 12: 30073 Bus error: 10           ./python.exe -c 'import sympy; sympy.test(subprocess=False, split="1/6")'
$ git bisect bad
7b14c2ef194b6eed79670aa9d7e29ab8e2256a56 is the first bad commit
commit 7b14c2ef194b6eed79670aa9d7e29ab8e2256a56
Author: Mark Shannon <[email protected]>
Date:   Mon Jan 16 12:35:21 2023 +0000

    GH-100982: Add `COMPARE_AND_BRANCH` instruction (GH-100983)

:040000 040000 a01bf959d999386154fb4dfef5d568c0288cb316 dbd2b3a4a61424abfa5f9e726973b9d03bd24bfd M	Doc
:040000 040000 0fac1d9d6fc2e2066a40e527653ecdae70011c23 d3479ece49974dc513a50d3bd1d5684834e95970 M	Include
:040000 040000 58458255181b3044deeb7c7f852c89e37d398508 f24643aa3a7f1afbdd58639b6e4496f59f1da4f9 M	Lib
:040000 040000 3a605b820cd18bd683911c0a274405ab875a6c79 11052d23b2fb46e6cb0964518d8c215dbc5b3bcd M	Misc
:040000 040000 9364b4ad219faab45e789b146bd93c5694865d79 9a9702c99b36e043768cfde5b3dbdb97ba8972f3 M	Objects
:040000 040000 40b1d7e8c2b8c5af2fd1186256c49bebd3e9c954 83b3b65a9a27d8cb683313f6d280e9bfd20f16ca M	Python
:040000 040000 0cae5606a7c50631954bb014dd67b0384f201210 12b0ea6eb5eafad436ccf7564bacd0a72f40ab59 M	Tools

That's 7b14c2e from gh-100983.

I will do a bit more testing with the parent commit (b1a74a1) to confirm that the bug is not seen there.

CC @markshannon

Reacted by Owain Davies

OTheDev commented on Feb 26, 2023
OTheDev
on Feb 26, 2023
Contributor
More actions
Full test suite green for me in b1a74a1.
markshannon commented on Feb 26, 2023
markshannon
on Feb 26, 2023
Member
More actions
Does running the sympy/functions/special/tests/test_error_functions.py test repeatedly by itself cause the segfault?
OTheDev commented on Feb 26, 2023
OTheDev
on Feb 26, 2023
Contributor
More actions
Attempting to run the full test suite at 7b14c2e, I encounter a segfault at

... sympy/diffgeom/tests/test_diffgeom.py[16] ................ [OK] sympy/diffgeom/tests/test_function_diffgeom_book.py[4] ..zsh: segmentation fault ./python.exe

However, running sympy/diffgeom/tests/test_function_diffgeom_book.py repeatedly (100 times) doesn't seem to reproduce the segfault - likewise for sympy/functions/special/tests/test_error_functions.py.
oscarbenjamin commented on Feb 26, 2023
oscarbenjamin
on Feb 26, 2023
ContributorAuthor
More actions

I will do a bit more testing with the parent commit (b1a74a1) to confirm that the bug is not seen there.

I confirm the full test suite passes with the parent commit so I'm confident that the problem begines with 7b14c2e.

Does running the sympy/functions/special/tests/test_error_functions.py test repeatedly by itself cause the segfault?

I tried something similar to this. Exactly where it crashes is different on different runs but for example it sometimes crashes in the diffgeom tests so I made a script to run just the diffgeom tests 100 times and that didn't give any crash. I'll try again with:

import sympy for n in range(1000): print() print('-------------------------------------- n =', n) print() sympy.test('test_error_functions', subprocess=False)

It's possible that it would be different by setting the environment variable SYMPY_USE_CACHE=no although that makes everything much slower.

How many repetitions should I expect to need?

dtrodrigues commented on Feb 26, 2023

dtrodrigues

Contributor

More actions

The crash @corona10 mentioned is reproducible with --with-pydebug enabled. A simpler reproducer is

from sympy.core.symbol import symbols
x = symbols('x')
assert x > 0

The bytecode for both is below

b1a74a1 (good):


  4           0 RESUME                   0

  5           2 LOAD_GLOBAL              1 (NULL + symbols)
             14 LOAD_CONST               1 ('x')
             16 CALL                     1
             26 STORE_FAST               0 (x)

  6          28 LOAD_FAST                0 (x)
             30 LOAD_CONST               2 (0)
             32 COMPARE_OP              68 (>)
             36 POP_JUMP_IF_TRUE         2 (to 42)
             38 LOAD_ASSERTION_ERROR
             40 RAISE_VARARGS            1
        >>   42 LOAD_CONST               0 (None)
             44 RETURN_VALUE

7b14c2e (bad):


  4           0 RESUME                   0

  5           2 LOAD_GLOBAL              1 (NULL + symbols)
             14 LOAD_CONST               1 ('x')
             16 CALL                     1
             26 STORE_FAST               0 (x)

  6          28 LOAD_FAST                0 (x)
             30 LOAD_CONST               2 (0)
             32 COMPARE_AND_BRANCH      68 (>)
             36 POP_JUMP_IF_TRUE         2 (to 42)
             38 LOAD_ASSERTION_ERROR
             40 RAISE_VARARGS            1
        >>   42 LOAD_CONST               0 (None)
             44 RETURN_VALUE

with the difference between the two being COMPARE_OP vs COMPARE_AND_BRANCH.

I can't cause this alone to cause a segfault when --with-pydebug is not enabled.

Edit: Removing https://github.com/sympy/sympy/blob/sympy-1.11.1/sympy/core/tests/test_relational.py#L860-L869 from the sympy test suite allow sthe subset of the test suite which I was running on (split='1/12') to run without segfaulting anymore. Adding the test back in reintroduces the segfault that appears later on.

Reacted by Oleg Iarygin, Donghee Na and Owain Davies

oscarbenjamin commented on Feb 26, 2023

ContributorAuthor

More actions

A simpler reproducer is

from sympy.core.symbol import symbols
x = symbols('x')
assert x > 0

Okay this starts to make sense. A comparison like x > 0 will call x.__gt__(0) which would usually be expected to return a bool. However Symbol.__gt__ instead returns a symbolic GreaterThan(x, 0) object to represent the inequality symbolically:

>>> cond = x > 0
>>> cond
x > 0
>>> type(cond)
<class 'sympy.core.relational.StrictGreaterThan'>
>>> bool(cond)  # it isn't known if cond is True or False
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/oscar/current/sympy/sympy.git/sympy/core/relational.py", line 510, in __bool__
    raise TypeError("cannot determine truth value of Relational")
TypeError: cannot determine truth value of Relational

The COMPARE_AND_BRANCH opcode evaluates this here:

Lines 1854 to 1857 in 7b14c2e

	PyObject *cond = PyObject_RichCompare(left, right, oparg>>4);
	Py_DECREF(left);
	Py_DECREF(right);
	ERROR_IF(cond == NULL, error);

Now I think:

PyObject_RichCompare will raise an exception so it returns NULL.
Py_DECREF(left) decrefs the Symbol object x.
cond == NULL evaluates as true so we do the ERROR_IF.

I'm not sure what ERROR_IF does:

Line 68 in 7b14c2e

#define ERROR_IF(cond, labelname) ((void)0)

I guess it means goto error;. I'm not sure what happens then.

oscarbenjamin commented on Feb 26, 2023

ContributorAuthor

More actions

2. Py_DECREF(left) decrefs the Symbol object x.

To be clear it is the Symbol object refcount that causes the crash. Using --with-pydebug and the short repro above:

$ ./python.exe t.py 
Traceback (most recent call last):
  File "/Users/enojb/current/sympy/cpython.git/t.py", line 3, in <module>
    assert x > 0
           ^^^^^
  File "/Users/enojb/current/sympy/cpython.git/sympy/core/relational.py", line 511, in __bool__
    raise TypeError("cannot determine truth value of Relational")
TypeError: cannot determine truth value of Relational
Modules/gcmodule.c:115: gc_decref: Assertion "gc_get_refs(g) > 0" failed: refcount is too small
Enable tracemalloc to get the memory block allocation traceback

object address  : 0x103808410
object refcount : 10
object type     : 0x7f80fd043a30
object type name: Symbol
object repr     : x

Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
Python runtime state: finalizing (tstate=0x0000000101fe7000)

Current thread 0x0000000109b74600 (most recent call first):
  Garbage-collecting
  <no Python frame>
Abort trap: 6

oscarbenjamin commented on Feb 26, 2023

ContributorAuthor

More actions

PyObject_RichCompare will raise an exception so it returns NULL.

No it won't. It will return the GreaterThan:

Lines 1854 to 1869 in 7b14c2e

	PyObject *cond = PyObject_RichCompare(left, right, oparg>>4);
	Py_DECREF(left);
	Py_DECREF(right);
	ERROR_IF(cond == NULL, error);
	assert(_Py_OPCODE(next_instr[1]) == POP_JUMP_IF_FALSE \|\|
	_Py_OPCODE(next_instr[1]) == POP_JUMP_IF_TRUE);
	bool jump_on_true = _Py_OPCODE(next_instr[1]) == POP_JUMP_IF_TRUE;
	int offset = _Py_OPARG(next_instr[1]);
	int err = PyObject_IsTrue(cond);
	Py_DECREF(cond);
	if (err < 0) {
	goto error;
	}
	if (jump_on_true == (err != 0)) {
	JUMPBY(offset);
	}

Then PyObject_IsTrue returns -1 because GreaterThan.__bool__ raises. Then we hit the err < 0 and goto error;.

oscarbenjamin commented on Feb 26, 2023

ContributorAuthor

More actions

Before 7b14c2e the COMPARE_OP code did not call PyObject_IsTrue:

Lines 1839 to 1854 in b1a74a1

	inst(COMPARE_OP, (unused/1, left, right -- res)) {
	_PyCompareOpCache cache = (_PyCompareOpCache )next_instr;
	if (ADAPTIVE_COUNTER_IS_ZERO(cache->counter)) {
	assert(cframe.use_tracing == 0);
	next_instr--;
	_Py_Specialize_CompareOp(left, right, next_instr, oparg);
	DISPATCH_SAME_OPARG();
	}
	STAT_INC(COMPARE_OP, deferred);
	DECREMENT_ADAPTIVE_COUNTER(cache->counter);
	assert((oparg >> 4) <= Py_GE);
	res = PyObject_RichCompare(left, right, oparg>>4);
	Py_DECREF(left);
	Py_DECREF(right);
	ERROR_IF(res == NULL, error);
	}

oscarbenjamin commented on Feb 26, 2023