GeezerVison

Description

This fixes the AI (3.10) litellm flake. The test that fails changes from run to run. Recent example: test_binary_content_encoding_mixed_content[False-False], which failed 7 times on master between Sep 30 and Oct 13 (job). Other failures were in test_no_integration[False-False] and test_binary_content_encoding_image_url[True-False]. All 10 litellm failures I looked at on master have the same cause:

/usr/lib/python3.10/threading.py:1011: in _bootstrap_inner
    _sys.settrace(_trace_hook)
E   RuntimeError: Cannot install a trace function while another trace function is being installed

Root cause

• On CPython < 3.12, _PyEval_SetTrace uses a static int reentrant flag shared by the whole process. The flag stays set while audit hooks run (python/cpython#98257, fixed in 3.12).
• filelock 3.30.0 through 4.0.2 registers a Python audit hook (sys.addaudithook(_audit_fork_safety)) when it is imported. The py3.10 litellm envs pin filelock==3.32.4, and huggingface_hub imports it during the first test.
• pytest-cov calls threading.settrace, so every new thread calls sys.settrace() while it starts up. If the GIL switches while another thread is inside filelock's hook, the new thread's settrace() raises and the thread dies before it runs.
• The tests create a new single-worker executor (reset_litellm_executor), and LiteLLM finishes the gen_ai.chat span in a success callback submitted to it. A dead worker means the callback never runs, so the test gets len(chat_spans) == 0.

Fix
filelock 4.0.3 fixed this upstream: it no longer registers the hook on CPython < 3.12 (_SETTRACE_SURVIVES_AUDIT_HOOKS). This PR adds a py3.10,py3.11 rule, filelock>=4.0.3, to the litellm config, then regenerates tox.ini with scripts/generate-test-files.sh --only litellm.

• Only the four py3.10-litellm-* blocks change; they now pin filelock==4.1.0. py3.12 and later are unaffected and unchanged.
• Re-resolving those four blocks also moved other transitive pins, for example anthropic 1.0.0 → 1.13.0.
• releases.jsonl is unchanged. --only would have pruned it, so I restored it. package_dependencies.jsonl only gains the 3 new resolution entries.

No retries, sleeps, skipped tests or weaker assertions.

Reproduction: getsentry/repro#60
It is a standalone script with no litellm and no pytest. On Python 3.10 with filelock 3.32.4, it lost 14 of 32,000 callbacks in one run, and 3 of 5 shorter runs (16,000 callbacks each) reproduced it. Under the same conditions it lost 0 of 32,000 with filelock 4.0.3, and 0 on Python 3.12.

Original suite, run locally (py3.10-litellm-v1.83.14, with coverage):

• With default settings: 0 of 20 runs failed.
• With GIL switching set to happen more often (sys.setswitchinterval(1e-6) via a -p plugin; no test code changed): 4 of 20 runs hit the RuntimeError, and 3 of 20 failed.

This is a draft until validation of the fix against both the repro and the original suite is posted here.

Possible follow-up, not changed here: py3.11-anthropic-v0.16.0 also pins filelock==3.32.4.

Issues

• resolves: Resolve RuntimeError: Cannot install a trace function while another trace function is being installed #7014
• Linear: PY-2660

via Junior system actor event.

--

View Junior Session [Sentry]