GeezerVison is showing github.com — Android buildbot failures after updating to API level 35 · Issue #142387 · python/cpython
Android buildbot failures after updating to API level 35 #142387Labels OS-androidinfraCI, GitHub Actions, buildbots, Dependabot, etc.CI, GitHub Actions, buildbots, Dependabot, etc.type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error DescriptionBug report
Bug description:
This update was done in #142289 because the API level 34 system image was much larger, and was filling up the disk on GitHub Actions. It works fine in GitHub Actions, but caused different problems in the two Android buildbots:
On the aarch64 buildbot, the new system image failed to start, with its logs showing a repeated loop of system services crashing and restarting:
12-05 14:50:56.223 4745 4745 F DEBUG : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
12-05 14:50:56.223 4745 4745 F DEBUG : Build fingerprint: 'Android/sdk_slim_arm64/emulator64_arm64:14/UE1A.230829.036.A1/11228894:userdebug/test-keys'
12-05 14:50:56.223 4745 4745 F DEBUG : Revision: '0'
12-05 14:50:56.223 4745 4745 F DEBUG : ABI: 'arm64'
12-05 14:50:56.223 4745 4745 F DEBUG : Timestamp: 2025-12-05 14:50:56.190712855+0000
12-05 14:50:56.223 4745 4745 F DEBUG : Process uptime: 1s
12-05 14:50:56.223 4745 4745 F DEBUG : Cmdline: /system/bin/surfaceflinger
12-05 14:50:56.223 4745 4745 F DEBUG : pid: 4738, tid: 4742, name: surfaceflinger >>> /system/bin/surfaceflinger <<<
12-05 14:50:56.223 4745 4745 F DEBUG : uid: 1000
12-05 14:50:56.223 4745 4745 F DEBUG : tagged_addr_ctrl: 0000000000000001 (PR_TAGGED_ADDR_ENABLE)
12-05 14:50:56.223 4745 4745 F DEBUG : pac_enabled_keys: 000000000000000f (PR_PAC_APIAKEY, PR_PAC_APIBKEY, PR_PAC_APDAKEY, PR_PAC_APDBKEY)
12-05 14:50:56.223 4745 4745 F DEBUG : signal 6 (SIGABRT), code -1 (SI_QUEUE), fault addr --------
12-05 14:50:56.223 4745 4745 F DEBUG : Abort message: 'EGLContext creation failed'
12-05 14:50:56.223 4745 4745 F DEBUG : x0 0000000000000000 x1 0000000000001286 x2 0000000000000006 x3 000000785e5ba4a0
12-05 14:50:56.223 4745 4745 F DEBUG : x4 0000000000000010 x5 0000000000000010 x6 0000000000000010 x7 7f7f7f7f7f7f7f7f
12-05 14:50:56.223 4745 4745 F DEBUG : x8 00000000000000f0 x9 0000007afdc33090 x10 0000000000000001 x11 0000007afdc86058
12-05 14:50:56.223 4745 4745 F DEBUG : x12 000000785e5ba470 x13 000000000000001b x14 000000785e5ba630 x15 00014295acd0344e
12-05 14:50:56.223 4745 4745 F DEBUG : x16 0000007afdcf3d08 x17 0000007afdcc7e90 x18 000000785d54c000 x19 0000000000001282
12-05 14:50:56.223 4745 4745 F DEBUG : x20 0000000000001286 x21 00000000ffffffff x22 000000785e5babd8 x23 0000000000000002
12-05 14:50:56.223 4745 4745 F DEBUG : x24 0000000000000000 x25 0000000000000000 x26 000000785e5baff8 x27 00000000000fc000
12-05 14:50:56.223 4745 4745 F DEBUG : x28 00000000000fe000 x29 000000785e5ba520
12-05 14:50:56.223 4745 4745 F DEBUG : lr 0000007afdc779b8 sp 000000785e5ba480 pc 0000007afdc779e4 pst 0000000000001000
EGL-related messages leading up to the crash (from a different run):
01-25 17:47:44.958 4473 4486 I EGL_emulation: Opening libGLESv1_CM_emulation.so
01-25 17:47:44.958 4473 4486 I EGL_emulation: Opening libGLESv2_emulation.so
01-25 17:47:44.959 4473 4486 I RenderEngine: EGL information:
01-25 17:47:44.959 4473 4486 I RenderEngine: vendor : Android
01-25 17:47:44.959 4473 4486 I RenderEngine: version : 1.4 Android META-EGL
01-25 17:47:44.959 4473 4486 I RenderEngine: extensions: EGL_ANDROID_front_buffer_auto_refresh EGL_ANDROID_get_native_client_buffer EGL_ANDROID_presentation_time EGL_EXT_surface_CTA861_3_metadata EGL_EXT_surface_SMPTE2086_metadata EGL_KHR_get_all_proc_addresses EGL_KHR_swap_buffers_with_damage EGL_ANDROID_get_frame_timestamps EGL_ANDROID_image_native_buffer EGL_ANDROID_native_fence_sync EGL_ANDROID_recordable EGL_KHR_fence_sync EGL_KHR_gl_texture_2D_image EGL_KHR_image EGL_KHR_image_base EGL_KHR_wait_sync
01-25 17:47:44.959 4473 4486 I RenderEngine: Client API: OpenGL_ES
01-25 17:47:44.959 4473 4486 I RenderEngine: EGLSurface: 8-8-8-8, config=0x2
01-25 17:47:44.959 4473 4486 E EGL_emulation: eglCreateContext: EGL_BAD_CONFIG: no ES 3 support
01-25 17:47:44.959 4473 4486 E EGL_emulation: tid 4486: eglCreateContext(1816): error 0x3005 (EGL_BAD_CONFIG)
01-25 17:47:44.959 4473 4486 F RenderEngine: EGLContext creation failed
01-25 17:47:44.959 4473 4486 F libc : Fatal signal 6 (SIGABRT), code -1 (SI_QUEUE) in tid 4486 (surfaceflinger), pid 4473 (surfaceflinger)
I guessed this was caused by the emulator not being compatible with the emulator system image, so I installed the current emulator version 36.3.10, but it made no difference. Neither did 36.2.12, which is what I'd used to test the update on my own aarch64 Mac, or 35.6.11, which is the last in the 35 series. Unfortunately I didn't keep a note of what the previously installed version was.
The same thing happens with several other system images, even the API level 34 one we were previously using, either from the existing snapshot or a freshly created one.
I've taken the aarch64 buildbot offline until this is fixed. Next things to try:
- Check disk space, since that's been a problem on this machine before.
- Move the SDK and .android directories out of the way and recreate them from scratch.
- Reboot the machine.
- Check for OS updates.
- The message is related to OpenGL, so maybe it'll make a difference to:
- Log in graphically
- Start the emulator with different GPU or OpenGL settings
- Try running an emulator interactively through Android Studio
On the x86_64 buildbot, the tests are running to completion, but giving errors about not being able to contact pythontest.net. It's possible this is caused by an emulator / system image incompatibility, but be careful in case the same thing happens here.
CPython versions tested on:
CPython main branch
Operating systems tested on:
Other
Linked PRs
Activitymhsmith commented on Jan 19, 2026 The following SDK packages were installed on the x86_64 buildbot:
Path | Version | Description | Location
------- | ------- | ------- | -------
build-tools;34.0.0 | 34.0.0 | Android SDK Build-Tools 34 | build-tools/34.0.0
build-tools;35.0.0 | 35.0.0 | Android SDK Build-Tools 35 | build-tools/35.0.0
cmake;3.22.1 | 3.22.1 | CMake 3.22.1 | cmake/3.22.1
emulator | 35.1.21 | Android Emulator | emulator
ndk;26.2.11394342 | 26.2.11394342 | NDK (Side by side) 26.2.11394342 | ndk/26.2.11394342
ndk;27.1.12297006 | 27.1.12297006 | NDK (Side by side) 27.1.12297006 | ndk/27.1.12297006
ndk;27.2.12479018 | 27.2.12479018 | NDK (Side by side) 27.2.12479018 | ndk/27.2.12479018
ndk;27.3.13750724 | 27.3.13750724 | NDK (Side by side) 27.3.13750724 | ndk/27.3.13750724
platform-tools | 35.0.2 | Android SDK Platform-Tools | platform-tools
platforms;android-34 | 3 | Android SDK Platform 34 | platforms/android-34
platforms;android-35 | 2 | Android SDK Platform 35 | platforms/android-35
system-images;android-34;aosp_atd;x86_64 | 2 | AOSP ATD Intel x86_64 Atom System Image | system-images/android-34/aosp_atd/x86_64
system-images;android-35;aosp_atd;x86_64 | 1 | AOSP ATD Intel x86_64 Atom System Image | system-images/android-35/aosp_atd/x86_64
At first it seemed like some of the tests involving pythontest.net were passing, as there were some "fetching" lines in the log in places such as test_hashlib. But it turns out these tests were actually being skipped when the download failed.
Upgrading emulator to version 36.3.10 was enough to fix the problem on this worker. mhsmith commented on Jan 25, 2026 As of today, the aarch64 buildbot had been up for 222 days, and had the following SDK packages installed:
Path | Version | Description | Location
------- | ------- | ------- | -------
build-tools;34.0.0 | 34.0.0 | Android SDK Build-Tools 34 | build-tools/34.0.0
build-tools;35.0.0 | 35.0.0 | Android SDK Build-Tools 35 | build-tools/35.0.0
cmake;3.22.1 | 3.22.1 | CMake 3.22.1 | cmake/3.22.1
emulator | 35.6.11 | Android Emulator | emulator
ndk;27.2.12479018 | 27.2.12479018 | NDK (Side by side) 27.2.12479018 | ndk/27.2.12479018
ndk;27.3.13750724 | 27.3.13750724 | NDK (Side by side) 27.3.13750724 | ndk/27.3.13750724
platform-tools | 35.0.2 | Android SDK Platform-Tools | platform-tools
platforms;android-34 | 3 | Android SDK Platform 34 | platforms/android-34
platforms;android-35 | 2 | Android SDK Platform 35 | platforms/android-35
system-images;android-34;aosp_atd;arm64-v8a | 2 | AOSP ATD ARM 64 v8a System Image | system-images/android-34/aosp_atd/arm64-v8a
system-images;android-35;aosp_atd;arm64-v8a | 1 | AOSP ATD ARM 64 System Image | system-images/android-35/aosp_atd/arm64-v8a
system-images;android-35;default;arm64-v8a | 2 | ARM 64 v8a System Image | system-images/android-35/default/arm64-v8a
Check disk space, since that's been a problem on this machine before
There was 37 GB free, much more than the 12 GB where it's had issues before.
Move the SDK and .android directories out of the way and recreate them from scratch.
Removed .android and allowed it to be recreated. Made no difference.
Removed android-sdk and recreated it from commandlinetools-mac-14742923_latest.zip, downloaded from https://developer.android.com/studio. Made no difference.
Reboot the machine
Made no difference, although it did unintentionally cause the buildbot worker to start.
The message is related to OpenGL, so maybe it'll make a difference to log in graphically.
Made no difference, regardless of whether I logged in as the admin user (ec2-user), buildbot user (android), or ran android.py in a terminal within the android user's graphical session – though that was probably reusing a Gradle daemon started in the SSH session, so it wouldn't have made any difference anyway. mhsmith commented on Jan 28, 2026
Check for OS updates
Updating macOS from 15.3.1 to 15.7.3 allowed the emulator to start! So my best guess is that there was some issue with that macOS version which wasn't exposed until we re-ran the Gradle-managed emulator snapshot creation process, which we probably hadn't done since the previous OS update.
However, the emulator was unable to access the internet, causing the same test failures as seen in the x86_64 buildbot above. All the emulator versions mentioned above had the same problem, but reducing the system image API level to 34 or lower made the problem go away.
So I think we'd better just treat the aosp_atd image as untrustworthy on API level 35. I'll make a PR to reduce it to 33, which while not as small as 35, is still much smaller than 34:
4249 /Users/android/android-sdk/system-images/android-33/aosp_atd
8333 /Users/android/android-sdk/system-images/android-34/aosp_atd
1688 /Users/android/android-sdk/system-images/android-35/aosp_atd
Reacted by Hugo van Kemenade mhsmith commented on Apr 7, 2026 Since this PR was merged, the API level was reduced again to 32 in #142912.
While testing #148054, I deleted the existing AVD on the aarch64 buildbot, and it failed to recreate with the same "EGLContext creation failed" errors as above. However, the problem isn't specific to that PR; it also happens on the main branch now that the old emulator is gone.
I don't know why the emulator is so fragile on this machine; I've never had such problems on my own Mac. mhsmith commented on Apr 7, 2026 Although the system image was reinstalled, it's still at version 1, which I assume is the only version. The emulator version is also unchanged:
emulator | 35.6.11 | Android Emulator | emulator
system-images;android-32;aosp_atd;arm64-v8a | 1 | AOSP ATD ARM 64 v8a System Image | system-images/android-32/aosp_atd/arm64-v8a
The macOS version is also unchanged at 15.7.3. However, the uptime command showed that the machine was last rebooted 7 days ago (March 30). Logging in graphically gave me an alert that the machine had restarted because of a problem:
panic(cpu 1 caller 0xfffffe0027584ee0): IOKit Daemon (kernelmanagerd) stall[0], (60s): registry root held busy, kernelmanagerd has not checked in @IOService.cpp:5829
Details
Debugger message: panic
Memory ID: 0x6
OS release type: User
OS version: 24G419
Kernel version: Darwin Kernel Version 24.6.0: Wed Nov 5 21:33:59 PST 2025; root:xnu-11417.140.69.705.2~1/RELEASE_ARM64_T8112
Fileset Kernelcache UUID: 8EF81EBFB2DA156B9DB72F21B3FBCE3A
Kernel UUID: 2BDF559B-BD91-31AC-8F09-345BD5A73C49
Boot session UUID: 4A8AA60F-DF67-4BF7-B95C-75A2D7D46F51
iBoot version: iBoot-11881.140.96
iBoot Stage 2 version: iBoot-13822.61.10
secure boot?: YES
roots installed: 0
Paniclog version: 14
KernelCache slide: 0x000000001e640000
KernelCache base: 0xfffffe0025644000
Kernel slide: 0x000000001e648000
Kernel text base: 0xfffffe002564c000
Kernel text exec slide: 0x000000001fdcc000
Kernel text exec base: 0xfffffe0026dd0000
mach_absolute_time: 0x76a71d50
Epoch Time: sec usec
Boot : 0x69ca527a 0x0009c958
Sleep : 0x00000000 0x00000000
Wake : 0x00000000 0x00000000
Calendar: 0x69ca52c9 0x0004dfcc
Zone info:
Zone map: 0xfffffe1016000000 - 0xfffffe3616000000
. VM : 0xfffffe1016000000 - 0xfffffe15e2000000
. RO : 0xfffffe15e2000000 - 0xfffffe187c000000
. GEN0 : 0xfffffe187c000000 - 0xfffffe1e48000000
. GEN1 : 0xfffffe1e48000000 - 0xfffffe2414000000
. GEN2 : 0xfffffe2414000000 - 0xfffffe29e0000000
. GEN3 : 0xfffffe29e0000000 - 0xfffffe2fac000000
. DATA : 0xfffffe2fac000000 - 0xfffffe3616000000
Metadata: 0xfffffec8fc010000 - 0xfffffec905810000
Bitmaps : 0xfffffec905810000 - 0xfffffec909e24000
Extra : 0 - 0
CORE 0 recently retired instr at 0xfffffe0026f94028
CORE 1 recently retired instr at 0xfffffe0026f92784
CORE 2 recently retired instr at 0xfffffe0026f94028
CORE 3 recently retired instr at 0xfffffe0026f94028
CORE 4 recently retired instr at 0xfffffe0026f94028
CORE 5 recently retired instr at 0xfffffe0026f94028
CORE 6 recently retired instr at 0xfffffe0026f94028
CORE 7 recently retired instr at 0xfffffe0026f94028
TPIDRx_ELy = {1: 0xfffffe241651a7b8 0: 0x0000000000000001 0ro: 0x000000016af0f0e0 }
CORE 0 PVH locks held: None
CORE 1 PVH locks held: None
CORE 2 PVH locks held: None
CORE 3 PVH locks held: None
CORE 4 PVH locks held: None
CORE 5 PVH locks held: None
CORE 6 PVH locks held: None
CORE 7 PVH locks held: None
CORE 0: PC=0xfffffe0026e6849c, LR=0xfffffe0026e6849c, FP=0xfffffeca0ae6bed0
CORE 1 is the one that panicked. Check the full backtrace for details.
CORE 2: PC=0xfffffe0026e6849c, LR=0xfffffe0026e6849c, FP=0xfffffeca0ba47ed0
CORE 3: PC=0xfffffe002a131dbc, LR=0xfffffe002a131e08, FP=0xfffffeca0b8a2c10
CORE 4: PC=0xfffffe0026e6849c, LR=0xfffffe0026e6849c, FP=0xfffffeca0a967ed0
CORE 5: PC=0xfffffe0026e684a0, LR=0xfffffe0026e6849c, FP=0xfffffeca0b63fed0
CORE 6: PC=0xfffffe0026e684a0, LR=0xfffffe0026e6849c, FP=0xfffffeca0ba23ed0
CORE 7: PC=0xfffffe0026e684a0, LR=0xfffffe0026e6849c, FP=0xfffffeca0a17bed0
Compressor Info: 0% of compressed pages limit (OK) and 0% of segments limit (OK) with 0 swapfiles and OK swap space
Panicked task 0xfffffe2eabf68e38: 248 pages, 4 threads: pid 110: watchdogd
Panicked thread: 0xfffffe241651a7b8, backtrace: 0xfffffeca0b777490, tid: 2149
lr: 0xfffffe0026e2c72c fp: 0xfffffeca0b777520
lr: 0xfffffe0026f8a7dc fp: 0xfffffeca0b777590
lr: 0xfffffe0026f889b0 fp: 0xfffffeca0b777650
lr: 0xfffffe0026dd78b0 fp: 0xfffffeca0b777660
lr: 0xfffffe0026e2ca68 fp: 0xfffffeca0b777a30
lr: 0xfffffe00276bbfc4 fp: 0xfffffeca0b777a50
lr: 0xfffffe0027584ee0 fp: 0xfffffeca0b777b30
lr: 0xfffffe0026f51f70 fp: 0xfffffeca0b777b60
lr: 0xfffffe0026e33578 fp: 0xfffffeca0b777c00
lr: 0xfffffe0026e0a22c fp: 0xfffffeca0b777c60
lr: 0xfffffe0026e1dbc0 fp: 0xfffffeca0b777cb0
lr: 0xfffffe0026e1dfd8 fp: 0xfffffeca0b777da0
lr: 0xfffffe0026f7d954 fp: 0xfffffeca0b777e50
lr: 0xfffffe0026f88e80 fp: 0xfffffeca0b777f10
lr: 0xfffffe0026dd78b0 fp: 0xfffffeca0b777f20
lr: 0x0000000190c35c34 fp: 0x0000000000000000
last started kext at 981389383: com.apple.driver.AppleUSBODD 556 (addr 0xfffffe0025c6a390, size 1941)
loaded kexts:
com.apple.driver.AppleUSBODD 556
com.apple.iokit.SCSITaskUserClient 500.120.2
com.apple.driver.AppleUSBMassStorageInterfaceNub 556
com.apple.driver.CoreKDL 1
com.apple.driver.AppleBiometricServices 1
com.apple.driver.DiskImages.ReadWriteDiskImage 493.0.0
com.apple.driver.DiskImages.UDIFDiskImage 493.0.0
com.apple.driver.DiskImages.RAMBackingStore 493.0.0
com.apple.driver.DiskImages.FileBackingStore 493.0.0
com.apple.iokit.AppleBCM5701Ethernet 11.0.0
com.apple.driver.SEPHibernation 1
com.apple.driver.AppleUSBDeviceNCM 5.0.0
com.apple.driver.AppleThunderboltIP 4.0.3
com.apple.driver.AppleFileSystemDriver 3.0.1
com.apple.nke.l2tp 1.9
com.apple.filesystems.tmpfs 1
com.apple.driver.AppleSmartBatteryManager 161.0.0
com.apple.filesystems.nfs 1
com.apple.filesystems.lifs 1
com.apple.filesystems.apfs 2332.140.13
com.apple.IOTextEncryptionFamily 1.0.0
com.apple.filesystems.hfs.kext 683.120.3
com.apple.security.BootPolicy 1
com.apple.BootCache 40
com.apple.AppleFSCompression.AppleFSCompressionTypeZlib 1.0.0
com.apple.AppleFSCompression.AppleFSCompressionTypeDataless 1.0.0d1
com.apple.driver.AppleAOPVoiceTrigger 440.4
com.apple.driver.AppleTypeCRetimer 1.0.0
com.apple.AppleEmbeddedSimpleSPINORFlasher 1
com.apple.driver.AppleCS42L84Audio 840.26
com.apple.driver.ApplePMP 1
com.apple.driver.AppleSmartIO2 1
com.apple.driver.AppleSN012776Amp 840.26
com.apple.driver.AppleT8020SOCTuner 1
com.apple.driver.AppleT8112CLPC 1
com.apple.driver.AppleM68Buttons 1.0.0d1
com.apple.driver.ApplePMPFirmware 1
com.apple.AGXG14G 329.2
com.apple.driver.AppleProResHW 475.10
com.apple.driver.AppleAVD 865
com.apple.driver.AppleAVE2 803.73.1
com.apple.driver.AppleJPEGDriver 7.6.9
com.apple.driver.AppleMobileDispH14G-DCP 140.0
com.apple.driver.usb.AppleSynopsysUSB40XHCI 1
com.apple.driver.AudioDMAController-T8112 460.8.1
com.apple.driver.AppleSPIMC 1
com.apple.driver.AppleSamsungSerial 1.0.0d1
com.apple.driver.AppleSerialShim 1
com.apple.driver.AppleEventLogHandler 1
com.apple.driver.AppleS5L8960XNCO 1
com.apple.driver.AppleT8112PMGR 1
com.apple.driver.AppleS8000AES 1
com.apple.driver.AppleS5L8920XPWM 1.0.0d1
com.apple.driver.AppleS8000DWI 1.0.0d1
com.apple.driver.AppleInterruptControllerV2 1.0.0d1
com.apple.driver.AppleBluetoothModule 1
com.apple.driver.AppleBCMWLANBusInterfacePCIe 1
com.apple.driver.AppleT8110DART 1
com.apple.driver.AppleS5L8940XI2C 1.0.0d2
com.apple.driver.AppleT8110 1
com.apple.iokit.IOUserEthernet 1.0.1
com.apple.driver.usb.AppleUSBUserHCI 1
com.apple.iokit.IOKitRegistryCompatibility 1
com.apple.iokit.EndpointSecurity 1
com.apple.driver.AppleDiskImages2 385.140.4
com.apple.AppleSystemPolicy 2.0.0
com.apple.kec.InvalidateHmac 1
com.apple.kec.AppleEncryptedArchive 1
com.apple.iokit.IOSCSIMultimediaCommandsDevice 500.120.2
com.apple.iokit.IOBDStorageFamily 1.8
com.apple.iokit.IODVDStorageFamily 1.8
com.apple.iokit.IOCDStorageFamily 1.8
com.apple.driver.usb.cdc 5.0.0
com.apple.driver.AppleUSBAudio 750.4
com.apple.iokit.IOAudioFamily 600.2
com.apple.vecLib.kext 1.2.0
com.apple.driver.AppleConvergedIPCOLYBTControl 1
com.apple.driver.AppleConvergedPCI 1
com.apple.driver.AppleBluetoothDebug 1
com.apple.driver.AppleMesaSEPDriver 100.99
com.apple.iokit.IOBiometricFamily 1
com.apple.driver.DiskImages.KernelBacked 493.0.0
com.apple.driver.AppleUVDMDriver 1.0.0
com.apple.driver.AppleUVDM 1.0.0
com.apple.driver.AppleThunderboltPCIUpAdapter 4.1.1
com.apple.driver.AppleThunderboltDPOutAdapter 8.5.1
com.apple.driver.usb.AppleEmbeddedUSBXHCIPCI 1
com.apple.driver.usb.AppleUSBXHCIPCI 1.2
com.apple.driver.AppleXsanScheme 3
com.apple.driver.AppleSEPHDCPManager 1.0.1
com.apple.driver.AppleTrustedAccessory 1
com.apple.iokit.AppleSEPGenericTransfer 1
com.apple.driver.usb.networking 5.0.0
com.apple.driver.AppleThunderboltDPInAdapter 8.5.1
com.apple.driver.AppleThunderboltDPAdapterFamily 8.5.1
com.apple.driver.AppleThunderboltUSBDownAdapter 1.0.4
com.apple.driver.AppleThunderboltPCIDownAdapter 4.1.1
com.apple.driver.AppleBTM 1.0.1
com.apple.nke.ppp 1.9
com.apple.driver.AppleBSDKextStarter 3
com.apple.kext.triggers 1.0
com.apple.driver.IOHIDPowerSource 1
com.apple.driver.AppleCallbackPowerSource 1
com.apple.filesystems.hfs.encodings.kext 1
com.apple.driver.AppleSyntheticGameController 12.6.1
com.apple.driver.usb.AppleUSBHostPacketFilter 1.0
com.apple.driver.AppleDCPDPTXProxy 1.0.0
com.apple.driver.DCPDPFamilyProxy 1
com.apple.driver.AppleDiagnosticDataAccessReadOnly 1.0.0
com.apple.driver.AppleHPM 3.4.4
com.apple.driver.AppleCSEmbeddedAudio 840.26
com.apple.driver.AppleAOPAudio 440.12
com.apple.driver.AppleEmbeddedAudio 840.26
com.apple.iokit.AppleARMIISAudio 440.17
com.apple.driver.IISAudioIsolatedStreamECProxy 440.17
com.apple.driver.ExclavesAudioKext 240.34
com.apple.driver.ApplePassthroughPPM 3.0
com.apple.driver.AppleA7IOP-MXWrap-v1 1.0.2
com.apple.driver.AppleT8103PCIeCPIODMA 1
com.apple.driver.AppleThunderboltNHI 7.2.81
com.apple.iokit.IONVMeFamily 2.1.0
com.apple.driver.AppleNANDConfigAccess 1.0.0
com.apple.driver.AppleMobileApNonce 1
com.apple.driver.AppleSPU 1
com.apple.AGXFirmwareKextG14GRTBuddy 1
com.apple.AGXFirmwareKextRTBuddy64 329.2
com.apple.driver.AppleMCA2-T8112 940.3
com.apple.driver.AppleEmbeddedAudioLibs 420.3
com.apple.iokit.IOGPUFamily 104.6.3
com.apple.driver.AppleH11ANEInterface 8.600.2
com.apple.driver.DCPAVFamilyProxy 1
com.apple.iokit.IOMobileGraphicsFamily-DCP 343.0.0
com.apple.driver.AppleDCP 1
com.apple.driver.AppleFirmwareKit 1
com.apple.driver.AppleFirmwareUpdateKext 1
com.apple.iokit.IOMobileGraphicsFamily 343.0.0
com.apple.driver.AppleM2ScalerCSCDriver 265.0.0
com.apple.driver.AppleStockholmControl 1.0.0
com.apple.driver.AppleSPMIPMU 1.0.1
com.apple.driver.AppleDialogPMU 1.0.1
com.apple.driver.AppleSPMI 1.0.1
com.apple.driver.AppleUSBXDCIARM 1.0
com.apple.driver.AppleUSBXDCI 1.0
com.apple.iokit.IOUSBDeviceFamily 2.0.0
com.apple.driver.usb.AppleSynopsysUSBXHCI 1
com.apple.driver.usb.AppleUSBXHCI 1.2
com.apple.driver.AppleEmbeddedUSBHost 1
com.apple.driver.usb.AppleUSBHub 1.2
com.apple.driver.usb.AppleUSBHostCompositeDevice 1.2
com.apple.driver.AppleT8112TypeCPhy 1
com.apple.driver.AppleT8103TypeCPhy 1
com.apple.driver.AppleDisplayCrossbar 1.0.0
com.apple.iokit.IODisplayPortFamily 1.0.0
com.apple.driver.AppleTypeCPhy 1
com.apple.driver.AppleSART 1
com.apple.driver.ApplePMGR 1
com.apple.driver.AppleA7IOP-ASCWrap-v4 1.0.2
com.apple.driver.AppleARMWatchdogTimer 1
com.apple.driver.AppleT8112PCIe 1
com.apple.driver.AppleT8103PCIeC 1
com.apple.iokit.IOThunderboltFamily 9.3.3
com.apple.iokit.IOPortFamily 1.0
com.apple.driver.AppleBluetoothDebugService 1
com.apple.driver.AppleBCMWLANCore 1.0.0
com.apple.iokit.IO80211Family 1200.13.1
com.apple.driver.IOImageLoader 1.0.0
com.apple.driver.AppleOLYHAL 1
com.apple.iokit.IOAVBFamily 1320.3
com.apple.plugin.IOgPTPPlugin 1340.12
com.apple.driver.AppleGPIOICController 1.0.2
com.apple.driver.AppleFireStormErrorHandler 1
com.apple.driver.AppleMultiFunctionManager 1
com.apple.driver.corecapture 1.0.4
com.apple.driver.AppleEmbeddedPCIE 1
com.apple.driver.ApplePIODMA 1
com.apple.iokit.IOTimeSyncFamily 1340.12
com.apple.driver.DiskImages 493.0.0
com.apple.iokit.IOGraphicsFamily 599
com.apple.iokit.IOBluetoothFamily 9.0.0
com.apple.driver.AppleUIO 1
com.apple.driver.AppleSSE 1.0
com.apple.driver.AppleSEPKeyStore 2
com.apple.driver.AppleUSBTDM 556
com.apple.iokit.IOUSBMassStorageDriver 259.100.1
com.apple.iokit.IOPCIFamily 2.9
com.apple.iokit.IOUSBHostFamily 1.2
com.apple.driver.AppleUSBHostMergeProperties 1.2
com.apple.driver.usb.AppleUSBCommon 1.0
com.apple.driver.AppleSMC 3.1.9
com.apple.driver.RTBuddy 1.0.0
com.apple.driver.AppleEmbeddedTempSensor 1.0.0
com.apple.driver.AppleARMPMU 1.0
com.apple.iokit.IOAccessoryManager 1.0.0
com.apple.driver.AppleOnboardSerial 1.0
com.apple.iokit.IOSerialFamily 11
com.apple.iokit.IOSCSIBlockCommandsDevice 500.120.2
com.apple.iokit.IOSCSIArchitectureModelFamily 500.120.2
com.apple.driver.AppleRSMChannel 1
com.apple.iokit.IORSMFamily 1
com.apple.driver.AppleLockdownMode 1
com.apple.driver.AppleIPAppender 1.0
com.apple.iokit.IOSkywalkFamily 1.0
com.apple.driver.mDNSOffloadUserClient 1.0.1b8
com.apple.iokit.IONetworkingFamily 3.4
com.apple.driver.AppleFDEKeyStore 28.30
com.apple.driver.AppleEffaceableStorage 1.0
com.apple.driver.AppleCredentialManager 1.0
com.apple.driver.AppleSEPManager 1.0.1
com.apple.driver.IODARTFamily 1
com.apple.driver.AppleA7IOP 1.0.2
com.apple.driver.IOSlaveProcessor 1
com.apple.driver.AppleBiometricSensor 2
com.apple.iokit.IOHIDFamily 2.0.0
com.apple.AUC 1.0
com.apple.iokit.IOSurface 372.5.2
com.apple.iokit.IOAVFamily 1.0.0
com.apple.iokit.IOHDCPFamily 1.0.0
com.apple.iokit.IOCECFamily 1
com.apple.iokit.IOAudio2Family 1.0
com.apple.driver.AppleIISController 440.2
com.apple.driver.AppleAudioClockLibs 420.3
com.apple.driver.FairPlayIOKit 72.13.0
com.apple.driver.AppleARMPlatform 1.0.2
com.apple.iokit.IOSlowAdaptiveClockingFamily 1.0.0
com.apple.iokit.IOReportFamily 47
com.apple.security.quarantine 4
com.apple.security.sandbox 300.0
com.apple.iokit.IOStorageFamily 2.1
com.apple.kext.AppleMatch 1.0.0d1
com.apple.driver.AppleMobileFileIntegrity 1.0.5
com.apple.iokit.CoreAnalyticsFamily 1
com.apple.security.AppleImage4 7.0.0
com.apple.kext.CoreTrust 1
com.apple.iokit.IOCryptoAcceleratorFamily 1.0.1
com.apple.kec.pthread 1
com.apple.kec.Libm 1
com.apple.kec.Compression 1.0
com.apple.kec.corecrypto 14.0
** Stackshot Succeeded ** Bytes Traced 169019 (Uncompressed 412912) **
mhsmith commented on Apr 7, 2026 Rebooting made no difference, but updating the emulator from version 35.6.11 to 36.5.10 fixed the problem. mhsmith commented on Jun 17, 2026
On the x86_64 buildbot, the tests are running to completion, but giving errors about not being able to contact pythontest.net.
These recently appeared intermittently on the aarch64 buildbot, on the main and 3.15 branches only. There are several network-related failures, but the most common ones are:
======================================================================
FAIL: test_ftp_error (test.test_urllib2.HandlerTests.test_ftp_error)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/data/user/0/org.python.testbed/files/python/lib/python3.15/urllib/request.py", line 1531, in ftp_open
host = socket.gethostbyname(host)
socket.gaierror: [Errno 7] No address associated with hostname
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/data/user/0/org.python.testbed/files/python/lib/python3.15/test/test_urllib2.py", line 815, in test_ftp_error
urlopen("ftp://www.pythontest.net/")
~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/user/0/org.python.testbed/files/python/lib/python3.15/urllib/request.py", line 489, in open
response = self._open(req, data)
File "/data/user/0/org.python.testbed/files/python/lib/python3.15/urllib/request.py", line 506, in _open
result = self._call_chain(self.handle_open, protocol, protocol +
'_open', req)
File "/data/user/0/org.python.testbed/files/python/lib/python3.15/urllib/request.py", line 466, in _call_chain
result = func(*args)
File "/data/user/0/org.python.testbed/files/python/lib/python3.15/urllib/request.py", line 1533, in ftp_open
raise URLError(msg)
urllib.error.URLError: <urlopen error [Errno 7] No address associated with hostname>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/data/user/0/org.python.testbed/files/python/lib/python3.15/test/test_urllib2.py", line 817, in test_ftp_error
self.assertEqual(raised.reason,
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
f"ftp error: {exception.args[0]}")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: gaierror(7, 'No address associated with hostname') != 'ftp error: 500 OOPS: cannot change directory:/nonexistent'
----------------------------------------------------------------------
...
======================================================================
ERROR: test_getcode (test.test_urllibnet.urlopenNetworkTests.test_getcode)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/data/user/0/org.python.testbed/files/python/lib/python3.15/test/test_urllibnet.py", line 107, in test_getcode
self.assertEqual(e.exception.code, 404)
^^^^^^^^^^^^^^^^
AttributeError: 'URLError' object has no attribute 'code'
----------------------------------------------------------------------
These errors first appeared on 05-08, followed by 06-05, 06-09 and 06-11. On 06-15 there was a different network-related failure. After that, it started failing in various ways in about 1 run out of every 5.
Upgrading emulator to version 36.3.10 was enough to fix the problem on [the x86_64] worker.
The aarch64 worker was already on 36.5.10 as a result of the issues in April above. There have been no macOS version updates or reboots since then, or any other changes on the machine as far as I know. mhsmith commented on Jun 19, 2026 I tried updating the emulator to the current version 36.6.11. At first, it failed to start up about half of the time, and when it did start it was recreating the snapshot every time. After removing the gradle-managed directory and allowing it to be recreated, things now appear to be normal. Testing with the following script:
( while true; do if ! python Platforms/Android test --managed maxVersion -- -c 'from test import support; print(support.open_urlresource("http://www.pythontest.net/hashlib/blake2b.txt").read()[:50])'; then exit 1; fi; done )
It passed for 15 iterations in a row, and never gave the "This may be transient, so continuing to wait" message which indicates that the snapshot is being recreated. mhsmith commented on Jun 19, 2026 After reactivating the buildbot with emulator 36.6.11, it failed 1 out of the next 7 runs. I was able to reproduce this manually by running the above script for 65 iterations on the 3.15 branch, and then again after 11 iterations. I also reproduced it by running test_ftp_error for 45 and 29 iterations, so there doesn't appear to be any difference between the two methods. In both cases, each iteration took about 20 seconds. Rebooting made no difference.
I also tried emulator 36.3.10, which has been working fine on the x86_64 buildbot for the last 6 months, but it failed to start with the same EGLContext errors as above. Rebooting made no difference.
Going back to emulator 36.6.11, I tried the system image android-35/aosp_atd, which previously failed consistently on this machine. But now, it succeeds consistently, reaching 200 iterations of the script. So I'll make a PR to switch to this image.
MetadataMetadataOS-androidinfraCI, GitHub Actions, buildbots, Dependabot, etc.CI, GitHub Actions, buildbots, Dependabot, etc.type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error DevelopmentNo branches or pull requestsIssue actions
|