GeezerVison is showing github.com — gh-137973: Add a non-parallel test plan to the iOS testbed project by freakboy3742 · Pull Request #138018 · python/cpython
Conversation
Modifies the iOS testbed project to add a test plan.
Adding a test plan to the project has three main benefits:
Firstly, it provides the Xcode standard location for defining test arguments. This doesn't change anything for the documented use of the iOS testbed; but for anyone familiar with Xcode, it will be less surprising that using the app's Info.plist as a way to configure test arguments.
Secondly, it allows defining an LLDB startup configuration, removing the need to manually disable signal breakpoints. This means the Xcode project will work out of the box, without the manual configuration of LLDB ignores that was previously required (and documented).
Lastly, and most importantly, it allows the default behavior of the test suite to be configured as non-parallel. The default behavior for iOS Xcode projects is to run in parallel, which means Xcode tries to launch 2 simulators; and because there will be 2 simulators, it's necessary for Xcode to clone the original simulator to provide independent copies.
The second simulator is never used for anything, because the CPython test suite is (at the Xcode level) a single XCTestCase. However, generating the clone and then launching 2 simulators obviously takes more time than just using the simulator that is ready.
However - more importantly, when running in parallel, Xcode doesn't stream test log output of the tests to the console (because parallel tests could cause stdout collisions). This required the iOS testbed to implement a moderately complex locking and simulator discovery mechanism to find the UUID of the test simulator, and then live stream the logs of that simulator so the user could view them.
So - by turning off parallelism, we can eliminate all that complexity, and just run the Xcode project as-is, and test output will appear in the standard Xcode output.
For added bonus points - the simulator locking and discovery logic is the cause of the issues with the recent 20250811 macos-15 image update on GitHub Actions - so by massively simplifying things, we can also restore service on GitHub Actions.
For more added bonus points - if the system running the tests is under load, the separate log streaming approach can drop log messages. This doesn't happen if Xcode displays the test logs directly.
📚 Documentation preview 📚: https://cpython-previews--138018.org.readthedocs.build/
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
ned-deily
left a comment
•
edited
Loading
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I did not do a detailed review of each change but, at a quick glance, I didn't see any obvious problems. With these changes, I found my standard make testios setup to run without any issues (which wasn't always the case with iOS simulators) and the wall clock time from start to finish seemed to be significantly less. So looking good to me!
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
Thanks @freakboy3742 for the PR 🌮🎉.. I'm working now to backport this PR to: 3.13, 3.14.
🐍🍒⛏🤖
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
miss-islington
pushed a commit
to miss-islington/cpython
that referenced
this pull request
Aug 22, 2025
…ect (pythonGH-138018)
Modifies the iOS testbed project to add a test plan. This simplifies the iOS
test runner, as we can now use the built-in log streaming to see test results.
It also allows for some other affordances, like providing a default LLDB config,
and using a standardized mechanism for specifying test arguments.
(cherry picked from commit 2ba2287)
Co-authored-by: Russell Keith-Magee <[email protected]>
miss-islington
pushed a commit
to miss-islington/cpython
that referenced
this pull request
Aug 22, 2025
…ect (pythonGH-138018)
Modifies the iOS testbed project to add a test plan. This simplifies the iOS
test runner, as we can now use the built-in log streaming to see test results.
It also allows for some other affordances, like providing a default LLDB config,
and using a standardized mechanism for specifying test arguments.
(cherry picked from commit 2ba2287)
Co-authored-by: Russell Keith-Magee <[email protected]>
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️
Hi! The buildbot iOS ARM64 Simulator 3.x (tier-3) has failed when building commit 2ba2287.
What do you need to do:
- Don't panic.
- Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
- Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1380/builds/4442) and take a look at the build logs.
- Check if the failure is related to this commit (2ba2287) or if it is a false positive.
- If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.
You can take a look at the buildbot page here:
https://buildbot.python.org/#/builders/1380/builds/4442
Summary of the results of the build (if available):
Click to see traceback logs
remote: Enumerating objects: 26, done.
remote: Counting objects: 16% (1/6)
remote: Counting objects: 33% (2/6)
remote: Counting objects: 50% (3/6)
remote: Counting objects: 66% (4/6)
remote: Counting objects: 83% (5/6)
remote: Counting objects: 100% (6/6)
remote: Counting objects: 100% (6/6), done.
remote: Compressing objects: 16% (1/6)
remote: Compressing objects: 33% (2/6)
remote: Compressing objects: 50% (3/6)
remote: Compressing objects: 66% (4/6)
remote: Compressing objects: 83% (5/6)
remote: Compressing objects: 100% (6/6)
remote: Compressing objects: 100% (6/6), done.
remote: Total 26 (delta 0), reused 0 (delta 0), pack-reused 20 (from 1)
From https://github.com/python/cpython
* branch main -> FETCH_HEAD
Note: switching to '2ba2287b85eea3cc3a71d77c6bcf9eb5670ca05d'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:
git switch -c <new-branch-name>
Or undo this operation with:
git switch -
Turn off this advice by setting config variable advice.detachedHead to false
HEAD is now at 2ba2287b85e gh-137973: Add a non-parallel test plan to the iOS testbed project (#138018)
Switched to and reset branch 'main'
configure: WARNING: no system libmpdec found; falling back to pure-Python version for the decimal module
configure: WARNING: pkg-config is missing. Some dependencies may not be detected correctly.
configure: WARNING: no system libmpdec found; falling back to pure-Python version for the decimal module
configure: WARNING: pkg-config is missing. Some dependencies may not be detected correctly.
ld: warning: ignoring duplicate libraries: '-lm'
ld: warning: ignoring duplicate libraries: '-lm'
--- xcodebuild: WARNING: Using the first of multiple matching destinations:
{ platform:iOS Simulator, id:4A7F2ACB-DEB3-48C5-83AB-082A7477C83B, OS:18.2, name:iPhone SE (3rd generation) }
{ platform:iOS Simulator, id:4A7F2ACB-DEB3-48C5-83AB-082A7477C83B, OS:18.2, name:iPhone SE (3rd generation) }
make: *** [testios] Error 65
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
freakboy3742
added a commit
that referenced
this pull request
Aug 22, 2025
…ject (GH-138018) (#138039)
Modifies the iOS testbed project to add a test plan. This simplifies the iOS
test runner, as we can now use the built-in log streaming to see test results.
It also allows for some other affordances, like providing a default LLDB config,
and using a standardized mechanism for specifying test arguments.
(cherry picked from commit 2ba2287)
Co-authored-by: Russell Keith-Magee <[email protected]>
⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️
Hi! The buildbot iOS ARM64 Simulator 3.x (tier-3) has failed when building commit 2ba2287.
What do you need to do:
- Don't panic.
- Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
- Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1380/builds/4444) and take a look at the build logs.
- Check if the failure is related to this commit (2ba2287) or if it is a false positive.
- If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.
You can take a look at the buildbot page here:
https://buildbot.python.org/#/builders/1380/builds/4444
Summary of the results of the build (if available):
Click to see traceback logs
Note: switching to '2ba2287b85eea3cc3a71d77c6bcf9eb5670ca05d'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:
git switch -c <new-branch-name>
Or undo this operation with:
git switch -
Turn off this advice by setting config variable advice.detachedHead to false
HEAD is now at 2ba2287b85e gh-137973: Add a non-parallel test plan to the iOS testbed project (#138018)
Switched to and reset branch 'main'
configure: WARNING: no system libmpdec found; falling back to pure-Python version for the decimal module
configure: WARNING: pkg-config is missing. Some dependencies may not be detected correctly.
configure: WARNING: no system libmpdec found; falling back to pure-Python version for the decimal module
configure: WARNING: pkg-config is missing. Some dependencies may not be detected correctly.
ld: warning: ignoring duplicate libraries: '-lm'
ld: warning: ignoring duplicate libraries: '-lm'
--- xcodebuild: WARNING: Using the first of multiple matching destinations:
{ platform:iOS Simulator, id:4A7F2ACB-DEB3-48C5-83AB-082A7477C83B, OS:18.2, name:iPhone SE (3rd generation) }
{ platform:iOS Simulator, id:4A7F2ACB-DEB3-48C5-83AB-082A7477C83B, OS:18.2, name:iPhone SE (3rd generation) }
make: *** [testios] Error 65
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️
Hi! The buildbot iOS ARM64 Simulator 3.13 (tier-3) has failed when building commit a73515e.
What do you need to do:
- Don't panic.
- Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
- Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1386/builds/1550) and take a look at the build logs.
- Check if the failure is related to this commit (a73515e) or if it is a false positive.
- If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.
You can take a look at the buildbot page here:
https://buildbot.python.org/#/builders/1386/builds/1550
Summary of the results of the build (if available):
Click to see traceback logs
Note: switching to 'a73515e7475c7abc7614c9f59d497af5b8709585'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:
git switch -c <new-branch-name>
Or undo this operation with:
git switch -
Turn off this advice by setting config variable advice.detachedHead to false
HEAD is now at a73515e7475 [3.13] gh-137973: Add a non-parallel test plan to the iOS testbed project (GH-138018) (#138039)
Switched to and reset branch '3.13'
configure: WARNING: no system libmpdec found; falling back to bundled libmpdec (deprecated and scheduled for removal in Python 3.16)
configure: WARNING: pkg-config is missing. Some dependencies may not be detected correctly.
configure: WARNING: no system libmpdec found; falling back to bundled libmpdec (deprecated and scheduled for removal in Python 3.16)
configure: WARNING: pkg-config is missing. Some dependencies may not be detected correctly.
--- xcodebuild: WARNING: Using the first of multiple matching destinations:
{ platform:iOS Simulator, id:4A7F2ACB-DEB3-48C5-83AB-082A7477C83B, OS:18.2, name:iPhone SE (3rd generation) }
{ platform:iOS Simulator, id:4A7F2ACB-DEB3-48C5-83AB-082A7477C83B, OS:18.2, name:iPhone SE (3rd generation) }
make: *** [testios] Error 65
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
The buildbot failures are due to concurrency issues; I've modified the buildbot config to limit concurrent builds until I can develop a more permanent solution.
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
freakboy3742
added a commit
to freakboy3742/cpython
that referenced
this pull request
Feb 3, 2026
…ed project (pythonGH-138018) (python#138039)
Modifies the iOS testbed project to add a test plan. This simplifies the iOS
test runner, as we can now use the built-in log streaming to see test results.
It also allows for some other affordances, like providing a default LLDB config,
and using a standardized mechanism for specifying test arguments.
(cherry picked from commit 2ba2287)
Co-authored-by: Russell Keith-Magee <[email protected]>
freakboy3742
added a commit
to freakboy3742/cpython
that referenced
this pull request
Feb 3, 2026
…ed project (pythonGH-138018) (python#138039)
Modifies the iOS testbed project to add a test plan. This simplifies the iOS
test runner, as we can now use the built-in log streaming to see test results.
It also allows for some other affordances, like providing a default LLDB config,
and using a standardized mechanism for specifying test arguments.
(cherry picked from commit 2ba2287)
Co-authored-by: Russell Keith-Magee <[email protected]>
freakboy3742
added a commit
to freakboy3742/cpython
that referenced
this pull request
Feb 3, 2026
…ed project (pythonGH-138018) (python#138039)
Modifies the iOS testbed project to add a test plan. This simplifies the iOS
test runner, as we can now use the built-in log streaming to see test results.
It also allows for some other affordances, like providing a default LLDB config,
and using a standardized mechanism for specifying test arguments.
(cherry picked from commit 2ba2287)
Co-authored-by: Russell Keith-Magee <[email protected]>
freakboy3742
added a commit
to freakboy3742/cpython
that referenced
this pull request
Jun 12, 2026
…ed project (pythonGH-138018) (python#138039)
Modifies the iOS testbed project to add a test plan. This simplifies the iOS
test runner, as we can now use the built-in log streaming to see test results.
It also allows for some other affordances, like providing a default LLDB config,
and using a standardized mechanism for specifying test arguments.
(cherry picked from commit 2ba2287)
Co-authored-by: Russell Keith-Magee <[email protected]>
freakboy3742
added a commit
to freakboy3742/cpython
that referenced
this pull request
Jun 12, 2026
…ed project (pythonGH-138018) (python#138039)
Modifies the iOS testbed project to add a test plan. This simplifies the iOS
test runner, as we can now use the built-in log streaming to see test results.
It also allows for some other affordances, like providing a default LLDB config,
and using a standardized mechanism for specifying test arguments.
(cherry picked from commit 2ba2287)
Co-authored-by: Russell Keith-Magee <[email protected]>
freakboy3742
added a commit
to freakboy3742/cpython
that referenced
this pull request
Oct 2, 2026
…ed project (pythonGH-138018) (python#138039)
Modifies the iOS testbed project to add a test plan. This simplifies the iOS
test runner, as we can now use the built-in log streaming to see test results.
It also allows for some other affordances, like providing a default LLDB config,
and using a standardized mechanism for specifying test arguments.
(cherry picked from commit 2ba2287)
Co-authored-by: Russell Keith-Magee <[email protected]>
freakboy3742
added a commit
to freakboy3742/cpython
that referenced
this pull request
Oct 2, 2026
…ed project (pythonGH-138018) (python#138039)
Modifies the iOS testbed project to add a test plan. This simplifies the iOS
test runner, as we can now use the built-in log streaming to see test results.
It also allows for some other affordances, like providing a default LLDB config,
and using a standardized mechanism for specifying test arguments.
(cherry picked from commit 2ba2287)
Co-authored-by: Russell Keith-Magee <[email protected]>
freakboy3742
added a commit
to freakboy3742/cpython
that referenced
this pull request
Oct 12, 2026
…ed project (pythonGH-138018) (python#138039)
Modifies the iOS testbed project to add a test plan. This simplifies the iOS
test runner, as we can now use the built-in log streaming to see test results.
It also allows for some other affordances, like providing a default LLDB config,
and using a standardized mechanism for specifying test arguments.
(cherry picked from commit 2ba2287)
Co-authored-by: Russell Keith-Magee <[email protected]>
Uh oh!
There was an error while loading. Please reload this page.
Assignees
No one assigned
Milestone
No milestone
Development
Successfully merging this pull request may close these issues.
Uh oh!
There was an error while loading. Please reload this page.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
|