GeezerVison is showing github.com — gh-109845: Make test_ftplib more stable under load by serhiy-storchaka · Pull Request #109912 · python/cpython
Conversation
serhiy-storchaka
commented
Sep 26, 2023
•
edited by bedevere-app
Bot
Loading
Uh oh!
There was an error while loading. Please reload this page.
recv() can return partial data cut in the middle of a multibyte character. Test raw binary data instead of data incorrectly decoded by parts.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
recv() can return partial data cut in the middle of a multibyte
character. Test raw binary data instead of data incorrectly decoded by parts.
|
|
# the dummy data returned by server over the data channel when
|
|
|
# RETR, LIST, NLST, MLSD commands are issued
|
|
|
RETR_DATA = 'abcde12345\r\n' * 1000 + 'non-ascii char \xAE\r\n'
|
|
|
RETR_DATA = 'abcde\xB9\xB2\xB3\xA4\xA6\r\n' * 1000
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It makes the bug easier reproducible.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
vstinner reacted with thumbs up emoji
|
I confirm that the PR fix the issue.
I can easily reproduce the issue with this script in a few seconds: #109845 (comment)
With this PR, I can no longer reproduce the issue:
vstinner@mona$ ./python -m test test_ftplib -m test_storlines -u all -j10 --forever -R 3:3
(...)
0:02:35 load avg: 9.30 [350] test_ftplib passed
beginning 6 repetitions
123456
......
^C
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
Thanks @serhiy-storchaka for the PR 🌮🎉.. I'm working now to backport this PR to: 3.11, 3.12.
🐍🍒⛏🤖
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
miss-islington
pushed a commit
to miss-islington/cpython
that referenced
this pull request
Sep 26, 2023
…9912)
recv() can return partial data cut in the middle of a multibyte
character. Test raw binary data instead of data incorrectly decoded by parts.
(cherry picked from commit 2ef2fff)
Co-authored-by: Serhiy Storchaka <[email protected]>
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
miss-islington
pushed a commit
to miss-islington/cpython
that referenced
this pull request
Sep 26, 2023
…9912)
recv() can return partial data cut in the middle of a multibyte
character. Test raw binary data instead of data incorrectly decoded by parts.
(cherry picked from commit 2ef2fff)
Co-authored-by: Serhiy Storchaka <[email protected]>
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
serhiy-storchaka
added a commit
that referenced
this pull request
Sep 26, 2023
…GH-109920)
recv() can return partial data cut in the middle of a multibyte
character. Test raw binary data instead of data incorrectly decoded by parts.
(cherry picked from commit 2ef2fff)
Co-authored-by: Serhiy Storchaka <[email protected]>
⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️
Hi! The buildbot ARM64 macOS 3.11 has failed when building commit 4e66eca.
What do you need to do:
- Don't panic.
- Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
- Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/1030/builds/1399) and take a look at the build logs.
- Check if the failure is related to this commit (4e66eca) or if it is a false positive.
- If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.
You can take a look at the buildbot page here:
https://buildbot.python.org/all/#builders/1030/builds/1399
Failed tests:
Failed subtests:
- test_access_parameter - test.test_mmap.MmapTests.test_access_parameter
Summary of the results of the build (if available):
==
Click to see traceback logs
Traceback (most recent call last):
File "/Users/buildbot/buildarea/3.11.pablogsal-macos-m1.macos-with-brew/build/Lib/multiprocessing/resource_tracker.py", line 239, in main
cache[rtype].remove(name)
KeyError: '/psm_0ec5f1cc'
Traceback (most recent call last):
File "/Users/buildbot/buildarea/3.11.pablogsal-macos-m1.macos-with-brew/build/Lib/test/test_mmap.py", line 258, in test_access_parameter
m = mmap.mmap(f.fileno(), mapsize, prot=prot)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
PermissionError: [Errno 1] Operation not permitted
Traceback (most recent call last):
File "/Users/buildbot/buildarea/3.11.pablogsal-macos-m1.macos-with-brew/build/Lib/multiprocessing/resource_tracker.py", line 239, in main
cache[rtype].remove(name)
KeyError: '/psm_786bbeff'
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️
Hi! The buildbot s390x RHEL7 LTO 3.x has failed when building commit 2ef2fff.
What do you need to do:
- Don't panic.
- Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
- Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/402/builds/5478) and take a look at the build logs.
- Check if the failure is related to this commit (2ef2fff) or if it is a false positive.
- If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.
You can take a look at the buildbot page here:
https://buildbot.python.org/all/#builders/402/builds/5478
Failed tests:
Failed subtests:
- test_freeze_simple_script - test.test_tools.test_freeze.TestFreeze.test_freeze_simple_script
Summary of the results of the build (if available):
==
Click to see traceback logs
Traceback (most recent call last):
File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z.lto/build/Lib/test/test_tools/test_freeze.py", line 32, in test_freeze_simple_script
outdir, scriptfile, python = helper.prepare(script, outdir)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z.lto/build/Tools/freeze/test/freeze.py", line 146, in prepare
copy_source_tree(srcdir, SRCDIR)
File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z.lto/build/Tools/freeze/test/freeze.py", line 95, in copy_source_tree
shutil.copytree(oldroot, newroot, ignore=ignore_non_src)
File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z.lto/build/Lib/shutil.py", line 588, in copytree
return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z.lto/build/Lib/shutil.py", line 542, in _copytree
raise Error(errors)
shutil.Error: [('/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z.lto/build/build/test_python_21697æ/@test_21697_tmpæ_dir/sym1', '/tmp/test_python_t5j05b15/tmphy1lw6tl/cpython/build/test_python_21697æ/@test_21697_tmpæ_dir/sym1', "[Errno 2] No such file or directory: '/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z.lto/build/build/test_python_21697æ/@test_21697_tmpæ_dir/sym1'"), ('/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z.lto/build/build/test_python_21697æ/@test_21697_tmpæ_dir/sym2', '/tmp/test_python_t5j05b15/tmphy1lw6tl/cpython/build/test_python_21697æ/@test_21697_tmpæ_dir/sym2', "[Errno 2] No such file or directory: '/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z.lto/build/build/test_python_21697æ/@test_21697_tmpæ_dir/sym2'")]
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️
Hi! The buildbot s390x Fedora Clang 3.x has failed when building commit 2ef2fff.
What do you need to do:
- Don't panic.
- Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
- Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/3/builds/4684) and take a look at the build logs.
- Check if the failure is related to this commit (2ef2fff) or if it is a false positive.
- If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.
You can take a look at the buildbot page here:
https://buildbot.python.org/all/#builders/3/builds/4684
Failed tests:
Failed subtests:
- test_freeze_simple_script - test.test_tools.test_freeze.TestFreeze.test_freeze_simple_script
Summary of the results of the build (if available):
==
Click to see traceback logs
Traceback (most recent call last):
File "/home/dje/cpython-buildarea/3.x.edelsohn-fedora-z.clang/build/Lib/test/test_tools/test_freeze.py", line 32, in test_freeze_simple_script
outdir, scriptfile, python = helper.prepare(script, outdir)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/dje/cpython-buildarea/3.x.edelsohn-fedora-z.clang/build/Tools/freeze/test/freeze.py", line 146, in prepare
copy_source_tree(srcdir, SRCDIR)
File "/home/dje/cpython-buildarea/3.x.edelsohn-fedora-z.clang/build/Tools/freeze/test/freeze.py", line 95, in copy_source_tree
shutil.copytree(oldroot, newroot, ignore=ignore_non_src)
File "/home/dje/cpython-buildarea/3.x.edelsohn-fedora-z.clang/build/Lib/shutil.py", line 588, in copytree
return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/dje/cpython-buildarea/3.x.edelsohn-fedora-z.clang/build/Lib/shutil.py", line 542, in _copytree
raise Error(errors)
shutil.Error: [('/home/dje/cpython-buildarea/3.x.edelsohn-fedora-z.clang/build/build/test_python_3659703æ/@test_3659708_tmpæ', '/tmp/test_python_y5lypxfd/tmpywa5a_cx/cpython/build/test_python_3659703æ/@test_3659708_tmpæ', '`/home/dje/cpython-buildarea/3.x.edelsohn-fedora-z.clang/build/build/test_python_3659703æ/@test_3659708_tmpæ` is a named pipe')]
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️
Hi! The buildbot PPC64LE RHEL8 3.11 has failed when building commit 4e66eca.
What do you need to do:
- Don't panic.
- Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
- Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/997/builds/857) and take a look at the build logs.
- Check if the failure is related to this commit (4e66eca) or if it is a false positive.
- If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.
You can take a look at the buildbot page here:
https://buildbot.python.org/all/#builders/997/builds/857
Failed tests:
Summary of the results of the build (if available):
==
Click to see traceback logs
remote: Enumerating objects: 5, done.
remote: Counting objects: 20% (1/5)
remote: Counting objects: 40% (2/5)
remote: Counting objects: 60% (3/5)
remote: Counting objects: 80% (4/5)
remote: Counting objects: 100% (5/5)
remote: Counting objects: 100% (5/5), done.
remote: Compressing objects: 20% (1/5)
remote: Compressing objects: 40% (2/5)
remote: Compressing objects: 60% (3/5)
remote: Compressing objects: 80% (4/5)
remote: Compressing objects: 100% (5/5)
remote: Compressing objects: 100% (5/5), done.
remote: Total 5 (delta 0), reused 3 (delta 0), pack-reused 0
From https://github.com/python/cpython
* branch 3.11 -> FETCH_HEAD
Note: switching to '4e66eca489a33dc81d676037a456e85c50c81123'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:
git switch -c <new-branch-name>
Or undo this operation with:
git switch -
Turn off this advice by setting config variable advice.detachedHead to false
HEAD is now at 4e66eca489 [3.11] gh-109845: Make test_ftplib more stable under load (GH-109912) (GH-109920)
Switched to and reset branch '3.11'
renaming build/scripts-3.11/pydoc3 to build/scripts-3.11/pydoc3.11
renaming build/scripts-3.11/idle3 to build/scripts-3.11/idle3.11
renaming build/scripts-3.11/2to3 to build/scripts-3.11/2to3-3.11
renaming build/scripts-3.11/pydoc3 to build/scripts-3.11/pydoc3.11
renaming build/scripts-3.11/idle3 to build/scripts-3.11/idle3.11
renaming build/scripts-3.11/2to3 to build/scripts-3.11/2to3-3.11
renaming build/scripts-3.11/pydoc3 to build/scripts-3.11/pydoc3.11
renaming build/scripts-3.11/idle3 to build/scripts-3.11/idle3.11
renaming build/scripts-3.11/2to3 to build/scripts-3.11/2to3-3.11
Timeout (0:15:00)!
Thread 0x00007fffa8724d40 (most recent call first):
File "/home/buildbot/buildarea/3.11.cstratak-RHEL8-ppc64le/build/Lib/selectors.py", line 415 in select
File "/home/buildbot/buildarea/3.11.cstratak-RHEL8-ppc64le/build/Lib/subprocess.py", line 2108 in _communicate
File "/home/buildbot/buildarea/3.11.cstratak-RHEL8-ppc64le/build/Lib/subprocess.py", line 1209 in communicate
File "/home/buildbot/buildarea/3.11.cstratak-RHEL8-ppc64le/build/Lib/test/test_gdb.py", line 108 in run_gdb
File "/home/buildbot/buildarea/3.11.cstratak-RHEL8-ppc64le/build/Lib/test/test_gdb.py", line 225 in get_stack_trace
File "/home/buildbot/buildarea/3.11.cstratak-RHEL8-ppc64le/build/Lib/test/test_gdb.py", line 1003 in test_basic_command
File "/home/buildbot/buildarea/3.11.cstratak-RHEL8-ppc64le/build/Lib/unittest/case.py", line 579 in _callTestMethod
File "/home/buildbot/buildarea/3.11.cstratak-RHEL8-ppc64le/build/Lib/unittest/case.py", line 623 in run
File "/home/buildbot/buildarea/3.11.cstratak-RHEL8-ppc64le/build/Lib/unittest/case.py", line 678 in __call__
File "/home/buildbot/buildarea/3.11.cstratak-RHEL8-ppc64le/build/Lib/unittest/suite.py", line 122 in run
File "/home/buildbot/buildarea/3.11.cstratak-RHEL8-ppc64le/build/Lib/unittest/suite.py", line 84 in __call__
File "/home/buildbot/buildarea/3.11.cstratak-RHEL8-ppc64le/build/Lib/unittest/suite.py", line 122 in run
File "/home/buildbot/buildarea/3.11.cstratak-RHEL8-ppc64le/build/Lib/unittest/suite.py", line 84 in __call__
File "/home/buildbot/buildarea/3.11.cstratak-RHEL8-ppc64le/build/Lib/unittest/suite.py", line 122 in run
File "/home/buildbot/buildarea/3.11.cstratak-RHEL8-ppc64le/build/Lib/unittest/suite.py", line 84 in __call__
File "/home/buildbot/buildarea/3.11.cstratak-RHEL8-ppc64le/build/Lib/unittest/runner.py", line 217 in run
File "/home/buildbot/buildarea/3.11.cstratak-RHEL8-ppc64le/build/Lib/test/support/__init__.py", line 1137 in _run_suite
File "/home/buildbot/buildarea/3.11.cstratak-RHEL8-ppc64le/build/Lib/test/support/__init__.py", line 1265 in run_unittest
File "/home/buildbot/buildarea/3.11.cstratak-RHEL8-ppc64le/build/Lib/test/libregrtest/runtest.py", line 298 in _test_module
File "/home/buildbot/buildarea/3.11.cstratak-RHEL8-ppc64le/build/Lib/test/libregrtest/runtest.py", line 313 in regrtest_runner
File "/home/buildbot/buildarea/3.11.cstratak-RHEL8-ppc64le/build/Lib/test/libregrtest/runtest.py", line 356 in _load_run_test
File "/home/buildbot/buildarea/3.11.cstratak-RHEL8-ppc64le/build/Lib/test/libregrtest/runtest.py", line 396 in _runtest_env_changed_exc
File "/home/buildbot/buildarea/3.11.cstratak-RHEL8-ppc64le/build/Lib/test/libregrtest/runtest.py", line 253 in _runtest_capture_output_timeout_junit
File "/home/buildbot/buildarea/3.11.cstratak-RHEL8-ppc64le/build/Lib/test/libregrtest/runtest.py", line 280 in runtest
File "/home/buildbot/buildarea/3.11.cstratak-RHEL8-ppc64le/build/Lib/test/libregrtest/main.py", line 367 in rerun_failed_tests
File "/home/buildbot/buildarea/3.11.cstratak-RHEL8-ppc64le/build/Lib/test/libregrtest/main.py", line 847 in _main
File "/home/buildbot/buildarea/3.11.cstratak-RHEL8-ppc64le/build/Lib/test/libregrtest/main.py", line 802 in main
File "/home/buildbot/buildarea/3.11.cstratak-RHEL8-ppc64le/build/Lib/test/libregrtest/main.py", line 866 in main
File "/home/buildbot/buildarea/3.11.cstratak-RHEL8-ppc64le/build/Lib/test/__main__.py", line 2 in <module>
File "/home/buildbot/buildarea/3.11.cstratak-RHEL8-ppc64le/build/Lib/runpy.py", line 88 in _run_code
File "/home/buildbot/buildarea/3.11.cstratak-RHEL8-ppc64le/build/Lib/runpy.py", line 198 in _run_module_as_main
make: *** [Makefile:1849: buildbottest] Error 1
Cannot open file '/home/buildbot/buildarea/3.11.cstratak-RHEL8-ppc64le/build/build_oot/test-results.xml' for upload
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
macOS: test_access_parameter - test.test_mmap.MmapTests.test_access_parameter
Fixed by PR #109928.
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
csm10495
pushed a commit
to csm10495/cpython
that referenced
this pull request
Sep 28, 2023
…9912)
recv() can return partial data cut in the middle of a multibyte
character. Test raw binary data instead of data incorrectly decoded by parts.
Yhg1s
pushed a commit
that referenced
this pull request
Oct 2, 2023
…#109919)
gh-109845: Make test_ftplib more stable under load (GH-109912)
recv() can return partial data cut in the middle of a multibyte
character. Test raw binary data instead of data incorrectly decoded by parts.
(cherry picked from commit 2ef2fff)
Co-authored-by: Serhiy Storchaka <[email protected]>
Glyphack
pushed a commit
to Glyphack/cpython
that referenced
this pull request
Sep 2, 2024
…9912)
recv() can return partial data cut in the middle of a multibyte
character. Test raw binary data instead of data incorrectly decoded by parts.
Reviewers
vstinner
vstinner approved these changes
Uh oh!
There was an error while loading. Please reload this page.
Assignees
No one assigned
Milestone
No milestone
Development
Successfully merging this pull request may close these issues.
Uh oh!
There was an error while loading. Please reload this page.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
|