Description

JacobCoffee

opened on Aug 7, 2024

Issue body actions

I'm commencing OS upgrades for the pythontest service to the latest Ubuntu LTS - which is used for some network tests in the CPython test suite (see lines referencing pythontest.net.)

No changes should be necessary on anyones end, but if anyone happens to run their tests with internet access enabled some may fail during this upgrade.

### Tasks
- [x] Host provisioned
- [x] Data migrated
- [x] Test suite tested against current `pythontest` service
- [x] Test suite tested against new `pythontest` service
- [x] Test suite tested against bogus endpoint to ensure fails indeed... fail
- [x] Old production host shut down
- [x] New production host swapped out
- [x] Test suite tested against replaced `pythontest` service
- [x] DNS records swapped in Route53

Activity

1. JacobCoffee

   self-assigned this

   on Aug 7, 2024
2. JacobCoffee commented on Aug 7, 2024

   

   JacobCoffee

   on Aug 7, 2024

   MemberAuthor

   More actions

   TTL is 7200 for the DNS entries so they will take awhile to roll over.
3. JacobCoffee commented on Aug 7, 2024

   

   JacobCoffee

   on Aug 7, 2024

   MemberAuthor

   More actions

   DNS has propagated and tests are now passing

   cpython [⇣1] via C v15.0.0-clang via  pyenv took 5s
   ➜ ./python.exe -m test -v test_urllib2net -u all -j16
   == CPython 3.14.0a0 (heads/main:5b8a6c5186, Aug 6 2024, 16:03:59) [Clang 15.0.0 (clan
   == macOS-14.5-arm64-arm-64bit-Mach-O little-endian
   == Python build: debug
   == cwd: /Users/coffee/git/internal/python/cpython/build/test_python_worker_20375æ
   == CPU count: 16
   == encodings: locale=UTF-8 FS=utf-8
   == resources: all
   
   Using random seed: 681115882
   0:00:00 load avg: 2.32 Run 1 test in parallel using 1 worker process
   0:00:05 load avg: 2.21 [1/1] test_urllib2net passed
   test_close (test.test_urllib2net.CloseSocketTest.test_close) ... ok
   test_custom_headers (test.test_urllib2net.OtherNetworkTests.test_custom_headers) ... ok
   test_file (test.test_urllib2net.OtherNetworkTests.test_file) ... ok
   test_ftp (test.test_urllib2net.OtherNetworkTests.test_ftp) ... ok
   test_redirect_url_withfrag (test.test_urllib2net.OtherNetworkTests.test_redirect_url_withfrag) ... ok
   test_sites_no_connection_close (test.test_urllib2net.OtherNetworkTests.test_sites_no_connection_close) ... skipped 'XXX: http://www.imdb.com is gone'
   test_urlwithfrag (test.test_urllib2net.OtherNetworkTests.test_urlwithfrag) ... ok
   test_ftp_basic (test.test_urllib2net.TimeoutTest.test_ftp_basic) ... ok
   test_ftp_default_timeout (test.test_urllib2net.TimeoutTest.test_ftp_default_timeout) ... ok
   test_ftp_no_timeout (test.test_urllib2net.TimeoutTest.test_ftp_no_timeout) ... ok
   test_ftp_timeout (test.test_urllib2net.TimeoutTest.test_ftp_timeout) ... ok
   test_http_basic (test.test_urllib2net.TimeoutTest.test_http_basic) ... ok
   test_http_default_timeout (test.test_urllib2net.TimeoutTest.test_http_default_timeout) ... ok
   test_http_no_timeout (test.test_urllib2net.TimeoutTest.test_http_no_timeout) ... ok
   test_http_timeout (test.test_urllib2net.TimeoutTest.test_http_timeout) ... ok
   
   ----------------------------------------------------------------------
   Ran 15 tests in 5.375s
   
   OK (skipped=1)
   
   == Tests result: SUCCESS ==
   
   1 test OK.
   
   Total duration: 5.6 sec
   Total tests: run=15 skipped=1
   Total test files: run=1/1
   Result: SUCCESS
   
4. JacobCoffee

   closed this as completedon Aug 7, 2024
5. hugovk commented on Aug 7, 2024

   

   hugovk

   on Aug 7, 2024

   Member

   More actions

   Confirmed PRs are now passing, thank you for keeping the infra up-to-date!

   Reacted by Jacob Coffee
6. ned-deily commented on Aug 14, 2024

   

   ned-deily

   on Aug 14, 2024

   Member

   More actions

   We are seeing some ftp tests failing from macOS and iOS buildbots / CI with pythontest.net vsftpd apparently returning 500 OOPS: vsf_sysutil_bind errors. I think these are new errors but can't swear to it. @freakboy3742 noticed the failures with the iOS buildbots. Sounds like it might have to do with not enough passive ports or something similar. It could be that connections are taking a while to time out from some buildbot or CI instances. This is one of those things that just worked for years so perhaps the old OS configuration had tweaked something?

   Here are two examples from today:

   https://github.com/python/cpython/actions/runs/10380012215/job/28739153641
   https://buildbot.python.org/#/builders/1380/builds/1068/steps/10/logs/stdio
7. freakboy3742 commented on Aug 14, 2024

   

   freakboy3742

   on Aug 14, 2024

   Contributor

   More actions

   FWIW - I've also seen the failure on Linux as well (see 1, 2).

   There's possibly an interwoven issue with socket_helper. transient_internet; these errors all appear to be 500 service errors from an FTP service, and the helper is choking because that particular error doesn't set reason on the error, so what would be normally be marked as a resource skip is raising as an error.
8. ewdurbin commented on Aug 14, 2024

   

   ewdurbin

   on Aug 14, 2024

   Member

   More actions

   Looks like we are only opening 10 passive ports, which is the same as past hosts. Seems reasonable to just extend that further to ~100 ports.

   vsftpd.conf: https://github.com/python/psf-salt/blob/f348b8a70632f5bc238bba6fd826d5fb4f8cf6f6/salt/pythontest/config/vsftpd.conf#L35-L42
   firewall config: https://github.com/python/psf-salt/blob/f348b8a70632f5bc238bba6fd826d5fb4f8cf6f6/pillar/base/firewall/ftp.sls#L7

   Reacted by Jacob Coffee and Russell Keith-Magee
9. JacobCoffee commented on Aug 15, 2024

   

   JacobCoffee

   on Aug 15, 2024

   MemberAuthor

   More actions

   This is deployed so if you continue to see errors let me know

   Reacted by Russell Keith-Magee

Issue actions