GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — test_zlib: test_pair() and test_speech128() fail with s390x hardware accelerator · Issue #90781 · python/cpython

test_zlib: test_pair() and test_speech128() fail with s390x hardware accelerator #90781

New issue

Copy link

New issue

Copy link

Closed

test_zlib: test_pair() and test_speech128() fail with s390x hardware accelerator#90781

Copy link

Labels

3.10 (EOL)end of lifeend of life3.11only security fixesonly security fixes3.9 (EOL)end of lifeend of lifetestsTests in the Lib/test dirTests in the Lib/test dir

Description

vstinner

opened

on Feb 3, 2022

Issue body actions

BPO	46623
Nosy	@vstinner, @encukou
PRs	bpo-46623: Skip two test_zlib tests on s390x #31096

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2022-02-03.08:34:40.152>
labels = ['tests', '3.9', '3.10', '3.11']
title = 'test_zlib: test_pair() and test_speech128() fail with s390x hardware accelerator'
updated_at = <Date 2022-03-03.13:20:48.694>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2022-03-03.13:20:48.694>
actor = 'petr.viktorin'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Tests']
creation = <Date 2022-02-03.08:34:40.152>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 46623
keywords = ['patch']
message_count = 6.0
messages = ['412426', '413866', '413939', '413940', '413941', '414437']
nosy_count = 2.0
nosy_names = ['vstinner', 'petr.viktorin']
pr_nums = ['31096']
priority = 'normal'
resolution = None
stage = 'resolved'
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue46623'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

Activity

vstinner commented on Feb 3, 2022
vstinner
on Feb 3, 2022
MemberAuthor
More actions
test_pair() and test_speech128() tests of test_zlib fail on the s390x architecture if zlib uses the s390x hardware accelerator.

RHEL8 downstream issues (with most details):
https://bugzilla.redhat.com/show_bug.cgi?id=1974658

Fedora downstream issues:
https://bugzilla.redhat.com/show_bug.cgi?id=2038848

The s390x has a hardware accelerator for zlib. Depending if the hardware accelerator is used or not, the output (compress data) is different.

Also, test_zlib compress data in two different ways and then expect the same output. test_zlib pass with the software implementation which creates a single (final) compressed block. test_zlib fails with the hardware implementation which creates multiple compressed blocks (the last one is a final block).

Another reason the output differs is the FHT/DHT heuristic. The zlib deflate algorithm can analyze the data distribution and decide whether it wants to use a fixed-Huffman table (FHT) or a dynamic-Huffman table (DHT) for the next block, but the accelerator can't. Furthermore, looking at data in software would kill the accelerator performance. Therefore the following heuristic is used on s390x: the first 4k are compressed with FHT and the rest of the data with DHT. So, compress() creates a single FHT block. compressobj() creates a FHT block, a DHT block and a trailing block.

It is *not a bug* in zlib: the decompression gives back the original content as expected in all cases. The issue is that Python test_zlib makes too many assumptions on how "streamed" data should be compressed. The test expected that compressed data using different ways to call zlib would return the exact same compressed data. If an accelarator is used, it's not always the case.
vstinner
added
3.9 (EOL)end of lifeend of life

3.10 (EOL)end of lifeend of life

3.11only security fixesonly security fixes

testsTests in the Lib/test dirTests in the Lib/test dir
on Feb 3, 2022
vstinner commented on Feb 23, 2022
vstinner
on Feb 23, 2022
MemberAuthor
More actions
I checked os.uname() value on two buildbots, by looking at their test.pythoninfo:

os.uname: posix.uname_result(sysname='Linux', nodename='ztcpip3.pok.ibm.com', release='3.10.0-1160.53.1.el7.s390x', version='#1 SMP Thu Dec 16 04:33:52 EST 2021', machine='s390x')

os.uname: posix.uname_result(sysname='Linux', nodename='dje', release='5.10.0-11-s390x', version='#1 SMP Debian 5.10.92-1 (2022-01-18)', machine='s390x')

So the machine is 's390x'.
vstinner commented on Feb 24, 2022
vstinner
on Feb 24, 2022
MemberAuthor
More actions
New changeset 9475dc0 by Victor Stinner in branch 'main':
bpo-46623: Skip two test_zlib tests on s390x (GH-31096)
9475dc0
vstinner commented on Feb 24, 2022
vstinner
on Feb 24, 2022
MemberAuthor
More actions
Even if Python 3.9 and 3.10 are also affected, I prefer to not backport the change since it's not ideal.
vstinner
closed this as completed on Feb 24, 2022
vstinner
closed this as completed on Feb 24, 2022
vstinner commented on Feb 24, 2022
vstinner
on Feb 24, 2022
MemberAuthor
More actions
The Python 3.7 package on Fedora also skips these two tests using --ignore option of regrtest:

%ifarch s390x --ignore test_speech128 \\ --ignore test_pair \\ %endif

https://src.fedoraproject.org/rpms/python3.7/pull-request/38#request_diff

https://src.fedoraproject.org/rpms/python3.7/blob/rawhide/f/python3.7.spec#_1168
7 remaining items
vstinner
moved this to Done in Compression issues 🗜on Nov 3, 2022
mcepl
added 9 commits that reference this issue on Apr 3, 2024
skip two test_zlib tests failing on s390x
d76694a
skip two test_zlib tests failing on s390x
d16c9c7
skip two test_zlib tests failing on s390x
d1a0513
skip two test_zlib tests failing on s390x
c292258
skip two test_zlib tests failing on s390x
ca5fc03
skip two test_zlib tests failing on s390x
7344fdd
skip two test_zlib tests failing on s390x
d0d3226
skip two test_zlib tests failing on s390x
2873423
skip two test_zlib tests failing on s390x
933eff0
encukou
added 2 commits that reference this issue on Oct 16, 2024
pythongh-125041: pythongh-90781: test_zlib: For s390x HW acceleration…
d522856
[3.9] pythongh-125041: pythongh-90781: test_zlib: For s390x HW accele…
21bf2ba
pablogsal
added a commit that references this issue on Oct 22, 2024
[3.10] gh-125041: gh-90781: test_zlib: For s390x HW acceleration, ski…
3c63d80
ambv
added a commit that references this issue on Oct 28, 2024
[3.9] gh-125041: gh-90781: test_zlib: For s390x HW acceleration, skip…
bbe8756
mcepl
added a commit that references this issue on Jun 26, 2025
skip two tests failing on s390x
5783e07

Metadata

Assignees

No one assigned

Labels

3.10 (EOL)end of lifeend of life3.11only security fixesonly security fixes3.9 (EOL)end of lifeend of lifetestsTests in the Lib/test dirTests in the Lib/test dir

Projects

Compression issues 🗜
Status
Done
Show more project fields

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions

You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.

GeezerVison

test_zlib: test_pair() and test_speech128() fail with s390x hardware accelerator #90781

Description

Activity

vstinner commented on Feb 3, 2022

vstinner commented on Feb 23, 2022

vstinner commented on Feb 24, 2022

vstinner commented on Feb 24, 2022

vstinner commented on Feb 24, 2022

7 remaining items

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions