GeezerVisonThe World Wide Web, restored to 800 pixels of glory. |
| Home|About|Gaming cabinet|Archives|How it works|Starter sites |
GeezerVison is showing github.com — tarfile: extractfile(member).read() of a sparse member is slow due to repeated bytes concatenation · Issue #160774 · python/cpython Open Labels performancePerformance or resource usagePerformance or resource usagestdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directorytriagedThe issue has been accepted as valid by a triager.The issue has been accepted as valid by a triager.type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error DescriptionBug report
Holes take no space in the archive, so a small archive is enough to make this noticeable. Reproducerimport io, tarfile, time def build(n, blk=4096): # GNU sparse 0.1 (pax) member: n data blocks separated by holes. t = tarfile.TarInfo("GNUSparseFile.0/a.img") t.size = n * blk t.pax_headers = { "GNU.sparse.map": ",".join(f"{2 * i * blk},{blk}" for i in range(n)), "GNU.sparse.name": "a.img", "GNU.sparse.realsize": str(2 * n * blk), } bio = io.BytesIO() with tarfile.open(fileobj=bio, mode="w", format=tarfile.PAX_FORMAT) as tf: tf.addfile(t, io.BytesIO(b"D" * t.size)) return bio.getvalue() def read_time(n): with tarfile.open(fileobj=io.BytesIO(build(n))) as tf: f = tf.extractfile(tf.next()) t0 = time.perf_counter() f.read() return time.perf_counter() - t0 for n in (1000, 2000, 4000): print(f"{n} segments: {read_time(n):.2f}s") On a release build (macOS, Apple M3) the read of a sparse member with 4000 segments takes several seconds; the time roughly quadruples when the segment count doubles. With the change below each read is a fraction of a second. Suggested fixAccumulate the pieces in a list and join once, instead of repeated concatenation: - buf = b"" + chunks = [] while size > 0: ... - buf += b + chunks.append(b) else: - buf += NUL * length + chunks.append(NUL * length) size -= length self.position += length - return buf + return b"".join(chunks) I have a branch with this change and a regression test that reads a multi-segment sparse member through Activity
MetadataMetadataAssigneesLabelsperformancePerformance or resource usagePerformance or resource usagestdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directorytriagedThe issue has been accepted as valid by a triager.The issue has been accepted as valid by a triager.type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error ProjectsMilestoneRelationshipsDevelopmentIssue actions |
| You are visitor 01402503 — GeezerVison does not store the pages you look up. Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required. |