GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — tarfile: extractfile(member).read() of a sparse member is slow due to repeated bytes concatenation · Issue #160774 · python/cpython

tarfile: extractfile(member).read() of a sparse member is slow due to repeated bytes concatenation #160774

New issue
Copy link
New issue
Copy link
Open
Open
tarfile: extractfile(member).read() of a sparse member is slow due to repeated bytes concatenation#160774
Copy link
Assignees
devdavidejesus
Labels
performancePerformance or resource usagePerformance or resource usagestdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directorytriagedThe issue has been accepted as valid by a triager.The issue has been accepted as valid by a triager.type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

Description

@devdavidejesus
devdavidejesus
opened on Oct 27, 2026
Issue body actions

Bug report

_FileInFile.read() in Lib/tarfile.py builds its result with buf += b for each data segment and buf += NUL * length for each hole of a sparse member. Each bytes concatenation copies the whole accumulated buffer, so reading a sparse member with TarFile.extractfile(member).read() takes time proportional to the member size times the number of sparse segments.

Holes take no space in the archive, so a small archive is enough to make this noticeable. extract(), extractall() and reading the member in small chunks (for example with shutil.copyfileobj()) are not affected.

Reproducer

import io, tarfile, time

def build(n, blk=4096):
    # GNU sparse 0.1 (pax) member: n data blocks separated by holes.
    t = tarfile.TarInfo("GNUSparseFile.0/a.img")
    t.size = n * blk
    t.pax_headers = {
        "GNU.sparse.map": ",".join(f"{2 * i * blk},{blk}" for i in range(n)),
        "GNU.sparse.name": "a.img",
        "GNU.sparse.realsize": str(2 * n * blk),
    }
    bio = io.BytesIO()
    with tarfile.open(fileobj=bio, mode="w", format=tarfile.PAX_FORMAT) as tf:
        tf.addfile(t, io.BytesIO(b"D" * t.size))
    return bio.getvalue()

def read_time(n):
    with tarfile.open(fileobj=io.BytesIO(build(n))) as tf:
        f = tf.extractfile(tf.next())
        t0 = time.perf_counter()
        f.read()
        return time.perf_counter() - t0

for n in (1000, 2000, 4000):
    print(f"{n} segments: {read_time(n):.2f}s")

On a release build (macOS, Apple M3) the read of a sparse member with 4000 segments takes several seconds; the time roughly quadruples when the segment count doubles. With the change below each read is a fraction of a second.

Suggested fix

Accumulate the pieces in a list and join once, instead of repeated concatenation:

-        buf = b""
+        chunks = []
         while size > 0:
             ...
-                buf += b
+                chunks.append(b)
             else:
-                buf += NUL * length
+                chunks.append(NUL * length)
             size -= length
             self.position += length
-        return buf
+        return b"".join(chunks)

I have a branch with this change and a regression test that reads a multi-segment sparse member through extractfile().read() and in chunks, and compares both to the expected content. I can open a PR if this looks right.

Activity

  1. NekoAsakura
    added
    performancePerformance or resource usagePerformance or resource usage
    stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directory
    on Oct 27, 2026
  2. picnixz
    added
    type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error
    on Oct 28, 2026
  3. picnixz commented on Oct 28, 2026

    @picnixz
    picnixz
    on Oct 28, 2026
    Member
    More actions

    I believe this is a relevant change.

  4. picnixz
    added
    triagedThe issue has been accepted as valid by a triager.The issue has been accepted as valid by a triager.
    on Oct 28, 2026
  5. devdavidejesus commented on Oct 28, 2026

    @devdavidejesus
    devdavidejesus
    on Oct 28, 2026
    ContributorAuthor
    More actions

    I believe this is a relevant change.

    Roger that! I have three more PRs awaiting review. Once they’re merged, I’ll submit a PR for this issue.

  6. picnixz
    assigned
    devdavidejesus
    on Oct 29, 2026
  7. devdavidejesus
    added a commit that references this issue on Nov 6, 2026

    pythongh-160774: Speed up reading sparse tar members with extractfile…

    e126620
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

  • @devdavidejesus
    devdavidejesus

Labels

performancePerformance or resource usagePerformance or resource usagestdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directorytriagedThe issue has been accepted as valid by a triager.The issue has been accepted as valid by a triager.type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

    You are visitor 01402503 — GeezerVison does not store the pages you look up.
    Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.