GeezerVison

@vstinner I think this is ready for review if bytearray.resize seems like a reasonable feature add.

Please initialize new bytes to zero. We don't accept undefined behavior in Python.

I'm not sure that adding this API is needed since there is already a way to truncate a bytearray and to extend a bytearray using the existing API. You should run a benchmark to show that it's worth it, especially to extend a bytearray.

What is the way to extend the bytearray without copying byte by byte or having the length of the extension extra memory allocated?

What is the way to extend the bytearray without copying byte by byte or having the length of the extension extra memory allocated?

What you say. In short, ba[len(ba):] = b'\0' * extend.

Please run a benchmark to measure this compared to ba.extend().

@ZeroIntensity: It's already possible to implement extend() with the current bytearray API, it's just a convenient helper for: ba[len(ba):] = b'\0' * extend. So I don't see why other Python implementation couldn't implement it. They should already be able to resize a bytearray.

There are many ways to "extend" a bytearray. Another example:

>>> ba=bytearray(b'abc')
>>> ba += b'def'
>>> ba, len(ba)
(bytearray(b'abcdef'), 6)

Yeah, but I think that's a bit different than an explicit "resize." You can emulate it with that kind of thing, but it feels different than an actual method saying it will reallocate.

The method documentation should not promise any performance guarantee in general, it can be implemented in different ways (the implementation doesn't matter).

Can we clarify that with a CPython implementation detail note?

@ZeroIntensity can definitely add a note of "This is equivalent to bytearray += b'\0' * size". For more efficiency in JavaScript in particular I think this could map to https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/fill -- nevermind, just sets elements to 0, can't append.

(Planning to do next iteration + perf testing follow up today, yesterday got sucked into a BytesIO.readfrom(file, *, limit: int | None = None, expected: int | None = None) experiment)

test_bytes and test_capi are failing, you need to update the exception.

Working on them; Unfortunate part of "add suggestions", it adds "this person contributed" but doesn't let you do that + local changes before making push visible / alerting reviewers... Should have soon

I think I've addressed all review comments.

Tested performance on an optimized build (--with-lto --enable-optimizations)

$ python bm_bytearray_resize.py       
.....................
extend_bytes: Mean +- std dev: 142 ms +- 1 ms
.....................
extend_range: Mean +- std dev: 1.89 sec +- 0.07 sec
.....................
iadd: Mean +- std dev: 143 ms +- 1 ms
.....................
resize: Mean +- std dev: 48.7 ms +- 0.5 ms

import itertools
import pyperf

runner = pyperf.Runner()


# Aiming for what a file readall does in terms of resizing as get more data.
# Non-linear resizing up to a reaonably large read size.
blocksize = 1024
sizes = [
    blocksize,
    blocksize * 4,
    blocksize * 16,
    1_000_000,
    100_000_000,
    1_000_000_000,
    ]

def extend_bytes():
    """Extend by appending a temporary byte string"""
    ba = bytearray()
    for size in sizes:
        ba.extend(b'\0' * size)


def extend_range():
    """Resize by an iterator with length_hint"""
    ba = bytearray()
    for size in sizes:
        ba.extend(itertools.repeat(0, size))


def iadd():
    """Resize by using += """
    ba = bytearray()
    for size in sizes:
        ba += b'\0' * size


def resize():
    """Use resize"""
    ba = bytearray()
    for size in sizes:
        ba.resize(size)



runner.bench_func("extend_bytes", extend_bytes)
runner.bench_func("extend_range", extend_range)
runner.bench_func("iadd", iadd)
runner.bench_func("resize", resize)

The change mostly LGTM. I just made a new review for some details.

extend_bytes: Mean +- std dev: 142 ms +- 1 ms
extend_range: Mean +- std dev: 1.89 sec +- 0.07 sec
iadd: Mean +- std dev: 143 ms +- 1 ms
resize: Mean +- std dev: 48.7 ms +- 0.5 ms

Thanks for the benchmark. So resize is 2.9x faster than the existing fastest method to grow a bytearray (extend_bytes). So it's worth it to add a dedicated method.

Nice, I'm not surprised it is faster. Though what I care more about is that code using this is more readable than the other hacks. :)