GeezerVison is showing github.com — Idea: New v3 opt-in serialization format to save space · Issue #366 · igbinary/igbinary
Idea: New v3 opt-in serialization format to save space #366Description
- Use version \x00\x00\x00\x03 when this is stable
- Avoid wasting space emitting key representations when
array_is_list, i.e. when array keys are consecutive integers starting from 0 (have a fallback to start emitting unexpected keys with different values, if needed)
- Allow more than one reference group to the same php value to be distinctly unserialized (e.g.
$x = new stdClass; $y = $x; igbinary_serialize([&$x, &$x, &$y, &$y]);)
(TODO: I forget the exact issue. Was that fixed already for the general case, or was the issue mixes of references and non-references)
- Shorter representations for 1-character interned strings
- Shorter representations for private properties of serialized class
\x00$classname\x00$propname (and protected properties \x00*\x00$propname) (keys for parent classes and traits would be serialized normally)
- Don't bother creating string ids for empty strings (EDIT: that's already part of the v2 design?) and 1-character strings that would already be interned in modern php versions (support reusing per-request in RINIT in igbinary itself if practical for php versions without ZSTR_CHAR guaranteed to exist)
Opt-in can be done on per-serializer settings, to allow any migrations (e.g. redis, memcached) to be done separately.
The unserializer should recognize v3 from the header
Related to phpredis/phpredis#2194
Updates (October 19):
Implementation is broken/incomplete and the v3 serialization format will be changed, found on the fork branch igbinary-v3-format
- Shorter representations for very small non-negative integers (e.g. 0..7 can be X+(0..7)) is an idea I've seen in several other binary protocol formats, useful for
['key' => 1, 'x' => 0] where integers are used instead of booleans, or just from most integers being small (e.g. counts) when averaged out over common use cases
- Still haven't implemented for serializing lists
- Still haven't finished adding a way to represent creating a new reference group to the same php object for the general case
- Haven't benchmarked this yet
Activitytricky commented on Oct 19, 2022 This seems reasonable to me.
Additionally consider specific types for short integer indexed arrays (0..3), though this should be guided by real numbers rather than a gut feeling that short lists are common. TysonAndre commented on Oct 23, 2022 One other optimization to look into would be to indicate for strings whether or not they should be added to the strings table during serialization, and use that to improve performance when unserializing.
- the serializer's map of zend_string to their ids could be changed to also track the pointer to the unsigned char where the type was stored
It may help for extremely large data, but I'm less certain of this for small objects that fit in cache (e.g. due to branch prediction costs)
- If branch prediction when unserializing affects performance, igbinary could put logic in the serializer to only perform this step if the fraction of strings that aren't referenced is above a given threshold and/or the total data size is large enough
If there's a lot of small strings in arrays that are only used once (megabytes), for example, unserializing igbinary protocol v2 currently
- requires a second pass through all of that memory to decrement the reference count of the string (worse for extremely large values, when not in the processor's cache)
- Cause larger string ids to be generated
E.g. an example use case might be storing a distinct set of object ids as serialized data in memcached ([$id1 => true, $id2 => true]) alone or as part of an object
Whether this should actually reindex the string ids would depend on the unserializer performance in practice
- e.g. add NULL instead of the zend_string when unserializing, and emit an error if the unserialize data references a string id that's labelled as not being meant for use
- The serializer could track offsets of the enum type
Whether it's worth it in practice would probably depend on how it works for normal object sizes
/* 16 */ igbinary_v3_type_string8, /**< String. */
/* 17 */ igbinary_v3_type_string8_noref, /**< String. */
/* 18 */ igbinary_v3_type_string16, /**< String. */
/* 19 */ igbinary_v3_type_string16_noref, /**< String. */
/* 1a */ igbinary_v3_type_string32, /**< String. */
/* 1b */ igbinary_v3_type_string32_noref, /**< String. */
/* 1c */ igbinary_v3_type_string64, /**< String larger than 4GB (originally, php strings had a limit of 32-bit lengths). */
/* 1d */ igbinary_v3_type_string64_noref, /**< String larger than 4GB (originally, php strings had a limit of 32-bit lengths). */
In that example, the type & 1 would indicate if null should be saved to the strings table instead
This would also reduce memory usage __unserialize accepted distinct temporary strings but didn't keep the reference to it around permanently TysonAndre commented on Oct 23, 2022 For use cases where there's a lot of strings being unserialized that only occur once, igbinary performs worse than unserialize in benchmarks I've run, e.g. https://github.com/krakjoe/apcu/pull/454#issuecomment-1288182980, I expect probably because igbinary has to perform a second pass through all the strings in the string table and free them TysonAndre commented on Oct 23, 2022 Also, when igbinary.compact_strings=off, the v3 protocol could immediately use the _noref variants of all of those string types changed the title [-]Idea: New opt-in serialization format to save space[/-] [+]Idea: New v3 opt-in serialization format to save space[/+] on Nov 1, 2022
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|