GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — GitHub - Blosc/python-blosc2: A high-performance library for compressed ND arrays and columnar tables, with compute and indexing engines

BranchesTags
Open more actions menu

Latest commit

 

History

5,786 Commits
5,786 Commits

Folders and files

NameName
Last commit message
Last commit date
.devcontainer
.devcontainer
 
 
.github
.github
 
 
.guix/modules
.guix/modules
 
 
.skills/wasm32-pyodide-dev
.skills/wasm32-pyodide-dev
 
 
bench
bench
 
 
cmake
cmake
 
 
doc
doc
 
 
examples
examples
 
 
images
images
 
 
plans
plans
 
 
scripts
scripts
 
 
src/blosc2
src/blosc2
 
 
tests
tests
 
 
todo
todo
 
 
tools
tools
 
 
.gitattributes
.gitattributes
 
 
.gitignore
.gitignore
 
 
.guix-channel
.guix-channel
 
 
.pre-commit-config.yaml
.pre-commit-config.yaml
 
 
ADD_LAZYFUNCS.md
ADD_LAZYFUNCS.md
 
 
AGENTS.md
AGENTS.md
 
 
ANNOUNCE.rst
ANNOUNCE.rst
 
 
CMakeLists.txt
CMakeLists.txt
 
 
CONTRIBUTING.rst
CONTRIBUTING.rst
 
 
LICENSE.txt
LICENSE.txt
 
 
ONBOARDING.md
ONBOARDING.md
 
 
README.rst
README.rst
 
 
README_DEVELOPERS.md
README_DEVELOPERS.md
 
 
RELEASE_NOTES.md
RELEASE_NOTES.md
 
 
RELEASING.rst
RELEASING.rst
 
 
ROADMAP-TO-4.0.md
ROADMAP-TO-4.0.md
 
 
code_of_conduct.md
code_of_conduct.md
 
 
conftest.py
conftest.py
 
 
generate_version.py
generate_version.py
 
 
guix.scm
guix.scm
 
 
menudet-vs-numba-first-call.md
menudet-vs-numba-first-call.md
 
 
pyproject.toml
pyproject.toml
 
 
pytest.ini
pytest.ini
 
 
update_version.py
update_version.py
 
 

Repository files navigation

Python-Blosc2

A fast & compressed ndarray library with a flexible compute engine

Author:The Blosc development team
Contact:[email protected]
Github:https://github.com/Blosc/python-blosc2
Actions:actions
PyPi:version
NumFOCUS:numfocus
Code of Conduct:Contributor Covenant

What is Python-Blosc2?

Python-Blosc2 is a high-performance compressor, compute engine, and format for binary data containers that are portable and open-source. It comes with a lazy expression engine allowing for complex calculations on compressed data, whether stored in memory, on disk, or over the network (e.g., via Caterva2). It is especially optimized for storing and retrieving data from N-dimensional arrays (NDArray) and columnar tables (CTable), complemented by a query/indexing layer. The main use case is fast, compressed, out-of-core numerical data — especially when data is too large to fit comfortably in RAM.

C-Blosc2 is used under the hood as its compression backend. Written in C, and building on its predecessor C-Blosc, C-Blosc2 aims to be an extremely fast meta-compressor for binary data, supporting a diverse set of strategies, and with an extensible plugin architecture for a wide range of codecs and filters.

miniexpr is embedded as the compute engine backend. It delivers fast multithreaded evaluation for lazy expressions directly over compressed chunks, and powers Menudet, a little language for portable, JIT-accelerated array (and table columns) kernels that can be serialized on a architecture-independent way.

More info: https://www.blosc.org/python-blosc2/getting_started/overview.html

Installing

Binary packages are available for major OSes (Win, Mac, Linux) and platforms. Install from PyPI using pip:

pip install blosc2 --upgrade

Conda users can install from conda-forge:

conda install -c conda-forge python-blosc2

Command line tools

Two CLI tools are installed along with the package:

  • b2view: an interactive terminal browser (TUI) for TreeStore bundles (.b2d directories or .b2z files), with paged views of NDArray and CTable data of any size (walkthrough; requires pip install "blosc2[tui]").
  • parquet-to-blosc2: converts Parquet files to Blosc2 columnar table stores, and back (walkthrough; requires pip install "blosc2[parquet]").

Documentation

The documentation is available here:

https://blosc.org/python-blosc2/python-blosc2.html

You can find examples at:

https://github.com/Blosc/python-blosc2/tree/main/examples

A tutorial from PyData Global 2025 is available at:

https://github.com/Blosc/PyData-Global-2025-Tutorial

(Click here to watch the video recording of the tutorial)

It contains Jupyter notebooks explaining the main features of Python-Blosc2.

License

This software is licensed under a 3-Clause BSD license. A copy of the python-blosc2 license can be found in LICENSE.txt.

Discussion forum

Discussion about this package is welcome at:

https://github.com/Blosc/python-blosc2/discussions

Social feeds

Stay informed about the latest developments by following us in Mastodon, Bluesky or LinkedIn.

Thanks

Blosc2 is supported by the NumFOCUS foundation, the LEAPS-INNOV project and ironArray SLU, among many other donors. This allowed the following people to have contributed in an important way to the core development of the Blosc2 library:

  • Francesc Alted
  • Marta Iborra
  • Luke Shaw
  • Aleix Alcacer
  • Oscar Guiñón
  • Juan David Ibáñez
  • Ivan Vilata i Balaguer
  • Oumaima Ech.Chdig
  • Ricardo Sales Piquer

In addition, other people have participated in the project in different aspects:

  • Jan Sellner, contributed the mmap support for NDArray/SChunk objects.
  • Dimitri Papadopoulos, contributed a large bunch of improvements to many aspects of the project. His attention to detail is remarkable.
  • And many others that have contributed with bug reports, suggestions and improvements.

Developed using JetBrains IDEs.

JetBrains logo.

Citing Blosc

You can cite our work on the various libraries under the Blosc umbrella as follows:

@ONLINE{blosc,
  author = {{Blosc Development Team}},
  title = "{A fast, compressed and persistent data store library}",
  year = {2009-2026},
  note = {https://blosc.org}
}

Support Blosc for a Sustainable Future

If you find Blosc useful and want to support its development, please consider making a donation or contract to the Blosc Development Team. Thank you!

Compress Better, Compute Bigger

About

A high-performance library for compressed ND arrays and columnar tables, with compute and indexing engines

www.blosc.org/python-blosc2

Topics

columnar-storagecompressioncomputationallazy-evaluationndimensional-arraystabular-data

Resources

Readme
License

Code of conduct

Code of conduct

Contributing

Contributing
Activity
Custom properties

Stars

215 stars

Watchers

4 watching

Forks

73 forks
Report repository

Releases

Sponsor this project

Packages

Used by

Contributors

Languages

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.