Release 2.21.1

Bug Fixes and Other Changes

• Exclude stubdata and i18n BUILD.bazel from ICU 77 repository archive
• Fix ICU Unicode normalization data symbol name and link configuration in TF-Text (#1548)
• Update versions for 2.21.1 release (#1551)

Thanks to our Contributors

This release contains contributions from many people at Google, as well as:

C. Antonio Sánchez

Release 2.21.0

Bug Fixes and Other Changes

• Update TensorFlow Text WORKSPACE for latest Xcode.
• Integrate rules_ml_toolchain for hermetic CUDA rules.
• Remove now incorrect use_simple_allocator option.
• Upgrade TensorFlow to 2.21.0 and fix build dependencies (#1528)
• Update TensorFlow Text build scripts for newer manylinux
• Register ARM64 toolchain in WORKSPACE to fix build errors
• Reduce TF-Text wheel size by filtering ICU 77 data and stripping binaries.

Thanks to our Contributors

This release contains contributions from many people at Google, as well as:

C. Antonio Sánchez

Release 2.20.1

Bug Fixes and Other Changes

• Fixed a memory safety bug in FastWordpieceTokenizer concerning StringVocab lifetime. This prevents temporary copies that were previously invalidating std::string_view references to internal vocabulary data, ensuring memory stability during tokenization.
• Update Version for 2.20.1

Thanks to our Contributors

This release contains contributions from many people at Google, as well as:

C. Antonio Sánchez

Release 2.20.0

Bug Fixes and Other Changes

• Fix nightly builds.
• iterate over Unicode White_Space directly, rather than testing each of 1.1M code points
• Change input size from int16_t to int32_t to support large inputs.
• Ignore SentencePiece::BYTE during encoding instead of throwing error
• Fix TF-text for TF 2.20.
• Update requirements with latest tensorflow-metadata.
• Fix configure to pull bazelrc from TF 2.20.
• update sha256 for tensorflow 2.20.0
• Revert "update sha256 for tensorflow 2.20.0"
• Change strip_prefix logic to grab sha256 properly
• Limit array-record dependency to Linux x86_64 and aarch64.
• Pin array-record to 0.4.1 for darwin platform
• Adds build_bazel_apple_support for newer macOS builds

Bug Fixes and Other Changes

• Add docker build scripts and enable aarch64 pip wheels.
• Regenerate stubs with Mypy 1.13.0
• Handle the punctuation definition mismatch between different Unicode versions.
• Replace outdated select() on --cpu in core/kernels/sentencepiece/BUILD with platform API equivalent.
• cleanup of deprecated test methods
• Update tensorflow version.
• Limit dm-tree to 0.1.8.
• Remove srcs_version and python_version attributes, as they already default to "PY3"
• Remove invalid public_names_test
• Fix protobuf dependency.
• Update test files for new ICU version
• Init python_wheel_version_suffix_repository for TF dependency in tf-text project after tensorflow/tensorflow@805775f.
• Add numpy py dep and fix python dependencies.
• Add explicit tf-keras dependency.
• Update Version for 2.19.0-rc0
• Revert "Init python_wheel_version_suffix_repository for TF dependency in tf-text project after tensorflow/tensorflow@805775f."
• Refresh requirement lockfiles
• add @absl_py//absl:app dep to tensorflow_build_info
• Update Version for 2.19.0

Thanks to our Contributors

This release contains contributions from many people at Google, as well as:

C. Antonio Sánchez

Bug Fixes and Other Changes

• Add docker build scripts and enable aarch64 pip wheels.
• Regenerate stubs with Mypy 1.13.0
• Handle the punctuation definition mismatch between different Unicode versions.
• Replace outdated select() on --cpu in core/kernels/sentencepiece/BUILD with platform API equivalent.
• cleanup of deprecated test methods
• Update tensorflow version.
• Limit dm-tree to 0.1.8.
• Remove srcs_version and python_version attributes, as they already default to "PY3"
• Remove invalid public_names_test
• Fix protobuf dependency.
• Update test files for new ICU version
• Init python_wheel_version_suffix_repository for TF dependency in tf-text project after tensorflow/tensorflow@805775f.
• Add numpy py dep and fix python dependencies.
• Add explicit tf-keras dependency.
• Update Version for 2.19.0-rc0
• Revert "Init python_wheel_version_suffix_repository for TF dependency in tf-text project after tensorflow/tensorflow@805775f."
• Refresh requirement lockfiles
• add @absl_py//absl:app dep to tensorflow_build_info

Thanks to our Contributors

This release contains contributions from many people at Google, as well as:

C. Antonio Sánchez

Major Features and Improvements

• Add Python 3.12 and aarch64 support

Bug Fixes and Other Changes

• Add docker build scripts and enable aarch64 pip wheels.
• Replace std::string_view with absl::string_view
• Protect the fast wordpiece tokenizer from infinite looping.
• Update lock files
• Update version for 2.18.1
• Set HERMETIC_PYTHON_VERSION correctly

Release 2.18.0

Bug Fixes and Other Changes

• Fix out-of-bounds read in whitespace tokenizer
• Add unit test for fixed bounds check in IsWhitespace
• Add Hermetic CUDA rules.
• Remove tf/lite/kernels/shim:tf_headers from tf/core:framework
• Update version
• Update configure.sh

Release 2.18.0-rc0

Bug Fixes and Other Changes

• Fix out-of-bounds read in whitespace tokenizer
• Add unit test for fixed bounds check in IsWhitespace
• Add Hermetic CUDA rules.
• Remove tf/lite/kernels/shim:tf_headers from tf/core:framework
• Update version
• Update configure.sh

Release 2.17.0

Bug Fixes and Other Changes

• negative sampling excludes positive class
• revert html encoding
• much faster set-intersection based version
• Fix notebook failure with Keras 3.
• Remove tensorflow-macos from setup.py
• Update tensorflow-macos to 2.16.1
• Update version

Thanks to our Contributors

This release contains contributions from many people at Google, as well as:

Alex Shroyer, C. Antonio Sánchez, Maggie Zhang