Pinned Loading

1. tokenkit tokenkit Public

   A toolkit implementing advanced methods to transfer models and model knowledge across tokenizers.

   Python 72 5

2. zett zett Public

   Code for Zero-Shot Tokenizer Transfer

   Python 147 11

3. segment-any-text/wtpsplit segment-any-text/wtpsplit Public

   Toolkit to segment text into sentences or other semantic units in a robust, efficient and adaptable way.

   Python 1.4k 84

4. CPJKU/wechsel CPJKU/wechsel Public

   Code for WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models.

   Python 93 13

5. gerpt2 gerpt2 Public

   German small and large versions of GPT2.

   Python 21

6. nlprule nlprule Public

   A fast, low-resource Natural Language Processing and Text Correction library written in Rust.

   Rust 680 46