GPT-2-style LLM built from scratch in C/CUDA with hand-written backprop, BPE tokenizer, FlashAttention, pretraining, and SFT.

c nlp training machine-learning deep-learning neural-network openmp cuda cublas language-model from-scratch byte-pair-encoding gpt2 llm bpe-tokenizer flashattention trasformer

• Updated Jun 18, 2026
• Cuda

Go BPE tokenizer (Encoder+Decoder) for GPT2 and GPT3

go encoder decoder tokenizer transformer openai token codex bpe byte-pair-encoding gpt-2 gpt-3

• Updated Dec 2, 2024
• Go

Pre-trained subword embeddings in 275 languages, based on Byte-Pair Encoding.

multilingual python nlp word-embeddings embeddings byte-pair-encoding

• Updated Nov 5, 2026
• Python

A ridiculously fast Python BPE (Byte Pair Encoder) implementation written in Rust

python rust openai pypi-package bpe byte-pair-encoding huggingface tokenizers llm tiktoken bpe-tokenizer byte-pair-tokenizer

• Updated Mar 19, 2025
• Python

Kotlin multiplatform BPE tokenizer library for OpenAI models

kotlin tokenizer openai gpt bpe byte-pair-encoding tiktoken binary-p

• Updated Mar 8, 2026
• Kotlin

Python/Cython implementations and experimental data for the Pruned BPE and DH-BPE publications.

tokenizer tokenization bpe byte-pair-encoding subword-tokenization pre-tokenization pruned-bpe vocabulary-pruning dh-bpe minimum-token-dp-encoding

• Updated Sep 10, 2026
• Python

670 MB/s Swift tokenizer inspired by Marcel Roed’s gigatoken. Zero-copy, NEON/CRC32, WebAssembly, and Embedded Swift.

swift high-performance neon tokenizer zero-copy swift-package-manager webassembly wasm simd arm64 tokenization core-ai bpe byte-pair-encoding large-language-models llm tiktoken mlx-swift embedded-swift gigatoken

• Updated Jul 31, 2026
• Swift

An educational Python project for learning tokenization step by step by building character-level, byte-level, and BPE tokenizers from scratch.

python nlp docker tokenizer text-processing tokenization educational-project bpe byte-pair-encoding regextokenizer llm subword-tokenization bpe-tokenizer byte-pair-tokenizer char-tokeneizer

• Updated Oct 27, 2026
• Python

High performance unsupervised text tokenization for Ruby

unsupervised-learning word-segmentation tokenization npl bpe byte-pair-encoding

• Updated Dec 27, 2023
• Ruby

Byte-Pair Encoding (BPE) (subword-based tokenization) algorithm implementaions from scratch with python

python nlp natural-language-processing tokenizer data-preprocessing data-cleaning bpe byte-pair-encoding subword-tokenization

• Updated Jan 30, 2023
• Python

R package for Byte Pair Encoding based on YouTokenToMe

text-mining tokenization bpe byte-pair-encoding

• Updated Jun 13, 2026
• C++

Byte-level byte pair encoding (BPE) in Haskell

haskell tokenizer bpe byte-pair-encoding llm

• Updated Sep 17, 2026
• Haskell

Feature extraction from sequential data

natural-language-processing text-classification feature-extraction word-segmentation sequential-data vgram byte-pair-encoding

• Updated Jul 4, 2019
• C++

This is a tool that encrypts a sequence of words (or pieces of texts) using the AES-256 algorithm and encodes the encrypted result into a PNG image by linking each byte value to a specific color. It also decodes the before image to get back the original sequence of words

go golang encryption tdd clean-code wasm image-processing bit-manipulation aes-256 hexagonal-architecture byte-array bip39 png-decoder solid-principles encryption-decryption cold-wallet golang-wasm byte-pair-encoding pterm go-aes-256

• Updated May 24, 2026
• Go

The fastest tokenizer for LLMs, up to 30x+ faster.

tokenizer tokenization bpe byte-pair-encoding llm bpe-tokenizer

• Updated Jul 22, 2026
• Rust

Lightweight, header-only Byte Pair Encoding (BPE) trainer in modern C++17. Produces HuggingFace-compatible vocabularies for transformers and integrates with Modern Text Tokenizer.

nlp machine-learning natural-language-processing deep-learning tokenizer modern-cpp vocabulary transformers header-only text-processing no-dependencies tokenization c17 bpe byte-pair-encoding huggingface

• Updated Aug 8, 2025
• C++

Code repo for the paper "AutoGO: Automated Computation Graph Optimization for Neural Network Evolution", accepted to NeurIPS 2023.

deep-neural-networks deep-learning tensorflow tokenizer pytorch classification human-pose-estimation super-resolution semantic-segmentation neural-architecture-search pytorch-geometric byte-pair-encoding neurips-2023

• Updated Jan 12, 2026
• Python

Byte-Pair Encoding tokenizer for training large language models on huge datasets

python machine-learning deep-learning tokenizer chunking from-scratch bpe byte-pair-encoding large-language-models llm bpe-tokenizer

• Updated Jun 4, 2024
• Python

Генерация новостных заголовков

pytorch summarization reinforce abstractive-summarization pointer-generator byte-pair-encoding summatization

• Updated Nov 21, 2022
• Python

tokenization bpe byte-pair-encoding

• Updated Oct 23, 2026
• Python