🦀 Decoder-only LLM built from scratch in pure Rust using Candle — no Python, no PyTorch. Gated DeltaNet + sparse attention, fine-grained MoE, native video/document understanding, long-horizon tool agents, quantization-aware training. Scales: Tiny (25M) to Large (1.3B).

rust machine-learning deep-learning lora quantization candle from-scratch self-learning dora ai-agents mixture-of-experts quantization-aware-training sparse-attention llm large-language-model gguf multimodal-llm decoder-only grpo thinking-llm

• Updated Aug 21, 2026
• Rust

[ICCV 2025] DONUT: A Decoder-Only Model for Trajectory Prediction

autonomous-driving trajectory-prediction motion-prediction argoverse decoder-only iccv2025

• Updated Mar 23, 2026
• Python

Efficient encoder-decoder architecture for small language models (≤1B parameters) with cross-architecture knowledge distillation and vision-language capabilities

encoder-decoder vision-and-language llm decoder-only

• Updated Feb 7, 2025
• Python

使用Decoder-only的Transformer进行时序预测，包含SwiGLU和RoPE(Rotary Positional Embedding)，Time series prediction using Decoder-only Transformer, Including SwiGLU and RoPE(Rotary Positional Embedding)

time-series pytorch transformer rope time-series-prediction decoder-only rotary-positional-embedding swiglu

• Updated Jan 25, 2024
• Python

Code for paper "Modality Plug-and-Play: Elastic Modality Adaptation in Multimodal LLMs for Embodied AI"

deep-learning multimodal embodied-ai large-language-model decoder-only modality-adaptation

• Updated Jan 19, 2024
• Python

[ACL 2025] 🔍 Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignment

multilingual evaluation embeddings evaluation-metrics cross-lingual multilingual-nlp large-language-models decoder-only

• Updated Oct 31, 2026
• HTML

SAMPO: Scale-wise Autoregression with Motion Prompt for Generative World Models

generative-model var decoder-only

• Updated Apr 5, 2026
• Python

Minimal decoder-only seq2seq pipeline with proper causal masking, teacher forcing, Ignite training loop, and checkpointed inference

nlp machine-learning deep-learning pytorch transformer seq2seq-model pytorch-implementation autoregressive-models decoder-only causal-masking

• Updated Feb 23, 2026
• Python

Clean-room GPT-2/GPT-3 implementation: tokenizers, architecture blocks, training loop with AdamW + cosine decay, CLI scripts, inference tools, and pytest suite. Covers OpenWebText-10k & WikiText-103 workflows. Designed as an academic reference for understanding and scaling decoder-only transformers

nlp transformers pytorch gpu-acceleration language-model adamw gpt2 gpt3 cosine-decay decoder-only educational-implementation

• Updated Feb 18, 2026
• Python

ViAG: A Novel Framework for Fine-tuning Answer Generation models ultilizing Encoder-Decoder and Decoder-only Transformers's architecture

meteor question-answering bart llama rouge bleu-score encoder-decoder fine-tuning answer-generation t5 plms bartpho llm bertscore instruction-tuning qlora qwen decoder-only vit5

• Updated May 26, 2025
• Python

Implementation of the GPT-2 architecture using PyTorch, trained on the TinyStories dataset. Features custom training pipelines on Modal (cloud computing) and integration with the Hugging Face ecosystem.

modal transformers pytorch english-nlp gpt2 huggingface decoder-only bpe-tokenizer tiny-stories

• Updated Jan 1, 2026
• Python

Developed and pre-trained a 20.39M-parameter Punjabi GPT-style base model from scratch, including corpus preparation, tokenizer training, benchmark evaluation, and text generation, using a cleaned Punjabi corpus and local Apple Silicon GPU acceleration.

language-model gpt2tokenizer gpt2lmheadmodel llms generative-ai punjabi-language decoder-only tiny-model

• Updated Mar 12, 2026
• Jupyter Notebook

Tiny-K3 — a 969M-parameter (297M active) decoder-only language model built from scratch, combining MLA latent attention, Stable LatentMoE with Quantile-Balancing routing, and Attention Residuals. Trained on FineWeb-Edu. Fully HuggingFace-compatible.

deep-learning ml pytorch transformer moe attention language-model mixture-of-experts huggingface pretraining decoder-only

• Updated Sep 23, 2026
• Python

Open source implementation of AMALIA: A Fully Open Large Language Model for European Portuguese

open-source machine-learning deep-learning decoder inference text-generation pytorch artificial-intelligence portugal gpt quantization rope pytorch-implementation large-language-models llm causal-language-modeling llms llm-inference decoder-only group-query-attention

• Updated Oct 10, 2026
• Python

A decoder-only transformer specializing in long-horizon reasoning, software architecture, and autonomous software engineering.

python nlp machine-learning research deep-learning tokenizer pytorch artificial-intelligence transformer model-architecture gpt language-model autonomous-agents reasoning bpe byte-pair-encoding sentencepiece llm llm-training decoder-only

• Updated Jul 29, 2026
• Python

A from-scratch implementation of a scaled-down GPT-2 model in PyTorch, trained on the Snappfood dataset for sentiment-controlled Persian text generation.

deep-learning university-project text-generation transformer course-project self-attention review-generation gpt2 positional-embedding decoder-only persian-text-generation causal-attention

• Updated Nov 2, 2025
• Python

Implementing GPT2 from scratch for learning purposes.

transformer gpt-2 decoder-only

• Updated Jun 3, 2026
• Python

Train a decoder-only GPT language model from scratch for code and math reasoning — custom 16k BPE tokenizer, streaming data pipeline, ablation studies, and hardware-aware training on Apple Silicon (MPS) and CUDA.

machine-learning deep-learning tokenizer cuda pytorch transformer code-generation gpt mps language-model from-scratch apple-silicon llm nanogpt decoder-only

• Updated Jun 25, 2026
• Python

Decoder OID Certificado

certificate decoder brazil certificates brasil certification oid decoders brasileiro brazilian-portuguese decoder-utility decoder-only decoder-only-transformer

• Updated Aug 7, 2026
• HTML

deep-learning university-project text-generation transformer course-project self-attention review-generation gpt2 positional-embedding decoder-only persian-text-generation causal-attention

• Updated Oct 13, 2026
• Python