🧑‍🏫 60+ Implementations/tutorials of deep learning papers with side-by-side notes 📝; including transformers (original, xl, switch, feedback, vit, ...), optimizers (adam, adabelief, sophia, ...), gans(cyclegan, stylegan2, ...), 🎮 reinforcement learning (ppo, dqn), capsnet, distillation, ... 🧠

machine-learning reinforcement-learning deep-learning transformers pytorch transformer gan neural-networks literate-programming attention lora deep-learning-tutorial optimizers

• Updated Nov 2, 2026
• Python

🦁 Lion, new optimizer discovered by Google Brain using genetic algorithms that is purportedly better than Adam(w), in Pytorch

deep-learning artificial-intelligence optimizers evolutionary-search

• Updated Jul 9, 2026
• Python

Nx-powered Neural Networks

elixir deep-learning neural-networks nx optimizers

• Updated Oct 8, 2026
• Elixir

🐦 Opytimizer is a Python library consisting of meta-heuristic optimization algorithms.

python optimization artificial-intelligence meta-heuristic optimizers bioinspired meta-heuristic-optimization

• Updated Oct 26, 2026
• Python

A New Optimization Technique for Deep Neural Networks

deep-neural-networks optimizers optimization-technique gradient-centralization

• Updated Jan 13, 2022
• Python

RAdam implemented in Keras & TensorFlow

tensorflow keras adam optimizers radam rectified-adam

• Updated Jan 22, 2022
• Python

Keras/TF implementation of AdamW, SGDW, NadamW, Warm Restarts, and Learning Rate multipliers

tensorflow keras sgd adamw adamwr nadam optimizers learning-rate-multipliers warm-restarts

• Updated Jan 6, 2022
• Python

Implementation of the proposed Adam-atan2 from Google Deepmind in Pytorch

deep-learning artificial-intelligence stability adam optimizers

• Updated Sep 3, 2026
• Python

Summarize Massive Datasets using Submodular Optimization

greedy-algorithms combinatorial-optimization submodular-optimization data-summarization optimizers submodular-functions submodular submodular-information-measures

• Updated May 14, 2025
• Jupyter Notebook

Fast, Modern, and Low Precision PyTorch Optimizers

deep-learning pytorch optimizers

• Updated Oct 24, 2026
• Python

Code for the paper "Facial Emotion Recognition: State of the Art Performance on FER2013"

architecture tuning vggnet loss fer schedulers fer2013 saliency-maps optimizers facial-emotions

• Updated May 7, 2023
• Jupyter Notebook

Accelerated tensor operations and dynamic neural networks based on reverse mode automatic differentiation for every device that can run Swift - from watchOS to Linux

swift machine-learning deep-neural-networks deep-learning automatic-differentiation autograd recurrent-neural-networks recurrent-networks neural-networks derivatives convolutional-neural-networks tensor gradient-descent swift-machine-learning optimizers

• Updated Oct 12, 2026
• Swift

FrostNet: Towards Quantization-Aware Network Architecture Search

computer-vision deep-learning pytorch style-transfer classification object-detection quantization network-architecture semantic-segmentation optimizers quantization-aware-training int8-quantization post-quantization quantization-efficient-network

• Updated May 3, 2024
• Python

Instantly improve your optimizer with just 2 lines of code!

python machine-learning deep-neural-networks deep-learning neural-network tensorflow python3 pip python-package tensorflow2 optimizers optimization-technique gradient-centralization

• Updated Feb 9, 2022
• Python

Prodigy and Schedule-Free, together at last.

deep-learning pytorch artificial-intelligence optimizers

• Updated Sep 27, 2025
• Python

🌹 Rose: Range-Of-Slice Equilibration PyTorch optimizer. Stateless optimization through range-normalized gradient updates.

machine-learning ai deep-learning transformers optimizer pytorch artificial-intelligence sgd gradient-descent gpt lora adam-optimizer fine-tuning diffusion-models pretraining optimizers optimizer-algorithms large-language-models llm

• Updated Oct 9, 2026
• Python

DataPredict ML Library: 100+ Lua Models for Deep Learning and Reinforcement Learning 2026

machine-learning deep-neural-networks reinforcement-learning lua-library lua deep-learning clustering roblox scikit-learn deep-reinforcement-learning regression supervised-learning classification unsupervised-learning scikitlearn-machine-learning software-library optimizers

• Updated Nov 12, 2026
• HTML

Neutron: A pytorch based implementation of Transformer and its variants.

natural-language-processing python3 pytorch transformer seq2seq ensemble beam-search neural-machine-translation multi-gpu attention-is-all-you-need multilingual-nmt average-attention-network dynamic-sentence-sampling robust-neural-machine-translation average-models optimizers sentential-context context-aware-nmt relative-position dynamic-batch-size

• Updated Aug 10, 2023
• Python

Intergration to get optimizers information from the SolarEdge portal

python solaredge-api optimizers

• Updated Mar 19, 2025
• Python

Benchmarking Optimizers for LLM Pretraining. NeurIPS 2026, EurIPS 2025

benchmarking optimizers llm

• Updated May 3, 2026
• Python