Project
WeightLifting - transformer from scratch
Decoder-only char-level transformer (~10.8M params) built and trained from scratch in PyTorch.
- Role
- Sole author
- Repository
- github.com/jkarancs/WeightLifting
- PyTorch
- CUDA
- Weights & Biases
A nanoGPT-style decoder-only transformer (~10.8M params: 384 embedding / 6 heads / 6 layers, block size 256) implemented from scratch in PyTorch and trained on HuggingFace’s megaGymDataset on a 6GB GTX 1660 Ti. Mixed-precision (AMP), gradient clipping, checkpoint/resume of model and optimizer state, a CLI generation script (temperature/top-k), 11 tests, clean src/ layout, and W&B tracking. Trained to step 5000 (val loss 3.68 -> 0.063, ~6h). “I trained a model from scratch” artifact.