Motivation
Every top speedcuber has gone through the tedious process of deconstructing their solves. It is a tedious process of going frame-by-frame, matching each turn to a timestamp so viewers can follow along. This project was motivated by my desire to automate this through vision, aiding in both solve decomposition and coaching.
Model Architecture
The recognition side is built on R3D-18. Structurally, it's a plain ResNet-18, containing a stem, four residual stages, global average pooling, a final linear layer. In this architecture, however, every convolution is 3D, sliding over (time, height, width) instead of just (height, width), so a filter can respond to a pixel pattern moving in a particular direction across frames rather than to what a single frame looks like. The model weights were pretrained on the Kinetics dataset with 400 outputs; I swapped that last layer for a 19-way head (12 quarter turns, 6 cube rotations, and a NONE class).
Training
Model versioning and tracking was done through Weights & Biases, making all runs reproducible. Augmentation such as crop/zoom and brightness/contrast jitter helped immensely, but I deliberately held out vertical and horizontal flips to avoid mirroring moves the wrong direction. Additionally, I exposed how the backbone actually trains through an --unfreeze-layers flag, letting the residual stages fine-tune alongside the fc head. In practice, I found unfreezing just the last stage gave the best performance.
It was easier to record individual cube turns than whole-cube reorientation, leading to class imbalance. To address this, I used a class-weighted loss function, improving accuracy on rarer moves. On a 70/15/15 train/val/test split, I recorded a test accuracy of 95% on over 15,000 clips that I created.
Coaching Pipeline
Once I had move transcripts from my own speedsolves, I wanted to go further and turn them into real coaching. On their own, LLMs were not good at this: I tested one against Feliks Zemdegs' verified world record solves, and it could not even tell the cube was solved, confidently suggesting algorithms that were completely wrong. To fix this, I built a cube simulator that splits each solve into CFOP stages (Cross, F2L, OLL, PLL), the method most speedcubers use. OLL and PLL cases are matched against a database of canonical algorithms, built with SQLAlchemy on SQLite and served through a REST API, so feedback points to algorithms top speedcubers actually use.
I exposed these analysis functions as MCP tools, which made Gemini's coaching noticeably better. Every user has their own account, so their solves and feedback stay tied to them. My favorite feature is asking the LLM to compare a solve to my recent ones, so I can see whether I am actually improving over time.
Frontend
The project sits behind a React website, so a solve can be recorded, reviewed, and coached without touching the terminal, providing a usage point for everyday users. The raw video is deleted the moment the clips are confirmed segmented correctly, and there is an option to manually input previously deconstructed solves.
