An ML pipeline that converts an audio file into an editable four-part SATB (soprano, alto, tenor, bass) a cappella arrangement.
Architecture
We first use HT-Demucs, a pretrained audio-separation model, to isolate the lead vocal from the full mix. We then feed that isolated vocal to torchcrepe to determine the pitch at every moment, converting the lead vocal into a sequence of musical notes (sheet music).
The main part of our pipeline is a hybrid Transformer-LSTM model that harmonizes that lead melody into the other three voices. A Transformer encoder reads the whole lead melody at once, and its self-attention makes it good at taking in the full sequence and understanding context. From there, four independent LSTMs (one per voice) write each voice note by note, using cross-attention to continually check back on the lead melody so the harmony stays aligned with it. A rule-based post-processor then keeps the generated notes within human vocal ranges.
Training
There isn't much labeled a cappella sheet music out there, so we trained in two steps. First, we pretrained the model on 278 Bach chorales, which taught it the general rules of how four voices fit together. Then we fine-tuned it on 50 real a cappella songs at a much lower learning rate, so it could pick up that specific style without forgetting what it had already learned.
Results: voices awkwardly crossing over each other dropped from 5.7% to 1.3% of the time, and note-prediction accuracy rose from 23.3% to 37.6%, a 76% and 62% improvement respectively.
Deployment
The pipeline is served through a Streamlit web app, deployed on Hugging Face Spaces via Docker. You upload audio and download a MIDI file to open in whatever notation editor you like (MuseScore, GarageBand, etc.).
Demo
- Technical demo (video)
- Walkthrough demo (video)