Kyutai's MuScriptor doubles the state of the art in music transcription
What's the deal? KyutaiDealroom has a profile for this one. Try Dealroom →, in collaboration with Mirelo, has released MuScriptor, an open-source model that transcribes multi-instrument music recordings into MIDI. Feed it a recording — pop, classical, metal, or jazz — and it transcribes the notes played by every instrument, without needing to know which instruments are present.
Why now? Automatic music transcription has stalled since the MT3 model in 2022, held back by a shortage of real aligned audio-MIDI data. Earlier models trained mostly on audio synthesized from MIDI files, so they scored well on synthetic tests but produced noisy output on actual songs.
How it works: MuScriptor is a decoder-only transformer that predicts MT3-like tokens from a mel-spectrogram of a five-second audio excerpt, stitching longer tracks together chunk-by-chunk. Model sizes span from 100M to 1.3B parameters. Users can optionally tell it which instruments are present, both to customise output and to stabilise instrument identity across a track.
The training recipe: Kyutai trained the model in three stages. It pre-trained on 1.45 million synthesized MIDI files, then fine-tuned on 170,000 real recordings — 11,000 hours across most genres — which improved every metric by roughly 20 points. A final reinforcement-learning stage used 300 manually verified tracks to reduce missed notes and sharpen onset precision.
What's the endgame? On a held-out test set of 372 tracks, MuScriptor beat the previous state of the art, YourMT3+, across every measure. Its onset F1 score reached 60.4 against YourMT3+'s 32.5, and its multi-instrument F1 hit 48.2 versus 21.9.
What could go wrong? Kyutai concedes the transcriptions are still far from perfect. It recommends users supply a list of instruments to transcribe for more consistent results, and notes the model's tokenisation has limits.
Try it: Kyutai hosts the model at muscriptor.kyutai.org, where users can transcribe recordings, play back results, and download MIDI. Because it is open-source, it can also be run locally via the GitHub repo.
The signal: MuScriptor's leap shows that real-world data, not just cheap synthetic training sets, is what moves a stalled field forward. By open-sourcing a model that works on messy, real recordings, Kyutai is betting that accessible tooling can reopen a research area that had gone quiet for years.
Read more: Kyutai
Image credit: Generated with Gemini
