Skip to main content
The four stages of a fine-tuning pipeline: data collection, data processing, fine-tuning, testing.

Series 2 parts Since

What I learned fine-tuning speech models locally

The failures, dead ends and wrong turns from fine-tuning ASR models on consumer AMD hardware, with the numbers that made each one obvious in hindsight.

Most write-ups of fine-tuning show the run that worked. This is the other set.

Everything here happened while building and testing Listenr, and every number was measured on one machine with an AMD iGPU. Some of it is embarrassing, in the way that a mistake is embarrassing only after you know what it was. A fine-tune that made a model two and a half times worse. A GPU that could not allocate memory for a reason the error message never mentioned. A result that reversed direction when I changed how I counted the words.

None of it is a reason not to fine-tune ASR models locally. All of it is the part that usually gets left out.

All parts

  1. 01

    Using the hardware I already have to fine-tune Whisper (AMD isn't that bad)

    AMD makes compelling hardware and backs open source, but CUDA is still the default. Here's a little taste of how to change that.

  2. 02

    Fine-tuning made it worse, and I still do not know why

    I fine-tuned whisper-small on 7.7 hours of real spontaneous speech and made it 2.6 times worse. I had a tidy explanation for it, so I tested the explanation, and the test said I was wrong.