Fine-tuning made it worse, and I still do not know why

I fine-tuned whisper-small on 7.7 hours of real spontaneous speech and made it 2.6 times worse. The loss went down, the dev WER looked fine, and the model came out much less accurate than the one I started with. I had a tidy explanation for why. Then I ran the experiment that explanation predicted, and it came back the other way.

The setup

This was a shakedown, not an experiment. Listenr had just had a larger release, and I wanted to know whether the whole path still worked end to end on my own hardware: record, build a dataset, fine-tune, merge, evaluate. Not whether a few hours of audio would move a model.

Mozilla Common Voice Spontaneous Speech 4.0 English, CC0, 609 speakers, 7.66 hours of transcribed audio, using the corpus's own train/dev/test splits. LoRA fine-tuning on a single AMD GPU, 1000 steps.

Every signal I had looked healthy: 55 minutes of wall clock, final train loss 1.131, best dev WER 23.36. Then listenr eval --compare-base on the 337 held-out clips, using Whisper's English normalization:

whisper-small    base 10.8%  ->  fine-tuned 28.0%

Same clips, same normalization, one model before the fine-tune and one after.

The explanation I reached for

I took a sample of 400 training labels and read them. 41% contain "uh" or "um". 11% have no terminal punctuation at all. Here is one, verbatim:

I like almost no sports or exercise but uh yoga, swimming, ah, and walking or hiking

That line has both problems. Mozilla's transcribers write down what the speaker said, filler included, and do not force a full stop onto a fragment. Whisper does the opposite on both counts. Neither convention is wrong, they just disagree, and WER does not care which one you prefer: every "uh" the base model correctly omits is scored as a deletion. Listenr's strip_noise_tags drops bracketed [disfluency] markers, which is why I thought this was handled, but not a spoken "uh", because there "uh" is just a word.

So the story wrote itself. The fine-tune learned the corpus, house style included. A second set pointed the same way: on 23 clips of my own speech that neither model was trained on, labelled in Whisper's style rather than Mozilla's, whisper-small went from 14.4% to 16.6%. Worse again, as a style mismatch predicts.

It was a good hypothesis. It named a mechanism, fit the size of the effect, and predicted something I could check in an afternoon: clean the labels into Whisper's convention, retrain, and most of the damage should go away.

The test that killed it

Same corpus, same clips, same recipe, with one change made first: fillers removed from the training labels and terminal punctuation added, 728 of the 1472 training labels changed. Then retrain and evaluate on the same 337 held-out clips.

raw labels       base 10.8%  ->  fine-tuned 28.0%
cleaned labels   base 10.7%  ->  fine-tuned 35.6%

Cleaning the labels made it 7.6 points worse. The hypothesis predicted the opposite, in advance, so this is not a matter of interpretation. It is wrong.

The only other evidence I had for the cleaning story was a whisper-tiny run on cleaned corpus labels mixed with my own clips, which held WER exactly flat. It cannot carry the claim: different model, different training mixture, and a baseline four times higher with far more room to move.

What is established, and what is not

Established: fine-tuning whisper-small on 7.7 hours of this corpus makes it substantially worse. 10.8% to 28.0% with the labels as shipped, 10.7% to 35.6% with them cleaned. Two independent runs, same direction, large effect, and the first held its direction under a second text normalizer too.

Also established, and still interesting: 41% of a 400-label sample contains "uh" or "um". A true fact about the corpus, worth knowing before you train on it, and not the cause of anything I measured.

Not established: the cause. I do not have one.

What is left is a hypothesis, flagged as one. A model already at 10.7% on the target domain has little to gain from 7.7 hours of it against the roughly 680,000 hours it was pretrained on, while 1000 steps at the learning rate I used is plenty to do damage. That predicts the regression gets worse the better the base model already is, consistent with tiny and small behaving differently. Checking it needs a learning rate sweep I have not run.

The words I actually wanted

The whole point of the exercise was vocabulary. 23 clips of my own speech cover the words Whisper keeps mangling, and listenr eval --keyword reports per-word recall on them. Base against fine-tuned, for both model sizes:

| word | tiny base | tiny tuned | small base | small tuned | | ---------- | --------- | ---------- | ---------- | ----------- | | Claude | 0/3 | 0/3 | 2/3 | 2/3 | | Sonnet | 0/2 | 0/2 | 1/2 | 1/2 | | Opus | 0/1 | 0/1 | 1/1 | 1/1 | | Haiku | 0/1 | 0/1 | 0/1 | 1/1 | | PyTorch | 0/1 | 1/1 | 1/1 | 1/1 | | Moonshine | 0/1 | 0/1 | 0/1 | 0/1 | | Mistral | 0/1 | 0/1 | 0/1 | 0/1 | | ChatGPT | 0/1 | 0/1 | 0/1 | 0/1 | | Kubernetes | 0/1 | 0/1 | 0/1 | 0/1 |

Read the base columns first, because they are the finding. whisper-small already produces Claude, Sonnet, Opus and PyTorch out of the box. whisper-tiny produces none of them. For this vocabulary the gap is model size, not missing training data.

The fine-tune moved two cells, each on a single clip, and neither is a result. It could not have taught these words in any case: the corpus contains zero occurrences of claude, anthropic, openai, llm, rocm or pytorch. It is lifestyle speech. Whatever the adapter learned, it did not learn this vocabulary, because this vocabulary was never in front of it.

That is worth knowing before you spend a weekend recording. The cheap experiment is to run the next size up and see whether the word you care about was already there.

Two more runs, and the limit they found

So I tried to teach the words directly, with two more runs on whisper-tiny. The numbers below look like a failure. They are not. I had 2.7 minutes of audio. Nobody should expect that to reshape a speech model. I wanted to know that the path from a handful of recordings to a merged model runs cleanly, and roughly where the ceiling sits.

The first trained on the keyword clips alone: 18 clips, 2.7 minutes of audio, 5 held out, 300 steps, 100 epochs. The second mixed those clips into the corpus, with the corpus labels rewritten into Whisper's convention first, keyword clips oversampled 20x to 19.7% of the training set.

| Training data | Claude base | Claude tuned | WER base | WER tuned | | ------------------------------------ | ----------- | ------------ | -------- | --------- | | keyword clips alone | 0/3 | 1/3 | 34.2% | 39.7% | | corpus plus keywords, labels cleaned | 0/3 | 0/3 | 34.2% | 34.2% |

Every other target word stayed at zero in both.

Training on the target clips alone bought a flicker of learning, one word in one clip, and paid 5.5 WER points for it. That is overfitting on 2.7 minutes of audio. Mixing in a much larger pile of general speech prevented that damage completely, and prevented the learning with it.

I do not think the setup I used can teach a word the model has never heard. LoRA only adjusts a thin slice of the network, and I had frozen the half that listens to the audio, so nothing was ever going to learn what an unfamiliar word sounds like. On top of that, 18 clips covering 14 different words means most words appeared exactly once. One example is not training, it is a coincidence waiting to happen.

The next attempt wants more recordings per word and more of the model left unfrozen. Until I run that, the honest statement is that I found a limit, not a result.

The cheapest way to find out you are wrong

Nothing I watched during training could have caught this. Loss curves, dev WER, hours of audio, speaker counts: all healthy. Dev WER especially, because it is scored against labels from the same corpus the training set comes from, so it falls as the model gets better at imitating that corpus, whether or not that is what I want. It only becomes a regression when you compare against something else, which here was the base model on the same clips.

The other lesson is smaller and more useful. An explanation that only fits what you have already seen is cheap, because you built it knowing the answer. The one worth having predicts something you have not measured yet. Mine did, checking took an afternoon, and it came back negative. That afternoon is the whole difference between this post and a confident wrong one.

The run I am not going to report

I ran the same recipe on whisper-tiny at the same time, and I am leaving that number out, because I do not have one I trust.

listenr eval --compare-base, with Whisper's English normalization, said tiny improved a lot. An independent re-run on the same 337 clips with the same two checkpoints, using a naive normalizer that only lowercases, strips punctuation and collapses whitespace, said the opposite: 35.8% base against 39.1% fine-tuned. Two measurements of the same two files, pointing in different directions.

The reason is the "uh" problem again, from the other side. Whisper's own scoring strips filler words before comparing, so under it the mismatch mostly vanishes. The simpler scoring keeps them, which punishes the base model for leaving out fillers the corpus wrote down. Same two files, two defensible ways to count, two different answers.

I also checked whether the gap was big enough to mean anything at all, and it is not. On 337 clips the difference sits inside the range you would expect from chance, before you even get to the argument about which scoring is right.

A result that flips sign when you change the text normalizer is not a result. The small regression survives both normalizers and is large enough to see through the noise. The tiny number does not, so it does not get published.

One related trap in the same tooling

listenr build-dataset was reshuffling the corpus into a fresh 80/10/10 split, discarding the train/dev/test division the corpus ships with. Common Voice keeps speakers disjoint across those splits deliberately, so reshuffling scatters the same 609 speakers across all three and the test set fills with voices the model trained on, inflating anything you measure. A --preserve-splits flag existed but was unreachable, because validate_entry dropped source_split first. Fixed in listenr 0.2.1, and the dataset behind every number above was rechecked: zero speakers shared between train and test.

Where that leaves the case for your own data

What two runs support is narrow: a model already good at your domain is not obviously improved by a few more hours of the same, and it can get a lot worse. Hours are not the unit of value. That does not make recording yourself pointless, because the argument for your own audio was never volume. It is that a general model has the least evidence for how you talk and which words you use, which is the gap the keyword table above measures. What I no longer have is a story about how to close it: a result I trust, an explanation I disproved, and one hypothesis left to test.

The rest of this work is in the Listenr series.