Good speech data is the hard part
Whisper is impressive and its vocabulary stopped in 2023. Fine-tuning fixes that, but only if you have high quality data, and getting that is the part nobody wants to do.
Locally fine-tuning Whisper and ASR models with Listenr · Part 1 of 2
Off-the-shelf Whisper models are impressive. Whisper is able to translate and transcribe a wide variety of languages while using consumer hardware. One reason it's so good came from being able to leverage way more semi-self-supervised data than models before it, using something like 680,000 hours of recorded audio clips and transcriptions. That data includes many languages (to varying degrees of accuracy) meaning the model can accurately process a wide range of accents, languages, and recording conditions well. But it's not perfect. Depending on the model variant, the WER is like 9.3% for English and because the model was released in 2023, so too is the model's vocabulary. Anything that wasn't included in the training data won't be transcribed well, meaning sentences that have new word, new technical jargon or slang won't be particularly accurate.
So other than waiting around for OpenAI to release a new version trained on new data, a potential fix for that is fine-tuning. It's a lighter weight process for updating model weights, reusing the core weights of the original model and greatly reducing the complexity, cost and resourcing otherwise required when starting from scratch.
However fine-tuning still requires data. And not just any data, but high quality data that actually can be used to address any of the models shortcomings. Getting this data can be quite a challenge for even well funded teams looking to improve the model.
The natural starting point is one of the established open datasets: Common Voice, LibriSpeech, VCTK, or one of the many derivatives built on top of them. They're free, reasonably well-documented, and genuinely useful with some extra processing.
But there is a catch: almost all of them are read speech (Common Voice does have "spontaneous speech" now, though the dataset is much smaller for now), the quality of audio greatly varies and accuracy of transcriptions/metadata is also not guaranteed.
A read-speech corpus has a particular cadence. A participant sits down, reads a sentence off a screen, waits for the recording light, and reads the next one. The recordings are clean, carefully segmented, and nothing like how people talk when they are thinking, explaining, or in the middle of a conversation. The vocabulary is also frozen at collection time. In my case, I tend to speak a lot of technical jargon, meaning words like LoRA, ROCm, GPT, or any number of domain-specific words simply do not exist in datasets assembled five years ago.
Dataset quality is generally improving though, with initiatives like Mozilla's Data Collective and Common Voice helping push the market forward for ASR/voice datasets. But the realistic question is whether those datasets contain enough material that matches your specific use case. Accent, domain, speaking style, acoustic environment all need to be considered. For my usecase, I wanted my voice and a specific list of words to be better understood which these datasets wouldn't really help with.
That means needing to collect and process my own data. This gives me the most control but also the most work to do. Getting tens of hours of high quality, transcribed and relevant data isn't a trivial endeavour. I don't have the resources to just pay someone to have a conversation and then painstakingly transcribe the audio. Nor would that actually reflect my voice. Hiring a professional annotation service is also out of the question because of cost and effort involved and costs wouldn't scale if I wanted to keep adding to the dataset.
After looking at other projects that aid in voice collection, I realized it's probably simpler for me to just create my own solution. I didn't want to learn a specific workflow and then still need to do additional bespoke processing, I didn't want to use any online/cloud services and I wanted to make the most out of the hardware I was already using.
So I created Listenr. Initially, I focused on the collection, transcribing and validation of conversational audio and then turned it into a full service, end-to-end fine-tuning pipeline.
The dataset-building side of the project works by streaming an audio source (like a microphone) that's always on, passing the data to a VAD-enabled ASR service that generates the initial transcript, saves the audio and then uses an LLM to further improve the raw transcript with a series of prompts and helpers (I can pass in a list of words I want it to generally watch out and replace).
Automating the Hard Part
Whisper is already good enough to use to get a raw transcript in real time. Whisper-Tiny or Whisper-Base are the smallest models and easily run on a dedicated GPU. They're not super accurate, but that's fine for this case - we leverage VAD and the small model to optimize what audio clips are actually being saved to disk. Otherwise, there'd be way too much useless data being saved and a more complex audio processing/segmentation step.
The raw transcripts can then also being improved using an LLM. Not just any LLM either, but local, private ones. Local LLMs are now capable enough to clean up the rough edges of a raw ASR transcript by fixing punctuation, correcting obvious homophones, expanding contractions, flagging poorly structured transcripts, and using conversation context to help shape transcripts.
Neither of those steps requires a human in the loop for a first pass and when continuously running in the background, a growing, labeled corpus of my own voice data naturally grows over time. And with none of the icky privacy-infringing services or ballooning costs that commercial tools might incur.
The rest of this series is what that turned into. The walkthrough covers running it end to end, and what a speech dataset looks like on disk covers the manifest and the splits in detail.