Skip to main content
Listenr CLI streaming real-time ASR transcriptions to Lemonade — Whisper-Tiny loaded alongside gpt-oss-20b-mxfp4-GGUF.

Series 1 part Since

Locally fine-tuning Whisper and ASR models with Listenr

A running log of building Listenr, a privacy-first tool for capturing real-world conversational audio and using it to fine-tune Whisper speech recognition models locally on AMD hardware.

The goal is simple: as an individual, build enough high-quality audio clips and transcriptions to meaningfully improve a speech recognition model — privately, cheaply, and with a scalable process.

fine-tuning pipeline

That means solving three problems:

Data Collection:

  • Capture — a frictionless way to record real-world conversational audio without sending anything to the cloud.
  • Label — automated transcription plus LLM-assisted post-processing to get clean, accurate ground-truth text.

Data Processing

  • Build - Take the disparate data and create useful datasets

Fine-tuning

  • Train — a reproducible fine-tuning pipeline that runs on consumer hardware (AMD especially) and produces a model that actually works.
  • Test - a quantitative way to measure if the results are useful.

This series aims to document the whole journey to enable others to do to the same!

All parts

  1. 01

    How I locally fine-tuned Whisper using my own voice data and some effort

    Off-the-shelf Whisper models are impressive but struggle with personal vocabulary, accents, and jargon not present in their training data. This post covers why standard open datasets fall short for personal fine-tuning and how I built Listenr to continuously capture and transcribe my own conversational audio as a training set.