Using the hardware I already have to fine-tune Whisper (AMD isn't that bad)

I use AMD because the hardware is good and the company actually backs open source. ROCm is open, the driver is in the kernel, and they fund things like Lemonade, which Listenr leans on for local inference. The catch is that the software stack is not the easiest to work with, which is something I discovered first-hand developing Listenr.

Generally, NVIDIA has lead the development of CUDA, which is the layer connecting hardware people have and the software people write. It's the default assumption baked into the whole ecosystem. AMD has been catching up, and quickly, but defaults are sticky and that is where the friction comes from.

Dependency hell, the Python special

I always tell myself every time I create a new Python project that it'll be different - I'll leverage uv from the get-go and not have to think about wheels or subdependencies. But... that generally never works out once heavier compute is required. In this case, I started with pip (oops) and issues arose with Torch. Torch is not a direct dependency of Listenr. It arrives underneath transformers and peft, which means pip resolves it without anything in the chain ever being asked to think about your hardware. On Linux the default torch wheel on PyPI is a CUDA build.

Nothing errors. Install succeeds, training starts, loss goes down (eventually), checkpoints get written. But beyond small test runs, training is so very slow, because anything that runs is running on your CPU.

It's kind of obvious that torch isn't setup correctly, but deterministically, you can check via:

>>> import torch
>>> torch.__version__
'2.12.0+cu130'          # a CUDA build
>>> torch.cuda.is_available()
False

On a working AMD setup you want to see a ROCm build and a real device:

>>> torch.__version__
'2.9.1+rocm7.2.0'
>>> torch.cuda.is_available()
True
>>> torch.cuda.get_device_name(0)
'AMD Radeon Graphics'

Why this is a container problem

The other half is that ROCm is not a pip package. It is a user space stack that has to match the kernel driver, and only the kernel side lives on the host. If you have a modern AMD GPU or want to actually get the most out of your hardware, you'll need to install ROCm. I tried installing it directly in on my host machine (running fedora 44) and it was a mess. AMD doesn't actually support it out of the box (IIRC, only ubuntu and debian(?)) meaning there's no simple dnf command or binary to download. I tried working around it but I'm really not dedicated enough to build from source and maintain ROCm dependencies/updates etc. This means I am out of luck doing it 'the simple way'.

But thankfully AMD ships a docker image with PyTorch and ROCm that "just works". It assembled and matched OS+kernel+user space dependencies and all that out of the box, which offers a sane solution that doesn't involve building from source by hand. Listenr's Dockerfile builds on rocm/pytorch:rocm7.2_ubuntu24.04_py3.12_pytorch_release_2.9.1 and pins the torch that image ships so a later pip install cannot quietly swap in the PyPI one.

By leveraging the docker image, Listenr is able to make the end to end path repeatable for people who don't really care about kernel and particularly on AMD. Docker takes the hardest part off the table, since you are no longer matching a user space stack against your kernel by hand. It is not completely hands off, there is still pip to manage on top, but that is the part Listenr packages up so the pipeline runs on someone else's machine without them needing to know any of this.

gfx1151 was support out of the box?!

gfx1151 is the architecture ID for the GPU in Strix Halo, the same way a CPU has a model number. ROCm targets these explicitly.

When I first looked at this, ROCm did not support gfx1151, because the hardware was new. It does now, yay but this is a reminder new hardware takes some time to be supported. So the thing I expected to be the hard part turned out to need nothing at all: no mucking around with faking device IDs, no compatibility flag. The GPU reports itself correctly and it works.

torch.cuda.get_device_capability(0)   ->   (11, 5)

That is how software goes in the real world, stuff changes, sometimes for the positive even.

Precision matters (obviously)

Training works by doing a lot of math on a lot of numbers. The ELI5 version I understand is that learning happens by adding tiny amounts to the model's weights, over and over. Those tiny amounts are the actual "learning" and they can be infinitesimally small. The smaller the steps, the more subtly you can improve the model without overshooting (and getting worse results).

The catch is that a number format only has so many bits to work with, and those bits get split between how big a number can be and how finely you can distinguish one from the next. Go too coarse and your tiny learning step gets rounded away to nothing when it is added to a much larger weight, so that part of the model just stops learning. Go too fine and you are paying for decimal places that never changed the answer.

Fewer bits also means real speed, because GPUs ship dedicated matrix hardware for the small formats. It is not that big formats fall back to software, they run fine, there is simply far less silicon devoted to them.

To illustrate this simply, I generally use a Radeon 8060S locally when fine-tuning. Look at the throughput of matrix multiplication across the different precision variants:

| precision | 4096 squared | | --------- | ------------- | | fp32 | 2.45 TFLOP/s | | bf16 | 23.69 TFLOP/s | | fp16 | 22.90 TFLOP/s |

The unit for that speed is TFLOP/s, trillions of floating point operations per second. One multiply-add is one operation, and training is essentially nothing but multiply-adds, so this number is a decent proxy for how fast a run will go.

In practical terms, training wouldn't be that bad but there is still a penalty for being too precise. Below is the actual wall clock times for fine-tuning whisper tiny on my machine. Same model, same data, same 1000 steps, with two different precision settings:

| precision | per step | 1000 steps | | --------- | -------- | ---------- | | bf16 | 1.03 s | 17m14s | | fp32 | 1.65 s | about 28m |

Roughly 1.6 times, not ten. Training spends a lot of its time on things precision does not touch: loading audio, extracting features, the optimiser step. The matrix multiplies get ten times faster and the run gets one and a half times faster, which is still quite a large difference by again highlights the difference of synthetic benchmarks and the actual training run.

Also I only just learned that bf16 and fp16 land within a few percent of each other, so picking between those two is not about speed. Both are 16 bits, they just split them differently: fp16 keeps more decimal places, bf16 keeps more range. Training cares about range, because gradients get very small, so bf16 is the one you want.

To round this all off, here are end to end fine-tuning runs, 1000 steps each on about 7.7 hours of audio on my Strix Halo machine:

| model | wall clock | samples/s | | ------------- | ---------- | --------- | | whisper-tiny | 17m14s | 15.47 | | whisper-small | 55m27s | 4.81 |

An hour to fine-tune a 244M parameter speech model on an integrated GPU is genuinely reasonable in my opinion.

Working with Docker isn't completely painless

So, the PyTorch image is great, yay. But it also required lsitenr to flesh out a ton of docker-specific passthroughs and tinkering. It was the first time I ever configured a docker image to actually do compute heavy GPU workloads. Thankfully with the help of LLMs and documentation, I was able to figure it out. Specifically worth mentioning is --ipc=host, without it, every allocation fails, including a trivial matrix multiply:

Memory critical error by agent node-0 ... Reason: Memory in use.

The message says memory, the problem is IPC. Compose already sets it, so this only occurs when you are running a container by hand to debug something, which is exactly when you are least inclined to suspect it.

Memory on an APU

Strix Halo has one pool of memory shared between CPU and GPU which is pretty nifty and part of why Macs M processors are particularly good running models locally. My machine has 128 GB, with 96 GiB assigned to the GPU in firmware, which means the OS only sees about 31 GiB.

The practical consequence is that anything else using the GPU is competing with your training run. Lemonade was holding Whisper-Large-v3-Turbo at 4.6 GiB while I was benchmarking. POST /v1/unload frees it.

What the training actually produced

The hardware works, Listenr (generally) works, the data and its sources are solid and I finally can tweak actual real models on my own machine end to end. However the results were another matter: fine-tuning made whisper-small substantially worse, which I have written that up in detail in Fine-tuning made it worse.

I also unconvered a few bugs found along the way which are fixed upstream in listenr 0.2.1.

Also remember not to be silly

If you run OpenSnitch or any firewall, remember to allow the container access the internet (or set it up in way that you don't need the interet). I did not, and spent a while staring at DNS failures that look exactly like a broken Dockerfile when in fact nothing was downloading because I would not to let it. Whoops.