I built a tiny neural audio codec. Then I listened to the results.

I implemented a small vector-quantized audio autoencoder to examine what gets lost when continuous samples become discrete code indices. The model trains, the reconstruction improves, and the quality numbers make it impossible to pretend that a tiny bitrate means good audio.

So I built a small vector-quantized audio codec from scratch with NumPy. No pretrained encoder, no codec service and no magic downloadable model. Just a nonlinear encoder, a trained discrete codebook, a nonlinear decoder and enough bookkeeping to see where it struggles.

The input is mono 8-kHz PCM split into 64-sample frames. A learned encoder maps each frame into a 12-dimensional latent vector. I quantize that vector against 32 learned embeddings using squared Euclidean distance, keep the winning code index, and reconstruct the frame using a learned decoder.

In simplified notation:

z = tanh(x @ W_encoder + b_encoder)
k = argmin_j ||z - codebook[j]||^2
q = codebook[k]
x_reconstructed = tanh(q @ W_decoder + b_decoder)

The nearest-codeword choice is discrete, so gradient descent doesn't flow through it directly. The prototype uses a straight-through approximation for the encoder path, a commitment penalty to keep the encoder near selected codewords, and a centroid-style codebook update. This is a tiny research implementation, not the architecture of a modern high-quality neural speech codec.

I generated a repeatable dataset of harmonic, voiced synthetic signals with different fundamental frequencies. After 280 training steps, the local run reported:

That is a substantial reduction in reconstruction error from the untrained model. It is emphatically not good speech reconstruction. The bitrate calculation counts five bits per 64-sample frame, excluding codebook weights, model storage, transport and framing. You cannot compare 625 bit/s and 64 kbit/s without also comparing fidelity and implementation cost.

The 8-bit μ-law baseline has dramatically better SNR on the same synthetic input. It's not a learned neural codec, and it isn't being used as a telephone-transport conformance test. It's there to stop me from congratulating myself over a very compressed but fairly mangled signal.

A useful production-oriented codec study would add real licensed speech, multilingual and speaker-separated holdouts, train/validation/test isolation, objective quality measures, listening tests (MOS or ABX), packet-loss concealment, decoder speed, model size, memory use and streaming latency. I'd test whether residual VQ, temporal context and learned decoder architectures improve the quality-versus-bitrate curve rather than optimizing bitrate in isolation.

There's also an operational reason to care about discrete latent representations: they can become tokens for downstream generative speech models. But quantized latents are not automatically good generative targets. Codebook collapse, temporal alignment and speaker/prosody preservation are separate experiments.

I like research projects where the numbers force me to change my mind. This was one of them. The training code works and the reconstruction improves; the result also tells me exactly how far away this toy architecture is from good voice quality.

Related: Neural Audio Codec & Vector Quantization · Speech & Audio ML projects · Streaming speech inference.