Project 02 / 05 · Generative audio modeling
Implemented and testedNeural Audio Codec & Vector Quantization
I implemented and trained a small nonlinear vector-quantized audio autoencoder with a learned encoder, decoder and discrete codebook, then compared measured reconstruction error to an 8-bit companding baseline.
A low bitrate on its own proves nothing. Compression, reconstruction fidelity, latency, packet loss and model cost pull in different directions; the representation has to earn its keep.
What I implemented
- Implemented a NumPy encoder/decoder with tanh nonlinearities, vector-codebook nearest-neighbor assignments and a straight-through gradient approximation.
- Trained and measured the model on generated voiced signals, recording reconstruction MSE, signal-to-noise ratio, code utilization and index-only bitrate.
- Built a separate 8-bit μ-law companding comparison and WAV reconstruction output so both the compression tradeoff and its limitations are inspectable.
Data and execution path
- 01
Synthetic voiced waveform or mono PCM16 input
- 02
64-sample framing and learned nonlinear encoder
- 03
Nearest-codeword quantization (32 entries)
- 04
Discrete code indices and codebook utilization
- 05
Learned nonlinear decoder and waveform reconstruction
- 06
MSE / SNR / bitrate against companding baseline
Experiment and results
On the local synthetic voiced corpus, 280 training steps reduced frame reconstruction MSE from 0.04164 to 0.01217, reaching 5.24 dB SNR. Five index bits per 64-sample frame imply 625 bit/s at 8 kHz before framing and model overhead. An 8-bit μ-law sample baseline achieved 37.89 dB SNR at 64 kbit/s on the same data. That quality gap is the important result—not a headline compression ratio.
These results are from the specified local dataset or offline rendering pipeline. I keep them separate from real speech-corpus and live-call metrics because the populations and failure modes differ.
System-level constraints
- Quality is a perceptual measurement as well as a numerical one; speech/music holdouts and ABX/MOS listening tests reveal failures synthetic SNR can miss.
- Streaming transport adds packet loss, decoder buffering, memory pressure and real-time deadlines that index bitrate alone misses.
- Residual VQ and temporal modeling change the bitrate, quality and latency frontier; they cannot be compared by compression ratio alone.
- Codebook collapse, model/decoder versioning and latent compatibility are distinct failure modes, not mere training hyperparameters.
What the measurements establish
This is a small trained neural codec prototype, not a production-quality speech codec, neural audio tokenizer for a generative model, or proof of perceptual quality. The test audio is synthetic; headers, real transport, model weights and listening-test scores are excluded from the idealized index bitrate.
I can make the bitrate astonishingly small. Making the reconstructed audio sound good is the part that requires actual research.