Making a 3B Hindi text-to-speech model speak in real time on a 4 GB laptop GPU

The result first. rumik-oss-1 is a 3-billion-parameter Hindi, English and Indic text-to-speech model. The published engine for it needs an H100. A from-scratch engine with int4 weights and hand-written GPU kernels now streams its speech at about 98 audio tokens per second on a 4 GB laptop GPU, where real time is 100. The first sound leaves the speaker about half a second after you press Speak, the whole thing fits in 2.5 GB of VRAM, and the speech is unchanged to the ear and to an ASR model.

int4 weights: bringagi/rumik-oss-1-int4 (CC-BY-NC 4.0). Engine code: neoandmatrix/rumik-fast (private at the time of writing).

What "real time" means here

Rumik generates speech as a stream of audio tokens for the Mimi codec: 8 tokens describe 80 ms of sound. Playback without gaps therefore needs 100 tokens per second, every second, plus the time to turn tokens into waveform. On an H100, the reference engine by Ashwin Hegde does 301 tok/s. Nobody had measured the model on anything smaller.

The useful mental model for decoding is simple. Every token requires reading all of the model's weights once. At bf16, Rumik's 36 layers are about 5.6 GB. The laptop's memory bus moves 192 GB/s. So the bf16 ceiling on the laptop is 192 / 5.6 ≈ 34 tok/s even with perfect code, a third of real time. Software alone cannot fix that; the bytes have to shrink. This is the whole plan: fewer bytes per token, then kernels that actually reach the memory bus.

Step 1: measure the reference on a smaller GPU (cloud L4)

Ashwin's engine, unchanged, on a $0.80/h L4 (300 GB/s):

The engine was already at 79% of the L4's memory bandwidth. It was not slow; the GPU was small. And a second problem appeared that the H100 hides: the Mimi codec took 24 ms of every 80 ms frame on the same GPU.

Step 2: int8 and int4 weights (cloud L4)

Replacing the four projection matrices per layer with torchao quantized tensors, everything else untouched:

int4 made Rumik real time on an L4 with no measurable loss (the 0.073 on Hindi is the ASR writing a mixed sentence's English words in Devanagari, identical for all three). Scaling the same 68% bandwidth efficiency to the laptop predicted about 80 tok/s. Not enough. The laptop would need better kernels than torchao's.

Step 3: an int4 file and a first kernel on the laptop

A short CPU job exported the weights to a plain int4 format: every projection quantised in groups of 128 columns with an fp16 scale and minimum per group, two 4-bit codes per byte. 2.13 GB, not the 1.6 GB I guessed: the embedding table alone is 0.57 GB at int8, because the vocabulary is 277,000 entries.

A Triton GEMV (matrix-vector product, the operation that dominates batch-1 decoding) for that layout reached 99–152 GB/s on the laptop, just behind PyTorch's own int4 kernel and well behind what cuBLAS reaches with bf16 (140–168 GB/s). The whole step, 108 projections and the audio head in one CUDA graph, took 14 ms: at most 72 tok/s before attention and the codec.

Two laptop-specific discoveries from profiling that step:

• The GPU throttles. The first step took 10.1 ms, sustained steps 12.7 ms. The SM clock fell from 1,600 to 1,280 MHz at the 35 W power limit. Neither the OS performance mode nor nvidia-powerd lifted it. Timings also drift with heat (10.1 ms cold, 14.9 ms warm), so every comparison has to be back to back.
• Mimi is fine on a laptop. One frame decodes in 6.4 ms on the 3050, 8% of the budget. The L4's 24 ms was contention with the language model, not the codec's cost.

Step 4: our own engine speaks (59 tok/s)

Rather than keep patching a reference engine built for a different GPU, I wrote one from the model card and the Cohere2 architecture it derives from. Per layer, the q, k, v, gate and up projections are fused into a single 25,088-row int4 matrix, so three GEMVs per layer instead of seven. A static KV cache in length buckets, an audio-only output head, top-k sampling on the GPU with the Gumbel-max trick, one CUDA graph per bucket, and one device-to-host read per 8-token frame.

Correctness before speed: the layer output matches transformers' Cohere2 layer on the same dequantised weights to within 1% (bf16 noise), and the decode step agrees with the prefill logits on the argmax. First speech from the laptop: 58–60 tok/s, 0.6× real time, 770 ms to the first frame. Judged by ear later: about 9 out of 10 clips are clean speech; numbers come out digit by digit and emotion tags produce quiet audio, both of which are model behaviour to fix in a front-end, not in the engine.

The profile said where the time went: a 16.8 ms step, of which 12.7 ms was int4 GEMVs and about 4 ms was ~1,500 tiny kernels for norms, rotary embeddings, attention, residual adds and sampling.

Step 5: fuse the small things (75 tok/s)

Four Triton kernels replaced those 1,500 launches with about 260: embedding plus LayerNorm; a "post" kernel that applies rotary position, writes the KV cache and does the SiLU gate; an online-softmax attention kernel; and a fused residual-add plus LayerNorm. Same tokens out, 4 ms down to 1.5 ms, and real generation went from 59 to 75 tok/s. Two lessons about tuning on a laptop: autotune results are noisy (13.0 vs 14.6 ms for the same config in two runs), and a config that wins in isolation can lose inside the real step. Tune inside the real CUDA graph and pin the result.

Step 6: flash-decoding for long clips

Attention over a 2,000-position cache was taking 2.6 ms per step, so 18-second lines ran at 54 tok/s while short ones ran at 75. Splitting the attention over 16 chunks per KV head, computing the 4 query heads that share a KV head in one tensor-core tile, and combining the partial results in a small second kernel brought it to 1.1 ms. Long clips now run at the same speed as short ones.

Step 7: why the GEMV was stuck, and a CUDA kernel (93 tok/s)

The int4 GEMV sat at 112 GB/s on a 176 GB/s bus and no Triton variant moved it. Two measurements found the reason:

• A read-only twin of the kernel, identical except that it never does the maths, reached 153 GB/s, as fast as cuBLAS. So memory was not the limit.
• Counting instructions in the compiled SASS showed about 6 instructions per weight to unpack a nibble, convert it to a float and multiply-accumulate. On 16 SMs at 1.3 GHz, that arithmetic was the ceiling.

A hand-written CUDA kernel uses the fp16 "magic number" trick: a LOP3 instruction places each 4-bit code into the mantissa of an fp16 whose exponent is fixed, so the value 1024 + code appears without any integer-to-float conversion, and one HFMA2 then applies the scale and minimum to two values at once. About 1.6 instructions per weight. 167 GB/s, 95% of peak, on the largest matrix. One bug (a minimum term counted four times) was caught with known-answer inputs; the model's output distribution is unchanged (KL 0.007, top-1 agreement 40/40). Real generation: 93 tok/s.

Step 8: a prefill that stays int4

The prompt stage still dequantised the weights to temporary bf16 copies for every request, which took 0.7–2.2 s. A Triton matmul that keeps the weights int4, unpacks one group at a time, runs on tensor cores and applies scale and minimum to the result brought the prefill to 50–230 ms with the same first-token distribution. One gotcha worth knowing: Triton silently specialises on integer arguments that are multiples of 16, so a 112-token prompt triggered a 1.3 s recompile. Another: on battery the GPU is power-capped, so speed numbers are only valid on AC.

Steps 9–10: streaming to the speaker (97 tok/s, zero stalls)

The codec normally decodes a whole clip at once. An exact streaming version keeps the convolution state between frames on top of the transformer cache (66 dB signal-to-noise against a full decode) and feeds a PortAudio player. Then the arithmetic bit: at 94 tok/s each 80 ms frame takes 84.5 ms to generate, so playback falls behind by 5.6% of the clip, and the start delay has to cover that. Streaming is only as good as being faster than real time, and the engine was 6% short.

The remaining 0.7 ms per token after the 36 layers was the audio head, torch.topk and about 25 small ops. Three Triton kernels replaced the sampler (top-k by sorting int64 keys that pack the float bits with the index). The audio head moved to int8 with a new CUDA kernel; int4 was measurably too lossy for the head (KL 0.10–0.15 against 0.013 for int8). Live streaming reached 97 tok/s with no stalls on any of the 8 test lines: first sound at 0.45–0.7 s for 5–8 s lines, 1.3–1.5 s for 18 s lines, where the wait is the safety buffer for being 3% slower than real time.

A small web UI finished it: voice, tone, accent and pace dropdowns, buttons for <laugh> and <sigh> tags, live waveform, and a history of takes. Audio streams to the browser as 80 ms int16 frames over a chunked HTTP response and the Web Audio API schedules them; the first audio reaches the page about 0.3 s after Speak.

A detour: the laptop engine on an H100

Out of curiosity, and for about $0.65, the engine ran unchanged on a rented H100 next to the bf16 reference engine.

Twice the speed of the reference, with less than a quarter of its memory. But look at the efficiency column. On the laptop the engine reads weights at 78% of peak bandwidth; on the H100 it manages 30%. Seventeen times the bandwidth bought seven times the speed.

Re-tuning the CUDA kernel's block shapes for 132 SMs instead of 16 gained 2% (673 → 690 tok/s). The isolated numbers explain why: the big fused GEMV reaches 61% of peak, but the 2 MB o_proj takes 3.2 µs, which is the launch latency of any kernel, and the step is about 230 launches. At 1.45 ms per token, the fixed cost per kernel is the bottleneck, not the bytes. "Memory-bound" was true on the laptop and false on the H100 for the same code. Getting further there means a different design, with far fewer kernels per layer, which is not the design that suits a 16-SM laptop GPU. I stopped here.

What I would tell someone starting this

1. Do the bytes-per-token arithmetic first. Weights ÷ bandwidth is the ceiling. If it is below your target, no kernel will save you; quantise.
2. int4 with group-128 scale and minimum was free for this model on an ASR-measured basis, for every matrix except the output head, which needed int8.
3. Measure what is actually limiting. A read-only twin of a kernel and an instruction count in SASS answered in an hour what weeks of Triton variants had not.
4. Count kernel launches, not just bytes. 1,500 small kernels per token cost as much as a third of the weights.
5. Benchmark inside the real CUDA graph. Per-call launch overhead in a micro-benchmark lies about small kernels, and laptop clocks drift with heat and power. Compare back to back, on AC.
6. A kernel tuned for one GPU is tuned for that GPU. The same code is memory-bound on 16 SMs and launch-bound on 132.
7. Streaming needs margin. Being 3% slower than real time costs you half a second of start delay on every line.

Cost, licences, credits

Laptop time was free. Cloud GPUs: roughly $2 in total across the L4 runs, the int4 export and the H100 detour.

The weights, including the int4 version, are rumik-oss-1 by Rumik AI, a derivative of Cohere Labs' tiny-aya-fire: CC-BY-NC 4.0 plus the Cohere Labs acceptable-use policy, non-commercial use only. The Mimi codec is Kyutai's (CC-BY-4.0). The int4 model card's NOTICE lists every change.

Thanks to Rumik AI, Cohere Labs and Kyutai for the models; to Ashwin Hegde for the rumik-latency engine that gave steps 1 and 2 their baseline; and to Mrinaal Arora, whose latency write-up framed the problem.