Locking the picture to the music: audio clock, sidechain, and beat

Part 6. Why the demo’s master clock is the sound card

AudioQueue through plain FFI, time taken from the audio device instead of a timer, smooth clock correction without jumps, and one kick-drum envelope that pumps both the sound and the camera.

NoteWhere we are

We have the nebula (parts 2–4) and a finished track in memory (part 5). Today the track starts playing, and the picture stays in step with it until the last frame.

The problem of two clocks

The first idea: start the sound, take Instant::now(), and draw frames by it. That fails for two reasons:

1. Unknown startup latency. Between “start playback” and the first sound from the speakers there are tens or hundreds of milliseconds: device initialization, buffers, Bluetooth headphones.
2. Drift. The sound card’s crystal and the system timer are different oscillators. Over 150 seconds they diverge noticeably, and the kick starts arriving before or after the flash.

Hence the demoscene rule: the master clock is the sound. The picture asks the sound card which sample is playing right now and draws the frame for that moment.

sequenceDiagram
  participant HW as Audio device
  participant AQ as AudioQueue thread
  participant MT as Main thread (frames)
  AQ->>HW: buffer 1 (2048 samples)
  AQ->>HW: buffers 2 and 3, in advance
  loop every ~46 ms
    HW-->>AQ: buffer played, need the next one
    AQ->>AQ: fill(): copy from the finished track
    AQ->>HW: enqueue
  end
  loop every 16.6 ms
    MT->>AQ: AudioQueueGetCurrentTime()
    AQ-->>MT: sampleTime = position at the speakers
    MT->>MT: t → frame
  end

AudioQueue through plain FFI

Output goes through AudioQueue from AudioToolbox: a system queue of buffers that calls our callback whenever it needs more data [1]. We take no bindings and declare the C functions and structs directly in Rust. That pays off in part 7: this module survives the move to no_std almost untouched.

The most delicate part is the struct layout. It must match Apple’s headers byte for byte:

Stream format

src/audio.rs

#[repr(C)]
struct StreamFormat {               // AudioStreamBasicDescription
    sample_rate: f64,
    format_id: u32,                 // 'lpcm'
    format_flags: u32,              // float | packed
    bytes_per_packet: u32,          // 8: two channels × f32
    frames_per_packet: u32,         // 1
    bytes_per_frame: u32,           // 8
    channels_per_frame: u32,        // 2
    bits_per_channel: u32,          // 32
    reserved: u32,
}

const FORMAT_LINEAR_PCM: u32 = u32::from_be_bytes(*b"lpcm");  // a FourCC as a number
const FLAG_FLOAT: u32 = 1;
const FLAG_PACKED: u32 = 8;

Buffer

src/audio.rs

#[repr(C)]
struct QueueBuffer {                // AudioQueueBuffer
    capacity: u32,
    data: *mut c_void,              // samples go here
    byte_size: u32,                 // how many bytes are actually filled
    user_data: *mut c_void,
    packet_desc_capacity: u32,
    packet_descs: *mut c_void,
    packet_desc_count: u32,
}

Functions

src/audio.rs (abridged)

#[link(name = "AudioToolbox", kind = "framework")]
unsafe extern "C" {
    fn AudioQueueNewOutput(format: *const StreamFormat, callback: Callback,
        user: *mut c_void, run_loop: *const c_void, run_loop_mode: *const c_void,
        flags: u32, out: *mut Queue) -> i32;
    fn AudioQueueAllocateBuffer(queue: Queue, bytes: u32, out: *mut *mut QueueBuffer) -> i32;
    fn AudioQueueEnqueueBuffer(queue: Queue, buffer: *mut QueueBuffer,
        packet_descs: u32, descs: *const c_void) -> i32;
    fn AudioQueueStart(queue: Queue, at: *const TimeStamp) -> i32;
    fn AudioQueueGetCurrentTime(queue: Queue, timeline: *mut c_void,
        out: *mut TimeStamp, discontinuity: *mut u8) -> i32;
}

The callback runs on the AudioQueue thread when a buffer has finished playing. Its job is to copy the next chunk of the finished track and put the buffer back in the queue:

src/audio.rs

/// Runs on the AudioQueue's own thread; it is the only code touching `Track`
/// after `play` returns.
extern "C" fn fill(user: *mut c_void, queue: Queue, buffer: *mut QueueBuffer) {
    let track = unsafe { &mut *(user as *mut Track) };
    let buf = unsafe { &mut *buffer };
    let n = buf.capacity as usize / 4;                      // how many f32 fit
    let dst = unsafe { std::slice::from_raw_parts_mut(buf.data as *mut f32, n) };
    let src = track.data.get(track.pos..).unwrap_or(&[]);
    let k = n.min(src.len());
    dst[..k].copy_from_slice(&src[..k]);
    dst[k..].fill(0.0);                                     // silence after the track ends
    track.pos += n;
    buf.byte_size = buf.capacity;
    unsafe { AudioQueueEnqueueBuffer(queue, buffer, 0, null()) };
}

Three buffers of 2048 frames give about 46 ms per buffer and 140 ms of headroom for the whole queue. Before starting, we call fill for each buffer ourselves, so the queue does not begin with silence.

TipThread safety without a mutex

The track reaches the callback as a raw pointer to a Box that is never freed. That is safe because after play, only the AudioQueue thread touches Track. The main thread reads only the time, and that comes from the system, not from our struct.

The audio clock

AudioQueueGetCurrentTime returns sampleTime, the number of the sample playing right now. Divide by the sample rate and you get seconds:

src/audio.rs

pub fn time(&self) -> Option<f32> {
    let mut ts = TimeStamp::default();
    let err = unsafe { AudioQueueGetCurrentTime(self.queue, null_mut(), &mut ts, null_mut()) };
    (err == 0 && ts.sample_time > 0.0).then(|| (ts.sample_time / self.rate) as f32)
}

Until the device has started, the function returns an error and we answer None.

A clock without jumps

Taking the audio time directly is risky. Updates from the device may arrive in steps, and the animation would stutter. So smooth wall-clock time w stays the basis, and an offset o is added to it and gently pulled towards the audio time a:

\varepsilon = a - (w + o), \qquad o \leftarrow \begin{cases} a - w, & \text{first reading or } |\varepsilon| > 0.1\text{ s} \\ o + 0.05\,\varepsilon & \text{otherwise} \end{cases}, \qquad t = \max(0,\; w + o)

It is a simple phase-locked loop: on a large error the clock jumps immediately; on a small one it is pulled 5% per frame and converges smoothly within about a second.

src/main.rs

/// Demo time in seconds, locked to what the speakers are playing.
fn clock(&self) -> f32 {
    let iv = self.ivars();
    let wall = iv.start.elapsed().as_secs_f32();
    if let Some(a) = iv.audio.as_ref().and_then(Audio::time) {
        let err = a - (wall + iv.offset.get());
        if !iv.synced.get() || err.abs() > 0.1 {
            iv.offset.set(a - wall);                  // jump
            iv.synced.set(true);
        } else {
            iv.offset.set(iv.offset.get() + err * 0.05);  // gentle correction
        }
    }
    (wall + iv.offset.get()).max(0.0)
}

A debug run, first frames:

DBG wall 0.030 audio Some(0.24412699) t 0.2441
DBG wall 0.055 audio Some(0.26866212) t 0.2687
DBG wall 0.057 audio Some(0.27072564) t 0.2707

By the first frame, the sound has already been playing for 0.24 s: while the main thread was compiling the shader, the queue managed to start. The very first reading set the offset, and the picture matched the sound from frame one. The audio time advances in small increments without big steps, so the smoothing here is insurance rather than a necessity.

beat: one envelope, two consumers

We do not need to analyze the music to find the kicks: we wrote it. The kick_on function from part 5 knows which steps have a kick. beat(t) finds the most recent kick and returns a decaying envelope:

\text{beat}(t) = e^{-7\,(t - t_\text{kick})}

src/music.rs

pub fn beat(t: f32) -> f32 {
    let cur = floor(t / STEP) as i32;
    for k in 0..16 {                         // look back no further than one bar
        let step = cur - k;
        if step < 0 {
            break;
        }
        if kick_on(step as u32) {
            return exp(-(t - step as f32 * STEP) * 7.0);
        }
    }
    0.0
}

One bar of the “core” scene. The beat envelope jumps to 1 on every kick and decays. The same shape, inverted and scaled, sets the gain of the pad and bass.

The same function is used in three places:

Table 1: Consumers of beat.

Where	Formula	Effect
Sound, duck bus	gain × (1 − 0.6·beat)	The pad and bass dip under the kick: sidechain
Shader, camera	focal length × (1 − 0.03·beat)	On each kick the picture “flinches”, widening slightly
Shader, exposure	1.3 + 0.35·beat	A flash of brightness on each kick, +27%

src/main.rs

let params = Params {
    // ...
    beat: music::beat(t),      // the same function that shapes the sidechain
};

One subtlety. In the mixdown (part 5) the envelope is computed incrementally, while each frame uses beat(t). The formula is the same, though: exp(-(t - t_kick) * 7). So the sound and the picture pump in the same phase.

Listen to the moment the kick first comes in, at 0:30. The pad starts “breathing” in time with it:

And in the “flow”, where the sidechain works on every beat:

Summary

The whole demo now depends on one quantity: the time reported by the sound card.

flowchart TB
  HW["Sound card: sampleTime"] --> CLK["clock(): t"]
  CLK --> TL["timeline(t): scene, phase, brightness"]
  CLK --> CZ["camera_z(t): where the camera is"]
  CLK --> BT["beat(t): kick drum hit"]
  TL --> P["Params"]
  CZ --> P
  BT --> P
  P --> SH["Shader: the frame"]

Scenes, harmony, flight, light, and pulse are all computed from one function of time. That is what directing an intro means. One task remains: fitting all of this into 64 kilobytes.

Series: Demoscene 2026

Part 6 of 7 · suggested reading order

Previous: A synthesizer in 600 lines: a soundtrack without a single sample

Next: 338 KB → 11.7 KB: Rust without std, Objective-C without bindings, and a shell dropper

References

[1]

Apple Inc. Audio Queue Services. https://developer.apple.com/documentation/audiotoolbox/audio-queue-services