The non-obvious step: when the user spins their wheel, you don't just change zoom. The world point under the cursor must stay under the cursor through the entire zoom. Otherwise the canvas "jumps" away from the cursor at every notch — the cardinal sin of pan/zoom UI.

The fix is two lines of math:

#![allow(unused)]
fn main() {
let world_pivot = viewport.screen_to_world(pivot);
let new_zoom = (viewport.zoom * factor).clamp(min, max);
viewport.zoom = new_zoom;
viewport.offset = pivot - world_pivot * new_zoom;

</div>
</div>

Take the world point under the cursor BEFORE the zoom. Change the
zoom. Solve `offset` so that the same world point lands under the
same pixel cursor AFTER the zoom. Done.
}

The transform

flowchart LR
    A[world point] -->|"× zoom + offset"| B[screen point]
    B -->|"− offset"| C[screen point − offset]
    C -->|"÷ zoom"| D[world point]

A Viewport2D { offset, zoom } represents a world-to-screen transform via screen = world * zoom + offset. screen_to_world is the inverse. The controller mutates this struct in response to host input.

Inputs the controller wires

y_delta < 0 (browser convention: scroll up) zooms in; y_delta > 0 zooms out. The host's adapter normalises whichever sign convention the source uses.

Quickstart

#![allow(unused)]
fn main() {
use glam::Vec2;
use wisp_interaction::{PanZoomController, Viewport2D};

let mut ctrl = PanZoomController::new();  // Figma defaults
let mut viewport = Viewport2D::identity();

// Pan with middle mouse:
ctrl.pan_begin(Vec2::new(100.0, 100.0));
ctrl.pan_drag(Vec2::new(150.0, 200.0), &mut viewport);
ctrl.pan_end();

// Zoom around cursor (wheel up):
ctrl.wheel_zoom(Vec2::new(300.0, 200.0), -1.0, &mut viewport);
// World point under (300, 200) is still under (300, 200) after zoom.
}

Clamps

The controller exposes min_zoom: 0.01 and max_zoom: 100.0 by default — Figma-equivalent. Tighten for chart canvases that don't want users zooming past readability; loosen for the recorder's editor surface where the user might want a full-document overview.

Adapters — winit and web-sys

One normalised vocabulary, many input sources. The adapters are pure translation functions — testable without ever opening a window.

The pointer-event lineage (1968 → 2013)

Tracing the family tree of every input event your finger fires today:

flowchart TD
    A[Engelbart mouse — 1968] --> B[Xerox Alto — 1973]
    B --> C[Apple Lisa ADB — 1983]
    C --> D[Microsoft serial mouse — 1987]
    D --> E[USB HID — 1996]
    E --> F[W3C MouseEvent — 2000]
    F --> G[W3C TouchEvent — 2011]
    G --> H[W3C PointerEvent — 2013]
    H --> I["this crate's adapter::web"]
    E --> J[winit 0.30 — 2024]
    J --> K["this crate's adapter::winit"]

Engelbart's original 1968 mouse reported one button press over a single wire. The Xerox Alto added three buttons and a serial protocol. Apple's ADB and Microsoft's serial-then-PS/2 standardised the transport. USB HID (Human Interface Devices) collapsed keyboards, mice, gamepads, and tablet pens into one report format in 1996. The W3C standardised the browser-side surface in three passes: MouseEvent (2000), TouchEvent (2011), and finally PointerEvent (2013) — a single typed event covering mouse, touch, and pen with a stable pointerId per contact.

wisp-interaction lives at the bottom of that tree. Its adapter module translates whichever event source your host owns (winit on native windows, PointerEvent in browsers) into one normalised vocabulary: InputEvent, PointerId, WheelDelta, KeyCode.

Pure-function adapters

```admonish important title="No addEventListener here" The adapter modules ship pure translation functions only: translate_mouse_button, translate_scroll, translate_key_code, etc. No event-loop wiring. That belongs in the host crate — the recorder app, the storybook bundle — whichever owns the window or canvas.

This split makes the translation independently testable (and indeed, we have 6 winit-translation tests that run on every CI matrix runner without ever opening a window). The host's wiring code is mechanical and platform-specific; the translation correctness is where the bugs live.

## The `KeyCode` enum

87 variants matching the intersection of `winit::keyboard::KeyCode`
and W3C UI Events Code strings (`"KeyW"`, `"Digit1"`, `"ArrowUp"`,
`"MetaLeft"` → `SuperLeft`). Letters, digits, F-keys, navigation,
modifiers, punctuation. Anything outside this set returns `None`
from `translate_key_code` — adapters drop unmapped keys silently
(Numpad, IME-only, dead keys).

## The translation table

| Winit / web-sys | wisp-interaction | Notes |
|---|---|---|
| `MouseButton::{Left,Right,Middle,Back,Forward,Other(n)}` | `MouseButton::{Left,Right,Middle,Back,Forward,Other(n)}` | 1:1 |
| W3C `PointerEvent.button` (i16) | same | 0=Left, 1=Middle, 2=Right, ... |
| `MouseScrollDelta::LineDelta` | `WheelDelta::Line(Vec2)` | Mouse-wheel notches |
| `MouseScrollDelta::PixelDelta` | `WheelDelta::Pixel(Vec2)` | Trackpad / touch surface |
| W3C `WheelEvent.deltaMode == LINE` | `WheelDelta::Line` | |
| W3C `WheelEvent.deltaMode == PIXEL` | `WheelDelta::Pixel` | PAGE is treated as Pixel |
| winit `Touch::id: u64` | `PointerId::Touch(u64)` | |
| W3C `pointer_id (i32) + pointerType != "mouse"` | `PointerId::Touch(u64)` | |
| `ModifiersState::SHIFT \| CONTROL \| ALT \| SUPER` | `ModifierState { shift, ctrl, alt, super_key }` | Per-event snapshot |

## `FocusLost` → release all keys

When the window or canvas loses focus, the OS may stop sending
key-up events. `WindowEvent::Focused(false)` / DOM `blur` should
trigger `ButtonInput::release_all()` so phantom-held keys don't
stick around when the user tabs back in.

## Adapter usage

```rust
# #[cfg(feature = "winit")]
# fn demo() {
use wisp_interaction::adapter::winit::{
    translate_mouse_button, translate_modifiers, mouse_button_event,
    pointer_location,
};
use winit::event::{MouseButton, ElementState};

let modifiers = translate_modifiers(winit::keyboard::ModifiersState::SHIFT);
let event = mouse_button_event(
    translate_mouse_button(MouseButton::Left),
    matches!(ElementState::Pressed, ElementState::Pressed),
    modifiers,
);
// `event: InputEvent::MouseButton(...)` — feed to ButtonInput<T>
// or PointerDispatcher.
# }

DPR + viewport coordinates

AnimationTriggers — wire pointer events to animations

Disney's "anticipation" and "follow-through" make a button-press feel substantial. AnimationTriggers is the sugar that wires Pointer<Click> to a Driver::play(...) call.

Disney's 12 Principles (1981)

In 1981, two legendary Disney animators — Frank Thomas and Ollie Johnston — published The Illusion of Life: Disney Animation, a book distilling fifty years of studio craft into twelve numbered principles. The list is one of the great instruction manuals in any visual medium. Principle #1 is squash and stretch: when a ball hits the ground, it flattens before bouncing back. Principle #2 is anticipation: before a character swings a bat, they wind back — your eye reads the wind-back as "something is about to happen." Principle #6 is slow in and slow out: motion accelerates and decelerates, never starts or stops abruptly. Principle #7 is arcs: natural movement traces curves, not straight lines.

The reason a click-to-trigger button in software UI feels good is that someone applied these principles. The button pre-shrinks slightly when pressed (anticipation), then springs back beyond its rest size before settling (squash-and-stretch + follow-through). The whole motion takes 200–400 ms and lives on a spring curve, not a linear ramp (slow in / slow out). That's what AnimationTriggers wires up — the connection between "user clicked" and "Driver, play the bounce tween" — without requiring a dependency from wisp-animation to wisp-interaction.

Why glue lives here, not in wisp-animation

Shape

sequenceDiagram
    participant User
    participant Adapter
    participant Dispatcher
    participant Registry
    participant Triggers
    participant Driver
    User->>Adapter: click
    Adapter->>Dispatcher: on_pointer_press / release
    Dispatcher->>Registry: lookup (NodeId, EventKind::Click)
    Registry->>Triggers: invoke registered Fn()
    Triggers->>Driver: driver.play(tween)

AnimationTriggers is a thin wrapper over CallbackRegistry. It exposes ergonomic methods (on_click, on_hover_enter, on_press_release, on_drag) that take no-arg closures the host can wire to anything.

Cooldown debouncer

Quickstart

#![allow(unused)]
fn main() {
use std::cell::Cell;
use std::rc::Rc;
use wisp_interaction::{AnimationTriggers, CallbackRegistry, Cooldown, cooldown_action};
let mut registry = CallbackRegistry::new();
let my_button_node = wisp::scene::Stage::new().root();

// Simple: no debounce.
{
    let mut t = AnimationTriggers::new(&mut registry);
    t.on_click(my_button_node, move || {
        // driver.play(bounce_tween);
    });
}

// Anti-spam with a 300ms gate. Replace `now_secs` with your monotonic clock.
let now_secs = Rc::new(Cell::new(0.0_f32));
{
    let cooldown = Rc::new(Cooldown::new(0.3, move || {
        // driver.play(bounce_tween);
    }));
    let clock = now_secs.clone();
    let action = cooldown_action(cooldown, move || clock.get());
    let mut t = AnimationTriggers::new(&mut registry);
    t.on_click(my_button_node, action);
}
}

Hover, press-release, drag

The triggers cover the three other Disney-principle patterns:

• Hover preview — on_hover_enter / on_hover_leave fade a preview tooltip in/out. Anticipation principle: the tooltip pre-appears as the cursor approaches.
• Press-and-hold — on_press_release toggles state on press, releases on lift. Mute / unmute, momentary buttons.
• Drag-to-trigger — on_drag fires start when the press promotes past the 5px threshold, end on release. Pull-to-refresh, swipe-to-dismiss.

All four wire through the same CallbackRegistry. Drop down to the registry directly if you need access to the full Pointer<E> payload (the event's local_pos, modifier state, etc.).

decode — overview

The decoder side of the recorder. Turns video sources (MP4 files, ScreenCaptureKit captures, …) into a stream of BGRA frames that wisp's VideoTexture::upload_bgra consumes directly.

Why a trait

The recorder uses GStreamer as the single media stack (see stack — and AUT-144 for the "no ffmpeg-next" decision). Multiple implementations still hide behind the trait:

• gstreamer_pipe — CLI-subprocess via gst-launch-1.0. Ships today; powers both the decode integration tests and the playback player.
• gstreamer-rs Rust bindings (future, encode-side) — needed for the appsrc-fed encoder pipeline where wisp pushes BGRA frames into the encoder in-process.
• MockVideoStream — deterministic synthesized frames; no external deps. Used by playback_demo and the wisp story harnesses.

Each backend is a non-trivial integration, but the consumer — wisp's per-frame upload path — is uniform: it wants Vec<u8> BGRA at known dimensions, ticked at known timestamps. VideoStream is that uniform contract.

Current state

• ✅ M-DEC.1 — trait + MockVideoStream (synthesizes scrolling gradient, no external deps, drives the playback_demo example).
• ✅ M-DEC.2 — real MP4 decode via GStreamer CLI-subprocess (gstreamer_pipe).

End-to-end proof point

The playback_demo example pulls 8 frames from MockVideoStream, uploads each through VideoTexture::upload_bgra, renders via wisp's Sprite pipeline, and writes the result to disk.

The motion is the gradient phase-shifting frame to frame — proves the per-frame upload path actually replaces the texture each tick (a static output would be a regression).

Run with:

cargo run -p screen-wisp --example playback_demo

Decode API ref

media — architecture

Linear: AUT-96

The media crate is the home for GStreamer-backed audio + video capture, playback orchestration, and the data models that wisp (the renderer) and app (the Tauri/Leptos shell) consume. It's the M-MEDIA track's foundation — every subsequent ticket (M-MEDIA.1 through M-MEDIA.22) adds to one of its modules.

api

Three-way responsibility split

Boundaries are load-bearing. Crossing them once would bloat every wisp consumer (storybook, headless export, future plugins) with GStreamer's build footprint + license obligations.

Layering

graph TD
    App["<b>app</b><br/>(Tauri shell + Leptos UI)<br/>• calls media::commands::*<br/>• feeds wisp typed data"]
    App --> Media
    App --> Wisp

    subgraph siblings ["sibling crates — no direct dep between them"]
        Media["<b>media</b><br/>• GStreamer<br/>• timing model<br/>• histogram<br/>• manifest"]
        Wisp["<b>wisp</b><br/>• render<br/>• sprite/graphics<br/>• text + mask<br/>• filter + blend"]
    end

    Media -. typed data: VideoFrame,<br/>WaveformBarRect,<br/>MediaTime, … .-> Wisp

Build-on-decode

The decode crate already carries the BGRA-frame contract used throughout the project (VideoFrame, VideoStream, and the existing GstreamerPipeStream CLI-pipe pattern). media builds on top of it — re-exports VideoFrame / VideoStream under video and consumes the CLI-pipe pattern in M-MEDIA.6 / .13 / .16 for video capture and webcam intake.

GStreamer integration choice — CLI-pipe

Spawn gst-launch-1.0 as a child process and pipe raw bytes through fdsrc / fdsink. Not gstreamer-rs.

• Zero compile-time dependency on libgstreamer. Works on any machine with brew install gstreamer / apt install gstreamer1.0-tools, no gst-build setup needed.
• The CLI pipeline doubles as runnable documentation — you can paste it into a terminal.
• Upgrading to gstreamer-rs later is a one-line swap at the call site, because the public surface (VideoStream, AudioStream trait + chunk types) hides the transport.

Lessons captured in CLAUDE.md and the GStreamer-integration project memory: fdsink fd=1 for stdout, rawvideoparse before mp4mux to synthesize PTS, drop-kill the child on shutdown, skip-guard every integration test, include PATH in spawn errors.

Module index

Every cell marked "scaffolded" is a module that exists today, compiles, and contains the planned-surface comment that the next chunk converts into real types + tests + an mdBook chapter of its own.

Track sequencing

The 23 M-MEDIA chunks land on the m-media branch as one big PR. Order is numeric and follows the dependency chain:

• P0 (AUT-96..103) — crate + probe + clock + audio model + mock sources + GStreamer capture (audio + video) + A/V sync harness.
• P1 (AUT-104..110) — histogram → waveform → Wisp render → gst histogram → texture handoff → gst video → synced scene.
• P2 (AUT-111..117) — live mic / webcam / playback harness / cursor / device enumeration / manifest / Leptos seam.
• P3 (AUT-118) — end-to-end smoke.

Media clock + timestamp model

Linear: AUT-98

Every audio chunk, video frame, cursor event, and visualization window in the recorder is stamped against one shared timeline. This module is that timeline's vocabulary.

api

Types

Why nanoseconds (i64)

Internal representation is i64 nanoseconds, signed so a pre-origin offset is representable.

• i64::MAX ns ≈ 292 years — comfortable headroom for any recorder session.
• f64 seconds drops below 1 µs precision past ~10⁹ s; nanoseconds stay exact through arithmetic.
• Integer math for from_sample / to_sample round-trips exactly for any sample rate (44.1 kHz included), thanks to round-half-up in to_sample. Without rounding, 44.1 kHz drifts -1 sample per conversion.

Sample / frame conversions

#![allow(unused)]
fn main() {
use media::clock::MediaTime;

// 30 fps, frame 90 → 3.0 s.
assert!((MediaTime::from_frame(90, 30.0).as_seconds() - 3.0).abs() < 1e-9);

// 48 kHz, sample 48 000 → 1.0 s exactly (nanos-level).
assert_eq!(MediaTime::from_sample(48_000, 48_000).as_nanos(), 1_000_000_000);

// 44.1 kHz round-trips exactly thanks to round-half-up.
let t = MediaTime::from_sample(1, 44_100);
assert_eq!(t.to_sample(44_100), 1);
}

Two clock modes

#![allow(unused)]
fn main() {
use media::clock::{MediaClock, MediaDuration, MediaTime};

// Production — anchored to Instant::now().
let live = MediaClock::wall_clock();
let _t = live.now();

// Tests + headless examples — driven by advance_by, byte-exact reproducible.
let mock = MediaClock::manual(MediaTime::ZERO);
mock.advance_by(MediaDuration::from_millis(20));
assert!((mock.now().as_seconds() - 0.020).abs() < 1e-12);
}

MediaClock::assign(value) attaches the current timestamp:

#![allow(unused)]
fn main() {
use media::clock::{MediaClock, MediaTime, Timestamped};
let clock = MediaClock::manual(MediaTime::from_seconds(2.5));
let ts: Timestamped<&str> = clock.assign("hello");
assert_eq!(ts.value, "hello");
}

The synthetic A/V sync harness (M-MEDIA.7) uses MediaClock::manual so the test's drift assertion is deterministic across hosts. Live capture (M-MEDIA.5/.6) uses MediaClock::wall_clock.

Arithmetic

MediaTime + MediaDuration  → MediaTime      (forward in time)
MediaTime - MediaDuration  → MediaTime      (backward in time)
MediaTime - MediaTime      → MediaDuration  (interval between)
MediaDuration ± MediaDuration → MediaDuration

Saturating arithmetic prevents overflow at the i64 boundary.

MediaDuration::abs() is provided specifically for drift reporting — A/V sync logs typically want the magnitude rather than the signed delta.

Audio data model

Linear: AUT-99

Every audio buffer in the recorder flows through one type: AudioChunk — a timestamped slice of normalized f32 samples with its AudioFormat. The GStreamer capture path (M-MEDIA.5), the deterministic mock sources (M-MEDIA.4), the histogram quantizer (M-MEDIA.8), and the live microphone path (M-MEDIA.15) all produce / consume this type.

api

Why normalized f32

• It's what every downstream visualization wants — AudioHistogram's RMS / peak math runs cleaner on floats than on integers.
• It's what GStreamer's audioconvert ! audio/x-raw,format=F32LE produces natively — capture pipelines don't have to re-quantize.
• Future device-capture backends (cpal, coreaudio-rs) emit f32 as their preferred shape too — no buffer layout churn at the seam.

The SampleFormat enum exists so capture-side code can declare its input layout (F32 / I16 / U8) before normalization. Internally, AudioChunk::samples is always &[f32].

Interleave order — planar-per-frame

Stereo: [L₀, R₀, L₁, R₁, …]. Mono: [s₀, s₁, …]. Matches GStreamer raw-audio, cpal, coreaudio-rs. No re-layout needed at the capture seam.

Validation

[AudioChunk::new] rejects:

• samples.len() % channels != 0 — each frame must carry exactly one sample per channel.
• channels == 0.
• sample_rate == 0.

These are the three "is this a meaningful chunk?" checks. The remaining shape questions (clipping, NaN, DC offset) are visualization concerns, not data-model concerns.

Derived metrics

AudioChunk::peak() and AudioChunk::rms() are pre-computed shortcuts used by M-MEDIA.8 (histogram quantization) and capture-side regression checks. They run in O(n) over the buffer; cache the result if you need it more than once per chunk.

Quick start

#![allow(unused)]
fn main() {
use media::audio::{AudioChunk, AudioFormat};
use media::clock::MediaTime;

let fmt = AudioFormat::mono_f32(48_000);
let samples = vec![0.0_f32; 48_000]; // 1.0 s of silence at 48 kHz.
let chunk = AudioChunk::new(fmt, samples, MediaTime::ZERO).expect("valid");
assert_eq!(chunk.frame_count(), 48_000);
assert!((chunk.duration().as_seconds() - 1.0).abs() < 1e-9);
}

Deterministic mock audio sources

Linear: AUT-100

Three sources that emit timestamped [AudioChunk]s with byte-exact reproducible samples — no microphone, no GStreamer. Every test that wants "this is what audio looks like" without a live device uses one of these.

Left to right: SineWaveSource(440 Hz, A=0.7) — five cycles fitting the panel; SilenceSource — flat zero everywhere; StepPulseSource(120) — a single 1.0 spike at frame 120, silent otherwise. Rendered against a white backdrop per the CLAUDE.md asset-choice rule (audio shape needs light backing to read).

api

The three shapes

Each source advances an internal frame counter on next_chunk(frames) and stamps PTS via [MediaTime::from_sample]. Two successive next_chunk(48_000) calls on a 48 kHz source produce chunks with pts = 0 s and pts = 1 s — no rounding drift.

Why three shapes

The histogram quantizer in M-MEDIA.8 has three correctness assertions:

1. Silence → zero bars. Trivial; SilenceSource is the input.
2. Sine wave → stable RMS. A pure sinusoid at amplitude A has RMS = A/√2 ≈ 0.7071·A. Constant across buckets. SineWaveSource is the input; the test asserts (observed - A/√2).abs() < tolerance.
3. Pulse → expected peak. A single 1.0 spike at frame K should show up as peak == 1.0 exactly in the bucket containing frame K, and peak < epsilon everywhere else. StepPulseSource is the input.

These are the same three properties M-MEDIA.4 tests today, and the same three M-MEDIA.8 will assert against the histogram output. Mock sources let the entire audio-visualization stack be TDD'd before any real audio infrastructure exists.

Quick start

#![allow(unused)]
fn main() {
use media::audio::AudioFormat;
use media::mock_audio::{SineWaveSource, SilenceSource, StepPulseSource};

let fmt = AudioFormat::mono_f32(48_000);

// Sine wave at 440 Hz, amplitude 0.5. RMS ≈ 0.5 / √2.
let mut sine = SineWaveSource::new(fmt, 440.0, 0.5);
let chunk = sine.next_chunk(48_000); // 1 s
assert!((chunk.rms() - 0.5 / 2.0_f32.sqrt()).abs() < 0.01);

// 100 ms of silence.
let mut hush = SilenceSource::new(fmt);
assert!(hush.next_chunk(4_800).peak() < f32::EPSILON);

// Single spike at frame 7.
let mut pulse = StepPulseSource::new(fmt, 7);
let chunk = pulse.next_chunk(16);
assert!((chunk.samples()[7] - 1.0).abs() < f32::EPSILON);
}

GStreamer audio capture

Linear: AUT-101

Spawns gst-launch-1.0 with a pipeline that emits normalized F32LE raw audio on stdout, then chunks the byte stream into AudioChunks with contiguous PTS.

Round-tripped through the bundled MP3 fixture

The bundled 34.9-s sample-audio.mp3 (deterministic 440 Hz sine, generated locally via gstreamer's audiotestsrc + lamemp3enc — license-clean by construction) feeds GstreamerAudioCapture::from_file. After decode you get a stream of AudioChunks carrying normalized f32 samples.

First 100 ms of the decoded MP3. The flat lead-in is the MP3 decoder's priming samples (mpeg-layer-3 has a brief encoder delay); the visible sine cycles afterward are the 440 Hz tone the fixture encodes. Rendered as a min/max envelope per pixel column so the high-frequency content reads as a band rather than aliased noise.

Same audio, quantized at 200 ms buckets across the full 34.9 s. Uniform bars — a pure tone has constant amplitude so every bucket has the same peak and rms. RMS sits ≈ 0.707 × peak (the canonical sinusoid identity A/√2).

api

Two source modes

GstreamerAudioCapture::test_source(format, freq_hz)
  audiotestsrc wave=sine freq=F
    ! audioconvert
    ! audioresample
    ! audio/x-raw,format=F32LE,rate=R,channels=C,layout=interleaved
    ! fdsink fd=1

GstreamerAudioCapture::from_file(path, format)
  filesrc location=PATH
    ! decodebin
    ! audioconvert
    ! audioresample
    ! audio/x-raw,format=F32LE,rate=R,channels=C,layout=interleaved
    ! fdsink fd=1

test_source is the AUT-101 deliverable — deterministic, no external file required. from_file is the companion fixture path that decodes a real audio file (the bundled crates/media/tests/fixtures/sample-audio.mp3, a deterministic 35-s 440 Hz sine).

Real fixtures matter for downstream tests: M-MEDIA.8 (audio histogram) needs to assert numeric correctness on real-world signal, not just mock data. The fixture's pure 440 Hz sine has a known RMS (amplitude / √2) so the assertions stay tight.

Lifecycle

GstreamerAudioCapture owns the child process. Drop kills the child and waits — without this, gst-launch-1.0 keeps decoding into a dropped pipe and burns CPU. Matches the decode::GstreamerPipeStream pattern.

#![allow(unused)]
fn main() {
use media::audio::AudioFormat;
use media::gstreamer_audio::GstreamerAudioCapture;

let fmt = AudioFormat::mono_f32(48_000);
let mut cap = GstreamerAudioCapture::test_source(fmt, 440.0)?;

// Read 100 ms chunks for 1 second.
for _ in 0..10 {
    let chunk = cap.next_chunk(4_800)?;
    println!(
        "pts={:.3}s rms={:.3} peak={:.3}",
        chunk.pts().as_seconds(),
        chunk.rms(),
        chunk.peak(),
    );
}
Ok::<(), media::gstreamer_audio::Error>(())
}

Format support

Only SampleFormat::F32 is supported at construction — the pipeline caps it to F32LE explicitly and converting integer formats at this seam would push sample-format complexity into the public API for no payoff. [crate::audio::SampleFormat::I16] / U8 exist for future capture backends that need to declare an upstream layout, but the public from_file / test_source constructors require F32. A non-F32 format returns Error::UnsupportedFormat.

Integration tests

4 tests in crates/media/tests/gstreamer_audio_integration.rs, each skip-guarded via media::gstreamer::is_available:

Manual regression

just gate                           # runs the integration tests via nextest
                                    # (skips silently if GStreamer absent)
cargo run -p media --example gst_audio_dump  # planned M-MEDIA.11 follow-up

GStreamer video capture

Linear: AUT-102

Spawns gst-launch-1.0 with a videotestsrc-fed pipeline that emits raw BGRA frames on stdout, then chunks the byte stream into VideoFrames — the same type decode::VideoStream returns.

3 seconds of videotestsrc SMPTE colorbars at 320×240, 30 fps. Same content GstreamerVideoCapture::test_source emits frame-by-frame into a VideoFrame stream; the animated ball in the bottom-left corner is videotestsrc's built-in "is this actually moving" indicator. Captured directly via gst-launch-1.0 + x264enc + mp4mux (see just snapshots-media-video) so the chapter carries a concrete artifact independent of the Rust test path.

api

Pipeline

videotestsrc is-live=false
  ! videoconvert
  ! video/x-raw,format=BGRA,width=W,height=H,framerate=F/1
  ! fdsink fd=1

videotestsrc's default pattern is SMPTE colorbars with a tiny animated ball — successive frames carry different bytes, which makes visual smoke checks cheap.

AUT-102 only covers the videotestsrc path. M-MEDIA.16 (live webcam) will add an autovideosrc variant; M-MEDIA.17 (playback harness) will add a filesrc ! decodebin variant that decodes the existing crates/decode/tests/fixtures/sample.mp4.

Quick start

#![allow(unused)]
fn main() {
use media::gstreamer_video::GstreamerVideoCapture;

let mut cap = GstreamerVideoCapture::test_source(640, 360, 30)?;
for _ in 0..30 {
    let frame = cap.next_frame()?;
    println!(
        "frame {} pts={:.4}s {}×{} bgra-bytes={}",
        frame.frame_index,
        frame.pts_seconds,
        frame.width,
        frame.height,
        frame.bgra.len(),
    );
}
Ok::<(), media::gstreamer_video::Error>(())
}

Lifecycle

Drop kills the child + waits. Same pattern as gstreamer_audio and decode::GstreamerPipeStream — a dropped pipe without an explicit kill keeps the gst-launch process decoding into the void.

PTS

PTS is computed from frame_index / framerate via MediaTime::from_frame. At 30 fps, frame 90's PTS is exactly 3.0 s — no rounding drift across long captures (the rounding in MediaTime's sample/frame helpers ensures the integer round-trip stays tight).

Integration tests

3 tests in crates/media/tests/gstreamer_video_integration.rs, all skip-guarded via media::gstreamer::is_available:

A/V sync harness

Linear: AUT-103

Combines GstreamerAudioCapture

• GstreamerVideoCapture into one harness that reports per-stream timing and inter-stream drift. With synthetic audiotestsrc + videotestsrc the assertion stays deterministic — live capture (M-MEDIA.15 / .16) will reuse this exact harness with autoaudiosrc / autovideosrc.

The two streams the harness drives

Video side — videotestsrc SMPTE colorbars (3 s, 320×240, 30 fps).

Audio side — first 100 ms of the bundled 440 Hz MP3 fixture.

The harness pulls audio chunks (audio_chunk_frames per call — 4 800 frames / 100 ms at 48 kHz in the default config) and one video frame per video tick until the configured MediaDuration is covered. Both streams stamp their own PTS via MediaTime; the SyncReport records the end-of-window timestamp for each stream (chunk.pts + chunk.duration for audio, frame.pts + 1/fps for video) so the drift calculation is symmetric.

api

What "drift" means here

Each stream stamps its own PTS via MediaTime::from_sample / MediaTime::from_frame. For synthetic sources, both PTS values are derived from per-stream counters at construction-time rates, so the per-stream PTS is the timeline. The harness reports:

SyncReport::drift_within(tolerance) returns bool for assertion ergonomics. Display formats a compact one-line summary.

Quick start

#![allow(unused)]
fn main() {
use media::sync::{SyncConfig, run};
use media::clock::MediaDuration;

let report = run(SyncConfig::deterministic_1s())?;
println!("{report}");
assert!(report.drift_within(MediaDuration::from_millis(50)));
Ok::<(), media::sync::Error>(())
}

Configuration

SyncConfig::deterministic_1s returns the default: 48 kHz mono audio + 64×36 30 fps video for 1 second. For longer captures (matches the AUT-103 ticket's 5–10 s recommendation):

#![allow(unused)]
fn main() {
use media::audio::AudioFormat;
use media::clock::MediaDuration;
use media::sync::{SyncConfig, run};

let cfg = SyncConfig {
    audio_format: AudioFormat::stereo_f32(48_000),
    audio_frequency_hz: 1_000.0,
    video_width: 640,
    video_height: 360,
    video_framerate: 30,
    audio_chunk_frames: 4_800,
    duration: MediaDuration::from_seconds(5.0),
};
let report = run(cfg)?;
Ok::<(), media::sync::Error>(())
}

Integration tests

4 tests in crates/media/tests/sync_harness_integration.rs, each skip-guarded via media::gstreamer::is_available:

Manual regression

cargo run -p media --example gst_sync_dump  # planned follow-up

The integration tests run a 1-second capture for CI speed; the ticket's recommended 5–10 s manual regression is left as the example above.

Audio histogram quantization

Linear: AUT-104

Turns an AudioChunk into design-friendly rectangle bars for timeline / dope-sheet visualization. Each AudioBar carries its window's start time, duration, peak, and RMS.

Left to right: the same three mock sources as the mock-sources chapter, quantized at 50 ms buckets over 1 s of audio. Outer (light grey) extent = peak; inner (amber) bar = rms. Sine: every bucket carries the same height — constant amplitude. Silence: nothing. Pulse: one bucket carries the spike (peak = 1.0), the rest are zero.

api

Math

Multi-channel chunks collapse to a single bar series — every sample in the interleaved buffer counts toward the same bucket. That matches dope-sheet rendering (one row per audio track, not per channel) and keeps the math + tests simple. M-MEDIA.9 (geometry) handles mono vs stereo display modes.

Bucket size

Default range is 20–50 ms — dope-sheet readability sweet spot (≈ 20–50 bars per second of audio). 10 ms is supported for tests + zoom views.

Bucket-duration arithmetic uses the MediaTime::from_sample round-half-up path, so bar start_time values are exact — bar[i+1] .start = bar[i].start + bar[i].duration for every i. No gap-or-overlap drift across long captures.

Quick start

#![allow(unused)]
fn main() {
use media::audio::{AudioChunk, AudioFormat};
use media::clock::{MediaDuration, MediaTime};
use media::histogram::quantize;
use media::mock_audio::SineWaveSource;

let fmt = AudioFormat::mono_f32(48_000);
let mut src = SineWaveSource::new(fmt, 1_000.0, 0.6);
let chunk = src.next_chunk(48_000);                       // 1 s
let h = quantize(&chunk, MediaDuration::from_millis(50)); // 20 bars
let expected = 0.6 / 2.0_f32.sqrt();
for bar in h.bars.iter().skip(1).take(18) {
    assert!((bar.rms - expected).abs() < 0.05);
}
}

Three correctness assertions

M-MEDIA.4 (mock sources) lined up the three reference signals; this chunk's tests verify the corresponding three histogram behaviors:

The same three mock sources will drive M-MEDIA.10 (Wisp render) and M-MEDIA.11 (gst→histogram example) regression tests, so the correctness chain is uniform across the audio-visualization stack.

Tests

10 unit tests in crates/media/src/histogram.rs::tests. Bucket counts at 10 ms / 20 ms / 50 ms, silence → zero bars, sine → stable RMS, pulse → singular peak, empty chunk → empty histogram, contiguous bar timestamps, stereo collapses to mono bar series, Send + Sync. Full just gate green at 376 tests (366 + 10 new).

Waveform bar geometry

Linear: AUT-105

Maps an AudioHistogram to a list of axis-aligned rectangles that wisp's graphics pipeline can render directly.

api

Data flow

sequenceDiagram
    participant Src as AudioChunk source
    participant Hist as histogram::quantize
    participant Geom as waveform::mono_bars / stereo_bars
    participant Wisp as wisp::Graphics

    Src->>Hist: AudioChunk (PTS, samples)
    Hist->>Geom: AudioHistogram (peak + rms per bucket)
    Geom->>Wisp: Vec<WaveformBarRect>
    Wisp->>Wisp: graphics.rect(x, y, w, h) per bar

The histogram carries timing on a media timeline (each bar has start_time + duration); the geometry stage drops timing and lays bars out by index, with bar_width + bar_gap between adjacent left edges. Timeline-aligned layout (dope-sheet, scrubber) is the caller's job — this module is unit-agnostic.

Coordinate convention

Rectangles use a y-up convention (matching wisp NDC): x / y is the bottom-left corner, width / height are non-negative. Layout values use whatever unit the caller picks — NDC [-1, +1], screen pixels, normalized [0, 1]. The math doesn't care.

Two display modes

Quick start

#![allow(unused)]
fn main() {
use media::{
    audio::AudioFormat, clock::MediaDuration, histogram::quantize,
    mock_audio::SineWaveSource, waveform::{mono_bars, WaveformLayout},
};

let mut src = SineWaveSource::new(AudioFormat::mono_f32(48_000), 440.0, 0.6);
let chunk = src.next_chunk(48_000);                       // 1 s
let h = quantize(&chunk, MediaDuration::from_millis(50)); // 20 bars
let rects = mono_bars(&h, &WaveformLayout::ndc_default());
// rects.len() == 20; each rect.height == bar.peak * 0.4
}

Manual regression — four-bar table

A test crafts a 4-bar histogram with peak = [1.0, 0.5, 0.25, 0.0], lays it out anchored at y = 0, bar_width = 0.1, bar_gap = 0.02, max_height = 1.0, origin_x = 0.0. Expected geometry:

Stride = bar_width + bar_gap = 0.12. Height = peak × max_height. The bar-4 zero-height rect is preserved (geometry stays parallel to histogram order) so downstream renderers can use stable indices.

Next

Render synthetic audio histogram in Wisp takes this geometry list and draws it through wisp::Graphics for the first time.

Audio histogram in Wisp

Linear: AUT-106

The seam works. media quantizes audio into an AudioHistogram, media::waveform::mono_bars lays it out as rectangle geometry, and wisp's graphics pipeline draws those rectangles. Wisp never imports media.

A 440 Hz sine at amplitude 0.6, sampled at 48 kHz for 1 second, quantized at 50 ms (20 bars), mirrored about the centerline. Every bar is roughly the same height — that's the constant-amplitude sine made visible.

End-to-end path

sequenceDiagram
    participant Mock as media::SineWaveSource
    participant Hist as media::histogram::quantize
    participant Geom as media::waveform::mono_bars
    participant Wisp as wisp::Graphics

    Mock->>Hist: AudioChunk (1s, 48 kHz mono)
    Hist->>Geom: AudioHistogram (20 × peak, rms)
    Geom->>Wisp: Vec<WaveformBarRect>
    Wisp->>Wisp: graphics.draw_rect(x, y, w, h) ×20

Story code (excerpt)

#![allow(unused)]
fn main() {
use media::{audio::AudioFormat, clock::MediaDuration,
            histogram::quantize, mock_audio::SineWaveSource,
            waveform::{mono_bars, BarMetric, WaveformDisplayMode, WaveformLayout}};
use wisp::{Color, Fill, Graphics, math::Rect};

let mut src = SineWaveSource::new(AudioFormat::mono_f32(48_000), 440.0, 0.6);
let chunk = src.next_chunk(48_000);                       // 1 s
let histogram = quantize(&chunk, MediaDuration::from_millis(50)); // 20 bars

let layout = WaveformLayout {
    origin_x: -0.85, baseline_y: 0.0,
    bar_width: 0.075, bar_gap: 0.012,
    max_height: 1.1,
    color: [1.0, 0.74, 0.30, 1.0],
    metric: BarMetric::Peak,
    mode: WaveformDisplayMode::Mirrored,
};
let rects = mono_bars(&histogram, &layout);

let mut bars = Graphics::new();
let [r, g, b, a] = layout.color;
bars.fill(Fill::Solid(Color::rgba(r, g, b, a)));
for rect in &rects {
    bars.draw_rect(Rect::new(rect.x, rect.y, rect.width, rect.height));
}
}

The full story is in crates/wisp-storybook/src/stories/s_audio_histogram.rs.

Next

GStreamer audio → histogram example swaps the mock source for a real GStreamer-captured audio chunk so the same renderer can draw real microphone input.

GStreamer audio → histogram example

Linear: AUT-107

Closes the loop: the same histogram::quantize that drove the synthetic Wisp story accepts real-shape chunks from a GStreamer audiotestsrc. Swap audiotestsrc for a microphone (M-MEDIA.15) and the rest of the pipeline doesn't move.

What the example does

sequenceDiagram
    participant Probe as media::gstreamer::is_available
    participant Gst as gst-launch-1.0 audiotestsrc
    participant Cap as GstreamerAudioCapture
    participant Hist as histogram::quantize
    participant Out as stdout

    Probe-->>Probe: PATH lookup
    alt missing
        Probe-->>Out: skip message, exit 0
    else available
        Cap->>Gst: spawn pipeline (audiotestsrc, freq=440, 48 kHz f32 mono)
        Gst->>Cap: 1 s of raw PCM via fdsink
        Cap->>Hist: AudioChunk (48 000 frames, pts = 0)
        Hist->>Out: bucket / peak / rms summary
    end

Run

cargo run -p media --example gst_audio_histogram

Sample output on a machine with GStreamer installed:

GStreamer audiotestsrc → AudioHistogram
  source        : audiotestsrc freq=440.0 Hz
  format        : 1 ch, 48000 Hz, f32 LE
  chunk         : 48000 frames, pts = 0 ns
  bucket / bars : 50 ms × 20 bars
  peak max      : 0.8000
  rms  min..max : 0.5657 .. 0.5657
  first 5 bars  :
    pts=         0 ns  peak=0.8000  rms=0.5657
    pts=  50000000 ns  peak=0.8000  rms=0.5657
    …

Manual regression

Run the example and verify:

Next

Wisp media texture handoff for video frames — the video-side equivalent of this audio path. Takes a captured VideoFrame and uploads it to a wisp Texture so a Sprite can sample it.

Video texture handoff

Linear: AUT-108

The audio side of the seam (M-MEDIA.9–11) emits Vec<WaveformBarRect> geometry; the video side emits raw BGRA bytes. wisp's VideoTexture already knew how to receive them — this chunk formalizes the call site as a story so future code can copy the pattern.

A 128×72 synthetic VideoFrame (diagonal gradient + horizontal stripes) uploaded to a VideoTexture and drawn through the standard Sprite pipeline.

Handoff path

sequenceDiagram
    participant Src as decode::VideoFrame
    participant Tex as wisp::VideoTexture
    participant Spr as wisp::Sprite
    participant Gpu as wgpu Queue

    Src->>Tex: VideoTexture::new(app, w, h)
    Src->>Gpu: VideoTexture::upload_bgra(app, frame.bgra)
    Tex->>Spr: Sprite::from_texture(tex.texture().clone())
    Spr->>Spr: scene.add_child(root, sprite)

Story code (excerpt)

#![allow(unused)]
fn main() {
use media::VideoFrame;
use wisp::texture::video_texture::VideoTexture;
use wisp::Sprite;

let video_tex = VideoTexture::new(app, frame.width, frame.height);
video_tex.upload_bgra(app, &frame.bgra);

let mut sprite = Sprite::from_texture(video_tex.texture().clone());
sprite.container.transform.scale = Vec2::new(1.5, 1.5 * 72.0 / 128.0);
let _ = stage.add_child(stage.root(), sprite);
}

Full story: s_video_frame_handoff.rs.

Determinism

The synthetic frame depends only on (width, height), so the rendered PNG is identical run-to-run on the same GPU. Story is covered by story_smoke (no validation errors + visible pixels) and story_fingerprints (quadrant snapshot). Upload-orientation, format-byte-order, and colorspace regressions trip the snapshot because the gradient + stripe pattern has enough per-quadrant variance to detect them.

Next

GStreamer video test source through Wisp — swaps the synthetic frame for a real videotestsrc frame served via the existing M-MEDIA.6 capture pipeline.

GStreamer videotestsrc through Wisp

Linear: AUT-109

Real GStreamer frames through the same VideoTexture path that M-MEDIA.12 stood up with synthetic frames. Closes the loop: the recorder's video intake (M-MEDIA.6) feeds wisp's rendering pipeline (M-MEDIA.12) end-to-end.

Frame 0 of videotestsrc's default SMPTE-colorbars pattern at 320×180, captured via the M-MEDIA.6 CLI-pipe wrapper, uploaded to a wisp VideoTexture, and rendered through Sprite.

Run

cargo run -p media --example gst_video_to_wisp

Eight frames at 30 fps are captured and rendered. Output lands under target/gst-video-frames/frame_NN.png with a per-frame PTS log:

frame 00: pts = 0.000 s (         0 ns), index = 0
frame 01: pts = 0.033 s (  33333333 ns), index = 1
frame 02: pts = 0.067 s (  66666667 ns), index = 2
…
frame 07: pts = 0.233 s ( 233333333 ns), index = 7

Summary: 8 frames captured, 8 VideoTexture uploads,
         8 cumulative gst frames emitted.

The wiring

sequenceDiagram
    participant Gst as gst-launch-1.0 videotestsrc
    participant Cap as GstreamerVideoCapture
    participant Tex as wisp::VideoTexture
    participant Ren as wisp::Renderer
    participant Out as PNG

    loop frame i ∈ [0, 8)
        Gst->>Cap: BGRA frame via fdsink
        Cap->>Tex: upload_bgra(app, frame.bgra)
        Tex->>Ren: Stage with Sprite(tex)
        Ren->>Out: render_target.read_pixels → PNG
    end

Manual regression

After running, verify:

Real-webcam capture (M-MEDIA.16) drops in here without any of the wisp-side code changing. The video texture is format-agnostic at the contract level — caps to BGRA on the GStreamer side, upload, render.

Next

Synced video + audio histogram in one Wisp scene — combines this video render with the M-MEDIA.10 audio histogram so the two shows the recorder's two intake streams composed against one timeline.

Synced video + audio in one scene

Linear: AUT-110

The M-MEDIA P1 capstone. Video frames and an audio histogram render in the same wisp scene, anchored to a single MediaClock. Every seam shipped in M-MEDIA.0 through M-MEDIA.13 shows up at the same call site:

• MediaClock::manual (M-MEDIA.2) drives the timeline.
• VideoFrame (M-MEDIA.12) uploads BGRA to a wisp VideoTexture.
• SineWaveSource → AudioChunk → quantize (M-MEDIA.8) produces the histogram.
• mono_bars (M-MEDIA.9) lays the histogram out as rectangles.
• wisp::Graphics and wisp::Sprite draw both layers in one pass.

Frame 5 of 10. Video occupies the top half (a hue-rotating gradient that changes per frame); audio bars sit on the bottom (amber, scaled to the chunk's peak). The amplitude ramps 0.30 → 0.90 over the run, so bar heights grow monotonically frame-to-frame.

Per-frame loop

sequenceDiagram
    participant Clk as MediaClock::manual
    participant Aud as SineWaveSource + quantize
    participant Vid as synth_frame_bgra + VideoTexture
    participant Stg as wisp::Stage
    participant Out as PNG

    loop frame i ∈ [0, 10)
        Clk-->>Aud: t = clock.now()
        Clk-->>Vid: same t
        Aud->>Stg: histogram → mono_bars → Graphics rects
        Vid->>Stg: BGRA upload → Sprite
        Stg->>Out: renderer.render_stage → read_pixels → PNG
        Clk-->>Clk: clock.advance_by(100ms)
    end

Run

cargo run -p media --example synced_scene

frame 00: video.pts = 0.000 s | hist.window = [0.000 s, +20 ms ×  5 bars] amp=0.30 peak=0.300 rms=0.214
frame 01: video.pts = 0.100 s | hist.window = [0.100 s, +20 ms ×  5 bars] amp=0.37 peak=0.367 rms=0.261
…
frame 09: video.pts = 0.900 s | hist.window = [0.900 s, +20 ms ×  5 bars] amp=0.90 peak=0.900 rms=0.642

Ten 640×360 PNGs land under target/synced-scene/frame_NN.png.

Reproducibility

Manual regression

What this unlocks

Live capture (M-MEDIA.15 / .16) swaps the synthetic sources for real hardware-backed ones without changing the wisp side of the call. Editor-UI integration (M-MEDIA.21) wires the same composition into the Tauri shell. The scene shape doesn't change — only the data sources do.

playback — overview

The middle layer between decode and wisp. Player owns a boxed VideoStream and a VideoTexture, and pumps decoded frames into the GPU at the source's frame rate while the shell ticks it once per render frame.

Contract

sequenceDiagram
    participant Shell as Shell<br/>(Tauri / winit)
    participant Player
    participant Stream as Box&lt;dyn VideoStream&gt;<br/>(decode crate)
    participant Texture as VideoTexture<br/>(wisp crate)
    participant Sprite as wisp::Sprite

    loop once per render frame
        Shell ->> Player: tick(dt)
        Note over Player: if Playing,<br/>elapsed += dt
        loop while elapsed ≥ next_due
            Player ->> Stream: next_frame()
            Stream -->> Player: VideoFrame (BGRA)
            Player ->> Texture: upload_bgra(&frame.bgra)
            Note over Player: next_due +=<br/>1.0 / frame_rate()
        end
        Player -->> Shell: frames_uploaded: u32
        Note over Shell: redraw only when<br/>frames_uploaded > 0
        Sprite ->> Texture: sample (on screen)
    end

tick returns the number of frames it actually uploaded so the shell can drive a redraw signal off it (no re-render needed when no new frame is due).

Transport

Anti-regression contract (tests in tests/timing.rs)

• Paused player does not advance — elapsed == 0 after a 1 s tick.
• First tick uploads the t=0 frame even with a 1 ms dt (no off-by-one causing the first frame to be skipped).
• 1 s of wallclock at 60 Hz render against a 30 fps source pulls ~30 frames (29..=31 inclusive — boundary tolerance is documented).
• Stream exhaustion transitions to Ended cleanly.
• Pause freezes both the state and the texture (next tick does not re-upload the held frame).
• duration_hint matches frame_count / frame_rate.

Visual proof — 30 ticks of the timed_playback example

Run with:

cargo run -p playback --example timed_playback

Each tick where the player uploaded a frame is captured below — that's the gradient phase advancing across 1 s of wallclock as the player catches the timestamps that come due.

Player API · PlayState

Real MP4 → wisp playback

This is the first chunk that puts the complete data path together:

sequenceDiagram
    participant File as sample.mp4
    participant Discoverer as gst-discoverer-1.0
    participant Launch as gst-launch-1.0<br/>(filesrc → decodebin →<br/>videoconvert → BGRA → fdsink)
    participant Stream as GstreamerPipeStream<br/>(impl VideoStream)
    participant Player
    participant Texture as VideoTexture
    participant Renderer as wisp::Sprite +<br/>Renderer
    participant Output as RenderTexture → PNG

    File ->> Discoverer: probe
    Discoverer -->> Player: width / height / fps / duration
    File ->> Launch: decode
    Launch -->> Stream: raw BGRA bytes on stdout
    Stream ->> Player: next_frame() → VideoFrame
    Player ->> Texture: upload_bgra(frame)
    Player ->> Renderer: render_stage(stage)
    Renderer ->> Output: read_pixels → PNG

How to run

The committed test fixture ships with the repo, so the example is runnable with no setup beyond brew install gstreamer:

cargo run -p playback --example play_file

Custom video:

cargo run -p playback --example play_file -- /path/to/video.mp4

Output (running against tests/fixtures/sample.mp4)

The fixture is the M-DEC.1 mock-stream gradient, encoded once with x264 into an 11 KB MP4. The example pulls it back through GStreamer + Player

• wisp and writes one PNG per render-tick where a frame was uploaded.

The first tick pulls two frames (catch-up — t=0 and t=1/30 s are both due before the wallclock has advanced past either), then it settles to one frame per tick. The Player exits to Ended cleanly when the GStreamer pipe returns EOF.

Why GStreamer (and why CLI not crate)

GStreamer is LGPL and modular: decodebin auto-selects the best codec plugin for the file, including hardware decoders (vtdec on macOS, vah264dec on Linux, nvh264dec with NVIDIA). Switching backends never touches our code.

The CLI-pipe approach trades a fork for zero compile-time integration. For the player loop that's one fork for the whole stream, not per-frame — overhead is amortised. The Rust-bound integration (gstreamer-rs) is queued as M-DEC.3+; the VideoStream trait makes that swap a one-line change at the call site.

What's now possible

With this chunk landed, the recorder has 4 of 6 stages on the path to "first MP4 plays in Tauri-Leptos via wisp":

• ✅ M-DEC.1 — VideoStream trait + MockVideoStream
• ✅ M-PLAY.1 — Player state machine + frame pump
• ✅ M-DEC.2 — GstreamerPipeStream (real MP4 decode)
• ✅ M-INT.1 — Trunk + Leptos in Tauri (replace the vanilla JS frontend)
• ✅ M-INT.2 — Tauri serves Trunk bundle + OS file-drop wiring
• ✅ M-PREVIEW.1 — Native winit sibling window with the wisp surface
• ✅ M-PLAY.2 — Tauri↔player IPC for transport controls

GstreamerPipeStream · Player · Decode overview · Player overview

preview — overview

A native winit window with a wgpu surface that wisp renders into. Plays an MP4 end-to-end through the decode → playback → wisp stack:

sequenceDiagram
    participant Window as winit::Window
    participant Wgpu as wgpu Instance /<br/>Adapter / Device / Queue
    participant App as wisp::Application<br/>(from_wgpu)
    participant Surface as wgpu::Surface
    participant Player
    participant Stage
    participant Renderer

    Note over Window,Wgpu: boot
    Window ->> Wgpu: create_surface(window)
    Wgpu ->> App: from_wgpu(instance, adapter, device, queue)
    Wgpu ->> Surface: configure(width, height, format)

    loop Per RedrawRequested
        Window ->> Player: tick(dt)
        Note over Player: uploads next frame<br/>to VideoTexture
        Player ->> Stage: build one-Sprite scene<br/>(centered, aspect-fit)
        Stage ->> Renderer: render_stage(surface_view)
        Renderer ->> Surface: surface_texture.present()
    end

Why it exists

The recorder's editor view is a native sibling window — not the Tauri webview — because GPU-accelerated 4K video preview behind WebKit's compositor isn't a viable path. preview proves the contract: wisp can attach to a host-supplied wgpu device and render every frame into the host's surface.

Application::from_wgpu is the seam. The Tauri shell will wire two separate windows (the WebKit-backed shell window + a winit-backed preview window) and hand wisp the wgpu objects from the latter.

Run it

# Default fixture (committed test MP4):
cargo run -p preview

# Custom video:
cargo run -p preview -- /path/to/video.mp4

The window title is screen — preview. Close it to exit; playback will also auto-exit when the stream reaches EOF.

Headless asset generator

render_offscreen is the same pipeline driven against a RenderTexture instead of a winit surface. Used both as the chapter's PNG source and as a CI-safe smoke test for the from_wgpu codepath:

cargo run -p preview --example render_offscreen

Anti-regression

• aspect_fit_scale is unit-tested for matching/wider/taller/zero cases.
• tests/render_smoke.rs exercises the full Application::from_wgpu → Player → Renderer → RenderTexture path and asserts the rendered output is non-trivial (not the clear color, not uniform).

Native winit window

The fifth chunk on the path to "first MP4 plays in the recorder via wisp". Builds a real OS window via [winit] 0.30 and hands the wgpu surface to wisp through [Application::from_wgpu].

What landed

• crates/preview/ — new crate with both a [lib] (pure helpers) and a [[bin]] (the winit app).
• Application::from_wgpu is the seam between embedding host (winit, in the future a winit child of Tauri) and wisp's Renderer.
• The window auto-sizes to the source video (clamped to a 640×360 minimum so it isn't comically small for tiny test fixtures).
• The sprite is centered and aspect_fit_scale-letterboxed against the current surface dimensions, so resizing the window keeps the video proportional.

Visual proof — render_offscreen example

The example rebuilds the same render path against an offscreen RenderTexture (no winit window required), so it runs in CI and produces a deterministic asset:

cargo run -p preview --example render_offscreen

Five frames into a 800×450 surface (16:9), with the 480×270 fixture letterboxed to fit.

What you're looking at. The committed sample.mp4 test fixture is a deterministic synthetic gradient (the M-DEC.1 mock-stream encoded once with x264 into an 11 KB MP4) — the visible "gradient look" is the fixture content, not a rendering bug. The chapter's claim is that the winit → wgpu → Application::from_wgpu → Player::tick → Renderer path delivers decoded frames into the surface; the horizontal-phase advance frame-to-frame is the proof. For a more representative example of decoded-video output, see the media chapters which use videotestsrc SMPTE colorbars.

app-ui — overview

The Leptos CSR app that becomes the recorder's HTML/UI layer. This is the chunk where the components workshopped in ui-storybook leave their gallery and live in a real shell.

Architecture

graph LR
    App["<b>crates/app/</b><br/>Tauri shell (Rust binary)<br/>tauri.conf.json<br/>frontendDist → app-ui/dist (M-INT.2)"]
    AppUi["<b>crates/app-ui/</b><br/>Leptos CSR app (WASM)<br/>Trunk-built dist/<br/>Mounts &lt;App&gt; to &lt;body&gt;"]
    Storybook["<b>crates/ui-storybook/</b><br/>Component library<br/>SSR snapshot tests<br/>visual gallery"]

    App -->|frontendDist points at| AppUi
    AppUi -->|reuses DropZone,<br/>PlayerControls, RecordingToolbar,<br/>StatusBar, Card, DopeSheet| Storybook

Three layers, three crates, one direction of dependency. Adding a new component once means it becomes available in:

1. The storybook gallery (regression-tested via SSR + insta).
2. The shell app (consumed via use ui_storybook::components::*).
3. The mdBook chapter (assets regenerated via just snapshots-ui).

Current state — M-INT.1

Pure UI composition, no Tauri IPC, no real file ingestion:

• RecordingToolbar at the top showing the idle state.
• DropZone (idle) as the main surface — clicking it flips a Leptos signal into the loaded view.
• PlayerControls (paused) + a placeholder gradient surface where the wisp render output will eventually go (M-PREVIEW.1).
• StatusBar (ready) at the bottom.

No real wiring yet — the click → loaded transition is a demo affordance that lets reviewers exercise both views before the actual file-drop event lands in M-INT.2.

How to run

just app-ui          # dev server with hot reload, opens browser
just app-ui-build    # production build → crates/app-ui/dist/

just app-ui-build produces a static bundle that the Tauri shell will serve verbatim once M-INT.2 wires tauri.conf.json to point at it.

Roadmap

• ✅ M-INT.1 — this chunk; Trunk builds, components render, gate green.
• ⏳ M-INT.2 — Tauri serves the Trunk bundle; OS file-drop events flip the loaded signal.
• ⏳ M-PREVIEW.1 — native winit sibling window with the wisp surface rendering the active video frame.
• ⏳ M-PLAY.2 — Tauri↔player IPC for transport (load / play / pause / seek dispatched from Leptos signals to the native player loop).

Tauri ↔ Leptos integration (M-INT.2)

The chunk that joins the Leptos shell to the Tauri webview, then forwards OS-level drag-drop events into a Leptos signal.

Data flow

sequenceDiagram
    participant OS
    participant Shell as crates/app/src/main.rs<br/>(Tauri shell)
    participant Bridge as crates/app-ui/index.html<br/>(JS bridge)
    participant App as crates/app-ui/src/app.rs<br/>(Leptos)
    participant View as &lt;App&gt; view

    OS ->> Shell: drag-drop event
    Note over Shell: on_window_event(<br/>WindowEvent::DragDrop)
    Shell -->> Bridge: window.emit("file-dropped", path)
    Note over Bridge: __TAURI__.event.listen
    Bridge -->> App: window.dispatchEvent(<br/>CustomEvent "file-dropped")
    Note over App: install_file_drop_listener()<br/>addEventListener
    App ->> App: set_loaded.set(Some(path))
    App -->> View: swap drop-zone view → player view

Every hop is a one-liner. No tauri-sys crate, no JS-side state — the bridge is one addEventListener per direction.

Tauri side (crates/app/src/main.rs)

#![allow(unused)]
fn main() {
.on_window_event(|window, event| {
    if let WindowEvent::DragDrop(DragDropEvent::Drop { paths, .. }) = event
        && let Some(path) = paths.first()
    {
        let payload = path.to_string_lossy().into_owned();
        if let Err(err) = window.emit("file-dropped", payload) {
            eprintln!("failed to emit file-dropped event: {err}");
        }
    }
})
}

Tauri 2's WindowEvent::DragDrop fires automatically when tauri.conf.json has "dragDropEnabled": true (the default for our window).

JS bridge (crates/app-ui/index.html)

<script>
  window.addEventListener("DOMContentLoaded", () => {
    if (window.__TAURI__ && window.__TAURI__.event) {
      window.__TAURI__.event.listen("file-dropped", (event) => {
        window.dispatchEvent(new CustomEvent("file-dropped", {
          detail: event.payload
        }));
      });
    }
  });
</script>

Why a CustomEvent instead of calling Leptos directly: it keeps the WASM bundle dependency-free of Tauri's JS API. Leptos's web-sys listener works against any browser; the bridge degrades to a no-op when window.__TAURI__ is absent (e.g. running under trunk serve for component review).

Leptos side (crates/app-ui/src/app.rs)

#![allow(unused)]
fn main() {
fn install_file_drop_listener(set_loaded: WriteSignal<Option<String>>) {
    let Some(window) = web_sys::window() else { return; };
    let closure = Closure::wrap(Box::new(move |event: web_sys::Event| {
        if let Ok(ce) = event.dyn_into::<CustomEvent>()
            && let Some(path) = ce.detail().as_string()
        {
            set_loaded.set(Some(path));
        }
    }) as Box<dyn FnMut(_)>);
    let _ = window.add_event_listener_with_callback(
        "file-dropped",
        closure.as_ref().unchecked_ref(),
    );
    closure.forget();   // app-lifetime listener — never removed
}
}

Closure::forget is intentional: the listener lives for the whole app lifetime. Dropping it would silently de-register the handler.

Tauri config (tauri.conf.json)

{
  "build": {
    "frontendDist": "../app-ui/dist",
    "beforeDevCommand": "cd ../app-ui && trunk serve --port 1420",
    "beforeBuildCommand": "cd ../app-ui && trunk build --release",
    "devUrl": "http://localhost:1420"
  }
}

cargo tauri dev automatically runs trunk serve in crates/app-ui/ and waits for it on port 1420 before opening the webview. cargo tauri build runs trunk build --release first; the resulting crates/app-ui/dist/ becomes the bundled webview content.

How to run

# Dev — hot-reload Leptos via Trunk + reload the Tauri webview on save.
cd crates/app && cargo tauri dev

# Production — Trunk build + Tauri build, single binary.
cd crates/app && cargo tauri build

Drop any file on the running window; the path appears under "Preview surface · " in the player view.

Drag-over visual feedback (M-POLISH.1)

Tauri 2 fires four DragDropEvent variants — Enter, Over, Drop, Leave. M-INT.2 only handled Drop. M-POLISH.1 adds Enter and Leave so the drop-zone shows the user "yes, this drop will land here":

#![allow(unused)]
fn main() {
// crates/app/src/main.rs
match drag {
    DragDropEvent::Enter { .. } => emit("file-drag-enter"),
    DragDropEvent::Leave => emit("file-drag-leave"),
    DragDropEvent::Drop { paths, .. } => {
        emit("file-dropped", path);
        emit("file-drag-leave");  // reset visual after drop
    }
    _ => {}  // Over and any future variants
}
}

The JS bridge re-emits as browser CustomEvents (parallel to file-dropped / player-status). Leptos installs an is_dragging signal that flips on enter/leave; the existing DropZoneState::Active variant the storybook already shipped (M-UI.1) provides the visual.

Tier-2 e2e (crates/app-e2e/tests/golden_path.rs) uses __test_drag_enter and __test_drag_leave debug-only commands — parallel to __test_drop_file since WebDriver can't synthesize OS-level drag events.

App-shell visual references

The drop-zone surface (idle) at boot — exactly what the user sees before any file is dropped:

DropZone story · Editor mock composition

Player IPC — Tauri commands + status events (M-PLAY.2)

The chunk that lifts playback::Player out of standalone-binary territory and behind a Tauri IPC surface. The Leptos shell can now drive a real Rust player from a button click, and the player can push state changes back without polling.

Data flow

sequenceDiagram
    participant UI as Leptos UI<br/>(transport buttons)
    participant Bridge as __TAURI__.core<br/>(JS bridge)
    participant Cmd as Tauri commands<br/>(crates/app)
    participant Session as PlayerSession<br/>(playback crate)
    participant Tick as Tick thread

    UI ->> Bridge: PlayerControls on_toggle (invoke)
    Bridge ->> Cmd: player_play / player_pause
    Cmd ->> Session: .play() / .pause()

    UI ->> Bridge: DropZone file path (invoke)
    Bridge ->> Cmd: player_open
    Cmd ->> Session: .open(path)

    rect rgb(245, 245, 250)
        Note over Tick: every 33 ms
        Tick ->> Session: .tick(dt) → .status()
        Session -->> Cmd: PlayerStatus
        Cmd -->> Bridge: emit("player-status")
        Bridge -->> UI: CustomEvent
    end

Every IPC hop is a one-liner. The bridge in index.html exposes three top-level helpers (__screenOpen / __screenPlay / __screenPause) and re-emits Tauri's player-status event as a browser CustomEvent. No tauri-sys crate; the WASM bundle stays dependency-free of Tauri's JS API.

Tauri commands (crates/app/src/commands.rs)

#![allow(unused)]
fn main() {
#[tauri::command]
pub fn player_open(state: State<'_, PlayerSession>, path: String) -> Result<PlayerStatus, String> {
    state.open(&PathBuf::from(path))
}

#[tauri::command]
pub fn player_play(state: State<'_, PlayerSession>) { state.play(); }

#[tauri::command]
pub fn player_pause(state: State<'_, PlayerSession>) { state.pause(); }

#[tauri::command]
pub fn player_status(state: State<'_, PlayerSession>) -> PlayerStatus { state.status() }
}

The four commands are thin wrappers around PlayerSession. The session itself is pure Rust (no Tauri types), so its lifecycle is testable end-to-end without booting Tauri — see crates/app/tests/player_session.rs (6 tests covering empty/open/play/ pause/tick/error paths).

Tick thread (crates/app/src/main.rs)

#![allow(unused)]
fn main() {
fn spawn_tick_thread(app_handle: tauri::AppHandle) {
    thread::spawn(move || {
        let mut last: Option<PlayerStatus> = None;
        loop {
            thread::sleep(TICK_INTERVAL);                  // 33 ms
            let session = app_handle.state::<PlayerSession>();
            session.tick();
            let status = session.status();
            if status_changed(last.as_ref(), &status) {
                let _ = app_handle.emit("player-status", &status);
                last = Some(status);
            }
        }
    });
}
}

status_changed throttles emits to:

• every state transition (Empty → Paused → Playing → Ended),
• every 100 ms of elapsed_ms change while playing (10 Hz UI updates).

A 33 ms tick × always-emit would be 30 events / sec hitting the webview; the 10 Hz throttle keeps the IPC bandwidth flat at the cost of slightly choppy timer animation. The play/pause UI flip is still instantaneous because the state change emits immediately.

JS bridge (crates/app-ui/index.html)

<script>
  window.addEventListener("DOMContentLoaded", () => {
    if (window.__TAURI__?.event) {
      window.__TAURI__.event.listen("player-status", (event) => {
        window.dispatchEvent(new CustomEvent("player-status", {
          detail: event.payload
        }));
      });
    }
  });

  window.__screenOpen  = (path) => window.__TAURI__?.core?.invoke("player_open", { path });
  window.__screenPlay  = ()     => window.__TAURI__?.core?.invoke("player_play");
  window.__screenPause = ()     => window.__TAURI__?.core?.invoke("player_pause");
</script>

Outbound is a thin core.invoke wrapper; inbound is the same CustomEvent re-emit pattern M-INT.2 introduced for file-dropped. Both directions degrade to no-ops when window.__TAURI__ is absent — a trunk serve dev session against the standalone Leptos shell still flips the drop-zone-to-player view via the demo affordance.

Leptos side (crates/app-ui/src/player_ipc.rs + app.rs)

#![allow(unused)]
fn main() {
#[wasm_bindgen]
extern "C" {
    #[wasm_bindgen(js_namespace = window, js_name = "__screenOpen", catch)]
    pub fn screen_open(path: &str) -> Result<JsValue, JsValue>;
    #[wasm_bindgen(js_namespace = window, js_name = "__screenPlay", catch)]
    pub fn screen_play() -> Result<JsValue, JsValue>;
    #[wasm_bindgen(js_namespace = window, js_name = "__screenPause", catch)]
    pub fn screen_pause() -> Result<JsValue, JsValue>;
}

pub fn install_player_status_listener(set_status: WriteSignal<PlayerStatus>) {
    /* CustomEvent listener — same shape as install_file_drop_listener */
    /* parses CE.detail() via serde-wasm-bindgen, calls set_status.set */
}
}

The PlayerStatus and SessionState types are mirrored on the Leptos side (Deserialize matches the Rust-side Serialize's rename_all = "lowercase" form). The mirror lives in crates/app-ui/src/player_ipc.rs and must stay in sync with crates/app/src/player_session.rs — they're a contract pair.

The transport buttons in <PlayerView> are wrapped in a reactive closure that re-renders <PlayerControls> whenever player_status changes, with an on_toggle: Callback<()> that picks screen_play or screen_pause based on the current state.

Testable surface (crates/app/tests/player_session.rs)

• empty_session_reports_empty — fresh session, nothing loaded.
• open_transitions_to_paused_with_metadata — open the test fixture, assert width/height/fps come through.
• play_pause_lifecycle — round-trip play/pause/play.
• tick_advances_elapsed_when_playing — confirms wallclock pumping.
• tick_is_noop_when_empty — guard for the always-running tick thread.
• open_with_invalid_path_errors — error string flows out cleanly.

How to run

# Dev — hot-reload via Trunk, Tauri webview on top.
cd crates/app && cargo tauri dev

# Drop the test fixture onto the window:
#   crates/decode/tests/fixtures/sample.mp4
# The status bar reflects the player's metadata; the play button toggles
# the Rust-side player; the timer ticks at 10 Hz.

App-shell visual references

The player view, post-drop, with its transport bar wired to the IPC commands. (Component-level layout is unchanged from M-INT.2 — what changed is the wiring underneath, not the rendered HTML.)

Visible playback — <video> element bound to convertFileSrc (M-PLAY.3)

The IPC plumbing above tracks state, but on its own renders no pixels. M-PLAY.3 wires the user-visible playback surface: an HTML5 <video> element whose src is derived from the dropped path via Tauri 2's convertFileSrc JS helper.

sequenceDiagram
    participant Drop as Drop event
    participant Signal as loaded signal
    participant Convert as window.__screenConvertFileSrc
    participant Video as &lt;video&gt; element<br/>(node_ref=video_ref)
    participant Toggle as PlayerControls toggle
    participant State as Tauri state<br/>(screen_play/pause)
    participant Event as player-status event

    Drop ->> Signal: file dropped
    Signal ->> Convert: video_src() resolves path
    Note over Convert: returns asset:// or<br/>http://asset.localhost URL
    Convert -->> Video: src= asset URL

    par user gesture (sync)
        Toggle ->> Video: video.play() / pause()
    and Tauri state mirror
        Toggle ->> State: screen_play() / screen_pause()
    end

    Event ->> Signal: Effect listens
    Signal ->> Video: catch-up play / pause / future seek

Why two paths to the <video> element

WebKit blocks programmatic .play() outside a user gesture. So:

• Click handler drives <video> synchronously inside the Callback<()>. The browser sees this as user-initiated and allows playback to start.
• Effect::new over player_status is the catch-up path for state changes that aren't user clicks — Tauri pushing Ended on EOF, future seek commands, etc. Idempotent: it only acts when video.paused() doesn't already match the target state, so it doesn't fight the click handler.

Why HTML5 video for the playback surface (and not wisp)

The recorder's editor preview surface will eventually be a winit-child window driven by wisp (so we can apply filters, transforms, animation). But for the MVP "user dropped a file and wants to see it", HTML5 <video> with the asset protocol is:

• one element, no decoder integration,
• hardware-accelerated by the WebView,
• scrub-bar/seek/audio for free.

The Tauri-side PlayerSession keeps running alongside — it owns the gstreamer-decoded VideoTexture that future wisp-rendered surfaces will read. Two-source-of-truth is a deliberate trade for shipping the playback MVP today.

tauri.conf.json requirements

The assetProtocol.scope must include the dropped file's path. Our config uses ["**"] (any local file). For a production build we'd tighten to user-selected directories.

"security": {
  "assetProtocol": {
    "enable": true,
    "scope": ["**"]
  }
}

Without "enable": true, convertFileSrc returns the path unchanged and the <video> element fails to load with a CSP / protocol error.

PlayerSession API · Tauri commands · PlayerControls component · Tauri ↔ Leptos integration (M-INT.2)

Testing the recorder shell — three tiers (M-TEST.1 / .2)

screen-app (Tauri shell) and app-ui (Leptos webview) sit at the integrated top of the stack. Their regression surface is different in shape from the library crates below — there's an OS process, a WebView, an IPC channel, and a JS bridge between Rust and Rust. The tests that matter are correspondingly stratified.

The three tiers

Adding a tier-N test does NOT replace tier-(N-1). They overlap deliberately — tier 0 fires fast on every save; tier 2 fires once per CI run; in-between tier 1 catches the wire-format and registration regressions that tier 0 can't reach (no IPC dispatch) and tier 2 is too expensive to keep paged in.

Tier 0 — chunk-level (existing pattern)

Direct tests against the data types and functions in each crate. Already the dominant test layer in the workspace. See _docs/TESTING.md for the broader strategy. For screen-app specifically:

#![allow(unused)]
fn main() {
// crates/app/tests/player_session.rs
#[test]
fn play_pause_lifecycle() {
    let session = PlayerSession::new();
    session.open(Path::new(FIXTURE)).unwrap();
    session.play();
    assert_eq!(session.status().state, SessionState::Playing);
    session.pause();
    assert_eq!(session.status().state, SessionState::Paused);
}
}

These tests bypass Tauri entirely. If PlayerSession is correct, but the Tauri registration layer has a typo, these still pass.

Tier 1 — IPC harness (M-TEST.1)

Uses tauri::test::mock_builder() to spin up a Tauri runtime in-process. No window, no WebView, no WASM. Commands are dispatched the way the Leptos frontend dispatches them.

#![allow(unused)]
fn main() {
// crates/app/tests/commands.rs
fn build_app() -> tauri::App<MockRuntime> {
    mock_builder()
        .manage(PlayerSession::new())
        .invoke_handler(tauri::generate_handler![
            commands::player_open,
            commands::player_play,
            commands::player_pause,
            commands::player_status,
        ])
        .build(mock_context(noop_assets()))
        .expect("build mock app")
}

#[test]
fn player_open_then_play_then_pause_via_ipc() {
    let app = build_app();
    let webview = main_webview(&app);
    invoke(&webview, "player_open", InvokeBody::Json(json!({ "path": FIXTURE })));
    invoke(&webview, "player_play", InvokeBody::default());
    let value = invoke(&webview, "player_status", InvokeBody::default());
    let status: PlayerStatus = serde_json::from_value(value).unwrap();
    assert_eq!(status.state, SessionState::Playing);
}
}

What this catches that Tier 0 misses:

• A typo in tauri::generate_handler![commands::playerr_play] — compile fails; this test would never run.
• A missing .manage(PlayerSession::new()) — Tier 0 still passes (it builds the session directly); Tier 1 fails at runtime when the command tries to read State.
• A #[serde(rename_all = "lowercase")] accidentally dropped from SessionState — Tier 0 doesn't serialize anything; Tier 1's serde_shape_uses_lowercase_session_state test fails because the payload now reads "Empty" instead of "empty".

Cost: each test pays the same wisp Application::new() boot (~200 ms on Apple Silicon) as Tier 0. Total Tier 1 runtime is ~1 s.

Setup required: tauri = { version = "2", features = ["test"] } in [dev-dependencies]. Cargo unifies dep + dev-dep features, so the release binary also gets the test module compiled in (a known cargo wart). The footprint is small enough to accept.

Tier 2 — WebDriver e2e (M-TEST.2)

The real thing: launches the built screen-app binary, drives it through WebDriver via tauri-driver, asserts on rendered DOM state.

#![allow(unused)]
fn main() {
// crates/app-e2e/tests/golden_path.rs
#[tokio::test]
async fn open_play_pause_via_ui() {
    let app = E2eApp::start().await;
    let driver = app.client();

    // OS-level file drop can't be scripted via WebDriver, so the test
    // calls a debug-only Tauri command (`__test_drop_file`) that emits
    // the same `file-dropped` event the real drag-drop handler emits.
    driver.execute(
        "return window.__TAURI__.core.invoke('__test_drop_file', { path: arguments[0] })",
        vec![FIXTURE.into()],
    ).await?;

    driver.wait().for_element(Locator::Css(".player-controls")).await?;
    driver.find(Locator::Css(".player-toggle")).await?.click().await?;
    driver.wait_for(|d| async {
        d.find(Locator::Css(".player-toggle-playing")).await.is_ok()
    }).await?;
}
}

What this catches that Tier 1 misses:

• index.html JS bridge regressions (e.g. dropping __screenPlay globals, breaking the __TAURI__.event.listen re-emit).
• Leptos rendering / hydration issues (e.g. <PlayerControls> not re-rendering when player_status changes).
• CSS layout regressions that hide the play button off-screen.
• Cross-process timing — events arriving after the listener is wired, Promise resolution order on the JS side, etc.

Platform support

tauri-driver works well on Linux (WebKitGTK has solid WebDriver support via webkit2gtk-driver) and Windows (Edge WebView2 + msedgedriver). macOS is the gap — Apple's WKWebView WebDriver support is half-implemented and tauri-driver doesn't reliably drive it. The community pattern is "Linux CI gates everything; mac is manual smoke before tagging."

Tier 2 is intentionally local-only — the just e2e recipe is not part of the CI gate. tauri-driver + WebKitGTK under xvfb on GitHub-hosted Ubuntu runners proved flaky enough that the skip-or-fail signal stopped being useful. Contributors run it locally before opening any PR that touches crates/app/, crates/app-ui/, or crates/app-e2e/. The recipe detects the host OS:

• Linux: runs xvfb-run cargo nextest run -p app-e2e.
• macOS: prints a clear "skipping — Tauri WKWebView WebDriver doesn't drive reliably" message and exits 0; do a manual smoke via cargo tauri dev instead.

Local prerequisites (Linux)

# Cargo plugin:
cargo install --locked tauri-driver

# System packages (Debian/Ubuntu):
sudo apt-get install -y webkit2gtk-driver xvfb

Then:

just e2e

File-drop simulation — the trick

WebDriver clients can't synthesize OS-level drag-drop events. Tauri 2's WindowEvent::DragDrop only fires from real OS drops, not JS code. The solution: a debug-only Tauri command that emits the same file-dropped event the real handler emits.

#![allow(unused)]
fn main() {
#[cfg(debug_assertions)]
#[tauri::command]
pub fn __test_drop_file(app: tauri::AppHandle, path: String) -> Result<(), String> {
    app.emit("file-dropped", path).map_err(|e| e.to_string())
}
}

Gated by #[cfg(debug_assertions)] and a corresponding #[cfg(...)] in the generate_handler! macro, so the test entry point is excluded from release builds. Tests invoke it via window.__TAURI__.core.invoke('__test_drop_file', { path }).

The real OS drag-drop path stays untouched. Tier 1 + Tier 2 cover it collectively: Tier 1 verifies the player_open command (the handler side of the file-drop chain), Tier 2 verifies the full UI flow assuming a file-dropped event was emitted.

When to add tests at each tier

• New chunk with pure Rust logic → Tier 0.
• New #[tauri::command] exposed to the frontend → add a Tier 1 case with the exact JSON body shape the frontend will send.
• New event emitted from Rust → received by Leptos → add a Tier 2 case asserting the rendered DOM responds to a synthesized event.
• New cross-process timing concern (race conditions, ordering) → Tier 2 only. Tier 1 doesn't have wall-clock semantics.

What's still missing

• Visual regression for the integrated shell. The ui-storybook SSR snapshots cover individual components, but a "full editor mock under the live IPC" diff isn't captured today. Could be added under Tier 2 with headless-screenshot
  • image::compare once the e2e suite stabilizes.
• macOS automation. Tracked as future work; the broader Tauri community is still iterating on this.

Tray icon → AppShell → NavigationRail routing — M-TRAY.0..4

cargo run -p screen-app puts a small filled-circle icon on the macOS menubar. Left-click toggles the main app window; the window shows the full ui-storybook AppShell (with NavigationRail on the left) and the NavigationRail items switch the right-pane content. Click the tray icon again to hide.

What ships across the four tickets

End-to-end flow

sequenceDiagram
    participant User
    participant Tauri as Tauri shell (main.rs)
    participant State as TrayState (commands.rs)
    participant Window as `tray-popover` window
    participant Bundle as app-ui wasm bundle
    participant Shell as AppShellRoot (app_shell_mount.rs)

    User->>Tauri: Left-click tray icon
    Tauri->>State: TrayPopoverState::on_click() → Action::Show
    State->>Window: window.show() + set_focus()
    Window->>Bundle: Load index.html?surface=recorder
    Bundle->>Bundle: parse_surface_from_query() → AppSection::Record
    Bundle->>Shell: AppShellRoot { initial: Record }
    Shell->>Shell: RwSignal::new(Record)
    Shell->>User: AppShell with NavigationRail (Record active)
    User->>Shell: Click "Library" in NavigationRail
    Shell->>Shell: on_select(Library) → signal.set(Library)
    Shell->>Window: history.replaceState(?surface=library)
    Shell->>User: Right-pane swaps to "Library" placeholder
    User->>Tauri: Left-click tray icon
    Tauri->>State: TrayPopoverState::on_click() → Action::Hide
    State->>Window: window.hide()

Architecture decisions

```admonish important title="AppShell stays state-free; app-ui owns the signal" The original ticket spec for M-TRAY.2 suggested adding an initial_surface prop to AppShell. The M-TRAY.1 audit found that AppShell is pure slot composition — it has no internal signal for the prop to drive. The active-surface state lives in crates/app-ui (specifically in AppShellRoot's RwSignal<AppSection>), and it flows into AppShell's rail and main slots from there. Result: the M-TRAY.2 ticket pivoted from "add prop to AppShell" to "add on_select: Callback<AppSection> to NavigationRail" — a smaller, cleaner change.

```admonish warning title="`NavigationRail` items were inert until M-TRAY.2"
The buttons in `NavigationRail` rendered correctly under SSR + CSR but carried no `on:click` handler. M-TRAY.2 added an optional `Callback<AppSection>` prop and wired the click. Existing stories that don't pass the callback are unchanged in SSR HTML output — Leptos `on:click` doesn't produce an HTML attribute, only a runtime listener attached during CSR mount.

How to verify locally

• cargo run -p screen-app — launches the binary. Tray icon appears on the macOS menubar (or Windows tray / Linux app-indicator). Left-click toggles the main window.
• cargo run -p screen-app --example regen-tray-icons — regenerates the three tray.png raster outputs from the SVG source. Idempotent; commit any changes.
• just dev-appshell — runs trunk serve --features tray-appshell-preview from crates/app-ui, mounting the AppShell directly in a browser at http://localhost:8080. NavRail clicks are inert in this mode (M-TRAY.1 dev affordance only — the full routing lives behind ?surface=).
• cargo test -p app-ui --lib — runs the 7 routing round-trip tests in crates/app-ui/src/routing.rs.
• cargo nextest run -p screen-app --lib — runs the 4 TrayPopoverState state-machine tests.

What this closes vs what's deferred

Closes: the M-TRAY.0..4 sequence end-to-end. Tray → AppShell → NavRail surface switching all work on macOS; cross-OS compile paths are gate-green.

Deferred to M-RECORDER-V1:

• Cross-process surface persistence (LocalStorage / tauri-plugin-store).
• Multi-display window positioning under the tray click (M-RECP.1).
• The small TrayRecordPopover quick-record window as a separate surface — tray-popover today opens the full AppShell, not the AUT-132 small popover. Worth filing as a separate "tray quick-record popover" track post-V0.
• wasm-bindgen-test interaction smoke for NavigationRail click → signal-change. Skipped pending the headless-browser CI setup.

Webcam-bubble overlay — M-BUBBLE.0 + .1 + .3

cargo run -p screen-app --features custom-protocol (or just test-recorder) puts a "Show webcam bubble" toggle in the Recorder surface of the AppShell. Clicking it reveals a borderless, transparent, always-on-top 200×200 Tauri window — the future home of the recognisable Screen-Studio-style floating webcam circle. For v0 the bubble shows an indigo "Webcam" placeholder; the live wisp-rendered canvas inside it is M-BUBBLE.2, which is blocked on the M-CAM.3 pipeline (see "Blockers downstream" below).

The bubble's position is persisted across hide / show cycles AND across app launches — drag the bubble anywhere on screen, hide it, reopen it: it reappears at the spot you left it. After a display unplug (so the saved position lands off-screen) the bubble falls back to a sensible default (bottom-right of the primary monitor, 16 px inset).

What ships across the two tickets

End-to-end flow

sequenceDiagram
    participant User
    participant Bundle as app-ui wasm bundle (in the AppShell webview)
    participant Tauri as Tauri shell (main.rs / commands.rs)
    participant State as BubbleState (commands.rs)
    participant Disk as bubble-position.txt
    participant Window as `webcam-bubble` window

    User->>Bundle: Click "Show webcam bubble" button
    Bundle->>Tauri: __TAURI__.core.invoke("toggle_webcam_bubble")
    Tauri->>State: BubbleVisibility::on_click() → Show
    Tauri->>State: last_position?
    alt In-memory cache hit
        State-->>Tauri: Some(pos)
    else Cold launch, no in-memory state
        Tauri->>Disk: read bubble-position.txt
        Disk-->>Tauri: "x,y\n" → BubblePosition
        Tauri->>State: cache it for next show
    else No persisted file, or persisted pos off-screen
        Tauri->>Tauri: default_position(primary_monitor)
    end
    Tauri->>Window: set_position(physical) BEFORE show()
    Tauri->>Window: window.show()
    Window->>Bundle: Load index.html?mount=bubble
    Bundle->>Bundle: parse_mount_point() → MountPoint::Bubble
    Bundle->>User: <BubbleRoot /> with indigo placeholder
    User->>Window: Drag bubble to new spot
    Window->>Tauri: WindowEvent::Moved(physical)
    Tauri->>State: update_bubble_position_from_event(x, y)
    User->>Bundle: Click "Show webcam bubble" again
    Bundle->>Tauri: __TAURI__.core.invoke("toggle_webcam_bubble")
    Tauri->>State: BubbleVisibility::on_click() → Hide
    Tauri->>Window: outer_position()
    Window-->>Tauri: PhysicalPosition(x, y)
    Tauri->>State: cache the position
    Tauri->>Disk: write "x,y\n"
    Tauri->>Window: window.hide()

Coordinate-system contract

Persistence file format

Why M-BUBBLE.1 ships v0, not full

The original ticket scoped per-pixel hit-testing: on macOS, a custom NSView subclass via objc2 overrides hitTest: to return nil for pixels outside the inscribed circle — so clicks on the four transparent corners pass through to whatever's underneath, but clicks on the visible circle still hit the bubble (drag-to-move works). The original "hover toggle" alternative (auto-disable click-through when the cursor enters the visible area, re-enable when it leaves) doesn't work: macOS's setIgnoresMouseEvents(true) filters at the NSWindow level, so the webview never receives the mouseenter event that's supposed to flip it back to false. Chicken-and-egg.

The v0 ship target here is a user-driven toggle: a button in the AppShell that flips the whole bubble between "interactive" (drag works, corners catch clicks) and "click-through" (whole window passes mouse events through). To turn click-through off the user clicks the AppShell button — the bubble itself can't receive the click while passthrough is on, so the out-of-band trigger is required.

The proper per-pixel hitTest: is filed as the v1 follow-up under the same ticket. It needs a small NSView subclass injected at window-creation time (via Tauri's plugin hook + objc2) plus Windows SetWindowRgn(CreateEllipticRgn) and Linux X11/Wayland shape-extension equivalents. Substantial native plumbing per OS; the v0 toggle is an honest middle step.

Why snap-on-drag is inert in v0

The pure-Rust snap_to_nearest_corner helper is fully implemented + tested in crates/app/src/recp/bubble_position.rs — given a current position + monitor bounds + a snap radius, it returns the snapped position OR None if the bubble is far from every corner.

It's not wired to WindowEvent::Moved yet. The reason: calling window.set_position(snapped) from inside the Moved handler triggers another Moved event for the new position. Without a "last-snap-applied-was" guard or a leading-edge debounce, that's an infinite event loop that pegs the OS event queue.

Wiring it cleanly requires either:

1. A small last_snap_applied: AtomicI32 × 2 to short-circuit re-snapping to the same coords, OR
2. A trailing debounce (250 ms via tokio::time::sleep) that fires snap only after the drag stops, OR
3. A separate WindowEvent::MouseUp / DragEnd signal Tauri 2 doesn't expose today on all OSes.

Option 1 is the obvious choice for a follow-up; the math is the load-bearing piece and it's tested. Filed as a v1 polish ticket alongside the resize-handle work that's also deferred from AUT-276.

Tests

• BubbleVisibility state machine (4 tests) — round-trips, default-state, ten-alternating-clicks parity check.
• MountPoint parsing (4 tests) — ?surface=… wins over ?mount=…, unknown queries fall through to DropZone, ?mount=bubble lands in the new Bubble mount.
• default_position — bottom-right of monitor with inset; respects secondary-monitor offsets.
• is_on_any_monitor — true for fully-inside, true for partial overlap, false for fully-off-screen and for positions that assumed a now-gone secondary display.
• snap_to_nearest_corner — snaps to bottom-right when near, snaps to top-left when near, chooses nearest corner when two are in range, returns None for dead-center, respects monitor offsets.
• BubbleState + persistence helpers — encode_position / decode_position round-trip; rejects malformed inputs (missing comma, non-integer, empty); tolerates whitespace + missing trailing newline; update_bubble_position_from_event updates the in-memory cache atomically.

Total: 26+ unit tests covering the M-BUBBLE.0 + .3 surface.

Manually verifiable

Blockers downstream

Cross-link

• Tauri tray → AppShell flow — the existing tray-popover machinery this work parallels (same state-machine shape, same URL-routed mount pattern).

macOS permissions — embedded Info.plist

The recorder needs three macOS TCC (Transparency, Consent and Control) permissions: Camera, Microphone, and Screen Recording. macOS gates these by requiring the requesting app to declare each one in its Info.plist via the corresponding NS*UsageDescription string. Without the declaration, macOS silently returns empty results from APIs like AVCaptureDevice.devices(for: .video) — never prompting the user, never showing the app in System Settings → Privacy & Security.

This bites two ways: dev builds and downloaded .app bundles.

Single source of truth

Two ingestion paths read the same file:

flowchart LR
    Plist[crates/app/Info.plist]
    DevBin[target/debug/screen-app<br/>Mach-O __TEXT,__info_plist section]
    ProdApp[screen-app.app/Contents/Info.plist]
    Plist -->|"tauri::generate_context! auto-embed<br/>(tauri-codegen, dev+macOS)"| DevBin
    Plist -->|"cargo tauri build auto-detects file<br/>next to tauri.conf.json"| ProdApp

Dev binary — Mach-O section embed (auto-embedded by Tauri)

cargo run -p screen-app produces a raw Mach-O at target/debug/screen-app. There's no screen-app.app/Contents/Info.plist filesystem path for TCC to read. Apple's fallback for command-line tools: a __TEXT,__info_plist section embedded in the binary itself (the same mechanism /usr/bin/screencapture, /usr/bin/pmset, etc. use).

tauri::generate_context!() auto-embeds this section for every debug macOS build. The code path lives in tauri-codegen-2.6.1/src/context.rs::context_codegen, branch target == Target::MacOS && dev && !running_tests — it reads Info.plist next to tauri.conf.json, merges bundle name / version, and emits the same embed_plist::embed_info_plist! call we used to make manually.

```admonish note title="History: manual embed_plist! was removed in M-MIC.1" PR #47 originally added an explicit embed_plist::embed_info_plist!("../Info.plist") in crates/app/src/main.rs. Once Tauri 2.6.1+'s auto-embed was confirmed working, the manual macro call was redundant — and worse, it emitted the same _EMBED_INFO_PLIST symbol as the auto-embed, which broke every integration test in screen-app at link time with symbol _EMBED_INFO_PLIST is already defined. M-MIC.1 (AUT-278) removed the manual call + the embed_plist dep; the auto-embed is now the only path.

Verify the embed worked:

```bash
otool -s __TEXT __info_plist target/debug/screen-app | head -20

You'll see the PLIST DTD declaration in the hex dump — that's the signal.

Bundled .app — Tauri bundler auto-detect

cargo tauri build produces target/release/bundle/macos/screen-app.app. Tauri's bundler:

1. Reads crates/app/tauri.conf.json for the standard keys (CFBundleIdentifier ← identifier, CFBundleVersion ← version, etc.).
2. Looks for Info.plist next to tauri.conf.json and merges its keys into the generated bundle plist.
3. Writes the merged result to screen-app.app/Contents/Info.plist.

No explicit bundle.macOS.infoPlist config field needed — Tauri 2 detects the file by convention.

What the user actually sees

First launch (dev binary OR downloaded .app):

1. App tries to enumerate cameras (or open mic, or capture screen).
2. macOS reads the NS*UsageDescription string.
3. System prompt appears: "screen-app would like to access the camera" followed by our string.
4. User clicks Allow or Don't Allow.

Subsequent launches: silent. The grant is cached under the bundle/binary's TCC entry. The app now appears in System Settings → Privacy & Security → Camera (and Microphone, Screen Recording) with a toggle the user can flip later.

What's in the Info.plist

Twelve keys, four concerns:

Bundle identity (three keys)

<key>CFBundleIdentifier</key>      <string>com.screen.app</string>
<key>CFBundleName</key>            <string>screen-app</string>
<key>CFBundleShortVersionString</key> <string>0.1.0</string>
<key>CFBundleVersion</key>         <string>0.1.0</string>

CFBundleIdentifier is the TCC key — macOS pairs the user's permission grant against this string. Once any user has granted Camera (or Mic, or Screen Recording) to com.screen.app, that grant survives rebuilds as long as this string stays the same. Renaming it = every user is re-prompted. One-way decision. Keep in sync with identifier in tauri.conf.json.

CFBundleName is the prompt display name — the macOS dialog says "screen-app would like to access the camera" using this string.

CFBundleShortVersionString + CFBundleVersion keep the TCC grant stable across rebuilds on macOS versions that pair the grant with (id, version). Match version in tauri.conf.json.

Minimum OS version (one key)

<key>LSMinimumSystemVersion</key>  <string>13.0</string>

ScreenCaptureKit's video API was introduced in macOS 12.3, but the audio API (SCStreamConfiguration.capturesAudio — M-AUDIO-SYS.0 / AUT-280) is macOS 13.0+. The recorder's full capture surface (display + window + mic + system audio + per-app audio) requires 13.0; we declare the floor here so macOS gatekeeps launch on older systems instead of letting the user run + silently fail at first system-audio capture.

File-system access strings (four keys)

<key>NSDocumentsFolderUsageDescription</key>
<string>Screen needs Documents folder access to save and read your screen recordings.</string>

<key>NSDownloadsFolderUsageDescription</key>
<string>Screen needs Downloads folder access to save and read your screen recordings.</string>

<key>NSDesktopFolderUsageDescription</key>
<string>Screen needs Desktop folder access to save and read your screen recordings.</string>

<key>NSRemovableVolumesUsageDescription</key>
<string>Screen needs removable-volume access to save recordings to external drives.</string>

Cover the recorder's programmatic read/write paths into user-owned folders. Not needed for explicit file-picker flows — when the user explicitly chooses a file via an NSOpenPanel / NSSavePanel (Tauri's file-dialog plugin uses these, and so does drag/drop), macOS treats the selection as an implicit grant and no Info.plist string is required.

Permission usage strings (three keys — the load-bearing ones)

<key>NSCameraUsageDescription</key>
<string>Screen needs camera access to record your webcam alongside your screen.</string>

<key>NSMicrophoneUsageDescription</key>
<string>Screen needs microphone access to record audio with your recordings.</string>

<key>NSScreenCaptureUsageDescription</key>
<string>Screen needs screen recording access to capture your display.</string>

These are user-facing — they appear verbatim in the macOS prompt. Edit them to explain why you need the permission, not what the permission technically grants.

```admonish note title="NSScreenCaptureUsageDescription covers more than you'd think" This one string is the TCC gate for all of: full-display capture, specific-window capture, system audio output capture, and per-process audio capture. ScreenCaptureKit's audio capture path uses the screen-recording TCC entry rather than the microphone one — counterintuitive but baked into the framework.

The recorder thus needs only the three strings above to cover every flavour of capture we plan to support.

## Audio capture paths — verified TCC mapping (AUT-283)

M-AUDIO.PERMS (AUT-283) verified the per-path TCC mapping for the three audio capture flavours by attempting each path against a fresh `tccutil reset All com.screen.app` state and observing which prompt macOS surfaced.

| Capture path | Triggered by | TCC category | Info.plist key |
| --- | --- | --- | --- |
| Microphone (M-MIC.1 / AUT-278) | `gst-launch-1.0 ! autoaudiosrc !` opens AVAudioSession | **Microphone** | `NSMicrophoneUsageDescription` |
| System audio (M-AUDIO-SYS.0 / AUT-280) | `SCStreamConfiguration.setCapturesAudio(true)` | **Screen Recording** | `NSScreenCaptureUsageDescription` |
| Per-process audio (M-AUDIO-SYS.1 / AUT-281) | `SCContentFilter.initWithDisplay_includingApplications_…` | **Screen Recording** (same entry) | `NSScreenCaptureUsageDescription` |

```admonish important title="One Screen Recording grant covers both SCK audio paths"
The system-audio + per-process-audio paths share the same TCC entry. Once the user grants Screen Recording (either for video capture or for the first system-audio attempt), every subsequent SCK audio call is silent. **The user sees the prompt once, not twice.**

Verified by attempting `cargo run -p media --example system_audio_smoke` on a freshly-reset TCC state: the error message returned by SCK was *"The user declined TCCs for application, window, display capture"* — confirming the SCK audio path engages the Screen Recording TCC entry, not Microphone.

Platform-quirk reminders

Resetting permissions during development

If you grant + later want to re-test the first-run prompt path:

# Reset all camera grants for our bundle id
tccutil reset Camera com.screen.app

# Same for the other two
tccutil reset Microphone com.screen.app
tccutil reset ScreenCapture com.screen.app

# Or reset every TCC entry for our bundle (nuclear)
tccutil reset All com.screen.app

Next launch: macOS prompts again.

Diagnostic: "No cameras detected" silently

If the Recorder surface shows the empty-state "No cameras detected" and you suspect a permission issue, media::list_cameras now logs PATH + raw stderr on failure (M-CAM.3 diagnostics commit). Run with verbose logging:

RUST_LOG=info just test-recorder 2>&1 | grep list_cameras

If the log shows gst-device-monitor exited 0 but parser found 0 cameras, it's a TCC permission issue — verify the binary has the __TEXT,__info_plist section embedded via the otool command above.

Camera-pipeline worker — M-CAM.3 (gst layer)

start_preview now spawns a real GStreamer subprocess and pulls BGRA frames into Rust. After the macOS permission prompt resolves, the PreviewLifecycle state machine transitions Starting → Running and stays there until stop_preview (which drops the worker, cancels the loop, joins the thread, and kills the gst child via the Drop impl on media::gstreamer_video::GstreamerVideoCapture).

Diagnostics architecture

flowchart LR
    subgraph Worker thread
        Loop[next_frame loop]
        Loop -->|on every frame| Atomic[atomic counters]
        Loop -->|once per session| Dump[encode + write PNG]
    end
    subgraph "Tauri state (PreviewDiagnostics)"
        Atomic
        Dump
    end
    subgraph "Webview (Leptos)"
        Poll[setInterval 500ms] --> IPC[__screenPreviewDiagnostics]
        IPC --> Render[CameraDiagnostics view]
    end
    IPC -->|read snapshot| Atomic
    IPC -->|read dump path| Dump

Architecture

sequenceDiagram
    participant User
    participant Leptos as Leptos (Recorder surface)
    participant Cmd as start_preview (commands.rs)
    participant Handle as CameraPipelineHandle (Tauri state)
    participant Pipe as CameraPipeline (worker)
    participant Gst as gst-launch-1.0 child
    participant Life as PreviewLifecycle (Tauri state)

    User->>Leptos: select camera
    Leptos->>Cmd: __TAURI__.invoke("start_preview", { cameraId })
    Cmd->>Life: try_start() → Starting
    Cmd->>Pipe: CameraPipeline::spawn(app)
    Pipe->>Pipe: thread::spawn("camera-pipeline")
    Pipe->>Gst: GstreamerVideoCapture::from_default_camera(480, 480, 30)
    Note right of Gst: macOS prompt fires here on first run
    Cmd->>Handle: install(pipeline)
    Cmd-->>Leptos: Ok(())

    loop frames
        Gst-->>Pipe: BGRA bytes via stdout pipe
        Pipe->>Life: mark_running() (idempotent)
    end

    User->>Leptos: stop
    Leptos->>Cmd: __TAURI__.invoke("stop_preview")
    Cmd->>Life: try_stop() → Stopping
    Cmd->>Handle: shutdown() → drops CameraPipeline
    Pipe->>Pipe: cancel.store(true)
    Pipe->>Pipe: handle.join()
    Note right of Gst: Drop on GstreamerVideoCapture kills + reaps the child
    Cmd->>Life: finish_stop() → Idle

Thread-affinity contract

Drop-safety

The Drop impl on CameraPipeline flips the cancel flag, joins the thread, and triggers Drop on the GstreamerVideoCapture inside the worker — which kills + reaps the gst-launch child. The chain is:

Tauri State<CameraPipelineHandle>::install(new_pipeline)
  → Mutex::lock → Option::replace(Some(new)) → old Option<CameraPipeline> dropped
    → CameraPipeline::drop
      → cancel.store(true)
      → JoinHandle::join (blocks until worker exits)
        → GstreamerVideoCapture::drop inside the worker
          → Child::kill + Child::wait

So a re-entrant start_preview while a session is already running cleanly tears down the previous session before starting the new one. The smoke test in M-RECP.4 (AUT-265 — no zombie gst processes after app quit) is the regression guard.

Tests

• Pure-state: CameraPipelineHandle install / shutdown / is_active round-trip (no thread spawn — that requires a real tauri::AppHandle + real gst install + camera).
• Compile-time invariants: PREVIEW_WIDTH == PREVIEW_HEIGHT (the circle mask the follow-up adds requires square input), PREVIEW_FPS == 30 || 60 (round targets cameras support natively).

Real end-to-end testing requires hardware. The CI gate runtime-skips when gst-launch isn't on PATH (per the existing gstreamer_available() pattern in crates/decode/tests/gstreamer_integration.rs).

Manually verifiable

Cross-link

• Webcam bubble overlay — the M-BUBBLE.0/.3/.1 window infrastructure that will host the rendered pixels.
• Tray → AppShell flow — the existing surface the camera pipeline mounts under.

Audio capture — microphone + system audio (M-AUDIO)

The recorder captures audio from three sources, each going through a different OS path but converging on the same f32 PCM shape downstream:

flowchart LR
    Mic[Microphone<br/>built-in / USB / Bluetooth]
    Speakers[Speakers<br/>system audio output]
    Apps[Per-app audio<br/>e.g. Spotify only]

    Mic -->|gst-launch-1.0 autoaudiosrc| MicWorker[MicCapturePipeline<br/>worker thread]
    Speakers -->|SCK SCStream<br/>capturesAudio=true| SckSession[SystemAudioStream]
    Apps -->|SCK SCContentFilter<br/>includingApplications| SckSession

    MicWorker -->|f32 PCM chunks| Encoder[Encoder<br/>M-RECORD]
    SckSession -->|f32 PCM chunks| Encoder

    classDef gst fill:#14532d,stroke:#22c55e,color:#dcfce7
    classDef sck fill:#312e81,stroke:#818cf8,color:#e0e7ff
    classDef sink fill:#1e293b,stroke:#94a3b8,color:#f1f5f9
    class MicWorker gst
    class SckSession sck
    class Encoder sink

Microphone (M-MIC chain)

The mic chain mirrors the camera chain end-to-end. Three tickets, three layers.

M-MIC.0 — device enumeration (AUT-277)

media::list_microphones() -> Vec<MicrophoneDevice> spawns gst-device-monitor-1.0 Audio/Source and parses the text output. Returns {id, label, is_default, channels, sample_rate_hz} per attached input.

Two intentional deltas from the camera enumerator:

• is_default uses gst's explicit signal, not "first in list". On macOS the properties: block contains is-default = true|false; a Bluetooth headset can be third-listed but flagged default. The first-listed heuristic remains as a fallback for backends that omit the property.
• channels + sample_rate_hz come from the first caps line (the device's preferred native format). Either field degrades to 0 ("unknown") when absent; downstream defaults to 48 kHz / 2 channels.
• ID prefix is mic- so the ID space can't collide with cam- at the IPC layer.

cargo run -p media --example list_microphones

M-MIC.1 — capture worker (AUT-278)

media::gstreamer_audio::GstreamerAudioCapture::from_microphone(mic_id, format) spawns:

gst-launch-1.0 ! autoaudiosrc ! audioconvert ! audioresample
              ! audio/x-raw,format=F32LE,rate=48000,channels=2
              ! fdsink fd=1

crates/app/src/audio/pipeline.rs::MicCapturePipeline owns a dedicated thread that pulls 100 ms PCM chunks (4800 frames @ 48 kHz) into Rust. MicLifecycle { Idle, Starting, Running, Stopping } (mirror of PreviewLifecycle) tracks state; Drop cancels + joins, and GstreamerAudioCapture's own Drop kills + reaps the gst child.

M-MIC.2 — picker UI (AUT-279)

<MicPicker /> renders below <CameraPicker /> in the Recorder surface. Click the trigger → real attached mics enumerate via Tauri IPC; click a row → start_mic_capture(mic_id) fires the worker (triggering NSMicrophoneUsageDescription prompt on first run); the last-used mic id persists to LocalStorage so re-opens land on the same device.

Unlike the camera picker, the mic picker does not auto-start the worker on mount. Recording audio without the user clicking would be surprising even for a default mic — opt-in is the cleanest UX.

System audio (M-AUDIO-SYS chain)

System audio capture (what plays through the speakers — YouTube, Spotify, conference calls) uses ScreenCaptureKit, not gst. There's no general gst element to tap system audio on macOS without a kernel extension (BlackHole, Loopback); SCK is Apple's blessed post-13.0 path.

M-AUDIO-SYS.0 — SCK system audio (AUT-280)

media::sck_audio::SystemAudioStream opens an SCStream with SCStreamConfiguration.capturesAudio = true against an SCContentFilter covering the primary display. An SCStreamOutput delegate (defined via objc2::define_class!) receives CMSampleBuffer audio on SCK's dispatch queue, extracts Float32 PCM from the AudioBufferList (handles both interleaved and planar layouts), and forwards onto an mpsc::Sender. The caller's next_chunk(frames) blocks on the receiver until enough PCM has buffered.

excludesCurrentProcessAudio defaults to true to prevent a feedback loop (the recorder capturing its own output back). Override only for the meta-recording case (recording a tutorial of using the recorder).

cargo run -p media --example system_audio_smoke

M-AUDIO-SYS.1 — per-process filter (AUT-281)

SystemAudioStream::set_app_filter(AudioAppFilter) rebuilds the SCContentFilter via updateContentFilter_completionHandler for hot-swap (no audio gap vs tear-down + recreate).

AudioAppFilter variants carry bundle ids, not PIDs:

#![allow(unused)]
fn main() {
pub enum AudioAppFilter {
    AllAudio,
    OnlyApps(Vec<String>),     // bundle ids
    ExcludeApps(Vec<String>),  // bundle ids
}
}

PIDs are re-resolved at filter-apply time so a Spotify crash + restart is followed transparently. list_audio_apps() enumerates every running app via SCShareableContent.applications, deduped by bundle id (Chrome's per-renderer processes collapse to one row).

cargo run -p media --example list_audio_apps

M-AUDIO-SYS.2 — picker UI (AUT-282)

<SystemAudioPicker /> renders below <MicPicker /> in the Recorder surface. Two-button header: a master on/off toggle that starts/stops the SCK session, and an expand button that opens a per-app multi-select dropdown.

sequenceDiagram
    participant User
    participant Picker as SystemAudioPicker
    participant IPC as Tauri commands
    participant State as SystemAudioCaptureState
    participant SCK as SCStream

    User->>Picker: click master toggle (Off → On)
    Picker->>IPC: start_system_audio_capture
    IPC->>State: start(config)
    State->>SCK: SystemAudioStream::new
    SCK-->>State: ready (or TCC denial)
    State-->>IPC: Ok / Err
    IPC-->>Picker: surface error inline if Err

    User->>Picker: expand dropdown
    Picker->>IPC: list_audio_apps
    IPC->>SCK: SCShareableContent.applications
    SCK-->>IPC: Vec<AudioApp>
    IPC-->>Picker: render checklist

    User->>Picker: toggle Spotify checkbox
    Picker->>IPC: set_system_audio_filter(OnlyApps(["com.spotify.client"]))
    IPC->>State: set_filter
    State->>SCK: updateContentFilter (hot-swap)

Selected bundle ids round-trip through LocalStorage (screen.system_audio.selected_bundle_ids key), so a Spotify selection survives across launches. Master toggle reverts on start failure with the SCK error surfaced inline (most commonly TCC denial).

Permissions (M-AUDIO.PERMS / AUT-283)

All three audio paths are gated by TCC entries declared in crates/app/Info.plist. Verified end-to-end by attempting each path on a freshly-reset TCC state:

See macOS permissions — embedded Info.plist for the full TCC + bundle-id story.

Recorder Page — live composition of the design components

The Recorder surface is the user's home screen: workspace badge, capture-mode tabs (Screen / Window / Area), source preview with dimensions, camera + mic rows with inline expandable pickers, system-audio multi-select, on-screen options, and the primary record button. crates/app-ui/src/recorder_page.rs composes the design from the storybook presentational components into this single surface, owning the live signal state and the IPC wiring.

Layout

flowchart TB
  subgraph header["header"]
    Workspace["Workspace badge (N)"]
    Tabs["CaptureModeTabs"]
  end
  subgraph body["body"]
    Display["DisplaySourceCard (live screen_ipc)"]
    Camera["CaptureSourceRow → DevicePickerMenu"]
    Preview["CameraPreview canvas (M-PIX.8)"]
    Mic["CaptureSourceRow → DevicePickerMenu"]
    Audio["SystemAudioRow → SystemAudioAppList"]
    OnScreen["OnScreenOptionsPopover"]
  end
  subgraph footer["footer"]
    Controls["RecordingControlsFooter<br/>(AutoZoomSelect + CountdownSelect + StartRecordingButton)"]
  end
  header --> body --> footer

Signal → view-model conversions

Each presentational component takes a view-model struct. The live page derives that struct from signals on every render:

sequenceDiagram
  autonumber
  participant IPC as camera_ipc / mic_ipc / screen_ipc / system_audio_ipc
  participant Sig as Live signals
  participant VM as view-model fn
  participant Comp as presentational component
  IPC->>Sig: refresh_cameras / refresh_mics / refresh_displays / refresh_audio_apps
  Note over Sig: RwSignal<Vec<…>>, Option<String>, bool, etc.
  Sig->>VM: camera_view() / mic_view() / display_card_view() / system_audio_view()
  VM->>Comp: CaptureSourceView, DeviceOptionView, DisplaySourceView, SystemAudioView
  Comp-->>Sig: (presentational; no callbacks fire from rendering)

The OpenPicker enum (None | Camera | Microphone | SystemAudio | OnScreen) keeps the four expand-states mutually exclusive — opening one closes the others.

Permissions are untouched

This refactor is pure render layer. The TCC flow (camera / microphone / screen-recording prompts), the request_all_permissions command, and the MicLifecycle / CameraPermission state machines all keep their existing public signatures. The device_state_for(permission, empty) helper converts the IPC-returned CameraPermission into the storybook's DevicePickerState so the visual three-state UI (Populated / Empty / PermissionNeeded) renders correctly without the IPC layer knowing about storybook types.

Tested helpers

recorder_page.rs exposes seven pure helpers — all under #[cfg(test)] mod tests:

The pure-function split keeps the presentational view-model conversions covered without needing a wasm32 test harness — the same approach the legacy *_picker.rs files used for resolve_default / selected_label.

ui-storybook — overview

The HTML/Leptos counterpart to wisp-storybook. Every shipped UI component appears here as a Leptos #[component], with its SSR HTML locked to an insta snapshot — same regression discipline as wisp's quadrant fingerprints.

Linear: AUT-120

Rendered demos live under assets/ui/. Each is a complete standalone HTML file with the storybook stylesheet inlined; open it in a browser tab for a live render.

Regenerate with just snapshots-ui.

Workbench layout

flowchart LR
    Fixtures[fixtures/*] -->|owned mock structs| Stories
    Components[components/*] -->|Leptos components| Stories
    Stories[stories/*] -->|all_stories aggregator| Registry
    Registry --> Snapshot[tests/snapshots.rs<br/>SSR-to-HTML snapshot]
    Registry --> Exporter[bin/export_stories.rs<br/>assets/ui/<id>.html]
    Exporter --> mdBook[mdBook chapters<br/>&lt;iframe src=...&gt;]

Three sources of truth feed the storybook:

• crates/ui-storybook/src/components/ — the actual Leptos components. Subgroups: primitives, shell, menus, recorder, library, editor, cursor. Public types are re-exported at the components:: level so ui_storybook::components::Button resolves the same as components::primitives::Button.
• crates/ui-storybook/src/fixtures/ — owned mock data structs. Stories must not hand-roll inline mocks; every device / workspace / recording / track is built from a fixture so the structure stays single-sourced.
• crates/ui-storybook/src/stories/ — one file per component surface (primitives.rs, shell.rs, recorder.rs, editor.rs, menus.rs, library.rs, cursor.rs). Each module returns Vec<Story>; stories::all_stories() aggregates them in display order.

Visual baseline: rust-ui

The component set follows rust-ui's shadcn-style copy-paste convention but lives in this workspace so we can put it under the same gate as the rest of the code.

Visual style: zinc dark palette, subtle 1px borders, 6/10px radius corners, muted secondary text, accent for primary actions and the playhead.

Index

• Presentational contract — the rules that keep components stateless and reusable.
• Components — every component grouped by surface (primitives, shell, recorder, editor, …).
• Dope sheet — the editor timeline.

Presentational contract

Linear: AUT-120

Every Leptos component in crates/ui-storybook is presentational — it renders its props and emits callbacks, full stop. Application state, side effects, and runtime concerns live one layer up in crates/app-ui. This file is the explicit ruleset that keeps the boundary honest.

The rules

Why these rules exist

The contract also means each component slots into a different host unchanged: the same DropZone works in the Tauri app, in the storybook, in a hypothetical web preview, and in a future test harness. Internal state would tie the component to a specific host's lifecycle.

Wisp / canvas components

Components that need a <canvas> (the editor preview, the cursor preview canvas, the display source thumbnail) follow a two-path rule:

• A feature-gated Wisp-backed story / export path under #[cfg(feature = "csr")] that mounts wisp into the canvas.
• A deterministic non-Wisp fallback for SSR + mdBook — a static PNG sprite or a CSS-only placeholder. The story renders the fallback by default; tests only see deterministic HTML.

This is how UI-07 (DisplaySourceCard), UI-17 (WispCanvasHost), and UI-21 (CursorPreviewCanvas) all stay in-bounds.

Enforcement

Composition

Components compose by passing data + callbacks through plain props. A higher-level surface (e.g. TrayRecordPopover from UI-12) is built by stacking lower-level primitives (CaptureModeTabs / DisplaySourceCard / CaptureSourceRow / SystemAudioPickerList / OnScreenOptionsPopover / RecordingControlsFooter) — none of which import any of the others. The popover's parent in app-ui owns the state machine and threads selections back via callback props.

Empty subgroups

Some subgroups (menus, library, cursor) start empty — the follow-up tickets fill them in. The empty pub mod declarations in components/mod.rs keep the structure visible so authors know where new components belong instead of inventing parallel locations.

State boundaries

Linear: AUT-143

The shortest possible description of where state lives in this workspace:

sequenceDiagram
    autonumber
    participant App as "crates/app-ui (runtime state owner)"
    participant Comp as "ui-storybook component (controlled)"
    participant User as "User"

    App->>Comp: view-model props (snapshot of state)
    User->>Comp: click / keypress / drag
    Comp->>App: callback (on_click / on_select / …)
    App->>App: update signal / dispatch Tauri command
    App->>Comp: next view-model on re-render

What goes where

Examples

#![allow(unused)]
fn main() {
// ✅ Good — controlled, callback-out
#[component]
pub fn ToggleSwitch(
    checked: bool,
    on_change: Option<Callback<bool>>,
) -> impl IntoView { /* … */ }
}

#![allow(unused)]
fn main() {
// ❌ Bad — owns app state, calls runtime services
#[component]
pub fn ToggleSwitch() -> impl IntoView {
    let (checked, set_checked) = signal(false);          // ← no signals in components
    Effect::new(move |_| {                               // ← no effects either
        tauri::invoke("preference_set", ...);            // ← no invoke
    });
    // …
}
}

Story-only interactive wrappers can still create a signal to make the demo clickable in the browser — that's allowed as long as it lives in stories/ and isn't exported from components/.

Allowed in components

Story-only wrappers

If a CSR demo needs internal state (e.g. a dropdown that opens on click for the browser preview), wrap the controlled component in a story-only thin component:

#![allow(unused)]
fn main() {
// stories/my_story.rs — NOT exported from components
#[component]
fn DemoWrapper() -> impl IntoView {
    let open = RwSignal::new(false);
    view! {
        <SelectPill open=open.get() />
    }
}
}

The wrapper lives in stories/, not in components/, so the grep guardrail allows it. The exported SelectPill itself stays controlled.

Review checklist

Linear: AUT-143

Use this list when reviewing a PR that touches crates/ui-storybook/src/components/. Each line maps to a rule in the presentational contract.

File scan

• No use leptos::reactive::*; outside a cfg(test) block.
• No RwSignal::new, signal(, Effect::new, Effect::watch, Action::new in components/.
• No tauri::, invoke(, wasm_bindgen::start, web_sys::window().local_storage().
• No set_interval, setTimeout, gloo_timers.
• No std::fs, std::process, tokio::spawn inside components.
• No lazy_static!, OnceCell, Lazy-style globals.

Props / API surface

• Every visual state has a named prop (selected, open, disabled, loading, etc.) — not a derived internal bool.
• Optional callbacks are typed Option<Callback<T>> with #[prop(optional)]. None is the SSR-stable default.
• Long view-model structs decompose at the top of the component so the view! body reads as flat HTML.
• No Option<Option<T>> from accidentally wrapping Option<Children> in Some(...).

Story coverage

• Every new variant has at least one story in ui_storybook::stories::all_stories().
• Stories sweep at minimum: default + active/open + disabled + empty/overflow.
• Story id is kebab-case and matches the asset HTML filename.
• If the new component takes a Children slot, at least one story exercises it.

mdBook chapter

• New chapter under _docs/book/src/ui/chunks/<id>.md.
• Iframe embed of the default story near the top.
• States table.
• API code block.
• At least one admonish important for the non-obvious rule.
• Mermaid (no ASCII) if a diagram is needed.
• Listed in _docs/book/src/SUMMARY.md.

Gate

• just snapshots-ui re-exports the HTML assets.
• just gate is green (fmt + check + clippy + nextest + doctest + docs + snapshots-check + mermaid-check).
• PROGRESS.md has a new entry with files / tests / verified lines.

Wisp / canvas region only

• Component declares a CanvasBackendView-style enum.
• CSS fallback renders without any browser API.
• If a Wisp asset is referenced, the PNG is committed under _docs/book/src/assets/ui/.
• No direct wgpu import in the component.

Common foot-guns

• Show when=… closure is 'static — captures a bool, not a borrowed String.
• No Some(ToChildren::to_children(...)) — pass bare for optional slots.
• Component file under 100 lines per fn (clippy too_many_lines); split the view into helpers.
• Pre-existing rustdoc intra-doc links don't reference ComponentName::method — Leptos components are fns, not types.

Shared fixture library

Linear: AUT-142

Every storybook component reads its sample data from one of the per-surface fixture modules under crates/ui-storybook/src/fixtures/. This page is the index of what each fixture provides and why stable, deterministic fixture data matters.

Why fixtures matter

The component stories double as our SSR snapshot suite. If a story were to invent its own mock data inline, each story would drift into its own dialect of "what a recording looks like" and the snapshot diff between PRs would become noise — every component change would touch every story body.

Centralizing the canonical samples per surface means:

• Fixtures never use randomness, Instant::now, the local filesystem, or any OS state.
• Stable IDs (rec-01, ws-northwind, space-team) so snapshot diffs are reviewable.
• Fixtures map 1:1 to real DTOs — when the runtime crate lands a real Recording struct, the fixtures can be replaced (not rewritten) by mappers.

Module index

Contact sheet

The fixture-gallery story renders one tile per surface so designers can review the canonical data shape at a glance:

Adding a new fixture

1. Decide which per-surface module owns it (or create one).
2. Add the builder as pub fn sample_<thing>() -> Vec<...>.
3. If the new fixture should appear on the contact sheet, extend contact_sheet::default_ui_fixtures() and the fixtures-contact-sheet story.
4. Add a test asserting non-empty + stable IDs.

UI — components

Index of every shipped component, grouped by the same product-surface subfolders as crates/ui-storybook/src/components/. New components must land in the matching subgroup AND have at least one story registered in stories::all_stories().

Linear: AUT-120

Primitives

Button

Variants: Default, Outline, Ghost, Destructive, Secondary. Sizes: Sm, Md, Lg. Disabled state.

Open as live demo →

Button sizes

Open as live demo →

Card

Card, CardHeader { title, subtitle }, CardBody. Composable surface container — used everywhere the editor groups controls.

Open as live demo →

Shell

Drop zone

States: Idle, Active. Used as the editor's empty state and as the "drag your recording here" surface.

Status bar

Kinds: Ready, Busy, Error. Bottom-of-window strip — FPS, encoder, file size, transient detail line.

Recorder

Recording toolbar

States: Idle, Recording, Paused. The legacy single-row toolbar; the new compositions (UI-11 footer, UI-12 tray popover) live alongside it.

Editor + Player

• Dope sheet — full chapter with multi-track + dense variants.
• PlayerControls — three positions (paused at start / playing mid-clip / near end of clip).
• editor-mock composition.

Menus / Library / Cursor

Empty in UI-00 — the follow-up tickets land components into these subgroups:

• Menus: UI-03 (MenuShell, MenuItem, popover anchors), UI-05 (WorkspaceSwitcherMenu), UI-10 (OnScreenOptionsPopover).
• Library: UI-14 (LibrarySidebar + storage meter), UI-15 (RecordingCard + LibraryGrid).
• Cursor: UI-20 (CursorStudioShell + style picker), UI-21 (CursorPreviewCanvas).

Design tokens

Linear: AUT-121

Semantic CSS variables that every component reaches for instead of the raw hex literals. Adding a new product surface means picking from this list — not introducing a new --bg-purple-deep for one component. The raw zinc palette stays in style.css :root as implementation detail; semantic aliases are the public API.

Open as live demo →

Token table

Adding a new token

1. Add the variable to :root in style.css with a comment naming the surfaces that need it.
2. Add a row to the table above.
3. Use it from the component CSS — never reach for a raw zinc value.
4. If the new token can be derived from an existing one (alpha variant, hover state), prefer color-mix(...) in CSS over a new hex literal.

Surface primitives

Linear: AUT-121

Foundational rendering primitives — Surface, Badge, Divider, Kbd, IconTile. Every other UI surface composes from these.

Surface

Five kinds — pick one per surface. Drives the background, border, and shadow tokens applied.

Open as live demo →

#![allow(unused)]
fn main() {
use ui_storybook::components::{Surface, SurfaceKind};

view! {
    <Surface kind=SurfaceKind::Popover>
        <p>"Tray menu content here"</p>
    </Surface>
}
}

Badge

Open as live demo →

Six kinds, each maps to a recurring badge usage in the recorder:

Divider

Thin separator. Two orientations:

• Horizontal — full width, 1px tall. Default.
• Vertical — 1px wide, stretches to parent height. Use inline for toolbar separators.

Kbd

Keyboard-shortcut chip. Pass an ordered slice; each element renders as a <kbd>:

#![allow(unused)]
fn main() {
use ui_storybook::components::Kbd;

view! { <Kbd keys=vec!["⌘", "⇧", "R"] /> }
}

IconTile

Small square tile for inline icons / monograms. Five kinds:

Navigation rail

Linear: AUT-122

Left-edge nav for every product surface — record setup, library, editor, cursor studio, prefs. Structural only: the selected section is passed in from above; the rail never owns that state.

Open as live demo →

Selected per section

The active class is applied only to the matching item — no internal state, no router. Each story below renders the rail with a different active=AppSection::…:

Composition

flowchart LR
    Rail[NavigationRail] --> Workspace[WorkspaceBadge]
    Rail --> Items[NavItemView × N]
    Rail --> User[UserAvatar?]
    Workspace -->|on_workspace_click| Parent
    Items -->|active=AppSection| Parent
    User -->|on_user_click| Parent

The rail composes three primitives from shell/:

• WorkspaceBadge at the top — the red workspace tile + chevron. Visual only; the workspace switcher menu (UI-05) is the parent's job.
• A list of NavItemView rows, one per AppSection.
• UserAvatar at the bottom — optional. Pass None on surfaces that don't show a signed-in user.

API

#![allow(unused)]
fn main() {
use ui_storybook::components::{
    AppSection, NavItemView, NavigationRail,
    WorkspaceBadgeView, UserAvatarView,
};

view! {
    <NavigationRail
        items=items                // Vec<NavItemView>
        active=AppSection::Editor  // explicit prop
        workspace=workspace        // WorkspaceBadgeView
        user=user                  // Option<UserAvatarView>
    />
}
}

NavItemView lets every row carry icon, label, optional notification count, and a disabled flag (rendered with reduced opacity + aria-disabled).

States covered

App shell

Linear: AUT-122

Top-level layout with slots for rail / main / inspector / titlebar / footer. Each product screen (library, editor, cursor studio, prefs) mounts one AppShell with the panes it needs — chrome stays consistent across surfaces.

Open as live demo →

Slots

Composition

flowchart TD
    Shell[AppShell] --> Title[titlebar]
    Shell --> Body[body]
    Body --> Rail[rail<br/>NavigationRail]
    Body --> Main[main<br/>per-section content]
    Body --> Inspector[inspector?]
    Shell --> Foot[footer?]

API

#![allow(unused)]
fn main() {
use ui_storybook::components::{AppShell, NavigationRail, /* … */};

view! {
    <AppShell
        rail=ToChildren::to_children(move || view! { <NavigationRail … /> })
        main=ToChildren::to_children(move || view! { /* editor canvas / library grid / … */ })
        inspector=ToChildren::to_children(move || view! { /* property rows */ })
        titlebar=ToChildren::to_children(move || view! { <span>"Recording 02"</span> })
        footer=ToChildren::to_children(move || view! { /* status bar */ })
    />
}
}

Each slot accepts Children (a Box<dyn FnOnce() -> AnyView>). Optional slots accept Option<Children>; pass ToChildren::to_children(...) to populate them and omit the prop to skip.

What this unlocks

Every UI-14..21 ticket plugs into this shell. The library is AppShell { rail: …, main: LibraryGrid, inspector: None }. The editor is AppShell { rail: …, main: EditorCanvas, inspector: InspectorPanel, footer: TimelineSkeleton }. Cursor Studio is AppShell { rail: …, main: CursorPreviewCanvas, inspector: CursorStyleControls }. None of those slots care about the others.

Popover surface

Linear: AUT-123

Chrome shared by every tray menu / dropdown / on-screen-options popover. Owns the corner radius, drop shadow, header / body / footer slots, and a placement class the parent overlay layer uses to position the surface. The component itself doesn't compute coordinates — that's the parent's job.

Open as live demo →

With a footer

API

#![allow(unused)]
fn main() {
use ui_storybook::components::{
    PopoverSurface, PopoverPlacement, MenuList, MenuRow,
};

view! {
    <PopoverSurface
        placement=PopoverPlacement::BottomLeft
        width_px=300_u16
        title="Choose camera"
        footer=ToChildren::to_children(|| view! { /* primary action */ })
    >
        <MenuList label="Cameras">
            /* … rows */
        </MenuList>
    </PopoverSurface>
}
}

Placement classes

Menu row

Linear: AUT-123

The single row used inside every popover menu — workspace switcher, device pickers, system-audio picker, on-screen options, future editor menus. The shape covers ~95% of the recurring menu-row needs without bespoke CSS per menu.

Kinds

Slots

• leading — Option<Children>. Typically an IconTile (device / app / workspace flavor).
• title — required String. Truncates with text-overflow: ellipsis when the row narrows.
• subtitle — optional String. Same truncation behavior.
• badges — Vec<MenuBadgeView>. Each (label, BadgeKind) pair renders inline between the text and trailing slot.
• trailing — Option<Children>. Typically a Kbd shortcut chip or a chevron glyph.

Composition example

flowchart LR
    Popover[PopoverSurface] --> Header[header: title + description]
    Popover --> Body[body]
    Body --> List[MenuList]
    List --> Section[MenuSection]
    Section --> Row[MenuRow]
    Row --> Leading[leading: IconTile]
    Row --> Text[title + subtitle]
    Row --> Badges[badges]
    Row --> Trailing[trailing: Kbd]
    Popover --> Footer[footer: MenuFooter]

Long-label behavior

Titles + subtitles truncate at the row's max-width. The badges + trailing slot stay visible at full size; only the text column shrinks. This is why the row uses flex: 1 on the text column and flex-shrink: 0 on the badge / trailing columns.

Workspace switcher menu

Linear: AUT-125

Popover anchored to the rail's WorkspaceBadge. Pure composition of UI-03 menu primitives + UI-01 surface tokens — no bespoke CSS.

Open as live demo →

States covered

Composition

flowchart TD
    Menu[WorkspaceSwitcherMenu] --> Popover[PopoverSurface]
    Popover --> ListBody[MenuList]
    ListBody --> Group[MenuSection — Your workspaces]
    Group --> Row[MenuRow × N]
    Row --> Tile[span.icon-tile-workspace]
    Row --> Title[name]
    Row --> Sub[member count]
    Row --> Plan[Badge — Plan]
    ListBody --> Actions[MenuSection — Actions]
    Actions --> New[MenuRow — New workspace]
    Actions --> Settings[MenuRow — Workspace settings]
    Popover --> Footer[MenuFooter]

API

#![allow(unused)]
fn main() {
use ui_storybook::components::{WorkspaceSwitcherMenu, WorkspaceView};

view! {
    <WorkspaceSwitcherMenu
        workspaces=fixtures::workspaces::sample_workspace_views()
        selected_id="ws-northwind"
    />
}
}

Member count formatting

format_member_count(u32) -> String renders 1 member (singular) vs N members (plural). Unit-tested for both branches + zero.

Controls

Linear: AUT-124

Seven new control primitives that expand the Button vocabulary into everything the recorder + editor + cursor studio need. All stateless; all driven by props.

Icon buttons

IconButtonVariant { Ghost, Filled, Danger }. Supports pressed for toggle-style icon buttons and disabled. Accessible label is required since the button has no visible text.

Toggle switch

Segmented control

Vec<Segment> + active: String (the segment id). Each segment can have an optional leading icon glyph + disabled flag.

Slider

Slider { value, min, max, disabled, label, readout }. Pure visual — renders track + fill + thumb at the computed percent. slider_percent helper is pub and unit-tested for clamping behavior.

Color swatches

Circular tiles with an outer ring when selected. Used in the cursor studio color picker.

Meters

Audio-level bars driven by a normalized [0, 1] level. bar_count defaults to 12; danger=true switches the lit color from emerald to the action-record red (used for clipping). lit_segments is pub + unit-tested.

Select pill

Bundled in the meters demo above. Compact pill that opens a popover when clicked. The component renders the pill chrome only; the popover content is the parent's job (typically a PopoverSurface from UI-03).

Capture mode tabs

Linear: AUT-126

Three tabs at the top of the tray record popover: Screen / Window / Area. The first visible control on the record-from-tray path — it determines which setup UI is showing.

API

#![allow(unused)]
fn main() {
use ui_storybook::components::CaptureModeTabs;
use ui_storybook::fixtures::recorder::CaptureMode;

view! {
    <CaptureModeTabs
        selected=CaptureMode::Screen
        // optional — gray out modes the user can't enter:
        disabled_modes=vec![CaptureMode::Area]
    />
}
}

Composition

flowchart LR
    Tabs[CaptureModeTabs] -->|maps CaptureMode →| Segments[Segment × 3]
    Tabs --> SC[SegmentedControl]
    SC --> Segments

CaptureModeTabs is a thin wrapper that maps the three CaptureMode enum values to Segments and forwards them to SegmentedControl. The component itself owns no state; selected is a prop.

Display source card

Linear: AUT-127

Tray-popover card that shows which screen is selected for recording. Header carries the display name, size, favourite glyph, resolution pill, and chevron; body holds the mock preview frame.

States

API

#![allow(unused)]
fn main() {
use ui_storybook::components::{DisplaySourceCard, DisplaySourceView};
use ui_storybook::fixtures::devices::sample_display_source;

view! {
    <DisplaySourceCard view=sample_display_source(true) open=false />
}
}

DisplaySourceView carries the static metadata (name, size, dimensions, favorite, selected) plus a DisplayPreviewView for the preview frame.

Preview frame

DisplayPreviewFrame is the mocked canvas. It renders a titlebar strip + a body containing positioned PreviewWindowChips. Each chip is a colored rounded rect at a percentage offset / size — the deterministic "non-Wisp fallback" called out in the spec.

Aspect-ratio helper

aspect_ratio_css(num, den) -> String produces "<num> / <den>" for the CSS aspect-ratio property, falling back to "1 / 1" when the denominator is zero. Unit-tested.

Composition

flowchart TD
    Card[DisplaySourceCard] --> Header[header]
    Header --> Name[display-source-name]
    Header --> Size[size pill]
    Header --> Star[favourite ★]
    Header --> Dims[resolution pill]
    Header --> Chevron[chevron]
    Card --> Frame[DisplayPreviewFrame]
    Frame --> Titlebar[mac-window titlebar]
    Frame --> Body[preview body]
    Body --> Chip[PreviewWindowChip × N]
    Body --> Overlay[overlay label]
    Card --> Banner[unavailable banner?]

Capture source row

Linear: AUT-128

Collapsed row in the tray record popover for the camera + microphone slots. Five grid columns: leading icon tile, title + subtitle column, optional live meter, on/off toggle, expand chevron.

Camera · Microphone with meter

API

#![allow(unused)]
fn main() {
use ui_storybook::components::{
    CaptureSourceRow, CaptureSourceView, CaptureSourceKind,
};

view! {
    <CaptureSourceRow view=CaptureSourceView {
        id: "mic-built-in",
        kind: CaptureSourceKind::Microphone,
        title: "MacBook Pro Microphone",
        subtitle: "Built-in · 48 kHz",
        enabled: true,
        expanded: false,
        favorite: true,
        level: Some(0.45),
    } />
}
}

Composition

flowchart LR
    Row[CaptureSourceRow] --> Icon[IconTile · Device]
    Row --> Text[title + subtitle]
    Row --> Meter[Meter · microphone only]
    Row --> Toggle[ToggleSwitch]
    Row --> Chevron[chevron · expand]

Reuses IconTile (UI-01), Meter (UI-04), ToggleSwitch (UI-04). No new primitives — composition only.

Device picker menu

Linear: AUT-128

Popover that opens from the CaptureSourceRow chevron. Lists available devices for the camera or microphone slot. Composes UI-03 PopoverSurface + MenuList + a custom device-picker-row shape (thumbnail + name/detail + optional badge + optional meter + selected check).

States

API

#![allow(unused)]
fn main() {
use ui_storybook::components::{
    DevicePickerMenu, DevicePickerState, DeviceOptionView,
    CaptureSourceKind,
};

view! {
    <DevicePickerMenu
        kind=CaptureSourceKind::Camera
        devices=fixtures::devices::sample_camera_options()
        // optional — defaults to Populated
        state=DevicePickerState::Populated
    />
}
}

DeviceOptionView shape

#![allow(unused)]
fn main() {
pub struct DeviceOptionView {
    pub id: &'static str,
    pub name: &'static str,
    pub detail: &'static str,
    pub badge: Option<&'static str>,    // "Wireless", "New"
    pub selected: bool,                 // ✓ checkmark
    pub level: Option<f32>,             // microphone meter
    pub thumbnail: Option<DeviceThumb>, // camera thumbnail tile
}
}

Cameras typically set thumbnail + level = None. Microphones typically set level = Some(0..=1) + thumbnail = None. The component handles both gracefully.

Composition

flowchart TD
    Picker[DevicePickerMenu] --> Surface[PopoverSurface]
    Surface --> Body[body slot]
    Body --> Branch{state}
    Branch -->|Populated| List[MenuList → MenuSection × N → device-picker-row × N]
    Branch -->|Empty| EmptyState[icon + headline + subtitle]
    Branch -->|PermissionNeeded| PermState[warn icon + headline + subtitle]
    Surface --> Footer[MenuFooter — Connect/Pair button]

System audio picker

Linear: AUT-129

Tray-popover section for picking which apps' audio joins the recording. Two components: a collapsed row (selected count + overlapping app-icon stack + toggle) and an expanded list (filter chips + per-app rows with selection checkbox, Suggested badge, LIVE pulse, and meter).

States

API

#![allow(unused)]
fn main() {
use ui_storybook::components::{
    SystemAudioRow, SystemAudioAppList, AudioFilter,
};

// Collapsed row.
view! { <SystemAudioRow view=sample_system_audio_view(true, false, &selected, total) /> }

// Expanded list.
view! { <SystemAudioAppList apps=sample_audio_apps() active_filter=AudioFilter::Suggested /> }
}

```admonish important title="active_filter is cosmetic only" The filter chip just highlights "All", "None", or "Suggested". The component does NOT change apps.selected — the parent already applied the filter to the apps slice it passes in. Clicking a chip in production fires on_filter_select(AudioFilter) and the parent recomputes selection.

## Helpers

- `format_selection_count(selected: usize, total: usize) -> String`
  → `"4 of 7 apps"` (plural) / `"1 of 1 app"` (singular). 5 unit tests.
- `ICON_STACK_MAX: usize = 3` — overflow tips to a `+N` pill above
  that count. Tested.

## Composition

```mermaid
flowchart TD
    Row[SystemAudioRow] --> Stack[icon stack — first 3 + overflow pill]
    Row --> Count[selected_count / total_count]
    Row --> Toggle[ToggleSwitch]
    Row --> Chevron[chevron expand]
    List[SystemAudioAppList] --> Filters[filter chips — All / None / Suggested]
    List --> Rows[AudioAppView × N]
    Rows --> Check[checkbox]
    Rows --> Icon[app icon tile]
    Rows --> Text[name + context]
    Rows --> Suggested[Suggested badge?]
    Rows --> Live[LIVE pulse + label?]
    Rows --> Meter[Meter · 8-bar?]

Composition uses UI-01 Badge, UI-04 Meter, UI-04 ToggleSwitch — no new primitives.

On-screen options popover

Linear: AUT-130

Tray popover that controls what shows during recording — desktop cleanup, keypress overlay, sensitive-info auto-blur. Composes UI-03 PopoverSurface + UI-04 ToggleSwitch. Three options today; more can land by extending OnScreenOptionKind.

States

API

#![allow(unused)]
fn main() {
use ui_storybook::components::{
    OnScreenOptionsPopover, OnScreenOptionKind, OnScreenOptionView,
};
use ui_storybook::fixtures::recorder::sample_on_screen_options;

view! {
    <OnScreenOptionsPopover
        options=sample_on_screen_options(false)
        // optional — defaults to "Applies to this recording".
        applies_label="Applies to all recordings"
    />
}
}

OnScreenOptionKind

A stable enum so parents can pattern-match on a row instead of string-comparing:

• CleanDesktop — hide desktop icons + dock
• ShowKeys — render keypress badges over the recording
• BlurSensitiveInfo — auto-detect + blur sensitive regions

Composition

flowchart TD
    Popover[OnScreenOptionsPopover] --> Surface[PopoverSurface · title + description]
    Surface --> List[ul.on-screen-options]
    List --> Row[OnScreenOptionView × N]
    Row --> Toggle[ToggleSwitch]
    Row --> Text[title + description]
    Surface --> Footer[MenuFooter · applies_label + Done button]

Recording controls footer

Linear: AUT-131

Bottom row of the tray record popover — the final visible step before the first recording starts. Auto-zoom + countdown pills on the left, prominent red Start recording button on the right with the keyboard shortcut hint inside.

States

API

#![allow(unused)]
fn main() {
use ui_storybook::components::{
    RecordingControlsFooter, RecordingControlsView, StartRecordingState,
};
use ui_storybook::fixtures::recorder::sample_recording_controls;

view! {
    <RecordingControlsFooter
        view=sample_recording_controls(StartRecordingState::Ready)
        // optional callbacks; stories leave them unset.
    />
}
}

Composition

flowchart LR
    Footer[RecordingControlsFooter]
    Footer --> Zoom[AutoZoomSelect · SelectPill]
    Footer --> Countdown[CountdownSelect · SelectPill]
    Footer --> Start[StartRecordingButton]
    Start --> Glyph[●  / ◌  / ⚠ ]
    Start --> Label["Start recording"]
    Start --> Shortcut[ShortcutBadgeGroup · ⌘ ⇧ 2]

Save panel

The post-record Save panel. Once a recording stops it isn't written straight to disk — it's parked in a scratch file awaiting export, and this panel replaces the record/stop footer so the user can pick the output folder + container format, then Export or Discard. After a successful export it flips to a "Saved to <path>" confirmation with Reveal in Finder / Done.

It is presentational: the parent (app-ui's RecorderPage) owns the pending export, the chosen format, the in-flight flag, and the saved path, maps them into a SavePanelView, and wires the callbacks to the Tauri IPC commands.

States

stateDiagram-v2
    [*] --> Choosing: recording stopped (pending export)
    Choosing --> Exporting: Export (busy = true)
    Exporting --> Saved: move / transcode succeeds
    Exporting --> Choosing: export fails (pending restored)
    Choosing --> [*]: Discard (scratch deleted)
    Saved --> [*]: Done

API

#![allow(unused)]
fn main() {
use ui_storybook::components::recorder::{SaveFormat, SavePanel, SavePanelView};

view! {
    <SavePanel
        view=SavePanelView::Choosing {
            output_dir: "/Users/you/Movies/Screen".into(),
            format: SaveFormat::Mp4H264,
            busy: false,
        }
        // optional callbacks; stories leave them unset.
        // on_change_folder / on_format_change / on_discard /
        // on_export / on_reveal / on_done
    />
}
}

Tray record popover

Linear: AUT-132

Top-level composition: the floating black rounded window the user sees first. Pulls UI-02 → UI-11 together — workspace chip, capture-mode tabs, display source card, camera + microphone rows, system audio, on-screen overlay summary, and the recording controls footer.

States

API

#![allow(unused)]
fn main() {
use ui_storybook::components::recorder::{
    TrayRecordPopover, OpenRecorderPopoverKind,
};
use ui_storybook::fixtures::recorder::sample_tray_record_popover;

view! {
    <TrayRecordPopover view=sample_tray_record_popover(OpenRecorderPopoverKind::None) />
}
}

Composition

flowchart TD
    Popover[TrayRecordPopover] --> Header[Workspace chip + CaptureModeTabs]
    Popover --> Body[tray-record-body]
    Body --> Display[DisplaySourceCard]
    Body --> Sources[CaptureSourceRow × 2]
    Body --> Audio[SystemAudioRow + AppList]
    Body --> OnScreen[On-screen summary chip]
    Popover --> Footer[RecordingControlsFooter]
    Popover --> Overlay[OpenRecorderPopoverKind overlay]

Recording status button

Linear: AUT-133

Compact pill that replaces the red Start button after capture begins. Used in the system tray and any other small surface that needs the live recording status without the full footer.

States

API

#![allow(unused)]
fn main() {
use ui_storybook::components::recorder::{
    RecordingStatusButton, CompactRecordingState,
};

view! {
    <RecordingStatusButton
        state=CompactRecordingState::Recording {
            elapsed_label: "00:42".into(),
        }
        shortcuts=vec!["⌘".into(), "⇧".into(), "2".into()]
    />
}
}

Composition

stateDiagram-v2
    [*] --> Countdown
    Countdown --> Recording: countdown reaches 0
    Recording --> Paused: pause
    Paused --> Recording: resume
    Recording --> Stopping: stop
    Paused --> Stopping: stop
    Stopping --> Stopped: encoder done
    Recording --> Error: capture failure
    Paused --> Error: capture failure

Library sidebar

Linear: AUT-134

Left rail of the library screen. Primary nav rows (New / All / Starred / Shared / Inbox), SPACES, TAGS, and a bottom storage quota meter that turns red past 85%.

States

API

#![allow(unused)]
fn main() {
use ui_storybook::components::library::{LibrarySidebar, LibrarySidebarView};
use ui_storybook::fixtures::library::sample_library_sidebar;

view! {
    <LibrarySidebar view=sample_library_sidebar(/* inbox_unread */ 3) />
}
}

Recording card + library grid

Linear: AUT-135

The library's card primitive and the grid that lays cards out under a filter + sort + layout toolbar. Card lifecycle covers Ready, Processing (with percent overlay), and Failed.

Card states

Grid states

API

#![allow(unused)]
fn main() {
use ui_storybook::components::library::{
    LibraryGrid, RecordingCard, RecordingCardState,
};
use ui_storybook::fixtures::library::{sample_library_grid, sample_recording_cards};

view! { <LibraryGrid view=sample_library_grid() /> }
}

Editor shell

Linear: AUT-136

Structural shell for the editor screen — macOS-style title bar, top toolbar (16:9 / Crop / Annotate / Trim + Share / Export), and three slot regions for the canvas, inspector, and timeline.

States

API

#![allow(unused)]
fn main() {
use ui_storybook::components::editor::{EditorShell, EditorShellView};
use ui_storybook::fixtures::editor::sample_editor_shell;

view! {
    <EditorShell view=sample_editor_shell(/* has_clip_loaded */ true) />
    // optional children: canvas, inspector, timeline
}
}

Editor drop zone + Wisp canvas host

Linear: AUT-137

The editor's center canvas region. WispCanvasHost picks a backend — CssFallback (used by SSR + mdBook), a pre-rendered WispAsset { asset_path }, or WispRuntimeUnavailable (renders a warning banner). EditorDropZoneCanvas wraps the host in the dotted drop overlay + action cards + recent-clips strip the user sees when no clip is loaded.

States

API

#![allow(unused)]
fn main() {
use ui_storybook::components::editor::{
    EditorDropZoneCanvas, WispCanvasHost, CanvasBackendView,
};
use ui_storybook::fixtures::editor::sample_editor_drop_zone;

view! {
    <EditorDropZoneCanvas view=sample_editor_drop_zone(/* drag_active */ false) />
}
}

Composition

flowchart TD
    DropZone[EditorDropZoneCanvas]
    DropZone --> Host[WispCanvasHost]
    Host --> Backend{CanvasBackendView}
    Backend -->|CssFallback| Css[CSS checkered + label]
    Backend -->|WispAsset| Img[<img> pre-rendered]
    Backend -->|WispRuntimeUnavailable| Banner[Warning banner]
    DropZone --> Content[Drop overlay + headline + actions]
    DropZone --> Recent[Recent clips strip]

Inspector panel

Linear: AUT-138

Right-pane inspector used by both the editor and the cursor studio. Tab strip (Style / Cursor / Audio / Captions / AI) + zero-or-more PropertySection blocks rendering rows with slider / toggle / color swatch / pill controls.

States

API

#![allow(unused)]
fn main() {
use ui_storybook::components::editor::{InspectorPanel, InspectorTab};
use ui_storybook::fixtures::editor::sample_inspector_style_tab;

view! { <InspectorPanel view=sample_inspector_style_tab() /> }
}

Timeline skeleton

Linear: AUT-139

Layout-only timeline scaffold — transport row + per-track labels + dashed placeholder content. The real keyframe editing lives in DopeSheet; this skeleton is what the editor renders below the canvas when no clip is loaded or when a track has nothing on it.

States

API

#![allow(unused)]
fn main() {
use ui_storybook::components::editor::{TimelineSkeleton, TimelineView};
use ui_storybook::fixtures::editor::sample_timeline_skeleton;

view! { <TimelineSkeleton view=sample_timeline_skeleton() /> }
}

Cursor Studio shell + style picker

Linear: AUT-140

The bottom style strip in Cursor Studio (System / Arrow / Soft / Dot / Ring / Reticle / Tactile / Hide) plus the structural shell that wires the preview slot + inspector slot + picker together.

States

API

#![allow(unused)]
fn main() {
use ui_storybook::components::cursor::{CursorStudioShell, CursorStyle, CursorStylePicker};
use ui_storybook::fixtures::cursor::{sample_cursor_studio_shell, sample_cursor_style_picker};

view! { <CursorStudioShell view=sample_cursor_studio_shell() /> }
}

Cursor preview canvas + appearance controls

Linear: AUT-141

CursorPreviewCanvas reuses the same backend pattern as the editor canvas (CSS fallback / Wisp asset / runtime unavailable) so a pixel- accurate Wisp preview can drop in later without changing the component contract. CursorAppearancePanel composes UI-18 inspector primitives into APPEARANCE / CLICK EFFECT / MOTION / BEHAVIOR sections with a footer (Reset / Apply).

Preview states

Appearance panel states

API

#![allow(unused)]
fn main() {
use ui_storybook::components::cursor::{
    CursorPreviewCanvas, CursorPreviewBackend, CursorAppearancePanel,
};
use ui_storybook::fixtures::cursor::sample_cursor_appearance;

view! {
    <CursorPreviewCanvas backend=CursorPreviewBackend::CssFallback />
    <CursorAppearancePanel view=sample_cursor_appearance() />
}
}

Button — variants

Five variants in a single row: Default, Outline, Ghost, Destructive, Secondary. Same shape, same height, different role.

The variant decides only the surface treatment (bg-* / border-*) — the typography, padding, and corner radius come from btn + the size class. That separation is why we can ship a new variant by adding one CSS rule and one enum case.

Class hooks mirror rust-ui's, so swapping in Tailwind later is a search- and-replace, not a rewrite.

Open as standalone demo →

---

Button API · Components index

Button — sizes

Sm (28px), Md (34px, default), Lg (40px), plus a disabled state shown on the right.

Sizes scale font size, padding, and overall height in lockstep so that the optical mass scales linearly. The disabled state is just opacity: 0.5 + cursor: not-allowed — variant-agnostic.

Open as standalone demo →

---

ButtonSize · Components index

Card — header + body

Card is a surface container. CardHeader { title, subtitle } sits above a CardBody separated by a 1px divider. Used everywhere the editor groups related controls — recording metadata, timeline, export presets.

The composition pattern (<Card> <CardHeader/> <CardBody>…</CardBody> </Card>) is rust-ui-flavored: small composable building blocks rather than a single fat component with a dozen props.

Open as standalone demo →

---

Card API · Components index

Drop zone — idle

The recorder's import surface in its resting state. Dashed outline, neutral copy, optional keyboard hint chip ("⌘O to browse").

The Tauri shell hosts a single <DropZone> and flips it between idle and active in response to OS-level drag events (tauri::Window::on_window_event → WindowEvent::DragDrop). Pure presentational — the component takes a state: DropZoneState prop, the shell owns the signal that drives it.

Open as standalone demo →

---

DropZone API · Components index

Drop zone — active

The same component, state=Active. Solid accent border (sky-blue, the linear-keyframe color — visually consistent with the editor's "active" language across the app). Background tint shifts to the same sky at low alpha. Glyph picks up the accent.

The transform-scale (1.005) is intentional — small enough not to feel animated-for-its-own-sake, big enough to confirm the drag is recognized under fast cursor motion.

Headline + subtext swap to "Release to import" / "Will open in the editor" so users get a confirmation of what's about to happen before they let go.

Open as standalone demo →

---

DropZoneState · Components index

Player controls — paused

Transport bar at rest. Round play button (▶), 0:00 / 1:24 time display, scrub handle parked at the start.

The component is purely presentational: position is a 0.0..=1.0 fraction the parent owns. format_time rounds seconds to the nearest second so the display ticks once per beat rather than on every frame.

Open as standalone demo →

---

PlayerControls API · Components index

Player controls — playing

Same component, state=Playing, position=0.32. Toggle glyph swaps to ❚❚ (pause), the scrub fill draws to 32%, the handle sits on top of the fill's leading edge with a soft halo on hover.

Time display tracks position: position × duration rounded to whole seconds → 0:27 / 1:24.

Open as standalone demo →

---

PlayState · Components index

Player controls — near end

position=0.94. Confirms the handle sits inside the track at the right edge — no overflow past the rounded end-cap, no clipping. The margin-left: -6px on .player-scrub-handle exactly cancels the half-width so the handle's center aligns with the position percentage.

This story exists specifically to lock that boundary. Without it, a careless tweak to handle dimensions could push the dot off the end of the track and the SSR snapshot wouldn't catch the visual regression — only this kind of "edge case" story does.

Open as standalone demo →

---

PlayerControls · Components index

Recording toolbar — idle

The first surface a user sees when they open the recorder. Status reads "Ready", timer is 00:00, source picker shows the currently-selected display, and a single primary "Start recording" button (red, with a static white dot to telegraph what's about to happen) takes most of the visual weight on the right.

This state is intentionally one-button. We don't want a "Pause" or "Stop" button visible before recording starts — they'd be muted and dead, which is worse than absent.

Open as standalone demo →

---

RecordingToolbar API · Components index

Recording toolbar — recording

State swap to Recording. The dot turns red and pulses (CSS keyframes, ~1.4s loop), the status label colors red, the timer ticks (02:17 here for elapsed_seconds=137.0), and the action stack swaps to Pause (secondary) + Stop (outline).

The pulsing dot is the primary recording-is-on signal. It's visible peripherally — even when the user has the toolbar in the corner of their eye, the motion confirms capture.

The timer formats as M:SS until the hour, then H:MM:SS. Recordings longer than an hour are unusual but the format handles them cleanly.

Open as standalone demo →

---

RecordingState · Components index

Recording toolbar — paused

Paused. The dot stops pulsing and switches to the marker-yellow color (reuses the --kf-marker token, which is the same yellow the dope sheet uses for chapter markers — visual consistency across the app's "interrupted state" language).

Action stack is Resume (primary, red) + Stop (outline). The "Resume" button intentionally uses the same red as the initial "Start recording" — they're the same action, mechanically: begin/continue capture.

Timer freezes at the elapsed value (02:17 here) until resume.

Open as standalone demo →

---

RecordingToolbar · Components index

Status bar — ready

The slim bottom-of-app strip. Three telemetry cells (FPS / Encoder / Size), a flexible spacer, optional detail text, and a right-justified health pill. In Ready state the pill is green with no detail text; encoder shows H.264 · idle; size is 0 B.

The pill is the eye-grabber. Green = good, blue = working, red = broken. Same color language as the recording toolbar's dot but expressed as backgrounded pills (this is a passive surface; the recording toolbar is an active one).

Open as standalone demo →

---

StatusBar API · Components index

Status bar — encoding

StatusKind::Busy. Pill swaps to sky-blue with a pulsing dot (reuses @keyframes rec-pulse from the recording toolbar — same motion language across the app). Detail text reads Encoding · 38% next to the pill.

Encoder cell shows live bitrate (H.264 · 9.4 Mbps); size cell shows the file growing (23.0 MB after format_bytes(24_117_248)). The bytes formatter rolls through B → KB → MB → GB with appropriate fractional digits — a 1.07 GB recording reads more like a recording than a wall of digits.

Open as standalone demo →

---

StatusKind · Components index

Status bar — error

StatusKind::Error. Pill goes red, dot stops pulsing (the user shouldn't mistake an error for in-progress work). FPS reads 0 because the renderer is no longer ticking; encoder cell shows the codec without a live bitrate; size cell freezes at the last known value. Detail text carries the actual error — VideoToolbox: out of memory here.

Whatever appears in detail is the only specific information the user gets at this layer; a longer error log lives in the encoder card (a future chunk). The status bar is a glanceable summary, not a debug pane.

Open as standalone demo →

---

StatusBar · Components index

UI — dope sheet

The editor's timeline. Tracks are rows (video, cursor, audio, captions, effects), columns are time, dots are keyframes, the bright vertical line is the playhead.

A pure presentational component for now — interaction (drag, snap-to-frame, scrub) lives behind a future signal-driven variant. The visual + structural contract is locked in tests/snapshots.rs.

Multi-track baseline

Open as live demo →

Dense keyframes

Twelve keyframes alternating ease/linear on a TrackKind::Effect zoom track, in addition to the four standard tracks. Confirms the dot positioning math holds at high density.

Open as live demo →

Embedded in a card (composition)

The expected production placement: dope sheet wrapped in a Card with title and metadata in the header.

Open as live demo →

Keyframe glyphs

Dope sheet — multi-track

The editor's timeline. Tracks are rows (Video / Cursor / Audio / Caption), columns are time, dots are keyframes, and the bright vertical line is the playhead at t=3.4s.

A pure presentational component for now — interaction (drag, snap-to-frame, scrub) lives behind a future signal-driven variant. The visual + structural contract is locked in tests/snapshots.rs.

The keyframe glyph maps to KeyframeKind:

Open as standalone demo →

---

DopeSheet API · Dope sheet overview

Dope sheet — dense keyframes

Twelve keyframes alternating ease/linear on a TrackKind::Effect zoom track, in addition to the four standard tracks.

Confirms the dot positioning math holds at high density (no overlap or clipping at the seconds boundary). Also exercises the track-effect visual styling — pink-tinted row background — to validate the per-kind gradient approach.

The playhead is at t=5.1s — not on a frame boundary on purpose, to verify the floating-point positioning.

Open as standalone demo →

---

DopeSheet API · Dope sheet overview

Editor panel — card wrapping dope sheet

Composition: a Card with CardHeader ("Timeline · 4 tracks · 8.0s") and a CardBody containing the DopeSheet. This is the expected production placement in the editor — title and metadata up top, the timeline grid underneath.

It's also the canary that catches "two correct components compose incorrectly" bugs: padding, scroll behavior, the playhead's vertical extent all have to coexist with the card's overflow and corner radius. The SSR snapshot test locks this composition's HTML alongside the isolated views.

Open as standalone demo →

---

Card · DopeSheet · Components index

Editor mock — full composition

The whole editor mock in one story: Card (Recording 02 metadata) → preview placeholder (gradient surface) → PlayerControls (playing, mid-clip) → second Card (Timeline) → DopeSheet with the standard four tracks.

This is the reference composition the Tauri shell will mount once the Leptos integration lands (M-INT.1). Three things it locks:

1. Vertical rhythm. Card padding + 12px scrub gap + 14px between cards adds up to a comfortable density without scroll on a 720p editor view.
2. Color reuse. The preview gradient picks up the same sky/violet accents the dope-sheet keyframes use, so the eye reads the editor as one palette.
3. Component boundaries. Card is the only "chrome" — both the player surface and the timeline live inside their own Card, which is the pattern the rest of the editor will follow (settings panel, captions panel, export panel).

Open as standalone demo →

---

Components index · Dope sheet

The editor — Record → Edit → Export (M-EDIT)

If the recorder is the theatre, the editor is the cutting room — and a cutting room has a hundred-year history worth borrowing. In 1924 Iwan Serrurier built the Moviola so an editor could run film backward and forward and stop on a frame; the Steenbeck flatbed made that scrubbing fluid; cuts were a razor blade and a strip of tape; nothing was thrown away — outtakes hung in the trim bin; SMPTE timecode (1967) gave every frame an address; and in 1971 the CMX 600 dragged the whole craft from cutting the negative to non-linear editing — edit by decision list, never touch the source.

That arc is the design. This editor is the cutting room in software: a non-destructive edit decision list, a scrubbable flatbed over a forward-only decoder, frame-addressed by a single clock, re-rendered at export. Each chunk retraces one step:

The flow

flowchart LR
  REC[Record\nsource .mp4 + audio + click log] --> PROJ[EditProject\nthe decision list]
  PROJ --> EDIT[Edit\nsplice · speed · crop · zoom\non a timeline + dopesheet]
  EDIT --> PROJ
  PROJ --> PREVIEW[Preview\nwisp composes each frame\nat the playhead]
  PROJ --> EXPORT[Export\nre-render the timeline → .mp4]

The one idea: an edit is a decision list, not a re-cut negative

The editor never rewrites the recording. Every edit is a small, serializable value stored in an EditProject:

• an ordered list of timeline segments — slices of the source clip. Trimming moves a slice's edges; splitting replaces one slice with two; changing speed sets a slice's timescale.
• a list of zoom regions — cinematic punch-ins, each compiled to a keyframed transform at render time.
• one background / cursor / crop / aspect config — the produced "framing" look.

The renderer (wisp) and encoder (media) re-derive every frame from that model at preview and export time. Editing stays non-destructive, preview and export share one code path (so they match), and the whole edit model is exhaustively unit-testable without a GPU — the modern descendant of "never cut the negative."

Chapters

• The edit model — ED.1
• Edit operations + undo/redo — ED.2
• Random-access decode — ED.3
• Playback clock — ED.4
• Editor surface + handoff — ED.5
• Editor preview canvas — ED.6
• Playback transport — ED.7
• Timeline ruler + coordinate system — ED.8
• Video track + clip selection — ED.9
• Audio waveform lane — ED.10
• Splitting, ripple-delete + undo/redo — ED.11
• The zoom lane — ED.12
• Dopesheet keyframes + Easy Ease — ED.13
• Per-segment speed — ED.14
• Crop + aspect reframe — ED.15
• The zoom engine — ED.16
• Auto-zoom from click telemetry — ED.17
• Inspector Style tab — ED.18
• Inspector Cursor tab — ED.19
• Deferred export frame generator — ED.20
• End-to-end edited export — ED.21
• Export progress + cancel — ED.22
• Project persistence — .screenproj — ED.23
• Recordings library + open-in-editor — ED.24

The edit model — ED.1

crates/edit is the editor's spine: a pure, serializable model of an edit with no GPU, media, or UI dependencies — just data and the arithmetic that maps timeline time to source-clip time. Everything the editor does (trim, split, speed, crop, zoom, undo/redo, export) is an operation on, or a render of, this model. Keeping it dependency-light is what makes it exhaustively unit-testable on any machine.

The shape

classDiagram
  class EditProject {
    +ClipRef source
    +Vec~TimelineSegment~ segments
    +Vec~ZoomSegment~ zooms
    +BackgroundConfig background
    +CursorConfig cursor
    +Option~CropRect~ crop
    +AspectRatio aspect
    +u32 project_fps
    +project_duration() Frame
    +source_time(project_frame) Option~Frame~
  }
  class TimelineSegment {
    +Frame source_start
    +Frame source_end
    +f64 timescale
  }
  class ZoomSegment {
    +ZoomId id
    +Frame start
    +Frame end
    +f64 amount
    +ZoomMode mode
    +EditEase ease
  }
  EditProject "1" o-- "many" TimelineSegment
  EditProject "1" o-- "many" ZoomSegment
  EditProject "1" o-- "1" ClipRef

The edited video is the ordered concatenation of the segments. Trimming adjusts a segment's source_start / source_end; splitting replaces one segment with two adjacent ones sharing the cut frame; changing speed sets a segment's timescale. Zoom regions, the framing config, and the crop ride alongside — the renderer applies them per frame.

Project time → source time

Project time is frame-indexed at project_fps (default 30). The core of the model is source_time(project_frame): walk the segment list, accumulating each segment's project length, until you find the one containing the frame, then map the within-segment offset to a source frame using that segment's timescale.

flowchart TD
  A[project_frame] --> B{walk segments\nacc += project_len}
  B -->|frame < acc + len| C[offset = frame - acc]
  C --> D[source_start + offset × timescale]
  D --> E[source frame to decode]
  B -->|past last segment| F[None — end of timeline]

A timescale of 2.0 means 2× speed: a 100-source-frame slice occupies only 50 project frames, and project offset p maps to source frame 2p. Worked example for a three-segment project:

So source_time(375) lands in segment 1 at offset 75 → source frame 300 + 75×2 = 450, and the whole timeline is 750 project frames long.

See the edit rustdoc for the full API — EditProject, TimelineSegment, and ZoomSegment.

Edit operations + undo/redo — ED.2

Every timeline edit is an EditOp applied to the project — a small, validated, invariant-preserving mutation. Undo/redo wraps them in a History.

The operations

classDiagram
  class EditOp {
    <<enum>>
    Split { at }
    Trim { index, edge, to }
    RippleDelete { start, end }
    SetSpeed { index, timescale }
    AddZoom { zoom }
    RemoveZoom { id }
    MoveZoom { id, start, end }
  }

• Split cuts the segment under a project frame into two (a no-op on a boundary). Trim moves a segment's in/out point, clamped so it stays non-empty and inside the source. RippleDelete removes a project range and closes the gap — the surviving pieces simply concatenate. SetSpeed sets a segment's timescale (sanitized to a finite, positive multiplier).
• AddZoom / RemoveZoom / MoveZoom edit the zoom list, which stays sorted by start frame; ids are assigned fresh on insert so move/remove can address a zoom stably as the list changes.

Undo without fragile inverses

History::apply doesn't derive a per-operation inverse (which is error-prone for splits and ripples). Instead it applies the operation to a clone and commits only if the project actually changed:

sequenceDiagram
  participant C as caller
  participant H as History
  participant P as EditProject
  C->>H: apply(op)
  H->>P: next = current.clone()
  H->>P: next.apply(op)
  alt next != current
    H->>H: push current onto undo stack, clear redo
    H->>H: current = next
  else unchanged (no-op / error)
    H->>H: discard the clone, record nothing
  end

The lift-delete variant (delete a range but leave a black gap) needs a timeline "gap" item the segment model doesn't yet have; it is deferred (see _docs/ISSUES.md). Ripple-delete is the primary delete, which is what the timeline UI (ED.11) leads with.

Random-access decode — ED.3

The recorder's decoder (GstreamerPipeStream) streams BGRA frames forward only — perfect for playback, useless for an editor that scrubs to arbitrary frames. EditorVideoStream wraps it with frame-indexed seeking and a bounded decoded-frame cache.

How a seek resolves

flowchart TD
  A["frame(index)"] --> B{clamp to last frame}
  B --> C{in cache?}
  C -->|yes| R[return cached frame]
  C -->|no| D{"pipe missing,\nor already past index?"}
  D -->|yes| E[re-spawn pipe from frame 0]
  D -->|no| F[keep current pipe]
  E --> G[decode forward, caching each frame, until index]
  F --> G
  G --> R

A forward seek keeps pulling from the live pipe; a backward seek (before the pipe's current position) re-spawns from frame 0 and decodes up to the target. Every decoded frame is cached (LRU), so local scrubbing and repeated access are cheap, and export — which walks frames in order — never re-spawns.

Correctness is proven directly: the integration test decodes the fixture both ways and asserts frame(n) is byte-identical to forward-decoding to n; a separate test asserts cached frames don't bump spawn_count, and out-of-range indices clamp to the last frame.

Playback clock — ED.4

The recorder's player is wall-clock paced and 1× only. The editor needs a clock that is the single authority over time — EditorPlayer: seek to an exact frame, step a frame at a time, play at a chosen rate, honour in/out points, and loop.

stateDiagram-v2
  [*] --> Paused
  Paused --> Playing: play()
  Playing --> Paused: pause() / step()
  Playing --> Paused: reached out-point (no loop)
  Playing --> Playing: reached out-point (loop → in)
  Paused --> Paused: seek(frame) / set_rate(r)

Time is measured in project frames at the project fps. tick(dt) advances the playhead by dt × fps × rate; current_frame() is floor(elapsed × fps), clamped to the playable range [in, out). At the end of the range the clock loops back to the in-point or clamps to the last frame and pauses.

Editor surface + Record→Edit handoff — ED.5

?surface=editor routed to a placeholder (<h1>Editor</h1>). ED.5 activates it: the surface now renders the real EditorShell chrome — title bar, toolbar, and body layout — driven by the loaded EditProject, and wires the handoff that loads a finished recording into it.

The handoff

sequenceDiagram
  participant U as User
  participant JS as index.html bridge
  participant Cmd as open_in_editor (Tauri)
  participant UI as EditorSurface (Leptos)
  U->>JS: __screenOpenInEditor(path)
  JS->>Cmd: invoke("open_in_editor", { path })
  Cmd->>Cmd: probe metadata → EditProject::from_recording
  Cmd-->>JS: EditProject (serialized)
  JS->>UI: dispatch "editor-project" CustomEvent
  UI->>UI: deserialize → RwSignal<Option<EditProject>>
  UI->>U: jump to the editor, render EditorShell populated

open_in_editor (a thin Tauri command) probes the recording with gst-discoverer-1.0 and returns a default, untouched EditProject — one full-length real-time segment. The webview bridge re-emits it as an editor-project event; a Leptos listener deserializes it into a context signal that the surface reads.

```admonish important title="No mirror type — app-ui depends on edit" app-ui can't depend on screen-app (Tauri-native), but it can depend on the pure edit crate (serde-only, wasm-clean). So the command's payload deserializes straight into edit::EditProject — no hand-maintained IPC mirror struct to drift out of sync.

The surface maps the project onto the shell view-model (title = file name,
subtitle = `1920×1080 · 30 fps · m:ss`, toolbar enabled); with no clip it
shows the "No clip loaded" empty state. The canvas, timeline, and
inspector slots are filled by the chunks that follow — preview (ED.6),
transport (ED.7), timeline (ED.8), inspector (ED.18).

```admonish note title="Handoff trigger"
ED.5 lands the complete receiving mechanism (command → event → signal →
surface, with an auto-jump to the editor when a project loads). The
user-facing "Open in Editor" button on the recorder's save panel rides
in with the recordings Library (ED.24), which is where browsing and
re-opening past recordings lives.

Editor preview canvas — ED.6

The editor needs to show the frame under the playhead. EditorPreview composes it — and does so through the same compositor the recorder uses, so there's one render path to reason about (and, later, exact preview/export parity).

flowchart LR
  P["EditorPlayer.current_frame()"] --> S["EditorVideoStream.frame(n)"]
  S --> R["EditorPreview.render_frame(bgra)"]
  R --> C["RecordingCompose\n(wisp scene → RenderTexture)"]
  C --> B["composed BGRA"]
  B --> W["winit preview window"]
  B -.same path.-> X["export (ED.20)"]

EditorPreview wraps the proven RecordingCompose but feeds it from the seekable EditorVideoStream at the playhead instead of live capture slots. The recorded clip is already a fully-composited frame (any webcam bubble was baked in at record time), so it's shown full-frame; the scene's camera channel stays idle.

Playback transport — ED.7

Full basic playback: play/pause, frame-step, jump-to-ends, a scrubber, a speed selector, and a MM:SS.ff timecode — wired to the backend playhead.

sequenceDiagram
  participant UI as Transport bar (Leptos)
  participant JS as index.html bridge
  participant Cmd as editor_transport (Tauri)
  participant Sess as EditorSession (EditorPlayer)
  UI->>JS: __screenEditorTransport(action)
  JS->>Cmd: invoke("editor_transport", { action })
  Cmd->>Sess: apply(action)
  Sess-->>Cmd: EditorStatusView
  Cmd-->>JS: status
  JS->>UI: dispatch "editor-status" → RwSignal<EditorStatus>
  UI->>UI: timecode + scrubber re-render (fine-grained)

The clock lives in the backend — an EditorSession wrapping the EditorPlayer from ED.4. The webview is a thin transport that sends one enum-dispatched editor_transport command and renders the returned status.

The timecode formatter (format_timecode) renders MM:SS.ff (frames within the second) and is unit-tested across boundaries; the scrubber and timecode use fine-grained reactivity so only they re-render as the playhead advances — the speed selector and buttons stay put.

Timeline ruler + coordinate system — ED.8

The timeline needs one shared map between project frames and screen pixels — for the ruler, the lanes (ED.9–12), the playhead, and snapping (ED.11). That map is TimelineViewport: a zoom (px_per_frame), a scroll (scroll_frame), and the tick math.

flowchart LR
  F["project frame"] -- "× px_per_frame − scroll" --> PX["pixel x"]
  PX -- "÷ px_per_frame + scroll" --> F
  Z["zoom_at(factor, anchor_px)"] -. keeps anchor frame fixed .-> PX

• frame_to_px / px_to_frame round-trip exactly.
• zoom_at(factor, anchor_px) zooms while holding the frame under the anchor pixel fixed — so zooming centred on the playhead keeps the playhead put (a hard NLE expectation).
• pan_px scrolls, clamped so you can't scroll past either end.
• ruler_ticks emits labeled ticks at a "nice" second interval (1/2/5/10/15/30/60/…s) chosen so labels never crowd — frame-correct at every zoom.

The TimelineRuler component renders a fit-to-width ruler — the full-clip "global progress" view the ticket calls for, decoupled from per-lane zoom — with those tick labels, a reactive playhead bound to the editor status, and click-to-seek (clicked fraction → Seek).

Video track + clip selection — ED.9

The video lane shows the recording as its segments — proportional clip blocks along the timeline — and lets you select one to edit.

segment_spans is the pure layout: each TimelineSegment becomes a start_fraction / width_fraction of the project, so the lane tiles responsively at any width. A 2× segment is half as wide as its source span (it occupies half the project time) — width tracks project length, not source length, which is what keeps the lane in sync with the ruler after a speed change.

The VideoFilmstrip component renders those spans as clip blocks with duration labels; clicking one sets the selected-clip signal (a RwSignal<Option<usize>> in context) that the inspector (ED.18) and edit operations (ED.11) read. Because the spans are derived from the segment list every render, the lane re-flows automatically when a split or trim changes the segments.

Audio waveform lane — ED.10

For decades the picture editor cut blind to sound, then threaded a separate magnetic track onto the flatbed's soundhead so picture and audio ran in sync — and you cut on what you could hear. The waveform is that mag track made visible: you find the breath before a sentence, the click of a button, the silence to trim, all by eye. You cannot splice on a sound you cannot see.

downsample_peaks turns a sea of samples into one min/max pair per horizontal bucket — the peak envelope. Drawing every sample is both impossible (millions of them) and pointless (the screen has a few hundred pixels); the envelope is exactly what the eye reads. The AudioWaveform lane draws those buckets beneath the video track, aligned to the same timeline.

flowchart LR
  S["44.1k samples/s"] -- "min/max per bucket" --> E["~N envelope buckets"]
  E --> BARS["waveform bars under the video lane"]

Splitting, ripple-delete + undo/redo — ED.11

The most physical act in the cutting room: lay the film on the bench, drop the razor on the frame line, and you have two pieces where there was one. Lift one piece out and push the ends together and the cut closes — a ripple. And behind both, the safety net: the trim bin, where every offcut hangs so nothing is ever final. ED.11 brings all three to the timeline — S splits the clip under the playhead, Delete lifts the selected clip and closes the gap, ⌘Z / ⌘⇧Z walk the trim bin.

flowchart LR
  K["S at the playhead"] --> OP["EditOp::Split"]
  DEL["Delete on a selection"] --> OP2["EditOp::RippleDelete"]
  OP --> H["edit::History.apply"]
  OP2 --> H
  H --> SIG["project signal updates"]
  SIG --> STRIP["filmstrip re-flows"]
  H --> DUR["SetDuration → playback clock"]
  UNDO["⌘Z"] --> H2["History.undo / redo"] --> SIG

Each edit is just an EditOp against the (proptest-verified) History from ED.2 — the UI layer is thin. resolve_history reuses the running history when it belongs to the open clip (so the undo stack survives) or starts fresh when a different clip loads; segment_project_range turns the selected clip index into the [start, end) frames a ripple deletes. The result syncs into the reactive project signal the filmstrip already renders. Nothing here touches the negative — a split divides a segment's range, a ripple drops a segment from the list.

The zoom lane — ED.12

The rostrum operator never improvised a push-in. Every camera move was planned on an exposure sheet — ruled paper where each row was a frame and a margin column noted the moves: zoom in, frames 120–180; hold; zoom out. The move list was authored on paper, then executed by the stand. ED.12 is that exposure sheet as a timeline lane: each block is one planned push-in, and the zoom engine is the stand that executes it at preview and export.

flowchart LR
  ADD["+ Zoom at the playhead"] --> OP["EditOp::AddZoom"]
  OP --> H["edit::History.apply"]
  H --> ZL["project.zooms"]
  ZL --> LANE["ZoomLane blocks\n(laid out by fraction)"]
  LANE -->|select / ×| H
  ZL --> ENG["ED.16 engine compiles\neach to a push-in at render"]

The lane authors ZoomSegment values; it never renders pixels. A block is laid out by zoom_spans with the same fraction-of-duration math as the filmstrip, so the zoom lane lines up frame-for-frame with the video and audio tracks above and below it. + Zoom drops a default ~1.5 s, 1.6× region at the playhead through EditOp::AddZoom; clicking a block selects it; its × removes it — all via the proptest-verified History, so every add and remove is undoable from the trim bin.

Dopesheet keyframes + Easy Ease — ED.13

The animator's dope sheet was a frame-by-frame timing chart: which drawing on which frame, how a move accelerates and settles. A zoom is the same idea — a value (scale) keyframed across time and eased between the keys. ED.13 brings that chart to the selected zoom: ZoomDopesheet plots its keyframes and offers the easing presets, with Easy Ease front and centre.

flowchart LR
  SEL["selected zoom (ED.12)"] --> KF["zoom_keyframes\nidentity → full → full → identity"]
  KF --> PLOT["dopesheet markers"]
  EASE["ease preset (Easy Ease …)"] --> OP["EditOp::SetZoomEase"]
  OP --> H["edit::History.apply"]
  H --> Z["zoom.ease"]
  Z --> ENG["ED.16 engine eases between keys"]

zoom_keyframes is the pure, Track-shaped view of the engine's ramp: identity at both edges, full amount across the hold (a triangle when the ramps fill the window), the same ramp_frames the engine uses — so the dopesheet shows exactly what plays. The ease row commits EditOp::SetZoomEase through the shared History (undoable), and the engine eases between the keys with the chosen curve. The default and one-click favourite is Easy Ease (InOutCubic) — accelerate off the wide shot, settle into the detail.

Per-segment speed — ED.14

Speed was a lab trick long before it was a slider. On the optical printer, step-printing exposed each negative frame two or three times to make slow motion; skip-printing dropped frames for fast motion. The Steenbeck let an editor crank faster or slower to find the moment, but the speed that shipped was baked into the print at the lab. ED.14 makes it a value instead: a timescale on a segment, re-derived at preview and export so nothing is baked until you ask for it.

flowchart LR
  PRESET["speed preset (0.5×…4×)"] --> OP["EditOp::SetSpeed"]
  OP --> H["edit::History.apply"]
  H --> TS["segment.timescale"]
  TS --> DUR["project length shrinks / grows\n→ filmstrip re-flows\n→ clock re-syncs (SetDuration)"]
  TS --> PREV["preview re-times for free\nvia source_time (ED.4)"]
  TS --> EXP["export re-times + pitch-corrects\naudio (ED.21)"]

Each TimelineSegment carries a timescale: a 2× segment occupies half its source span in project time, a 0.5× segment twice as much. The ClipInspector shows presets for the selected clip; choosing one runs EditOp::SetSpeed through the shared History (undoable from the trim bin). Because speed is duration-changing, the edit re-flows the filmstrip and re-syncs the playback clock — the same SetDuration path a ripple uses.

Crop + aspect reframe — ED.15

Aspect ratio used to be a piece of metal. A hard matte in the camera or optical printer blacked the frame down to 16:9, or 4:3, or anamorphic 2.35 — the shape was masked into the negative. Releasing a 4:3 negative to a 16:9 screen meant pan-and-scan: an operator chose, frame by frame, which window of the image to keep. ED.15 makes both into values — the matte (aspect) and the chosen window (crop) — stored on the project and re-derived at export, so the same source reframes to a widescreen export or a vertical short without ever recutting it.

flowchart LR
  A["aspect preset (16:9 / 9:16 / 1:1 / 4:3)"] --> OPA["EditOp::SetAspect"]
  C["crop %  (X / Y / W / H)"] --> OPC["EditOp::SetCrop"]
  OPA --> H["edit::History.apply"]
  OPC --> H
  H --> P["project.aspect / project.crop"]
  P --> CANVAS["aspect.canvas_dims → export canvas"]
  P --> FRAME["crop sub-rect → render_framed / videocrop"]

The FramingInspector carries both: aspect-ratio presets (the matte) and four numeric crop fields as percentages (the window), with a reset to full frame. Each runs EditOp::SetAspect / EditOp::SetCrop through the shared History, so reframing is undoable. SetCrop sanitizes the rect to a valid in-frame sub-rect (non-zero extent, inside [0, 1]), and a full-frame crop is stored as no crop so the export can skip the videocrop element entirely.

The zoom engine — ED.16

On an animation stand the camera lived on a column above the artwork, and the operator pushed in by turning a screw drive — a slow, deliberate move from wide to tight, hold on the detail, then ease back out. That move, the rostrum push-in, is the single gesture that reads as "cinematic" in a screen recording: the viewer's eye is walked to exactly the thing that matters. ED.16 is that move in software — and, crucially, it is pure arithmetic, so the same function drives the live preview and the final export. What you scrub is what you ship.

flowchart LR
  Z["ZoomSegment\nstart · end · amount · target · ease"] --> F["zoom_at(seg, frame, ramp)"]
  P["project frame"] --> F
  F --> T["ZoomTransform\nscale · center_x · center_y"]
  T --> R["renderer scales the framed\nscreen about the focal point"]

A zoom is not stored as baked frames — it's a ZoomSegment value, and zoom_at recomputes the transform for any frame on demand. That's the editor's founding rule — never cut the negative — applied to motion: the zoom is a function of the frame, evaluated at preview and again at export, never a destructive bake.

The three-phase profile

Over a zoom's [start, end) window the scale follows an eased ramp-in to full amount, a flat hold, and a symmetric eased ramp-out back to no-zoom. The ramp length is clamped to half the window, so the two ramps can never overlap — a very short zoom degrades to a clean triangle (push-in straight into push-out) rather than fighting itself.

The easing is the segment's EditEase; its default, InOutCubic, is the "Easy Ease" feel — accelerate off the wide shot, decelerate into the detail. EditEase::eval maps a 0..1 ramp fraction to eased progress, and every curve is pinned to f(0)=0, f(1)=1 so the window's edges always meet no-zoom exactly.

From transform to pixels

The ZoomTransform now drives real pixels. EditorPreview::render_framed writes a single crop-then-zoom affine into the screen sprite's transform and composes through the recorder's proven wisp path, so the export generator and the live preview punch in through the same code — preview/export parity by construction. The screen sprite is centre-anchored at scale 2 (it fills NDC [-1, 1]); a ZoomTransform of scale z becomes sprite scale 2 · z with the focal point pinned in place via position += focal_ndc · (1 − z), where focal_ndc = (2·fx − 1, −(2·fy − 1)) — the − on y is wisp's +y-up convention (the decoded top-down frame is flipped bottom-up at upload). Crop composes underneath: the sub-rect is pre-scaled 2/w, 2/h and recentred, then the zoom rides on top.

Auto-zoom from click telemetry — ED.17

In a busy cutting room the assistant editor kept a continuity log — every take, every notable beat, marked against the footage so the editor knew where the action was without re-screening everything. A screen recording keeps its own continuity log for free: every place the user clicked is a place their attention went. ED.17 reads that log and marks up the timeline with proposed push-ins — the assistant's annotations, which the editor (you) then keeps, nudges, or throws away.

flowchart LR
  LOG["click log\n(frame + x,y per click)"] --> CL["cluster by time gap (~1s)"]
  CL --> WIN["per cluster: window =\n[first − lead, last + hold],\ntarget = centroid"]
  WIN --> ZS["ZoomSegment { Manual(x,y), max_zoom }"]
  ZS --> ENG["ED.16 engine compiles\neach to a push-in"]

auto_zoom_segments is pure arithmetic over a ClickEvent list: clicks within ~1 s cluster together; each cluster becomes a zoom that opens ~0.3 s before the first click, holds AutoZoomConfig::hold_time_ms past the last, and targets the cluster's centroid at max_zoom. Sub-half- second blips are dropped and adjacent windows are clamped so they never overlap. The output is an ordinary list of ZoomSegments — the zoom engine compiles them exactly like hand-authored ones, and the zoom lane renders them for editing.

Capturing the telemetry

The generator consumes a click log + a cursor track; the capture that produces them lives in app::cursor_capture (macOS first):

• Cursor position — a CursorPoller samples the global pointer at ~60 Hz via CGEventCreate(NULL) + CGEventGetLocation, which read the current position with no Input-Monitoring permission and no event tap. samples_to_track resamples the timestamped samples onto the project frame grid; normalize_cursor_to_frame maps display points into the [0, 1] CursorSample convention. Both are pure + unit-tested; the poller thread is runtime-only.

Inspector Style tab — ED.18

A finished print was never just tacked to the wall — it was mounted: matted with a border, sometimes float-mounted with a shadow so it stood off the backing board, set against a chosen backdrop. That presentation layer is exactly what turns a raw screen grab into something that looks produced — the recording floated on a gradient, padded, rounded, with a soft shadow. ED.18 is the mount: a Style panel that drives the project's BackgroundConfig.

flowchart LR
  SW["backdrop swatch"] --> OP["EditOp::SetBackground"]
  NUM["padding / radius / shadow"] --> OP
  OP --> H["edit::History.apply"]
  H --> BG["project.background"]
  BG --> R["render: backdrop + padded,\nrounded, shadowed screen"]

The StyleInspector offers backdrop swatches (gradients + flat fills) and numeric padding / corner-radius / shadow fields; each reads the current config, changes one field, and commits a SetBackground through the shared History (undoable). The swatches render the actual backdrop via source_css — the same CSS the live canvas backdrop will use — so what you pick is what you see.

From config to pixels

The Style panel authors the values; the renderer draws them. The framing is two layers, applied once when the config changes (the export sets it at generator construction; a live preview re-applies on edit) via EditorPreview::set_background:

• The backdrop is a full-NDC Graphics rect on the recorder's scene — a linear gradient (the default warm→cool diagonal), a flat color, or (later) a wallpaper. It carries no clip, so it renders in the advanced-dispatch Phase 1, behind everything.
• The screen is the recording sprite given a MaskShape::RoundedRect clip set to the padded window. The clip makes it a dispatched node — it composites over the backdrop in Phase 2, with the rounded-corner SDF cutting its alpha.

The clip lives in fixed output NDC (screen space, not transform-aware), so the rounded window is a stable frame while the zoom punch-in (ED.16) tightens inside it. Padding folds into the same screen transform as a centered shrink — scale *= k, position *= k, with k = 1 − 2·padding/axis — so it composes exactly with crop and zoom (pad ∘ zoom ∘ crop). Because the recorder never calls set_background_* / set_screen_clip, its scene is unchanged: no backdrop node, no screen clip, a plain full-bleed compose.

Inspector Cursor tab — ED.19

In a screen recording the cursor is the only performer on stage — and like any performer it reads better with a little grooming. Scaled up so it's findable, its motion smoothed the way a fluid-head dolly tames handheld jitter, a ripple on each click the way a clapperboard's snap marks the action, and politely off-stage when it isn't doing anything. ED.19 is the cursor's dressing room: a panel that edits one CursorConfig on the project.

flowchart LR
  CTL["size / smoothing / ripples /\nhide-static / auto-zoom"] --> OP["EditOp::SetCursor"]
  OP --> H["edit::History.apply"]
  H --> CUR["project.cursor"]
  CUR --> OVL["cursor overlay at render,\ndriven by the cursor track"]

The CursorInspector exposes size (clamped to a sane 25–400 %), smoothing, and three toggles — click ripples, hide-when-static, and auto-zoom on clicks (the switch that feeds ED.17). Each reads the current config, changes one field, and commits a SetCursor through the shared History. Because CursorConfig is Copy, the op is a plain field assignment — the by-reference apply refactor ED.18 introduced for the non-Copy background config carries it for free.

The overlay at render

The composited cursor is a single wisp Graphics node drawn over the framed screen by EditorPreview::render_framed_with_cursor, driven by the recorded cursor track (EditProject::cursor_track):

• a scaled, dark-outlined white arrow pointer (a single convex draw_polygon quad — tip, left edge, tail point, right barb — sized by size_pct),
• expanding, fading click ripples (draw_ellipse discs) at each recent click, radius growing + alpha decaying across a ~0.4 s window (ripples_at),
• smoothing as a pure EMA over the track (cursor_at),
• hide-when-static: a parked pointer fades off-stage (cursor_is_static), but a live click ripple keeps it visible — a click is an action worth showing even when the cursor hasn't moved.

Deferred export frame generator — ED.20

The optical printer was the lab's export stage: a camera and a projector locked together, re-photographing the cut negative one frame at a time onto fresh stock, honoring every splice and speed change as it advanced. ED.20 is that printer in software. Given an EditProject, the ExportFrameGenerator walks the project frames 0..project_duration and, for each, prints the right source frame onto the output.

flowchart LR
  F["project frame f"] --> ST["source_time(f)\n(trim · split · speed)"]
  ST --> DEC["EditorVideoStream.frame\n(seek + decode)"]
  DEC --> COMP["EditorPreview.render_frame\n(same compose as preview)"]
  COMP --> OUT["ExportFrame { bgra, pts, source_frame }"]
  OUT --> ENC["encoder (ED.21)"]

The key is that the timeline edits are already baked into the frame selection: source_time maps each project frame to its source frame, so a trimmed clip starts later, a split is invisible (two segments, one continuous source walk), and a 2× segment advances the source twice as fast. The generator composes through the same EditorPreview path the live preview uses, so the exported file matches the cut you scrubbed. Each ExportFrame carries a PTS computed with the same formula the live recorder's encoder feed uses, so timestamps line up when ED.21 hands the stream to the encoder.

End-to-end edited export — ED.21

The lab's last step was the answer print: the first complete, projectable reel struck from the cut negative — every splice, speed change, and trim finally baked into something you could screen. ED.21 strikes that print. export_edited_project drives the frame generator into the encoder and finalizes a real .mp4.

flowchart LR
  GEN["ExportFrameGenerator\n(retimed BGRA + pts)"] --> PUSH["LiveGstreamerEncoder\n.push_video_frame"]
  PUSH --> FIN["finalize → moov"]
  FIN --> MP4[".mp4"]
  MP4 -.verify.-> DEC["decode back:\ndims + retimed length"]

The encoder is the live recorder's LiveGstreamerEncoder, reused unchanged — the editor's export and the recorder's capture write through the exact same vtenc → h264parse → mp4mux pipeline, so there's one encode path to trust. The export loop is synchronous and polls a cancel flag plus calls an on_progress(done, total) callback once per frame — the hooks the export UI (ED.22) drives. The gst-guarded end-to-end test exports a 2×-speed project and then decodes the result back with EditorVideoStream, asserting the output is a valid container at the source dimensions whose length is the retimed (halved) duration — proof the edit reached the file, verified with our own decoder rather than a brittle byte golden.

The audio rides the same timeline

The generator bakes the visual edits in (trim/split/speed via source_time, the zoom punch-in (ED.16), crop/aspect (ED.15), and background framing (ED.18)). Audio gets the same treatment, with the project's "GStreamer owns the intake; Rust owns the edit arithmetic" split:

flowchart LR
  SRC["source .mp4"] --> DEC["decode_source_audio_f32\n(one gst pass → raw F32LE)"]
  DEC --> RT["retime_audio\n(pure: per-segment trim + speed,\nlinear-interp resample, concat)"]
  RT --> PUSH["encoder.push_audio_chunk"]
  PUSH --> REMUX["finalize remux (build_remux_args)"]
  REMUX --> MP4[".mp4 — video + retimed audio"]

One gst-launch pass decodes the whole source audio to raw interleaved F32LE; retime_audio then slices it in pure Rust — for each segment, the source sample-frame range resampled to that segment's project duration (a 2× segment emits half the sample-frames; linear interpolation keeps it click-free) — and the result feeds the encoder's audio scratch, so the recorder's existing finalize remux muxes it onto the retimed video. The end-to-end test builds an audio-bearing source and proves the export carries a retimed audio track.

Export progress + cancel UI — ED.22

Striking a print took minutes, and the lab tech didn't just walk away — they watched the footage counter climb and could halt a bad run before it wasted the whole reel. ED.22 is that counter and that stop button: an ExportBar that turns ED.21's export into a visible, interruptible job.

sequenceDiagram
  participant UI as ExportBar (Leptos)
  participant CMD as editor_export (Tauri)
  participant GEN as export_edited_project
  UI->>CMD: invoke(project, "mp4")
  CMD->>GEN: spawn_blocking(...)
  loop each frame
    GEN-->>CMD: on_progress(done, total)
    CMD-->>UI: emit editor-export-progress
  end
  UI->>CMD: editor_export_cancel (Cancel)
  CMD-->>GEN: cancel flag set
  GEN-->>UI: editor-export-done { path } / -error

The backend editor_export command runs the export on the blocking pool (so the webview stays live), deriving the output path beside the recordings folder and emitting editor-export-progress throttled to ~100 events over the run. A shared AtomicBool (raised by editor_export_cancel) is the stop button — the export loop polls it each frame. The progress / done / error events ride the same __TAURI__.event → CustomEvent → signal bridge the rest of the editor uses, landing in an ExportUiState the bar renders: Export when idle, a progress bar + Cancel while running, and the output path (or error) when done.

Project persistence — .screenproj — ED.23

When the CMX 600 turned editing into a decision list, that list became the thing you filed in the can beside the negative: reopen the can, feed the list back, and you had your cut — no frame of original ever copied. ED.23 is that can. Because an EditProject is only value types — segment ranges, zoom windows, framing config, a pointer at the source — saving it is just serializing the decision list to JSON, and reopening it is parsing the JSON back.

flowchart LR
  P["EditProject\n(the decision list)"] -->|to_screenproj| J[".screenproj (JSON)"]
  J -->|from_screenproj| P2["EditProject (identical)"]
  P -.points at.-> SRC["source .mp4 (untouched)"]
  P2 -.points at.-> SRC

edit::persist is the whole format: to_screenproj / from_screenproj over serde_json, pretty-printed so the file is human-readable and diff-friendly. The round-trip is lossless and pure — proven without touching the filesystem — so save↔reload is identical by construction. On the shell side, editor_save_project writes <recording>.screenproj beside the source and editor_load_project reads it back; the Save button in the editor lives next to Export.

```admonish note title="serde_json joins edit — and it stays wasm-clean" edit was deliberately serde-only; persistence promotes serde_json from a dev-dependency to a real one. It's wasm-safe (no filesystem of its own — the file I/O lives in the native app commands), so edit remains GPU-free and wasm-clean, and the round-trip test runs in the gate with no GPU or disk. The file carries a SCHEMA_VERSION so a future format change migrates rather than mis-parses.

Recordings library + open-in-editor — ED.24

Every cutting room had a shelf of cans — the finished reels, racked and labelled, ready to pull down onto the bench. ED.24 is that shelf: the recordings in your output folder, shown as tiles, each one a click away from the editor.

flowchart LR
  DIR["output folder"] --> CMD["list_recordings\n(scan → entries)"]
  CMD --> GRID["RecordingsLibrary grid"]
  GRID -->|click| OPEN["open_in_editor (ED.5)"]
  OPEN --> ED["Editor (nav flips)"]

The backend list_recordings command scans the recordings folder; the pure, tested recording_entries turns that listing into entries — every .mp4, newest first, flagged if a saved .screenproj (ED.23) sits beside it. The RecordingsLibrary grid renders them through the same __TAURI__.event → CustomEvent → signal bridge the rest of the editor uses, and a click reuses the ED.5 open_in_editor handoff before flipping the nav rail to the editor — closing the Record → Edit → Export loop the whole milestone set out to build.

M0 — wisp renderer

21 chunks delivered the renderer: Stage, transforms, sprites, graphics, text, filters, mesh. The visual evidence lives in its own book:

→ Wisp stories — every chunk's renderable output, with the SDF / batching / mask / text-pipeline deep dives.

→ Wisp overview — the public API tour: Container, Sprite, Graphics, Text, RenderTexture, the filter chain, the mask system.

For the chunk-level acceptance criteria and "Done when:" history, see _docs/milestone-0-renderer.md in the repo (kept as engineering record, not published to the book).

M1 — Tauri drop-zone + video player

11 chunks consolidated. Tauri 2 shell with vanilla HTML/CSS/JS frontend, MP4 drop-zone with convertFileSrc → <video> playback, view switching.

Full write-up in _docs/milestone-1-drop-zone-player.md.

The Leptos editor UI (Button / Card / DopeSheet) lives separately under ui-storybook and is not yet wired into the Tauri shell. Migration from vanilla JS to Leptos is queued as a future chunk.

API reference

The full rustdoc is generated by cargo doc and mounted under api/.

• wisp — the renderer.
• wisp-storybook — wgpu story gallery (lib + bin + headless exporter).
• ui-storybook — Leptos UI gallery (lib + headless exporter).
• screen-app — the Tauri shell.

Regenerate with just docs. For broken-link enforcement, just docs-strict.