Mathias Petersen Let’s talk ↗
MATHIAS//PETERSEN
← Project index

Observable media stack / 2024 to present

Styx

Why it matters Styx makes a camera frame's ownership, physical layout, copies, queue pressure, and teardown visible instead of hiding them behind a convenient image object.

I separated Styx from HeliOS so camera discovery, capture ownership, queue behavior, media transforms, recording, and lifecycle events could form one reusable contract.

Explore repository
StatusOpen-source library
Role in the architectureCamera I/O + frame ownership
My roleFounder & Lead Engineer
DisciplinesRobotics + embedded / Systems + tools / Runtime + media
01 / The work
Challenge

Camera APIs disagree on discovery, ownership, plane layout, blocking behavior, disconnects, and teardown.

A convenient image abstraction can hide megabytes of copying, while an unbounded consumer silently converts load into memory growth and stale frames.

Approach

I made leases, layouts, copy materialization, bounded queues, freshness, stop handles, service events, and health reports part of the public model.

Backend crates own physical APIs; opt-in codec and graph crates add capability; task-oriented builders compose the common capture-to-preview or recording lifecycle.

02 / Architecture model Frame ownership path

The bytes carry their history with them.

Camera backends expose different memory and shutdown rules. FrameLease keeps ownership and physical layout visible through queues, transforms, graph execution, and recording, so a host can identify real pixel copies and stale-frame pressure.

InputPhysical input

Camera · network · file · simulation

01Backend

Negotiate format, layout, and driver buffers

02FrameLease

Pooled or externally backed frame ownership

03Queue policy

Bound capacity · freshness · dropped-frame accounting

04Media stages

Decode · transform · encode · observe

System outputConsumer

Graph · recorder · stream · algorithm

Optional capabilitiesPay only for selected paths
  • V4L2 / libcamera
  • FFmpeg / JPEG
  • Daedalus graph
  • Recording / replay
03 / Implementation

The decisions beneath the summary.

01

One camera lifecycle

Give discovery, capture, reconnect, teardown, processing, preview, and recording one observable contract.

Linux camera APIs such as Video4Linux2 (V4L2) and libcamera, plus network cameras, files, and virtual sources, have different setup and failure behavior. An application should not rebuild the surrounding lifecycle for each. Styx separates backends from task-oriented builders and emits common device, frame, health, memory, and service events through capture-to-consumer workflows.

  • Hotplug behavior: arrival and removal events replace application-level loops that blindly retry a stale device path
  • Task-level API: MediaPipelineBuilder composes capture, optional decode and transforms, graph handoff, preview, and recording
  • Lifecycle contract: stop handles and service events make shutdown, disconnect, and reconnect visible to the host
02

Observable frame ownership

Call a path zero-copy only when the backing buffer and layout prove it.

FrameLease can wrap pooled memory or externally owned planes such as a V4L2 mmap buffer. Format, stride, planes, ownership, and access remain attached to the frame. Unsafe or unrepresentable external layouts deliberately materialize into shared backing, allowing metrics to distinguish a wrapper from a pixel copy.

  • Physical evidence: negotiated dimensions, pixel format, row stride, planes, and buffer size remain attached to a V4L2 frame
  • Narrow dependency path: an algorithm can use FrameLease without installing camera, codec, or service backends
  • Measurement boundary: benchmarks label wrapper construction separately from camera dequeue and full pipeline latency
03

Backpressure as product behavior

Bound memory and state what should happen when a consumer loses pace.

A camera can keep producing while a graph, encoder, recorder, or network consumer slows down. Styx exposes queue capacity, receive modes, freshness policy, dropped-frame accounting, and retained external-buffer pressure so a host can choose complete sequences or newest-frame latency explicitly.

  • Execution fit: blocking camera APIs stay on synchronous workers instead of being disguised as asynchronous device I/O
  • Service integration: asynchronous helpers coordinate receive and control without changing the physical capture model
  • Operator evidence: health and memory reports expose queue pressure, median and tail stage time, copies, bytes moved, and retained buffers
04

Capability-shaped integration

Keep physical backends, codecs, graph execution, and recording opt-in behind the same facade.

Crates and features separate V4L2, libcamera, FFmpeg, JPEG implementations, network cameras, files, simulation, Daedalus handoff, and recording. A small consumer can depend only on the frame transport, while HeliOS can build the full lifecycle without making every backend a mandatory transitive dependency.

  • Replay memory stays bounded: decoded file frames use a capacity-limited cache that a host can disable
  • Hardware test rows record frame rate, CPU load, copies, bytes moved, and median and tail timing rather than reporting only success
  • Benchmark discipline: lease and transform figures state the synthetic frame, operation, and measured boundary instead of implying camera-to-result performance
04 / Tradeoffs + limits

What this work does not claim.

01

The external-frame construction benchmark measures wrapper creation, not camera dequeue or full pipeline latency.

02

Representative camera and driver rows remain manual release validation because generic CI has no physical devices.

03

Codec performance differs sharply by format and backend; one fast packed transform does not imply every media stage is fast.

05 / Engineering review

Architecture, evidence, and limits.

Explore repository ↗

Architecture

  1. 01
    Frame ownership in the type model

    a lease can represent pooled memory or a borrowed driver buffer and makes the copy boundary observable

  2. 02
    Sync-first core around synchronous camera APIs

    blocking device work stays explicit, while async adapters coordinate receive and control

  3. 03
    Backends isolated from the facade

    V4L2, libcamera, FFmpeg, network, file, and simulation dependencies remain opt-in

  4. 04
    Bounded queues with explicit policy

    latency, dropped frames, and slow consumers become host decisions instead of accidental memory growth

Validation

  1. 01
    Microbenchmark scope

    wrapping an existing synthetic 4K YUYV buffer took about 21 ns versus 1.261 ms to duplicate its pixels; this isolates copy-avoidance overhead and is not camera or pipeline latency

  2. 02

    External-buffer tests cover representable and unsafe V4L2 plane layouts, verifying that the library borrows memory only when ownership and layout permit it and records when it must copy

  3. 03

    Queue and service examples exercise newest-frame, complete-sequence, stop, disconnect, and recording behavior through the same public lifecycle

06 / Project consequence

Why it matters in practice.

  • 01Lets a product swap physical, network, file, and virtual sources without rewriting discovery, reconnect, shutdown, health, and recording behavior
  • 02Makes the copy boundary inspectable by carrying driver or pool ownership and plane layout with each frame instead of assuming every path is zero-copy
  • 03Forces the host to choose between complete history and the freshest frame, preventing a slow consumer from silently becoming unbounded memory growth and stale latency
My ownership
  • Capture abstractions
  • Transform pipeline
  • Recording workflow
  • Runtime integration
Built with
  • Rust
  • Media
  • Zero-copy
  • Capture
  • Services