Mathias Petersen Let’s talk ↗
MATHIAS//PETERSEN
← Project index

Typed dataflow runtime / 2024 to present

Daedalus

Why it matters Daedalus separates what a user authored from what is safe and available to execute.

I built Daedalus so a product can expose editable processing graphs without pushing validation, conversions, memory movement, queue policy, or foreign-language lifecycle into UI code.

Explore repository
StatusOpen-source library
Role in the architecturePipeline description, planning + execution
My roleFounder & Lead Engineer
DisciplinesSystems + tools / Runtime + media
01 / The work
Challenge

An editable graph has to remain portable and understandable while its runtime handles incompatible payloads, automatic adapters, bounded streaming, CPU/GPU movement, plugin packaging, and long-lived foreign-language workers.

If those concerns are resolved inside handlers, mistakes appear late and every integration invents a different lifecycle.

Approach

I made the graph an intermediate representation, split registration and validation from planning, moved payload identity and residency into transport, and gave execution explicit queue and telemetry contracts.

A feature-gated facade and narrow foreign-function interface (FFI) let hosts take only the capability surface they need.

02 / Architecture model Plan before execution

A graph becomes a checked runtime plan.

A visual editor saves a portable node graph. Before live data starts, Daedalus checks that the host provides every node and conversion, chooses queue behavior and CPU or GPU placement, and produces the explicit plan workers execute.

InputAuthoring input

Versioned graph document

01Registry

Nodes · adapters · serializers · capabilities

02Planner

Validate types · resolve conversions · place work

03Transport

Identity · ownership · CPU/GPU residency

04Execution

Bounded CPU workers · optional wgpu backend

System outputRuntime output

Results + structured telemetry

Host boundaryOne model, several hosts
  • Rust facade
  • Persistent language workers
  • Python · Node · Java · C++ SDKs
03 / Implementation

The decisions beneath the summary.

01

Graph as an intermediate representation

Keep the user-authored pipeline portable, inspectable, and migratable before it becomes running workers.

Node declarations, typed ports, payload descriptors, edges, parameters, and graph metadata are data. A UI can save or edit that representation without constructing plugins, while a host can register only the capabilities it permits. Validation can return structural and type evidence against the exact document an operator authored.

  • Portable document: values, type descriptors, graph identity, and version metadata can be stored without loading a plugin
  • Host allowlist: the registry states which nodes, adapters, serializers, devices, and plugin packages are actually available
  • Plugin developer path: typed port handles, derive macros, package schemas, and quickstarts reduce registration boilerplate
02

Validation and lowering

Resolve conversions, placement, queue semantics, and unsupported options before the hot path starts.

The planner checks connections against registered capabilities, finds compatible adapter paths, lowers the graph into explicit runtime inputs, and chooses CPU or GPU handoffs. Bounded edge policies make continuity, newest-value behavior, or backpressure an authored decision rather than an accidental queue implementation.

  • Preflight failures: missing nodes, incompatible ports, unresolved conversions, and invalid scheduling requirements stop before workers receive data
  • Honest contracts: unsupported timeout, queue-depth, and restart options are rejected instead of appearing to work
  • Testable placement: a deterministic GPU mock checks planning and residency decisions without pretending generic CI validates graphics performance
03

Payload identity and residency

Put ownership and CPU/GPU movement in transport once, not in every node implementation.

A payload carries stable identity, layout, access mode, lifetime, ownership, and residency information. The planner can account for handoffs, and shared payload state can memoize a GPU download when one result fans out to several CPU consumers. Telemetry records handles, byte movement, adapter behavior, queue pressure, and node timing against the same lifecycle.

  • One ownership model: transport owns payload identity, lifetime, CPU or GPU residency, fan-out, and conversion primitives
  • Optional acceleration: mock, no-op, and wgpu backends sit behind features so graph-only consumers do not inherit a GPU stack
  • Operational evidence: telemetry exposes copies, byte movement, queue pressure, and resource transitions to the host and UI
04

Host and language boundaries

Expose one execution model through narrow, persistent host contracts.

The Rust facade takes an application from registry through planner into runtime without forcing optional GPU or foreign-function-interface (FFI) dependencies on graph-only consumers. Persistent workers carry the same plugin model into Python, Node, Java, and C/C++, while bounded stderr draining, lifecycle events, and payload handles prevent each SDK from inventing its own execution semantics.

  • Dependency control: a graph-only host can use the facade without installing GPU or language-worker dependencies
  • Measurement context: repository benchmarks separate direct warm calls, pooled worker dispatch, and cross-process handle lookup so unlike costs are not compared
  • Claim boundary: handle benchmarks measure metadata dispatch and do not claim that the referenced multi-megabyte payload was copied
04 / Tradeoffs + limits

What this work does not claim.

01

Persistent FFI workers do not yet promise automatic restart or cancellable I/O for every configuration.

02

GPU support is feature-gated and still requires target-hardware validation.

03

A typed and validated graph prevents structural mistakes; it cannot prove that a node’s algorithm is semantically correct.

05 / Engineering review

Architecture, evidence, and limits.

Explore repository ↗

Architecture

  1. 01
    Declarative graph as an intermediate representation

    a UI can save, inspect, migrate, and validate a pipeline without starting its workers

  2. 02
    Planning separated from execution

    type errors, missing adapters, and scheduling decisions are resolved before the hot path begins

  3. 03
    Memory residency in payload transport

    CPU and GPU movement plus fan-out reuse follow one policy instead of being reimplemented by each node

  4. 04
    Feature-gated facade and narrow FFI

    applications can use the graph core without inheriting wgpu, every plugin, or four language SDKs

Validation

  1. 01

    Planner fixtures cover incompatible ports, missing adapters, bounded-edge policies, and unsupported runtime promises before execution begins

  2. 02

    A deterministic mock GPU backend exercises placement and residency logic without claiming that generic CI validates target graphics hardware

  3. 03

    Runnable examples exercise bounded streaming, adapter insertion, telemetry, typed payload handles, GPU nodes, and persistent language workers through the public facade

06 / Project consequence

Why it matters in practice.

  • 01Gives a visual editor a portable graph document it can save, migrate, and validate without loading native or foreign-language workers
  • 02Moves type conversion, queue policy, CPU or GPU placement, and unsupported promises into planning so bad pipelines fail before live data starts
  • 03Uses one payload and telemetry contract across Rust, Python, Node, Java, and C++ workers, making copies and lifecycle transitions visible to the host
My ownership
  • Graph and type model
  • Planner surface
  • Plugin boundaries
  • Telemetry and SDK architecture
Built with
  • Rust
  • Dataflow graphs
  • Foreign-language plugins
  • Telemetry
  • Runtime planning