Mimir

Realtime cameras, microphones, chirplets, and field evidence organized into one coherent stream machine.

Viable Stream App

5 min read

Viable Stream App

Objective

Build the smallest live Mimir application that can ingest multiple local or networked video and audio feeds, hold a short synchronized window in memory, and emit coherent OBS-facing program outputs while Fensalir provides the operator UI.

The default reservoir horizon is five seconds. That delay is not decoration. It is the budget that lets late network packets, independent capture clocks, audio/video alignment, and debug controls converge before OBS sees the result. Mimir is explicitly trading latency and memory for throughput and extraction quality: the app may spend that buffered window lining streams up and crunching audio/video data to recover as much of the volumetric audio field and volumetric sensor field as possible. Do not optimize the window away unless a replacement invariant preserves that alignment and field-reconstruction budget.

Current Mechanism

Today the repo has two partial machines:

  • FFmpeg/SRT bridge scripts can capture sender video/audio and expose stable OBS Media Source endpoints.
  • The native LocalcastRuntime reservoir ABI can hold typed live sample handles, and Fensalir can consume native render packets for GPU fusion.

Those parts are useful, but they are not yet the viable app. The viable app is the join: ingest workers append live sample handles into one Fensalir-hosted runtime window, Fensalir renders video from that window, Faust/native audio builds synchronized stems from the same window, and OBS receives only the program outputs.

flowchart TD
    A["local camera workers"] --> R["Fensalir-hosted 5s LocalcastRuntime"]
    B["network video ingest workers"] --> R
    C["local mic/loopback workers"] --> R
    D["network audio ingest workers"] --> R
    R --> U["Fensalir debug/settings/output UI"]
    R --> V["Fensalir GPU video composition"]
    R --> F["Faust/native audio alignment + stems"]
    V --> O["OBS video output"]
    F --> P["OBS audio stem outputs"]

Invariants

  • One in-memory reservoir owns the live timing edge and expiry.
  • The five-second rolling window is a deliberate compute budget, not accidental latency. It exists so capture, sync, GPU fusion, and audio field extraction can choose coherence over immediacy.
  • Fensalir owns the running app, UI, debug state, settings changes, output management, GPU fusion/rendering, and Spout/video publication.
  • Faust/native DSP owns audio alignment, voice separation, suppression, volumetric sound-field extraction, stems, and spatial bed generation.
  • OBS receives synchronized program surfaces. Raw unsynchronized feeds may be diagnostic inputs, not broadcast truth.
  • Networked feeds are producers into the same reservoir. They do not own clocks, private buffers, or a parallel synchronization model.
  • Missing feeds become explicit absence or silent/blank placeholders inside the program model. Stale samples do not get stretched into looking current.
  • Calibration and diagnostics do not run the live hot path.

First Viable Cut

The first cut should be deliberately boring:

  1. Build Mimir as its own C# solution in this repo.
  2. Reference Fensalir as the windowing/rendering library and D3D12 bridge; do not copy engine host code into Mimir.
  3. Start a Fensalir-hosted LocalCastRuntime app with a five-second native runtime.
  4. Create one producer per configured feed using native source-id hashing.
  5. Ingest video feeds as timed camera-frame or render-packet handles.
  6. Ingest audio feeds as timed LocalcastAudioBlockDescriptor handles. Starfire’s Focusrite USB ASIO path is the current local target for mic plus loopback ingest.
  7. Expose Fensalir UI readouts for reservoir edge, window start, per-kind counts, feed health, delay estimate, output status, and current settings.
  8. Render one synchronized program video output from the reservoir.
  9. Emit separately controllable OBS audio stems from the same reservoir window.
  10. Keep the existing FFmpeg/SRT scripts as edge utilities for network capture and OBS compatibility, not as the stream authority.

Repo Shape

Mimir.slnx owns the Mimir app surface:

  • src/Mimir.App launches Fensalir and points it at the Mimir runtime assembly.
  • src/Mimir.Runtime implements the Fensalir client runtime factory. Mimir owns its scene state; Fensalir is the host/window/D3D12 bridge, not an embedded demo runtime.

Synchronization Runtime

Mimir.Runtime now owns the first synchronization spine:

  • MimirRuntime is the Fensalir client runtime. It starts from an empty Mimir scene and adds a Mimir-owned synchronization hub.
  • MimirSynchronizationHub owns stream source polling and appends samples into bounded buffers.
  • MimirRollingStreamBuffer owns one stream’s short history and evicts samples outside the configured window.
  • IMimirStreamSource is the adapter seam for local devices and network feeds.
  • MimirNativeIngestStreamSource is the low-overhead push seam for native camera/audio adapters that already own buffers or handles.
  • MimirVideoFrameDescriptor carries video shape, pixel format, device timestamp, stride, and optional native/GPU handle metadata. Leap stereo IR should enter here through a native driver path.
  • IMimirVideoCaptureDriver and MimirVideoCaptureDriverSource are the driver-facing seam for polling/pushing frames from LeapUVC/libusb, LeapC image APIs, Media Foundation, DirectShow, or a future GPU shared-texture capture.
  • MimirProcessStreamSource is a compatibility adapter for network or diagnostic command sources. It is not the six-camera local capture foundation.

The default buffer duration is five seconds. It can be overridden with MIMIR_SYNC_BUFFER_SECONDS or MIMIR_RESERVOIR_SECONDS.

Configured stream ids can be declared with comma- or semicolon-separated environment variables:

  • MIMIR_LOCAL_VIDEO_STREAMS
  • MIMIR_LOCAL_AUDIO_STREAMS
  • MIMIR_NETWORK_VIDEO_STREAMS
  • MIMIR_NETWORK_AUDIO_STREAMS

These create buffers immediately. Capture adapters still need to register IMimirStreamSource implementations to feed live samples into those buffers.

For local cameras, prefer native adapters that push sample handles directly through MimirNativeIngestStreamSource. Process-backed FFmpeg ingest is allowed for remote SRT/diagnostic edges, but local six-camera ingest should stay close to the driver, GPU, and capture APIs.

Leap should be treated as the timing-camera candidate only when the native adapter preserves device timestamps or captures a monotonic timestamp immediately on frame receipt.

Cut Line

Do not add a new registry, bridge daemon, JSON live store, OBS plugin, or per-feed buffer manager to make this happen. If the app needs one of those, first move the missing authority into LocalcastRuntime, Fensalir, or Faust/native DSP.

The app exists to make the live system coherent, not to preserve every deadline-era surface. Deadline fossils may be used as diagnostic witnesses. They do not get promoted because they still twitch.

Backlinks