Mimir

Realtime cameras, microphones, chirplets, and field evidence organized into one coherent stream machine.

Viable Stream App

6 min read

Viable Stream App

Objective

Build the smallest live Mimir application that can ingest multiple local or networked video and audio feeds, hold a short synchronized window in memory, and emit coherent Mimir program outputs while Eve provides operator surfaces and Yggdrasil publishes the site feed.

The default reservoir horizon is five seconds. That delay is not decoration. It is the budget that lets late network packets, independent capture clocks, audio/video alignment, and debug controls converge before Mimir emits the program result. Mimir is explicitly trading latency and memory for throughput and extraction quality: the app may spend that buffered window lining streams up and crunching audio/video data to recover as much of the volumetric audio field and volumetric sensor field as possible. Do not optimize the window away unless a replacement invariant preserves that alignment and field-reconstruction budget.

Current Mechanism

Today the repo has two partial machines:

  • FFmpeg/SRT bridge scripts can capture sender video/audio and expose stable compatibility endpoints.
  • The native LocalcastRuntime reservoir ABI can hold typed live sample handles, and Fensalir can consume native render packets for GPU fusion.

Those parts are useful, but they are not yet the viable app. The viable app is the join: ingest workers append live sample handles into one Fensalir-hosted runtime window, Fensalir renders video from that window, Faust/native audio builds synchronized stems from the same window, Mimir commits the scene graph, Eve lowers the operator surface, and Yggdrasil publishes the program output.

flowchart TD
    A["local camera workers"] --> R["Fensalir-hosted 5s LocalcastRuntime"]
    B["network video ingest workers"] --> R
    C["local mic/loopback workers"] --> R
    D["network audio ingest workers"] --> R
    R --> U["Fensalir debug/settings/output UI"]
    R --> S["Mimir program scene graph"]
    S --> U
    S --> V["Fensalir GPU video composition"]
    R --> F["Faust/native audio alignment + stems"]
    S --> E["Eve GUI/TUI operator surfaces"]
    V --> Y["Yggdrasil site publisher"]
    F --> Y
    V -. "compatibility" .-> O["OBS adapter"]
    F -. "compatibility" .-> O

Invariants

  • One in-memory reservoir owns the live timing edge and expiry.
  • The five-second rolling window is a deliberate compute budget, not accidental latency. It exists so capture, sync, GPU fusion, and audio field extraction can choose coherence over immediacy.
  • Mimir owns the program scene graph, source subscription policy, preview/control state, stats, and publication intent.
  • Fensalir owns the running app host, UI lowering, debug state, GPU fusion/rendering, and local program texture output.
  • Faust/native DSP owns audio alignment, voice separation, suppression, volumetric sound-field extraction, stems, and spatial bed generation.
  • Eve lowerers render controls and previews without owning scene truth.
  • The Yggdrasil publisher consumes synchronized program surfaces without owning a second composition.
  • OBS compatibility adapters may receive synchronized program surfaces. Raw unsynchronized feeds may be diagnostic inputs, not broadcast truth.
  • Networked feeds are producers into the same reservoir. They do not own clocks, private buffers, or a parallel synchronization model.
  • Missing feeds become explicit absence or silent/blank placeholders inside the program model. Stale samples do not get stretched into looking current.
  • Calibration and diagnostics do not run the live hot path.

First Viable Cut

The first cut should be deliberately boring:

  1. Build Mimir as its own C# solution in this repo.
  2. Reference Fensalir as the windowing/rendering library and D3D12 bridge; do not copy engine host code into Mimir.
  3. Start a Fensalir-hosted LocalCastRuntime app with a five-second native runtime.
  4. Create one producer per configured feed using native source-id hashing.
  5. Ingest video feeds as timed camera-frame or render-packet handles.
  6. Ingest audio feeds as timed LocalcastAudioBlockDescriptor handles. Starfire’s Focusrite USB ASIO path is the current local target for mic plus loopback ingest.
  7. Expose Fensalir UI readouts for reservoir edge, window start, per-kind counts, feed health, delay estimate, output status, and current settings.
  8. Render one synchronized Mimir program video output from the reservoir.
  9. Emit separately controllable program audio stems from the same reservoir window.
  10. Publish Eve GUI/TUI operator surfaces for preview, source selection, scene edits, stats, and output state.
  11. Add the Yggdrasil-facing publisher daemon that consumes Mimir program output without owning composition.
  12. Keep the existing FFmpeg/SRT scripts as edge utilities for network capture and OBS compatibility, not as the stream authority.

Repo Shape

Mimir.slnx owns the Mimir app surface:

  • src/Mimir.App launches Fensalir and points it at the Mimir runtime assembly.
  • src/Mimir.Runtime implements the Fensalir client runtime factory. Mimir owns its scene state; Fensalir is the host/window/D3D12 bridge, not an embedded demo runtime.

Synchronization Runtime

Mimir.Runtime now owns the first synchronization spine:

  • MimirRuntime is the Fensalir client runtime. It starts from an empty Mimir scene and adds a Mimir-owned synchronization hub.
  • MimirSynchronizationHub owns stream source polling and appends samples into bounded buffers.
  • MimirRollingStreamBuffer owns one stream’s short history and evicts samples outside the configured window.
  • IMimirStreamSource is the adapter seam for local devices and network feeds.
  • MimirNativeIngestStreamSource is the low-overhead push seam for native camera/audio adapters that already own buffers or handles.
  • MimirVideoFrameDescriptor carries video shape, pixel format, device timestamp, stride, and optional native/GPU handle metadata. Leap stereo IR should enter here through a native driver path.
  • IMimirVideoCaptureDriver and MimirVideoCaptureDriverSource are the driver-facing seam for polling/pushing frames from LeapUVC/libusb, LeapC image APIs, Media Foundation, DirectShow, or a future GPU shared-texture capture.
  • MimirProcessStreamSource is a compatibility adapter for network or diagnostic command sources. It is not the six-camera local capture foundation.

The default buffer duration is five seconds. It can be overridden with MIMIR_SYNC_BUFFER_SECONDS or MIMIR_RESERVOIR_SECONDS.

Configured stream ids can be declared with comma- or semicolon-separated environment variables:

  • MIMIR_LOCAL_VIDEO_STREAMS
  • MIMIR_LOCAL_AUDIO_STREAMS
  • MIMIR_NETWORK_VIDEO_STREAMS
  • MIMIR_NETWORK_AUDIO_STREAMS

These create buffers immediately. Capture adapters still need to register IMimirStreamSource implementations to feed live samples into those buffers.

For local cameras, prefer native adapters that push sample handles directly through MimirNativeIngestStreamSource. Process-backed FFmpeg ingest is allowed for remote SRT/diagnostic edges, but local six-camera ingest should stay close to the driver, GPU, and capture APIs.

Leap should be treated as the timing-camera candidate only when the native adapter preserves device timestamps or captures a monotonic timestamp immediately on frame receipt.

Cut Line

Do not add a new registry, bridge daemon, JSON live store, OBS plugin, or per-feed buffer manager to make this happen. If the app needs one of those, first move the missing authority into LocalcastRuntime, Fensalir, or Faust/native DSP.

The app exists to make the live system coherent, not to preserve every deadline-era surface. Deadline fossils may be used as diagnostic witnesses. They do not get promoted because they still twitch.

Backlinks