Media Processing Architecture

How OctoMY™ unifies audio and video processing across robotics, music, karaoke, and acoustic communication applications.

Design Philosophy

The media system is a buffet of powerful libraries that work well together. Consumers pick only the layers they need: a simple intercom uses just the device API, while a music tracker uses the full graph engine. The key constraint is one stack, no duplication -- every consumer uses the same device management, the same DSP primitives, the same format negotiation.

Why a unified media system?

OctoMY™ has many consumers of audio and video hardware:

Without unification, each consumer independently enumerates devices, manages lifecycles, and duplicates frame/sample conversion code. The unified system eliminates this duplication while letting each consumer use only the complexity it needs.

Architecture layers

                        APPLICATIONS
    +----------+----------+----------+----------+----------+----------+
    |   Bahn   |Singularity|  Banter  |  Gibber  | Intercom |  Voice   |
    | (tracker)| (karaoke) |(acoustic)|(FSK modem)|  (PTT)  |  (TTS)   |
    |          |          |          |          |          |          |
    | uses:    | uses:    | uses:    | uses:    | uses:    | uses:    |
    | engine + | engine + | device   | device   | device   | DSP      |
    | device   | device   | API +    | API +    | API only | units +  |
    | API      | API      | pump     | pump     |          | device   |
    +----+-----+----+-----+----+-----+----+-----+----+-----+----+-----+
         |          |          |          |          |          |
    +----+----------+----------+----------+----------+----------+-----+
    |               SHARED AUDIO STACK                                |
    |                                                                 |
    |  +----------------------------------------------------------+   |
    |  |  PROCESSORS (libbahn)                                    |   |
    |  |  Extend AudioNode with tracker state (sequencer data,    |   |
    |  |  live params, pause, perf tracking, serialization)       |   |
    |  +----------------------------------------------------------+   |
    |                          |                                      |
    |  +-----------------------+----------------------------------+   |
    |  |  GRAPH ENGINE (libmediacore) -- optional                 |   |
    |  |  AudioEngine, Schedule, AudioNode, AudioContext           |   |
    |  |  Generic graph processing -- no tracker knowledge         |   |
    |  +----------------------------------------------------------+   |
    |                                                                 |
    |  +----------------------------------------------------------+   |
    |  |  DSP UNITS (libmediacore) -- template library            |   |
    |  |  SinOsc, Saw, Tb303, Delay, Biquad, Drum, ...           |   |
    |  |  Usable standalone OR inside graph processors             |   |
    |  +----------------------------------------------------------+   |
    |                                                                 |
    |  +----------------------------------------------------------+   |
    |  |  AUDIO I/O (libaudio) -- always used                     |   |
    |  |  AudioSystem (facade), AudioSource, AudioSink             |   |
    |  |  AudioPump (engine<->device bridge, also direct mode)     |   |
    |  +----------------------------------------------------------+   |
    |                                                                 |
    |  +----------------------------------------------------------+   |
    |  |  DEVICE MANAGEMENT (libmediacore) -- foundation           |   |
    |  |  DeviceManager, DeviceCache, DeviceMatcher                |   |
    |  +----------------------------------------------------------+   |
    +-----------------------------------------------------------------+

Dependency direction

Dependencies flow strictly downward:

• libbahn (Processors) depends on libmediacore (AudioNode, units, engine)
• libaudio (AudioPump) depends on libmediacore (AudioEngine, Mixer)
• libmediacore has no dependency on libbahn or libaudio

This prevents circular dependencies and allows the graph engine to be used without any tracker knowledge.

Library structure

The media code is distributed across purpose-specific libraries:

Pro Tip

This is "Option B" from the original design evaluation -- fine-grained separation into purpose-specific libraries rather than a monolithic media library. The trade-off is more libraries to manage, but each has a clear, testable scope.

Device management

All device enumeration goes through a single DeviceManager singleton in libmediacore. This solves the historical problem of scattered QMediaDevices calls throughout the codebase.

Key features:

• TTL-based caching via DeviceCache -- avoids redundant hardware enumeration
• Fuzzy device matching via DeviceMatcher -- finds devices by name, USB ID, or string similarity
• Split initialization -- audio and video device enumeration initialize separately to avoid PipeWire crashes on Linux
• Hot-plug signals -- notifies consumers when USB devices are connected or disconnected

See Device Management Reference for API details.

Graph-based processing

The graph framework (originally from libbahn's music tracker) provides real-time audio/video processing with:

• DFS cycle detection and topology change tracking
• BFS compilation into an ordered Schedule of Job entries
• Single contiguous memory block with 32-byte aligned buffers (SIMD-ready)
• Zero allocation during processing -- all memory allocated at graph compilation time

The graph engine is optional. Simple consumers (banter, gibber, intercom) bypass it entirely and use direct callbacks. Complex consumers (bahn, singularity) build processing graphs. Singularity plays its song previews, backing tracks and jingles through one graph: QMediaPlayer only decodes, and the graph owns the device, so a device with 14 separate outputs works as well as a stereo one. See Channel layouts.

See Graph Framework Reference for API details.

Audio engine

The audio engine extracts bahn's proven processing infrastructure into libmediacore as generic interfaces, allowing any consumer to use graph-based or direct audio processing without tracker dependencies.

Key design decisions:

• AudioNode is the minimal processing interface (process + characteristics + prepare). Processor (libbahn) extends it with tracker state; non-tracker consumers implement AudioNode directly.
• AudioPump bridges processing to hardware via QIODevice pull model. Supports graph mode (AudioEngine) and direct mode (callback lambda) for simple pipelines.
• DSP units are template-based (Unit<T>) and usable standalone or inside graph processors. One canonical implementation per effect type -- no duplication between libaudio and libmediacore.
• AudioSystem is a facade providing deferred initialization, format negotiation with SampleRatePolicy hints, and role-based device assignment via AudioCharacteristics.

See Audio Engine Reference for API details.

Video streaming

Video frames travel from Agent to Remote using a dedicated StreamCourier over the existing comms infrastructure:

Agent                                       Remote
+--------------------+                      +--------------------+
| VideoSource        |                      | AgentClientWidget  |
|   |                |                      |   |                |
|   v                |                      |   v                |
| VisionStreamManager|  StreamCourier       | StreamCourier      |
|   |                | ==================> |   |                |
|   | JPEG encode    |  (UDP packets)       |   | JPEG decode    |
|   | chunk          |                      |   | reassemble     |
+--------------------+                      +--------------------+

Device-to-role mapping uses VisionCharacteristics -- persistent assignments like "this USB camera is the PRIMARY_VISION sensor" that survive reboots.

See Video Streaming Reference for protocol details.

GPU strategy

GPU compute is centralized in libconstruct using Qt6 RHI (Rendering Hardware Interface) for cross-platform support. Media libraries that need GPU acceleration (face detection, video processing) will access it through libconstruct's compute pipeline rather than managing their own GPU contexts.

Naming conventions

"Video" is about the pipe, "Vision" is about the purpose. Qt types (QCamera, QCameraDevice) stay as-is -- they are wrapped, not renamed.

See Also

• Media Reference - API reference for all media components
• Device Management - DeviceManager, DeviceCache, DeviceMatcher
• Graph Framework - Graph processing infrastructure
• Audio Engine - AudioNode, AudioEngine, AudioPump, AudioSystem, DSP units
• Video Streaming - Vision system and streaming protocol
• Couriers and Sessions - How StreamCourier fits into the comms system