Video Streaming

End-to-end video from Agent camera to Remote display, including device configuration, network transport, and role-based camera assignment.

Overview

Video streaming uses two parallel transport tracks:

Agent                                       Remote
+--------------------+                      +--------------------+
| VideoSource        |                      | AgentClientWidget  |
|   |                |                      |   |                |
|   v                |                      |   v                |
| VisionStreamManager|  StreamCourier       | StreamCourier      |
|   |                | ==================> |   |                |
|   | JPEG encode    |  (UDP, lossy)        |   | JPEG decode    |
|   | fragment       |                      |   | reassemble     |
+--------------------+                      +--------------------+

|   |                |  BlobCourier         |   |                |
|   | PNG encode     | ==================> |   | PNG decode     |
|   | (on request)   |  (UDP, reliable)     |   | snapshot view  |
+--------------------+                      +--------------------+

• Track 1 (StreamCourier): Continuous lossy video at 10-30 fps for live preview
• Track 2 (BlobCourier): On-demand lossless snapshots for inspection

Naming convention

VisionCharacteristics

Role-based camera assignment system. Each Agent node assigns its video sources to named vision roles that persist across reboots.

Location

src/libs/libvideo/video/
  VisionCharacteristics.hpp / .cpp

Predefined roles

Auto-assignment heuristic

When no explicit assignment exists for PrimaryVision:

1. If only one video source exists, assign it automatically
2. If Qt6 reports QCameraDevice::position() == FrontFace, prefer it
3. Otherwise, pick the source with the highest max resolution
4. User can always override via the Vision Designation Activity

Persistence

Assignments are stored in node settings keyed by USB vendor:product ID (via DeviceMatcher). This survives device re-enumeration and USB port changes.

Key API

class VisionCharacteristics : public QObject {
    void assign(VisionCharacteristic role, const QCameraDevice &device);
    void unassign(VisionCharacteristic role);
    QCameraDevice deviceFor(VisionCharacteristic role) const;
    void autoAssign(const QList<QCameraDevice> &available);
    void save(QSettings &settings) const;
    void load(const QSettings &settings);

signals:
    void assignmentChanged(VisionCharacteristic role);
};

StreamCourier

Real-time lossy media streaming courier. Handles frame sequencing, fragmentation, and flow-controlled delivery over UDP.

Location

src/libs/libcomms/comms/couriers/stream/
  StreamCourier.hpp / .cpp
  StreamHeader.hpp

Design decisions

• One courier for all media types -- video, audio, and future streams share the same protocol, distinguished by StreamType enum
• No retransmission -- lost fragments mean dropped frames (unlike BlobCourier)
• Adaptive quality -- reads RTT and packet loss from the session's existing ReliabilitySystem and FlowControl

Stream header

struct StreamHeader {
    StreamType type;        // Video, Audio, etc.
    quint8 streamId;        // Distinguishes multiple streams of same type
    quint32 frameNumber;    // Monotonically increasing per stream
    quint64 timestampUs;    // Microsecond capture timestamp
    quint16 fragmentIndex;  // Fragment within frame
    quint16 fragmentCount;  // Total fragments in this frame
    quint16 payloadSize;    // Bytes in this packet's payload
    quint8 codecId;         // Codec that encoded this frame
    quint8 flags;           // KEY_FRAME, END_OF_STREAM, etc.
};

Packet types

Frame fragmentation

Large encoded frames (e.g., 20KB JPEG at 1080p) are split into MTU-sized fragments. The receiver assembles by frameNumber + fragmentIndex. If any fragment is missing when the next frame starts arriving, the incomplete frame is discarded.

Integration with courier sets

StreamCourier is registered in both AgentClientCourierSet and RemoteClientCourierSet alongside existing couriers (AgentStateCourier, BlobCourier, etc.).

VisionStreamManager

Agent-side orchestrator that manages the video capture and streaming lifecycle.

Location

src/libs/libagent/agent/
  VisionStreamManager.hpp / .cpp

Responsibilities

• Starts/stops video capture based on VisionEnabled flag in AgentStateCourier
• Reads camera assignment from VisionCharacteristics
• Encodes frames (JPEG) and sends via StreamCourier
• Adapts quality/framerate based on flow control feedback

Lifecycle

1. Agent receives VisionEnabled = true from Remote via AgentStateCourier
2. VisionStreamManager opens the PrimaryVision camera via VideoSystem
3. Frames are JPEG-encoded, fragmented, and sent via StreamCourier
4. When VisionEnabled = false, camera is stopped and streaming ceases

VideoSystem

Video-specific device facade wrapping DeviceManager.

Location

src/libs/libvideo/video/
  VideoSystem.hpp / .cpp

Key API

class VideoSystem : public QObject {
    static VideoSystem& instance();
    QList<QCameraDevice> devices() const;
    QCameraDevice defaultDevice() const;
    QCameraDevice findDevice(const QString& name) const;
    QCameraFormat bestFormat(const QCameraDevice& device, QSize preferred = QSize()) const;
    void initialize();

signals:
    void devicesChanged();
};

VideoSource

Unified camera wrapper consolidating lifecycle management from multiple locations.

Location

src/libs/libvideo/source/
  VideoSource.hpp / .cpp

Key API

class VideoSource : public QObject {
    void setDevice(const QCameraDevice& device);
    void start();
    void stop();
    bool isActive() const;
    QVideoSink* videoSink() const;
    QMediaCaptureSession* captureSession() const;

signals:
    void frameReady(const QVideoFrame& frame);
};

FrameConverter

Consolidates four duplicated QVideoFrame-to-QImage conversion sites into one utility class.

Location

src/libs/libvideo/video/
  FrameConverter.hpp / .cpp

Consolidated call sites

Key API

class FrameConverter {
    static QImage toImage(const QVideoFrame& frame);
    static QImage toImage(const QVideoFrame& frame, QSize scaleTo);
    static QImage toGrayscale(const QVideoFrame& frame);
    static QPixmap toPixmap(const QVideoFrame& frame);
    static QImage errorImage(QSize size, const QString& message = "No Frame");
};

Snapshot protocol

Full-resolution snapshots use the existing BlobCourier for reliable delivery:

1. Remote sets SnapshotRequested flag in AgentStateCourier
2. Agent captures full-resolution frame from VideoSource
3. Agent encodes as lossless PNG and queues via BlobCourier (high priority)
4. Agent clears SnapshotRequested flag to acknowledge
5. Remote receives blob, decodes, and displays in SnapshotViewerActivity

See Also

• Media Architecture - Design rationale
• Device Management - DeviceManager, DeviceCache, DeviceMatcher
• Graph Framework - Graph-based processing
• Couriers and Sessions - How couriers work