Encryption

Reference for OctoMY™'s end-to-end encryption system covering handshake encryption, session key derivation, and courier data AEAD.

Key Point

Encryption operates at two levels: handshake packets use hybrid asymmetric encryption with the recipient's public key, while courier data uses symmetric XChaCha20-Poly1305 AEAD with a per-session key derived during the handshake. Both RSA and Ed25519 keys are supported, including cross-algorithm sessions.

Overview

Encryption is controlled at runtime via Comms::encryptionEnabled() (default: enabled).

Hybrid asymmetric encryption

Both RSA and Ed25519 implement hybrid encryption via ISigningKey::encrypt()/decrypt(). The wire format is self-describing via a version byte.

RSA hybrid (version 0x01)

[0x01][RSA-encrypted symKey: keySize/8 bytes][nonce: 24 bytes][AEAD ciphertext+tag: N+16 bytes]

1. Generate random 32-byte symmetric key
2. RSA-encrypt the symmetric key with recipient's RSA public key
3. Generate random 24-byte nonce
4. XChaCha20-Poly1305 AEAD encrypt the payload with the symmetric key

Ed25519 hybrid (version 0x02)

[0x02][ephemeral X25519 pubkey: 32 bytes][nonce: 24 bytes][AEAD ciphertext+tag: N+16 bytes]

1. Generate ephemeral Ed25519 keypair, convert to X25519
2. X25519 ECDH with recipient's public key (converted to X25519)
3. XChaCha20-Poly1305 AEAD encrypt the payload with the shared secret

Cross-algorithm support

The receiver reads the version byte to determine the decryption path. This means an RSA node can communicate with an Ed25519 node — each side encrypts with the recipient's key type, and the recipient decrypts with their own key.

Session key derivation

After the 3-way handshake completes, both sides derive a symmetric session key for AEAD courier encryption.

Ed25519 ↔ Ed25519 (ECDH path)

Both sides convert their Ed25519 identity keys to X25519:

sharedSecret = x25519KeyExchange(mySecretX25519, theirPublicX25519)
sessionKey = SHA256(sharedSecret || synNonce || synAckNonce)

Both sides compute the same result (X25519 ECDH is commutative).

RSA or cross-algorithm (key-transport path)

The initiator generates random 32-byte keyMaterial and includes it in the SYN (which is already hybrid-encrypted). The responder extracts it and echoes SHA256(keyMaterial) in the SYN-ACK as confirmation.

sessionKey = SHA256(keyMaterial || synNonce || synAckNonce)

Mixing in both nonces binds the session key to this specific handshake instance, preventing replay of captured SYN packets.

Courier data AEAD

After session key derivation, all courier data is encrypted with XChaCha20-Poly1305.

Wire format

[sessionID: 2 bytes][nonceCounter: 8 bytes][AEAD ciphertext+tag: N+16 bytes]

Nonce construction

The 24-byte AEAD nonce is constructed from the 64-bit counter:

[counter: 8 bytes big-endian][zeros: 16 bytes]

Each side maintains an independent monotonically increasing counter (mTxNonceCounter).

Plaintext format (before encryption)

[courierID: 4 bytes][payloadSize: 2 bytes][payload: N bytes]

Replay protection

A 64-bit sliding window detects replayed or very old packets:

• Nonces above the high watermark are accepted and the window advances
• Nonces within the 64-packet window are checked against the bitmap (duplicates rejected)
• Nonces below the window (older than 64 packets) are rejected

This design tolerates UDP reordering (packets within 64 of the latest are accepted in any order) while rejecting replays and very old packets.

Packet size budget

Handshake packets (hybrid encrypted)

Both fit within UDP MTU (1472 bytes for typical Ethernet).

Courier data packets (AEAD encrypted)

Overhead: 26 bytes (2 + 8 + 16) — leaves ~1446 bytes for courier payload, well above typical courier payloads and BLOB_CHUNK_SIZE (400 bytes).

Key infrastructure

ISigningKey interface

All key operations are algorithm-agnostic via ISigningKey:

KeyFactory

Creates keys by algorithm:

auto key = KeyFactory::create(KeyFactory::Algorithm::Ed25519);
auto key = KeyFactory::create(KeyFactory::Algorithm::RSA, 4096);
auto key = KeyFactory::fromPEM(pemString);
auto key = KeyFactory::fromVariantMap(map);

Implementations

Runtime configuration

// Enable/disable encryption
comms->setEncryptionEnabled(true);

// Check if session has derived key
if (session->sessionKeyReady()) {
    // AEAD courier encryption is active
}

When encryption is disabled, handshake packets are sent in cleartext and courier data is unencrypted. The session key derivation step is skipped.

Test coverage