QuicR - Media Delivery Protocol over QUIC

Internet-Draft	quicr	March 2022
Jennings & Nandakumar	Expires 5 September 2022	[Page]

Abstract

Recently new usecases have emerged requiring higher scalability of delivery for interactive realtime applications and much lower latency for streaming applications and a combination thereof. draft-jennings-moq-quicr-arch specifies architectural aspects of QuicR, a media delivery protocol based on publish/subscribe metaphor and Relay based delivery tree, that enables a wide range of realtime applications with different resiliency and latency needs.¶

This specification defines the protocol aspects of the QuicR architecture.¶

4. QuicR Protocol

4.1. Publish API and PUBLISH Message

Entities the want to send named objects will use Publish API to trigger PUBLISH messages from a QuicR client to the Origin server (via Relay(s) if present). The publish message identifies active flow of named objects, such a data can be originating from a given QuicR endpoint client or a might be relayed by other entities. In the latter case, the relaying entity MUST NOT change the name associated with the object being published unless the intermediary is a publisher. All the publishes MUST be authorized by the Origin server for¶

In general, the Publish API specifies following thing about the named objects being published.¶

  NAME           (String)
  RELIABLE       (Boolean)
  PRIORITY       (Enumeration)
  BESTBEFORE     (Number)
  TIMESTAMP      (Number)
  DISCARDABLE    (Boolean)
  Payload        (Bytes)

NAME: Every PUBLISH message MUST have a name to identify the object to the QuicR components (relays/origin-server/other clients).¶
PAYLOAD: End-to-End encrypted application data associated with the named object to be published (e.g audio sample, video frame, game move), which are typically timestamped buffers of application data.¶
RELIABLE: Boolean flag to indicate the if QuicR should use QUIC Streams (true value) or QUIC datagrams (false value).¶
BESTBEFORE: Time to live defines the time after which the named object can be discarded from the caches.¶
PRIORITY: Enumeration specifying relative priority of named objects being published by this end-point. This can help Relay to make dropping/caching decisions.¶
DISCARDABLE: Provides an hint to the relays for making drop decisions.¶

The QuicR PUBLISH message(s) are represented as below and are encapsulated in QUIC packets.¶

A> All the integer fields are variable length encoded¶

PUBLISH {
  ORIGIN            (String)
  NAME              (Number128)
  FLAGS             (Byte)
  FRAGMENT_ID       (Number16)
  BESTBEFORE        (Number64)
  TIMESTAMP         (Number64)
  PAYLOAD           (ByteArray)
}

Flags := Reserved (4) | IsDiscardable (1) | Priority (3)

Entities that intend to receive named objects will do so via subscriptions to the named objects. The Subscribe API triggers sending SUBSCRIBE messages. Subscriptions are sent from the QuicR clients to the origin server(s) (via relays, if present) and are typically processed by the relays. See {#relay_behavior} for further details. All the subscriptions MUST be authorized.¶

Subscriptions are typically long-lived transactions and they stay active until one of the following happens¶

a client local policy dictates expiration of a subscription.¶
optionally, a server policy dicates subscription expiration.¶
the underlying transport is disconnected.¶

When an explicit indication is preferred to indicate the expiry of subscription, it is indicated via SUBSCRIPTION_EXPIRY message.¶

Until the subscription is active for a given name, the Relay(s) should send named objects it receives to all the matching subscribers. The Relays may optionally cache the objects. A QuicR client can renew its subscriptions at any point by sending a new SUBSCRIBE message to the Origin server. Such subscriptions MUST refresh the existing subscriptions for that name. A renewal period of 5 seconds is RECOMMENDED.¶

SUBSCRIBE {
  ORIGIN             (String)
  SUBSCRIPTION_ID    (Number64)
  NAMES              [NAME]
  INTENT             [Enumeration]
}

NAME {
 name               (Number128)
 mask               (Number7)
}

The ORIGIN field identifies the Origin server for which this subscription is targeted. SUBSCRIPTION_ID is a randomly chosen to identify the subscription by a given client and is local to the client's session. NAMES identify the fully formed names or wildcarded names along with the appropriate bitmask length.¶

The INTENT field specifies how the Relay should provide the named objects to the client. Following options are defined for the INTENT¶

IMMEDIATE: Deliver any new objects it receives that match the name¶

CATHCH_UP: Deliver any new objects it receives and in addition send any previous objects it has received that are in the same group that matches the name.¶

WAIT_UP: Wait until an object that has a objectId that matches the name is received then start sending any objects that match the name.¶

MOST_RECENT: Deliver any new objects it receives and in addition send the most recent object it has received that is in the same group that matches the name.¶

4.2.1. Aggregating Subscriptions

Subscriptions are aggregated at entities that perform Relay Function. Aggregating subscriptions helps reduce the number of subscriptions for a given named objects in transit and also enables efficient disrtibution of published media with minimal copies between the client and the Origin server , as well as reduce the latencies when there are multiple subscribers for a given named object behind a common set of cloud server.¶

4.2.2. Wildcarded Names

The names used in SUBSCRIBE can be truncated by skipping the right most segments of the name that is application specific, in which case it will act as a wildcard subscription to all names that match the provided part of the name. The same is indicated via bitmask associated with the name in SUBSRIBE messages. Wildcard search on Relay(s) thus turns into a bitmask at the appropriate bit location of the hashed name.¶

For example, in an web conferencing use case, the client may subscribe to just the origin and meetingID to get all the media for a particular conference.¶

4.3. PUBLISH_INTENT and PUBLISH_INTENT_OK Message

The PUBLISH_INTENT message indicates the names chosen by a Publisher for transmitting named objects within a session. This message is sent to the Origin Server whenever a given publisher intends to publish on a new name (which can be at the beginning of the session or during mid session). This message is authorized by the Origin server and thus requires a mechanism to setup the initial trust (via out of band) between the publisher and the origin server.¶

A PUBLISH_INTENT message is represented as below:¶

 PUBLISH_INTENT {
  ORIGIN         (String)
  PUBLISHER_ID   (Number64)
  NAMES          [Number64]
 }

The ORIGIN field is used by the cloud Relays to choose the Origin server to forward the PUBLISH_INTENT message.¶

On a successful validation at the Origin server, a PUBLISH_INTENT_OK message is returned by the Origin server.¶

 PUBLISH_INTENT_OK {
  PUBLISHER_ID   (Number64)
  NAMES          [Number64]
 }

This message enables cloud relays to know the authorized names from a given Publisher. This helps to make caching decisions, dealing with collisions and so on.¶

A cloud relay may start caching the data associated with the names that has not been validated yet by the origin server and decide to flush its cache if no PUBLISHINTENTOK is received within a given implementation defined timeout. This is an optimization that would allow publishers to start transmitting the data without needing to wait a RTT.¶

Names chosen by the publishers MUST be unique with in a given session to avoid collisions. It is up to the application to define the necessary rules to ensure the uniqueness constraint. Cloud entities like Relays are agnostic to these rules and handle collisions by either overriding or dropping the associated data.¶

4.6. RELAY_REDIRECT Message

RELAY_REDIRECT message enables relay failover scenarios that is sent in response to PUBLISH, PUBLISH_INTENT and SUBSCRIBE messages indicating the new relay to the clients.¶

RELAY_REDIRECT
{
  RELAY_ADDRESS  (String)
}

4.7. Fragmentation and Reassembly

Application data may need to be fragmented to fit the underlying transport packet size requirements. QuicR protocol is responsbile for performing necessary fragmentation and reassembly. Each fragment needs to be small enough to send in a single QUIC packet. The low order bit is also a Last Fragment Flag to know the number of Fragments. The upper bits are used as a fragment counter with the first fragment starting at 1. The FRAGMENT_ID with in the PUBLISH message identifies the individual fragments.¶

5. Relay Function and Relays

Clients may be configured to connect to a local relay which then does a Publish/Subscribe for the appropriate named data towards the origin or towards another Relay. These relays can aggregate the subscriptions from multiple clients. This allows a relay in the LAN to aggregate request from multiple clients in subscription to the same data such that only one copy of the data flows across the WAN. In the case where there is only one client, this may still provides benefit in that a client that is experiencing loss on WIFI WAN has a very short RTT to the local relay so can recover the lost data much faster, and with less impact on end user QoE, than having to go across the LAN to recover the data.¶

Relays can also be deployed in classic CDN cache style for large scale streaming applications yet still provide much lower latency than traditional CDNs using Dash or HLS. Moving these relays into the 5G network close to clients may provide additional increase in QoE.¶

At a high level, Relay Function within QuicR architecture support store and forward behavior. Relay function can be realized in any component of the QuicR architecture depending on the application. Typical use-cases might require the intermediate servers (caches) and the origin server to implement the relay function. However the endpoint themselves can implement the Relay function in a Isomorphic deployment, if needed.¶

Non normatively, the Relay function is responsible carryout the following actions to enable the QuicR protocol:¶

On reception of SUBSCRIBE message, forward the message to the Origin server, and on the receipt of SUBSCRIBE_REPLY, store the subscriber info against the names in the NAMES_SUCCESS field of the SUBSCRIBE_REPLY message. If an entry for the name exists already, add the new subscriber to the list of Subscibers. [ See Subscribe Aggregations].¶
If there exists a matching named object for a subscription in the cache, forward the data to the subscriber(s) based on the subscribe INTENT.¶
On reception of PUBLISH_INTENT message, forward the message to the Origin server, and on the receipt of PUBLISH_INTENT_OK, store the names as authorized against a given publisher.¶
If a named object arrives at the relay via PUBLISH message , cache the name and the associated data, also distribute the same to all the active subscribers, if any, matching the given name.¶

The payload associated with a given PUBLISH message MUST not be cached longer than the BESTBEFORE time specified. Also to note, the local policies dicatated by the caching service provider can always overwrite the caching duration for the published data.¶

Relays MUST NOT modify the either the Name or the contents of PUBLISH/SUBSCRIBE messags expect for performing the necessary forwarding and caching operations as described above.¶

5.1. Relay fail over

A relay that wants to shutdown and use the redirect message to move traffic to a new relay. If a relay has failed and restarted or been load balanced to a different relay, the client will need to resubscribe to the new relay after setting up the connection.¶

TODO: Cluster so high reliable relays should share subscription info and publication to minimize of loss of data during a full over.¶

5.2. Relay Discovery

Local relays can be discovered via MDNS query to TODO. Cloud relays are discovered via application defined ways that is out of scope of this document. A Relay can send a message to client with the address of new relay. Client moves to the new relay with all of its Subscriptions and then Client unsubscribes from old relay and closes connection to it.¶

This allows for make before break transfer from one relay to another so that no data is lost during transition. One of the uses of this is upgrade of the Relay software during operation.¶

8. QuicR Usages

8.1. Real-time Conferencing Application

This subsection expands on using QuicR as the media delivery protocol for a real-time multiparty A/V conferencing applications.¶

8.1.1. Naming

Below provides one possible way to subdivide the application component portion for a conferencing scenario.¶

MeetingID: A identifier for the context of a single group session. Is unique with in scope of Origin.¶
SenderID: Identifies a single endpoint client within the MeetingID that publishes data.¶
SourceID: Identifies a stream of media or content from that Sender. Example: A client that was sending media from a camera, a mic, and screen share might use a different sourceID for each one. A scalable codec with multiple layers or simulcast streams each would use a different sourceID for each quality representation.¶
QualityID: Identifies media quality level - hi/low/medium¶

For A/V conference, the encoded media frames are formed into group of objects, where each group defines an independent decodeable object set. In case of video, each group comprises of objects on I-Frame boundaries, for audio a group would map to an encoded audio sample for codecs used today.¶

A conforming name is formatted as URLs like:¶

quicr://domain:port/MeetingID/SenderID/SourceID/QualityID/Group/Object¶

8.1.2. Manifest

As a prerequisite step, participants exchange their transmit and receive capabilities like sources, qualities, media types and so on, with application server (can be origin server). This is done out-of-band and is not in the scope of QuicR protocol.¶

However, as a outcome of this step is generation of the manifest data that describes the names, qualities and other information that aid in carrying out media delivery with QuicR protocol. This would for example setup unique SourceID sub-part of the application component for each media source or quality layers or a combination thereof. Similarly the SenderID may get mapped from a roster equivalent for the meeting.¶

A manifest may get updated several times during a session - either due to capabilities updates from existing participants or new participants joinings or so on.¶

Participants who wish to receive media from a given meeting in a web conference will do so by subscribing to the meeting's manifest. The manifest will list the name of the active publishers.¶

8.1.3. Example

Below picture depicts a simplified QuicR Publish/Subscribe protocol flow where participants exchange audio in a 3-party realtime audio conference.¶

                           ┌──────────┐
                           │          │
                           ▼          │
                 ┌───────────────────┐│  sub:meeting123/*
                 │ acme.meetings.com ││
                 └─▲─────────────┬───┘│
                   │             │    │
                   │   pub:alice │    │
                   │     medi┌───▼─────────────┐
 pub:alice  ┌──────┘         │      Relay      │────┐
 ,meeting1  │                └┬────────────────┘    │
   23..     │                 │ sub:meeting123/*    │
            │        pub:alice│      ▲▲             │
            │          media  │┌─────┘└──────┐      │
            │                 ▼│             │      │
      ┌───────────┐       ┌────────┐    ┌─────────┐ │
      │   Alice   │       │  Bob   │    │  Carl   │ │   pub:alice
      └───────────┘       └────────┘    └─────────┘ │     media
                                             ▲      │
                                             │      │
                                             └──────┘%

In the depicted protocol flow, Alice is the publisher while Bob and Carl are the subscribers. As part of joining into the conference, Bob and Carl subscribe to the name quicr://acme.meetings.com/meeting123/* to receive all the media streams being published for the meeting instance meeting123. Their subscriptions are sent to Origin Server via a Relay.The Relay aggregates the subscriptions from Bob and Carl, forwarding just one subscribe message. On Alice publishing her media stream fragments from a camera source to the Origin server, the same is forwarded to Relay. The relay will in turn forward the same to Bob and Carl based on their subscriptions.¶

Here is another scenario, where Alice has 2 media sources (mic, camera) and is able to send 2 simulcast streams for video (corresponding to 2 quality levels) and audio encoded via 2 different codecs, might have different sourceIDs as listed below¶

Source       --> SourceID
------------------------
mic_codec_1  --> audio_opus
mic_codec_2  --> audio_lyra
vid_simul_1  --> video_hi
vid_simul_2  --> video_low

Alice's SenderID -> Alice and she is joining meeting with id
meeting123

Names that Alice can publish includes:

quicr://acme.meetings.com/meeting123/Alice/audio_opus/...
quicr://acme.meetings.com/meeting123/Alice/audio_lyra/...
quicr://acme.meetings.com/meeting123/Alice/video_hi/...
quicr://acme.meetings.com/meeting123/Alice/video_low/...

Manifest encoded as json objects might capture the information as below.¶

{
    "origin": "acme.meetings.com"
    "meeting": "meeting123",
    publisher {
        id: "Alice",
        source: [
            {
                "type" : "audio",
                "sourceId": "alice-mic",
                "streams": [
                    {
                        "id": "audio_opus,
                        "codec": "opus",
                        "quality": "48khz",
                    }, {
                        "id" : "audio_lyra",
                        ....
                    }
                ]
            }, {
            "type": "video",
            "sourceId": "alice-cam",
            "streams": [
                {
                    "id": "video_hi",
                    "codec:" : "av1",
                    "quality" : "1920x720_60fps"
                },
                {
                    "id": "video_lo",
                    "codec:" : "av1",
                    "quality" : "640x480_30fps"
                }
            ],
            },
        ]
    }
}

With the names as above, any subscriber retrieving the manifest has the necessary information to send SUBSCRIBE for the named data of interest. The same happens when the manifest is updated once the session is in progress.¶

A sample subscribe for lyra audio and low-res video from Alice would have the form:¶

quicr://meetings.com/meeting123/alice/audio/alice-mic/audio_lyra/*¶

quicr://meetings.com/meeting123/alice/video/alice-cam/video_lo/*¶

8.2. Push To Talk Media Delivery Application

Frontline communications application like Push To Talk have close semblances with the publish/subscribe metaphor. In a typical setup, say a retail store, there are mutiple channels (bakery, garden) and retail workers with PTT communication devices access the chatter over such channels by tuning into them. In a typical use-case, the retails workers might want to tune into one or more channels of their interest and expect the media delivery system to deliver the media asynchornusly as talk spurts.¶

In general such a system needs the following:¶

A way for the end users to tune to their channels of interest and have these interests be longlived transactions.¶
A way for system to efficiently distribute the media to all the tuned in end users per channel.¶
A way for end user to catch up and playback when switching the channels.¶

8.2.1. Naming

In the case of PTT, the following mappings can be one way to defined the application subcomponents.¶

StoreID - Identifier for store at various locations.¶
ChannelID - Each PTT channel represents its own high level resource¶
SenderID - Authenticated user's Id who is actively checkedin into a frontline workplace (ex: a store).¶

With above division of Application Component, a sample QuicR named object might have following form¶

quicr://wallmart.com/store22/ch3/alice/groupID/objectID¶

In the above example, StoreID is store22, with ChannelID being channel 3 and the media corresponds to user, alice. The group of objects in this case map to the QuicR audio objects for audio delivery.¶

A SUBSCRIBE for all the PTT messages sent over channel 3, could be represented as¶

quicr://wallmart.com/store22/ch3/*¶

This will trigger the Relay Function to forward PTT audio from everyone on channel-3 to sent to the subscriber, as is typically expected in a frontline applications.¶

8.2.2. Manifest

For PTT application, a manifest describes various active PTT channels as the main resource. Subsribers who tune into channels typically get the names from the manifest to do so. Publishers publish their media to channels that they are authorized to.¶

8.2.2.1. Example

An example retail store scenario where users Alice, Bob subscribe to the bakery channel and Carl subscribes to the gardening channel. Also an annoucement from the store manager Tom, on bakery channel gets delivered to Alice and Bob but not Carl.¶

Alice: authorized to talk/listen on Channel Bakery
Bob:   authorized to talk/listen on Channel Bakery
Carl:  authorized to talk/listen on Channel GARDENING
Tom:   authorized to talk/listen on channels Bakery
       and Gardening

Bakery Channel Id -> bakery
Gardening Channel Id -> gardening

Manifest encoded as json objects might capture the information as below. [This encoded is for information purposes only.]¶

{
    "origin": "walmart.com",
    "storeId": "store22
    "channel": [
        {
            "name": "bakery",
            "id" : "bakery",
            "codec": "opus",
            "quality": "48khz",
            ....
        },
        {
            "name": "gardening",
            "id" : "gardening",
            "codec": "opus",
            "quality": "48khz",
            ...
        },
    ]
}

Alice and Bob shall subscribe to channel Bakery quicr://wamart.com/store22/bakery/* and Carl subscribe to channel Gargdening quicr://wamart.com/store22/gardening/*.¶

8.3. Low Latency Streaming Applications

A typical streaming application can be divided into 2 halves - media ingest and media distribution. Media ingestion is done via pushing one or more streams of different qualities to a central server. Media Distribution downstream is customized (via quality/rate adapation) per consumer by the central server.¶

One can model ingestion as sending PUBLISH mesages and the associated sources as publishers. Similarly, the consumers/end clients of the streaming media SUBSCRIBE to the media elements whose names are defined in the manifest. Manifest describes the name and qualities associated with media being published. The central severs (Origin) themselves act as publisher for producing streams with different qualities.¶

Streaming use-cases requiring lower latencies and high degree of realtime interactivity (chat for example) can easilt fit into QuicR's media delivery protocol.¶

Lower latencies can be achieved by the relay forwarding the data as they arrive to the subscribed clients.¶

Few sample scenarios that have such constrainsts are listed below:¶

Professional streamers (gamers/musicians) interacting with a live audience on social media, often via a directly coupled chat function in the viewing app/environment. A high degree of interactivity between the performer and the audience is required to enable engagement.¶
Live Auctions are another category of applications where an auction is broadcasted to serveral participants. The content must be delivered with low latency and more importantly within a well-defined sync across endpoints so customers trust the auction is fair.¶

Visual and aural quality are secondary in priority in these scenarios to sync and latency. This in turn increases revenue potential from a game/event.¶

8.3.1. Naming and Manifest Considerations

For downstream distribution of media data to clients with varying requirements, the central server (along with the source) generate different quality media representations. Each such quality is represented with a unique name and subscribers are made know of the same via the Manifest.¶

{
  "resource" : "jon.doe.music.live.tv",
  "streams: [
      {
      "id": "1234",
      "codec": "av1",
      "quality": "1920x720_60fps"
      },
      {
      "id": "5678",
      "codec": "av1",
      "quality": "3840x2160_30fps"
      },
      {
      "id": "9999",
      "codec": "av1",
      "quality": "640x480_30fps"
      },
  ]
}

Consumers end points subscribe to one or more names representing the quality based on their capabilities. This enables the relay to forward the ingested data to be sent as they arrive to the subscribers.¶

QuicR - Media Delivery Protocol over QUIC

Abstract

Status of This Memo

Copyright Notice

Table of Contents

1. Introduction

2. Contributing

3. Terminology