HESP - High Efficiency Streaming Protocol

3.1.1. Manifest Timestamp and Media Timestamp

A distinction is made between Manifest Timestamps and Media Timestamps in the definitions below. Media Timestamps MUST represent Presentation timestamps as they are given by the media data itself. Manifest Timestamps MUST define those same Media Timestamps, but with an additional offset such that the timestamps of all Tracks of a Presentation are aligned. This offset MUST be given by the Manifest for each Track or Switching Set.¶

For example, consider the first Segment of a Track with Media Timestamps starting at 0 seconds. This Track is part of the second Presentation of an HESP stream. The first Presentation has run for a significant amount of time and the second Presentation follows immediately after the first Presentation ends. In this case, the Manifest Timestamps of the second Presentation must succeed the Manifest Timestamps of the first Presentation. As a result, the Media Timestamps and Manifest Timestamps of the second Presentation now differ. An offset must be given by the Manifest in order to inform clients about this difference between timestamp types for each Track of the second Presentation.¶

An example of this distinction is given in Appendix A.1.2.¶

3.1.2. Sequence Numbers

Each Initialization Packet belonging to a Track is given a Sequence Number. This is a simple identifier used to retrieve specific Initialization Packets by the client. The Sequence Number is a positive integer that MUST be increased by 1 for each subsequent Initialization Packet of the same Track.¶

3.1.3. Calculating the Sequence Number of an Initialization Packet

To calculate the Sequence Number of a chosen Initialization Packet, the Manifest provides the client multiple values. Each active Presentation MUST contain both its current Manifest Timestamp and, for each Track, the Sequence Number of the latest Initialization Packet and the (constant) frame rate.¶

It is given that:¶

• the frame rate MUST be constant for each Track.¶
• Initialization Packets MUST become available in real-time.¶
• Sequence Numbers MUST be increased by 1 per Initialization Packet.¶
• the Sequence Number and Manifest Timestamp of the latest Initialization Packet for a Track are given by the Manifest.¶

As a result, it is possible to derive Sequence Numbers for arbitrary Initialization Packets. By taking the latest Sequence Number of a Track, its associated Manifest Timestamp, and the frame rate, it is possible to calculate the number of frames between the given Initialization Packet and the required Initialization Packet in the past or future. This amount can then be subtracted for that latest Sequence Number in order to derive the Sequence Number of the desired Initialization Packet.¶

For example, suppose the latest Sequence Number would be 103 at Manifest Timestamp 00:00:04.120 for a Track with a frame rate of 25 fps. To find the Sequence Number at Manifest Timestamp 00:00:01.360, the difference between both frames should be calculated. This is 2.760 seconds, or 2.760 * 25 = 69 frames. This means that the Sequence Number for the provided Manifest Timestamp is 103 - 69 = 34.¶

3.2.1. Additional JSON data types

YYYY-MM-DDThh:mm:ss.mmmTZD

where YYYY = four-digit year
      MM = two-digit month
      DD = two-digit day of the month
      hh = two digits of the hour (00 through 23)
      mm = two digits of a minute (00 through 59)
      ss = two digits of a second (00 through 59)
      mmm = three digits of a millisecond (000 through 999)
      TZD = time zone designator (Z or +hh:mm or -hh:mm)

3.2.2. ManifestType

This data type represents the root of the Manifest. The structure consists of general playback information and a collection of Presentations.¶

The table below gives the possible fields. The "Required?" column indicates if a field is REQUIRED (Y) or OPTIONAL (N).¶

3.2.3. TimeSource

A TimeSource is used to sync the packager and player internal clocks to make requests for data only once it is available.¶

3.2.4. ScaledValue

In order to avoid rounding issues introduced through floating-point numbers, this structure defines two integers. The total value is calculated by dividing value over scale.¶

3.2.5. PresentationType

This is the type definition of a Presentation.¶

3.2.6. TimeBounds

A TimeBounds structure denotes a time interval with a start and end time. A start or end time may be undefined. The desired behavior in such a situation depends on the specific usage of the structure.¶

These boundaries are inclusive at the start time and exclusive at the end time. I.e., if two TimeBounds need to be continuous, then the end time of the first one must equal the start time of the second.¶

3.2.7. AudioSwitchingSetType

This is the type definition of an audio Switching Set.¶

3.2.8. SwitchingSetProtection

The following fields are defined on the SwitchingSetProtection structure:¶

More information on content protection can be found in Section 7.¶

3.2.9. SwitchingSetProtectionSystem

The following fields are defined on the SwitchingSetProtectionSystem structure:¶

Additional attributes MAY be defined by specific protection systems such as license acquisition URLs, authorization-related URLs, specific initialization data (a default key ID),... These attributes SHOULD be defined on this SwitchingSetProtectionSystem structure; however, they will not be defined here. Depending on the scheme ID that is given here, a client SHOULD verify that these additional attributes exist.¶

3.2.10. VideoSwitchingSetType

This is the type definition of a video Switching Set.¶

3.2.11. MetadataSwitchingSetType

This is the type definition of a metadata Switching Set.¶

3.2.12. AudioTrackType

This is the type definition of an audio Track.¶

For any attributes that also exist on AudioSwitchingSetType, if both AudioSwitchingSetType and AudioTrackType have a value for this attribute, then only the value set by AudioTrackType must be considered (except for the identifier and label.)¶

3.2.13. VideoTrackType

This is the type definition of a video Track.¶

For any attributes that also exist on VideoSwitchingSetType, if both VideoSwitchingSetType and VideoTrackType have a value for this attribute, then only the value set by VideoTrackType must be considered (except for the identifier and label.)¶

3.2.14. Resolution

A Resolution contains the following elements:¶

The display aspect ratio belonging to this Resolution can be calculated by using this sample aspect ratio width and height, and the picture width and height as follows:¶

dar = (darWidth, darHeight)
darWidth = sarWidth * width
darHeight = sarHeight * height

3.2.15. MetadataTrackType

This is the type definition of a metadata Track.¶

3.2.16. SegmentType

This is the type definition of a Segment.¶

3.2.17. PresentationEventType

Timed metadata events can be embedded in the Manifest by using Presentation Events. Each PresentationEventType contains the following elements:¶

3.2.18. PresentationEventTimeBounds

The time boundaries of a PresentationEventType are defined as follows:¶

3.3.1. Manifest responses

If an error occurs, the player SHOULD attempt to retry the request. In case of consecutive unsuccessful requests, the media player SHOULD assume the content is unavailable and cease playback.¶

3.4.1. Content request URL resolution

A content request URL for a specific Track SHALL be constructed by applying relative resolution (as explained in Section 5.2 of RFC 3986 [RFC3986]) to each defined baseUrl or contentBaseUrl attribute relating to that Track, starting from the root of the Manifest. The URL of the Manifest SHALL be used as a base for the first resolution, after which the target URL of the previous resolution SHALL be used as the base URL of the next resolution.¶

This means that the content request URL for a Track's Initialization Stream is constructed as follows (in pseudocode):¶

T = manifestUrl
if isDefined(Manifest.contentBaseUrl):
      T = resolve(Manifest.contentBaseUrl, T)
if isDefined(Presentation.baseUrl):
      T = resolve(Presentation.baseUrl, T)
if isDefined(SwitchingSet.baseUrl):
      T = resolve(SwitchingSet.baseUrl, T)
if isDefined(Track.baseUrl):
      T = resolve(Track.baseUrl, T)
contentRequestURL = resolve(initializationPattern, T)

where initializationPattern is the attribute given by either SwitchingSet or Track (with Track taking precedence if given by both) and resolve(R, B) is relative resolution, where R is a URI reference and B is a base URI.¶

The content request URL for the Track's Continuation Stream can be constructed in the same manner if the initializationPattern is replaced with the continuationPattern. Both URLs MUST be unique for each unique Track.¶

An example of content addressing is given in Appendix A.1.3.¶

3.4.2. Requesting using an identifier and the content request URL

The content request URL MUST include the {initId} pattern for Initialization Stream URLs and the {segmentId} pattern for Continuation Stream URLs. These patterns are replaced in the actual content requests: {initId} is replaced with the requested Sequence Number of the Initialization Stream, and {segmentId} is replaced with the requested segment identifier of the Continuation Stream.¶

In order to add leading zeros to the identifier, the following can be added to the patterns initId and segmentId: :0(n)d where (n) is the minimal amount of characters the resulting identifier has (if the identifier is already longer, it will not be altered.)¶

For example, for a continuation segment with identifier 100, a following content request URL https://www.example.com/s-1/content-{segmentId:06d}.mp4 will resolve to https://www.example.com/s-1/content-000100.mp4. While the content request URL https://www.example.com/s-1/content-{segmentId:02d}.mp4 resolves to https://www.example.com/s-1/content-100.mp4¶

Further details about these requests are given in Section 4 and Section 5.¶

4.2.1. Video Initialization Packet

A video Initialization Packet MUST contain:¶

• A CMAF header, i.e., all information required to initialize the media decoder, stored in ISO Base Media File Format [ISOBMFF] boxes as specified by CMAF [CMAF]. The minimal required contents are defined in Section 4.2.3¶

  • An additional in-band metadata event containing extra HESP information, included in the aforementioned CMAF segment. The definition of such an event is denoted in Section 4.2.4.¶

• A CMAF segment, i.e., an independent media sample and timing information related to this sample, stored in ISOBMFF [ISOBMFF] boxes as specified by CMAF [CMAF].¶

4.2.2. Audio Initialization Packet

An audio Initialization Packet MUST contain:¶

• A CMAF header, i.e., all information required to initialize the media decoder, stored in ISO Base Media File Format [ISOBMFF] boxes as specified by CMAF [CMAF]. The minimal required contents are defined below.¶

• An additional in-band metadata event containing extra HESP information, included in the aforementioned CMAF segment. The definition of such an event is denoted below.¶

It MAY also contain an independent (audio) media sample, as is defined by the video Initialization Packet. However, it is heavily discouraged as it leads to more storage costs for the provider without any significant advantage.¶

4.2.3. CMAF header

A CMAF header is defined in ISO/IEC 23000-19:2020 [CMAF] as: "a sequence of CMAF constrained ISOBMFF boxes that do not reference any media samples (3.3.15), but are associated with a CMAF track (3.2.1) and necessary for the decoding of its CMAF fragments (3.1.1)." As such, the header MUST NOT contain any samples.¶

This means that for the ISO Base Media File Format, at least the following boxes MUST be given:¶

• An ftyp box.¶

• A moov box.¶

• One or more tracks and accompanying boxes, though stbl and other related boxes cannot give information about any actual sample entries.¶

4.2.4. Event message information

In order to pass information about the Continuation Stream, an in-band event in the form of an emsg box must be added to the Initialization Packet. The format of this box is defined at Section 6.2.1.1. The message_data field MUST contain the following two values (see Table 31):¶

• the Continuation Segment index (index), which defines the index of the Continuation Segment containing the next media sample.¶

• the Continuation Segment byte offset (offset), which defines the position of the next media sample in the Continuation Segment. This field is OPTIONAL if the next media sample is located at the start of the segment, in which case a byte-range request is not needed.¶

The message_data field of the emsg box could for example look like this:¶

{"index":1,"offset":1234}¶

4.3.1. Initialization Stream requests

Initialization Packets MUST be available through a basic GET request. They MUST be able to be retrieved through two methods:¶

4.3.2. Initialization Stream responses

5.1.1. Media content

The Continuation Stream is packaged as a regular CMAF stream, albeit with some encoding constraints, depending on the chosen profile (see Appendix C.)¶

Maximum Gain Profile¶

• Samples MUST only contain references to the one sample preceding it.¶
• Each CMAF chunk MUST contain at most one sample (for the lowest possible latency.)¶
• Long CMAF Segments (values of multiple minutes are possible.)¶

Compatibility Profile¶

• Regular sized CMAF Segments (recommendation: between 1 and 30 seconds)¶
• Chunk sizes range between one sample and one sub-GOP¶
• The following GOP structure MUST be followed: I B ... B P B ... B P B ... B P¶
• The number of B frames MAY vary from 0 (same behavior as Maximum Gain Profile) to 4 (recommended maximum)¶
• Each sub-GOP (B ... B P) MUST only reference one previous frame (allowing injection of keyframes)¶

5.3.1. Continuation Stream URLs

Section 3.4 defines how to create the correct URLs and how to request a continuation segment. The request content URL is used together with the desired segment identifier to retrieve the Segment.¶

5.3.2. Continuation Stream requests

Suppose a client needs to request a partial Continuation Segment, for example, starting at the byte offset given by metadata of an Initialization Packet. In that case, it MUST use one or more HTTP Range [RFC7233] Requests. The start and end of the range of each request MUST be defined.¶

Often, the total length of the Continuation Segment will not yet be known at the time of the request. If a client cannot define an accurate end value for the HTTP Range of a request, then 2^53 - 1 (9007199254740991) SHOULD be used.¶

5.3.3. Continuation Stream responses

A distinction is made between how responses should be returned, depending on the version of the HTTP protocol used.¶

6.2.1. In-band events

In order to deliver events in-band, root-level Event Message (emsg) boxes MUST be added to ongoing media Continuation Streams. The definition of an emsg box can be found in the CMAF [CMAF] specification.¶

These boxes should be appended to each Track of a Presentation to ensure all viewers can receive such an event. It is recommended to use out-of-band events if the included data is significantly large.¶

The HESP specification defines a few in-band events that it leverages for client initialization and Manifest updates. The structure of these events is given below.¶

6.2.2. Out-of-band events

Out-of-band events can be added to the Manifest through Presentation Events. The definition of a Presentation Event is given in Section 3.2.17. The contents of the event data are not predefined. It is possible to include arbitrary data as Base64 encoded data, and plaintext data can be included as-is. If needed, the data can be an URL that needs to be resolved separately, but this is up to the publisher of this data.¶

7.3.1. Initialization Stream

Initialization Packets MUST contain the following boxes:¶

• TrackEncryptionBox (tenc): This box contains default parameters regarding sample encryption.¶
• SchemeTypeBox (schm): This box identifies the protection scheme. scheme_type MUST be set to cbcs or cenc.¶
• ProtectionSystemSpecificHeaderBox (pssh): If the Manifest does not contain a pssh box that applies to this Track, then this box MUST be included in each Initialization Packet of each Track of the protected Switching Set. If the Manifest does contain a pssh box that applies to this Track, then this box MUST be disregarded.¶

In order to signal that the Track is encrypted, the stream type MUST equal encv for video Tracks and enca for audio Tracks.¶

Additionally, video Initialization Packets contain a sample. If that sample is encrypted, then the same requirements for CMAF fragments of a Continuation Segment also apply here: a senc box MUST be included, pssh, saiz and saio boxes MAY be included. The following section contains more information about these requirements.¶

7.3.2. Continuation Stream

CMAF fragments of a Continuation Segment MUST contain the following box:¶

• SampleEncryptionBox (senc): This box is used to store initialization vector data and information on subsample encryption. As each chunk in HESP must currently contain at most one sample, sample_count SHALL always 1.¶

The following boxes MAY be included:¶

• ProtectionSystemSpecificHeaderBox (pssh): If any updates need to be made to the underlying licensing system, a pssh box MAY be included.¶
• SampleAuxiliaryInformationSizesBox (saiz): This box is used to store the size of per-sample auxiliary information. It is only REQUIRED if such per-sample information exists.¶
• SampleAuxiliaryInformationOffsetsBox (saio): This box is used to store the offsets of per-sample auxiliary information. It is only REQUIRED if such per-sample information exists.¶

A.1.1. Retrieving the Manifest

The URL of the Manifest is given out of band to the client. The client sends out a GET request. In this case, let's suppose the Manifest is available at https://example.com/stream1/manifest.json. The client then makes such a request:¶

GET /stream1/manifest.json HTTP/1.1
Host: example.com
Accept: application/vnd.theo.hesp+json

The server responds with the following headers:¶

HTTP/1.1 200 OK
Content-Type: application/vnd.theo.hesp+json; charset=utf-8
Content-Length: 6867
Date: Wed, 31 Mar 2021 08:00:00 GMT

and the following body:¶

{
   "activePresentation":"1",
   "availabilityDuration":{
      "value":1500
   },
   "creationDate":"2021-03-31T08:00:00.000Z",
   "fallbackPollRate":300,
   "manifestVersion":"1.1.0",
   "streamType":"live",
   "presentations":[
      {
         "id":"0",
         "timeBounds":{
            "startTime":0,
            "endTime":972000000,
            "scale":90000
         },
         "audio":[
            {
               "id":"main-audio",
               "language":"eng",
               "baseUrl":"audio/",
               "channels":2,
               "codecs":"mp4a.40.2",
               "continuationPattern":"content-{segmentId}.mp4",
               "initializationPattern":"init-{initId}.mp4",
               "sampleRate":48000,
               "tracks":[
                  {
                     "id":"96kbps",
                     "activeSegment":1799,
                     "activeSequenceNumber":506249,
                     "averageBandwidth":96000,
                     "bandwidth":96000,
                     "baseUrl":"96k/",
                     "segmentDuration":{
                        "value":540000,
                        "scale":90000
                     },
                     "segments":[
                        {
                           "id":1799,
                           "timeBounds":{
                              "startTime":971460000,
                              "scale":90000
                           }
                        }
                     ]
                  }
               ]
            }
         ],
         "video":[
            {
               "id":"main-video",
               "baseUrl":"video/",
               "frameRate":{
                  "value":25
               },
               "continuationPattern":"content-{segmentId}.mp4",
               "initializationPattern":"init-{initId}.mp4",
               "tracks":[
                  {
                     "id":"720p",
                     "activeSegment":1799,
                     "activeSequenceNumber":269999,
                     "bandwidth":3000000,
                     "baseUrl":"720p/",
                     "codecs":"avc1.4d001f",
                     "resolution":{
                        "width":1280,
                        "height":720
                     },
                     "segmentDuration":{
                        "value":540000,
                        "scale":90000
                     },
                     "segments":[
                        {
                           "id":1799,
                           "timeBounds":{
                              "startTime":971460000,
                              "scale":90000
                           }
                        }
                     ]
                  }
               ]
            }
         ]
      },
      {
         "id":"1",
         "timeBounds":{
            "startTime":972000000,
            "scale":90000
         },
         "currentTime":{
            "value":1080000000,
            "scale":90000
         },
         "baseUrl":"https://otherexample.com/s2/",
         "audio":[
            {
               "id":"main-audio",
               "language":"eng",
               "baseUrl":"audio/",
               "channels":2,
               "codecs":"mp4a.40.2",
               "sampleRate":48000,
               "mediaTimeOffset":{
                  "value":-972000000,
                  "scale":90000
               },
               "tracks":[
                  {
                     "id":"128kbps",
                     "activeSegment":200,
                     "activeSequenceNumber":56250,
                     "averageBandwidth":128000,
                     "bandwidth":128000,
                     "continuationPattern":
                        "128k-content-{segmentId}.mp4",
                     "initializationPattern":
                        "128k-init-{initId}.mp4",
                     "segmentDuration":{
                        "value":540000,
                        "scale":90000
                     },
                     "segments":[
                        {
                           "id":200,
                           "timeBounds":{
                              "startTime":1080000000,
                              "scale":90000
                           }
                        }
                     ]
                  }
               ]
            }
         ],
         "video":[
            {
               "id":"main-video",
               "baseUrl":"video/",
               "frameRate":{
                  "value":25
               },
               "tracks":[
                  {
                     "id":"720p",
                     "activeSegment":200,
                     "activeSequenceNumber":29999,
                     "bandwidth":3000000,
                     "codecs":"avc1.4d001f",
                     "continuationPattern":
                        "720p-content-{segmentId}.mp4",
                     "initializationPattern":
                        "720p-init-{initId}.mp4",
                     "resolution":{
                        "width":1280,
                        "height":720
                     },
                     "segmentDuration":{
                        "value":540000,
                        "scale":90000
                     },
                     "segments":[
                        {
                           "id":200,
                           "timeBounds":{
                              "startTime":1080000000,
                              "scale":90000
                           }
                        }
                     ]
                  },
                  {
                     "id":"1080p",
                     "activeSegment":200,
                     "activeSequenceNumber":29999,
                     "bandwidth":5000000,
                     "codecs":"avc1.4d001f",
                     "continuationPattern":
                        "1080p-content-{segmentId}.mp4",
                     "initializationPattern":
                        "1080p-init-{initId}.mp4",
                     "resolution":{
                        "width":1920,
                        "height":1080
                     },
                     "segmentDuration":{
                        "value":540000,
                        "scale":90000
                     },
                     "segments":[
                        {
                           "id":200,
                           "timeBounds":{
                              "startTime":1080000000,
                              "scale":90000
                           }
                        }
                     ]
                  }
               ]
            }
         ]
      }
   ]
}

A.1.2. Timing information

In the above Manifest, we have two Presentations:¶

• The first Presentation with ID "0" is not active at the current time, though it was active previously. It contains one audio Track and one video Track. It is 3 hours long in total and starts at 00:00:00.000 (Manifest Timestamp.) No Track has a mediaTimeOffset defined, so the Manifest Timestamps match the Media Timestamps.¶
• The second Presentation with ID "1" is active at the current time. It contains one audio Track and two video Tracks. Its total duration is yet to be defined. Its start time is 03:00:00.000 in Manifest Time and its current time (at the moment the Manifest was retrieved) is 03:20:00.000. All Tracks have the same mediaTimeOffset defined. The Media Timestamps for this Presentation start at 00:00:00.000. That also means that for all the currently active Segments, the media data will contain a starting timestamp of 00:20:00.000.¶

The availabilityDuration of the Manifest is 25 minutes. As only 20 minutes of the second Presentation have elapsed, some Segments of the first Presentation are still available. Therefore, the first Presentation must still be included in the Manifest. Once 5 minutes have passed, the first Presentation can be left out of the Manifest.¶

A.1.3. Content addressing

The client uses the Manifest to derive the content request URLs of each Track.¶

For the audio Track of the first Presentation, the following parts are found:¶

AudioSwitchingSet.baseUrl = "audio/"
AudioTrack.baseUrl = "96k/"
AudioSwitchingSet.initializationPattern = "init-{initId}.mp4"
AudioSwitchingSet.continuationPattern = "content-{segmentId}.mp4"

The Manifest URL (https://example.com/stream1/manifest.json) is used as the base for resolution, and then relative resolution is applied to each of the parts above.¶

T = resolve(AudioSwitchingSet.baseUrl, manifestUrl)
  = "https://example.com/stream1/audio/"
T = resolve(AudioTrack.baseUrl, T)
  = "https://example.com/stream1/audio/96k/"
initBaseUrl = resolve(AudioSwitchingSet.initializationPattern, T)
  = "https://example.com/stream1/audio/96k/init-{initId}.mp4"
contBaseUrl = resolve(AudioSwitchingSet.continuationPattern, T)
  = "https://example.com/stream1/audio/96k/content-{segmentId}.mp4"

The final content request URLs are https://example.com/stream1/audio/96k/init-{initId}.mp4 and https://example.com/stream1/audio/96k/content-{segmentId}.mp4 for the Initialization Stream and Continuation Stream respectively.¶

For the audio Track of the second Presentation, the following parts are found:¶

Presentation.baseUrl = "https://otherexample.com/s2/"
AudioSwitchingSet.baseUrl = "audio/"
AudioTrack.initializationPattern = "128k-init-{initId}.mp4"
AudioTrack.continuationPattern = "128k-content-{segmentId}.mp4"

The Manifest URL (https://example.com/stream1/manifest.json) is used as the base for resolution, and then relative resolution is applied to each of the parts above.¶

T = resolve(Presentation.baseUrl, manifestUrl)
  = "https://otherexample.com/s2/"
T = resolve(AudioSwitchingSet.baseUrl, T)
  = "https://otherexample.com/s2/audio/"
initBaseUrl = resolve(AudioTrack.initializationPattern, T)
  = "https://otherexample.com/s2/audio/128k-init-{initId}.mp4"
contBaseUrl = resolve(AudioTrack.continuationPattern, T)
  = "https://otherexample.com/s2/audio/128k-content-{segmentId}.mp4"

The final content request URLs are https://otherexample.com/s2/audio/128k-init-{initId}.mp4 and https://otherexample.com/s2/audio/128k-content-{segmentId}.mp4 for the Initialization Stream and Continuation Stream respectively.¶

A.2.1. Retrieving Initialization Packets

The client decides to retrieve the only audio Track of the active Presentation. The URL pattern for the Initialization Stream of this Track is https://otherexample.com/s2/audio/128k-init-{initId}.mp4.¶

The client opts to retrieve the most recent Initialization Packet and sends out a request:¶

GET /s2/128k-init-now.mp4 HTTP/1.1
Host: otherexample.com
Accept: */*

The server responds with the following headers¶

HTTP/1.1 200 OK
Content-Type: audio/mp4
Content-Length: 742
Date: Wed, 31 Mar 2021 08:00:02 GMT

and a binary body containing the ISOBMFF media data of the audio Initialization Packet.¶

The client repeats this to retrieve one of the video Tracks.¶

A.2.2. Parsing offset information

The client parses the ISOBMFF boxes in the audio Initialization Packet. Once the emsg box with scheme ID urn:theo:hesp:2020 and value initdata is found, its message_data field is parsed.¶

In this case, message_data equals {"index":200,"offset":63275}. The following media data for the chosen audio Track is available in the Continuation Segment with ID 200 and at a byte offset of 63275.¶

A.3.1. Retrieving Continuation Segments

The client retrieves the Continuation Segment of the chosen audio Track. The URL pattern for the Continuation Stream of this Track is https://otherexample.com/s2/audio/128k-content-{segmentId}.mp4.¶

The client sends out a request for the segment with ID 200, starting at offset 63275:¶

GET /s2/128k-content-200.mp4 HTTP/1.1
Host: otherexample.com
Accept: */*
Range: bytes=63275-9007199254740991

As discussed in Section 5.3.2, a sufficiently large end byte value is chosen to ensure the entire range is retrieved.¶

The server responds with the following headers¶

HTTP/1.1 200 OK
Content-Type: audio/mp4
Content-Range: bytes 63275-9007199254740991/9007199254677716
Transfer-Encoding: chunked
Date: Wed, 31 Mar 2021 08:00:03 GMT

and a (chunked) binary body containing the ISOBMFF media data of the Continuation Segment.¶

The client repeats this for the chosen video Track and uses the retrieved media data to reach playback. The Manifest that was retrieved previously contains sufficient information to retrieve new Continuation Segments from this point on.¶

C.2.1. Example

Figure 5 depicts a part of H.264 output of 7 frames, sorted in decode order. The dependencies of each frame are shown with arrows.¶

            ______________________________________
            |                     |       |       |
            V                     |       |       |
+----+  +----+  +----+  +----+  +----+  +----+  +----+
|####|  |%%%%|  |%%%%|  |%%%%|  |@@@@|  |@@@@|  |@@@@|
| I1 |  | P4 |  | B2 |  | B3 |  | P7 |  | B5 |  | B6 |
+----+  +----+  +----+  +----+  +----+  +----+  +----+
    ^     |  ^     |       |        ^      |       |
    |_____|  |     |       |        |______|       |
    |        |     |       |        |              |
    |________|_____|       |        |______________|
    |                      |
    |______________________|