Source-Specific Media Format Parameters for H.264 and H.264 SVC
draft-lennox-avt-h264-source-fmtp-00

Status of this Memo

By submitting this Internet-Draft, each author represents that any applicable patent or other IPR claims of which he or she is aware have been or will be disclosed, and any of which he or she becomes aware will be disclosed, in accordance with Section 6 of BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as “work in progress.”

The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt.

The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html.

This Internet-Draft will expire on May 15, 2008.

Abstract

When used in the Session Description Protocol Offer/Answer model, several of the media format parameters for the H.264 video format, and for its Scalabile Video Codec (SVC) extension, describe characterics of the stream an endpoint is prepared to send, not of streams it is prepared to receive. If an endpoint wishes to send multiple streams, these parameters may be incompatible. This document defines how such media format parameters may be given on a per-source basis, using SDP source-specific fmtp attributes.

Table of Contents

1.  Introduction
2.  Terminology
3.  Overview
4.  Mapping of MIME parameters to SDP source attributes
    4.1.  Parameter Sets
    4.2.  Packetization Mode 2 Parameters
    4.3.  Capability Parameters
    4.4.  H264 SVC Parameters
    4.5.  Other Parameters
5.  Examples
6.  Backward Compatibility
7.  Security Considerations
8.  IANA Considerations
9.  Normative References
§  Author's Address
§  Intellectual Property and Copyright Statements

1.  Introduction

Unlike many media packetization formats for the Real-Time Transport Protocol (RTP) (Schulzrinne, H., Casner, S., Frederick, R., and V. Jacobson, “RTP: A Transport Protocol for Real-Time Applications,” July 2003.) [RFC3550], the the RTP packetization specifications for H.264 (Wenger, S., Hannuksela, M., Stockhammer, T., Westerlund, M., and D. Singer, “RTP Payload Format for H.264 Video,” February 2005.) [RFC3984] and for H.264's Scalable Video Coding extension (Wenger, S., Wang, Y., Schierl, T., and A. Eleftheriadis, “RTP Payload Format for SVC Video,” April 2010.) [I‑D.ietf‑avt‑rtp‑svc] define a number of MIME media type parameters that, when encoded in SDP (Handley, M., Jacobson, V., and C. Perkins, “SDP: Session Description Protocol,” July 2006.) [RFC4566], define characteristics of the media stream an endpoint is prepared to send, not of the streams it is prepared to receive. Most notably, streams' parameter sets (initial data required to correctly initialize a decoder) are encoded in SDP messages and sent out-of-band, to ensure that they are reliably received by a decoder before decoding begins.

Because RTP is fundamentally a group communication protocol, however, an RTP session may contain many different media streams. In this case, different H.264 and H.264 SVC streams in an RTP session may need to be described by different and incompatible values for these MIME media type parameters. For example, an endpoint may be switching between video streams encoded by separate video encoders. In this case, it is not possible, using only the mechanisms of [RFC3984] (Wenger, S., Hannuksela, M., Stockhammer, T., Westerlund, M., and D. Singer, “RTP Payload Format for H.264 Video,” February 2005.), to describe all the sources and to send their parameter sets out-of-band. An endpoint must instead fall back to sending parameter sets in-band, and retransmitting them with high enough frequency that there is a reasonably high likelihood of their being receceived successfully.

To solve this difficulty, this document uses the framework introduced by [I‑D.lennox‑mmusic‑sdp‑source‑attributes] (Lennox, J., “Source-Specific Media Attributes in the Session Description Protocol (SDP),” July 2007.) to describe MIME media format parameters of individual H.264 sources in SDP. This enables all the benefits of out-of-band H.264 source description in the case when multiple H.264 sources will be sent in an RTP session.

2.  Terminology

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 (Bradner, S., “Key words for use in RFCs to Indicate Requirement Levels,” March 1997.) [RFC2119] and indicate requirement levels for compliant implementations.

3.  Overview

When used with the SDP (Handley, M., Jacobson, V., and C. Perkins, “SDP: Session Description Protocol,” July 2006.) [RFC4566] offer/answer model (Rosenberg, J. and H. Schulzrinne, “An Offer/Answer Model with Session Description Protocol (SDP),” June 2002.) [RFC3264], several of the media format parameters of H.264 (Wenger, S., Hannuksela, M., Stockhammer, T., Westerlund, M., and D. Singer, “RTP Payload Format for H.264 Video,” February 2005.) [RFC3984] and H.264 SVC (Wenger, S., Wang, Y., Schierl, T., and A. Eleftheriadis, “RTP Payload Format for SVC Video,” April 2010.) [I‑D.ietf‑avt‑rtp‑svc] define characteristics of an RTP stream (media source) to be sent, not of a session receiver's capabilities. When multiple media sources are in use, it is sometimes useful to describe source characteristics individually for each source.

Per-source media format parameters are defined using the fmtp source parameter (Lennox, J., “Source-Specific Media Attributes in the Session Description Protocol (SDP),” July 2007.) [I‑D.lennox‑mmusic‑sdp‑source‑attributes]. This document updates the MIME type registration of video/H264 in [RFC3984] (Wenger, S., Hannuksela, M., Stockhammer, T., Westerlund, M., and D. Singer, “RTP Payload Format for H.264 Video,” February 2005.) and of video/H264-SVC in [I‑D.ietf‑avt‑rtp‑svc] (Wenger, S., Wang, Y., Schierl, T., and A. Eleftheriadis, “RTP Payload Format for SVC Video,” April 2010.) to specify the encoding of the MIME media format parameters into the fmtp source attribute.

4.  Mapping of MIME parameters to SDP source attributes

H.264 MIME media format parmaeters applicable to a specific source are are encoded into an "fmtp" source attribute for the H.264 payload type and the source being described. These parameters are expressed in the form of a semicolon-separated list of parameter=value pairs, the same syntax as the media-level fmtp value. For the purposes of discussion in this document, MIME media format parameters encoded into a source-specific fmtp attribute are called "source-specific parameters", while MIME media format parameters encoded into a media-level format attribute are called "media parameters".

Source-specific parameters only describe characteristics of a specific source might send, while media parameters describe all sources of a stream. Source-specific parameters do not override media parameters, though they do in some cases further constrain them or provide additional information.

4.1.  Parameter Sets

The "sprop-parameter-sets" parameter encodes H.264 sequence parameter set (SPS) and picture parameter set (PPS) Network Adaptation Layer NAL) units. These parameter sets provide essential information necessary to decode an H.264 bitstream; encoding them in SDP ensures that they are delivered reliably.

The "sprop-parameter-sets" parameter MAY be encoded as a source parameter. However, if the sprop-parameter-sets parameter is also present as a media parameter, the H.264 parameter sets described for each source MUST include all the H.264 parameter set described for the media.

In an SDP answer, the "sprop-parameter-sets" for a source MUST follow the same constraints as for the media. I.e., the parameter sets for a source described in an answer MUST be a superset of the parameter sets for the media in the offer, and if an offer indicates "parameter-add=0" (false) for the media, the corresponding answer MUST NOT add additonal parameter sets for any source.

4.2.  Packetization Mode 2 Parameters

The "sprop-deint-buf-req", "sprop-interleaving-depth", "sprop-max-don-diff", and "sprop-init-buf-time" parameters describe characteristics of H.264 media streams using packetization mode 2 (interleaved mode). According to [RFC3984] (Wenger, S., Hannuksela, M., Stockhammer, T., Westerlund, M., and D. Singer, “RTP Payload Format for H.264 Video,” February 2005.), the first two are mandatory for packetization mode 2 streams; the other two are optional. They specify the maximum size and time of the debuffering needed to deinterleave streams sent in interleaved mode.

These parameters MAY be included as source parameters, overriding any corresponding values at the media level for the source. However, if they are, they MUST be less than or equal to the value specified for the parameter (if any) at the media level. All constraints for these values specified by [RFC3984] (Wenger, S., Hannuksela, M., Stockhammer, T., Westerlund, M., and D. Singer, “RTP Payload Format for H.264 Video,” February 2005.) still apply.

4.3.  Capability Parameters

As defined in [RFC3984] (Wenger, S., Hannuksela, M., Stockhammer, T., Westerlund, M., and D. Singer, “RTP Payload Format for H.264 Video,” February 2005.), the meaning of the capability parameters ("max-mbps", "max-fs", "max-cpb", "max-dpb", "max-br", "redundant-pic-cap", and "max-rcmd-nalu-size") at the media level depends on a media stream's "direction" attribute. When the "direction" attribute is "sendonly", then the parameters describe the limits of media sources that the sender is capable of producing. When the "direction" attribute is "sendrecv" or "recvonly", then the parameters describe the limitations of what the receiver accepts.

When encoded as source parameters, these parameters always describe the limits of the source being described. In media streams whose "direction" is "sendonly", these parameters MUST be less than or equal to the values (if any) in the media parameters. In media streams whose "direction" is "sendrecv", source parameters for these values are unconstrained by stream's media parameters (which describe what the endpoint is willing to receive). However, in offer/answer mode, the values of these source parameters MUST be less than or equal to the values given in media parameters in the most recent (accepted) offer or answer for the stream.

(Media streams whose "direction" is "recvonly" do not encode any sources.)

4.4.  H264 SVC Parameters

The H.264 SVC parameters "sprop-scalability-info" and "sprop-layer-ids" are defined in [I‑D.ietf‑avt‑rtp‑svc] (Wenger, S., Wang, Y., Schierl, T., and A. Eleftheriadis, “RTP Payload Format for SVC Video,” April 2010.). They describe the scalability structure of an H.264 Scalable Video Codec stream.

These parameters MUST NOT be given both as source parameters and media parameters. They MUST be specified at one level at most.

4.5.  Other Parameters

The parameters "profile-level-id", "packetization-mode", and "parameter-add" MUST NOT be used as a source parameters.

5.  Examples

This section gives examples of SDP descriptions of media sessions containing H.264 source parameters. For brevity, only the media sections of the descriptions are given.

m=video 49170 RTP/AVP 96
a=rtpmap:96 H264/90000
a=fmtp:96 packetization-mode=1
a=ssrc:12345 cname:cif-stream@example.com
a=ssrc:12345 fmtp:96 sprop-parameter-sets=J0LgDZWWFglk,KM4Ecg==
a=ssrc:67890 cname:vga-stream@example.com
a=ssrc:67890 fmtp:96 sprop-parameter-sets=J0LgDZWWCgPZ,KM4Ecg==

The example in Figure 1 (Example: declaration of two sources with different parameter sets) shows two H.264 streams with different, incompatbile parameter sets. (Specifically, the two streams encode different image sizes.)

(TODO: add more examples.)

6.  Backward Compatibility

Unless a sender knows via some mechanism (not specified here or in [I‑D.lennox‑mmusic‑sdp‑source‑attributes] (Lennox, J., “Source-Specific Media Attributes in the Session Description Protocol (SDP),” July 2007.)) that a receiver definitely understands source parameters, it MUST send any parameter sets specified in sprop-parameter-sets source attributes in-band in the media stream as well. These parameter sets SHOULD be transmitted frequently enough that a receiver has a high probability of receiving them even in the presence of packet loss.

For H.264 SVC streams, the scalability information Supplemental Encoder Information (SEI) message encoded in an sprop-scalability-info parameter set SHOULD be similarly transmitted in-band as well.

7.  Security Considerations

Source-specific encoding of media format parameters does not add any additional security considerations beyond those of [RFC3984] (Wenger, S., Hannuksela, M., Stockhammer, T., Westerlund, M., and D. Singer, “RTP Payload Format for H.264 Video,” February 2005.) and [I‑D.ietf‑avt‑rtp‑svc] (Wenger, S., Wang, Y., Schierl, T., and A. Eleftheriadis, “RTP Payload Format for SVC Video,” April 2010.).

8.  IANA Considerations

This document updates the SDP encoding of the video/H264 MIME media type specified in [RFC3984] (Wenger, S., Hannuksela, M., Stockhammer, T., Westerlund, M., and D. Singer, “RTP Payload Format for H.264 Video,” February 2005.), and of the video/H264-SVC MIME media type specified in [I‑D.ietf‑avt‑rtp‑svc] (Wenger, S., Wang, Y., Schierl, T., and A. Eleftheriadis, “RTP Payload Format for SVC Video,” April 2010.).

9. Normative References

Author's Address

Full Copyright Statement

Copyright © The IETF Trust (2007).

This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights.

This document and the information contained herein are provided on an “AS IS” basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Intellectual Property

The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79.

Copies of IPR disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr.

The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at ietf-ipr@ietf.org.