| Network Working Group | J. Spittka |
| Internet-Draft | K. Vos |
| Intended status: Informational | Skype Technologies S.A. |
| Expires: January 02, 2012 | JM. Valin |
| Octasic Inc. | |
| July 2011 |
RTP Payload Format and File Storage Format for Opus Speech and Audio Codec
draft-spittka-payload-rtp-opus-00.xml
This document defines the Real-time Transport Protocol (RTP) payload format and file storage format for packetization of Opus encoded speech and audio data that is essential to integrate the codec in the most compatible way. Further, media type registrations are described for the RTP payload format and the file storage format.
The IETF has been notified of intellectual property rights claimed in regard to some or all of the specification contained in this document. For more information consult the online list of claimed rights.
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."
This Internet-Draft will expire on January 02, 2012.
Copyright (c) 2011 IETF Trust and the persons identified as the document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.
The Opus codec is a speech and audio codec developed within the IETF Internet Wideband Audio Codec working group [codec]. The codec has a very low algorithmic delay and is is highly scalable in terms of audio bandwidth, network bit rate, and complexity. Further, it provides different modes to efficiently encode speech signals as well as music signals, thus, making it the codec of choice for various applications using the Internet or similar networks.
This document defines the Real-time Transport Protocol (RTP) [RFC3550] payload format and file storage format for packetization of Opus encoded speech and audio data that is essential to integrate the Opus codec in the most compatible way. Further, media type registrations are described for the RTP payload format and the file storage format. More information on the Opus codec can be obtained from the following IETF draft [Opus].
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119].
The Opus speech and audio codec has been developed to encode speech signals as well as audio signals. Two different modes, a voice mode or an audio mode, may be chosen to allow the most efficient coding dependent on the type of input signal, the sampling frequency of the input signal, and the specific application.
The voice mode allows to efficiently encode voice signals at lower bit rates while the audio mode is optimized for audio signals at medium and higher bit rates.
The Opus speech and audio codec is highly scalable in terms of audio bandwidth, network bit rate, and complexity. Further, Opus allows to transmit stereo signals.
The Opus speech and audio codec is based on the SILK codec [SILK] and the CELT codec [CELT]. For more detailed information on how Opus operates, also refer to [Opus].
Opus supports five different audio bandwidths, 8000, 12000, 16000, 24000, and 48000 Hz sampling frequency, for the voice mode and four different audio bandwidths, 8000, 16000, 24000, and 48000 Hz sampling frequency, for the audio mode.
For low bit rate applications transmitting mostly speech signals the voice mode of Opus SHOULD be used. The voice mode allows to encode voice signals at 8000, 12000, 16000, 24000, and 48000 Hz sampling frequency.
A sampling rate of 8000 Hz SHOULD only be used to interface to PSTN networks or on low end devices that do not support greater than 8000 Hz sampling frequency. A sampling rate of 12000 Hz SHOULD be used for lower end devices that do not support greater than 12000 Hz sampling frequency or are under severe network bandwidth constrains (e.g. wireless devices). A sampling rate of 16000 Hz SHOULD be used for all-IP platforms that do not support greater than 16000 Hz sampling frequency. Higher sampling rates are recommended for all devices that support those high sampling rates and desire full-bandwidth speech at medium bit rates.
For applications desiring very low delay speech transmission as well as music transmission in trade off to a higher bit rate, the audio mode SHOULD be used. This mode supports audio sampling rates of 8000, 16000, 24000, and 48000 Hz.
The network bit rate is adaptive within the range specified in Table 1 for corresponding modes and audio sampling rates. The average target network bit rate can be defined and modified in real-time while the actual bit rate will be dependent on the settings of Opus and the input signal and may change over time.
| Mode | fs (Hz) | BR (kbps) |
|---|---|---|
| voice | 8000 | 6 - 20 |
| voice | 12000 | 7 - 25 |
| voice | 16000 | 8 - 30 |
| voice | 24000 | 18 - 28 |
| voice | 48000 | 24 - 32 |
| audio | 8000 | 20 - 28 |
| audio | 16000 | 24 - 32 |
| audio | 24000 | 28 - 40 |
| audio | 48000 | 32 - 128 |
Mode specifies the Opus mode of operation; fs specifies the audio sampling frequency in Hertz (Hz); BR specifies the network bit rate range in kilobits per second (kbps).
The voice mode will always use a variable bit rate at audio sampling rates of 8000, 12000, and 16000 Hz. The average target bit rate can be adjusted at any point in time. To avoid congestion of the connection the average target bit rate SHOULD be adjusted to the available network bandwidth. If no target bit rate is specified the average bit rate may go up to the highest bit rate specified in Table 1.
In voice mode at audio sampling rates higher than 16000 Hz, i.e. 24000, and 48000 Hz, and audio mode Opus can be operated in both variable and constant bit rate. The target bit rate can be adjusted at any point in time.
The Opus codec may, as described in Section 3.2.1, be operated with an adaptive bit rate. In that case, the bit rate will automatically be reduced for certain input signals like periods of silence. During continuous transmission the bit rate will be reduced, when the input signal allows to do so, but the transmission to the receiver itself will never be interrupted. Therefore, the received signal will maintain the same high level of quality over the full duration of a transmission while minimizing the average bit rate over time.
In cases where the bit rate of Opus needs to be reduced even further or in cases where only constant bit rate is available, the Opus encoder may be set to use discontinuous transmission (DTX), where parts of the encoded signal that correspond to periods of silence in the input speech or audio signal are not transmitted to the receiver.
On the receiving side, the non-transmitted parts will be handled by a frame loss concealment unit in the Opus decoder which generates a comfort noise signal to replace the non transmitted parts of the speech or audio signal.
The DTX mode of Opus will have a slightly lower speech or audio quality than the continuous mode. Therefore, it is RECOMMENDED to use Opus in the continuous mode unless restraints on network bandwidth are severe. The DTX mode can be engaged for operation in both adaptive or constant bit rate.
Complexity can be scaled to optimize for CPU resources in real-time, mostly in trade-off to network bit rate. Also, different modes of Opus have different complexity.
The voice mode of Opus allows for "in-band" forward error correction (FEC) data to be embedded into the bit stream of Opus. This FEC scheme adds redundant information about the previous packet (n-1) to the current output packet n. For each frame, the encoder decides whether to use FEC based on (1) an externally-provided estimate of the channel's packet loss rate; (2) an externally-provided estimate of the channel's capacity; (3) the sensitivity of the audio or speech signal to packet loss; (4) whether the receiving decoder has indicated it can take advantage of "in-band" FEC information. The decision to send "in-band" FEC information is entirely controlled by the encoder and therefore no special precautions for the payload or storage format have to be taken.
On the receiving side, the decoder can take advantage of this additional information when, in case of a packet loss, the next packet is available. In order to use the FEC data, the jitter buffer needs to provide access to payloads with the FEC data. The decoder API function has a flag to indicate that a FEC frame rather than a regular frame should be decoded. If no FEC data is available for the current frame, the decoder will consider the frame lost and invokes the frame loss concealment.
If the FEC scheme is not implemented on the receiving side, FEC SHOULD NOT be used, as it leads to an inefficient usage of network bandwidth. Decoder support for FEC SHOULD be indicated at the time a session is set up.
Opus allows for transmission of stereo audio signals. This operation will be signaled in the Opus payload and no special arrangements have to be made in the payload format. Any implementation of the Opus decoder MUST be capable to receive stereo signals.
If a decoder can not take advantage of the benefits of a stereo signal this SHOULD be indicated at the time a session is set up. In that case the sending side SHOULD NOT send stereo signals as it leads to an inefficient usage of network bandwidth.
The payload format for Opus consists of the RTP header and Opus payload data.
The format of the RTP header is specified in [RFC3550]. The Opus payload format uses the fields of the RTP header consistent with this specification.
The payload length of Opus is a multiple number of octets and therefore no padding is required. The payload MAY be padded by an integer number of octets according to [RFC3550].
The marker bit (M) of the RTP header has no function in combination with Opus and MAY be ignored.
The RTP payload type for Opus has not been assigned statically and is expected to be assigned dynamically.
The receiving side MUST be prepared to receive duplicates of RTP packets. Only one of those payloads MUST be provided to the Opus decoder for decoding and others MUST be discarded.
Opus supports 5 different sampling rates which may be adjusted during the duration of a call. The RTP timestamp clock frequency is defined as the highest supported sampling frequency of Opus, i.e. 48000 Hz, for all modes and sampling rates of Opus. The unit for the timestamp is samples. The RTP timestamp corresponds to the sample time of the first encoded sample in the encoded frame. For sampling rates lower than 48000 Hz the number of samples has to be multiplied with a multiplier according to Table 2 to determine the RTP timestamp.
| fs (Hz) | Multiplier |
|---|---|
| 8000 | 6 |
| 12000 | 4 |
| 16000 | 3 |
| 24000 | 2 |
| 48000 | 1 |
fs specifies the audio sampling frequency in Hertz (Hz); Multiplier is the value that the number of samples have to be multiplied with to calculate the RTP timestamp.
The Opus encoder can be set to output encoded frames representing 2.5, 5, 10, 20, 40, or 60 ms of speech or audio data. Further, an arbitrary number of frames can be combined into a packet. The maximum packet length is limited to the amount of encoded data representing 120 ms of speech or audio data. The packetization of encoded data is purely done by the Opus encoder and therefore only one packet output from the Opus encoder MUST be used as a payload.
Figure 1 shows the structure combined with the RTP header.
+----------+--------------+
|RTP Header| Opus Payload |
+----------+--------------+
Table 3 shows supported frame sizes for different modes and sampling rates of Opus and how the timestamp needs to be incremented for packetization.
| Mode | fs | 2.5 | 5 | 10 | 20 | 40 | 60 |
|---|---|---|---|---|---|---|---|
| ts incr | all | 120 | 240 | 480 | 960 | 1920 | 2880 |
| voice | 8000/12000/16000/24000/48000 | x | x | x | x | ||
| audio | 8000/16000/24000/48000 | x | x | x | x |
Mode specifies the Opus mode of operation; fs specifies the audio sampling frequency in Hertz (Hz); 2.5, 5, 10, 20, 40, and 60 represent the duration of encoded speech or audio data in a packet; ts incr specifies the value the timestamp needs to be incremented for the representing packet size. For multiple frames in a packet these values have to be multiplied with the respective number of frames.
The Opus storage format allows to store Opus encoded data into e.g. a file or an email attachment. The storage format consists of a header and a series of blocks containing encoded speech or audio frames. The storage format closely mimics the real-time payload format and allows to easily convert packets, e.g. received by a voicemail system, into a storage format and vice versa and therefore allowing maximum flexibility and low overhead. Please note that this storage format is not meant to be a robust storage format, nor the most efficient storage format. For a robust storage format that allows advanced functionality like e.g. seeking, a more advanced container format should be used.
Figure 2 shows an example of an Opus encoded file. Note that due to the potentially adaptive bit rate the packet length may be variable and no fixed block size can be defined for blocks containing encoded data.
+------------------+
| Header |
+-----------+------+
| block 1 |
+-----------+--+
| block 2 |
+--------------+--+
: ... :
+--------------+--+
| block n |
+-----------------+
An Opus storage header contains the following ASCII character string as a magic number:
"#!opus\n" (hexadecimal: 0x23 0x21 0x6f 0x70 0x75 0x73 0x0A)
Following the storage header, blocks of encoded data are stored in consecutive order in time according to Figure 2. Each block contains a block header followed by a payload according to Figure 3.
The block header contains information that, for an RTP-based session, can be derived from the IP and RTP headers: The number of octets contained in the subsequent payload and the RTP timestamp.
The number of octets in the payload is represented by 16 bits and the timestamp is specified by 32 bits. For the first block, the timestamp MAY be a random number. For the following blocks, the timestamp MUST be incremented according to the way timestamps are incremented when Opus payloads are transmitted over RTP.
0 16 48
+-------------------+----------------------------+-----------------
| # of octets | Timestamp | Payload
+-------------------+----------------------------+-----------------
The payload of each block in Figure 2 represents one packet of Opus encoded data the way as originally encoded by the Opus encoder. Information about frame size representing the duration of encoded speech or audio data, number of encoded frames, stereo information, and DTX is embedded into the payload of Opus and not subject to the storage format. It can be extracted from the payload during decoding of the encoded data.
During the usage of DTX no blocks are stored when the channel is inactive. Timestamps MUST be used to reassemble the decoded signal in a time-aligned way.
The adaptive nature of the Opus codec allows for an efficient congestion control.
The voice mode of Opus at audio sampling rates of 8000, 12000, and 16000 always runs with a variable bit rate. The average bit rate in that mode is dependent on the input signal and will especially decrease during silent periods. The voice mode at audio sampling rates of 24000 and 48000 Hz and the audio mode may run at a variable or constant bit rate. In either way, the target bit rate of Opus can be adjusted at any point in time and thus allowing for an efficient congestion control.
Furthermore, the amount of encoded speech or audio data encoded in a single packet can be used for congestion control since the transmission rate is inversely proportional to these frame sizes. A lower packet transmission rate reduces the amount of header overhead but at the same time increases latency and error sensitivity and should be done with care.
It is RECOMMENDED that congestion control is applied during the transmission of Opus encoded data.
One media subtype (audio/opus) has been defined and registered as described in the following section.
Media type registration is done according to [RFC4288] and [RFC4855].
Type name: audio
Subtype name: opus
Required parameters:
Optional parameters:
Encoding considerations:
Security considerations:
Interoperability considerations: none
Published specification: none
Applications that use this media type:
Additional information:
Person & email address to contact for further information:
Intended usage: COMMON
Restrictions on usage:
Author:
Change controller: TBD
The information described in the media type specification has a specific mapping to fields in the Session Description Protocol (SDP) [RFC4566], which is commonly used to describe RTP sessions. When SDP is used to specify sessions employing the Opus codec, the mapping is as follows:
Below are some examples of SDP session descriptions for Opus:
Example 1: Standard session with 48000 Hz clock rate
m=audio 54312 RTP/AVP 101
a=rtpmap:101 opus/48000
Example 2: 16000 Hz clock rate, maximum packet size of 40 ms, recommended packet size of 40 ms, maximum average bit rate of 20000 bps, stereo signals are preferred, FEC is allowed, DTX is not allowed
m=audio 54312 RTP/AVP 101
a=rtpmap:101 opus/48000
a=fmtp:101 maxcodedaudiobandwidth=16000; maxaveragebitrate=20000;
stereo=1; useinbandfec=1; usedtx=0
a=ptime:40
a=maxptime:40
When using the offer-answer procedure described in [RFC3264] to negotiate the use of Opus, the following considerations apply:
m=audio 54312 RTP/AVP 100
a=rtpmap:100 opus/48000
For declarative use of SDP such as in Session Announcement Protocol (SAP), [RFC2974], and RTSP, [RFC2326], for Opus, the following needs to be considered:
All RTP packets using the payload format defined in this specification are subject to the general security considerations discussed in the RTP specification [RFC3550] and any profile from e.g. [RFC3711] or [RFC3551].
This payload format transports Opus encoded speech or audio data, hence, security issues include confidentiality, integrity protection, and authentication of the speech or audio itself. The Opus payload format does not have any built-in security mechanisms. Any suitable external mechanisms, such as SRTP [RFC3711], MAY be used.
This payload format and the Opus encoding do not exhibit any significant non-uniformity in the receiver-end computational load and thus are unlikely to pose a denial-of-service threat due to the receipt of pathological datagrams.
TBD