Associating Time-codes with RTP streams
draft-ietf-avt-smpte-rtp-13.txt

Status of this Memo

By submitting this Internet-Draft, each author represents that any applicable patent or other IPR claims of which he or she is aware have been or will be disclosed, and any of which he or she becomes aware will be disclosed, in accordance with Section 6 of BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as “work in progress.”

The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt.

The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html.

This Internet-Draft will expire on December 27, 2008.

Abstract

This document describes a mechanism for associating time-codes, as defined by the Society of Motion Picture and Television Engineers (SMPTE), with media streams, in a way that is independent of the RTP payload format of the media stream itself.

Table of Contents

1.  Introduction
2.  Requirements Notation
3.  Design Goals
4.  Signaling Information
5.  In-stream Information
    5.1.  Compact Format of the Time-code
    5.2.  Full Format of the Time-code
    5.3.  Associations in RTCP
    5.4.  Associations in RTP
6.  Implementation Note
7.  Discussion
8.  Security Considerations
9.  IANA Considerations
10.  RFC Editor Considerations
11.  Acknowledgments
12.  Normative References
§  Author's Address
§  Intellectual Property and Copyright Statements

1.  Introduction

First a brief background on Time-codes [SMPTE‑12M] (Society of Motion Picture and Television Engineers, “Television, Audio and Film – Time and Control Code,” .).

The time-code system in common use is defined by the Society of Motion Picture and Television Engineers (SMPTE), and in it, time-codes count frames. A common form of the display looks like a normal clock value (hh:mm:ss.frame). When the frame rate is truly integer, then this can be a normal clock value, in that seconds tick by at the same rate as the seconds we know and love.

However, NTSC video infamously runs slightly slower than 30 frames/second. Some people call it 29.97 (which isn't quite right), but correctly a frame takes 1001 ticks of a 30000 tick/second clock. Be that as it may, SMPTE time codes count 30 of these frames and deem that to make a second.

This causes a SMPTE time-code display to 'run slow' compared to real-time. To ameliorate this, sometimes a format called drop-frame is used. Some of the frame numbers are skipped, so that the counter periodically 'catches up' (so some time-code seconds actually only have 28 frames in them).

It is worth noting that in neither case is the SMPTE time-code an accurate clock; in the first case, it runs slow, and in the second, the adjustments are abrupt and periodic – and still not quite accurate. Hence the rest of this document tries to be clear when referring to a second in a time-code as a ‘time-code second’.

However, SMPTE time-codes do run in real-time when used with systems with integral frames/second (e.g. film content at 24 frames/second, or PAL video).

This specification defines how to carry time-codes in RTP and RTCP, associate them with a media stream, and synchronize the time-codes with the RTP time-stamps. It uses the general RTP header extension mechanism [hdrext] (Singer, D., “A general mechanism for RTP Header Extensions,” October 2006.).

2.  Requirements Notation

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119] (Bradner, S., “Key words for use in RFCs to Indicate Requirement Levels,” March 1997.).

3.  Design Goals

What we desire is a system that allows us to associate a SMPTE time-code with some media in an RTP [RFC3550] (Schulzrinne, H., Casner, S., Frederick, R., and V. Jacobson, “RTP: A Transport Protocol for Real-Time Applications,” July 2003.) stream. Since in RTP all media has a clock already, we can often leverage that fact. If we treat the media as having ‘segments’ of time in which the time-code is simply counting up, then the time-code anywhere within a segment can be calculated if you know:

• the RTP timestamp of the start of the segment;
• the time-code of the start of the segment;
• the counting rate and other parameters of the time-code;
• the RTP timestamp where you want to know the time-code.

There are two cases to consider:

1. the time-codes are piece-wise continuous with only occasional discontinuities;
2. the continuity of the time-codes is not certain (or not known).

The first can be handled by providing details of the time-code axis and an initial mapping from RTP time to time-code time, and periodic mappings in RTCP packets. This is defined in section 5.3.

The second requires in-band signaling within the RTP packets themselves. This is defined in section 5.4.

There are applications where the transport of all 8 bytes of the SMPTE 12M timecode are important (e.g. when the date of the time-code must be known, or when the RTP transport is used as a transparent pipe). On the other hand, there are cases (e.g. when timecodes are used with compressed audio) when bandwidth is also important. To support both use cases, provision is made for both compact and full forms of the time-code.

Receivers MUST support timecodes in both RTCP and RTP, and both forms (compact and full) of the time-code. Senders, of course, are free to choose.

Note that the compact form allows frame numbers greater than the full form (a field of 6 bits vs. a full BCD digit and a 2-bit BCD digit, which gives a maximum transmitted value of 29). In some cases, the color frame flag (bit 11) is used to 'extend' the tens of frames field from 2 to 3 bits; however, such practices are outside the scope of this specification.

In the case that a presentation contains more than one stream senders MUST continue to send the standard RTP synchronization information in RTCP, even if the streams carry SMPTE time-codes that could be used for synchronization. In fact, when time-code is carried by more than one stream, this draft does not constrain the time-codes: at a given point in time, they may be the same, or they may differ (e.g. if they carry the original time-codes of different source material that was edited together).

4.  Signaling Information

If the recipient must ever calculate time-codes based on the RTP times, then some setup information is needed. This MUST be sent out-of-band, for example in a SIP offer/answer exchange. Since this is a general header extension [hdrext] (Singer, D., “A general mechanism for RTP Header Extensions,” October 2006.), appropriate signaling for those header extensions should be used.

The setup information should include:

1. the duration, in the RTP timescale, of a single frame-count in the 'frames' portion of the time-code (frame_duration)
2. the number of those frames that make a time-code second (frames_per_tc_second); framecounter values may be between 0 and (frames_per_tc_second - 1)
3. is-NTSC-drop-frame: should the usual 'left out numbers' of drop-frame be applied or not?

Note that other information we need to do the calculation (e.g. the clock rate of the RTP timestamp) is supplied already and assumed to be available.

For example, if associated with a video track with the common time-scale of 90000, then frame-duration of 3003 and frames-per-tc-second of 30 would yield a 'normal' SMPTE time-code for NTSC video. Similarly values of 3750 and 24 yield a time-code for 24 fps film content, and so on.

Note also that we supply explicitly the frame duration and frames/second, even though they are obviously closely related. These removes any ambiguity of what the counter values should be in the case of drop-frame counting. These three values MUST correspond with each other.

When SDP is used, these three parameters are transmitted as extensionattributes, as defined in the header extension specification [hdrext] (Singer, D., “A general mechanism for RTP Header Extensions,” October 2006.), with the following syntax. The form of the extension attributes is 'owned' by the extension name. These parameters to the extension do not need registration action beyond their documentation here. Note that the parameters are supplied as extension attributes, suitable for in-line use in RTP, even if in a given stream only the RTCP mapping is used.

  digit = "0"/"1"/"2"/"3"/"4"/"5"/"6"/"7"/"8"/"9"

  integer = 1*digit

  frame-duration = integer

  frames-per-tc-second = integer

  drop = "/drop"

  extensionattributes = frame-duration "/" frames-per-tc-second [drop]

If '/drop' is specified, then the first two frame numbers are omitted from the count of each minute, except for minutes 00, 10, 20, 30, 40, 50, as documented in SMPTE specification [SMPTE‑12M] (Society of Motion Picture and Television Engineers, “Television, Audio and Film – Time and Control Code,” .) section 4.2.2. (Note that this usually only applies to NTSC video.)

The URI used for the signaling is “urn:ietf:params:rtp-hdrext:smpte-tc”. This URI signals the possible presence of associations in RTCP or RTP, as defined below.

An example in SDP, for film material, on a stream with a timescale of 600, might be:

  a=extmap:4 urn:ietf:params:rtp-hdrext:smpte-tc 25/24

Another example, for drop-frame NTSC, on a stream with a timescale of 600, might be:

  a=extmap:4 urn:ietf:params:rtp-hdrext:smpte-tc 20/30/drop

5.  In-stream Information

5.1.  Compact Format of the Time-code

A compact binary SMPTE time-code in this design occupies 24 bits. It is NOT formatted in the BCD system, but uses binary fixed-width fields. It has the following structure:

sign(1)

-- 1 for negative, 0 for positive

hours (5 bits)

-- 0 to 23; the values 24-31 are reserved

minutes (6 bits)

-- 0 to 59; 60-63 are reserved

seconds (6 bits)

-- 0 to 59; 60-63 are reserved

frames(6 bits)

-- 0 to (frames-per-tc-second - 1)

Note that these fields are larger than the provision in SMPTE 12M where binary-coded decimal is used (and notably, where only two bits are provided for the tens digit of the frame count, so frame numbers above 39 cannot be represented).

5.2.  Full Format of the Time-code

A full SMPTE time-code occupies 64 bits. It is formatted exactly as defined in section 7 and 8 of SMPTE 12M [SMPTE‑12M] (Society of Motion Picture and Television Engineers, “Television, Audio and Film – Time and Control Code,” .), without the 16-bit syncword. The value of the "drop frame flag" MUST agree with the use of the "drop" indicator in the signalling.

Here are the bit assignments from SMPTE 12M, for information:

0--3

Units of frames

4--7

First binary group

8--9

Tens of frames

10

Drop frame flag

11

Color frame flag

12--15

Second binary group

16--19

Units of seconds

20--23

Third binary group

24--26

Tens of seconds

27

Polarity correction

28--31

Fourth binary group

32--35

Units of minutes

36--39

Fifth binary group

40--42

Tens of minutes

43

Binary group flag BGF0

44--47

Sixth binary group

48--51

Units of hours

52--55

Seventh binary group

56--57

Tens of hours

58

Binary group flag BGF1

59

Binary group flag BGF2

60--63

Eighth binary group

5.3.  Associations in RTCP

When the time-codes are piece-wise continuous, we then supply in RTCP packets an RTP timestamp and an SMPTE time-code, for the start of each run of calculable time-codes. This establishes the time-code for all RTP times greater than or equal to the one given, until a subsequent RTCP packet reestablishes the mapping.

Note that the RTP time-stamp in the RTCP mapping may not match the time-stamp of any frame in the media stream. For video, it normally would; but a time-stamp transition may happen part-way through a decoded audio frame. Since they share the same clock, the timing of that transition and the timing of the audio stream itself have the same accuracy.

The RTCP packets need not use the same RTP timestamp as the sender report (or transmission time) in the same RTCP packet. They can be sent ‘ahead of need’ if possible (e.g. for stored content, when the server can look-ahead) or just-in-time – sent as early feedback packets, for example following the rules in [RFC4585] (Ott, J., Wenger, S., Sato, N., Burmeister, C., and J. Rey, “Extended RTP Profile for Real-time Transport Control Protocol (RTCP)-Based Feedback (RTP/AVPF),” August 2006.), after a discontinuity in the time-code is detected, and allow media-buffering in the client the chance to ‘catch’ the RTCP before the matching RTP packet is processed and displayed.

The association is a new RTCP Control Packet Type, using the value 194 (see section 9). This control packet has one of the two following forms, differentiated by its length:

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |V=2|P|    SC   |PT=SMPTETC=194 |             length=3          |
   +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
   |                     SSRC of packet sender                     |
   +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
   |                         RTP timestamp                         |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |S|  hours  |  minutes  |  seconds  |  frames   |  reserved=0   |
   +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+

The fields S (sign), hours, minutes, seconds, and frames, are defined in section 5.1.

For this short form, the length takes the fixed value 3, indicating a control packet of 4 32-bit words.

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |V=2|P|    SC   |PT=SMPTETC=194 |             length=4          |
   +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
   |                     SSRC of packet sender                     |
   +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
   |                         RTP timestamp                         |
   +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
   |                          Full 8-byte                          |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                      SMPTE 12M timecode                       |
   +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+

For this full time-code, long form, the length takes the fixed value 4, indicating a control packet of 5 32-bit words.

5.4.  Associations in RTP

When the time-codes are not known to be piece-wise continuous, or absolute surety of mapping is desired, then the mapping can be placed into some or all of the RTP packets. This is a less desirable route; it uses the RTP header extension, which some terminals may find problematic. And clearly placing mapping information in every packet uses more bandwidth.

In as many RTP packets as needed (possibly all), a RTP header extension is used [hdrext] (Singer, D., “A general mechanism for RTP Header Extensions,” October 2006.) to associate an RTP time to a SMPTE time-code.

There are two forms of this header extension, again differentiated by their length. The short form associates a compact time-code with the RTP timestamp of the packet. The long form allows associates a full time-code with a timestamp offset from the RTP timestamp of the packet.

The short form has a length of 3 bytes (24 bits). The long form has a length of 12 bytes (96 bits), and consists of a full SMPTE 12M time-code, followed by a signed 32-bit offset D from the RTP timestamp. If the packet has timestamp T, this establishes an RTP to time-code association for the RTP time T+D.

6.  Implementation Note

This section contains a suggestion on how to calculate a time-code for a time T2, given an initial code at time T1, and the frame duration

It might seem that when drop-frame is used, there is a 'fence post' problem: how many minutes in which frame-numbers are dropped have past since the initial time-code? However, this can be avoided if all calculations are 'zero-based'; then the number of 'fence posts' is known.

  framesSinceTCzero := TimeCodeToFrameCount( initialTimeCode );
  framesSinceMapping := floor( (T2-T1)/frameDuration );
  totalFrames := framesSinceTCzero + framesSinceMapping;
  timeCode := FrameCountToTimeCode( totalFrames );

The SMPTE engineering guideline [smpte‑eg40] (SMPTE, “Conversion of Time Values Between SMPTE 12M Time Code, MPEG-2 PCR Time Base and Absolute Time,” August 2002.) contains all the appropriate equations, constants etc. for performing these and other conversions.

7.  Discussion

This design has the advantage of not requiring the introduction of new IP packets into the sessions or new data into the main data channel, using low-bandwidth (vanishingly low in the case of streams with no discontinuities), and is independent of the design of the RTP packets themselves: the RTP profile (including possibly encryption) and the RTP payload format. SMPTE time-codes can be associated with any RTP stream, including those with existing payload formats.

It might be argued that we could set the initial mapping also in the SDP, since RTCP packets might get lost. But this means that the SDP now has to have knowledge of the RTP random offset, which is nasty; and if one puts this RTCP packet into all sender reports, it's probably good enough. Then if you don't have time-codes, you don't have audio-video-sync either.

This associates the time-code with a particular media stream. An alternative would be to make it an RTP stream in its own right; but the data rate is so low, this seems egregious. And by packing it inline, we can do this backwards-compatible for gateways etc. that already handle dual-stream.

There is no way in this draft to detect that an RTCP packet has been lost, and that a mapping may be being used outside its intended range.

The design assumes that clients will hold mappings until they are superseded, and that a client may need to buffer some number of upcoming mappings.

8.  Security Considerations

SMPTE time-codes are only informative and there are no known security considerations from associating them with media streams.

9.  IANA Considerations

The RTCP packet type used for SMPTE time-code needs to be registered, in accordance with section 15 of [RFC3550] (Schulzrinne, H., Casner, S., Frederick, R., and V. Jacobson, “RTP: A Transport Protocol for Real-Time Applications,” July 2003.). The abbreviation is "SMPTETC", the full name is "SMPTE time-code mapping", the suggested value is 194, and the specification is this document.

The name smpte-tc needs to be registered into the rtp-hdrext section of the urn:ietf: namespace, referring to RFCxxxx.

10.  RFC Editor Considerations

The reference to an Internet Draft needs to be updated to the RFC when it is published (which should be before this draft).

RFCxxxx in the IANA considerations needs to be updated with the number of this RFC.

11.  Acknowledgments

Both Brian Link and John Lazzaro provided helpful comments on an initial draft. Colin Perkins was helpful in reviewing and dealing with the details. Ladan Gharai provided a thoughtful final review.

12. Normative References

Author's Address

Full Copyright Statement

Copyright © The IETF Trust (2008).

This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights.

This document and the information contained herein are provided on an “AS IS” basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Intellectual Property

The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79.

Copies of IPR disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr.

The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at ietf-ipr@ietf.org.