TOC 
Benchmarking Methodology WorkingS. Poretsky
GroupReef Point Networks
Internet-DraftV. Gurbani
Expires: May 22, 2008Bell Laboratories, Alcatel-Lucent
 C. Davids
 Illinois Institute of Technology
 November 19, 2007


Terminology for Benchmarking Session Initiation Protocol (SIP) Networking Devices
draft-poretsky-sip-bench-term-04

Status of this Memo

By submitting this Internet-Draft, each author represents that any applicable patent or other IPR claims of which he or she is aware have been or will be disclosed, and any of which he or she becomes aware will be disclosed, in accordance with Section 6 of BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as “work in progress.”

The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt.

The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html.

This Internet-Draft will expire on May 22, 2008.

Abstract

This document provides a terminology for benchmarking SIP performance in networking devices. Terms are included for test components, test setup parameters, and performance benchmark metrics for black-box benchmarking of SIP networking devices. The Performance Benchmark Metrics are obtained for the SIP Control Plane and Media Plane. The terms are intedned for use in a companion Methodology document for complete performance characterization of a device in a variety of network conditions making it possible to compare performance of different devices. It is critical to provide Test Setup Parameters and a Methodology document for SIP performance benchmarking because SIP allows a wide range of configuration and operational conditions that can influence performance benchmark measurements. It is necessary to have terminology and methodology standards to ensure that reported benchmarks have consistent definition and were obtained following the same procedures. Benchmarks can be applied to compare performance of a variety of SIP networking devices.


Table of Contents

1.  Terminology
2.  Introduction
3.  Term Definitions
    3.1.  Protocol Components
        3.1.1.  Session
        3.1.2.  Signaling Plane
        3.1.3.  Media Plane
        3.1.4.  Associated Media Stream
        3.1.5.  Invite-initiated Session (IS or I Session)
        3.1.6.  Session Attempting State
        3.1.7.  Session Established State
        3.1.8.  Session Disconnecting State
        3.1.9.  Non-INVITE-initiated Session (NIS or N-Session)
        3.1.10.  Registration
        3.1.11.  Successful IS Establishment
        3.1.12.  Unsuccessful IS Establishment
        3.1.13.  Successful NS Establishment
        3.1.14.  Unsuccessful NS Establishment
        3.1.15.  Successful IS Disconnection
        3.1.16.  Unsuccessful IS Disconnection
    3.2.  Test Components
        3.2.1.  Emulated Agent
        3.2.2.  Signaling Server
        3.2.3.  SIP-Aware Stateful Firewall
        3.2.4.  SIP Transport Protocol
    3.3.  Test Setup Parameters
        3.3.1.  IS Attempt
        3.3.2.  IS Duration
        3.3.3.  IS Signaling Duration
        3.3.4.  IS Media Duration
        3.3.5.  Session Attempt Rate
        3.3.6.  Media-Session Attempt Rate
        3.3.7.  NI-Session Attempt Rate
        3.3.8.  Disconnect Threshold
        3.3.9.  Establishment Threshold
        3.3.10.  Media Packet Size
        3.3.11.  Media Offered Load, per Media Stream
        3.3.12.  Media Offered Load, Aggregate
        3.3.13.  Media Session Hold Time
        3.3.14.  Loop Detection Option
        3.3.15.  Forking Option
    3.4.  Benchmarks
        3.4.1.  Maximum Registration Rate
        3.4.2.  Maximum Session Establishsment Rate
        3.4.3.  Session Capacity
        3.4.4.  Session Establishment Performance
        3.4.5.  Session Attempt Delay
        3.4.6.  Session Disconnect Delay
        3.4.7.  Standing Sessions
        3.4.8.  IM Rate
        3.4.9.  Presence Rate
4.  IANA Considerations
5.  Security Considerations
6.  Acknowledgments
7.  References
    7.1.  Normative References
    7.2.  Informational References
§  Authors' Addresses
§  Intellectual Property and Copyright Statements



 TOC 

1.  Terminology

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14, RFC 2119. RFC 2119 defines the use of these key words to help make the intent of standards track documents as clear as possible. While this document uses these keywords, this document is not a standards track document. The term Throughput is defined in RFC 2544.

Many SIP terms used in this document are defined in [RFC.3261] (Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston, A., Peterson, J., Sparks, R., Handley, M., and E. Schooler, “SIP: Session Initiation Protocol,” June 2002.).


 TOC 

2.  Introduction

Service Providers are now planning Voice Over IP (VoIP) and Multimedia network deployments using the IETF developed Session Initiation Protocol (SIP) [RFC.3261] (Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston, A., Peterson, J., Sparks, R., Handley, M., and E. Schooler, “SIP: Session Initiation Protocol,” June 2002.). SIP is a signaling protocol originally intended to be used for the dynamic establishment, disconnection and alteration of streams of media between end users. As it has evolved it is being used for a growing number of applications. Some of these applications do result in streams of media but instead create other services tailored to the end-users' immediate needs or preferences.

VoIP with SIP has led to development of new networking devices including SIP Server, Session Border Controller, and SIP-Aware Stateful Firewall. The mix of voice and IP functions in this variety of devices has produced inconsistencies in vendor reported performance metrics and has caused confusion in the service provider community. SIP allows a wide range of configuration and operational conditions that can influence performance benchmark measurements. It is important to be able to correlate a signalling measurement with the media plane measurements to determine the system performance. As SIP has increased deployments, the need to benchmark the performance of SIP devices is increasingly important.

When defining SIP performance benchmarks it is critical to also provide definitions for Test Setup Parameters and a corresponding Methodology document for SIP performance benchmarking. This enables benchmarks to be understood, fairly compared, and repeatable. The chosen benchmarking metrics and methodologies used to obtain those metrics should be based upon a common set of clearly defined terminology. This document provides the benchmarking terms for performance benchmarking the SIP control and media planes. Terms are included for Test Components, Test Setup Parameters, and Benchmarks. All benchmarks are black-box measurements of the SIP Control and Media Planes. It is intended that these terms be used in a companion Methodology document.

SIP is used to create a growing number of very different applications and features, some off which result in the creation of media streams, others of which do not. The set of benchmarking terms provided in this document is intended for use with any of these. SIP is frequently used to create streams of media. The control plane and the media plane are treated as orthogonal in this document. In order to characterize the performance of one or another application or feature it may be necessary to logically associate several of the benchmarking metrics provided here. Benchmarks to be obtained and compared for different types of Devices Under test (DUTs) such as SIP Proxy Server, SBC, P-CSCF, Proxy Server paired with a Firewall/NAT device, and P-CSCF paired with a Firewall/NAT device. Media benchmarks can also be made when testing Systems Under Test (SUTs).


 TOC 

3.  Term Definitions


 TOC 

3.1.  Protocol Components


 TOC 

3.1.1.  Session

Definition:
The combination of Signaling Plane and Media Plane protocol messages and processes that enable two or more participants to communicate.
Discussion:
SIP messages in the signaling plane can be used to create and manage applications for one or more end users. Most commonly the application in question resembles a traditional telephone call. SIP is often used to create and manage media streams in suspport of applications that resemble traditional telephone service. The prototype of these applications is a session - a collection of media streams under the control of SIP entities. The application, and by extension, the session, has components in both the signaling plane and the media plane. SIP includes definitions of a Call-ID, a dialogue and a transaction that support this application.A growing number of usages and applications do not require the creation of associated media. The first such usage was the REGISTER. Applications that use the IM and SUBSCRIBE/NOTIFY methods also do not require SIP to manage media streams. The terminology Invite-initiated Session (IS or I-Session) and Non-invite initiated Session (NIS or N-Session) are used to distinguish between these different usages.
A Session in the context of this document, is considered to be a vector with three components: (1) A component in the signaling plane (SIP messages), sess.sig (2) A media component in the media plane (RTP and SRTP streams for example), sess.med (3) A control component in the media plane (RTCP messages for example), sess.medc
An I-Session is expected to have non-null sess.sig and sess.med components. The use of control protocols in the media component is media dependent, thus the expected presence or absence of sess.medc is media dependent and test-case dependent. An N-Session is expected to have a non-null sess.sig component, but null sess.med and sess.medc components.
Packets in the Signaling Plane and Media Plane will be handled by different processes within the DUT. They will take different paths within a SUT. These different processes and paths may prooduce variations in performance. The terminology and benchmarks defined in this document and the methodology for their use are designed to enable us to compare performance of the DUT/SUT with reference to the type of SIP-supported application it is handling.
Note that one or more sessions can simultaneously exist between any participants. This is represented as an array session[x].
Sessions will be represented as a vector array with three components, as follows:
session->
session[x].sig, the signaling component
session[x].medc, the media control component (e.g. RTCP)
session[x].med[y], an array of associated media streams (e.g. RTP, SRTP, RTSP, MSRP). This media component may consist of zero or more media streams.
Measurement Units:
N/A.
Issues:
None.
See Also:
Media Plane
Signaling Plane
Invite-inititated Session (IS or I-Session)
Non-invite-inititated Session (NIS or N-Session)

Figure 1 below illustrates the vector character of the application or session described in this document. 





                |\
                |
                |   \
        sess.sig|
                |     \
                |
                |       \
                |         o
                |        /
                |       / |
                |      /
                |     /   |
                |    /
                |   /     |
                |  /
                | /       |   sess.medc
                |/_____________________
               /               /
              /           |
             /               /
 sess.med   /             |
           /_ _ _ _ _ _ _ _/
          /
         /
        /
       /

Figure 1. Application or Session Components

 Figure 1 

Figure 2 below illustrates the three states of a session. The EA attempts an I-session with the DUT/SUT. The state of the session following the INVITE and preceding the DUT/SUT's 200 OK is the Attempting state. The state of the session following the 200 OK sent by the DUT/SUT is the Established state. The Associated Media is shown as "media" connected to media ports M1 and M2 on the Tester. After the EA sends a BYE, the session enters the Disconnecting state. The session reverts to its Null state after the DUT/SUT sends a 200 OK to the EA. 




        EA	      DUT/SUT   M1	M2
	|		|	|	|
	|    INVITE	|	|	|
-----	|-------------->|	|	|
  	|		|	|	|
Attempting	|	|	|	|
 	|    200 OK	|	|	|
-----	|<--------------|	|	|
	|		|	|	|
	|		|	|	|
	|		|	| media	|
Established		|	|<----->|
	|		|	|<----->|
	|      BYE	|	|	|
-----	|-------------->|	|	|
	|		|	|	|
Disconnecting		|	|	|
	|   200 OK	|	|	|
-----	|<--------------|	|	|
	|		|	|	|

Figure 2. Basic SIP Test Topology

 Figure 2 

 TOC 

3.1.2.  Signaling Plane

Definition:
The control plane in which SIP messages [RFC.3261] (Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston, A., Peterson, J., Sparks, R., Handley, M., and E. Schooler, “SIP: Session Initiation Protocol,” June 2002.) are exchanged between SIP Agents [RFC.3261] (Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston, A., Peterson, J., Sparks, R., Handley, M., and E. Schooler, “SIP: Session Initiation Protocol,” June 2002.) to establish a connection for media exchange.
Discussion:
SIP messages are used to establish sessions in several ways: directly between two User Agents [RFC.3261] (Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston, A., Peterson, J., Sparks, R., Handley, M., and E. Schooler, “SIP: Session Initiation Protocol,” June 2002.), through a Proxy Server [RFC.3261] (Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston, A., Peterson, J., Sparks, R., Handley, M., and E. Schooler, “SIP: Session Initiation Protocol,” June 2002.), or through a series of Proxy Servers. The Signaling Plane MUST include the Session Description Protocol (SDP) [Ha06].
The Signaling Plane for a single Session is represented by session.sig.
Measurement Units:
N/A.
Issues:
None.
See Also:
Media Plane
Emulated Agents

 TOC 

3.1.3.  Media Plane

Definition:
The data plane in which one or more media streams [sc02] and their associated media control protocols [Sc03] are exchanged after a media connection has been created by the exchange of signaling messages in the Signaling Plane.
Discussion:
Media may also be known as the the "payload" or "bearer channel". The Media Plane MUST include the media control protocol, if one is used, and the media stream(s). Media is analagous Data. Example media are audio, video, whiteboard, and instant messaging service. The media stream is described in the SDP of the Signaling Plane.
The media for a single Session is represented by session.med. The media control protocol is represented by session.medc.
Measurement Units:
N/A.
Issues:
None.
See Also:
Signaling Plane

 TOC 

3.1.4.  Associated Media Stream

Definition:
Media that corresponds to an 'm' line in the SDP payload of the Signaling Plane.
Discussion:
Any media protocol MAY be used.
For any session's signaling component, represented as session.sig, there may be one or multiple associated media streams which are represented be a vector array session.med[y].
Measurement Units:
N/A.
Issues:
None.

 TOC 

3.1.5.  Invite-initiated Session (IS or I Session)

Definition:
A Session that is created by an exchange of messages in the Signaling Plane the first of which is a SIP INVITE transaction. [Ca02].
Discussion:
An IS is identified by the Call-ID, To-tag, and From-tag of the SIP message that sets them. These three fields are used to identify a SIP Dialog [RFC.3261] (Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston, A., Peterson, J., Sparks, R., Handley, M., and E. Schooler, “SIP: Session Initiation Protocol,” June 2002.).
An IS MAY have associated media. The inclusion of media is test case dependent.
An IS may be in one of several different states: Attempting , Established, Disconnecting.
Measurement Units:
N/A
Issues:
None
See Also:

 TOC 

3.1.6.  Session Attempting State

Definition:
The state in the Signaling Plane after the INVITE is sent by the EA and before the 200 OK is sent by the DUT/SUT.
Discussion:
The Session Attempting State includes possible re-INVITES as well as Redirects. It also includes all INVITES that are rejected for lack of authentication information.
Measurement Units:
N/A.
Issues:
None.
See Also:
Invite-initiated Session
Session Established State
Session Disconnecting State

 TOC 

3.1.7.  Session Established State

Definition:
The state in the Signaling Plane after the 200 OK for the initiating INVITE is sent by the DUT/SUT to the EA that sent the INVITE and before the EA sends a BYE.
Discussion:
The Session Established State includes possible re-INVITES as well as redirects. It also includes all INVITES that are rejected for lack of authentication information.
Measurement Units:
N/A.
Issues:
None.
See Also:
Invite-initiated Session
Session Attempting State
Session Disconnecting State

 TOC 

3.1.8.  Session Disconnecting State

Definition:
The state in the Signaling Plane after a BYE is sent by the EA and before a 200 OK reply to that BYE is received by the EA.
Discussion:
Measurement Units:
N/A.
Issues:
None.
See Also:
Invite-initiated Session
Session Attempting State
Session Established State

 TOC 

3.1.9.  Non-INVITE-initiated Session (NIS or N-Session)

Definition:
A session that is created by an exchange of messages in the Signaling Plane that does not include an initial SIP INVITE message.
Discussion:
An example of an NIS is a Session created by a SUBSCRIBE request.
Measurement Units:
N/A.
Issues:
None.
See Also:

 TOC 

3.1.10.  Registration

Definition:
A NIS whose initial SIP message is a REGISTER request.
Discussion:
Registrations represent a significant part of SIP network traffic and so contribute significantly to the amount of work that some elements of the DUT/SUT must perform. A Registration attempt MAY be sussessful or unsuccessful. A successful registration is determined by receipt of a 200 OK response. An unsuccessful registration is one that does not receive a 200 OK response.
Measurement Units:
N/A.
Issues:
None.
See Also:

 TOC 

3.1.11.  Successful IS Establishment

Definition:
An IS is said to have been successfully established if the following two conditions are met: (1) Sess.sig is in the established state, and (2) The media session described in the body of the signaling message is present.
Discussion:
Measurement Units:
N/A.
Issues:
None.
See Also:

 TOC 

3.1.12.  Unsuccessful IS Establishment

Definition:
An IS attempt is said to have been unsuccessfull if one or both of the following two conditions is met: (1) The session did not transition to the established state, and/or (2) The media session described in the body of the signaling message is not present.
Discussion:
Measurement Units:
N/A.
Issues:
None.
See Also:

 TOC 

3.1.13.  Successful NS Establishment

Definition:
An NS is said to have been successfully established if the non-INVITE request that represented its attempt received a 2xx or 3xx reply from the DUT/SUT before the expiration of the disconnect threshold timer.
Discussion:
Measurement Units:
N/A.
Issues:
None.
See Also:
Disconnection Threshold Timer

 TOC 

3.1.14.  Unsuccessful NS Establishment

Definition:
An NS attempt is unsuccessful if the non-INVITE request that represented its attempt did not receive a 2xx or 3xx reply from the DUT/SUTand.
Discussion:
Measurement Units:
N/A.
Issues:
None.
See Also:

 TOC 

3.1.15.  Successful IS Disconnection

Definition:
An IS disconnect is said to have been successful if the following two conditions apply: (1) There was a 200 OK to the BYE before the expiration of the Disconnection Threshold Timer. (2) There are no more elements of session.med present
Discussion:
Measurement Units:
N/A.
Issues:
None.
See Also:
Disconnection Threshold timer

 TOC 

3.1.16.  Unsuccessful IS Disconnection

Definition:
An IS disconnect is said to have been unsuccessful if either or both of the following two conditions apply: (1) There was not a 200 OK to the BYE before the expiration of the Disconnection Threshold Timer. (2) There are still some elements of session.med present
Discussion:
It is necessary to define the time period in which success and failure are determined. For example, it may be that the DUT is operating slowly and that it takes several minutes for it to act so as to remove the resources associated with the media. It is necessary to decide how much time is to be used to decide when the disconnect has failed.
Measurement Units:
N/A.
Issues:
None.
See Also:
Disconnect Threshold

 TOC 

3.2.  Test Components


 TOC 

3.2.1.  Emulated Agent

Definition:
Device in test topology that initates/responds to SIP messages as a session endpoint and sources/receives associated media for established connections.
Discussion:
The Emulated Agent (EA) is a function of the Tester. The Emulated Agent functions on the Signaling and Media Planes. The Tester MAY be configured to be multiple EAs.
Measurement Units:
N/A
Issues:
None.
See Also:
Media Plane
Signaling Plane

 TOC 

3.2.2.  Signaling Server

Definition:
Device in test topology that acts as a proxy on the `Signaling Plane between Emulated Agents. This device is either a DUT or component of a SUT.
Discussion:
The Signalling Server MUST be a RFC 3261 [RFC.3261] (Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston, A., Peterson, J., Sparks, R., Handley, M., and E. Schooler, “SIP: Session Initiation Protocol,” June 2002.) compliant device. It MAY be a Proxy Server, Session Border Controller (SBC), or other type of device that is RFC 3261 compliant. The Signalling Server functions only on the Signaling Plane.
Measurement Units:
NA
Issues:
None.
See Also:
Signaling Plane

 TOC 

3.2.3.  SIP-Aware Stateful Firewall

Definition:
Device in test topology that provides Denial-of-Service (DoS) Protection to the Signaling and Media Planes for the Emulated Agents and Signalling Server
Discussion:
The SIP-Aware Stateful Firewall MAY be an internal component or function of the Session Server. The SIP-Aware Stateful Firewall MAY be a standalone device. If it is a standalone device it MUST be paired with a Signalling Server. If it is a standalone device it MUST be benchmarked as a SUT. SIP-Aware Stateful Firewalls MAY include Network Address Translation (NAT) functionality. Ideally, the inclusion of the SIP-Aware Stateful Firewall as a SUT has no degradation to the measured performance benchmark metrics.
Measurement Units:
N/A
Issues:
None.
See Also:

 TOC 

3.2.4.  SIP Transport Protocol

Definition:
The protocol used for transport of the Signaling Plane messages.
Discussion:
Performance benchmarks may vary for the same SIP networking device depending upon whether TCP, UDP, TLS, SCTP, or another transport layer protocol is used. For this reason it MAY be necessary to measure the SIP Performance Benchmarks using these various transport protocols. Performance Benchmarks MUST report the SIP Transport Protocol used to obtain the benchmark results.
Measurement Units:
TCP or UDP
Issues:
None.
See Also:

 TOC 

3.3.  Test Setup Parameters


 TOC 

3.3.1.  IS Attempt

Definition:
An I-Session Attempt is an event in the Signaling Plane identified by the appearance of a new SIP Call-Id.
Discussion:
When successful, the Session Attempt results in the receipt of a 200 OK SIP message and the creation of a Media Session, session.med. This can be represented as a session.sig that produces a session.med and possibly a session.medc. There can be various reasons why a Session Attempt is unsuccessful. These include a lack of available ports on an element of the DUT/SUT or incorrect operation on the DUT/SUT. Unuccessful IS Attempts are identified and counted by the Recorder.
Measurement Units:
NA
Issues:
None.
See Also:
Session

 TOC 

3.3.2.  IS Duration

Definition:
The time for the first SIP message with the session&pos;s Call-ID until the last message for the session with that Call-ID.
Discussion:
The last message may be from either the signaling or the media plane. If it is in the media plane, sess.med(y) it means that some media resources continue to be associated with the session even after the final signalling messages associated with the session have been exchanged. There can be a variety of causes for such a situation, including network and system latency as well as failures to correlate the state machines of the signalling and media processing functions in the system.
The Session Duration configured on the Emulated Agent (EA) is the Intended Session Duration. When benchmarking Session Capacity the value of the Intended Session Duration MUST be infinite for all sessions.
The Session Duration measured on the DUT/SUT is the Measured Session Duration. The value of the Measured Session Duration MAY not equal the Intended Session Duration.
Measurement Units:
seconds
Issues:
None.
See Also:
Session Attempt Rate

 TOC 

3.3.3.  IS Signaling Duration

Definition:
The time from the first SIP message with the session's Call-ID until a 200 OK is sent in response to a BYE message with the same Call-ID as that identifies the session.
Discussion:
The Signaling Duration applies to session.sig
The Signaling Duration configured on the Emulated Agent (EA) is the Intended Signaling Duration. When benchmarking Session Capacity the value of the Intended Signaling Duration MUST be infinite for all sessions.
The Signaling Duration measured on the DUT/SUT is the Measured Signaling Duration. The value of the Measured Signaling Duration MAY not equal the Intended Signaling Duration.
Measurement Units:
Seconds
Issues:
None.
See Also:
Emulated Agent
DUT/SUT defined in companion methodology document

 TOC 

3.3.4.  IS Media Duration

Definition:
The time from the from the first media message created as defined in the SDP from the Signaling Plane until the last media message.
Discussion:
The Media Duration applies to session.med and session.medc.
Generally the first media message appears after the receipt of the 200 OK response to the SIP INVITE that bears the Call-Id, and before the sending of the ACK.
Measurement Units:
Seconds
Issues:
None.
See Also:

 TOC 

3.3.5.  Session Attempt Rate

Definition:
Configuration on the Emulated Agent for number of sessions to be established at the DUT per continuous one-second intervals.
Discussion:
The Session Attempt Rate can cause variation in performance benchmark measurements. Since this is the number of sessions configured on the Tester, some sessions may not be successfully established on the DUT. A sessio may be either an IS or an NIS.
For a fixed value of Session Attempt Rate, more stress may be incurred on the DUT/SUT when it processes sessions attempts and teardowns concurrently during each one-second interval.
Measurement Units:
session attempts per second (saps)
Issues:
None.
See Also:

 TOC 

3.3.6.  Media-Session Attempt Rate

Definition:
Configuration on the Emulated Agent for number of I-sessions to be established at the DUT per continuous one-second intervals.
Discussion:
The Media Session Attempt Rate can cause variation in performance benchmark measurements. This is the number of sessions configured on the Tester. Some attempts may not result in successful sessions established on the DUT. Media Sessions MUST be associated with an IS. By including this definition we leave open the possibility that there may be an IS that does not include a media description. In this document, however, we will assume that there is a one to one correspondence between IS session attempts and Media Session attempts.
Measurement Units:
session attempts per second (saps)
Issues:
None.
See Also:
I-Session

 TOC 

3.3.7.  NI-Session Attempt Rate

Definition:
Configuration on the Emulated Agent for number of NI-sessions to be established at the DUT per continuous one-second intervals.
Discussion:
The NI-session attempts include registrations, Instant Messages, and Presence-related messages.
Measurement Units:
session attempts per second (saps)
Issues:
None.
See Also:
Session Attempt Rate

 TOC 

3.3.8.  Disconnect Threshold

Definition:
Configuration on the Tester representing the amount of time that an actor (EA or DUT/SUT) will wait before declaring a disconnect failure.
Discussion:
This time duration is test dependent.
Measurement Units:
seconds
Issues:
None.
See Also:
session disconnect failure

 TOC 

3.3.9.  Establishment Threshold

Definition:
Configuration on the Tester representing the amount of time that an actor (EA or DUT/SUT) will wait before declaring a session establishment failure.
Discussion:
This time duration is test dependent.
Measurement Units:
seconds
Issues:
None.
See Also:
session establishment failure

 TOC 

3.3.10.  Media Packet Size

Definition:
Configuration on the Emulated Agent for a fixed size of packets used for media streams.
Discussion:
For a single benchmark test, all sessions use the same size packet for media streams. The size of packets can cause variation in performance benchmark measurements.
Measurement Units:
bytes
Issues:
None.
See Also:

 TOC 

3.3.11.  Media Offered Load, per Media Stream

Definition:
The constant amount of media traffic offered by the Emulated Agent to the DUT/SUT for each media stream.
Discussion:
For a single benchmark test, all sessions use the same Media Offered Load, per Media Stream.
Measurement Units:
packets per seccond (pps)
Issues:
None.
See Also:

 TOC 

3.3.12.  Media Offered Load, Aggregate

Definition:
The total amount of media traffic offered by the Emulated Agent to the DUT/SUT.
Discussion:
Measurement Units:
pps
Issues:
None.
See Also:

 TOC 

3.3.13.  Media Session Hold Time

Definition:
The amount of time during which media flows from the Tester to the DUT for a successful IS.
Discussion:
Measurement Units:
seconds
Issues:
None.
See Also:

 TOC 

3.3.14.  Loop Detection Option

Definition:
An option that causes a Proxy to check for loops in the routing of a SIP request before forwarding the request.
Discussion:
This is an optional process that a SIP proxy may employ. The method used to implement this option is not standardized. The option is described under Proxy Behavior in RFC 3261 in Section 16.3 Request Validation and that section also contains suggestions as to how the option could be implemented. Any procedure to detect loops will use processor cycles and hence could impact the performance of a proxy.
Measurement Units:
NA
Issues:
None.
See Also:

 TOC 

3.3.15.  Forking Option

Definition:
An option that enables a Proxy to fork requests to more than one destination.
Discussion:
This is an optional process that a SIP proxy may employ. The method used to implement this option is not standardized. The option is described under Proxy Behavior in RFC 3261 in Section 16.1. A proxy that uses forking must maintain state information and this will use processor cycles and memory. Thus the use of this option could impact the performance of a proxy and different implementations could produce different impacts.
Measurement Units:
seconds
Issues:
None.
See Also:

 TOC 

3.4.  Benchmarks


 TOC 

3.4.1.  Maximum Registration Rate

Definition:
The maximum number of registrations that can be successfully completed by the DUT/SUT in a given time period.
Discussion:
This benchmark is obtained with zero failure in which 100% of the registrations attempted by the Emulated Agent are successfully completed by the DUT/SUT. The maximum value is obtained by testing to failure. This means that the registration rate provisioned on the EA is raised progressively until a registration attempt failure is observed.
Measurement Units:
registrations per second (rps)
Issues:
None.
See Also:

 TOC 

3.4.2.  Maximum Session Establishsment Rate

Definition:
Maximum rate at which sessions can be successfully established. This is the maximum number of Sessions successfully established in continuous one-second intervals with the sessions remaining active.
Discussion:
This benchmark is obtained with zero failure in which 100% of the sessions introduced by the Emulated Agent are successfully established. The maximum value is obtained by testing to failure. This means that the session rate provisioned on the EA is raised progressively until a session establishment failure is observed. Sessions may be I-Sessions or NI-Sessions or a mix of both and will be defined in the particular test.
Measurement Units:
sessions per second (sps)
Issues:
None.
See Also:
I-Session
NI-Session
Session Attempt Rate

 TOC 

3.4.3.  Session Capacity

Definition:
The maximum number of sessions in the established state that can exist simultaneously on the DUT/SUT.
Discussion:
In order to make this measurement, the Session Duration MUST be set to infinity so that sessions remain established for the duration of the test. The Session Capacity must be reported with the Session Rate used to reach the maximum. Since Session Rate is a zero-loss measurement, there must be zero failures to achieve the Session Capacity. Sessions may be I-Sessions or NI-Sessions.
Measurement Units:
sessions
Issues:
None.
See Also:
Session Attempt Rate

 TOC 

3.4.4.  Session Establishment Performance

Definition:
The percentage of sessions that successfully establish for the duration of a benchmarking test as compared to the number of sessions attempted during that benchmarking test.
Discussion:
Session Establishment Performance is a benchmark to indicate session establishment success for the duration of a test. The duration for measuring this benchmark is to be specified in the Methodology. The Session Duration may be configured so that sessions terminate during the test duration. Established Sessions MAY be reported as the percentage of Session Attempts that failed or the percentage of Session Attempts that were successful.
Measurement Units:
Percentage, %
Issues:
None.
See Also:
Maximum Session Establishment Rate
Session Attempt Rate

 TOC 

3.4.5.  Session Attempt Delay

Definition:
The average time for a session to move from the attempting state to the established state.
Discussion:
Time is measured from when the EA sends the first INVITE for the call-ID in the case of an I-session. Time is measured from when the EA sends the first non-INVITE message in the case of an NI-session. Session Attempt Delay MUST be measured for every established session to calculate the average. Session Attempt Delay MUST be measured at the Maximum Session Establishment Rate.
Measurement Units:
msec
Issues:
None.
See Also:
Maximum Session Establishment Rate

 TOC 

3.4.6.  Session Disconnect Delay

Definition:
The average time that a session stays in the disconnecting state.
Discussion:
Time is measured from when the Emulated Agent sends the BYE request. Session Teardown Delay MUST be measured for every established session to calculate the average. Session Attempt Delay MUST be measured with the rate of teardowns configured to the value of the Maximum Session Establishment Rate.
Measurement Units:
msec
Issues:
None.
See Also:
Maximum Session Establishment Rate

 TOC 

3.4.7.  Standing Sessions

Definition:
The number of Sessions currently established on the DUT/SUT at any instant.
Discussion:
The number of Standing Sessions is influenced by the Session Duration and the Session Attempt Rate. Benchmarks MUST be reported with the maximum and average Standing Sessions for the DUT/SUT. In order to determine the maximum and average Standing Sessions on the DUT/SUT for the duration of the test it is necessary to make periodic measurements of the number of Standing Sessions on the DUT/SUT. The recommended value for the measurement period is 1 second.
Measurement Units:
Number of sessions
Issues:
None.
See Also:
Session Duration
Session Rate
Session Attempt Rate

 TOC 

3.4.8.  IM Rate

Definition:
Maximum number of IM messages completed successfully by the DUT/SUT.
Discussion:
For a UAS, the definition of success is the receipt of an IM request and the subsequent sending of a final response.
For a UAC, the definition of success is the sending of an IM request and the receipt of a final response to it. For a proxy, the definition of success is as follows:
A.
the number of IM requests it receives from the upstream client MUST be equal to the number of IM requests it sent to the downstream server; and
B.
the number of IM responses it receives from the downstream server MUST be equal to the number of IM requests sent to the downstream server; and
C.
the number of IM responses it sends to the upstream client MUST be equal to the number of IM requests it received from the upstream client.
Measurement Units:
IM messages per second
Issues:
None.
See Also:

 TOC 

3.4.9.  Presence Rate

Definition:
Maximum number of presence notifications sent out by the DUT/SUR acting as a Presence Agent [Ro04].
Discussion:
The intent of this benchmark is to assess the throughput of a Presence Agent (PA, see [Ro04]). The PA will accept subscriptions from watchers, and when the target of the subscription is registered with the PA (who is acting as a registrar), a notification is generated to the watcher. This benchmark will use the presence event package as documented in [Ro04]. The Presence Rate will be less than or equal to the Registration Rate.
Measurement Units:
Presence Notifications Per Second
See Also:
Registration Rate.

 TOC 

4.  IANA Considerations

This document requires no IANA considerations.


 TOC 

5.  Security Considerations

Documents of this type do not directly affect the security of Internet or corporate networks as long as benchmarking is not performed on devices or systems connected to production networks. Security threats and how to counter these in SIP and the media layer is discussed in RFC3261, RFC3550, and RFC3711 and various other drafts. This document attempts to formalize a set of common terminology for benchmarking SIP networks.


 TOC 

6.  Acknowledgments

The authors would like to thank Keith Drage and Daryl Malas for their contributions to this document.


 TOC 

7.  References


 TOC 

7.1. Normative References

[RFC.1242] Bradner, S., “Benchmarking Terminology for Network Interconnection Devices,” RFC 1242, July 1991 (TXT).
[RFC.2544] Bradner, S. and J. McQuaid, “Benchmarking Methodology for Network Interconnection Devices,” RFC 2544, July 1999 (TXT).
[RFC.3428] Campbell, B., “Session Initiation Protocol (SIP) Extension for Instant Messaging,” RFC 3428, December 2002 (TXT).
[RFC.4083] Garcia-Martin, M., “Input 3rd-Generation Partnership Project (3GPP) Release 5 Requirements on the Session Initiation Protocol (SIP),” RFC 4083, May 2005 (TXT).
[RFC.4566] Handley, M., “SDP: Session Description Protocol,” RFC 4566, July 2006 (TXT).
[I-D.sip-mib for SIP] Lingle, K., Mule, J., Maeng, J., and D. Walker, “Management Information Base for the Session Initiation Protocol (SIP),” draft-ietf-sip-mib-10 (work in progress), March 2006 (TXT).
[RFC.2285] Mandeville, R., “Benchmarking Terminology for LAN Switching Devices,” RFC 2285, February 1998 (TXT).
[I-D.malas] Malas, D., “SIP Performance Metrics,” draft-malas-performance-metrics-01 (work in progress), March 2007 (TXT).
[SIP Benchmarking methodology] Poretsky, S., Gurbani, V., and C. Davids, “SIP Performance Benchmarking Terminology,” draft-poretsky-bmwg-sip-meth-00 (work in progress), March 2007 (TXT).
[RFC.3261] Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston, A., Peterson, J., Sparks, R., Handley, M., and E. Schooler, “SIP: Session Initiation Protocol,” RFC 3261, June 2002 (TXT).
[RFC.3856] Rosenberg, J., “A Presence Event Package for the Session Initiation Protocol (SIP),” RFC 3856, August 2004 (TXT).
[RFC.3264] Schulzrinne, H. and J. Rosenberge, “An Offer/Answer Model with the Session Description Protocol (SDP),” RFC 3264, June 2002 (TXT).
[RFC.3550] Schulzrinne, H., “RTP: A Transport Protocol for Real-Time Applications,” RFC 3550, July 2003 (TXT).
[RFC.4475] Sparks, R., “Session Initiation Protocol (SIP) Torture Test Messages,” RFC 4475, March 2007 (TXT).

 TOC 

7.2. Informational References


 TOC 

Authors' Addresses

  Scott Porersky
  Reef Point Networks
  8 New England and Executive Park
  Burlington, MA 08103
  USA
Phone:  +1 508 439 9008
Email:  sporetsky@reefpoint.com
  
  Vijay K. Gurbani
  Bell Laboratories, Alcatel-Lucent
  2701 Lucent Lane
  Rm 9F-546
  Lisle, IL 60532
  USA
Phone:  +1 630 224 0216
Email:  vkg@alcatel-lucent.com
  
  Carol Davids
  Illinois Institute of Technology
  201 East Loop Road
  Wheaton, IL 60187
  USA
Email:  davids@iit.edu

 TOC 

Full Copyright Statement

Copyright © The IETF Trust (2007).

This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights.

This document and the information contained herein are provided on an “AS IS” basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Intellectual Property

The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79.

Copies of IPR disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr.

The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at ietf-ipr@ietf.org.