Framework for Performance Metric Development
draft-morton-perf-metrics-framework-00

Status of this Memo

By submitting this Internet-Draft, each author represents that any applicable patent or other IPR claims of which he or she is aware have been or will be disclosed, and any of which he or she becomes aware will be disclosed, in accordance with Section 6 of BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as “work in progress.”

The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt.

The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html.

This Internet-Draft will expire on April 19, 2008.

Abstract

This memo describes a framework and guidelines for the development of performance metrics that are beyond the scope of existing working group charters in the IETF. In this version, the memo refers to a Performance Metrics Entity, or PM Entity, which may in future be a working group or directorate or a combination of these two.

Requirements Language

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 (Bradner, S., “Key words for use in RFCs to Indicate Requirement Levels,” March 1997.) [RFC2119].

Table of Contents

1.  Introduction
    1.1.  Background and Motivation
    1.2.  Organization of this memo
2.  Purpose and Scope
3.  Metrics Development
    3.1.  Definitions of a Metric
    3.2.  Composed Metrics
    3.3.  Metric Specification
    3.4.  Classes of Metrics
    3.5.  Qualifying Metrics
    3.6.  Reporting Models
4.  Performance Metric Development Process
    4.1.  New Proposals for Metrics
    4.2.  Proposal Approval
    4.3.  PM Entity Interaction with other WGs
    4.4.  Standards Track Performance Metrics
5.  IANA Considerations
6.  Security Considerations
7.  Acknowledgements
8.  References
    8.1.  Normative References
    8.2.  Informative References
§  Authors' Addresses
§  Intellectual Property and Copyright Statements

1.  Introduction

There have always been some uncertainties about the performance and suitability of new technologies and applications for their intended audience, and the Internet is no exception. Most uncertainties are effectively addressed through quantified assessment of key performance indicators. Standardized performance metrics add the desirable features of consistent implementation, interpretation, and comparison.

There are at least three phases in the development of performance standards. They are:

1. Definition of a Performance Metric and its units of measure
2. Specification of a Method of Measurement
3. Specification of the Reporting Format

In some metric development activites, there are additional steps, such as setting numerical requirements or objectives. This memo will focus on metric definition, but it is worth noting that the products of this phase benefit when the challenges of measurement are considered.

In this version, the memo refers to a Performance Metrics Entity, or PM Entity, which may in future be a working group or directorate or a combination of these two.

1.1.  Background and Motivation

Although the IETF has two Working Groups dedicated to the development of performance metrics, they each have strict limitations in their charters:

- The Benchmarking Methodology WG has addressed a range of networking technologies and protocols in their long history (such as IEEE 802.3, ATM, Frame Relay, and Routing Protocols), but the charter strictly limits their performance characterizations to the laboratory environment.

- The IP Performance Metrics WG has the mandate to develop metrics applicable to live IP networks, but it is specifically prohibited from developing metrics that characterize traffic (such as a VoIP stream).

A BOF held at IETF-69 introduced the IETF community to the possibility of a generalized activity to define standardized performance metrics. The existence of a growing list of Internet-Drafts on performance metrics (with community interest in development, but in un-chartered areas) illustrates the need for additional performance work. The majority of people present at the BOF supported the proposition that IETF should be working in these areas, and no one objected to any of the proposals.

The IETF does have current and completed activities related to the reporting of application performance metrics (e.g. RAQMON) and is also actively involved in the development of reliable transport protocols which would affect the relationship between IP performance and application performance.

Thus there is a gap in the currently chartered coverage of IETF WGs: development of performance metrics for non-IP-layer protocols that can be used to characterize performance on live networks.

1.2.  Organization of this memo

This memo is divided in two major sections beyond the Purpose and Scope section. The first is a definition and description of a performance metric and its key aspects. The second defines a process to develop these metrics that is applicable to the IETF environment.

2.  Purpose and Scope

The purpose of this memo is to define a framework and a process for developing performance metrics in the IETF.

The scope includes metric definition for any protocol developed by the IETF. However, other frameworks exist for IETF chartered work [RFC2330] (Paxson, V., Almes, G., Mahdavi, J., and M. Mathis, “Framework for IP Performance Metrics,” May 1998.), and this memo is not intended to superceed the existing working methods.

Question: How do we write this scope so as not to conflict with IPPM and BMWG? Can we say that this framework may be useful to those activities as well, and leave it at that? Perhaps this will do:

This process is not intended to govern performance metric development in existing IETF WG, such as IPPM and BMWG. However, the framework and guidelines may be useful in these activities, and MAY be applied where appropriate.

3.  Metrics Development

This section provides key definitions and qualifications of performance metrics.

3.1.  Definitions of a Metric

A metric is a measure of an observable behavior of an application, protocol or other system. The definition of a metric often assumes some implicit or explicit underlying statistical process, and a metric is an estimate of a parameter of this process. If the assumed statistical process closely models the behavior of the system then the metric is "better" in the sense that it more accurately characterizes the state or behavior of the system.

A metric should serve some defined purpose. This may include the measurement of capacity, quantifying how bad some problem is, measurement of service level, problem diagnosis or location and other such uses. A metric may also be an input to some other process, for example the computation of a composite metric or a model or simulation of a system. Tests of the "usefulness" of a metric include:

(i) the degree to which its absence would cause significant loss of information on the behavior or state of the application or system being measured

(ii) the correlation between the metric and the quality of service / experience delivered to the user (person or other application)

(iii) the degree to which the metric is able to support the identification and location of problems affecting service quality.

For example, consider a distributed application operating over a network connection that is subject to packet loss. A Packet Loss Rate (PLR) metric is defined as the mean packet loss rate over some time period. If the application performs poorly over network connections with high packet loss rate and always performs well when the packet loss rate is zero then the PLR metric is useful to some degree. Some applications are sensitive to short periods of high loss (bursty loss). If both bursty and independent loss occur, it is possible that there may be periods during which the PLR metric would be the same however the application performance would be different - i.e. PLR is weakly correlated with application performance - which would suggest that the metric is weak.

3.2.  Composed Metrics

Some metrics may not be measured directly, but may be composed from metrics that have been measured. Usually the contribution metrics have a limited scope in time or space, and they can be combined to estimate the performance of some larger entity.

[I‑D.ietf‑ippm‑framework‑compagg] (Morton, A., “Framework for Metric Composition,” December 2009.) gives the framework for three types of composed metrics: Temporal Aggregation, Spatial Aggregation, and Spatial Composition.

Spatial Composition is described in [I‑D.ietf‑ippm‑spatial‑composition] (Morton, A. and E. Stephan, “Spatial Composition of Metrics,” April 2010.). Other memos in this framework are TBD.

>>>Not sure what's intended in this next one >>>

- Derivative / Composite metrics (e.g. combined effect of several different metrics)

3.3.  Metric Specification

Process for specifying a metric

3.4.  Classes of Metrics

Simplify process by identifying classes of metric

3.5.  Qualifying Metrics

Each metric SHOULD be assessed according to the following list of qualifications:

• Unambiguously defined?
• Units of Measure Specified?
• Measurement Errors Identified?
• Repeatable?
• Implementable?
• Assumptions concerning underlying process?
• Use cases?
• Correlation with application performance/ user experience?

(not sure that the last one is essential, it can be useful to characterize a network attribute without linking it to application performance/user experience)

3.6.  Reporting Models

A metric, or some set of metrics, may be measured over some time period, measured continuously, sampled, aggregated and/or combined into composite metrics and then reported using a "push" or "pull" model. Reporting protocols typically introduce some limitations and assumptions with regard to the definition of a metric.

4.  Performance Metric Development Process

4.1.  New Proposals for Metrics

The following entry criteria will be considered for each proposal.

Proposals SHOULD be prepared as Internet Drafts, describing the metrics and conforming to the qualifications above as much as possible.

Proposals SHOULD be vetted by the corresponding protocol development Working Group prior to discussion by the PM Entity. This aspect of the process includes an assessment of the need for the metrics proposed and assessment of the support for their development in IETF.

Proposals SHOULD include an assessment of interaction and/or overlap with work in other Standards Development Organizations.

Proposals SHOULD specify the intended audience and users of the metrics. The development process encourages participation by members of the intended audience.

4.2.  Proposal Approval

Who does this???

The IETF/IESG/Relevant ADs/Relevant WG/PM Entity ???

This section depends on the direction of the solution, or form that the PM Entity takes.

4.3.  PM Entity Interaction with other WGs

The PM Entity SHALL work in partnership with the related protocol development WG when considering an Internet Draft that specifies performance metrics for a protocol. A sufficient number of individuals with expertise must be willing to consult on the draft. If the related WG has concluded, comments on the proposal should still be sought from key RFC authors and former chairs, or from the WG mailing list if it was not closed.

A dedicated mailing list MAY be initiated for each work area, so that protocol experts can subscribe to and receive the message traffic that is relevant to their work.

In some cases, it will be appropriate to have the IETF session discussion during the related protocol WG session, to maximize visibility of the effort to that WG and expand the review.

4.4.  Standards Track Performance Metrics

The PM Entity will manage the progression of PM RFCs along the Standards Track. See [I‑D.bradner‑metricstest] (Bradner, S. and V. Paxson, “Advancement of metrics specifications on the IETF Standards Track,” August 2007.). This may include the preparation of test plans to examine different implementations of the metrics to ensure that the metric definitions are clear and unambiguous (depending on the final form of the draft above).

5.  IANA Considerations

This document makes no request of IANA.

Note to RFC Editor: this section may be removed on publication as an RFC.

6.  Security Considerations

In general, the existence of framework for performance metric development does not constitute a security issue for the Internet.

The security considerations that apply to any active measurement of live networks are relevant here as well. See [RFC4656] (Shalunov, S., Teitelbaum, B., Karp, A., Boote, J., and M. Zekauskas, “A One-way Active Measurement Protocol (OWAMP),” September 2006.).

7.  Acknowledgements

The authors would like to thank

8.  References

8.1. Normative References

8.2. Informative References

Authors' Addresses

Full Copyright Statement

Copyright © The IETF Trust (2007).

This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights.

This document and the information contained herein are provided on an “AS IS” basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Intellectual Property

The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79.

Copies of IPR disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr.

The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at ietf-ipr@ietf.org.