draft-ietf-bmwg-sip-bench-term-04.txt   draft-ietf-bmwg-sip-bench-term-05.txt 
Benchmarking Methodology Working C. Davids Benchmarking Methodology Working Group C. Davids
Group Illinois Institute of Technology Internet-Draft Illinois Institute of Technology
Internet-Draft V. Gurbani Expires: April 25, 2013 V. Gurbani
Expires: September 13, 2012 Bell Laboratories, Alcatel-Lucent Bell Laboratories,
Alcatel-Lucent
S. Poretsky S. Poretsky
Allot Communications Allot Communications
March 12, 2012 October 22, 2012
Terminology for Benchmarking Session Initiation Protocol (SIP) Terminology for Benchmarking Session Initiation Protocol (SIP)
Networking Devices Networking Devices
draft-ietf-bmwg-sip-bench-term-04 draft-ietf-bmwg-sip-bench-term-05
Abstract Abstract
This document provides a terminology for benchmarking SIP performance This document provides a terminology for benchmarking the SIP
in networking devices. Terms are included for test components, test performance of networking devices. The term performance in this
context means the capacity of the device- or system-under-test to
process SIP messages. Terms are included for test components, test
setup parameters, and performance benchmark metrics for black-box setup parameters, and performance benchmark metrics for black-box
benchmarking of SIP networking devices. The performance benchmark benchmarking of SIP networking devices. The performance benchmark
metrics are obtained for the SIP control plane and media plane. The metrics are obtained for the SIP signaling plane only. The terms are
terms are intended for use in a companion methodology document for intended for use in a companion methodology document for
complete performance characterization of a device in a variety of characterizing the performance of a SIP networking device under a
conditions making it possible to compare performance of different variety of conditions. The intent of the two documents is to enable
devices. It is critical to provide test setup parameters and a a comparison of the capacity of SIP networking devices. Test setup
methodology document for SIP performance benchmarking because SIP parameters and a methodology document are necessary because SIP
allows a wide range of configuration and operational conditions that allows a wide range of configuration and operational conditions that
can influence performance benchmark measurements. It is necessary to can influence performance benchmark measurements. A standard
have terminology and methodology standards to ensure that reported terminology and methodology will ensure that benchmarks have
benchmarks have consistent definition and were obtained following the consistent definition and were obtained following the same
same procedures. Benchmarks can be applied to compare performance of procedures.
a variety of SIP networking devices.
Status of this Memo Status of this Memo
This Internet-Draft is submitted in full conformance with the This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79. provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/. Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on April 25, 2013.
This Internet-Draft will expire on September 13, 2012.
Copyright Notice Copyright Notice
Copyright (c) 2012 IETF Trust and the persons identified as the Copyright (c) 2012 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License. described in the Simplified BSD License.
Table of Contents Table of Contents
1. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4 1. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 5
2. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 5 2. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.1. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.1. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.2. Benchmarking Models . . . . . . . . . . . . . . . . . . . 7 2.2. Benchmarking Models . . . . . . . . . . . . . . . . . . . 9
3. Term Definitions . . . . . . . . . . . . . . . . . . . . . . . 13 3. Term Definitions . . . . . . . . . . . . . . . . . . . . . . . 14
3.1. Protocol Components . . . . . . . . . . . . . . . . . . . 13 3.1. Protocol Components . . . . . . . . . . . . . . . . . . . 14
3.1.1. Session . . . . . . . . . . . . . . . . . . . . . . . 13 3.1.1. Session . . . . . . . . . . . . . . . . . . . . . . . 14
3.1.2. Signaling Plane . . . . . . . . . . . . . . . . . . . 16 3.1.2. Signaling Plane . . . . . . . . . . . . . . . . . . . 17
3.1.3. Media Plane . . . . . . . . . . . . . . . . . . . . . 16 3.1.3. Media Plane . . . . . . . . . . . . . . . . . . . . . 18
3.1.4. Associated Media . . . . . . . . . . . . . . . . . . . 17 3.1.4. Associated Media . . . . . . . . . . . . . . . . . . . 18
3.1.5. Overload . . . . . . . . . . . . . . . . . . . . . . . 17 3.1.5. Overload . . . . . . . . . . . . . . . . . . . . . . . 19
3.1.6. Session Attempt . . . . . . . . . . . . . . . . . . . 18 3.1.6. Session Attempt . . . . . . . . . . . . . . . . . . . 20
3.1.7. Established Session . . . . . . . . . . . . . . . . . 18 3.1.7. Established Session . . . . . . . . . . . . . . . . . 20
3.1.8. Invite-initiated Session (IS) . . . . . . . . . . . . 19 3.1.8. Invite-initiated Session (IS) . . . . . . . . . . . . 21
3.1.9. Non-INVITE-initiated Session (NS) . . . . . . . . . . 20 3.1.9. Non-INVITE-initiated Session (NS) . . . . . . . . . . 22
3.1.10. Session Attempt Failure . . . . . . . . . . . . . . . 20 3.1.10. Session Attempt Failure . . . . . . . . . . . . . . . 22
3.1.11. Standing Sessions Count . . . . . . . . . . . . . . . 21 3.1.11. Standing Sessions Count . . . . . . . . . . . . . . . 23
3.2. Test Components . . . . . . . . . . . . . . . . . . . . . 21 3.2. Test Components . . . . . . . . . . . . . . . . . . . . . 23
3.2.1. Emulated Agent . . . . . . . . . . . . . . . . . . . . 21 3.2.1. Emulated Agent . . . . . . . . . . . . . . . . . . . . 24
3.2.2. Signaling Server . . . . . . . . . . . . . . . . . . . 22 3.2.2. Signaling Server . . . . . . . . . . . . . . . . . . . 24
3.2.3. SIP-Aware Stateful Firewall . . . . . . . . . . . . . 22 3.2.3. SIP-Aware Stateful Firewall . . . . . . . . . . . . . 24
3.2.4. SIP Transport Protocol . . . . . . . . . . . . . . . . 23 3.2.4. SIP Transport Protocol . . . . . . . . . . . . . . . . 25
3.3. Test Setup Parameters . . . . . . . . . . . . . . . . . . 24 3.3. Test Setup Parameters . . . . . . . . . . . . . . . . . . 26
3.3.1. Session Attempt Rate . . . . . . . . . . . . . . . . . 24 3.3.1. Session Attempt Rate . . . . . . . . . . . . . . . . . 26
3.3.2. IS Media Attempt Rate . . . . . . . . . . . . . . . . 24 3.3.2. IS Media Attempt Rate . . . . . . . . . . . . . . . . 26
3.3.3. Establishment Threshold Time . . . . . . . . . . . . . 25 3.3.3. Establishment Threshold Time . . . . . . . . . . . . . 27
3.3.4. Session Duration . . . . . . . . . . . . . . . . . . . 25 3.3.4. Session Duration . . . . . . . . . . . . . . . . . . . 27
3.3.5. Media Packet Size . . . . . . . . . . . . . . . . . . 26 3.3.5. Media Packet Size . . . . . . . . . . . . . . . . . . 28
3.3.6. Media Offered Load . . . . . . . . . . . . . . . . . . 26 3.3.6. Media Offered Load . . . . . . . . . . . . . . . . . . 28
3.3.7. Media Session Hold Time . . . . . . . . . . . . . . . 27 3.3.7. Media Session Hold Time . . . . . . . . . . . . . . . 29
3.3.8. Loop Detection Option . . . . . . . . . . . . . . . . 27 3.3.8. Loop Detection Option . . . . . . . . . . . . . . . . 29
3.3.9. Forking Option . . . . . . . . . . . . . . . . . . . . 28 3.3.9. Forking Option . . . . . . . . . . . . . . . . . . . . 30
3.4. Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . 29 3.4. Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . 31
3.4.1. Registration Rate . . . . . . . . . . . . . . . . . . 29 3.4.1. Registration Rate . . . . . . . . . . . . . . . . . . 31
3.4.2. Session Establishment Rate . . . . . . . . . . . . . . 29 3.4.2. Session Establishment Rate . . . . . . . . . . . . . . 31
3.4.3. Session Capacity . . . . . . . . . . . . . . . . . . . 30 3.4.3. Session Capacity . . . . . . . . . . . . . . . . . . . 32
3.4.4. Session Overload Capacity . . . . . . . . . . . . . . 31 3.4.4. Session Overload Capacity . . . . . . . . . . . . . . 33
3.4.5. Session Establishment Performance . . . . . . . . . . 31 3.4.5. Session Establishment Performance . . . . . . . . . . 33
3.4.6. Session Attempt Delay . . . . . . . . . . . . . . . . 32 3.4.6. Session Attempt Delay . . . . . . . . . . . . . . . . 34
3.4.7. IM Rate . . . . . . . . . . . . . . . . . . . . . . . 32 3.4.7. IM Rate . . . . . . . . . . . . . . . . . . . . . . . 34
4. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 33 4. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 35
5. Security Considerations . . . . . . . . . . . . . . . . . . . 33 5. Security Considerations . . . . . . . . . . . . . . . . . . . 35
6. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 34 6. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 36
7. References . . . . . . . . . . . . . . . . . . . . . . . . . . 34 7. References . . . . . . . . . . . . . . . . . . . . . . . . . . 36
7.1. Normative References . . . . . . . . . . . . . . . . . . . 34 7.1. Normative References . . . . . . . . . . . . . . . . . . . 36
7.2. Informational References . . . . . . . . . . . . . . . . . 34 7.2. Informational References . . . . . . . . . . . . . . . . . 36
Appendix A. White Box Benchmarking Terminology . . . . . . . . . 35
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 35 Appendix A. White Box Benchmarking Terminology . . . . . . . . . 37
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 37
1. Terminology 1. Terminology
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in BCP 14, RFC2119 document are to be interpreted as described in BCP 14, RFC2119
[RFC2119]. RFC 2119 defines the use of these key words to help make [RFC2119]. RFC 2119 defines the use of these key words to help make
the intent of standards track documents as clear as possible. While the intent of standards track documents as clear as possible. While
this document uses these keywords, this document is not a standards this document uses these keywords, this document is not a standards
track document. The term Throughput is defined in RFC2544 [RFC2544]. track document. The term Throughput is defined in RFC2544 [RFC2544].
For the sake of clarity and continuity, this document adopts the For the sake of clarity and continuity, this document adopts the
template for definitions set out in Section 2 of RFC 1242 [RFC1242]. template for definitions set out in Section 2 of RFC 1242 [RFC1242].
Definitions are indexed and grouped together for ease of reference.
This document uses existing terminology defined in other BMWG work. The terms Device Under Test (DUT) and System Under Test (SUT) are
Examples include, but are not limited to: defined in the following BMWG documents:
Device under test (DUT) (c.f., Section 3.1.1 RFC 2285 [RFC2285]). Device Under Test (DUT) (c.f., Section 3.1.1 RFC 2285 [RFC2285]).
System under test (SUT) (c.f., Section 3.1.2, RFC 2285 [RFC2285]). System Under Test (SUT) (c.f., Section 3.1.2, RFC 2285 [RFC2285]).
Many commonly used SIP terms in this document are defined in RFC 3261 Many commonly used SIP terms in this document are defined in RFC 3261
[RFC3261]. For convenience the most important of these are [RFC3261]. For convenience the most important of these are
reproduced below. Use of these terms in this document is consistent reproduced below. Use of these terms in this document is consistent
with their corresponding definition in [RFC3261]. with their corresponding definition in [RFC3261].
o Call Stateful: A proxy is call stateful if it retains state for a o Call Stateful: A proxy is call stateful if it retains state for a
dialog from the initiating INVITE to the terminating BYE request. dialog from the initiating INVITE to the terminating BYE request.
A call stateful proxy is always transaction stateful, but the A call stateful proxy is always transaction stateful, but the
converse is not necessarily true. converse is not necessarily true.
o Stateful Proxy: A logical entity that maintains the client and o Stateful Proxy: A logical entity that maintains the client and
server transaction state machines defined by this specification server transaction state machines defined by this specification
during the processing of a request, also known as a transaction during the processing of a request, also known as a transaction
stateful proxy. The behavior of a stateful proxy is further stateful proxy. The behavior of a stateful proxy is further
defined in Section 16. A (transaction) stateful proxy is not the defined in Section 16 of RFC 3261 [RFC3261] . A transaction
same as a call stateful proxy. stateful proxy is not the same as a call stateful proxy.
o Stateless Proxy: A logical entity that does not maintain the o Stateless Proxy: A logical entity that does not maintain the
client or server transaction state machines defined in this client or server transaction state machines defined in this
specification when it processes requests. A stateless proxy specification when it processes requests. A stateless proxy
forwards every request it receives downstream and every response forwards every request it receives downstream and every response
it receives upstream. it receives upstream.
o Back-to-back User Agent: A back-to-back user agent (B2BUA) is a o Back-to-back User Agent: A back-to-back user agent (B2BUA) is a
logical entity that receives a request and processes it as a user logical entity that receives a request and processes it as a user
agent server (UAS). In order to determine how the request should agent server (UAS). In order to determine how the request should
be answered, it acts as a user agent client (UAC) and generates be answered, it acts as a user agent client (UAC) and generates
requests. Unlike a proxy server, it maintains dialog state and requests. Unlike a proxy server, it maintains dialog state and
skipping to change at page 5, line 12 skipping to change at page 6, line 12
established. Since it is a concatenation of a UAC and a UAS, no established. Since it is a concatenation of a UAC and a UAS, no
explicit definitions are needed for its behavior. explicit definitions are needed for its behavior.
o Loop: A request that arrives at a proxy, is forwarded, and later o Loop: A request that arrives at a proxy, is forwarded, and later
arrives back at the same proxy. When it arrives the second time, arrives back at the same proxy. When it arrives the second time,
its Request-URI is identical to the first time, and other header its Request-URI is identical to the first time, and other header
fields that affect proxy operation are unchanged, so that the fields that affect proxy operation are unchanged, so that the
proxy will make the same processing decision on the request it proxy will make the same processing decision on the request it
made the first time. Looped requests are errors, and the made the first time. Looped requests are errors, and the
procedures for detecting them and handling them are described by procedures for detecting them and handling them are described by
the protocol. the SIP protocol[RFC3261] and also by RFC 5393
2. Introduction 2. Introduction
Service Providers are now planning Voice Over IP (VoIP) and Service Providers and IT Organizations deliver Voice Over IP (VoIP)
Multimedia network deployments using the IETF developed Session and Multimedia network services based on the IETF Session Initiation
Initiation Protocol (SIP) [RFC3261]. SIP is a signaling protocol Protocol (SIP) [RFC3261]. SIP is a signaling protocol originally
originally intended to be used for the dynamic establishment, intended to be used to dynamically establish, disconnect and modify
disconnection and modification of streams of media between end users. streams of media between end users. As it has evolved it has been
As it has evolved it has been adopted for use in a growing number of adopted for use in a growing number of services and applications.
applications and features. Many of these result in the creation of a Many of these result in the creation of a media session, but some do
media stream, but some do not. Instead, they create other services not. Examples of this latter group include text messaging and
tailored to the end-users' immediate needs or preferences. The set subscription services. The set of benchmarking terms provided in
of benchmarking terms provided in this document is intended for use this document is intended for use with any SIP-enabled device
with any SIP-enabled device performing SIP functions in the interior performing SIP functions in the interior of the network, whether or
of the network. The performance of end-user devices is outside the not these result in the creation of media sessions. The performance
scope of this document. of end-user devices is outside the scope of this document.
VoIP with SIP has led to the development of new networking devices A number of networking devices have been developed to support SIP-
including SIP Server, Session Border Controllers (SBC), Back-to-back based VoIP services. These include SIP Servers, Session Border
user agents (B2BUA) and SIP-Aware Stateful Firewall. The mix of Controllers (SBC), Back-to-back User Agents (B2BUA), and SIP-Aware
voice and IP functions in these various devices has produced Stateful Firewalls. These devices contain a mix of voice and IP
inconsistencies in vendor reported performance metrics and has caused functions whose performance may be reported using metrics defined by
confusion in the service provider community. SIP allows a wide range the equipment manufacturer or vendor. The Service Provider or IT
of configuration and operational conditions that can influence Organization seeking to compare the performance of such devices will
performance benchmark measurements. When the device under test not be able to do so using these vendor-specific metrics, whose
terminates or relays both media and signaling, for example, it is conditions of test and algorithms for collection are often
important to be able to correlate a signaling measurement with the unspecified. SIP functional elements and the devices that include
media plane measurements to determine the system performance. As them can be configured many different ways and can be organized into
devices and their functions proliferate, the need to have a various topologies. These configuration and topological choices
consistent set of metrics to compare their performance becomes impact the value of any chosen signaling benchmark. Unless these
increasingly urgent. This document and its companion methodology conditions-of-test are defined, a true comparison of performance
document [I-D.ietf-bmwg-sip-bench-meth] provide a set of black-box metrics will not be possible. Some SIP-enabled network devices
benchmarks for describing and comparing the performance of devices terminate or relay media as well as signaling. The processing of
that incorporate the SIP User Agent Client and Server functions and media by the device impacts the signaling performance. As a result,
that operate in the network's core. the conditions-of-test must include information as to whether or not
the device under test processes media and if the device does process
media, a description of the media handled and the manner in which it
is handled. This document and its companion methodology document
[I-D.ietf-bmwg-sip-bench-meth] provide a set of black-box benchmarks
for describing and comparing the performance of devices that
incorporate the SIP User Agent Client and Server functions and that
operate in the network's core.
The definition of SIP performance benchmarks necessarily includes The definition of SIP performance benchmarks necessarily includes
definitions of Test Setup Parameters and a test methodology. These definitions of Test Setup Parameters and a test methodology. These
enable the Tester to perform benchmarking tests on different devices enable the Tester to perform benchmarking tests on different devices
and to achieve comparable and repeatable results. This document and to achieve comparable results. This document provides a common
provides a common set of well-defined terms for Test Components, Test set of definitions for Test Components, Test Setup Parameters, and
Setup Parameters, and Benchmarks. All the benchmarks defined are Benchmarks. All the benchmarks defined are black-box measurements of
black-box measurements of the SIP Control (Signaling) plane. The the SIP signaling plane. The Test Setup Parameters and Benchmarks
Test Setup Parameters and Benchmarks defined in this document are defined in this document are intended for use with the companion
intended for use with the companion Methodology document. Benchmarks Methodology document. Benchmarks of internal DUT characteristics
of internal DUT characteristics (also known as white-box benchmarks) (also known as white-box benchmarks) such as Session Attempt Arrival
such as Session Attempt Arrival Rate, which is measured at the DUT, Rate, which is measured at the DUT, are described in Appendix A to
are described in Appendix A to allow additional characterization of allow additional characterization of DUT behavior with different
DUT behavior with different distribution models. distribution models.
2.1. Scope 2.1. Scope
The scope of this work item is summarized as follows: The scope of this work item is summarized as follows:
o This terminology document describes SIP signaling (control- plane) o This terminology document describes SIP signaling performance
performance benchmarks for black-box measurements of SIP benchmarks for black-box measurements of SIP networking devices.
networking devices. Stress and debug scenarios are not addressed Stress and debug scenarios are not addressed in this work item.
in this work item.
o The DUT must be an RFC 3261 capable network equipment. This may o The DUT must be an RFC 3261 capable network equipment. This may
be a Registrar, Redirect Server, Stateless Proxy or Stateful be a Registrar, Redirect Server, Stateless Proxy or Stateful
Proxy. A DUT MAY also include a B2BUA, SBC functionality (this is Proxy. A DUT MAY also include a B2BUA, SBC functionality. The
referred to as the "Signaling Server".) The DUT MAY be a multi- DUT MAY be a multi-port SIP-to-switched network gateway
port SIP-to-switched network gateway implemented as a SIP UAC or implemented as a SIP UAC or UAS.
UAS. o The DUT MAY include an internal SIP Application Level Gateway
o The DUT MAY have an internal SIP Application Level Gateway (ALG), (ALG), firewall, and/or a Network Address Translator (NAT). This
firewall, and/or a Network Address Translator (NAT). This is is referred to as the "SIP Aware Stateful Firewall."
referred to as the "SIP Aware Stateful Firewall."
o The DUT or SUT MUST NOT be end user equipment, such as personal o The DUT or SUT MUST NOT be end user equipment, such as personal
digital assistant, a computer-based client, or a user terminal. digital assistant, a computer-based client, or a user terminal.
o The Tester acts as multiple "Emulated Agents" (EA) that initiate o The Tester acts as multiple "Emulated Agents" (EA) that initiate
(or respond to) SIP messages as session endpoints and source (or (or respond to) SIP messages as session endpoints and source (or
receive) associated media for established connections. receive) associated media for established connections.
o Control Signaling in presence of Media o SIP Signaling in presence of Media
* The media performance is not benchmarked in this work item. * The media performance is not benchmarked in this work item.
* It is RECOMMENDED that control plane benchmarks are performed * It is RECOMMENDED that SIP signaling plane benchmarks be
with media present, but this is optional. performed with media present, but this is optional.
* The SIP INVITE requests MUST include the SDP body. * The SIP INVITE requests MUST include the SDP body.
* The type of DUT dictates whether the associated media streams * The type of DUT dictates whether the associated media streams
traverse the DUT or SUT. Both scenarios are within the scope traverse the DUT or SUT. Both scenarios are within the scope
of this work item. of this work item.
* SIP is frequently used to create media streams; the control * SIP is frequently used to create media streams; the signaling
plane and media plane are treated as orthogonal to each other plane and media plane are treated as orthogonal to each other
in this document. While many devices support the creation of in this document. While many devices support the creation of
media streams, benchmarks that measure the performance of these media streams, benchmarks that measure the performance of these
streams are outside the scope of this document and its streams are outside the scope of this document and its
companion methodology document [I-D.ietf-bmwg-sip-bench-meth]. companion methodology document [I-D.ietf-bmwg-sip-bench-meth].
Tests may be performed with or without the creation of media Tests may be performed with or without the creation of media
streams. The presence or absence of media streams MUST be streams. The presence or absence of media streams MUST be
noted as a condition of the test as the performance of SIP noted as a condition of the test as the performance of SIP
devices may vary accordingly. Even if the media is used during devices may vary accordingly. Even if the media is used during
benchmarking, only the SIP performance will be benchmarked, not benchmarking, only the SIP performance will be benchmarked, not
the media performance or quality. the media performance or quality.
o Both INVITE and non-INVITE scenarios (such as Instant Messages or o Both INVITE and non-INVITE scenarios (such as Instant Messages or
IM) are addressed in this document. However, benchmarking SIP IM) are addressed in this document. However, benchmarking SIP
presence is not a part of this work item. presence is not a part of this work item.
o Different transport mechanisms -- such as UDP, TCP, SCTP, or TLS o Different transport mechanisms -- such as UDP, TCP, SCTP, or TLS
-- may be used; however, the specific transport mechanism MUST be -- may be used. The specific transport mechanism MUST be noted as
noted as a condition of the test as the performance of SIP devices a condition of the test as the performance of SIP devices may vary
may vary accordingly. accordingly.
o Looping and forking options are also considered since they impact o Looping and forking options are also considered since they impact
processing at SIP proxies. processing at SIP proxies.
o REGISTER and INVITE requests may be challenged or remain o REGISTER and INVITE requests may be challenged or remain
unchallenged for authentication purpose as this may impact the unchallenged for authentication purpose. Whether or not the
performance benchmarks. Any observable performance degradation REGISTER and INVITE requests are challenged is a condition of test
due to authentication is of interest to the SIP community. which will be recorded along with other such parameters which may
Whether or not the REGISTER and INVITE requests are challenged is impact the SIP performance of the device or system under test.
a condition of test and will be recorded and reported. o Re-INVITE requests are not considered in scope of this work item
o Re-INVITE requests are not considered in scope of this work item. since the benchmarks for INVITEs are based on the dialog created
by the INVITE and not on the transactions that take place within
that dialog.
o Only session establishment is considered for the performance o Only session establishment is considered for the performance
benchmarks. Session disconnect is not considered in the scope of benchmarks. Session disconnect is not considered in the scope of
this work item. this work item. This is because our goal is to determine the
maximum throughput of the device or system under test, that is the
number of simultaneous SIP sessions that the device or system can
support. It is true that there are BYE requests being created
during the test process. These transactions do contribute to the
load on the device or system under test and thus are accounted for
in the metric we derive. We do not seek a separate metric for the
number of BYE transactions a device or system can support.
o SIP Overload [I-D.ietf-soc-overload-design] is within the scope of o SIP Overload [I-D.ietf-soc-overload-design] is within the scope of
this work item. We test to failure and then can continue to this work item. We test to failure and then can continue to
observe and record the behavior of the system after failures are observe and record the behavior of the system after failures are
recorded. The cause of failure is not within the scope of this recorded. The cause of failure is not within the scope of this
work. We note the failure and may continue to test until a work. We note the failure and may continue to test until a
different failure or condition is encountered. Considerations on different failure or condition is encountered. Considerations on
how to handle overload are deferred to work progressing in the SOC how to handle overload are deferred to work progressing in the SOC
working group [I-D.ietf-soc-overload-control]. Vendors are, of working group [I-D.ietf-soc-overload-control]. Vendors are, of
course, free to implement their specific overload control behavior course, free to implement their specific overload control behavior
as the expected test outcome if it is different from the IETF as the expected test outcome if it is different from the IETF
skipping to change at page 8, line 8 skipping to change at page 9, line 20
2.2. Benchmarking Models 2.2. Benchmarking Models
This section shows ten models to be used when benchmarking SIP This section shows ten models to be used when benchmarking SIP
performance of a networking device. Figure 1 shows shows the performance of a networking device. Figure 1 shows shows the
configuration needed to benchmark the tester itself. This model will configuration needed to benchmark the tester itself. This model will
be used to establish the limitations of the test apparatus. be used to establish the limitations of the test apparatus.
+--------+ Signaling request +--------+ +--------+ Signaling request +--------+
| +----------------------------->| | | +----------------------------->| |
| Tester | | Tester | | Tester | | Tester |
| | Signaling response | EA | | EA | Signaling response | EA |
| |<-----------------------------+ | | |<-----------------------------+ |
+--------+ +--------+ +--------+ +--------+
/|\ /|\ /|\ /|\
| Media | | Media |
+=========================================+ +=========================================+
Figure 1: Baseline performance of the Emulated Agent without a DUT Figure 1: Baseline performance of the Emulated Agent without a DUT
present present
Figure 2 shows the DUT playing the role of a user agent client (UAC), Figure 2 shows the DUT playing the role of a user agent client (UAC),
skipping to change at page 9, line 13 skipping to change at page 10, line 24
without associated media without associated media
Figure 4 shows the DUT plays the role of a user agent client (UAC), Figure 4 shows the DUT plays the role of a user agent client (UAC),
initiating requests and absorbing responses. This model can be used initiating requests and absorbing responses. This model can be used
as a baseline performance for the DUT acting as a UAC with associated as a baseline performance for the DUT acting as a UAC with associated
media. media.
+--------+ Signaling request +--------+ +--------+ Signaling request +--------+
| +----------------------------->| | | +----------------------------->| |
| DUT | | Tester | | DUT | | Tester |
| | Signaling response | EA | | | Signaling response | (EA) |
| |<-----------------------------+ | | |<-----------------------------+ |
| |<============ Media =========>| | | |<============ Media =========>| |
+--------+ +--------+ +--------+ +--------+
Figure 4: Baseline performance for DUT acting as a user agent client Figure 4: Baseline performance for DUT acting as a user agent client
with associated media with associated media
Figure 5 shows the DUT plays the role of a user agent server (UAS), Figure 5 shows the DUT plays the role of a user agent server (UAS),
absorbing the requests and sending responses. This model can be used absorbing the requests and sending responses. This model can be used
as a baseline performance for the DUT acting as a UAS with associated as a baseline performance for the DUT acting as a UAS with associated
media. media.
+--------+ Signaling request +--------+ +--------+ Signaling request +--------+
| +----------------------------->| | | +----------------------------->| |
| Tester | | DUT | | Tester | | DUT |
| EA | Response | | | (EA) | Response | |
| |<-----------------------------+ | | |<-----------------------------+ |
| |<============ Media =========>| | | |<============ Media =========>| |
+--------+ +--------+ +--------+ +--------+
Figure 5: Baseline performance for DUT acting as a user agent server Figure 5: Baseline performance for DUT acting as a user agent server
with associated media with associated media
Figure 6 shows that the Tester acts as the initiating and responding Figure 6 shows that the Tester acts as the initiating and responding
EA as the DUT/SUT forwards Session Attempts. EA as the DUT/SUT forwards Session Attempts.
skipping to change at page 10, line 27 skipping to change at page 11, line 31
Figure 7 is used when performing those same benchmarks with Figure 7 is used when performing those same benchmarks with
Associated Media traversing the DUT/SUT. Associated Media traversing the DUT/SUT.
+--------+ Session +--------+ Session +--------+ +--------+ Session +--------+ Session +--------+
| | Attempt | | Attempt | | | | Attempt | | Attempt | |
| |<------------+ |<------------+ | | |<------------+ |<------------+ |
| | | | | | | | | | | |
| | Response | | Response | | | | Response | | Response | |
| Tester +------------>| DUT +------------>| Tester | | Tester +------------>| DUT +------------>| Tester |
| | | | | (EA) | | (EA) | | | | (EA) |
| | Media | | Media | | | | Media | | Media | |
| |<===========>| |<===========>| | | |<===========>| |<===========>| |
+--------+ +--------+ +--------+ +--------+ +--------+ +--------+
Figure 7: DUT/SUT performance benchmark for session establishment Figure 7: DUT/SUT performance benchmark for session establishment
with media traversing the DUT with media traversing the DUT
Figure 8 is to be used when performing those same benchmarks with Figure 8 is to be used when performing those same benchmarks with
Associated Media, but the media does not traverse the DUT/SUT. Associated Media, but the media does not traverse the DUT/SUT.
Again, the benchmarking of the media is not within the scope of this Again, the benchmarking of the media is not within the scope of this
work item. The SIP control signaling is benchmarked in the presence work item. The SIP control signaling is benchmarked in the presence
of Associated Media to determine if the SDP body of the signaling and of Associated Media to determine if the SDP body of the signaling and
the handling of media impacts the performance of the DUT/SUT. the handling of media impacts the performance of the DUT/SUT.
+--------+ Session +--------+ Session +--------+ +--------+ Session +--------+ Session +--------+
| | Attempt | | Attempt | | | | Attempt | | Attempt | |
| |<------------+ |<------------+ | | |<------------+ |<------------+ |
| | | | | | | | | | | |
| | Response | | Response | | | | Response | | Response | |
| Tester +------------>| DUT +------------>| Tester | | Tester +------------>| DUT +------------>| Tester |
| | | | | (EA) | | (EA) | | | | (EA) |
| | | | | | | | | | | |
+--------+ +--------+ +--------+ +--------+ +--------+ +--------+
/|\ /|\ /|\ /|\
| Media | | Media |
+=============================================+ +=============================================+
Figure 8: DUT/SUT performance benchmark for session establishment Figure 8: DUT/SUT performance benchmark for session establishment
with media external to the DUT with media external to the DUT
Figure 9 is used when performing benchmarks that require one or more Figure 9 is used when performing benchmarks that require one or more
intermediaries to be in the signaling path. The intent is to gather intermediaries to be in the signaling path. The intent is to gather
benchmarking statistics with a series of DUTs in place. In this benchmarking statistics with a series of DUTs in place. In this
topology, the media is delivered end-to-end and does not traverse the topology, the media is delivered end-to-end and does not traverse the
DUT. DUT.
SUT SUT
'--------------------------^^^^^^^^-----------------------` ------------------^^^^^^^^-------------
/ \ / \
+------+ Session +---+ Session +---+ Session +------+ +------+ Session +---+ Session +---+ Session +------+
| | Attempt | | Attempt | | Attempt | | | | Attempt | | Attempt | | Attempt | |
| |<---------+ |<---------+ |<---------+ | | |<---------+ |<---------+ |<---------+ |
| | | | | | | | | | | | | | | |
| | Response | | Response | | Response | | | | Response | | Response | | Response | |
|Tester+--------->|DUT+--------->|DUT|--------->|Tester| |Tester+--------->|DUT+--------->|DUT|--------->|Tester|
| | | | | | | | | (EA) | | | | | | (EA) |
| | | | | | | | | | | | | | | |
+------+ +---+ +---+ +------+ +------+ +---+ +---+ +------+
/|\ /|\ /|\ /|\
| Media | | Media |
+=============================================+ +=============================================+
Figure 9: DUT/SUT performance benchmark for session establishment Figure 9: DUT/SUT performance benchmark for session establishment
with multiple DUTs and end-to-end media with multiple DUTs and end-to-end media
Figure 10 is used when performing benchmarks that require one or more Figure 10 is used when performing benchmarks that require one or more
intermediaries to be in the signaling path. The intent is to gather intermediaries to be in the signaling path. The intent is to gather
benchmarking statistics with a series of DUTs in place. In this benchmarking statistics with a series of DUTs in place. In this
topology, the media is delivered hop-by-hop through each DUT. topology, the media is delivered hop-by-hop through each DUT.
SUT SUT
'--------------------------^^^^^^^^-----------------------` -----------------^^^^^^^^-------------
/ \ / \
+------+ Session +---+ Session +---+ Session +------+ +------+ Session +---+ Session +---+ Session +------+
| | Attempt | | Attempt | | Attempt | | | | Attempt | | Attempt | | Attempt | |
| |<---------+ |<---------+ |<---------+ | | |<---------+ |<---------+ |<---------+ |
| | | | | | | | | | | | | | | |
| | Response | | Response | | Response | | | | Response | | Response | | Response | |
|Tester+--------->|DUT+--------->|DUT|--------->|Tester| |Tester+--------->|DUT+--------->|DUT|--------->|Tester|
| | | | | | | | | (EA) | | | | | | (EA) |
| | | | | | | | | | | | | | | |
| |<========>| |<========>| |<========>| | | |<========>| |<========>| |<========>| |
+------+ Media +---+ Media +---+ Media +------+ +------+ Media +---+ Media +---+ Media +------+
Figure 10: DUT/SUT performance benchmark for session establishment Figure 10: DUT/SUT performance benchmark for session establishment
with multiple DUTs and hop- by-hop media with multiple DUTs and hop- by-hop media
Figure 11 illustrates the SIP signaling for an Established Session. Figure 11 illustrates the SIP signaling for an Established Session.
The Tester acts as the EAs and initiates a Session Attempt with the The Tester acts as the EAs and initiates a Session Attempt with the
DUT/SUT. When the Emulated Agent (EA) receives a 200 OK from the DUT/SUT. When the Emulated Agent (EA) receives a 200 OK from the
skipping to change at page 13, line 7 skipping to change at page 14, line 7
sourced from the EA, is optional. When Associated Media is used, it sourced from the EA, is optional. When Associated Media is used, it
may traverse the DUT/SUT depending upon the type of DUT/SUT. The may traverse the DUT/SUT depending upon the type of DUT/SUT. The
Associated Media is shown in Figure 11 as "Media" connected to media Associated Media is shown in Figure 11 as "Media" connected to media
ports M1 and M2 on the EA. After the EA sends a BYE, the session ports M1 and M2 on the EA. After the EA sends a BYE, the session
disconnects. Performance test cases for session disconnects are not disconnects. Performance test cases for session disconnects are not
considered in this work item (the BYE request is shown for considered in this work item (the BYE request is shown for
completeness.) completeness.)
EA DUT/SUT M1 M2 EA DUT/SUT M1 M2
| | | | | | | |
| INVITE | | | | INVITE | | |
--------+--------------->| | | ---------+-------------->| | |
| | | | | | | |
Attempting | | | Attempting | | |
| 200 OK | | | | 200 OK | | |
--------|<-------------- | | | ---------+<--------------| | |
| ACK | | |
|-------------->| | |
| | | | | | | |
| | | | | | | |
| | | Media | | | | Media |
Established | |<=====>| Established | |<=====>|
| | | | | | | |
| BYE | | | | BYE | | |
--------+--------------> | | | --------+--------------> | | |
| | | | | | | |
Disconnecting | | | Disconnecting | | |
| 200 OK | | | | 200 OK | | |
--------|<-------------- | | | --------|<-------------- | | |
| | | | | | | |
Figure 11: Basic SIP test topology Figure 11: Invite-initiated Session States
3. Term Definitions 3. Term Definitions
3.1. Protocol Components 3.1. Protocol Components
3.1.1. Session 3.1.1. Session
Definition: Definition:
The combination of signaling and media messages and processes that The combination of signaling and media messages and processes that
enable two or more participants to communicate. support a SIP-based service.
Discussion: Discussion:
SIP messages in the signaling plane can be used to create and SIP messages are used to create and manage services for end users.
manage applications for one or more end users. SIP is often used Often, these services include the creation of media streams that
to create and manage media streams in support of applications. A are defined in the SDP body of a SIP message and carried in RTP
session always has a signaling component and may have a media protocol data units. However, SIP messages can also be used to
component. Therefore, a Session may be defined as signaling only create Instant Message services and subscription services, and
or a combination of signaling and media (c.f. Associated Media, such services are not associated with media streams. SIP reserves
see Section 3.1.4). SIP includes definitions of a Call-ID, a the term "session" to describe services that are analogous to
dialogue and a transaction that support this application. A telephone calls on a circuit switched network. SIP reserves the
growing number of usages and applications do not require the term "dialog" to refer to a signaling-only relationship between
creation of associated media. The first such usage was the User Agent peers. SIP reserves the term "transaction" to refer to
REGISTER. Applications that use the MESSAGE and SUBSCRIBE/NOTIFY the brief communication between a client and a server that lasts
methods also do not require SIP to manage media streams. The only until the final response to the SIP request. None of these
terminology Invite-initiated Session (IS) and Non-invite initiated terms describes the entity whose performance we want to benchmark.
Session (NS) are used to distinguish between these different For example, the MESSAGE request does not create a dialog and can
usages. be sent either within or outside of a dialog. It is not
associated with media, but it resembles a phone call in its
dependence on human rather than machine initiated responses. The
SUBSCRIBE method does create a dialog between the originating end-
user and the subscription service. It too is not associated with
a media session. In light of these observations we have extended
the term "session" to include SIP-based services that are not
initiated by INVITE requests and that do not have associated
media. In this extended definition, a session always has a
signaling component and may also have a media component. Thus, a
session can be defined as signaling-only or a combination of
signaling and media. We define the term "Associated Media", see
Section 3.1.4, to describe the situation in which media is
associated with a SIP dialog. The terminology "Invite-initiated
Session" (IS) Section 3.1.8 and "Non-invite-Initiated Session"
(NS) (add xref target="NS") are used to distinguish between these
two types of session. An Invite-initiated Session is a session as
defined in SIP. The performance of a device or system that
supports Invite-initiated Sessions that do not create media
sessions, "Invite-initiated Sessions without Associated Media",
can be measured and is of interest for comparison and as a
limiting case. The REGISTER request can be considered to be a
"Non-invite-initiated Session without Associated Media." A
separate set of benchmarks is provided for REGISTER requests since
most implementations of SIP-based services require this request
and since a registrar may be a device under test.
A Session in the context of this document, is considered to be a A Session in the context of this document, can be considered to be
vector with three components: a vector with three components:
1. A component in the signaling plane (SIP messages), sess.sig; 1. A component in the signaling plane (SIP messages), sess.sig;
2. A media component in the media plane (RTP and SRTP streams for 2. A media component in the media plane (RTP and SRTP streams for
example), sess.med (which may be null); example), sess.med (which may be null);
3. A control component in the media plane (RTCP messages for 3. A control component in the media plane (RTCP messages for
example), sess.medc (which may be null). example), sess.medc (which may be null).
An IS is expected to have non-null sess.sig and sess.med An IS is expected to have non-null sess.sig and sess.med
components. The use of control protocols in the media component components. The use of control protocols in the media component
is media dependent, thus the expected presence or absence of is media dependent, thus the expected presence or absence of
skipping to change at page 14, line 41 skipping to change at page 16, line 22
Note that one or more sessions can simultaneously exist between Note that one or more sessions can simultaneously exist between
any participants. This can be the case, for example, when the EA any participants. This can be the case, for example, when the EA
sets up both an IM and a voice call through the DUT/SUT. These sets up both an IM and a voice call through the DUT/SUT. These
sessions are represented as an array session[x]. sessions are represented as an array session[x].
Sessions will be represented as a vector array with three Sessions will be represented as a vector array with three
components, as follows: components, as follows:
session-> session->
session[x].sig, the signaling component session[x].sig, the signaling component
session[x].medc, the media control component (e.g. RTCP) session[x].medc[y], the media control component (e.g. RTCP)
session[x].med[y], an array of associated media streams (e.g. session[x].med[y], an array of associated media streams (e.g.
RTP, SRTP, RTSP, MSRP). This media component may consist of zero RTP, SRTP, RTSP, MSRP). This media component may consist of zero
or more media streams. or more media streams.
Figure 12 models the vectors of the session. Figure 12 models the vectors of the session.
Measurement Units: Measurement Units:
N/A. N/A.
Issues: Issues:
None. None.
See Also: See Also:
Media Plane Media Plane
Signaling Plane Signaling Plane
Associated Media Associated Media
Invite-initiated Session (IS) Invite-initiated Session (IS)
skipping to change at page 15, line 44 skipping to change at page 17, line 31
/ / / /
/ | / |
/ / / /
sess.med / | sess.med / |
/_ _ _ _ _ _ _ _/ /_ _ _ _ _ _ _ _/
/ /
/ /
/ /
/ /
Figure 12: Application or session components Figure 12: Session components
3.1.2. Signaling Plane 3.1.2. Signaling Plane
Definition: Definition:
The control plane in which SIP messages [RFC3261] are exchanged The plane in which SIP messages [RFC3261] are exchanged between
between SIP Agents [RFC3261] to establish a connection for media SIP Agents [RFC3261].
exchange.
Discussion: Discussion:
SIP messages are used to establish sessions in several ways: SIP messages are used to establish sessions in several ways:
directly between two User Agents [RFC3261], through a Proxy Server directly between two User Agents [RFC3261], through a Proxy Server
[RFC3261], or through a series of Proxy Servers. The Signaling [RFC3261], or through a series of Proxy Servers. The Session
Plane MUST include the Session Description Protocol (SDP). Description Protocol is included in the Signaling Plane. (SDP).
The Signaling Plane for a single Session is represented by The Signaling Plane for a single Session is represented by
session.sig. session.sig.
Measurement Units: Measurement Units:
N/A. N/A.
Issues: Issues:
None. None.
See Also: See Also:
Media Plane Media Plane
EAs EAs
3.1.3. Media Plane 3.1.3. Media Plane
Definition: Definition:
The data plane in which one or more media streams and their The data plane in which one or more media streams and their
associated media control protocols are exchanged after a media associated media control protocols are exchanged between User
connection has been created by the exchange of signaling messages Agents after a media connection has been created by the exchange
in the Signaling Plane. of signaling messages in the Signaling Plane.
Discussion: Discussion:
Media may also be known as the "bearer channel". The Media Plane Media may also be known as the "bearer channel". The Media Plane
MUST include the media control protocol, if one is used, and the MUST include the media control protocol, if one is used, and the
media stream(s). Examples of media are audio, video, whiteboard, media stream(s). Examples of media are audio and video. The
and instant messaging service. The media stream is described in media streams are described in the SDP of the Signaling Plane.
the SDP of the Signaling Plane.
The media for a single Session is represented by session.med. The The media for a single Session is represented by session.med. The
media control protocol is represented by session.medc. media control protocol for a single media description is
represented by session.medc.
Measurement Units: Measurement Units:
N/A. N/A.
Issues: Issues:
None. None.
See Also: See Also:
Signaling Plane Signaling Plane
skipping to change at page 17, line 18 skipping to change at page 19, line 4
See Also: See Also:
Signaling Plane Signaling Plane
3.1.4. Associated Media 3.1.4. Associated Media
Definition: Definition:
Media that corresponds to an 'm' line in the SDP payload of the Media that corresponds to an 'm' line in the SDP payload of the
Signaling Plane. Signaling Plane.
Discussion: Discussion:
Any media protocol MAY be used. Any media protocol MAY be used.
For any session's signaling component, represented as session.sig, For any session's signaling component, session.sig, there may be
there may be one or multiple associated media streams which are zero, one, or multiple associated media streams. When there are
represented be a vector array session.med[y], which is referred to multiple media streams, these are represented be a vector array
as the Associated Media. session.med[y]. When there are multiple media streams there will
be multiple media control protocol descriptions as well. They are
represented by a vector array session.medc[y].
Measurement Units: Measurement Units:
N/A. N/A.
Issues: Issues:
None. None.
3.1.5. Overload 3.1.5. Overload
Definition: Definition:
Overload is defined as the state where a SIP server does not have Overload is defined as the state where a SIP server does not have
sufficient resources to process all incoming SIP messages sufficient resources to process all incoming SIP messages
[I-D.ietf-soc-overload-design]. The distinction between an [I-D.ietf-soc-overload-design].
overload condition and other failure scenarios is outside the
scope of this document which is blackbox testing.
Discussion: Discussion:
Under overload conditions, all or a percentage of Session Attempts The distinction between an overload condition and other failure
will fail due to lack of resources. SIP server resources may scenarios is outside the scope of black box testing and of this
include CPU processing capacity, network bandwidth, input/output document. Under overload conditions, all or a percentage of
queues, or disk resources. Any combination of resources may be Session Attempts will fail due to lack of resources. In black box
fully utilized when a SIP server (the DUT/SUT) is in the overload testing the cause of the failure is not explored. The fact that a
condition. For proxy-only type of devices, overload issues will failure occurred for whatever reason, will trigger the tester to
be dominated by the number of signaling messages they can handle reduce the offered load, as described in the companion methodology
in a unit time before their throughput starts to drop. document, [I-D.ietf-bmwg-sip-bench-meth]. SIP server resources
may include CPU processing capacity, network bandwidth, input/
For UA-type of network devices (e.g., gateways), overload must output queues, or disk resources. Any combination of resources
necessarily include both the signaling traffic and media streams. may be fully utilized when a SIP server (the DUT/SUT) is in the
It is expected that the amount of signaling that a UA can handle overload condition. For proxy-only type of devices, it is
is inversely proportional to the amount of media streams currently expected that the proxy will be driven into overload based on the
handled by that UA. delivery rate of signaling requests.
For UA-type of network devices such as gateways, it is expected
that the UA will be driven into overload based on the volume of
media streams it is processing.
Measurement Units: Measurement Units:
N/A. N/A.
Issues: Issues:
The issue of overload in SIP networks is currently a topic of The issue of overload in SIP networks is currently a topic of
discussion in the SIPPING WG. The normal response to an overload discussion in the SIPPING WG. The normal response to an overload
stimulus -- sending a 503 response -- is considered inadequate and stimulus -- sending a 503 response -- is considered inadequate and
new response codes and behaviors may be specified in the future. new response codes and behaviors may be specified in the future.
From the perspective of this document, all these responses will be From the perspective of this document, all these responses will be
considered to be failures. There is thus no dependency between considered to be failures. There is thus no dependency between
this document and the ongoing work on the treatment of overload this document and the ongoing work on the treatment of overload
failure. failure.
3.1.6. Session Attempt 3.1.6. Session Attempt
Definition: Definition:
A SIP Session for which the EA has sent the SIP INVITE or A SIP request sent by the EA that has not received a final
SUBSCRIBE NOTIFY and has not yet received a message response from response.
the DUT/SUT.
Discussion: Discussion:
The attempted session may be an IS or an NS. The Session Attempt The attempted session may be Invite Initiated or Non-invite
includes SIP INVITEs and SUBSCRIBE/NOTIFY messages. It also Initiated. When counting the number of session attempts we
includes all INVITEs that are rejected for lack of authentication include all INVITEs that are rejected for lack of authentication
information. information. The EA needs to record the total number of session
attempts including those attempts that are routinely rejected by a
proxy that requires the UA to authenticate itself. The EA is
provisioned to deliver a specific number of session attempts per
second. But the EA must also count the actual number of session
attempts per given tie interval.
Measurement Units: Measurement Units:
N/A. N/A.
Issues: Issues:
None. None.
See Also: See Also:
Session Session
Session Attempt Rate Session Attempt Rate
skipping to change at page 19, line 4 skipping to change at page 20, line 45
Issues: Issues:
None. None.
See Also: See Also:
Session Session
Session Attempt Rate Session Attempt Rate
Invite-initiated Session Invite-initiated Session
Non-Invite initiated Session Non-Invite initiated Session
3.1.7. Established Session 3.1.7. Established Session
Definition: Definition:
A SIP session for which the EA acting as the UE/UA has received a A SIP session for which the EA acting as the UE/UA has received a
200 OK message from the DUT/SUT. 200 OK message.
Discussion: Discussion:
An Established Session MAY be type INVITE-Session (IS) or Non- An Established Session MAY be Invite Initiated or Non-invite
INVITE Session (NS). Initiated.
Measurement Units: Measurement Units:
N/A. N/A.
Issues: Issues:
None. None.
See Also: See Also:
Invite-initiated Session Invite-initiated Session
Session Attempting State Session Attempting State
Session Disconnecting State Session Disconnecting State
3.1.8. Invite-initiated Session (IS) 3.1.8. Invite-initiated Session (IS)
Definition: Definition:
A Session that is created by an exchange of messages in the A Session that is created by an exchange of messages in the
Signaling Plane, the first of which is a SIP INVITE request. Signaling Plane, the first of which is a SIP INVITE request.
Discussion: Discussion:
An IS is identified by the Call-ID, To-tag, and From-tag of the When an IS becomes an Established Session its signaling component
SIP message that establishes the session. These three fields are is identified by the SIP dialog parameter values, Call-ID, To-tag,
used to identify a SIP Dialog (RFC3261 [RFC3261]). An IS may have and From-tag (RFC3261 [RFC3261]). An IS may have zero, one or
Associated Media description in the SDP body. An IS may have multiple Associated Media descriptions in the SDP body. The
multiple Associated Media streams. The inclusion of media is test inclusion of media is test case dependent. An IS is successfully
case dependent. An IS is successfully established if the established if the following two conditions are met:
following two conditions are met:
1. Sess.sig is established by the end of Establishment Threshold 1. Sess.sig is established by the end of Establishment Threshold
Time (c.f. Section 3.3.3), and Time (c.f. Section 3.3.3), and
2. If a media session is described in the SDP body of the 2. If a media session is described in the SDP body of the
signaling message, then the media session is established by signaling message, then the media session is established by
the end of Establishment Threshold Time (c.f. Section 3.3.3). the end of Establishment Threshold Time (c.f. Section 3.3.3).
An SBC or B2BUA may receive media from a calling or called
party before a signaling dialog is established and certainly
before a confirmed dialog is established. The EA can be built
in such a way that it does not send early media or it needs to
include a parameter that indicates when it will send media.
This parameter must be included in the list of test setup
parameters in Section 5.1 of [I-D.ietf-bmwg-sip-bench-meth]
Measurement Units: Measurement Units:
N/A. N/A.
Issues: Issues:
None. None.
See Also: See Also:
Session Session
Non-Invite initiated Session Non-Invite initiated Session
Associated Media Associated Media
3.1.9. Non-INVITE-initiated Session (NS) 3.1.9. Non-INVITE-initiated Session (NS)
Definition: Definition:
A session that is created by an exchange of messages in the A session that is created by an exchange of SIP messages in the
Signaling Plane that does not include an initial SIP INVITE Signaling Plane the first of which is not a SIP INVITE message.
message.
Discussion: Discussion:
An NS is successfully established if the Session Attempt via a An NS is successfully established if the Session Attempt via a
non- INVITE request results in the EA receiving a 2xx reply from non- INVITE request results in the EA receiving a 2xx reply before
the DUT/SUT before the expiration of the Establishment Threshold the expiration of the Establishment Threshold timer (c.f.,
timer (c.f., Section 3.3.3). An example of a NS is a session Section 3.3.3). An example of a NS is a session created by the
created by the SUBSCRIBE request. SUBSCRIBE request.
Measurement Units: Measurement Units:
N/A. N/A.
Issues: Issues:
None. None.
See Also: See Also:
Session Session
Invite-initiated Session Invite-initiated Session
skipping to change at page 21, line 32 skipping to change at page 23, line 32
Duration and the Session Attempt Rate. Benchmarks MUST be Duration and the Session Attempt Rate. Benchmarks MUST be
reported with the maximum and average Standing Sessions for the reported with the maximum and average Standing Sessions for the
DUT/SUT for the duration of the test. In order to determine the DUT/SUT for the duration of the test. In order to determine the
maximum and average Standing Sessions on the DUT/SUT for the maximum and average Standing Sessions on the DUT/SUT for the
duration of the test it is necessary to make periodic measurements duration of the test it is necessary to make periodic measurements
of the number of Standing Sessions on the DUT/SUT. The of the number of Standing Sessions on the DUT/SUT. The
recommended value for the measurement period is 1 second. Since recommended value for the measurement period is 1 second. Since
we cannot directly poll the DUT/SUT, we take the number of we cannot directly poll the DUT/SUT, we take the number of
standing sessions on the DUT/SUT to be the number of distinct standing sessions on the DUT/SUT to be the number of distinct
calls as measured by the number of distinct Call-IDs that the EA calls as measured by the number of distinct Call-IDs that the EA
is processing at the time of measurement. is processing at the time of measurement. The EA must make that
count available for viewing ad recording.
Measurement Units: Measurement Units:
Number of sessions Number of sessions
Issues: Issues:
None. None.
See Also: See Also:
Session Duration Session Duration
Session Attempt Rate Session Attempt Rate
Session Attempt Rate Session Attempt Rate
Emulated Agent
3.2. Test Components 3.2. Test Components
3.2.1. Emulated Agent 3.2.1. Emulated Agent
Definition: Definition:
A device in test topology that initiates/responds to SIP messages A device in the test topology that initiates/responds to SIP
as one or more session endpoints and, wherever applicable, messages as one or more session endpoints and, wherever
sources/receives Associated Media for Established Sessions. applicable, sources/receives Associated Media for Established
Sessions.
Discussion: Discussion:
The EA functions in the signaling and media planes. The Tester The EA functions in the Signaling and Media Planes. The Tester
may act as multiple EAs. may act as multiple EAs.
Measurement Units: Measurement Units:
N/A N/A
Issues: Issues:
None. None.
See Also: See Also:
Media Plane Media Plane
Signaling Plane Signaling Plane
Established Session Established Session
Associated Media Associated Media
3.2.2. Signaling Server 3.2.2. Signaling Server
Definition: Definition:
Device in test topology that acts to create sessions between EAs Device in the test topology that acts to create sessions between
in the media plane. This device is either a DUT or component of a EAs. This device is either a DUT or a component of a SUT.
SUT.
Discussion: Discussion:
The DUT MUST be a RFC 3261 capable network equipment such as a The DUT MUST be an RFC 3261 capable network equipment such as a
Registrar, Redirect Server, User Agent Server, Stateless Proxy, or Registrar, Redirect Server, User Agent Server, Stateless Proxy, or
Stateful Proxy. A DUT MAY also include B2BUA or SBC. Stateful Proxy. A DUT MAY also include B2BUA or SBC.
Measurement Units: Measurement Units:
NA NA
Issues: Issues:
None. None.
See Also: See Also:
Signaling Plane Signaling Plane
3.2.3. SIP-Aware Stateful Firewall 3.2.3. SIP-Aware Stateful Firewall
Definition: Definition:
Device in test topology that provides Denial-of-Service (DoS) Device in the test topology that provides protection against
Protection to the Signaling and Media Planes for the EAs and various types of security threats to which the Signaling and Media
Signaling Server Planes of the EAs and Signaling Server are vulnerable.
Discussion: Discussion:
The SIP-Aware Stateful Firewall MAY be an internal component or Threats may include Denial-of-Service, theft of service and misuse
function of the Session Server. The SIP-Aware Stateful Firewall of service.he SIP-Aware Stateful Firewall MAY be an internal
MAY be a standalone device. If it is a standalone device it MUST component or function of the Session Server. The SIP-Aware
be paired with a Signaling Server. If it is a standalone device Stateful Firewall MAY be a standalone device. If it is a
it MUST be benchmarked as part of a SUT. SIP-Aware Stateful standalone device it MUST be paired with a Signaling Server. If
Firewalls MAY include Network Address Translation (NAT) it is a standalone device it MUST be benchmarked as part of a SUT.
functionality. Ideally, the inclusion of the SIP-Aware Stateful SIP-Aware Stateful Firewalls MAY include Network Address
Firewall as a SUT has no degradation to the measured performance Translation (NAT) functionality. Ideally, the inclusion of the
benchmarks. SIP-Aware Stateful Firewall in the SUT does not lower the measured
values of the performance benchmarks.
Measurement Units: Measurement Units:
N/A N/A
Issues: Issues:
None. None.
See Also: See Also:
3.2.4. SIP Transport Protocol 3.2.4. SIP Transport Protocol
skipping to change at page 24, line 10 skipping to change at page 26, line 12
Issues: Issues:
None. None.
See Also: See Also:
3.3. Test Setup Parameters 3.3. Test Setup Parameters
3.3.1. Session Attempt Rate 3.3.1. Session Attempt Rate
Definition: Definition:
Configuration of the EA for the number of sessions that the EA Configuration of the EA for the number of sessions per second that
attempts to establish with the DUT/SUT over a specified time the EA attempts to establish using the services of the DUT/SUT.
interval.
Discussion: Discussion:
The Session Attempt Rate can cause variation in performance The Session Attempt Rate is the number of sessions per second that
benchmark measurements. Since this is the number of sessions the EA sends toward the DUT/SUT. Some of the sessions attempted
configured on the Tester, some sessions may not be successfully may not result in a session being established. A session in this
established on the DUT. A session may be either an IS or an NS. case may be either an IS or an NS.
Measurement Units: Measurement Units:
Session attempts per second Session attempts per second
Issues: Issues:
None. None.
See Also: See Also:
Session Session
Session Attempt Session Attempt
3.3.2. IS Media Attempt Rate 3.3.2. IS Media Attempt Rate
Definition: Definition:
Configuration on the EA for number of ISs with Associated Media to Configuration on the EA for the rate, measured in sessions per
be established at the DUT per continuous one- second time second, at which the EA attempts to establish INVITE-initiated
intervals. sessions with Associated Media, using the services of the DUT/SUT.
Discussion: Discussion:
Note that a Media Session MUST be associated with an IS. In this An IS is not required to include a media description. The IS
document we assume that there is a one to one correspondence Media Attempt Rate defines the number of media sessions we are
between IS session attempts and Media Session attempts. By trying to create, not the number of media sessions that are
including this definition we leave open the possibility that there actually created. Some attempts might not result in successful
may be an IS that does not include a media description. Also note
that the IS Media Attempt Rate defines the number of media
sessions we are trying to create, not the number of media sessions
that are actually created. Variations in the Media Session
Attempt Rate might cause variations in performance benchmark
measurements. Some attempts might not result in successful
sessions established on the DUT. sessions established on the DUT.
Measurement Units: Measurement Units:
session attempts per second (saps) session attempts per second (saps)
Issues: Issues:
None. None.
See Also: See Also:
IS IS
3.3.3. Establishment Threshold Time 3.3.3. Establishment Threshold Time
Definition: Definition:
Configuration of the EA for representing the amount of time that Configuration of the EA for representing the amount of time that
an EA will wait before declaring a Session Attempt Failure. an EA will wait before declaring a Session Attempt Failure.
skipping to change at page 28, line 4 skipping to change at page 29, line 44
Issues: Issues:
None. None.
See Also: See Also:
Associated Media Associated Media
Established Session Established Session
Invite-initiated Session (IS) Invite-initiated Session (IS)
3.3.8. Loop Detection Option 3.3.8. Loop Detection Option
Definition: Definition:
An option that causes a Proxy to check for loops in the routing of An option that causes a Proxy to check for loops in the routing of
a SIP request before forwarding the request. a SIP request before forwarding the request.
Discussion: Discussion:
This is an optional process that a SIP proxy may employ; the This is an optional process that a SIP proxy may employ; the
process is described under Proxy Behavior in RFC 3261 in Section process is described under Proxy Behavior in RFC 3261 [RFC3261] in
16.3 Request Validation and that section also contains suggestions Section 16.3 Request Validation and that section also contains
as to how the option could be implemented. Any procedure to suggestions as to how the option could be implemented. Any
detect loops will use processor cycles and hence could impact the procedure to detect loops will use processor cycles and hence
performance of a proxy. could impact the performance of a proxy.
Measurement Units: Measurement Units:
NA NA
Issues: Issues:
None. None.
See Also: See Also:
3.3.9. Forking Option 3.3.9. Forking Option
skipping to change at page 29, line 46 skipping to change at page 31, line 43
3.4.2. Session Establishment Rate 3.4.2. Session Establishment Rate
Definition: Definition:
The maximum number of sessions that can be successfully completed The maximum number of sessions that can be successfully completed
by the DUT/SUT in a given time period without session by the DUT/SUT in a given time period without session
establishment failures in that time period. establishment failures in that time period.
Discussion: Discussion:
This benchmark is obtained with zero failure in which 100% of the This benchmark is obtained with zero failure in which 100% of the
sessions attempted by the emulated Agent are successfully sessions attempted by the Emulated Agent are successfully
completed by the DUT/SUT. The session attempt rate provisioned on completed by the DUT/SUT. The session attempt rate provisioned on
the EA is raised and lowered as described in the algorithm in the the EA is raised and lowered as described in the algorithm in the
accompanying methodology document, until a traffic load at the accompanying methodology document, until a traffic load at the
given attempt rate over the sustained period of time identified by given attempt rate over the sustained period of time identified by
T in the algorithm completes without any failed session attempts. T in the algorithm completes without any failed session attempts.
Sessions may be IS or NS or a mix of both and will be defined in Sessions may be IS or NS or a mix of both and will be defined in
the particular test. the particular test.
Measurement Units: Measurement Units:
sessions per second (sps) sessions per second (sps)
Issues: Issues:
None. None.
See Also: See Also:
skipping to change at page 31, line 24 skipping to change at page 33, line 21
simultaneously on the DUT/SUT until it stops responding to Session simultaneously on the DUT/SUT until it stops responding to Session
Attempts. Attempts.
Discussion: Discussion:
Session Overload Capacity is measured after the Session Capacity Session Overload Capacity is measured after the Session Capacity
is measured. The Session Overload Capacity is greater than or is measured. The Session Overload Capacity is greater than or
equal to the Session Capacity. When benchmarking Session Overload equal to the Session Capacity. When benchmarking Session Overload
Capacity, continue to offer Session Attempts to the DUT/SUT after Capacity, continue to offer Session Attempts to the DUT/SUT after
the first Session Attempt Failure occurs and measure Established the first Session Attempt Failure occurs and measure Established
Sessions until no there is no SIP message response for the Sessions until no there is no SIP message response for the
duration of the Establishment Threshold. It is worth noting that duration of the Establishment Threshold. Note that the Session
the Session Establishment Performance is expected to decrease Establishment Performance is expected to decrease after the first
after the first Session Attempt Failure occurs. Session Attempt Failure occurs.
Units: Units:
Sessions Sessions
Issues: Issues:
None. None.
See Also: See Also:
Overload Overload
Session Capacity Session Capacity
skipping to change at page 32, line 38 skipping to change at page 34, line 38
Definition: Definition:
The average time measured at the EA for a Session Attempt to The average time measured at the EA for a Session Attempt to
result in an Established Session. result in an Established Session.
Discussion: Discussion:
Time is measured from when the EA sends the first INVITE for the Time is measured from when the EA sends the first INVITE for the
call-ID in the case of an IS. Time is measured from when the EA call-ID in the case of an IS. Time is measured from when the EA
sends the first non-INVITE message in the case of an NS. Session sends the first non-INVITE message in the case of an NS. Session
Attempt Delay MUST be measured for every established session to Attempt Delay MUST be measured for every established session to
calculate the average. Session Attempt Delay MUST be measured at calculate the average. Session Attempt Delay MUST be measured at
the Maximum Session Establishment Rate. the Session Establishment Rate.
Measurement Units: Measurement Units:
Seconds Seconds
Issues: Issues:
None. None.
See Also: See Also:
Maximum Session Establishment Rate Session Establishment Rate
3.4.7. IM Rate 3.4.7. IM Rate
Definition: Definition:
Maximum number of IM messages completed by the DUT/SUT. Maximum number of IM messages completed by the DUT/SUT.
Discussion: Discussion:
For a UAS, the definition of success is the receipt of an IM For a UAS, the definition of success is the receipt of an IM
request and the subsequent sending of a final response. request and the subsequent sending of a final response.
For a UAC, the definition of success is the sending of an IM For a UAC, the definition of success is the sending of an IM
request and the receipt of a final response to it. For a proxy, request and the receipt of a final response to it. For a proxy,
skipping to change at page 34, line 29 skipping to change at page 36, line 29
Network Interconnect Devices", RFC 2544, March 1999. Network Interconnect Devices", RFC 2544, March 1999.
[RFC3261] Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston, [RFC3261] Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston,
A., Peterson, J., Sparks, R., Handley, M., and E. A., Peterson, J., Sparks, R., Handley, M., and E.
Schooler, "SIP: Session Initiation Protocol", RFC 3261, Schooler, "SIP: Session Initiation Protocol", RFC 3261,
June 2002. June 2002.
[I-D.ietf-bmwg-sip-bench-meth] [I-D.ietf-bmwg-sip-bench-meth]
Davids, C., Gurbani, V., and S. Poretsky, "Methodology for Davids, C., Gurbani, V., and S. Poretsky, "Methodology for
Benchmarking SIP Networking Devices", Benchmarking SIP Networking Devices",
draft-ietf-bmwg-sip-bench-meth-03 (work in progress), draft-ietf-bmwg-sip-bench-meth-04 (work in progress),
March 2011. March 2012.
7.2. Informational References 7.2. Informational References
[RFC2285] Mandeville, R., "Benchmarking Terminology for LAN [RFC2285] Mandeville, R., "Benchmarking Terminology for LAN
Switching Devices", RFC 2285, February 1998. Switching Devices", RFC 2285, February 1998.
[RFC1242] Bradner, S., "Benchmarking terminology for network [RFC1242] Bradner, S., "Benchmarking terminology for network
interconnection devices", RFC 1242, July 1991. interconnection devices", RFC 1242, July 1991.
[RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V. [RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V.
skipping to change at page 35, line 9 skipping to change at page 37, line 9
[I-D.ietf-soc-overload-design] [I-D.ietf-soc-overload-design]
Hilt, V., Noel, E., Shen, C., and A. Abdelal, "Design Hilt, V., Noel, E., Shen, C., and A. Abdelal, "Design
Considerations for Session Initiation Protocol (SIP) Considerations for Session Initiation Protocol (SIP)
Overload Control", draft-ietf-soc-overload-design-08 (work Overload Control", draft-ietf-soc-overload-design-08 (work
in progress), July 2011. in progress), July 2011.
[I-D.ietf-soc-overload-control] [I-D.ietf-soc-overload-control]
Gurbani, V., Hilt, V., and H. Schulzrinne, "Session Gurbani, V., Hilt, V., and H. Schulzrinne, "Session
Initiation Protocol (SIP) Overload Control", Initiation Protocol (SIP) Overload Control",
draft-ietf-soc-overload-control-07 (work in progress), draft-ietf-soc-overload-control-10 (work in progress),
January 2012. October 2012.
Appendix A. White Box Benchmarking Terminology Appendix A. White Box Benchmarking Terminology
Session Attempt Arrival Rate Session Attempt Arrival Rate
Definition: Definition:
The number of Session Attempts received at the DUT/SUT over a The number of Session Attempts received at the DUT/SUT over a
specified time period. specified time period.
Discussion: Discussion:
 End of changes. 84 change blocks. 
293 lines changed or deleted 344 lines changed or added

This html diff was produced by rfcdiff 1.41. The latest version is available from http://tools.ietf.org/tools/rfcdiff/