draft-ietf-bmwg-sip-bench-term-08.txt   draft-ietf-bmwg-sip-bench-term-09.txt 
Benchmarking Methodology Working C. Davids Benchmarking Methodology Working Group C. Davids
Group Illinois Institute of Technology Internet-Draft Illinois Institute of Technology
Internet-Draft V. Gurbani Intended status: Informational V. Gurbani
Intended status: Informational Bell Laboratories, Alcatel-Lucent Expires: August 18, 2014 Bell Laboratories,
Expires: July 12, 2013 S. Poretsky Alcatel-Lucent
S. Poretsky
Allot Communications Allot Communications
January 8, 2013 February 14, 2014
Terminology for Benchmarking Session Initiation Protocol (SIP) Terminology for Benchmarking Session Initiation Protocol (SIP) Devices:
Networking Devices Basic session setup and registration
draft-ietf-bmwg-sip-bench-term-08 draft-ietf-bmwg-sip-bench-term-09
Abstract Abstract
This document provides a terminology for benchmarking the SIP This document provides a terminology for benchmarking the Session
performance of networking devices. The term performance in this Initiation Protocol (SIP) performance of devices. Methodology
context means the capacity of the device- or system-under-test to related to benchmarking SIP devices is described in the companion
process SIP messages. Terms are included for test components, test methodology document. Using these two documents, benchmarks can be
setup parameters, and performance benchmark metrics for black-box obtained and compared for different types of devices such as SIP
benchmarking of SIP networking devices. The performance benchmark Proxy Servers, Registrars and Session Border Controllers. The term
metrics are obtained for the SIP signaling plane only. The terms are "performance" in this context means the capacity of the device-under-
intended for use in a companion methodology document for test (DUT) to process SIP messages. Media streams are used only to
characterizing the performance of a SIP networking device under a study how they impact the signaling behavior. The intent of the two
variety of conditions. The intent of the two documents is to enable documents is to provide a normalized set of tests that will enable an
a comparison of the capacity of SIP networking devices. Test setup objective comparison of the capacity of SIP devices. Test setup
parameters and a methodology document are necessary because SIP parameters and a methodology is necessary because SIP allows a wide
allows a wide range of configuration and operational conditions that range of configuration and operational conditions that can influence
can influence performance benchmark measurements. A standard performance benchmark measurements. A standard terminology and
terminology and methodology will ensure that benchmarks have methodology will ensure that benchmarks have consistent definition
consistent definition and were obtained following the same and were obtained following the same procedures.
procedures.
Status of this Memo Status of this Memo
This Internet-Draft is submitted in full conformance with the This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79. provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/. Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on July 12, 2013. This Internet-Draft will expire on August 18, 2014.
Copyright Notice Copyright Notice
Copyright (c) 2013 IETF Trust and the persons identified as the Copyright (c) 2014 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License. described in the Simplified BSD License.
Table of Contents Table of Contents
1. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 5 1. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4
2. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 6 2. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.1. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.1. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.2. Benchmarking Models . . . . . . . . . . . . . . . . . . . 9 3. Term Definitions . . . . . . . . . . . . . . . . . . . . . . . 7
3. Term Definitions . . . . . . . . . . . . . . . . . . . . . . . 14 3.1. Protocol Components . . . . . . . . . . . . . . . . . . . 7
3.1. Protocol Components . . . . . . . . . . . . . . . . . . . 14 3.1.1. Session . . . . . . . . . . . . . . . . . . . . . . . 7
3.1.1. Session . . . . . . . . . . . . . . . . . . . . . . . 14 3.1.2. Signaling Plane . . . . . . . . . . . . . . . . . . . 10
3.1.2. Signaling Plane . . . . . . . . . . . . . . . . . . . 17 3.1.3. Media Plane . . . . . . . . . . . . . . . . . . . . . 11
3.1.3. Media Plane . . . . . . . . . . . . . . . . . . . . . 18 3.1.4. Associated Media . . . . . . . . . . . . . . . . . . . 11
3.1.4. Associated Media . . . . . . . . . . . . . . . . . . . 18 3.1.5. Overload . . . . . . . . . . . . . . . . . . . . . . . 12
3.1.5. Overload . . . . . . . . . . . . . . . . . . . . . . . 19 3.1.6. Session Attempt . . . . . . . . . . . . . . . . . . . 12
3.1.6. Session Attempt . . . . . . . . . . . . . . . . . . . 20 3.1.7. Established Session . . . . . . . . . . . . . . . . . 13
3.1.7. Established Session . . . . . . . . . . . . . . . . . 20 3.1.8. Invite-Initiated Session (IS) . . . . . . . . . . . . 14
3.1.8. Invite-initiated Session (IS) . . . . . . . . . . . . 21 3.1.9. Non-INVITE-Initiated Session (NS) . . . . . . . . . . 14
3.1.9. Non-INVITE-initiated Session (NS) . . . . . . . . . . 22 3.1.10. Session Attempt Failure . . . . . . . . . . . . . . . 15
3.1.10. Session Attempt Failure . . . . . . . . . . . . . . . 22 3.2. Test Components . . . . . . . . . . . . . . . . . . . . . 15
3.1.11. Standing Sessions Count . . . . . . . . . . . . . . . 23 3.2.1. Emulated Agent . . . . . . . . . . . . . . . . . . . . 15
3.2. Test Components . . . . . . . . . . . . . . . . . . . . . 23 3.2.2. Signaling Server . . . . . . . . . . . . . . . . . . . 16
3.2.1. Emulated Agent . . . . . . . . . . . . . . . . . . . . 24 3.2.3. SIP Transport Protocol . . . . . . . . . . . . . . . . 16
3.2.2. Signaling Server . . . . . . . . . . . . . . . . . . . 24 3.3. Test Setup Parameters . . . . . . . . . . . . . . . . . . 17
3.2.3. SIP-Aware Stateful Firewall . . . . . . . . . . . . . 24 3.3.1. Session Attempt Rate . . . . . . . . . . . . . . . . . 17
3.2.4. SIP Transport Protocol . . . . . . . . . . . . . . . . 25 3.3.2. Establishment Threshold Time . . . . . . . . . . . . . 17
3.3. Test Setup Parameters . . . . . . . . . . . . . . . . . . 26 3.3.3. Session Duration . . . . . . . . . . . . . . . . . . . 18
3.3.1. Session Attempt Rate . . . . . . . . . . . . . . . . . 26 3.3.4. Media Packet Size . . . . . . . . . . . . . . . . . . 18
3.3.2. IS Media Attempt Rate . . . . . . . . . . . . . . . . 26 3.3.5. Codec Type . . . . . . . . . . . . . . . . . . . . . . 19
3.3.3. Establishment Threshold Time . . . . . . . . . . . . . 27 3.4. Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . 19
3.3.4. Session Duration . . . . . . . . . . . . . . . . . . . 27 3.4.1. Session Establishment Rate . . . . . . . . . . . . . . 19
3.3.5. Media Packet Size . . . . . . . . . . . . . . . . . . 28 3.4.2. Registration Rate . . . . . . . . . . . . . . . . . . 20
3.3.6. Media Offered Load . . . . . . . . . . . . . . . . . . 28 3.4.3. Registration Attempt Rate . . . . . . . . . . . . . . 21
3.3.7. Media Session Hold Time . . . . . . . . . . . . . . . 29 4. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 21
3.3.8. Loop Detection Option . . . . . . . . . . . . . . . . 29 5. Security Considerations . . . . . . . . . . . . . . . . . . . 21
3.3.9. Forking Option . . . . . . . . . . . . . . . . . . . . 30 6. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 22
3.4. Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . 31 7. References . . . . . . . . . . . . . . . . . . . . . . . . . . 22
3.4.1. Registration Rate . . . . . . . . . . . . . . . . . . 31 7.1. Normative References . . . . . . . . . . . . . . . . . . . 22
3.4.2. Session Establishment Rate . . . . . . . . . . . . . . 31 7.2. Informational References . . . . . . . . . . . . . . . . . 22
3.4.3. Session Capacity . . . . . . . . . . . . . . . . . . . 32 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 23
3.4.4. Session Overload Capacity . . . . . . . . . . . . . . 33
3.4.5. Session Establishment Performance . . . . . . . . . . 33
3.4.6. Session Attempt Delay . . . . . . . . . . . . . . . . 34
3.4.7. IM Rate . . . . . . . . . . . . . . . . . . . . . . . 34
4. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 35
5. Security Considerations . . . . . . . . . . . . . . . . . . . 35
6. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 35
7. References . . . . . . . . . . . . . . . . . . . . . . . . . . 36
7.1. Normative References . . . . . . . . . . . . . . . . . . . 36
7.2. Informational References . . . . . . . . . . . . . . . . . 36
Appendix A. White Box Benchmarking Terminology . . . . . . . . . 37
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 37
1. Terminology 1. Terminology
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in BCP 14, RFC2119 document are to be interpreted as described in BCP 14, RFC2119
[RFC2119]. RFC 2119 defines the use of these key words to help make [RFC2119]. RFC 2119 defines the use of these key words to help make
the intent of standards track documents as clear as possible. While the intent of standards track documents as clear as possible. While
this document uses these keywords, this document is not a standards this document uses these keywords, this document is not a standards
track document. The term Throughput is defined in RFC2544 [RFC2544]. track document. The term Throughput is defined in RFC2544 [RFC2544].
For the sake of clarity and continuity, this document adopts the For the sake of clarity and continuity, this document adopts the
template for definitions set out in Section 2 of RFC 1242 [RFC1242]. template for definitions set out in Section 2 of RFC 1242 [RFC1242].
The terms Device Under Test (DUT) and System Under Test (SUT) are The term Device Under Test (DUT) is defined in the following BMWG
defined in the following BMWG documents: documents:
Device Under Test (DUT) (c.f., Section 3.1.1 RFC 2285 [RFC2285]). Device Under Test (DUT) (c.f., Section 3.1.1 RFC 2285 [RFC2285]).
System Under Test (SUT) (c.f., Section 3.1.2, RFC 2285 [RFC2285]).
Many commonly used SIP terms in this document are defined in RFC 3261 Many commonly used SIP terms in this document are defined in RFC 3261
[RFC3261]. For convenience the most important of these are [RFC3261]. For convenience the most important of these are
reproduced below. Use of these terms in this document is consistent reproduced below. Use of these terms in this document is consistent
with their corresponding definition in [RFC3261]. with their corresponding definition in the base SIP specification
[RFC3261] as amended by [RFC4320], [RFC5393] and [RFC6026].
o Call Stateful: A proxy is call stateful if it retains state for a o Call Stateful: A proxy is call stateful if it retains state for a
dialog from the initiating INVITE to the terminating BYE request. dialog from the initiating INVITE to the terminating BYE request.
A call stateful proxy is always transaction stateful, but the A call stateful proxy is always transaction stateful, but the
converse is not necessarily true. converse is not necessarily true.
o Stateful Proxy: A logical entity that maintains the client and o Stateful Proxy: A logical entity that maintains the client and
server transaction state machines defined by this specification server transaction state machines defined by this specification
during the processing of a request, also known as a transaction during the processing of a request, also known as a transaction
stateful proxy. The behavior of a stateful proxy is further stateful proxy. The behavior of a stateful proxy is further
defined in Section 16 of RFC 3261 [RFC3261] . A transaction defined in Section 16 of RFC 3261 [RFC3261] . A transaction
stateful proxy is not the same as a call stateful proxy. stateful proxy is not the same as a call stateful proxy.
skipping to change at page 6, line 5 skipping to change at page 5, line 5
it receives upstream. it receives upstream.
o Back-to-back User Agent: A back-to-back user agent (B2BUA) is a o Back-to-back User Agent: A back-to-back user agent (B2BUA) is a
logical entity that receives a request and processes it as a user logical entity that receives a request and processes it as a user
agent server (UAS). In order to determine how the request should agent server (UAS). In order to determine how the request should
be answered, it acts as a user agent client (UAC) and generates be answered, it acts as a user agent client (UAC) and generates
requests. Unlike a proxy server, it maintains dialog state and requests. Unlike a proxy server, it maintains dialog state and
must participate in all requests sent on the dialogues it has must participate in all requests sent on the dialogues it has
established. Since it is a concatenation of a UAC and a UAS, no established. Since it is a concatenation of a UAC and a UAS, no
explicit definitions are needed for its behavior. explicit definitions are needed for its behavior.
o Loop: A request that arrives at a proxy, is forwarded, and later
arrives back at the same proxy. When it arrives the second time,
its Request-URI is identical to the first time, and other header
fields that affect proxy operation are unchanged, so that the
proxy will make the same processing decision on the request it
made the first time. Looped requests are errors, and the
procedures for detecting them and handling them are described by
the SIP protocol[RFC3261] and also by RFC 5393
2. Introduction 2. Introduction
Service Providers and IT Organizations deliver Voice Over IP (VoIP) Service Providers and IT Organizations deliver Voice Over IP (VoIP)
and Multimedia network services based on the IETF Session Initiation and Multimedia network services based on the IETF Session Initiation
Protocol (SIP) [RFC3261]. SIP is a signaling protocol originally Protocol (SIP) [RFC3261]. SIP is a signaling protocol originally
intended to be used to dynamically establish, disconnect and modify intended to be used to dynamically establish, disconnect and modify
streams of media between end users. As it has evolved it has been streams of media between end users. As it has evolved it has been
adopted for use in a growing number of services and applications. adopted for use in a growing number of services and applications.
Many of these result in the creation of a media session, but some do Many of these result in the creation of a media session, but some do
not. Examples of this latter group include text messaging and not. Examples of this latter group include text messaging and
subscription services. The set of benchmarking terms provided in subscription services. The set of benchmarking terms provided in
this document is intended for use with any SIP-enabled device this document is intended for use with any SIP-enabled device
performing SIP functions in the interior of the network, whether or performing SIP functions in the interior of the network, whether or
not these result in the creation of media sessions. The performance not these result in the creation of media sessions. The performance
of end-user devices is outside the scope of this document. of end-user devices is outside the scope of this document.
A number of networking devices have been developed to support SIP- A number of networking devices have been developed to support SIP-
based VoIP services. These include SIP Servers, Session Border based VoIP services. These include SIP Servers, Session Border
Controllers (SBC), Back-to-back User Agents (B2BUA), and SIP-Aware Controllers (SBC) and Back-to-back User Agents (B2BUA). These
Stateful Firewalls. These devices contain a mix of voice and IP devices contain a mix of voice and IP functions whose performance may
functions whose performance may be reported using metrics defined by be reported using metrics defined by the equipment manufacturer or
the equipment manufacturer or vendor. The Service Provider or IT vendor. The Service Provider or IT Organization seeking to compare
Organization seeking to compare the performance of such devices will the performance of such devices will not be able to do so using these
not be able to do so using these vendor-specific metrics, whose vendor-specific metrics, whose conditions of test and algorithms for
conditions of test and algorithms for collection are often collection are often unspecified. SIP functional elements and the
unspecified. SIP functional elements and the devices that include devices that include them can be configured many different ways and
them can be configured many different ways and can be organized into can be organized into various topologies. These configuration and
various topologies. These configuration and topological choices topological choices impact the value of any chosen signaling
impact the value of any chosen signaling benchmark. Unless these benchmark. Unless these conditions-of-test are defined, a true
conditions-of-test are defined, a true comparison of performance comparison of performance metrics across multiple vendor
metrics will not be possible. Some SIP-enabled network devices implementations will not be possible. Some SIP-enabled devices
terminate or relay media as well as signaling. The processing of terminate or relay media as well as signaling. The processing of
media by the device impacts the signaling performance. As a result, media by the device impacts the signaling performance. As a result,
the conditions-of-test must include information as to whether or not the conditions-of-test must include information as to whether or not
the device under test processes media and if the device does process the device under test processes media and if the device does process
media, a description of the media handled and the manner in which it media, a description of the media handled and the manner in which it
is handled. This document and its companion methodology document is handled. This document and its companion methodology document
[I-D.ietf-bmwg-sip-bench-meth] provide a set of black-box benchmarks [I-D.ietf-bmwg-sip-bench-meth] provide a set of black-box benchmarks
for describing and comparing the performance of devices that for describing and comparing the performance of devices that
incorporate the SIP User Agent Client and Server functions and that incorporate the SIP User Agent Client and Server functions and that
operate in the network's core. operate in the network's core.
The definition of SIP performance benchmarks necessarily includes The definition of SIP performance benchmarks necessarily includes
definitions of Test Setup Parameters and a test methodology. These definitions of Test Setup Parameters and a test methodology. These
enable the Tester to perform benchmarking tests on different devices enable the Tester to perform benchmarking tests on different devices
and to achieve comparable results. This document provides a common and to achieve comparable results. This document provides a common
set of definitions for Test Components, Test Setup Parameters, and set of definitions for Test Components, Test Setup Parameters, and
Benchmarks. All the benchmarks defined are black-box measurements of Benchmarks. All the benchmarks defined are black-box measurements of
the SIP signaling plane. The Test Setup Parameters and Benchmarks the SIP signaling plane. The Test Setup Parameters and Benchmarks
defined in this document are intended for use with the companion defined in this document are intended for use with the companion
Methodology document. Benchmarks of internal DUT characteristics Methodology document.
(also known as white-box benchmarks) such as Session Attempt Arrival
Rate, which is measured at the DUT, are described in Appendix A to
allow additional characterization of DUT behavior with different
distribution models.
2.1. Scope 2.1. Scope
The scope of this work item is summarized as follows: The scope of this work item is summarized as follows:
o This terminology document describes SIP signaling performance o This terminology document describes SIP signaling performance
benchmarks for black-box measurements of SIP networking devices. benchmarks for black-box measurements of SIP networking devices.
Stress and debug scenarios are not addressed in this work item. Stress and debug scenarios are not addressed in this work item.
o The DUT must be an RFC 3261 capable network equipment. This may o The DUT must be an RFC 3261 capable network equipment. This may
be a Registrar, Redirect Server, Stateless Proxy or Stateful be a Registrar, Redirect Server, Stateless Proxy or Stateful
Proxy. A DUT MAY also include a B2BUA, SBC functionality. The Proxy. A DUT MAY also include a B2BUA, SBC functionality.
DUT MAY be a multi-port SIP-to-switched network gateway o The DUT MUST NOT be end user equipment, such as personal digital
implemented as a SIP UAC or UAS. assistant, a computer-based client, or a user terminal.
o The DUT MAY include an internal SIP Application Level Gateway
(ALG), firewall, and/or a Network Address Translator (NAT). This
is referred to as the "SIP Aware Stateful Firewall."
o The DUT or SUT MUST NOT be end user equipment, such as personal
digital assistant, a computer-based client, or a user terminal.
o The Tester acts as multiple "Emulated Agents" (EA) that initiate o The Tester acts as multiple "Emulated Agents" (EA) that initiate
(or respond to) SIP messages as session endpoints and source (or (or respond to) SIP messages as session endpoints and source (or
receive) associated media for established connections. receive) associated media for established connections.
o SIP Signaling in presence of Media o SIP Signaling in presence of Media
* The media performance is not benchmarked in this work item. * The media performance is not benchmarked in this work item.
* It is RECOMMENDED that SIP signaling plane benchmarks be * Some tests require media, but the use of media is limited to
performed with media present, but this is optional. observing the performance of SIP signaling. Tests that require
* The SIP INVITE requests MUST include the SDP body. media will annotate the media characteristics as a condition of
test.
* The type of DUT dictates whether the associated media streams * The type of DUT dictates whether the associated media streams
traverse the DUT or SUT. Both scenarios are within the scope traverse the DUT. Both scenarios are within the scope of this
of this work item. work item.
* SIP is frequently used to create media streams; the signaling * SIP is frequently used to create media streams; the signaling
plane and media plane are treated as orthogonal to each other plane and media plane are treated as orthogonal to each other
in this document. While many devices support the creation of in this document. While many devices support the creation of
media streams, benchmarks that measure the performance of these media streams, benchmarks that measure the performance of these
streams are outside the scope of this document and its streams are outside the scope of this document and its
companion methodology document [I-D.ietf-bmwg-sip-bench-meth]. companion methodology document [I-D.ietf-bmwg-sip-bench-meth].
Tests may be performed with or without the creation of media Tests may be performed with or without the creation of media
streams. The presence or absence of media streams MUST be streams. The presence or absence of media streams MUST be
noted as a condition of the test as the performance of SIP noted as a condition of the test as the performance of SIP
devices may vary accordingly. Even if the media is used during devices may vary accordingly. Even if the media is used during
benchmarking, only the SIP performance will be benchmarked, not benchmarking, only the SIP performance will be benchmarked, not
the media performance or quality. the media performance or quality.
o Both INVITE and non-INVITE scenarios (such as Instant Messages or o Both INVITE and non-INVITE scenarios (registrations) are addressed
IM) are addressed in this document. However, benchmarking SIP in this document. However, benchmarking SIP presence or
presence is not a part of this work item. subscribe-notify extensions is not a part of this work item.
o Different transport mechanisms -- such as UDP, TCP, SCTP, or TLS o Different transport -- such as UDP, TCP, SCTP, or TLS -- may be
-- may be used. The specific transport mechanism MUST be noted as used. The specific transport mechanism MUST be noted as a
a condition of the test as the performance of SIP devices may vary condition of the test as the performance of SIP devices may vary
accordingly. accordingly.
o Looping and forking options are also considered since they impact
processing at SIP proxies.
o REGISTER and INVITE requests may be challenged or remain o REGISTER and INVITE requests may be challenged or remain
unchallenged for authentication purpose. Whether or not the unchallenged for authentication purpose. Whether or not the
REGISTER and INVITE requests are challenged is a condition of test REGISTER and INVITE requests are challenged is a condition of test
which will be recorded along with other such parameters which may which will be recorded along with other such parameters which may
impact the SIP performance of the device or system under test. impact the SIP performance of the device or system under test.
o Re-INVITE requests are not considered in scope of this work item o Re-INVITE requests are not considered in scope of this work item
since the benchmarks for INVITEs are based on the dialog created since the benchmarks for INVITEs are based on the dialog created
by the INVITE and not on the transactions that take place within by the INVITE and not on the transactions that take place within
that dialog. that dialog.
o Only session establishment is considered for the performance o Only session establishment is considered for the performance
benchmarks. Session disconnect is not considered in the scope of benchmarks. Session disconnect is not considered in the scope of
this work item. This is because our goal is to determine the this work item. This is because our goal is to determine the
maximum capacity of the device or system under test, that is the maximum capacity of the device or system under test, that is the
number of simultaneous SIP sessions that the device or system can number of simultaneous SIP sessions that the device or system can
support. It is true that there are BYE requests being created support. It is true that there are BYE requests being created
during the test process. These transactions do contribute to the during the test process. These transactions do contribute to the
load on the device or system under test and thus are accounted for load on the device or system under test and thus are accounted for
in the metric we derive. We do not seek a separate metric for the in the metric we derive. We do not seek a separate metric for the
number of BYE transactions a device or system can support. number of BYE transactions a device or system can support.
o SIP Overload [RFC6357] is within the scope of this work item. We
test to failure and then can continue to observe and record the
behavior of the system after failures are recorded. The cause of
failure is not within the scope of this work. We note the failure
and may continue to test until a different failure or condition is
encountered. Considerations on how to handle overload are
deferred to work progressing in the SOC working group
[I-D.ietf-soc-overload-control]. Vendors are, of course, free to
implement their specific overload control behavior as the expected
test outcome if it is different from the IETF recommendations.
However, such behavior MUST be documented and interpreted
appropriately across multiple vendor implementations. This will
make it more meaningful to compare the performance of different
SIP overload implementations.
o IMS-specific scenarios are not considered, but test cases can be o IMS-specific scenarios are not considered, but test cases can be
applied with 3GPP-specific SIP signaling and the P-CSCF as a DUT. applied with 3GPP-specific SIP signaling and the P-CSCF as a DUT.
2.2. Benchmarking Models
This section shows ten models to be used when benchmarking SIP
performance of a networking device. Figure 1 shows shows the
configuration needed to benchmark the tester itself. This model will
be used to establish the limitations of the test apparatus.
+--------+ Signaling request +--------+
| +----------------------------->| |
| Tester | | Tester |
| EA | Signaling response | EA |
| |<-----------------------------+ |
+--------+ +--------+
/|\ /|\
| Media |
+=========================================+
Figure 1: Baseline performance of the Emulated Agent without a DUT
present
Figure 2 shows the DUT playing the role of a user agent client (UAC),
initiating requests and absorbing responses. This model can be used
to baseline the performance of the DUT acting as an UAC without
associated media.
+--------+ Signaling request +--------+
| +----------------------------->| |
| DUT | | Tester |
| | Signaling response | EA |
| |<-----------------------------+ |
+--------+ +--------+
Figure 2: Baseline performance for DUT acting as a user agent client
without associated media
Figure 3 shows the DUT playing the role of a user agent server (UAS),
absorbing the requests and sending responses. This model can be used
as a baseline performance for the DUT acting as a UAS without
associated media.
+--------+ Signaling request +--------+
| +----------------------------->| |
| Tester | | DUT |
| EA | Response | |
| |<-----------------------------+ |
+--------+ +--------+
Figure 3: Baseline performance for DUT acting as a user agent server
without associated media
Figure 4 shows the DUT plays the role of a user agent client (UAC),
initiating requests and absorbing responses. This model can be used
as a baseline performance for the DUT acting as a UAC with associated
media.
+--------+ Signaling request +--------+
| +----------------------------->| |
| DUT | | Tester |
| | Signaling response | (EA) |
| |<-----------------------------+ |
| |<============ Media =========>| |
+--------+ +--------+
Figure 4: Baseline performance for DUT acting as a user agent client
with associated media
Figure 5 shows the DUT plays the role of a user agent server (UAS),
absorbing the requests and sending responses. This model can be used
as a baseline performance for the DUT acting as a UAS with associated
media.
+--------+ Signaling request +--------+
| +----------------------------->| |
| Tester | | DUT |
| (EA) | Response | |
| |<-----------------------------+ |
| |<============ Media =========>| |
+--------+ +--------+
Figure 5: Baseline performance for DUT acting as a user agent server
with associated media
Figure 6 shows that the Tester acts as the initiating and responding
EA as the DUT/SUT forwards Session Attempts.
+--------+ Session +--------+ Session +--------+
| | Attempt | | Attempt | |
| |<------------+ |<------------+ |
| | | | | |
| | Response | | Response | |
| Tester +------------>| DUT +------------>| Tester |
| (EA) | | | | (EA) |
| | | | | |
+--------+ +--------+ +--------+
Figure 6: DUT/SUT performance benchmark for session establishment
without media
Figure 7 is used when performing those same benchmarks with
Associated Media traversing the DUT/SUT.
+--------+ Session +--------+ Session +--------+
| | Attempt | | Attempt | |
| |<------------+ |<------------+ |
| | | | | |
| | Response | | Response | |
| Tester +------------>| DUT +------------>| Tester |
| (EA) | | | | (EA) |
| | Media | | Media | |
| |<===========>| |<===========>| |
+--------+ +--------+ +--------+
Figure 7: DUT/SUT performance benchmark for session establishment
with media traversing the DUT
Figure 8 is to be used when performing those same benchmarks with
Associated Media, but the media does not traverse the DUT/SUT.
Again, the benchmarking of the media is not within the scope of this
work item. The SIP control signaling is benchmarked in the presence
of Associated Media to determine if the SDP body of the signaling and
the handling of media impacts the performance of the DUT/SUT.
+--------+ Session +--------+ Session +--------+
| | Attempt | | Attempt | |
| |<------------+ |<------------+ |
| | | | | |
| | Response | | Response | |
| Tester +------------>| DUT +------------>| Tester |
| (EA) | | | | (EA) |
| | | | | |
+--------+ +--------+ +--------+
/|\ /|\
| Media |
+=============================================+
Figure 8: DUT/SUT performance benchmark for session establishment
with media external to the DUT
Figure 9 is used when performing benchmarks that require one or more
intermediaries to be in the signaling path. The intent is to gather
benchmarking statistics with a series of DUTs in place. In this
topology, the media is delivered end-to-end and does not traverse the
DUT.
SUT
------------------^^^^^^^^-------------
/ \
+------+ Session +---+ Session +---+ Session +------+
| | Attempt | | Attempt | | Attempt | |
| |<---------+ |<---------+ |<---------+ |
| | | | | | | |
| | Response | | Response | | Response | |
|Tester+--------->|DUT+--------->|DUT|--------->|Tester|
| (EA) | | | | | | (EA) |
| | | | | | | |
+------+ +---+ +---+ +------+
/|\ /|\
| Media |
+=============================================+
Figure 9: DUT/SUT performance benchmark for session establishment
with multiple DUTs and end-to-end media
Figure 10 is used when performing benchmarks that require one or more
intermediaries to be in the signaling path. The intent is to gather
benchmarking statistics with a series of DUTs in place. In this
topology, the media is delivered hop-by-hop through each DUT.
SUT
-----------------^^^^^^^^-------------
/ \
+------+ Session +---+ Session +---+ Session +------+
| | Attempt | | Attempt | | Attempt | |
| |<---------+ |<---------+ |<---------+ |
| | | | | | | |
| | Response | | Response | | Response | |
|Tester+--------->|DUT+--------->|DUT|--------->|Tester|
| (EA) | | | | | | (EA) |
| | | | | | | |
| |<========>| |<========>| |<========>| |
+------+ Media +---+ Media +---+ Media +------+
Figure 10: DUT/SUT performance benchmark for session establishment
with multiple DUTs and hop- by-hop media
Figure 11 illustrates the SIP signaling for an Established Session.
The Tester acts as the EAs and initiates a Session Attempt with the
DUT/SUT. When the EA receives a 200 OK from the DUT/SUT that session
is considered to be an Established Session. The illustration
indicates three states of the session bring created by the EA - (1)
Attempting, (2) Established, and (3) Disconnecting. Sessions can be
one of two type: Invite-Initiated Session (IS) or Non-Invite
Initiated Session (NS). Failure for the DUT/SUT to successfully
respond within the Establishment Threshold Time is considered a
Session Attempt Failure. SIP Invite messages MUST include the SDP
body to specify the Associated Media. Use of Associated Media, to be
sourced from the EA, is optional. When Associated Media is used, it
may traverse the DUT/SUT depending upon the type of DUT/SUT. The
Associated Media is shown in Figure 11 as "Media" connected to media
ports M1 and M2 on the EA. After the EA sends a BYE, the session
disconnects. Performance test cases for session disconnects are not
considered in this work item (the BYE request is shown for
completeness.)
EA DUT/SUT M1 M2
| | | |
| INVITE | | |
---------+-------------->| | |
| | | |
Attempting | | |
| 200 OK | | |
---------+<--------------| | |
| ACK | | |
|-------------->| | |
| | | |
| | | |
| | | Media |
Established | |<=====>|
| | | |
| BYE | | |
--------+--------------> | | |
| | | |
Disconnecting | | |
| 200 OK | | |
--------|<-------------- | | |
| | | |
Figure 11: Invite-initiated Session States
3. Term Definitions 3. Term Definitions
3.1. Protocol Components 3.1. Protocol Components
3.1.1. Session 3.1.1. Session
Definition: Definition:
The combination of signaling and media messages and processes that The combination of signaling and media messages and processes that
support a SIP-based service. support a SIP-based service.
skipping to change at page 15, line 23 skipping to change at page 8, line 19
with a media session. with a media session.
In light of the above observations we have extended the term In light of the above observations we have extended the term
"session" to include SIP-based services that are not initiated by "session" to include SIP-based services that are not initiated by
INVITE requests and that do not have associated media. In this INVITE requests and that do not have associated media. In this
extended definition, a session always has a signaling component extended definition, a session always has a signaling component
and may also have a media component. Thus, a session can be and may also have a media component. Thus, a session can be
defined as signaling-only or a combination of signaling and media. defined as signaling-only or a combination of signaling and media.
We define the term "Associated Media", see Section 3.1.4, to We define the term "Associated Media", see Section 3.1.4, to
describe the situation in which media is associated with a SIP describe the situation in which media is associated with a SIP
dialog. The terminology "Invite-initiated Session" (IS) dialog. The terminology "Invite-Initiated Session" (IS)
Section 3.1.8 and "Non-invite-Initiated Session" (NS) Section 3.1.8 and "Non-invite-Initiated Session" (NS)
Section 3.1.9 are used to distinguish between these two types of Section 3.1.9 are used to distinguish between these two types of
session. An Invite-initiated Session is a session as defined in session. An Invite-Initiated Session is a session as defined in
SIP. The performance of a device or system that supports Invite- SIP. The performance of a device or system that supports Invite-
initiated Sessions that do not create media sessions, "Invite- Initiated Sessions that do not create media sessions, "Invite-
initiated Sessions without Associated Media", can be measured and Initiated Sessions without Associated Media", can be measured and
is of interest for comparison and as a limiting case. The is of interest for comparison and as a limiting case. The
REGISTER request can be considered to be a "Non-invite-initiated REGISTER request can be considered to be a "Non-invite-Initiated
Session without Associated Media." A separate set of benchmarks Session without Associated Media." A separate set of benchmarks
is provided for REGISTER requests since most implementations of is provided for REGISTER requests since most implementations of
SIP-based services require this request and since a registrar may SIP-based services require this request and since a registrar may
be a device under test. be a device under test.
A Session in the context of this document, can be considered to be A Session in the context of this document, can be considered to be
a vector with three components: a vector with three components:
1. A component in the signaling plane (SIP messages), sess.sig; 1. A component in the signaling plane (SIP messages), sess.sig;
2. A media component in the media plane (RTP and SRTP streams for 2. A media component in the media plane (RTP and SRTP streams for
skipping to change at page 16, line 7 skipping to change at page 9, line 7
An IS is expected to have non-null sess.sig and sess.med An IS is expected to have non-null sess.sig and sess.med
components. The use of control protocols in the media component components. The use of control protocols in the media component
is media dependent, thus the expected presence or absence of is media dependent, thus the expected presence or absence of
sess.medc is media dependent and test-case dependent. An NS is sess.medc is media dependent and test-case dependent. An NS is
expected to have a non-null sess.sig component, but null sess.med expected to have a non-null sess.sig component, but null sess.med
and sess.medc components. and sess.medc components.
Packets in the Signaling Plane and Media Plane will be handled by Packets in the Signaling Plane and Media Plane will be handled by
different processes within the DUT. They will take different different processes within the DUT. They will take different
paths within a SUT. These different processes and paths may paths within a network. These different processes and paths may
produce variations in performance. The terminology and benchmarks produce variations in performance. The terminology and benchmarks
defined in this document and the methodology for their use are defined in this document and the methodology for their use are
designed to enable us to compare performance of the DUT/SUT with designed to enable us to compare performance of the DUT with
reference to the type of SIP-supported application it is handling. reference to the type of SIP-supported application it is handling.
Note that one or more sessions can simultaneously exist between Note that one or more sessions can simultaneously exist between
any participants. This can be the case, for example, when the EA any participants. This can be the case, for example, when the EA
sets up both an IM and a voice call through the DUT/SUT. These sets up both an IM and a voice call through the DUT. These
sessions are represented as an array session[x]. sessions are represented as an array session[x].
Sessions will be represented as a vector array with three Sessions will be represented as a vector array with three
components, as follows: components, as follows:
session-> session->
session[x].sig, the signaling component session[x].sig, the signaling component
session[x].medc[y], the media control component (e.g. RTCP) session[x].medc[y], the media control component (e.g. RTCP)
session[x].med[y], an array of associated media streams (e.g. session[x].med[y], an array of associated media streams (e.g.
RTP, SRTP, RTSP, MSRP). This media component may consist of zero RTP, SRTP, RTSP, MSRP). This media component may consist of zero
or more media streams. or more media streams.
Figure 12 models the vectors of the session. Figure 1 models the vectors of the session.
Measurement Units: Measurement Units:
N/A. N/A.
Issues: Issues:
None. None.
See Also: See Also:
Media Plane Media Plane
Signaling Plane Signaling Plane
Associated Media Associated Media
Invite-initiated Session (IS) Invite-Initiated Session (IS)
Non-invite-initiated Session (NS) Non-Invite-Initiated Session (NS)
|\ |\
| |
| \ | \
sess.sig| sess.sig|
| \ | \
| |
| \ | \
| o | o
| / | /
| / | | / |
skipping to change at page 17, line 31 skipping to change at page 10, line 31
/ / / /
/ | / |
/ / / /
sess.med / | sess.med / |
/_ _ _ _ _ _ _ _/ /_ _ _ _ _ _ _ _/
/ /
/ /
/ /
/ /
Figure 12: Session components Figure 1: Session components
3.1.2. Signaling Plane 3.1.2. Signaling Plane
Definition: Definition:
The plane in which SIP messages [RFC3261] are exchanged between The plane in which SIP messages [RFC3261] are exchanged between
SIP Agents [RFC3261]. SIP Agents [RFC3261].
Discussion: Discussion:
SIP messages are used to establish sessions in several ways: SIP messages are used to establish sessions in several ways:
directly between two User Agents [RFC3261], through a Proxy Server directly between two User Agents [RFC3261], through a Proxy Server
skipping to change at page 19, line 37 skipping to change at page 12, line 37
The distinction between an overload condition and other failure The distinction between an overload condition and other failure
scenarios is outside the scope of black box testing and of this scenarios is outside the scope of black box testing and of this
document. Under overload conditions, all or a percentage of document. Under overload conditions, all or a percentage of
Session Attempts will fail due to lack of resources. In black box Session Attempts will fail due to lack of resources. In black box
testing the cause of the failure is not explored. The fact that a testing the cause of the failure is not explored. The fact that a
failure occurred for whatever reason, will trigger the tester to failure occurred for whatever reason, will trigger the tester to
reduce the offered load, as described in the companion methodology reduce the offered load, as described in the companion methodology
document, [I-D.ietf-bmwg-sip-bench-meth]. SIP server resources document, [I-D.ietf-bmwg-sip-bench-meth]. SIP server resources
may include CPU processing capacity, network bandwidth, input/ may include CPU processing capacity, network bandwidth, input/
output queues, or disk resources. Any combination of resources output queues, or disk resources. Any combination of resources
may be fully utilized when a SIP server (the DUT/SUT) is in the may be fully utilized when a SIP server (the DUT) is in the
overload condition. For proxy-only type of devices, it is overload condition. For proxy-only (or intermediary) devices, it
expected that the proxy will be driven into overload based on the is expected that the proxy will be driven into overload based on
delivery rate of signaling requests. the delivery rate of signaling requests.
For UA-type of network devices such as gateways, it is expected
that the UA will be driven into overload based on the volume of
media streams it is processing.
Measurement Units: Measurement Units:
N/A. N/A.
Issues:
The issue of overload in SIP networks is currently a topic of
discussion in the SIPPING WG. The normal response to an overload
stimulus -- sending a 503 response -- is considered inadequate and
new response codes and behaviors may be specified in the future.
From the perspective of this document, all these responses will be
considered to be failures. There is thus no dependency between
this document and the ongoing work on the treatment of overload
failure.
3.1.6. Session Attempt 3.1.6. Session Attempt
Definition: Definition:
A SIP request sent by the EA that has not received a final
response. A SIP INVITE or REGISTER request sent by the EA that has not
received a final response.
Discussion: Discussion:
The attempted session may be Invite Initiated or Non-invite The attempted session may be Invite Initiated or Non-invite-
Initiated. When counting the number of session attempts we Initiated. When counting the number of session attempts we
include all INVITEs that are rejected for lack of authentication include all INVITEs that are rejected for lack of authentication
information. The EA needs to record the total number of session information. The EA needs to record the total number of session
attempts including those attempts that are routinely rejected by a attempts including those attempts that are routinely rejected by a
proxy that requires the UA to authenticate itself. The EA is proxy that requires the UA to authenticate itself. The EA is
provisioned to deliver a specific number of session attempts per provisioned to deliver a specific number of session attempts per
second. But the EA must also count the actual number of session second. But the EA must also count the actual number of session
attempts per given tie interval. attempts per given tie interval.
Measurement Units: Measurement Units:
N/A. N/A.
Issues: Issues:
None. None.
See Also: See Also:
Session Session
Session Attempt Rate Session Attempt Rate
Invite-initiated Session Invite-Initiated Session
Non-Invite initiated Session Non-Invite-Initiated Session
3.1.7. Established Session 3.1.7. Established Session
Definition: Definition:
A SIP session for which the EA acting as the UE/UA has received a A SIP session for which the EA acting as the UE/UA has received a
200 OK message. 200 OK message.
Discussion: Discussion:
An Established Session MAY be Invite Initiated or Non-invite An Established Session MAY be Invite Initiated or Non-Invite-
Initiated. Initiated. Early dialogues for INVITE requests are out of scope
for this work.
Measurement Units: Measurement Units:
N/A. N/A.
Issues: Issues:
None. None.
See Also: See Also:
Invite-initiated Session
Session Attempting State
Session Disconnecting State
3.1.8. Invite-initiated Session (IS) Invite-Initiated Session
3.1.8. Invite-Initiated Session (IS)
Definition: Definition:
A Session that is created by an exchange of messages in the A Session that is created by an exchange of messages in the
Signaling Plane, the first of which is a SIP INVITE request. Signaling Plane, the first of which is a SIP INVITE request.
Discussion: Discussion:
When an IS becomes an Established Session its signaling component When an IS becomes an Established Session its signaling component
is identified by the SIP dialog parameter values, Call-ID, To-tag, is identified by the SIP dialog parameter values, Call-ID, To-tag,
and From-tag (RFC3261 [RFC3261]). An IS may have zero, one or and From-tag (RFC3261 [RFC3261]). An IS may have zero, one or
multiple Associated Media descriptions in the SDP body. The multiple Associated Media descriptions in the SDP body. The
inclusion of media is test case dependent. An IS is successfully inclusion of media is test case dependent. An IS is successfully
established if the following two conditions are met: established if the following two conditions are met:
1. Sess.sig is established by the end of Establishment Threshold 1. Sess.sig is established by the end of Establishment Threshold
Time (c.f. Section 3.3.3), and Time (c.f. Section 3.3.2), and
2. If a media session is described in the SDP body of the 2. If the DUT is a device, such as an SBC or B2BUA, that may
signaling message, then the media session is established by receive media from a calling or called party before a
the end of Establishment Threshold Time (c.f. Section 3.3.3). signaling dialog is established or confirmed , this fact needs
An SBC or B2BUA may receive media from a calling or called to be taken into account when the EA is built. Any additional
party before a signaling dialog is established and certainly parameters that need to be configured in the EA must be added
before a confirmed dialog is established. The EA can be built to the list of test setup parameters in Section 5.1 of
in such a way that it does not send early media or it needs to [I-D.ietf-bmwg-sip-bench-meth]
include a parameter that indicates when it will send media.
This parameter must be included in the list of test setup
parameters in Section 5.1 of [I-D.ietf-bmwg-sip-bench-meth]
Measurement Units: Measurement Units:
N/A. N/A.
Issues: Issues:
None. None.
See Also: See Also:
Session Session
Non-Invite initiated Session Non-Invite-Initiated Session
Associated Media Associated Media
3.1.9. Non-INVITE-initiated Session (NS) 3.1.9. Non-INVITE-Initiated Session (NS)
Definition: Definition:
A session that is created by an exchange of SIP messages in the A session that is created by an exchange of SIP messages in the
Signaling Plane the first of which is not a SIP INVITE message. Signaling Plane the first of which is not a SIP INVITE message.
Discussion: Discussion:
An NS is successfully established if the Session Attempt via a An NS is successfully established if the Session Attempt via a
non- INVITE request results in the EA receiving a 2xx reply before non- INVITE request results in the EA receiving a 2xx reply before
the expiration of the Establishment Threshold timer (c.f., the expiration of the Establishment Threshold timer (c.f.,
Section 3.3.3). An example of a NS is a session created by the Section 3.3.2). For the purpose of this document, a NS is a
SUBSCRIBE request. session created only by the REGISTER request and no other request.
Measurement Units: Measurement Units:
N/A. N/A.
Issues: Issues:
None. None.
See Also: See Also:
Session Session
Invite-initiated Session Invite-Initiated Session
3.1.10. Session Attempt Failure 3.1.10. Session Attempt Failure
Definition: Definition:
A session attempt that does not result in an Established Session. A session attempt that does not result in an Established Session.
Discussion: Discussion:
The session attempt failure may be indicated by the following The session attempt failure may be indicated by the following
observations at the EA: observations at the EA:
1. Receipt of a SIP 4xx, 5xx, or 6xx class response to a Session 1. Receipt of a SIP 3xx-, 4xx-, 5xx-, or 6xx-class response to a
Attempt. Session Attempt.
2. The lack of any received SIP response to a Session Attempt 2. The lack of any received SIP response to a Session Attempt
within the Establishment Threshold Time (c.f. Section 3.3.3). within the Establishment Threshold Time (c.f. Section 3.3.2).
Measurement Units: Measurement Units:
N/A. N/A.
Issues: Issues:
None. None.
See Also: See Also:
Session Attempt Session Attempt
3.1.11. Standing Sessions Count
Definition:
The number of Sessions currently established on the DUT/SUT at any
instant.
Discussion:
The number of Standing Sessions is influenced by the Session
Duration and the Session Attempt Rate. Benchmarks MUST be
reported with the maximum and average Standing Sessions for the
DUT/SUT for the duration of the test. In order to determine the
maximum and average Standing Sessions on the DUT/SUT for the
duration of the test it is necessary to make periodic measurements
of the number of Standing Sessions on the DUT/SUT. The
recommended value for the measurement period is 1 second. Since
we cannot directly poll the DUT/SUT, we take the number of
standing sessions on the DUT/SUT to be the number of distinct
calls as measured by the number of distinct Call-IDs that the EA
is processing at the time of measurement. The EA must make that
count available for viewing and recording.
Measurement Units:
Number of sessions
Issues:
None.
See Also:
Session Duration
Session Attempt Rate
Session Attempt Rate
Emulated Agent
3.2. Test Components 3.2. Test Components
3.2.1. Emulated Agent 3.2.1. Emulated Agent
Definition: Definition:
A device in the test topology that initiates/responds to SIP A device in the test topology that initiates/responds to SIP
messages as one or more session endpoints and, wherever messages as one or more session endpoints and, wherever
applicable, sources/receives Associated Media for Established applicable, sources/receives Associated Media for Established
Sessions. Sessions.
Discussion: Discussion:
The EA functions in the Signaling and Media Planes. The Tester The EA functions in the Signaling and Media Planes. The Tester
skipping to change at page 24, line 31 skipping to change at page 16, line 24
See Also: See Also:
Media Plane Media Plane
Signaling Plane Signaling Plane
Established Session Established Session
Associated Media Associated Media
3.2.2. Signaling Server 3.2.2. Signaling Server
Definition: Definition:
Device in the test topology that acts to create sessions between Device in the test topology that facilitates the creation of
EAs. This device is either a DUT or a component of a SUT. sessions between EAs. This device is the DUT.
Discussion: Discussion:
The DUT MUST be an RFC 3261 capable network equipment such as a The DUT is a RFC3261-capable network intermediary such as a
Registrar, Redirect Server, User Agent Server, Stateless Proxy, or Registrar, Redirect Server, User Agent Server, Stateless Proxy,
Stateful Proxy. A DUT MAY also include B2BUA or SBC. Stateful Proxy, B2BUA or SBC.
Measurement Units: Measurement Units:
NA NA
Issues: Issues:
None. None.
See Also: See Also:
Signaling Plane Signaling Plane
3.2.3. SIP-Aware Stateful Firewall 3.2.3. SIP Transport Protocol
Definition:
Device in the test topology that provides protection against
various types of security threats to which the Signaling and Media
Planes of the EAs and Signaling Server are vulnerable.
Discussion:
Threats may include Denial-of-Service, theft of service and misuse
of service. The SIP-Aware Stateful Firewall MAY be an internal
component or function of the Session Server. The SIP-Aware
Stateful Firewall MAY be a standalone device. If it is a
standalone device it MUST be paired with a Signaling Server. If
it is a standalone device it MUST be benchmarked as part of a SUT.
SIP-Aware Stateful Firewalls MAY include Network Address
Translation (NAT) functionality. Ideally, the inclusion of the
SIP-Aware Stateful Firewall in the SUT does not lower the measured
values of the performance benchmarks.
Measurement Units:
N/A
Issues:
None.
See Also:
3.2.4. SIP Transport Protocol
Definition: Definition:
The protocol used for transport of the Signaling Plane messages. The protocol used for transport of the Signaling Plane messages.
Discussion: Discussion:
Performance benchmarks may vary for the same SIP networking device Performance benchmarks may vary for the same SIP networking device
depending upon whether TCP, UDP, TLS, SCTP, or another transport depending upon whether TCP, UDP, TLS, SCTP, websockets [RFC7118]
layer protocol is used. For this reason it MAY be necessary to or any future transport layer protocol is used. For this reason
measure the SIP Performance Benchmarks using these various it is necessary to measure the SIP Performance Benchmarks using
transport protocols. Performance Benchmarks MUST report the SIP these various transport protocols. Performance Benchmarks MUST
Transport Protocol used to obtain the benchmark results. report the SIP Transport Protocol used to obtain the benchmark
results.
Measurement Units: Measurement Units:
TCP,UDP, SCTP, TLS over TCP, TLS over UDP, or TLS over SCTP While these are not units of measure, they are attributes that are
one of many factors that will contribute to the value of the
measurements to be taken. TCP, UDP, SCTP, TLS over TCP, TLS over
UDP, TLS over SCTP, and websockets are among the possible values
to be recorded as part of the test.
Issues: Issues:
None. None.
See Also: See Also:
3.3. Test Setup Parameters 3.3. Test Setup Parameters
3.3.1. Session Attempt Rate 3.3.1. Session Attempt Rate
Definition: Definition:
Configuration of the EA for the number of sessions per second that Configuration of the EA for the number of sessions per second that
the EA attempts to establish using the services of the DUT/SUT. the EA attempts to establish using the services of the DUT.
Discussion: Discussion:
The Session Attempt Rate is the number of sessions per second that The Session Attempt Rate is the number of sessions per second that
the EA sends toward the DUT/SUT. Some of the sessions attempted the EA sends toward the DUT. Some of the sessions attempted may
may not result in a session being established. A session in this not result in a session being established. A session in this case
case may be either an IS or an NS. may be either an IS or an NS.
Measurement Units: Measurement Units:
Session attempts per second Session attempts per second
Issues: Issues:
None. None.
See Also: See Also:
Session Session
Session Attempt Session Attempt
3.3.2. IS Media Attempt Rate 3.3.2. Establishment Threshold Time
Definition:
Configuration on the EA for the rate, measured in sessions per
second, at which the EA attempts to establish INVITE-initiated
sessions with Associated Media, using the services of the DUT/SUT.
Discussion:
An IS is not required to include a media description. The IS
Media Attempt Rate defines the number of media sessions we are
trying to create, not the number of media sessions that are
actually created. Some attempts might not result in successful
sessions established on the DUT.
Measurement Units:
session attempts per second (saps)
Issues:
None.
See Also:
IS
3.3.3. Establishment Threshold Time
Definition: Definition:
Configuration of the EA for representing the amount of time that Configuration of the EA that represents the amount of time that an
an EA will wait before declaring a Session Attempt Failure. EA client will wait for a response from an EA server before
declaring a Session Attempt Failure.
Discussion: Discussion:
This time duration is test dependent. This time duration is test dependent.
It is RECOMMENDED that the Establishment Threshold Time value be It is RECOMMENDED that the Establishment Threshold Time value be
set to Timer B (for ISs) or Timer F (for NSs) as specified in RFC set to Timer B (for ISs) or Timer F (for NSs) as specified in RFC
3261, Table 4 [RFC3261]. Following the default value of T1 3261, Table 4 [RFC3261]. Following the default value of T1
(500ms) specified in the table and a constant multiplier of 64 (500ms) specified in the table and a constant multiplier of 64
gives a value of 32 seconds for this timer (i.e., 500ms * 64 = gives a value of 32 seconds for this timer (i.e., 500ms * 64 =
32s). 32s).
Measurement Units: Measurement Units:
seconds Seconds
Issues: Issues:
None. None.
See Also: 3.3.3. Session Duration
session establishment failure
3.3.4. Session Duration
Definition: Definition:
Configuration of the EA that represents the amount of time that Configuration of the EA that represents the amount of time that
the SIP dialog is intended to exist between the two EAs associated the SIP dialog is intended to exist between the two EAs associated
with the test. with the test.
Discussion: Discussion:
The time at which the BYE is sent will control the Session The time at which the BYE is sent will control the Session
Duration Duration.
Normally the Session Duration will be the same as the Media
Session Hold Time. However, it is possible that the dialog
established between the two EAs can support different media
sessions at different points in time. Providing both parameters
allows the testing agency to explore this possibility.
Measurement Units: Measurement Units:
seconds seconds
Issues: Issues:
None. None.
See Also: See Also:
Media Session Hold Time
3.3.5. Media Packet Size
Definition:
Configuration on the EA for a fixed size of packets used for media
streams.
Discussion:
For a single benchmark test, all sessions use the same size packet
for media streams. The size of packets can cause variation in
performance benchmark measurements.
Measurement Units:
bytes
Issues:
None.
See Also:
3.3.6. Media Offered Load
Definition:
Configuration of the EA for the constant rate of Associated Media
traffic offered by the EA to the DUT/SUT for one or more
Established Sessions of type IS.
Discussion:
The Media Offered Load to be used for a test MUST be reported with
three components:
1. per Associated Media stream;
2. per IS;
3. aggregate.
For a single benchmark test, all sessions use the same Media
Offered Load per Media Stream. There may be multiple Associated
Media streams per IS. The aggregate is the sum of all Associated
Media for all IS.
Measurement Units:
packets per second (pps)
Issues:
None.
See Also:
Established Session
Invite Initiated Session
Associated Media
3.3.7. Media Session Hold Time 3.3.4. Media Packet Size
Definition: Definition:
Parameter configured at the EA, that represents the amount of time Configuration on the EA for a fixed number of frames or samples to
that the Associated Media for an Established Session of type IS be sent in each RTP packet of the media session.
will last.
Discussion: Discussion:
The Associated Media streams may be bi-directional or uni-
directional as indicated in the test methodology.
Normally the Media Session Hold Time will be the same as the
Session Duration. However, it is possible that the dialog
established between the two EAs can support different media
sessions at different points in time. Providing both parameters
allows the testing agency to explore this possibility.
Measurement Units:
seconds
Issues: For a single benchmark test, media sessions use a defined number
None. of samples or frames per RTP packet. If two SBCs, for example,
used the same codec but one puts more frames into the RTP packet,
See Also: this might cause variation in the performance benchmark results.
Associated Media
Established Session
Invite-initiated Session (IS)
3.3.8. Loop Detection Option
Definition:
An option that causes a Proxy to check for loops in the routing of
a SIP request before forwarding the request.
Discussion:
This is an optional process that a SIP proxy may employ; the
process is described under Proxy Behavior in RFC 3261 [RFC3261] in
Section 16.3 Request Validation and that section also contains
suggestions as to how the option could be implemented. Any
procedure to detect loops will use processor cycles and hence
could impact the performance of a proxy.
Measurement Units: Measurement Units:
N/A An integer number of frames or samples, depending on whether
hybrid- or sample-based codec are used, respectively.
Issues: Issues:
None. None.
See Also: See Also:
3.3.9. Forking Option 3.3.5. Codec Type
Definition: Definition:
An option that enables a Proxy to fork requests to more than one The name of the codec used to generate the media session.
destination.
Discussion: Discussion
This is an process that a SIP proxy may employ to find the UAS. For a single benchmark text, all sessions use the same size packet
The option is described under Proxy Behavior in RFC 3261 in for media streams. The size of packets can cause a variation in
Section 16.1. A proxy that uses forking must maintain state the performance benchmark measurements.
information and this will use processor cycles and memory. Thus
the use of this option could impact the performance of a proxy and
different implementations could produce different impacts.
SIP supports serial or parallel forking. When performing a test,
the type of forking mode MUST be indicated.
Measurement Units: Measurement Units:
The number of endpoints that will receive the forked invitation. This is a textual name (alphanumeric) assigned to uniquely
A value of 1 indicates that the request is destined to only one identify the codec.
endpoint, a value of 2 indicates that the request is forked to two
endpoints, and so on. This is an integer value ranging between 1
and N inclusive, where N is the maximum number of endpoints to
which the invitation is sent.
Type of forking used, namely parallel or serial.
Issues: Issues:
None. None.
See Also: See Also:
3.4. Benchmarks 3.4. Benchmarks
3.4.1. Registration Rate 3.4.1. Session Establishment Rate
Definition:
The maximum number of registrations that can be successfully
completed by the DUT/SUT in a given time period without
registration failures in that time period.
Discussion:
This benchmark is obtained with zero failure in which 100% of the
registrations attempted by the EA are successfully completed by
the DUT/SUT. The registration rate provisioned on the Emulated
Agent is raised and lowered as described in the algorithm in the
companion methodology draft [I-D.ietf-bmwg-sip-bench-meth] until a
traffic load consisting of registrations at the given attempt rate
over the sustained period of time identified by T in the algorithm
completes without failure.
Measurement Units:
registrations per second (rps)
Issues:
None.
See Also:
3.4.2. Session Establishment Rate
Definition: Definition:
The maximum number of sessions that can be successfully completed The maximum value of the Session Attempt Rate that the DUT can
by the DUT/SUT in a given time period without session handle for an extended, pre-defined, period with zero failures.
establishment failures in that time period.
Discussion: Discussion:
This benchmark is obtained with zero failure in which 100% of the This benchmark is obtained with zero failure in which 100% of the
sessions attempted by the Emulated Agent are successfully sessions attempted by the Emulated Agent are successfully
completed by the DUT/SUT. The session attempt rate provisioned on completed by the DUT. The session attempt rate provisioned on the
the EA is raised and lowered as described in the algorithm in the EA is raised and lowered as described in the algorithm in the
accompanying methodology document, until a traffic load at the accompanying methodology document, until a traffic load at the
given attempt rate over the sustained period of time identified by given attempt rate over the sustained period of time identified by
T in the algorithm completes without any failed session attempts. T in the algorithm completes without any failed session attempts.
Sessions may be IS or NS or a mix of both and will be defined in Sessions may be IS or NS or a mix of both and will be defined in
the particular test. the particular test.
Measurement Units: Measurement Units:
sessions per second (sps) sessions per second (sps)
Issues: Issues:
None. None.
See Also: See Also:
Invite-initiated Sessions Invite-Initiated Sessions
Non-INVITE initiated Sessions Non-Invite-Initiated Sessions
Session Attempt Rate
3.4.3. Session Capacity
Definition:
The maximum value of Standing Sessions Count achieved by the DUT/
SUT during a time period T in which the EA is sending session
establishment messages at the Session Establishment Rate.
Discussion:
Sessions may be IS or NS. If they are IS they can be with or
without media. When benchmarking Session Capacity for sessions
with media it is required that these sessions be permanently
established, i.e., they remain active for the duration of the
test. In the signaling plane, this requirement means that the
dialog lasts as long as the test lasts. When media is present,
the Media Session Hold Time MUST be set to infinity so that
sessions remain established for the duration of the test. If the
DUT/SUT is dialog-stateful, then we expect its performance will be
impacted by setting Media Session Hold Time to infinity, since the
DUT/SUT will need to allocate resources to process and store the
state information. The report of the Session Capacity must
include the Session Establishment Rate at which it was measured.
Measurement Units:
sessions
Issues:
None.
See Also:
Established Session
Session Attempt Rate Session Attempt Rate
Session Attempt Failure
3.4.4. Session Overload Capacity
Definition:
The maximum number of Established Sessions that can exist
simultaneously on the DUT/SUT until it stops responding to Session
Attempts.
Discussion: 3.4.2. Registration Rate
Session Overload Capacity is measured after the Session Capacity
is measured. The Session Overload Capacity is greater than or
equal to the Session Capacity. When benchmarking Session Overload
Capacity, continue to offer Session Attempts to the DUT/SUT after
the first Session Attempt Failure occurs and measure Established
Sessions until there is no SIP message response for the duration
of the Establishment Threshold. Note that the Session
Establishment Performance is expected to decrease after the first
Session Attempt Failure occurs.
Units:
Sessions
Issues:
None.
See Also:
Overload
Session Capacity
Session Attempt Failure
3.4.5. Session Establishment Performance
Definition:
The percent of Session Attempts that become Established Sessions
over the duration of a benchmarking test.
Discussion:
Session Establishment Performance is a benchmark to indicate
session establishment success for the duration of a test. The
duration for measuring this benchmark is to be specified in the
Methodology. The Session Duration SHOULD be configured to
infinity so that sessions remain established for the entire test
duration.
Session Establishment Performance is calculated as shown in the
following equation:
Session Establishment = Total Established Sessions
Performance --------------------------
Total Session Attempts
Session Establishment Performance may be monitored real-time
during a benchmarking test. However, the reporting benchmark MUST
be based on the total measurements for the test duration.
Measurement Units:
Percent (%)
Issues:
None.
See Also:
Established Session
Session Attempt
3.4.6. Session Attempt Delay
Definition: Definition:
The average time measured at the EA for a Session Attempt to The maximum value of the Registration Attempt Rate that the DUT
result in an Established Session. can handle for an extended, pre-defined, period with zero
failures.
Discussion: Discussion:
Time is measured from when the EA sends the first INVITE for the This benchmark is obtained with zero failures in which 100% of the
call-ID in the case of an IS. Time is measured from when the EA registrations attempted by the EA are successfully completed by
sends the first non-INVITE message in the case of an NS. Session the DUT. The registration rate provisioned on the Emulated Agent
Attempt Delay MUST be measured for every established session to is raised and lowered as described in the algorithm in the
calculate the average. Session Attempt Delay MUST be measured at companion methodology draft [I-D.ietf-bmwg-sip-bench-meth] until a
the Session Establishment Rate. traffic load consisting of registration attempts at the given
attempt rate over the period of time necessary to attempt N
registrations completes without failure, where N is a parameter
specified in the algorithm and recorded in the Test Setup Report.
This benchmark is described separately from the Session
Establishment Rate (Section 3.4.1), although it could be
considered a special case of that benchmark, since a REGISTER
request is a request for a Non-Invite-Initiated session. It is
defined separately because it is a very important benchmark for
most SIP installations. An example demonstrating its use is an
avalanche restart, where hundreds of thousands of end points
register simultaneously following a power outage. In such a case,
an authoritative measurement of the capacity of the device to
register endpoints is useful to the network designer. Finally, in
certain controlled networks, there appears to be a difference in
the registration rate of new endpoints registering versus existing
endpoints refreshing their registrations. This benchmark can
capture these differences as well.
Measurement Units: Measurement Units:
Seconds registrations per second (rps)
Issues: Issues:
None. None.
See Also: See Also:
Session Establishment Rate
3.4.7. IM Rate 3.4.3. Registration Attempt Rate
Definition: Definition:
Maximum number of IM messages completed by the DUT/SUT. Configuration of the EA for the number of registrations per second
that the EA attempts to send to the DUT.
Discussion: Discussion:
For a UAS, the definition of success is the receipt of an IM The Registration Attempt Rate is the number of registration
request and the subsequent sending of a final response. requests per second that the EA sends toward the DUT.
For a UAC, the definition of success is the sending of an IM
request and the receipt of a final response to it. For a proxy,
the definition of success is as follows:
A. the number of IM requests it receives from the upstream client
MUST be equal to the number of IM requests it sent to the
downstream server; and
B. the number of IM responses it receives from the downstream
server MUST be equal to the number of IM requests sent to the
downstream server; and
C. the number of IM responses it sends to the upstream client
MUST be equal to the number of IM requests it received from
the upstream client.
Measurement Units: Measurement Units:
IM messages per second Registrations per second (rps)
Issues: Issues:
None. None.
See Also: See Also: Non-Invite-Initiated Session
4. IANA Considerations 4. IANA Considerations
This document requires no IANA considerations. This document requires no IANA considerations.
5. Security Considerations 5. Security Considerations
Documents of this type do not directly affect the security of Documents of this type do not directly affect the security of
Internet or corporate networks as long as benchmarking is not Internet or corporate networks as long as benchmarking is not
performed on devices or systems connected to production networks. performed on devices or systems connected to production networks.
skipping to change at page 36, line 21 skipping to change at page 22, line 29
Requirement Levels", BCP 14, RFC 2119, March 1997. Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for
Network Interconnect Devices", RFC 2544, March 1999. Network Interconnect Devices", RFC 2544, March 1999.
[RFC3261] Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston, [RFC3261] Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston,
A., Peterson, J., Sparks, R., Handley, M., and E. A., Peterson, J., Sparks, R., Handley, M., and E.
Schooler, "SIP: Session Initiation Protocol", RFC 3261, Schooler, "SIP: Session Initiation Protocol", RFC 3261,
June 2002. June 2002.
[RFC5393] Sparks, R., Lawrence, S., Hawrylyshen, A., and B. Campen,
"Addressing an Amplification Vulnerability in Session
Initiation Protocol (SIP) Forking Proxies", RFC 5393,
December 2008.
[RFC4320] Sparks, R., "Actions Addressing Identified Issues with the
Session Initiation Protocol's (SIP) Non-INVITE
Transaction", RFC 4320, January 2006.
[RFC6026] Sparks, R. and T. Zourzouvillys, "Correct Transaction
Handling for 2xx Responses to Session Initiation Protocol
(SIP) INVITE Requests", RFC 6026, September 2010.
[I-D.ietf-bmwg-sip-bench-meth] [I-D.ietf-bmwg-sip-bench-meth]
Davids, C., Gurbani, V., and S. Poretsky, "Methodology for Davids, C., Gurbani, V., and S. Poretsky, "Methodology for
Benchmarking SIP Networking Devices", Benchmarking SIP Networking Devices",
draft-ietf-bmwg-sip-bench-meth-08 (work in progress), draft-ietf-bmwg-sip-bench-meth-08 (work in progress),
January 2013. January 2013.
7.2. Informational References 7.2. Informational References
[RFC2285] Mandeville, R., "Benchmarking Terminology for LAN [RFC2285] Mandeville, R., "Benchmarking Terminology for LAN
Switching Devices", RFC 2285, February 1998. Switching Devices", RFC 2285, February 1998.
skipping to change at page 36, line 47 skipping to change at page 23, line 20
Applications", STD 64, RFC 3550, July 2003. Applications", STD 64, RFC 3550, July 2003.
[RFC3711] Baugher, M., McGrew, D., Naslund, M., Carrara, E., and K. [RFC3711] Baugher, M., McGrew, D., Naslund, M., Carrara, E., and K.
Norrman, "The Secure Real-time Transport Protocol (SRTP)", Norrman, "The Secure Real-time Transport Protocol (SRTP)",
RFC 3711, March 2004. RFC 3711, March 2004.
[RFC6357] Hilt, V., Noel, E., Shen, C., and A. Abdelal, "Design [RFC6357] Hilt, V., Noel, E., Shen, C., and A. Abdelal, "Design
Considerations for Session Initiation Protocol (SIP) Considerations for Session Initiation Protocol (SIP)
Overload Control", RFC 6357, August 2011. Overload Control", RFC 6357, August 2011.
[I-D.ietf-soc-overload-control] [RFC7118] Baz Castillo, I., Millan Villegas, J., and V. Pascual,
Gurbani, V., Hilt, V., and H. Schulzrinne, "Session "The Websocket Protocol as a Transport for the Session
Initiation Protocol (SIP) Overload Control", Initiation Protocol (SIP)", RFC 7118, January 2014.
draft-ietf-soc-overload-control-11 (work in progress),
November 2012.
Appendix A. White Box Benchmarking Terminology
Session Attempt Arrival Rate
Definition:
The number of Session Attempts received at the DUT/SUT over a
specified time period.
Discussion:
Sessions Attempts are indicated by the arrival of SIP INVITES OR
SUBSCRIBE NOTIFY messages. Session Attempts Arrival Rate
distribution can be any model selected by the user of this
document. It is important when comparing benchmarks of different
devices that same distribution model was used. Common
distributions are expected to be Uniform and Poisson.
Measurement Units:
Session attempts/sec
Issues:
None.
See Also:
Session Attempt
Authors' Addresses Authors' Addresses
Carol Davids Carol Davids
Illinois Institute of Technology Illinois Institute of Technology
201 East Loop Road 201 East Loop Road
Wheaton, IL 60187 Wheaton, IL 60187
USA USA
Phone: +1 630 682 6024 Phone: +1 630 682 6024
 End of changes. 88 change blocks. 
837 lines changed or deleted 248 lines changed or added

This html diff was produced by rfcdiff 1.41. The latest version is available from http://tools.ietf.org/tools/rfcdiff/