draft-ietf-bmwg-sip-bench-term-00.txt   draft-ietf-bmwg-sip-bench-term-01.txt 
Benchmarking Methodology Working S. Poretsky Benchmarking Methodology Working S. Poretsky
Group Allot Communications Group Allot Communications
Internet-Draft V. Gurbani Internet-Draft V. Gurbani
Expires: September 5, 2009 Bell Laboratories, Alcatel-Lucent Expires: August 12, 2010 Bell Laboratories, Alcatel-Lucent
C. Davids C. Davids
Illinois Institute of Technology Illinois Institute of Technology
March 4, 2009 February 8, 2010
Terminology for Benchmarking Session Initiation Protocol (SIP) Terminology for Benchmarking Session Initiation Protocol (SIP)
Networking Devices Networking Devices
draft-ietf-bmwg-sip-bench-term-00 draft-ietf-bmwg-sip-bench-term-01
Abstract
This document provides a terminology for benchmarking SIP performance
in networking devices. Terms are included for test components, test
setup parameters, and performance benchmark metrics for black-box
benchmarking of SIP networking devices. The performance benchmark
metrics are obtained for the SIP control plane and media plane. The
terms are intended for use in a companion methodology document for
complete performance characterization of a device in a variety of
conditions making it possible to compare performance of different
devices. It is critical to provide test setup parameters and a
methodology document for SIP performance benchmarking because SIP
allows a wide range of configuration and operational conditions that
can influence performance benchmark measurements. It is necessary to
have terminology and methodology standards to ensure that reported
benchmarks have consistent definition and were obtained following the
same procedures. Benchmarks can be applied to compare performance of
a variety of SIP networking devices.
Status of this Memo Status of this Memo
This Internet-Draft is submitted to IETF in full conformance with the This Internet-Draft is submitted to IETF in full conformance with the
provisions of BCP 78 and BCP 79. provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet- other groups may also distribute working documents as Internet-
Drafts. Drafts.
skipping to change at page 1, line 36 skipping to change at page 2, line 9
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt. http://www.ietf.org/ietf/1id-abstracts.txt.
The list of Internet-Draft Shadow Directories can be accessed at The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html. http://www.ietf.org/shadow.html.
This Internet-Draft will expire on September 5, 2009. This Internet-Draft will expire on August 12, 2010.
Copyright Notice Copyright Notice
Copyright (c) 2009 IETF Trust and the persons identified as the Copyright (c) 2010 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents in effect on the date of Provisions Relating to IETF Documents
publication of this document (http://trustee.ietf.org/license-info). (http://trustee.ietf.org/license-info) in effect on the date of
Please review these documents carefully, as they describe your rights publication of this document. Please review these documents
and restrictions with respect to this document. carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
Abstract include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
This document provides a terminology for benchmarking SIP performance described in the BSD License.
in networking devices. Terms are included for test components, test
setup parameters, and performance benchmark metrics for black-box
benchmarking of SIP networking devices. The performance benchmark
metrics are obtained for the SIP control plane and media plane. The
terms are intended for use in a companion methodology document for
complete performance characterization of a device in a variety of
conditions making it possible to compare performance of different
devices. It is critical to provide test setup parameters and a
methodology document for SIP performance benchmarking because SIP
allows a wide range of configuration and operational conditions that
can influence performance benchmark measurements. It is necessary to
have terminology and methodology standards to ensure that reported
benchmarks have consistent definition and were obtained following the
same procedures. Benchmarks can be applied to compare performance of
a variety of SIP networking devices.
Table of Contents Table of Contents
1. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 5 1. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 5
2. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 6 2. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.1. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.1. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.2. Benchmarking Models . . . . . . . . . . . . . . . . . . . 8 2.2. Benchmarking Models . . . . . . . . . . . . . . . . . . . 8
3. Term Definitions . . . . . . . . . . . . . . . . . . . . . . . 11 3. Term Definitions . . . . . . . . . . . . . . . . . . . . . . . 11
3.1. Protocol Components . . . . . . . . . . . . . . . . . . . 11 3.1. Protocol Components . . . . . . . . . . . . . . . . . . . 11
3.1.1. Session . . . . . . . . . . . . . . . . . . . . . . . 11 3.1.1. Session . . . . . . . . . . . . . . . . . . . . . . . 11
skipping to change at page 3, line 42 skipping to change at page 3, line 42
3.3.3. Establishment Threshold Time . . . . . . . . . . . . . 23 3.3.3. Establishment Threshold Time . . . . . . . . . . . . . 23
3.3.4. Media Packet Size . . . . . . . . . . . . . . . . . . 24 3.3.4. Media Packet Size . . . . . . . . . . . . . . . . . . 24
3.3.5. Media Offered Load . . . . . . . . . . . . . . . . . . 24 3.3.5. Media Offered Load . . . . . . . . . . . . . . . . . . 24
3.3.6. Media Session Hold Time . . . . . . . . . . . . . . . 25 3.3.6. Media Session Hold Time . . . . . . . . . . . . . . . 25
3.3.7. Loop Detection Option . . . . . . . . . . . . . . . . 25 3.3.7. Loop Detection Option . . . . . . . . . . . . . . . . 25
3.3.8. Forking Option . . . . . . . . . . . . . . . . . . . . 26 3.3.8. Forking Option . . . . . . . . . . . . . . . . . . . . 26
3.4. Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . 26 3.4. Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . 26
3.4.1. Registration Rate . . . . . . . . . . . . . . . . . . 26 3.4.1. Registration Rate . . . . . . . . . . . . . . . . . . 26
3.4.2. Session Establishment Rate . . . . . . . . . . . . . . 27 3.4.2. Session Establishment Rate . . . . . . . . . . . . . . 27
3.4.3. Session Capacity . . . . . . . . . . . . . . . . . . . 28 3.4.3. Session Capacity . . . . . . . . . . . . . . . . . . . 28
3.4.4. Session Overload Capacity . . . . . . . . . . . . . . 28 3.4.4. Session Overload Capacity . . . . . . . . . . . . . . 29
3.4.5. Session Establishment Performance . . . . . . . . . . 29 3.4.5. Session Establishment Performance . . . . . . . . . . 29
3.4.6. Session Attempt Delay . . . . . . . . . . . . . . . . 30 3.4.6. Session Attempt Delay . . . . . . . . . . . . . . . . 30
3.4.7. IM Rate . . . . . . . . . . . . . . . . . . . . . . . 30 3.4.7. IM Rate . . . . . . . . . . . . . . . . . . . . . . . 31
4. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 31 4. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 31
5. Security Considerations . . . . . . . . . . . . . . . . . . . 31 5. Security Considerations . . . . . . . . . . . . . . . . . . . 31
6. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 31 6. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 32
7. References . . . . . . . . . . . . . . . . . . . . . . . . . . 32 7. References . . . . . . . . . . . . . . . . . . . . . . . . . . 32
7.1. Normative References . . . . . . . . . . . . . . . . . . . 32 7.1. Normative References . . . . . . . . . . . . . . . . . . . 32
7.2. Informational References . . . . . . . . . . . . . . . . . 32 7.2. Informational References . . . . . . . . . . . . . . . . . 32
Appendix A. White Box Benchmarking Terminology . . . . . . . . . 33 Appendix A. White Box Benchmarking Terminology . . . . . . . . . 33
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 33 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 33
1. Terminology 1. Terminology
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
skipping to change at page 7, line 24 skipping to change at page 7, line 24
2.1. Scope 2.1. Scope
The scope of this work item is summarized as follows: The scope of this work item is summarized as follows:
o This terminology document describes SIP signaling (control- plane) o This terminology document describes SIP signaling (control- plane)
performance benchmarks for black-box measurements of SIP performance benchmarks for black-box measurements of SIP
networking devices. Stress and debug scenarios are not addressed networking devices. Stress and debug scenarios are not addressed
in this work item. in this work item.
o The DUT must be an RFC 3261 capable network equipment. This may o The DUT must be an RFC 3261 capable network equipment. This may
be a Registrar, Redirect Server, Stateless Proxy or Stateful be a Registrar, Redirect Server, Stateless Proxy or Stateful
Proxy. A DUT MAY also include a B2BUA, SBC, or P-CSCF Proxy. A DUT MAY also include a B2BUA, SBC functionality (this is
functionality (this is referred to as the "Signaling Server".) referred to as the "Signaling Server".) The DUT MAY be a multi-
The DUT MAY be a multi-port SIP-to-switched network gateway port SIP-to-switched network gateway implemented as a SIP UAC or
implemented as a SIP UAC or UAS. UAS.
o The DUT MAY have an internal SIP Application Level Gateway (ALG), o The DUT MAY have an internal SIP Application Level Gateway (ALG),
firewall, and/or a Network Address Translator (NAT). This is firewall, and/or a Network Address Translator (NAT). This is
referred to as the "SIP Aware Stateful Firewall." referred to as the "SIP Aware Stateful Firewall."
o The DUT or SUT MUST NOT be end user equipment, such as personal o The DUT or SUT MUST NOT be end user equipment, such as personal
digital assistant, a computer-based client, or a user terminal. digital assistant, a computer-based client, or a user terminal.
o The Tester acts as multiple "Emulated Agents" that initiate (or o The Tester acts as multiple "Emulated Agents" that initiate (or
respond) to SIP messages as session endpoints and source (or respond to) SIP messages as session endpoints and source (or
receive) associated media for established connections. receive) associated media for established connections.
o Control Signaling in presence of Media o Control Signaling in presence of Media
* The media performance is not benchmarked in this work item. * The media performance is not benchmarked in this work item.
* It is RECOMMENDED that control plane benchmarks are performed * It is RECOMMENDED that control plane benchmarks are performed
with media present, but this is optional. with media present, but this is optional.
* The SIP INVITE requests MUST include the SDP body. * The SIP INVITE requests MUST include the SDP body.
* The type of DUT dictates whether the associated media streams * The type of DUT dictates whether the associated media streams
traverse the DUT or SUT. Both scenarios are within the scope traverse the DUT or SUT. Both scenarios are within the scope
of this work item. of this work item.
* SIP is frequently used to create media streams; the control * SIP is frequently used to create media streams; the control
plane and media plane are treated as orthogonal to each other plane and media plane are treated as orthogonal to each other
in this document. While many devices support the creation of in this document. While many devices support the creation of
media streams, benchmarks that measure the performance of these media streams, benchmarks that measure the performance of these
streams are outside the scope of this document and its streams are outside the scope of this document and its
companion methodology document [I-D.ietf-bmwg-sip-bench-meth]. companion methodology document [I-D.ietf-bmwg-sip-bench-meth].
Tests may be performed with or without the creation of media Tests may be performed with or without the creation of media
streams. The presence or absence of media streams must (or streams. The presence or absence of media streams MUST be
MUST) be noted as a condition of the test as the performance of noted as a condition of the test as the performance of SIP
SIP devices may vary accordingly. Even if the media is used devices may vary accordingly. Even if the media is used during
during benchmarking, only the SIP performance will be benchmarking, only the SIP performance will be benchmarked, not
benchmarked, not the media performance or quality. the media performance or quality.
o Both INVITE and non-INVITE scenarios (such as Instant Messages or o Both INVITE and non-INVITE scenarios (such as Instant Messages or
IM) are addressed in this document. However, benchmarking SIP IM) are addressed in this document. However, benchmarking SIP
presence is not a part of this work item. presence is not a part of this work item.
o Different transport mechanisms -- such as UDP, TCP, SCTP, or TLS o Different transport mechanisms -- such as UDP, TCP, SCTP, or TLS
-- may be used; however, the specific transport mechanism MUST be -- may be used; however, the specific transport mechanism MUST be
noted as a condition of the test as the performance of SIP devices noted as a condition of the test as the performance of SIP devices
may vary accordingly. may vary accordingly.
o Looping and forking options are also considered since they impact o Looping and forking options are also considered since they impact
processing at SIP proxies. processing at SIP proxies.
o REGISTER and INVITE requests may be challenged or remain o REGISTER and INVITE requests may be challenged or remain
unchallenged for authentication purpose as this may impact the unchallenged for authentication purpose as this may impact the
performance benchmarks. Any observable performance degradation performance benchmarks. Any observable performance degradation
due to authentication is considered in the interest to the SIP due to authentication is of interest to the SIP community.
community.
o Re-INVITE requests are not considered in scope of this work item. o Re-INVITE requests are not considered in scope of this work item.
o Only session establishment is considered for the performance o Only session establishment is considered for the performance
benchmarks. Session disconnect is not considered in the scope of benchmarks. Session disconnect is not considered in the scope of
this work item. this work item.
o SIP Overload [I-D.ietf-sipping-overload-reqs] is within the scope o SIP Overload [I-D.ietf-sipping-overload-reqs] is within the scope
of this work item, but considerations on how to handle overload of this work item, but considerations on how to handle overload
are deferred to work progressing in the SIPPING working group are deferred to work progressing in the SIPPING working group
[I-D.ietf-sipping-overload-design]. Vendors are, of course, free [I-D.ietf-sipping-overload-design]. Vendors are, of course, free
to implement their specific overload control behavior as the to implement their specific overload control behavior as the
expected test outcome if it is different from the IETF expected test outcome if it is different from the IETF
skipping to change at page 9, line 5 skipping to change at page 9, line 5
o IMS-specific scenarios are not considered, but test cases can be o IMS-specific scenarios are not considered, but test cases can be
applied with 3GPP-specific SIP signaling and the P-CSCF as a DUT. applied with 3GPP-specific SIP signaling and the P-CSCF as a DUT.
2.2. Benchmarking Models 2.2. Benchmarking Models
This section shows the five models to be used when benchmarking SIP This section shows the five models to be used when benchmarking SIP
performance of a networking device. Figure 1 shows a configuration performance of a networking device. Figure 1 shows a configuration
in which the Tester acting as the Emulated agents is in loopback in which the Tester acting as the Emulated agents is in loopback
testing itself for the purpose of baselining its performance. testing itself for the purpose of baselining its performance.
+--------+ Signaling request +--------+ +--------+ Signaling request +--------+
| +----------------------------->| | | +----------------------------->| |
| Tester | | Tester | | Tester | | Tester |
| EA | Signaling response | EA | | EA | Signaling response | EA |
| |<-----------------------------+ | | |<-----------------------------+ |
+--------+ +--------+ +--------+ +--------+
Figure 1: Test topology 1 - Emulated agent (EA) baseline performance Figure 1: Test topology 1 - Emulated agent (EA) baseline performance
measurement measurement
Figure 2 shows the basic configuration for benchmarking the Figure 2 shows the basic configuration for benchmarking the
Registration of the DUT/SUT. Registration of the DUT/SUT.
+--------+ Registration +--------+ +--------+ Registration +--------+
| +----------------------------->| | | +----------------------------->| |
| Tester | | DUT | | Tester | | DUT |
| EA | Response | | | EA | Response | |
| |<-----------------------------+ | | |<-----------------------------+ |
+--------+ +--------+ +--------+ +--------+
Figure 2: Test topology 2 - Emulated agent (EA) registration to DUT/ Figure 2: Test topology 2 - Emulated agent (EA) registration to DUT/
SUT SUT
Figure 3 shows that the Tester acts as the initiating and responding Figure 3 shows that the Tester acts as the initiating and responding
Emulated Agents as the DUT/SUT forwards Session Attempts. Emulated Agents as the DUT/SUT forwards Session Attempts.
+--------+ Session +--------+ Session +--------+ +--------+ Session +--------+ Session +--------+
| | Attempt | | Attempt | | | | Attempt | | Attempt | |
| |<------------+ |<------------+ | | |<------------+ |<------------+ |
| | | | | | | | | | | |
| | Response | | Response | | | | Response | | Response | |
| Tester +------------>| DUT +------------>| Tester | | Tester +------------>| DUT +------------>| Tester |
| | | | | (EA) | | (EA) | | | | (EA) |
| | | | | | | | | | | |
+--------+ +--------+ +--------+ +--------+ +--------+ +--------+
Figure 3: Test topology 3 - DUT/SUT performance benchmark for session Figure 3: Test topology 3 - DUT/SUT performance benchmark for session
establishment without media establishment without media
Figure 4 is used when performing those same benchmarks with Figure 4 is used when performing those same benchmarks with
Associated Media traversing the DUT/SUT. Associated Media traversing the DUT/SUT.
+--------+ Session +--------+ Session +--------+ +--------+ Session +--------+ Session +--------+
| | Attempt | | Attempt | | | | Attempt | | Attempt | |
| |<------------+ |<------------+ | | |<------------+ |<------------+ |
| | | | | | | | | | | |
| | Response | | Response | | | | Response | | Response | |
| Tester +------------>| DUT +------------>| Tester | | Tester +------------>| DUT +------------>| Tester |
| | | | | (EA) | | | | | | (EA) |
| | Media | | Media | | | | Media | | Media | |
| |<===========>| |<===========>| | | |<===========>| |<===========>| |
+--------+ +--------+ +--------+ +--------+ +--------+ +--------+
Figure 4: Test topology 4 - DUT/SUT performance benchmark for session Figure 4: Test topology 4 - DUT/SUT performance benchmark for session
establishment with media traversing the DUT establishment with media traversing the DUT
Figure 5 is to be used when performing those same benchmarks with Figure 5 is to be used when performing those same benchmarks with
Associated Media, but the media does not traverse the DUT/SUT. Associated Media, but the media does not traverse the DUT/SUT.
Again, the benchmarking of the media is not within the scope of this Again, the benchmarking of the media is not within the scope of this
work item. The SIP control signaling is benchmarked in the presence work item. The SIP control signaling is benchmarked in the presence
of Associated Media to determine if the SDP body of the signaling and of Associated Media to determine if the SDP body of the signaling and
the handling of media impacts the performance of the DUT/SUT. the handling of media impacts the performance of the DUT/SUT.
+--------+ Session +--------+ Session +--------+ +--------+ Session +--------+ Session +--------+
| | Attempt | | Attempt | | | | Attempt | | Attempt | |
| |<------------+ |<------------+ | | |<------------+ |<------------+ |
| | | | | | | | | | | |
| | Response | | Response | | | | Response | | Response | |
| Tester +------------>| DUT +------------>| Tester | | Tester +------------>| DUT +------------>| Tester |
| | | | | (EA) | | | | | | (EA) |
| | | | | | | | | | | |
+--------+ +--------+ +--------+ +--------+ +--------+ +--------+
/|\ /|\ /|\ /|\
| Media | | Media |
+=============================================+ +=============================================+
Figure 5: Test topology 5 - DUT/SUT performance benchmark for session Figure 5: Test topology 5 - DUT/SUT performance benchmark for session
establishment with media external to the DUT establishment with media external to the DUT
Figure 6 illustrates the SIP signaling for an Established Session. Figure 6 illustrates the SIP signaling for an Established Session.
The Tester acts as the Emulated Agent(s) and initiates a Session The Tester acts as the Emulated Agent(s) and initiates a Session
Attempt with the DUT/SUT. When the Emulated Agent (EA) receives a Attempt with the DUT/SUT. When the Emulated Agent (EA) receives a
200 OK from the DUT/SUT that session is considered to be an 200 OK from the DUT/SUT that session is considered to be an
Established Session. Sessions can be one of two type: Invite- Established Session. The illustration indicates three states of the
Initiated Session (IS) or Non-Invite Initiated Session (NS). Failure session bring created by the EA - Attempting, Established, and
for the DUT/SUT to successfully respond within the Establishment Disconnecting. Sessions can be one of two type: Invite-Initiated
Threshold Time is considered a Session Attempt Failure. SIP Invite Session (IS) or Non-Invite Initiated Session (NS). Failure for the
messages MUST include the SDP body to specify the Associated Media. DUT/SUT to successfully respond within the Establishment Threshold
Use of Associated Media, to be sourced from the EA, is optional. Time is considered a Session Attempt Failure. SIP Invite messages
When Associated Media is used, it may traverse the DUT/SUT depending MUST include the SDP body to specify the Associated Media. Use of
upon the type of DUT/SUT. The Associated Media is shown in Figure 6 Associated Media, to be sourced from the EA, is optional. When
as "Media" connected to media ports M1 and M2 on the EA. After the Associated Media is used, it may traverse the DUT/SUT depending upon
EA sends a BYE, the session disconnects. Performance test cases for the type of DUT/SUT. The Associated Media is shown in Figure 6 as
"Media" connected to media ports M1 and M2 on the EA. After the EA
sends a BYE, the session disconnects. Performance test cases for
session disconnects are not considered in this work item (the BYE session disconnects are not considered in this work item (the BYE
request is shown for completeness.) request is shown for completeness.)
EA DUT/SUT M1 M2 EA DUT/SUT M1 M2
| | | | | | | |
| INVITE | | | | INVITE | | |
--------+-------------->| | | --------+-------------->| | |
| | | | | | | |
Attempting | | | Attempting | | |
| 200 OK | | | | 200 OK | | |
--------|<--------------| | | --------|<--------------| | |
| | | | | | | |
| | | | | | | |
| | | Media | | | | Media |
Established | |<=====>| Established | |<=====>|
| | | | | | | |
| BYE | | | | BYE | | |
--------+-------------->| | | --------+-------------->| | |
| | | | | | | |
Disconnecting | | | Disconnecting | | |
| 200 OK | | | | 200 OK | | |
--------|<--------------| | | --------|<--------------| | |
| | | | | | | |
Figure 6: Basic SIP test topology Figure 6: Basic SIP test topology
3. Term Definitions 3. Term Definitions
3.1. Protocol Components 3.1. Protocol Components
3.1.1. Session 3.1.1. Session
Definition: Definition:
skipping to change at page 14, line 4 skipping to change at page 14, line 4
Issues: Issues:
None. None.
See Also: See Also:
Media Plane Media Plane
Signaling Plane Signaling Plane
Associated Media Associated Media
Invite-initiated Session (IS) Invite-initiated Session (IS)
Non-invite-initiated Session (NS) Non-invite-initiated Session (NS)
|\ |\
| |
| \ | \
sess.sig| sess.sig|
| \ | \
| |
| \ | \
| o | o
| / | /
| / | | / |
| / | /
| / | | / |
| / | /
| / | | / |
| / | /
| / | sess.medc | / | sess.medc
|/_____________________ |/_____________________
/ / / /
/ | / |
/ / / /
sess.med / | sess.med / |
/_ _ _ _ _ _ _ _/ /_ _ _ _ _ _ _ _/
/
/ /
/ /
/ /
/
Figure 7: Application or session components Figure 7: Application or session components
3.1.2. Signaling Plane 3.1.2. Signaling Plane
Definition: Definition:
The control plane in which SIP messages [RFC3261] are exchanged The control plane in which SIP messages [RFC3261] are exchanged
between SIP Agents [RFC3261] to establish a connection for media between SIP Agents [RFC3261] to establish a connection for media
exchange. exchange.
skipping to change at page 15, line 24 skipping to change at page 15, line 24
3.1.3. Media Plane 3.1.3. Media Plane
Definition: Definition:
The data plane in which one or more media streams and their The data plane in which one or more media streams and their
associated media control protocols are exchanged after a media associated media control protocols are exchanged after a media
connection has been created by the exchange of signaling messages connection has been created by the exchange of signaling messages
in the Signaling Plane. in the Signaling Plane.
Discussion: Discussion:
Media may also be known as the the "payload" or "bearer channel". Media may also be known as the "payload" or "bearer channel". The
The Media Plane MUST include the media control protocol, if one is Media Plane MUST include the media control protocol, if one is
used, and the media stream(s). Media is analogous Data. Example used, and the media stream(s). Examples of media are audio,
media are audio, video, whiteboard, and instant messaging service. video, whiteboard, and instant messaging service. The media
The media stream is described in the SDP of the Signaling Plane. stream is described in the SDP of the Signaling Plane.
The media for a single Session is represented by session.med. The The media for a single Session is represented by session.med. The
media control protocol is represented by session.medc. media control protocol is represented by session.medc.
Measurement Units: Measurement Units:
N/A. N/A.
Issues: Issues:
None. None.
See Also: See Also:
skipping to change at page 16, line 46 skipping to change at page 16, line 46
is inversely proportional to the amount of media streams currently is inversely proportional to the amount of media streams currently
handled by that UA. handled by that UA.
Measurement Units: Measurement Units:
N/A. N/A.
Issues: Issues:
The issue of overload in SIP networks is currently a topic of The issue of overload in SIP networks is currently a topic of
discussion in the SIPPING WG. The normal response to an overload discussion in the SIPPING WG. The normal response to an overload
stimulus -- sending a 503 response -- is considered inadequate. stimulus -- sending a 503 response -- is considered inadequate.
The exact manner in how to handle overload is not determined yet The exact method for handling overload is not determined yet and
and as solutions are engineered, this draft will be updated as solutions are engineered, this draft will be updated
accordingly (Note: this draft now has a dependency on the overload accordingly (Note: this draft now has a dependency on the Overload
work in SIPPING). working group created by Dispatch.
3.1.6. Session Attempt 3.1.6. Session Attempt
Definition: Definition:
A SIP Session for which the Emulated Agent has sent the SIP INVITE A SIP Session for which the Emulated Agent has sent the SIP INVITE
or SUBSCRIBE NOTIFY and has not yet received a message response or SUBSCRIBE NOTIFY and has not yet received a message response
from the DUT/SUT. from the DUT/SUT.
Discussion: Discussion:
The attempted session may be an IS or an NS. The Session Attempt The attempted session may be an IS or an NS. The Session Attempt
skipping to change at page 18, line 15 skipping to change at page 18, line 15
3.1.8. Invite-initiated Session (IS) 3.1.8. Invite-initiated Session (IS)
Definition: Definition:
A Session that is created by an exchange of messages in the A Session that is created by an exchange of messages in the
Signaling Plane, the first of which is a SIP INVITE request. Signaling Plane, the first of which is a SIP INVITE request.
Discussion: Discussion:
An IS is identified by the Call-ID, To-tag, and From-tag of the An IS is identified by the Call-ID, To-tag, and From-tag of the
SIP message that establishes the session. These three fields are SIP message that establishes the session. These three fields are
used to identify a SIP Dialog (RFC3261 [RFC3261]). An IS MAY have used to identify a SIP Dialog (RFC3261 [RFC3261]). An IS may have
Associated Media description in the SDP body. An IS MAY have Associated Media description in the SDP body. An IS may have
multiple Associated Media streams. The inclusion of media is test multiple Associated Media streams. The inclusion of media is test
case dependent. An IS is successfully established if the case dependent. An IS is successfully established if the
following two conditions are met: following two conditions are met:
1. Sess.sig is established by the end of Establishment Threshold 1. Sess.sig is established by the end of Establishment Threshold
Time (c.f. Section 3.3.3), and Time (c.f. Section 3.3.3), and
2. The media session described in the SDP body of the signaling 2. The media session described in the SDP body of the signaling
message is present. message is present.
Measurement Units: Measurement Units:
N/A. N/A.
skipping to change at page 18, line 45 skipping to change at page 18, line 45
3.1.9. Non-INVITE-initiated Session (NS) 3.1.9. Non-INVITE-initiated Session (NS)
Definition: Definition:
A session that is created by an exchange of messages in the A session that is created by an exchange of messages in the
Signaling Plane that does not include an initial SIP INVITE Signaling Plane that does not include an initial SIP INVITE
message. message.
Discussion: Discussion:
An NS is successfully established if the Session Attempt via a An NS is successfully established if the Session Attempt via a
non- INVITE request results in the EA receiving a 2xx or 3xx reply non- INVITE request results in the EA receiving a 2xx reply from
from the DUT/SUT before the expiration of the Establishment the DUT/SUT before the expiration of the Establishment Threshold
Threshold timer (c.f., Section 3.3.3). An example of a NS is a timer (c.f., Section 3.3.3). An example of a NS is a session
session created by the SUBSCRIBE request. created by the SUBSCRIBE request.
Measurement Units: Measurement Units:
N/A. N/A.
Issues: Issues:
None. None.
See Also: See Also:
Session Session
Invite-initiated Session Invite-initiated Session
skipping to change at page 20, line 44 skipping to change at page 20, line 44
See Also: See Also:
Media Plane Media Plane
Signaling Plane Signaling Plane
Established Session Established Session
Associated Media Associated Media
3.2.2. Signaling Server 3.2.2. Signaling Server
Definition: Definition:
Device in test topology that acts as a proxy on the signaling Device in test topology that acts to create sessions between
plane between Emulated Agents. This device is either a DUT or Emulated Agents in the media plane. This device is either a DUT
component of a SUT. or component of a SUT.
Discussion: Discussion:
The DUT MUST be a RFC 3261 capable network equipment such as a The DUT MUST be a RFC 3261 capable network equipment such as a
Registrar, Redirect Server, User Agent Server, Stateless Proxy, or Registrar, Redirect Server, User Agent Server, Stateless Proxy, or
Stateful Proxy. A DUT MAY also include B2BUA, SBC or P-CSCF Stateful Proxy. A DUT MAY also include B2BUA or SBC.
functionality.
Measurement Units: Measurement Units:
NA NA
Issues: Issues:
None. None.
See Also: See Also:
Signaling Plane Signaling Plane
skipping to change at page 21, line 32 skipping to change at page 21, line 31
Definition: Definition:
Device in test topology that provides Denial-of-Service (DoS) Device in test topology that provides Denial-of-Service (DoS)
Protection to the Signaling and Media Planes for the Emulated Protection to the Signaling and Media Planes for the Emulated
Agents and Signaling Server Agents and Signaling Server
Discussion: Discussion:
The SIP-Aware Stateful Firewall MAY be an internal component or The SIP-Aware Stateful Firewall MAY be an internal component or
function of the Session Server. The SIP-Aware Stateful Firewall function of the Session Server. The SIP-Aware Stateful Firewall
MAY be a standalone device. If it is a standalone device it MUST MAY be a standalone device. If it is a standalone device it MUST
be paired with a Signaling Server. If it is a standalone device be paired with a Signaling Server. If it is a standalone device
it MUST be benchmarked as a SUT. SIP-Aware Stateful Firewalls MAY it MUST be benchmarked as part of a SUT. SIP-Aware Stateful
include Network Address Translation (NAT) functionality. Ideally, Firewalls MAY include Network Address Translation (NAT)
the inclusion of the SIP-Aware Stateful Firewall as a SUT has no functionality. Ideally, the inclusion of the SIP-Aware Stateful
degradation to the measured performance benchmarks. Firewall as a SUT has no degradation to the measured performance
benchmarks.
Measurement Units: Measurement Units:
N/A N/A
Issues: Issues:
None. None.
See Also: See Also:
3.2.4. SIP Transport Protocol 3.2.4. SIP Transport Protocol
skipping to change at page 25, line 18 skipping to change at page 25, line 18
None. None.
See Also: See Also:
Established Session Established Session
Invite Initiated Session Invite Initiated Session
Associated Media Associated Media
3.3.6. Media Session Hold Time 3.3.6. Media Session Hold Time
Definition: Definition:
Configuration of the Emulated Agent for the amount of time Parameter configured at the Emulated Agent, that represents the
measured at the Emulated Agent for the Associated Media for an amount of time that the Associated Media for an Established
Established Session of type IS media to flow between the Emulated Session of type IS will last .
Agent and the DUT/SUT.
Discussion: Discussion:
The Associated Media streams may be bi-directional or uni- The Associated Media streams may be bi-directional or uni-
directional as indicated in the test methodology. directional as indicated in the test methodology.
Measurement Units: Measurement Units:
seconds seconds
Issues: Issues:
None. None.
skipping to change at page 26, line 30 skipping to change at page 26, line 30
This is an process that a SIP proxy may employ to find the UAS. This is an process that a SIP proxy may employ to find the UAS.
The option is described under Proxy Behavior in RFC 3261 in The option is described under Proxy Behavior in RFC 3261 in
Section 16.1. A proxy that uses forking must maintain state Section 16.1. A proxy that uses forking must maintain state
information and this will use processor cycles and memory. Thus information and this will use processor cycles and memory. Thus
the use of this option could impact the performance of a proxy and the use of this option could impact the performance of a proxy and
different implementations could produce different impacts. different implementations could produce different impacts.
SIP supports serial or parallel forking. When performing a test, SIP supports serial or parallel forking. When performing a test,
the type of forking mode must be indicated. the type of forking mode must be indicated.
Measurement Units: Measurement Units:
The number of locations the proxy is asked to fork to. As the The number of endpoints that will receive the forked invitation.
locations are varied, the time it takes to return a final response A value of 1 indicates that the request is destined to only one
is measured (in seconds). endpoint, a value of 2 indicates that the request is forked to two
Type of forking used: parallel or serial. endpoints, and so on. This is an integer value ranging between 1
and N inclusive, where N is the maximum number of endpoints to
which the invitation is sent.
Type of forking used, namely parallel or serial.
Issues: Issues:
None. None.
See Also: See Also:
3.4. Benchmarks 3.4. Benchmarks
3.4.1. Registration Rate 3.4.1. Registration Rate
skipping to change at page 27, line 28 skipping to change at page 27, line 31
See Also: See Also:
3.4.2. Session Establishment Rate 3.4.2. Session Establishment Rate
Definition: Definition:
The maximum average rate at which the DUT/SUT can successfully The maximum average rate at which the DUT/SUT can successfully
establish sessions. establish sessions.
Discussion: Discussion:
This is the average number of sessions successfully established in This metric is an average of maximas. Each maximum is measured in
continuous one-second intervals with prior sessions remaining a separate sample. The Session Establishment Rate is the average
active. The session establishment rate is calculated using the of the maximas established in each individual sample. In each
following equation (n = number of samples): sample, the maximum in question is the number of sessions
successfully established in continuous one-second intervals with
prior sessions remaining active. This maximum is designated in
the equation below as "rate in sample i". The session
establishment rate is calculated using the following equation (n =
number of samples):
n n
-- --
\ rate at time i \ rate at sample i
/ /
-- --
i = 1 i = 1
--------------------- ---------------------
(n) (n)
This benchmark is obtained with zero failure in which 100% of the In each sample, the maximum is obtained by testing to failure.
sessions introduced by the Emulated Agent are successfully With zero failure, 100% of the sessions introduced by the Emulated
established. The maximum value is obtained by testing to failure. Agent are successfully established. The maximum value is obtained
This means that the session rate provisioned on the EA is raised by testing to failure. This means that the session rate
progressively until a Session Attempt Failure is observed. provisioned on the EA is raised progressively until a Session
Sessions may be IS or NS or a a mix of both and will be defined in Attempt Failure is observed. The maximum rate is the rate
the particular test. acheived in the interval prior to the interval in which the
failure is observed. Sessions may be IS or NS or a a mix of both
and will be defined in the particular test.
Measurement Units: Measurement Units:
sessions per second (sps) sessions per second (sps)
Issues: Issues:
None. None.
See Also: See Also:
Invite-initiated Sessions Invite-initiated Sessions
Non-INVITE initiated Sessions Non-INVITE initiated Sessions
skipping to change at page 29, line 38 skipping to change at page 30, line 4
Session Capacity Session Capacity
Session Attempt Failure Session Attempt Failure
3.4.5. Session Establishment Performance 3.4.5. Session Establishment Performance
Definition: Definition:
The percent of Session Attempts that become Established Sessions The percent of Session Attempts that become Established Sessions
over the duration of a benchmarking test. over the duration of a benchmarking test.
Discussion: Discussion:
Session Establishment Performance is a benchmark to indicate Session Establishment Performance is a benchmark to indicate
session establishment success for the duration of a test. The session establishment success for the duration of a test. The
duration for measuring this benchmark is to be specified in the duration for measuring this benchmark is to be specified in the
Methodology. The Session Duration SHOULD be configured to Methodology. The Session Duration SHOULD be configured to
infinity so that sessions remain established for the entire test infinity so that sessions remain established for the entire test
duration. duration.
Session Establishment Performance is calculated as shown in the Session Establishment Performance is calculated as shown in the
following equation: following equation:
Session Establishment = Total Established Sessions Session Establishment = Total Established Sessions
Performance -------------------------- Performance --------------------------
Total Session Attempts Total Session Attempts
Session Establishment Performance may be monitored real-time Session Establishment Performance may be monitored real-time
during a benchmarking test. However, the reporting benchmark MUST during a benchmarking test. However, the reporting benchmark MUST
be based on the total measurements for the test duration. be based on the total measurements for the test duration.
Measurement Units: Measurement Units:
Percent (%) Percent (%)
Issues: Issues:
None. None.
skipping to change at page 32, line 20 skipping to change at page 32, line 32
Network Interconnect Devices", RFC 2544, March 1999. Network Interconnect Devices", RFC 2544, March 1999.
[RFC3261] Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston, [RFC3261] Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston,
A., Peterson, J., Sparks, R., Handley, M., and E. A., Peterson, J., Sparks, R., Handley, M., and E.
Schooler, "SIP: Session Initiation Protocol", RFC 3261, Schooler, "SIP: Session Initiation Protocol", RFC 3261,
June 2002. June 2002.
[I-D.ietf-bmwg-sip-bench-meth] [I-D.ietf-bmwg-sip-bench-meth]
Poretsky, S., Gurbani, V., and C. Davids, "Methodology for Poretsky, S., Gurbani, V., and C. Davids, "Methodology for
Benchmarking SIP Networking Devices", Benchmarking SIP Networking Devices",
draft-ietf-bmwg-sip-bench-meth-00 (work in progress), draft-ietf-bmwg-sip-bench-meth-01 (work in progress),
March 2009. January 2010.
7.2. Informational References 7.2. Informational References
[RFC2285] Mandeville, R., "Benchmarking Terminology for LAN [RFC2285] Mandeville, R., "Benchmarking Terminology for LAN
Switching Devices", RFC 2285, February 1998. Switching Devices", RFC 2285, February 1998.
[RFC1242] Bradner, S., "Benchmarking terminology for network [RFC1242] Bradner, S., "Benchmarking terminology for network
interconnection devices", RFC 1242, July 1991. interconnection devices", RFC 1242, July 1991.
[RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V. [RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V.
Jacobson, "RTP: A Transport Protocol for Real-Time Jacobson, "RTP: A Transport Protocol for Real-Time
Applications", STD 64, RFC 3550, July 2003. Applications", STD 64, RFC 3550, July 2003.
[RFC3711] Baugher, M., McGrew, D., Naslund, M., Carrara, E., and K. [RFC3711] Baugher, M., McGrew, D., Naslund, M., Carrara, E., and K.
Norrman, "The Secure Real-time Transport Protocol (SRTP)", Norrman, "The Secure Real-time Transport Protocol (SRTP)",
RFC 3711, March 2004. RFC 3711, March 2004.
[I-D.ietf-sipping-overload-design] [I-D.ietf-sipping-overload-design]
Hilt, V., "Design Considerations for Session Initiation Hilt, V., Noel, E., Shen, C., and A. Abdelal, "Design
Protocol (SIP) Overload Control", Considerations for Session Initiation Protocol (SIP)
draft-ietf-sipping-overload-design-00 (work in progress), Overload Control", draft-ietf-sipping-overload-design-02
October 2008. (work in progress), July 2009.
[I-D.ietf-sipping-overload-reqs] [I-D.ietf-sipping-overload-reqs]
Rosenberg, J., "Requirements for Management of Overload in Rosenberg, J., "Requirements for Management of Overload in
the Session Initiation Protocol", the Session Initiation Protocol",
draft-ietf-sipping-overload-reqs-05 (work in progress), draft-ietf-sipping-overload-reqs-05 (work in progress),
July 2008. July 2008.
Appendix A. White Box Benchmarking Terminology Appendix A. White Box Benchmarking Terminology
Session Attempt Arrival Rate Session Attempt Arrival Rate
skipping to change at page 33, line 34 skipping to change at page 34, line 9
Issues: Issues:
None. None.
See Also: See Also:
Session Attempt Session Attempt
Authors' Addresses Authors' Addresses
Scott Poretsky Scott Poretsky
Allot Communications Allot Communications
67 South Bedford Street, Suite 400 300 TradeCenter, Suite 4680
Burlington, MA 08103 Woburn, MA 08101
USA USA
Phone: +1 508 309 2179 Phone: +1 508 309 2179
Email: sporetsky@allot.com Email: sporetsky@allot.com
Vijay K. Gurbani Vijay K. Gurbani
Bell Laboratories, Alcatel-Lucent Bell Laboratories, Alcatel-Lucent
1960 Lucent Lane 1960 Lucent Lane
Rm 9C-533 Rm 9C-533
Naperville, IL 60566 Naperville, IL 60566
USA USA
Phone: +1 630 224 0216 Phone: +1 630 224 0216
Email: vkg@alcatel-lucent.com Email: vkg@alcatel-lucent.com
 End of changes. 47 change blocks. 
194 lines changed or deleted 210 lines changed or added

This html diff was produced by rfcdiff 1.38. The latest version is available from http://tools.ietf.org/tools/rfcdiff/