draft-ietf-bmwg-sip-bench-term-01.txt   draft-ietf-bmwg-sip-bench-term-02.txt 
Benchmarking Methodology Working S. Poretsky Benchmarking Methodology Working S. Poretsky
Group Allot Communications Group Allot Communications
Internet-Draft V. Gurbani Internet-Draft V. Gurbani
Expires: August 12, 2010 Bell Laboratories, Alcatel-Lucent Expires: January 13, 2011 Bell Laboratories, Alcatel-Lucent
C. Davids C. Davids
Illinois Institute of Technology Illinois Institute of Technology
February 8, 2010 July 12, 2010
Terminology for Benchmarking Session Initiation Protocol (SIP) Terminology for Benchmarking Session Initiation Protocol (SIP)
Networking Devices Networking Devices
draft-ietf-bmwg-sip-bench-term-01 draft-ietf-bmwg-sip-bench-term-02
Abstract Abstract
This document provides a terminology for benchmarking SIP performance This document provides a terminology for benchmarking SIP performance
in networking devices. Terms are included for test components, test in networking devices. Terms are included for test components, test
setup parameters, and performance benchmark metrics for black-box setup parameters, and performance benchmark metrics for black-box
benchmarking of SIP networking devices. The performance benchmark benchmarking of SIP networking devices. The performance benchmark
metrics are obtained for the SIP control plane and media plane. The metrics are obtained for the SIP control plane and media plane. The
terms are intended for use in a companion methodology document for terms are intended for use in a companion methodology document for
complete performance characterization of a device in a variety of complete performance characterization of a device in a variety of
skipping to change at page 2, line 9 skipping to change at page 2, line 9
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt. http://www.ietf.org/ietf/1id-abstracts.txt.
The list of Internet-Draft Shadow Directories can be accessed at The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html. http://www.ietf.org/shadow.html.
This Internet-Draft will expire on August 12, 2010. This Internet-Draft will expire on January 13, 2011.
Copyright Notice Copyright Notice
Copyright (c) 2010 IETF Trust and the persons identified as the Copyright (c) 2010 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
skipping to change at page 3, line 23 skipping to change at page 3, line 23
3.1.1. Session . . . . . . . . . . . . . . . . . . . . . . . 11 3.1.1. Session . . . . . . . . . . . . . . . . . . . . . . . 11
3.1.2. Signaling Plane . . . . . . . . . . . . . . . . . . . 14 3.1.2. Signaling Plane . . . . . . . . . . . . . . . . . . . 14
3.1.3. Media Plane . . . . . . . . . . . . . . . . . . . . . 15 3.1.3. Media Plane . . . . . . . . . . . . . . . . . . . . . 15
3.1.4. Associated Media . . . . . . . . . . . . . . . . . . . 15 3.1.4. Associated Media . . . . . . . . . . . . . . . . . . . 15
3.1.5. Overload . . . . . . . . . . . . . . . . . . . . . . . 16 3.1.5. Overload . . . . . . . . . . . . . . . . . . . . . . . 16
3.1.6. Session Attempt . . . . . . . . . . . . . . . . . . . 17 3.1.6. Session Attempt . . . . . . . . . . . . . . . . . . . 17
3.1.7. Established Session . . . . . . . . . . . . . . . . . 17 3.1.7. Established Session . . . . . . . . . . . . . . . . . 17
3.1.8. Invite-initiated Session (IS) . . . . . . . . . . . . 18 3.1.8. Invite-initiated Session (IS) . . . . . . . . . . . . 18
3.1.9. Non-INVITE-initiated Session (NS) . . . . . . . . . . 18 3.1.9. Non-INVITE-initiated Session (NS) . . . . . . . . . . 18
3.1.10. Session Attempt Failure . . . . . . . . . . . . . . . 19 3.1.10. Session Attempt Failure . . . . . . . . . . . . . . . 19
3.1.11. Standing Sessions . . . . . . . . . . . . . . . . . . 19 3.1.11. Standing Sessions Count . . . . . . . . . . . . . . . 19
3.2. Test Components . . . . . . . . . . . . . . . . . . . . . 20 3.2. Test Components . . . . . . . . . . . . . . . . . . . . . 20
3.2.1. Emulated Agent . . . . . . . . . . . . . . . . . . . . 20 3.2.1. Emulated Agent . . . . . . . . . . . . . . . . . . . . 20
3.2.2. Signaling Server . . . . . . . . . . . . . . . . . . . 20 3.2.2. Signaling Server . . . . . . . . . . . . . . . . . . . 20
3.2.3. SIP-Aware Stateful Firewall . . . . . . . . . . . . . 21 3.2.3. SIP-Aware Stateful Firewall . . . . . . . . . . . . . 21
3.2.4. SIP Transport Protocol . . . . . . . . . . . . . . . . 21 3.2.4. SIP Transport Protocol . . . . . . . . . . . . . . . . 21
3.3. Test Setup Parameters . . . . . . . . . . . . . . . . . . 22 3.3. Test Setup Parameters . . . . . . . . . . . . . . . . . . 22
3.3.1. Session Attempt Rate . . . . . . . . . . . . . . . . . 22 3.3.1. Session Attempt Rate . . . . . . . . . . . . . . . . . 22
3.3.2. IS Media Attempt Rate . . . . . . . . . . . . . . . . 22 3.3.2. IS Media Attempt Rate . . . . . . . . . . . . . . . . 22
3.3.3. Establishment Threshold Time . . . . . . . . . . . . . 23 3.3.3. Establishment Threshold Time . . . . . . . . . . . . . 23
3.3.4. Media Packet Size . . . . . . . . . . . . . . . . . . 24 3.3.4. Session Duration . . . . . . . . . . . . . . . . . . . 24
3.3.5. Media Offered Load . . . . . . . . . . . . . . . . . . 24 3.3.5. Media Packet Size . . . . . . . . . . . . . . . . . . 24
3.3.6. Media Session Hold Time . . . . . . . . . . . . . . . 25 3.3.6. Media Offered Load . . . . . . . . . . . . . . . . . . 25
3.3.7. Loop Detection Option . . . . . . . . . . . . . . . . 25 3.3.7. Media Session Hold Time . . . . . . . . . . . . . . . 25
3.3.8. Forking Option . . . . . . . . . . . . . . . . . . . . 26 3.3.8. Loop Detection Option . . . . . . . . . . . . . . . . 26
3.4. Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . 26 3.3.9. Forking Option . . . . . . . . . . . . . . . . . . . . 26
3.4.1. Registration Rate . . . . . . . . . . . . . . . . . . 26 3.4. Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . 27
3.4.2. Session Establishment Rate . . . . . . . . . . . . . . 27 3.4.1. Registration Rate . . . . . . . . . . . . . . . . . . 27
3.4.3. Session Capacity . . . . . . . . . . . . . . . . . . . 28 3.4.2. Session Establishment Rate . . . . . . . . . . . . . . 28
3.4.4. Session Overload Capacity . . . . . . . . . . . . . . 29 3.4.3. Session Capacity . . . . . . . . . . . . . . . . . . . 29
3.4.5. Session Establishment Performance . . . . . . . . . . 29 3.4.4. Session Overload Capacity . . . . . . . . . . . . . . 30
3.4.6. Session Attempt Delay . . . . . . . . . . . . . . . . 30 3.4.5. Session Establishment Performance . . . . . . . . . . 30
3.4.6. Session Attempt Delay . . . . . . . . . . . . . . . . 31
3.4.7. IM Rate . . . . . . . . . . . . . . . . . . . . . . . 31 3.4.7. IM Rate . . . . . . . . . . . . . . . . . . . . . . . 31
4. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 31 4. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 32
5. Security Considerations . . . . . . . . . . . . . . . . . . . 31 5. Security Considerations . . . . . . . . . . . . . . . . . . . 32
6. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 32 6. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 33
7. References . . . . . . . . . . . . . . . . . . . . . . . . . . 32 7. References . . . . . . . . . . . . . . . . . . . . . . . . . . 33
7.1. Normative References . . . . . . . . . . . . . . . . . . . 32 7.1. Normative References . . . . . . . . . . . . . . . . . . . 33
7.2. Informational References . . . . . . . . . . . . . . . . . 32 7.2. Informational References . . . . . . . . . . . . . . . . . 33
Appendix A. White Box Benchmarking Terminology . . . . . . . . . 33
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 33 Appendix A. White Box Benchmarking Terminology . . . . . . . . . 34
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 34
1. Terminology 1. Terminology
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in BCP 14, RFC2119 document are to be interpreted as described in BCP 14, RFC2119
[RFC2119]. RFC 2119 defines the use of these key words to help make [RFC2119]. RFC 2119 defines the use of these key words to help make
the intent of standards track documents as clear as possible. While the intent of standards track documents as clear as possible. While
this document uses these keywords, this document is not a standards this document uses these keywords, this document is not a standards
track document. The term Throughput is defined in RFC2544 [RFC2544]. track document. The term Throughput is defined in RFC2544 [RFC2544].
skipping to change at page 6, line 26 skipping to change at page 6, line 26
Service Providers are now planning Voice Over IP (VoIP) and Service Providers are now planning Voice Over IP (VoIP) and
Multimedia network deployments using the IETF developed Session Multimedia network deployments using the IETF developed Session
Initiation Protocol (SIP) [RFC3261]. SIP is a signaling protocol Initiation Protocol (SIP) [RFC3261]. SIP is a signaling protocol
originally intended to be used for the dynamic establishment, originally intended to be used for the dynamic establishment,
disconnection and modification of streams of media between end users. disconnection and modification of streams of media between end users.
As it has evolved it has been adopted for use in a growing number of As it has evolved it has been adopted for use in a growing number of
applications and features. Many of these result in the creation of a applications and features. Many of these result in the creation of a
media stream, but some do not. Instead, they create other services media stream, but some do not. Instead, they create other services
tailored to the end-users' immediate needs or preferences. The set tailored to the end-users' immediate needs or preferences. The set
of benchmarking terms provided in this document is intended for use of benchmarking terms provided in this document is intended for use
with any SIP-enabled device performing SIP functions in the core of with any SIP-enabled device performing SIP functions in the interior
the network. The performance of end-user devices is outside the of the network. The performance of end-user devices is outside the
scope of this document. scope of this document.
VoIP with SIP has led to the development of new networking devices VoIP with SIP has led to the development of new networking devices
including SIP Server, Session Border Controllers (SBC), Back-to-back including SIP Server, Session Border Controllers (SBC), Back-to-back
user agents (B2BUA) and SIP-Aware Stateful Firewall. The mix of user agents (B2BUA) and SIP-Aware Stateful Firewall. The mix of
voice and IP functions in these various devices has produced voice and IP functions in these various devices has produced
inconsistencies in vendor reported performance metrics and has caused inconsistencies in vendor reported performance metrics and has caused
confusion in the service provider community. SIP allows a wide range confusion in the service provider community. SIP allows a wide range
of configuration and operational conditions that can influence of configuration and operational conditions that can influence
performance benchmark measurements. When the device under test performance benchmark measurements. When the device under test
skipping to change at page 8, line 21 skipping to change at page 8, line 21
o Different transport mechanisms -- such as UDP, TCP, SCTP, or TLS o Different transport mechanisms -- such as UDP, TCP, SCTP, or TLS
-- may be used; however, the specific transport mechanism MUST be -- may be used; however, the specific transport mechanism MUST be
noted as a condition of the test as the performance of SIP devices noted as a condition of the test as the performance of SIP devices
may vary accordingly. may vary accordingly.
o Looping and forking options are also considered since they impact o Looping and forking options are also considered since they impact
processing at SIP proxies. processing at SIP proxies.
o REGISTER and INVITE requests may be challenged or remain o REGISTER and INVITE requests may be challenged or remain
unchallenged for authentication purpose as this may impact the unchallenged for authentication purpose as this may impact the
performance benchmarks. Any observable performance degradation performance benchmarks. Any observable performance degradation
due to authentication is of interest to the SIP community. due to authentication is of interest to the SIP community.
Whether or not the REGISTER and INVITE requests are challenged is
a condition of test and will be recorded and reported.
o Re-INVITE requests are not considered in scope of this work item. o Re-INVITE requests are not considered in scope of this work item.
o Only session establishment is considered for the performance o Only session establishment is considered for the performance
benchmarks. Session disconnect is not considered in the scope of benchmarks. Session disconnect is not considered in the scope of
this work item. this work item.
o SIP Overload [I-D.ietf-sipping-overload-reqs] is within the scope o SIP Overload [I-D.ietf-sipping-overload-reqs] is within the scope
of this work item, but considerations on how to handle overload of this work item. We test to failure and then can continue to
are deferred to work progressing in the SIPPING working group observe and record the behavior of the system after failures are
[I-D.ietf-sipping-overload-design]. Vendors are, of course, free recorded. The cause of failure is not within the scope of this
to implement their specific overload control behavior as the work. We note the failure and may continue to test until a
expected test outcome if it is different from the IETF different failure or condition is encountered. Considerations on
recommendations. However, such behavior MUST be documented and how to handle overload are deferred to work progressing in the
interpreted appropriately across multiple vendor implementations. SIPPING working group [I-D.ietf-sipping-overload-design]. Vendors
This will make it more meaningful to compare the performance of are, of course, free to implement their specific overload control
different SIP overload implementations. behavior as the expected test outcome if it is different from the
IETF recommendations. However, such behavior MUST be documented
and interpreted appropriately across multiple vendor
implementations. This will make it more meaningful to compare the
performance of different SIP overload implementations.
o IMS-specific scenarios are not considered, but test cases can be o IMS-specific scenarios are not considered, but test cases can be
applied with 3GPP-specific SIP signaling and the P-CSCF as a DUT. applied with 3GPP-specific SIP signaling and the P-CSCF as a DUT.
2.2. Benchmarking Models 2.2. Benchmarking Models
This section shows the five models to be used when benchmarking SIP This section shows the five models to be used when benchmarking SIP
performance of a networking device. Figure 1 shows a configuration performance of a networking device. Figure 1 shows a configuration
in which the Tester acting as the Emulated agents is in loopback in which the Tester acting as the Emulated agents is in loopback
testing itself for the purpose of baselining its performance. testing itself for the purpose of baselining its performance.
skipping to change at page 11, line 18 skipping to change at page 11, line 18
Associated Media is used, it may traverse the DUT/SUT depending upon Associated Media is used, it may traverse the DUT/SUT depending upon
the type of DUT/SUT. The Associated Media is shown in Figure 6 as the type of DUT/SUT. The Associated Media is shown in Figure 6 as
"Media" connected to media ports M1 and M2 on the EA. After the EA "Media" connected to media ports M1 and M2 on the EA. After the EA
sends a BYE, the session disconnects. Performance test cases for sends a BYE, the session disconnects. Performance test cases for
session disconnects are not considered in this work item (the BYE session disconnects are not considered in this work item (the BYE
request is shown for completeness.) request is shown for completeness.)
EA DUT/SUT M1 M2 EA DUT/SUT M1 M2
| | | | | | | |
| INVITE | | | | INVITE | | |
--------+-------------->| | | --------+--------------->| | |
| | | | | | | |
Attempting | | | Attempting | | |
| 200 OK | | | | 200 OK | | |
--------|<--------------| | | --------|<-------------- | | |
| | | | | | | |
| | | | | | | |
| | | Media | | | | Media |
Established | |<=====>| Established | |<=====>|
| | | | | | | |
| BYE | | | | BYE | | |
--------+-------------->| | | --------+--------------> | | |
| | | | | | | |
Disconnecting | | | Disconnecting | | |
| 200 OK | | | | 200 OK | | |
--------|<--------------| | | --------|<-------------- | | |
| | | | | | | |
Figure 6: Basic SIP test topology Figure 6: Basic SIP test topology
3. Term Definitions 3. Term Definitions
3.1. Protocol Components 3.1. Protocol Components
3.1.1. Session 3.1.1. Session
skipping to change at page 15, line 24 skipping to change at page 15, line 24
3.1.3. Media Plane 3.1.3. Media Plane
Definition: Definition:
The data plane in which one or more media streams and their The data plane in which one or more media streams and their
associated media control protocols are exchanged after a media associated media control protocols are exchanged after a media
connection has been created by the exchange of signaling messages connection has been created by the exchange of signaling messages
in the Signaling Plane. in the Signaling Plane.
Discussion: Discussion:
Media may also be known as the "payload" or "bearer channel". The Media may also be known as the "bearer channel". The Media Plane
Media Plane MUST include the media control protocol, if one is MUST include the media control protocol, if one is used, and the
used, and the media stream(s). Examples of media are audio, media stream(s). Examples of media are audio, video, whiteboard,
video, whiteboard, and instant messaging service. The media and instant messaging service. The media stream is described in
stream is described in the SDP of the Signaling Plane. the SDP of the Signaling Plane.
The media for a single Session is represented by session.med. The The media for a single Session is represented by session.med. The
media control protocol is represented by session.medc. media control protocol is represented by session.medc.
Measurement Units: Measurement Units:
N/A. N/A.
Issues: Issues:
None. None.
See Also: See Also:
skipping to change at page 16, line 22 skipping to change at page 16, line 22
N/A. N/A.
Issues: Issues:
None. None.
3.1.5. Overload 3.1.5. Overload
Definition: Definition:
Overload is defined as the state where a SIP server does not have Overload is defined as the state where a SIP server does not have
sufficient resources to process all incoming SIP messages sufficient resources to process all incoming SIP messages
[I-D.ietf-sipping-overload-reqs]. [I-D.ietf-sipping-overload-reqs]. The distinction between an
overload condition and other failure scenarios is outside the
scope of this document which is blackbox testing.
Discussion: Discussion:
Under such conditions, all or a percentage of Session Attempts Under overload conditions, all or a percentage of Session Attempts
will fail due to lack of resources. SIP server resources may will fail due to lack of resources. SIP server resources may
include CPU processing capacity, network bandwidth, input/output include CPU processing capacity, network bandwidth, input/output
queues, or disk resources. Any combination of resources may be queues, or disk resources. Any combination of resources may be
fully utilized when a SIP server (the DUT/SUT) is in the overload fully utilized when a SIP server (the DUT/SUT) is in the overload
condition. For proxy-only type of devices, overload issues will condition. For proxy-only type of devices, overload issues will
be dominated by the number of signaling messages they can handle be dominated by the number of signaling messages they can handle
in a unit time before their throughput starts to drop. in a unit time before their throughput starts to drop.
For UA-type of network devices (e.g., gateways), overload must For UA-type of network devices (e.g., gateways), overload must
necessarily include both the signaling traffic and media streams. necessarily include both the signaling traffic and media streams.
It is expected that the amount of signaling that a UA can handle It is expected that the amount of signaling that a UA can handle
is inversely proportional to the amount of media streams currently is inversely proportional to the amount of media streams currently
handled by that UA. handled by that UA.
Measurement Units: Measurement Units:
N/A. N/A.
Issues: Issues:
The issue of overload in SIP networks is currently a topic of The issue of overload in SIP networks is currently a topic of
discussion in the SIPPING WG. The normal response to an overload discussion in the SIPPING WG. The normal response to an overload
stimulus -- sending a 503 response -- is considered inadequate. stimulus -- sending a 503 response -- is considered inadequate and
The exact method for handling overload is not determined yet and new response codes and behaviors may be specified in the future.
as solutions are engineered, this draft will be updated From the perspective of this document, all these responses will be
accordingly (Note: this draft now has a dependency on the Overload considered to be failures. There is thus no dependency between
working group created by Dispatch. this document and the ongoing work on the treatment of overload
failure.
3.1.6. Session Attempt 3.1.6. Session Attempt
Definition: Definition:
A SIP Session for which the Emulated Agent has sent the SIP INVITE A SIP Session for which the Emulated Agent has sent the SIP INVITE
or SUBSCRIBE NOTIFY and has not yet received a message response or SUBSCRIBE NOTIFY and has not yet received a message response
from the DUT/SUT. from the DUT/SUT.
Discussion: Discussion:
The attempted session may be an IS or an NS. The Session Attempt The attempted session may be an IS or an NS. The Session Attempt
skipping to change at page 17, line 34 skipping to change at page 17, line 35
See Also: See Also:
Session Session
Session Attempt Rate Session Attempt Rate
Invite-initiated Session Invite-initiated Session
Non-Invite initiated Session Non-Invite initiated Session
3.1.7. Established Session 3.1.7. Established Session
Definition: Definition:
A SIP session for which the Emulated Agent acting as the UE/UA has A SIP session for which the Emulated Agent acting as the UE/UA has
received a 200OK message from the DUT/SUT. received a 200 OK message from the DUT/SUT.
Discussion: Discussion:
An Established Session MAY be type INVITE-Session (IS) or Non- An Established Session MAY be type INVITE-Session (IS) or Non-
INVITE Session (NS). INVITE Session (NS).
Measurement Units: Measurement Units:
N/A. N/A.
Issues: Issues:
None. None.
skipping to change at page 18, line 22 skipping to change at page 18, line 25
Discussion: Discussion:
An IS is identified by the Call-ID, To-tag, and From-tag of the An IS is identified by the Call-ID, To-tag, and From-tag of the
SIP message that establishes the session. These three fields are SIP message that establishes the session. These three fields are
used to identify a SIP Dialog (RFC3261 [RFC3261]). An IS may have used to identify a SIP Dialog (RFC3261 [RFC3261]). An IS may have
Associated Media description in the SDP body. An IS may have Associated Media description in the SDP body. An IS may have
multiple Associated Media streams. The inclusion of media is test multiple Associated Media streams. The inclusion of media is test
case dependent. An IS is successfully established if the case dependent. An IS is successfully established if the
following two conditions are met: following two conditions are met:
1. Sess.sig is established by the end of Establishment Threshold 1. Sess.sig is established by the end of Establishment Threshold
Time (c.f. Section 3.3.3), and Time (c.f. Section 3.3.3), and
2. The media session described in the SDP body of the signaling 2. If a media session is described in the SDP body of the
message is present. signaling message, then the media session is established by
the end of Establishment Threshold Time (c.f. Section 3.3.3).
Measurement Units: Measurement Units:
N/A. N/A.
Issues: Issues:
None. None.
See Also: See Also:
Session Session
Non-Invite initiated Session Non-Invite initiated Session
skipping to change at page 19, line 25 skipping to change at page 19, line 25
3.1.10. Session Attempt Failure 3.1.10. Session Attempt Failure
Definition: Definition:
A session attempt that does not result in an Established Session. A session attempt that does not result in an Established Session.
Discussion: Discussion:
The session attempt failure may be indicated by the following The session attempt failure may be indicated by the following
observations at the Emulated Agent: observations at the Emulated Agent:
1. Receipt of a SIP 4xx, 5xx, or 6xx class response to a Session 1. Receipt of a SIP 4xx, 5xx, or 6xx class response to a Session
Attempt. Attempt.
2. The lack of any received SIP response to a Session Attempt. 2. The lack of any received SIP response to a Session Attempt
within the Establishment Threshold Time (c.f. Section 3.3.3).
Measurement Units: Measurement Units:
N/A. N/A.
Issues: Issues:
None. None.
See Also: See Also:
Session Attempt Session Attempt
3.1.11. Standing Sessions 3.1.11. Standing Sessions Count
Definition: Definition:
The number of Sessions currently established on the DUT/SUT at any The number of Sessions currently established on the DUT/SUT at any
instant. instant.
Discussion: Discussion:
The number of Standing Sessions is influenced by the Session The number of Standing Sessions is influenced by the Session
Duration and the Session Attempt Rate. Benchmarks MUST be Duration and the Session Attempt Rate. Benchmarks MUST be
reported with the maximum and average Standing Sessions for the reported with the maximum and average Standing Sessions for the
DUT/SUT. In order to determine the maximum and average Standing DUT/SUT. In order to determine the maximum and average Standing
skipping to change at page 20, line 13 skipping to change at page 20, line 14
measurement period is 1 second. measurement period is 1 second.
Measurement Units: Measurement Units:
Number of sessions Number of sessions
Issues: Issues:
None. None.
See Also: See Also:
Session Duration Session Duration
Session Rate Session Attempt Rate
Session Attempt Rate Session Attempt Rate
3.2. Test Components 3.2. Test Components
3.2.1. Emulated Agent 3.2.1. Emulated Agent
Definition: Definition:
A device in test topology that initiates/responds to SIP messages A device in test topology that initiates/responds to SIP messages
as one or more session endpoints and, wherever applicable, as one or more session endpoints and, wherever applicable,
sources/receives Associated Media for Established Sessions. sources/receives Associated Media for Established Sessions.
skipping to change at page 23, line 10 skipping to change at page 23, line 10
Session Session
Session Attempt Session Attempt
3.3.2. IS Media Attempt Rate 3.3.2. IS Media Attempt Rate
Definition: Definition:
Configuration on the Emulated Agent for number of ISs with Configuration on the Emulated Agent for number of ISs with
Associated Media to be established at the DUT per continuous one- Associated Media to be established at the DUT per continuous one-
second time intervals. second time intervals.
Discussion: Discussion:
The Media Session Attempt Rate can cause variation in performance Note that a Media Session MUST be associated with an IS. In this
benchmark measurements. This is the number of sessions configured document we assume that there is a one to one correspondence
on the Tester. Some attempts may not result in successful between IS session attempts and Media Session attempts. By
sessions established on the DUT. Media Sessions MUST be including this definition we leave open the possibility that there
associated with an IS. By including this definition we leave open may be an IS that does not include a media description. Also note
the possibility that there may be an IS that does not include a that the IS Media Attempt Rate defines the number of media
media description. In this document, however, we will assume that sessions we are trying to create, not the number of media sessions
there is a one to one correspondence between IS session attempts that are actually created. Variations in the Media Session
and Media Session attempts. Attempt Rate might cause variations in performance benchmark
measurements. Some attempts might not result in successful
sessions established on the DUT.
Measurement Units: Measurement Units:
session attempts per second (saps) session attempts per second (saps)
Issues: Issues:
None. None.
See Also: See Also:
IS IS
skipping to change at page 23, line 46 skipping to change at page 24, line 4
Discussion: Discussion:
This time duration is test dependent. This time duration is test dependent.
It is RECOMMENDED that the Establishment Threshold Time value be It is RECOMMENDED that the Establishment Threshold Time value be
set to Timer B (for ISs) or Timer F (for NSs) as specified in RFC set to Timer B (for ISs) or Timer F (for NSs) as specified in RFC
3261, Table 4 [RFC3261]. Following the default value of T1 3261, Table 4 [RFC3261]. Following the default value of T1
(500ms) specified in the table and a constant multiplier of 64 (500ms) specified in the table and a constant multiplier of 64
gives a value of 32 seconds for this timer (i.e., 500ms * 64 = gives a value of 32 seconds for this timer (i.e., 500ms * 64 =
32s). 32s).
Measurement Units: Measurement Units:
seconds seconds
Issues: Issues:
None. None.
See Also: See Also:
session establishment failure session establishment failure
3.3.4. Media Packet Size 3.3.4. Session Duration
Definition:
Configuration of the Emulated Agent that represents the amount of
time that the SIP dialog is intended to exist between the two EAs
associated with the test.
Discussion:
The time at which the BYE is sent will control the Session
Duration
Normally the Session Duration will be the same as the Media
Session Hold Time. However, it is possible that the dialog
established between the two EAs can support different media
sessions at different points in time. Providing both parameters
allows the testing agency to explore this possibility.
Measurement Units:
seconds
Issues:
None.
See Also:
Media Session Hold Time
3.3.5. Media Packet Size
Definition: Definition:
Configuration on the Emulated Agent for a fixed size of packets Configuration on the Emulated Agent for a fixed size of packets
used for media streams. used for media streams.
Discussion: Discussion:
For a single benchmark test, all sessions use the same size packet For a single benchmark test, all sessions use the same size packet
for media streams. The size of packets can cause variation in for media streams. The size of packets can cause variation in
performance benchmark measurements. performance benchmark measurements.
Measurement Units: Measurement Units:
bytes bytes
Issues: Issues:
None. None.
See Also: See Also:
3.3.5. Media Offered Load 3.3.6. Media Offered Load
Definition: Definition:
Configuration of the Emulated Agent for the constant rate of Configuration of the Emulated Agent for the constant rate of
Associated Media traffic offered by the Emulated Agent to the DUT/ Associated Media traffic offered by the Emulated Agent to the DUT/
SUT for one or more Established Sessions of type IS. SUT for one or more Established Sessions of type IS.
Discussion: Discussion:
The Media Offered Load to be used for a test MUST be reported with The Media Offered Load to be used for a test MUST be reported with
three components: three components:
1. per Associated Media stream; 1. per Associated Media stream;
skipping to change at page 25, line 15 skipping to change at page 25, line 42
packets per second (pps) packets per second (pps)
Issues: Issues:
None. None.
See Also: See Also:
Established Session Established Session
Invite Initiated Session Invite Initiated Session
Associated Media Associated Media
3.3.6. Media Session Hold Time 3.3.7. Media Session Hold Time
Definition: Definition:
Parameter configured at the Emulated Agent, that represents the Parameter configured at the Emulated Agent, that represents the
amount of time that the Associated Media for an Established amount of time that the Associated Media for an Established
Session of type IS will last . Session of type IS will last.
Discussion: Discussion:
The Associated Media streams may be bi-directional or uni- The Associated Media streams may be bi-directional or uni-
directional as indicated in the test methodology. directional as indicated in the test methodology.
Normally the Media Session Hold Time will be the same as the
Session Duration. However, it is possible that the dialog
established between the two EAs can support different media
sessions at different points in time. Providing both parameters
allows the testing agency to explore this possibility.
Measurement Units: Measurement Units:
seconds seconds
Issues: Issues:
None. None.
See Also: See Also:
Associated Media Associated Media
Established Session Established Session
Invite-initiated Session (IS) Invite-initiated Session (IS)
3.3.7. Loop Detection Option 3.3.8. Loop Detection Option
Definition: Definition:
An option that causes a Proxy to check for loops in the routing of An option that causes a Proxy to check for loops in the routing of
a SIP request before forwarding the request. a SIP request before forwarding the request.
Discussion: Discussion:
This is an optional process that a SIP proxy may employ; the This is an optional process that a SIP proxy may employ; the
process is described under Proxy Behavior in RFC 3261 in Section process is described under Proxy Behavior in RFC 3261 in Section
16.3 Request Validation and that section also contains suggestions 16.3 Request Validation and that section also contains suggestions
as to how the option could be implemented. Any procedure to as to how the option could be implemented. Any procedure to
skipping to change at page 26, line 13 skipping to change at page 26, line 47
performance of a proxy. performance of a proxy.
Measurement Units: Measurement Units:
NA NA
Issues: Issues:
None. None.
See Also: See Also:
3.3.8. Forking Option 3.3.9. Forking Option
Definition: Definition:
An option that enables a Proxy to fork requests to more than one An option that enables a Proxy to fork requests to more than one
destination. destination.
Discussion: Discussion:
This is an process that a SIP proxy may employ to find the UAS. This is an process that a SIP proxy may employ to find the UAS.
The option is described under Proxy Behavior in RFC 3261 in The option is described under Proxy Behavior in RFC 3261 in
Section 16.1. A proxy that uses forking must maintain state Section 16.1. A proxy that uses forking must maintain state
information and this will use processor cycles and memory. Thus information and this will use processor cycles and memory. Thus
the use of this option could impact the performance of a proxy and the use of this option could impact the performance of a proxy and
different implementations could produce different impacts. different implementations could produce different impacts.
SIP supports serial or parallel forking. When performing a test, SIP supports serial or parallel forking. When performing a test,
the type of forking mode must be indicated. the type of forking mode MUST be indicated.
Measurement Units: Measurement Units:
The number of endpoints that will receive the forked invitation. The number of endpoints that will receive the forked invitation.
A value of 1 indicates that the request is destined to only one A value of 1 indicates that the request is destined to only one
endpoint, a value of 2 indicates that the request is forked to two endpoint, a value of 2 indicates that the request is forked to two
endpoints, and so on. This is an integer value ranging between 1 endpoints, and so on. This is an integer value ranging between 1
and N inclusive, where N is the maximum number of endpoints to and N inclusive, where N is the maximum number of endpoints to
which the invitation is sent. which the invitation is sent.
Type of forking used, namely parallel or serial. Type of forking used, namely parallel or serial.
skipping to change at page 27, line 27 skipping to change at page 28, line 16
registrations per second (rps) registrations per second (rps)
Issues: Issues:
None. None.
See Also: See Also:
3.4.2. Session Establishment Rate 3.4.2. Session Establishment Rate
Definition: Definition:
The maximum average rate at which the DUT/SUT can successfully The average maximum rate at which the DUT/SUT can successfully
establish sessions. establish sessions.
Discussion: Discussion:
This metric is an average of maximas. Each maximum is measured in This metric is an average of maxima. Each maximum is measured in
a separate sample. The Session Establishment Rate is the average a separate sample. The Session Establishment Rate is the average
of the maximas established in each individual sample. In each of the maximas established in each individual sample. In each
sample, the maximum in question is the number of sessions sample, the maximum in question is the number of sessions
successfully established in continuous one-second intervals with successfully established in continuous one-second intervals with
prior sessions remaining active. This maximum is designated in prior sessions remaining active. This maximum is designated in
the equation below as "rate in sample i". The session the equation below as "rate in sample i". The session
establishment rate is calculated using the following equation (n = establishment rate is calculated using the following equation (n =
number of samples): number of samples):
n n
skipping to change at page 28, line 8 skipping to change at page 28, line 42
\ rate at sample i \ rate at sample i
/ /
-- --
i = 1 i = 1
--------------------- ---------------------
(n) (n)
In each sample, the maximum is obtained by testing to failure. In each sample, the maximum is obtained by testing to failure.
With zero failure, 100% of the sessions introduced by the Emulated With zero failure, 100% of the sessions introduced by the Emulated
Agent are successfully established. The maximum value is obtained Agent are successfully established. The maximum value is obtained
by testing to failure. This means that the session rate by testing to failure. This means that the Session Attempt Rate
provisioned on the EA is raised progressively until a Session provisioned on the EA is raised progressively until a Session
Attempt Failure is observed. The maximum rate is the rate Attempt Failure is observed. The maximum rate is the rate
acheived in the interval prior to the interval in which the acheived in the interval prior to the interval in which the
failure is observed. Sessions may be IS or NS or a a mix of both failure is observed. Sessions may be IS or NS or a a mix of both
and will be defined in the particular test. and will be defined in the particular test.
Measurement Units: Measurement Units:
sessions per second (sps) sessions per second (sps)
Issues: Issues:
None. None.
See Also: See Also:
Invite-initiated Sessions Invite-initiated Sessions
Non-INVITE initiated Sessions Non-INVITE initiated Sessions
Session Attempt Rate Session Attempt Rate
3.4.3. Session Capacity 3.4.3. Session Capacity
Definition: Definition:
The maximum number of Established Sessions that can exist The maximum value of Standing Sessions Count achieved by the DUT/
simultaneously on the DUT/SUT until Session Attempt Failure SUT during the process of steadily increasing the number of
occurs. Session Attempts per unit time, before the first Session Attempt
Failure occurs.
Discussion: Discussion:
When benchmarking Session Capacity it is assumed that sessions are When benchmarking Session Capacity for sessions with media it is
permanently established (i.e., they remain active for the duration required that these sessions be permanently established (i.e.,
of the test.) In order to make this measurement, the Session they remain active for the duration of the test.) This can be
Duration MUST be set to infinity so that sessions remain achieved by causing the EA not to send a BYE for the duration of
established for the duration of the test. The Session Capacity the testing. In the signaling plane, this requirement means that
must be reported with the Session Attempt Rate used to reach the the dialog lasts as long as the test lasts. In order to test
maximum. Since Session Attempt Rate is a zero-loss measurement, Session Capacity for sessions with media, the Media Session Hold
there must be zero failures to achieve the Session Capacity. The Time MUST be set to infinity so that sessions remain established
maximum is indicated at the Emulated Agent by arrival of a SIP for the duration of the test. If the DUT/SUT is dialog-stateful,
4xx, 5xx, or 6xx response from the DUT/SUT. Sessions may be IS or then we expect its performance will be impacted by setting Media
NS. Session Hold Time to infinity, since the DUT/SUT will need to
allocate resources to process and store the state information.
The Session Capacity must be reported with the Session Attempt
Rate used to reach the maximum. Since Session Attempt Rate is a
zero-loss measurement, there must be zero failures to achieve the
Session Capacity. The maximum is indicated at the Emulated Agent
by arrival of a SIP 4xx, 5xx, or 6xx response from the DUT/SUT.
Sessions may be IS or NS.
Measurement Units: Measurement Units:
sessions sessions
Issues: Issues:
None. None.
See Also: See Also:
Established Session Established Session
Session Attempt Rate Session Attempt Rate
skipping to change at page 32, line 4 skipping to change at page 32, line 42
This document requires no IANA considerations. This document requires no IANA considerations.
5. Security Considerations 5. Security Considerations
Documents of this type do not directly affect the security of Documents of this type do not directly affect the security of
Internet or corporate networks as long as benchmarking is not Internet or corporate networks as long as benchmarking is not
performed on devices or systems connected to production networks. performed on devices or systems connected to production networks.
Security threats and how to counter these in SIP and the media layer Security threats and how to counter these in SIP and the media layer
is discussed in RFC3261 [RFC3261], RFC 3550 [RFC3550], RFC3711 is discussed in RFC3261 [RFC3261], RFC 3550 [RFC3550], RFC3711
[RFC3711] and various other drafts. This document attempts to [RFC3711] and various other drafts. This document attempts to
formalize a set of common terminology for benchmarking SIP networks. formalize a set of common terminology for benchmarking SIP networks.
Packets with unintended and/or unauthorized DSCP or IP precedence
values may present security issues. Determining the security
consequences of such packets is out of scope for this document.
6. Acknowledgments 6. Acknowledgments
The authors would like to thank Keith Drage, Cullen Jennings, Daryl The authors would like to thank Keith Drage, Cullen Jennings, Daryl
Malas, Al Morton, and Henning Schulzrinne for invaluable Malas, Al Morton, and Henning Schulzrinne for invaluable
contributions to this document. contributions to this document.
7. References 7. References
7.1. Normative References 7.1. Normative References
skipping to change at page 32, line 32 skipping to change at page 33, line 29
Network Interconnect Devices", RFC 2544, March 1999. Network Interconnect Devices", RFC 2544, March 1999.
[RFC3261] Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston, [RFC3261] Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston,
A., Peterson, J., Sparks, R., Handley, M., and E. A., Peterson, J., Sparks, R., Handley, M., and E.
Schooler, "SIP: Session Initiation Protocol", RFC 3261, Schooler, "SIP: Session Initiation Protocol", RFC 3261,
June 2002. June 2002.
[I-D.ietf-bmwg-sip-bench-meth] [I-D.ietf-bmwg-sip-bench-meth]
Poretsky, S., Gurbani, V., and C. Davids, "Methodology for Poretsky, S., Gurbani, V., and C. Davids, "Methodology for
Benchmarking SIP Networking Devices", Benchmarking SIP Networking Devices",
draft-ietf-bmwg-sip-bench-meth-01 (work in progress), draft-ietf-bmwg-sip-bench-meth-02 (work in progress),
January 2010. July 2010.
7.2. Informational References 7.2. Informational References
[RFC2285] Mandeville, R., "Benchmarking Terminology for LAN [RFC2285] Mandeville, R., "Benchmarking Terminology for LAN
Switching Devices", RFC 2285, February 1998. Switching Devices", RFC 2285, February 1998.
[RFC1242] Bradner, S., "Benchmarking terminology for network [RFC1242] Bradner, S., "Benchmarking terminology for network
interconnection devices", RFC 1242, July 1991. interconnection devices", RFC 1242, July 1991.
[RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V. [RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V.
skipping to change at page 34, line 32 skipping to change at page 35, line 20
Phone: +1 630 224 0216 Phone: +1 630 224 0216
Email: vkg@alcatel-lucent.com Email: vkg@alcatel-lucent.com
Carol Davids Carol Davids
Illinois Institute of Technology Illinois Institute of Technology
201 East Loop Road 201 East Loop Road
Wheaton, IL 60187 Wheaton, IL 60187
USA USA
Phone: +1 630 682 6024
Email: davids@iit.edu Email: davids@iit.edu
 End of changes. 45 change blocks. 
98 lines changed or deleted 154 lines changed or added

This html diff was produced by rfcdiff 1.38. The latest version is available from http://tools.ietf.org/tools/rfcdiff/