draft-ietf-bmwg-sip-bench-term-09.txt   draft-ietf-bmwg-sip-bench-term-10.txt 
Benchmarking Methodology Working Group C. Davids Benchmarking Methodology Working Group C. Davids
Internet-Draft Illinois Institute of Technology Internet-Draft Illinois Institute of Technology
Intended status: Informational V. Gurbani Intended status: Informational V. Gurbani
Expires: August 18, 2014 Bell Laboratories, Expires: November 29, 2014 Bell Laboratories,
Alcatel-Lucent Alcatel-Lucent
S. Poretsky S. Poretsky
Allot Communications Allot Communications
February 14, 2014 May 28, 2014
Terminology for Benchmarking Session Initiation Protocol (SIP) Devices: Terminology for Benchmarking Session Initiation Protocol (SIP) Devices:
Basic session setup and registration Basic session setup and registration
draft-ietf-bmwg-sip-bench-term-09 draft-ietf-bmwg-sip-bench-term-10
Abstract Abstract
This document provides a terminology for benchmarking the Session This document provides a terminology for benchmarking the Session
Initiation Protocol (SIP) performance of devices. Methodology Initiation Protocol (SIP) performance of devices. Methodology
related to benchmarking SIP devices is described in the companion related to benchmarking SIP devices is described in the companion
methodology document. Using these two documents, benchmarks can be methodology document. Using these two documents, benchmarks can be
obtained and compared for different types of devices such as SIP obtained and compared for different types of devices such as SIP
Proxy Servers, Registrars and Session Border Controllers. The term Proxy Servers, Registrars and Session Border Controllers. The term
"performance" in this context means the capacity of the device-under- "performance" in this context means the capacity of the device-under-
skipping to change at page 2, line 4 skipping to change at page 2, line 4
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/. Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on August 18, 2014. This Internet-Draft will expire on November 29, 2014.
Copyright Notice Copyright Notice
Copyright (c) 2014 IETF Trust and the persons identified as the Copyright (c) 2014 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
skipping to change at page 5, line 29 skipping to change at page 5, line 29
of end-user devices is outside the scope of this document. of end-user devices is outside the scope of this document.
A number of networking devices have been developed to support SIP- A number of networking devices have been developed to support SIP-
based VoIP services. These include SIP Servers, Session Border based VoIP services. These include SIP Servers, Session Border
Controllers (SBC) and Back-to-back User Agents (B2BUA). These Controllers (SBC) and Back-to-back User Agents (B2BUA). These
devices contain a mix of voice and IP functions whose performance may devices contain a mix of voice and IP functions whose performance may
be reported using metrics defined by the equipment manufacturer or be reported using metrics defined by the equipment manufacturer or
vendor. The Service Provider or IT Organization seeking to compare vendor. The Service Provider or IT Organization seeking to compare
the performance of such devices will not be able to do so using these the performance of such devices will not be able to do so using these
vendor-specific metrics, whose conditions of test and algorithms for vendor-specific metrics, whose conditions of test and algorithms for
collection are often unspecified. SIP functional elements and the collection are often unspecified.
devices that include them can be configured many different ways and
can be organized into various topologies. These configuration and SIP functional elements and the devices that include them can be
topological choices impact the value of any chosen signaling configured many different ways and can be organized into various
benchmark. Unless these conditions-of-test are defined, a true topologies. These configuration and topological choices impact the
comparison of performance metrics across multiple vendor value of any chosen signaling benchmark. Unless these conditions-of-
implementations will not be possible. Some SIP-enabled devices test are defined, a true comparison of performance metrics across
terminate or relay media as well as signaling. The processing of multiple vendor implementations will not be possible.
media by the device impacts the signaling performance. As a result,
the conditions-of-test must include information as to whether or not Some SIP-enabled devices terminate or relay media as well as
the device under test processes media and if the device does process signaling. The processing of media by the device impacts the
media, a description of the media handled and the manner in which it signaling performance. As a result, the conditions-of-test must
is handled. This document and its companion methodology document include information as to whether or not the device under test
[I-D.ietf-bmwg-sip-bench-meth] provide a set of black-box benchmarks processes media. If the device processes media during the test, a
for describing and comparing the performance of devices that description of the media must be provided. This document and its
incorporate the SIP User Agent Client and Server functions and that companion methodology document [I-D.ietf-bmwg-sip-bench-meth] provide
operate in the network's core. a set of black-box benchmarks for describing and comparing the
performance of devices that incorporate the SIP User Agent Client and
Server functions and that operate in the network's core.
The definition of SIP performance benchmarks necessarily includes The definition of SIP performance benchmarks necessarily includes
definitions of Test Setup Parameters and a test methodology. These definitions of Test Setup Parameters and a test methodology. These
enable the Tester to perform benchmarking tests on different devices enable the Tester to perform benchmarking tests on different devices
and to achieve comparable results. This document provides a common and to achieve comparable results. This document provides a common
set of definitions for Test Components, Test Setup Parameters, and set of definitions for Test Components, Test Setup Parameters, and
Benchmarks. All the benchmarks defined are black-box measurements of Benchmarks. All the benchmarks defined are black-box measurements of
the SIP signaling plane. The Test Setup Parameters and Benchmarks the SIP signaling plane. The Test Setup Parameters and Benchmarks
defined in this document are intended for use with the companion defined in this document are intended for use with the companion
Methodology document. Methodology document.
2.1. Scope 2.1. Scope
The scope of this work item is summarized as follows: The scope of this document is summarized as follows:
o This terminology document describes SIP signaling performance o This terminology document describes SIP signaling performance
benchmarks for black-box measurements of SIP networking devices. benchmarks for black-box measurements of SIP networking devices.
Stress and debug scenarios are not addressed in this work item. Stress and debug scenarios are not addressed in this document.
o The DUT must be an RFC 3261 capable network equipment. This may o The DUT must be RFC 3261 capable network equipment. This may be a
be a Registrar, Redirect Server, Stateless Proxy or Stateful Registrar, Redirect Server, Stateless Proxy or Stateful Proxy. A
Proxy. A DUT MAY also include a B2BUA, SBC functionality. DUT MAY also include a B2BUA, SBC functionality.
o The DUT MUST NOT be end user equipment, such as personal digital o The DUT MUST NOT be end user equipment, such as personal digital
assistant, a computer-based client, or a user terminal. assistant, a computer-based client, or a user terminal.
o The Tester acts as multiple "Emulated Agents" (EA) that initiate o The Tester acts as multiple "Emulated Agents" (EA) that initiate
(or respond to) SIP messages as session endpoints and source (or (or respond to) SIP messages as session endpoints and source (or
receive) associated media for established connections. receive) associated media for established connections.
o SIP Signaling in presence of Media o SIP Signaling in presence of media
* The media performance is not benchmarked in this work item. * The media performance is not benchmarked.
* Some tests require media, but the use of media is limited to * Some tests require media, but the use of media is limited to
observing the performance of SIP signaling. Tests that require observing the performance of SIP signaling. Tests that require
media will annotate the media characteristics as a condition of media will annotate the media characteristics as a condition of
test. test.
* The type of DUT dictates whether the associated media streams * The type of DUT dictates whether the associated media streams
traverse the DUT. Both scenarios are within the scope of this traverse the DUT. Both scenarios are within the scope of this
work item. document.
* SIP is frequently used to create media streams; the signaling * SIP is frequently used to create media streams; the signaling
plane and media plane are treated as orthogonal to each other plane and media plane are treated as orthogonal to each other
in this document. While many devices support the creation of in this document. While many devices support the creation of
media streams, benchmarks that measure the performance of these media streams, benchmarks that measure the performance of these
streams are outside the scope of this document and its streams are outside the scope of this document and its
companion methodology document [I-D.ietf-bmwg-sip-bench-meth]. companion methodology document [I-D.ietf-bmwg-sip-bench-meth].
Tests may be performed with or without the creation of media Tests may be performed with or without the creation of media
streams. The presence or absence of media streams MUST be streams. The presence or absence of media streams MUST be
noted as a condition of the test as the performance of SIP noted as a condition of the test as the performance of SIP
devices may vary accordingly. Even if the media is used during devices may vary accordingly. Even if the media is used during
benchmarking, only the SIP performance will be benchmarked, not benchmarking, only the SIP performance will be benchmarked, not
the media performance or quality. the media performance or quality.
o Both INVITE and non-INVITE scenarios (registrations) are addressed o Both INVITE and non-INVITE scenarios (registrations) are addressed
in this document. However, benchmarking SIP presence or in this document. However, benchmarking SIP presence or
subscribe-notify extensions is not a part of this work item. subscribe-notify extensions is not a part of this document.
o Different transport -- such as UDP, TCP, SCTP, or TLS -- may be o Different transport -- such as UDP, TCP, SCTP, or TLS -- may be
used. The specific transport mechanism MUST be noted as a used. The specific transport mechanism MUST be noted as a
condition of the test as the performance of SIP devices may vary condition of the test as the performance of SIP devices may vary
accordingly. accordingly.
o REGISTER and INVITE requests may be challenged or remain o REGISTER and INVITE requests may be challenged or remain
unchallenged for authentication purpose. Whether or not the unchallenged for authentication purpose. Whether or not the
REGISTER and INVITE requests are challenged is a condition of test REGISTER and INVITE requests are challenged is a condition of test
which will be recorded along with other such parameters which may which will be recorded along with other such parameters which may
impact the SIP performance of the device or system under test. impact the SIP performance of the device or system under test.
o Re-INVITE requests are not considered in scope of this work item o Re-INVITE requests are not considered in scope of this document
since the benchmarks for INVITEs are based on the dialog created since the benchmarks for INVITEs are based on the dialog created
by the INVITE and not on the transactions that take place within by the INVITE and not on the transactions that take place within
that dialog. that dialog.
o Only session establishment is considered for the performance o Only session establishment is considered for the performance
benchmarks. Session disconnect is not considered in the scope of benchmarks. Session disconnect is not considered in the scope of
this work item. This is because our goal is to determine the this document. This is because our goal is to determine the
maximum capacity of the device or system under test, that is the maximum capacity of the device or system under test, that is the
number of simultaneous SIP sessions that the device or system can number of simultaneous SIP sessions that the device or system can
support. It is true that there are BYE requests being created support. It is true that there are BYE requests being created
during the test process. These transactions do contribute to the during the test process. These transactions do contribute to the
load on the device or system under test and thus are accounted for load on the device or system under test and thus are accounted for
in the metric we derive. We do not seek a separate metric for the in the metric we derive. We do not seek a separate metric for the
number of BYE transactions a device or system can support. number of BYE transactions a device or system can support.
o IMS-specific scenarios are not considered, but test cases can be o IMS-specific scenarios are not considered, but test cases can be
applied with 3GPP-specific SIP signaling and the P-CSCF as a DUT. applied with 3GPP-specific SIP signaling and the P-CSCF as a DUT.
skipping to change at page 7, line 42 skipping to change at page 7, line 42
Definition: Definition:
The combination of signaling and media messages and processes that The combination of signaling and media messages and processes that
support a SIP-based service. support a SIP-based service.
Discussion: Discussion:
SIP messages are used to create and manage services for end users. SIP messages are used to create and manage services for end users.
Often, these services include the creation of media streams that Often, these services include the creation of media streams that
are defined in the SDP body of a SIP message and carried in RTP are defined in the SDP body of a SIP message and carried in RTP
protocol data units. However, SIP messages can also be used to protocol data units. However, SIP messages can also be used to
create Instant Message services and subscription services, and create instant message services and subscription services, and
such services are not associated with media streams. SIP reserves such services are not associated with media streams. SIP reserves
the term "session" to describe services that are analogous to the term "session" to describe services that are analogous to
telephone calls on a circuit switched network. SIP reserves the telephone calls on a circuit switched network. SIP reserves the
term "dialog" to refer to a signaling-only relationship between term "dialog" to refer to a signaling-only relationship between
User Agent peers. SIP reserves the term "transaction" to refer to User Agent peers. SIP reserves the term "transaction" to refer to
the brief communication between a client and a server that lasts the brief communication between a client and a server that lasts
only until the final response to the SIP request. None of these only until the final response to the SIP request. None of these
terms describes the entity whose performance we want to benchmark. terms describes the entity whose performance we want to benchmark.
For example, the MESSAGE request does not create a dialog and can For example, the MESSAGE request does not create a dialog and can
be sent either within or outside of a dialog. It is not be sent either within or outside of a dialog. It is not
skipping to change at page 14, line 49 skipping to change at page 14, line 49
Associated Media Associated Media
3.1.9. Non-INVITE-Initiated Session (NS) 3.1.9. Non-INVITE-Initiated Session (NS)
Definition: Definition:
A session that is created by an exchange of SIP messages in the A session that is created by an exchange of SIP messages in the
Signaling Plane the first of which is not a SIP INVITE message. Signaling Plane the first of which is not a SIP INVITE message.
Discussion: Discussion:
An NS is successfully established if the Session Attempt via a An NS is successfully established if the Session Attempt via a
non- INVITE request results in the EA receiving a 2xx reply before non-INVITE request results in the EA receiving a 2xx reply before
the expiration of the Establishment Threshold timer (c.f., the expiration of the Establishment Threshold timer (c.f.,
Section 3.3.2). For the purpose of this document, a NS is a Section 3.3.2). For the purpose of this document, a NS is a
session created only by the REGISTER request and no other request. session created only by the REGISTER request and no other request.
Measurement Units: Measurement Units:
N/A. N/A.
Issues: Issues:
None. None.
skipping to change at page 17, line 23 skipping to change at page 17, line 23
Issues: Issues:
None. None.
See Also: See Also:
3.3. Test Setup Parameters 3.3. Test Setup Parameters
3.3.1. Session Attempt Rate 3.3.1. Session Attempt Rate
Definition: Definition:
Configuration of the EA for the number of sessions per second that Configuration of the EA for the number of sessions per second
the EA attempts to establish using the services of the DUT. (sps) that the EA attempts to establish using the services of the
DUT.
Discussion: Discussion:
The Session Attempt Rate is the number of sessions per second that The Session Attempt Rate is the number of sessions per second that
the EA sends toward the DUT. Some of the sessions attempted may the EA sends toward the DUT. Some of the sessions attempted may
not result in a session being established. A session in this case not result in a session being established. A session in this case
may be either an IS or an NS. may be either an IS or an NS.
Measurement Units: Measurement Units:
Session attempts per second Session attempts per second
skipping to change at page 19, line 46 skipping to change at page 19, line 46
3.4. Benchmarks 3.4. Benchmarks
3.4.1. Session Establishment Rate 3.4.1. Session Establishment Rate
Definition: Definition:
The maximum value of the Session Attempt Rate that the DUT can The maximum value of the Session Attempt Rate that the DUT can
handle for an extended, pre-defined, period with zero failures. handle for an extended, pre-defined, period with zero failures.
Discussion: Discussion:
This benchmark is obtained with zero failure in which 100% of the This benchmark is obtained with zero failure. The session attempt
sessions attempted by the Emulated Agent are successfully rate provisioned on the EA is raised and lowered as described in
completed by the DUT. The session attempt rate provisioned on the the algorithm in the accompanying methodology document
EA is raised and lowered as described in the algorithm in the [I-D.ietf-bmwg-sip-bench-meth], until a traffic load over the
accompanying methodology document, until a traffic load at the period of time necessary to attempt N sessions completes without
given attempt rate over the sustained period of time identified by failure, where N is a parameter specified in the algorithm and
T in the algorithm completes without any failed session attempts. recorded in the Test Setup Report. Sessions may be IS or NS or a
Sessions may be IS or NS or a mix of both and will be defined in mix of both and will be defined in the particular test.
the particular test.
Measurement Units: Measurement Units:
sessions per second (sps) sessions per second (sps)
Issues: Issues:
None. None.
See Also: See Also:
Invite-Initiated Sessions Invite-Initiated Sessions
Non-Invite-Initiated Sessions Non-Invite-Initiated Sessions
Session Attempt Rate Session Attempt Rate
3.4.2. Registration Rate 3.4.2. Registration Rate
Definition: Definition:
The maximum value of the Registration Attempt Rate that the DUT The maximum value of the Registration Attempt Rate that the DUT
can handle for an extended, pre-defined, period with zero can handle for an extended, pre-defined, period with zero
failures. failures.
Discussion: Discussion:
This benchmark is obtained with zero failures in which 100% of the This benchmark is obtained with zero failures. The registration
registrations attempted by the EA are successfully completed by rate provisioned on the Emulated Agent is raised and lowered as
the DUT. The registration rate provisioned on the Emulated Agent described in the algorithm in the companion methodology draft
is raised and lowered as described in the algorithm in the [I-D.ietf-bmwg-sip-bench-meth], until a traffic load consisting of
companion methodology draft [I-D.ietf-bmwg-sip-bench-meth] until a registration attempts at the given attempt rate over the period of
traffic load consisting of registration attempts at the given time necessary to attempt N registrations completes without
attempt rate over the period of time necessary to attempt N failure, where N is a parameter specified in the algorithm and
registrations completes without failure, where N is a parameter recorded in the Test Setup Report.
specified in the algorithm and recorded in the Test Setup Report.
This benchmark is described separately from the Session This benchmark is described separately from the Session
Establishment Rate (Section 3.4.1), although it could be Establishment Rate (Section 3.4.1), although it could be
considered a special case of that benchmark, since a REGISTER considered a special case of that benchmark, since a REGISTER
request is a request for a Non-Invite-Initiated session. It is request is a request for a Non-Invite-Initiated session. It is
defined separately because it is a very important benchmark for defined separately because it is a very important benchmark for
most SIP installations. An example demonstrating its use is an most SIP installations. An example demonstrating its use is an
avalanche restart, where hundreds of thousands of end points avalanche restart, where hundreds of thousands of end points
register simultaneously following a power outage. In such a case, register simultaneously following a power outage. In such a case,
an authoritative measurement of the capacity of the device to an authoritative measurement of the capacity of the device to
register endpoints is useful to the network designer. Finally, in register endpoints is useful to the network designer.
certain controlled networks, there appears to be a difference in Additionally, in certain controlled networks, there appears to be
the registration rate of new endpoints registering versus existing a difference between the registration rate of new endpoints and
endpoints refreshing their registrations. This benchmark can the registering rate of existing endpoints (register refreshes).
capture these differences as well. This benchmark can capture these differences as well.
Measurement Units: Measurement Units:
registrations per second (rps) registrations per second (rps)
Issues: Issues:
None. None.
See Also: See Also:
3.4.3. Registration Attempt Rate 3.4.3. Registration Attempt Rate
skipping to change at page 22, line 43 skipping to change at page 22, line 43
[RFC4320] Sparks, R., "Actions Addressing Identified Issues with the [RFC4320] Sparks, R., "Actions Addressing Identified Issues with the
Session Initiation Protocol's (SIP) Non-INVITE Session Initiation Protocol's (SIP) Non-INVITE
Transaction", RFC 4320, January 2006. Transaction", RFC 4320, January 2006.
[RFC6026] Sparks, R. and T. Zourzouvillys, "Correct Transaction [RFC6026] Sparks, R. and T. Zourzouvillys, "Correct Transaction
Handling for 2xx Responses to Session Initiation Protocol Handling for 2xx Responses to Session Initiation Protocol
(SIP) INVITE Requests", RFC 6026, September 2010. (SIP) INVITE Requests", RFC 6026, September 2010.
[I-D.ietf-bmwg-sip-bench-meth] [I-D.ietf-bmwg-sip-bench-meth]
Davids, C., Gurbani, V., and S. Poretsky, "Methodology for Davids, C., Gurbani, V., and S. Poretsky, "SIP Performance
Benchmarking SIP Networking Devices", Benchmarking Methodology",
draft-ietf-bmwg-sip-bench-meth-08 (work in progress), draft-ietf-bmwg-sip-bench-meth-10 (work in progress),
January 2013. May 2014.
7.2. Informational References 7.2. Informational References
[RFC2285] Mandeville, R., "Benchmarking Terminology for LAN [RFC2285] Mandeville, R., "Benchmarking Terminology for LAN
Switching Devices", RFC 2285, February 1998. Switching Devices", RFC 2285, February 1998.
[RFC1242] Bradner, S., "Benchmarking terminology for network [RFC1242] Bradner, S., "Benchmarking terminology for network
interconnection devices", RFC 1242, July 1991. interconnection devices", RFC 1242, July 1991.
[RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V. [RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V.
 End of changes. 19 change blocks. 
63 lines changed or deleted 64 lines changed or added

This html diff was produced by rfcdiff 1.41. The latest version is available from http://tools.ietf.org/tools/rfcdiff/