draft-ietf-bmwg-sip-bench-term-03.txt   draft-ietf-bmwg-sip-bench-term-04.txt 
Benchmarking Methodology Working C. Davids Benchmarking Methodology Working C. Davids
Group Illinois Institute of Technology Group Illinois Institute of Technology
Internet-Draft V. Gurbani Internet-Draft V. Gurbani
Expires: September 15, 2011 Bell Laboratories, Alcatel-Lucent Expires: September 13, 2012 Bell Laboratories, Alcatel-Lucent
S. Poretsky S. Poretsky
Allot Communications Allot Communications
March 14, 2011 March 12, 2012
Terminology for Benchmarking Session Initiation Protocol (SIP) Terminology for Benchmarking Session Initiation Protocol (SIP)
Networking Devices Networking Devices
draft-ietf-bmwg-sip-bench-term-03 draft-ietf-bmwg-sip-bench-term-04
Abstract Abstract
This document provides a terminology for benchmarking SIP performance This document provides a terminology for benchmarking SIP performance
in networking devices. Terms are included for test components, test in networking devices. Terms are included for test components, test
setup parameters, and performance benchmark metrics for black-box setup parameters, and performance benchmark metrics for black-box
benchmarking of SIP networking devices. The performance benchmark benchmarking of SIP networking devices. The performance benchmark
metrics are obtained for the SIP control plane and media plane. The metrics are obtained for the SIP control plane and media plane. The
terms are intended for use in a companion methodology document for terms are intended for use in a companion methodology document for
complete performance characterization of a device in a variety of complete performance characterization of a device in a variety of
skipping to change at page 1, line 49 skipping to change at page 1, line 49
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/. Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on September 15, 2011. This Internet-Draft will expire on September 13, 2012.
Copyright Notice Copyright Notice
Copyright (c) 2011 IETF Trust and the persons identified as the Copyright (c) 2012 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License. described in the Simplified BSD License.
Table of Contents Table of Contents
1. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4 1. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4
2. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 5 2. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.1. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.1. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.2. Benchmarking Models . . . . . . . . . . . . . . . . . . . 7 2.2. Benchmarking Models . . . . . . . . . . . . . . . . . . . 7
3. Term Definitions . . . . . . . . . . . . . . . . . . . . . . . 12 3. Term Definitions . . . . . . . . . . . . . . . . . . . . . . . 13
3.1. Protocol Components . . . . . . . . . . . . . . . . . . . 12 3.1. Protocol Components . . . . . . . . . . . . . . . . . . . 13
3.1.1. Session . . . . . . . . . . . . . . . . . . . . . . . 13 3.1.1. Session . . . . . . . . . . . . . . . . . . . . . . . 13
3.1.2. Signaling Plane . . . . . . . . . . . . . . . . . . . 15 3.1.2. Signaling Plane . . . . . . . . . . . . . . . . . . . 16
3.1.3. Media Plane . . . . . . . . . . . . . . . . . . . . . 16 3.1.3. Media Plane . . . . . . . . . . . . . . . . . . . . . 16
3.1.4. Associated Media . . . . . . . . . . . . . . . . . . . 16 3.1.4. Associated Media . . . . . . . . . . . . . . . . . . . 17
3.1.5. Overload . . . . . . . . . . . . . . . . . . . . . . . 17 3.1.5. Overload . . . . . . . . . . . . . . . . . . . . . . . 17
3.1.6. Session Attempt . . . . . . . . . . . . . . . . . . . 18 3.1.6. Session Attempt . . . . . . . . . . . . . . . . . . . 18
3.1.7. Established Session . . . . . . . . . . . . . . . . . 18 3.1.7. Established Session . . . . . . . . . . . . . . . . . 18
3.1.8. Invite-initiated Session (IS) . . . . . . . . . . . . 19 3.1.8. Invite-initiated Session (IS) . . . . . . . . . . . . 19
3.1.9. Non-INVITE-initiated Session (NS) . . . . . . . . . . 19 3.1.9. Non-INVITE-initiated Session (NS) . . . . . . . . . . 20
3.1.10. Session Attempt Failure . . . . . . . . . . . . . . . 20 3.1.10. Session Attempt Failure . . . . . . . . . . . . . . . 20
3.1.11. Standing Sessions Count . . . . . . . . . . . . . . . 20 3.1.11. Standing Sessions Count . . . . . . . . . . . . . . . 21
3.2. Test Components . . . . . . . . . . . . . . . . . . . . . 21 3.2. Test Components . . . . . . . . . . . . . . . . . . . . . 21
3.2.1. Emulated Agent . . . . . . . . . . . . . . . . . . . . 21 3.2.1. Emulated Agent . . . . . . . . . . . . . . . . . . . . 21
3.2.2. Signaling Server . . . . . . . . . . . . . . . . . . . 21 3.2.2. Signaling Server . . . . . . . . . . . . . . . . . . . 22
3.2.3. SIP-Aware Stateful Firewall . . . . . . . . . . . . . 22 3.2.3. SIP-Aware Stateful Firewall . . . . . . . . . . . . . 22
3.2.4. SIP Transport Protocol . . . . . . . . . . . . . . . . 22 3.2.4. SIP Transport Protocol . . . . . . . . . . . . . . . . 23
3.3. Test Setup Parameters . . . . . . . . . . . . . . . . . . 23 3.3. Test Setup Parameters . . . . . . . . . . . . . . . . . . 24
3.3.1. Session Attempt Rate . . . . . . . . . . . . . . . . . 23 3.3.1. Session Attempt Rate . . . . . . . . . . . . . . . . . 24
3.3.2. IS Media Attempt Rate . . . . . . . . . . . . . . . . 23 3.3.2. IS Media Attempt Rate . . . . . . . . . . . . . . . . 24
3.3.3. Establishment Threshold Time . . . . . . . . . . . . . 24 3.3.3. Establishment Threshold Time . . . . . . . . . . . . . 25
3.3.4. Session Duration . . . . . . . . . . . . . . . . . . . 25 3.3.4. Session Duration . . . . . . . . . . . . . . . . . . . 25
3.3.5. Media Packet Size . . . . . . . . . . . . . . . . . . 25 3.3.5. Media Packet Size . . . . . . . . . . . . . . . . . . 26
3.3.6. Media Offered Load . . . . . . . . . . . . . . . . . . 26 3.3.6. Media Offered Load . . . . . . . . . . . . . . . . . . 26
3.3.7. Media Session Hold Time . . . . . . . . . . . . . . . 26 3.3.7. Media Session Hold Time . . . . . . . . . . . . . . . 27
3.3.8. Loop Detection Option . . . . . . . . . . . . . . . . 27 3.3.8. Loop Detection Option . . . . . . . . . . . . . . . . 27
3.3.9. Forking Option . . . . . . . . . . . . . . . . . . . . 27 3.3.9. Forking Option . . . . . . . . . . . . . . . . . . . . 28
3.4. Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . 28 3.4. Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . 29
3.4.1. Registration Rate . . . . . . . . . . . . . . . . . . 28 3.4.1. Registration Rate . . . . . . . . . . . . . . . . . . 29
3.4.2. Session Establishment Rate . . . . . . . . . . . . . . 29 3.4.2. Session Establishment Rate . . . . . . . . . . . . . . 29
3.4.3. Session Capacity . . . . . . . . . . . . . . . . . . . 30 3.4.3. Session Capacity . . . . . . . . . . . . . . . . . . . 30
3.4.4. Session Overload Capacity . . . . . . . . . . . . . . 31 3.4.4. Session Overload Capacity . . . . . . . . . . . . . . 31
3.4.5. Session Establishment Performance . . . . . . . . . . 31 3.4.5. Session Establishment Performance . . . . . . . . . . 31
3.4.6. Session Attempt Delay . . . . . . . . . . . . . . . . 32 3.4.6. Session Attempt Delay . . . . . . . . . . . . . . . . 32
3.4.7. IM Rate . . . . . . . . . . . . . . . . . . . . . . . 32 3.4.7. IM Rate . . . . . . . . . . . . . . . . . . . . . . . 32
4. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 33 4. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 33
5. Security Considerations . . . . . . . . . . . . . . . . . . . 33 5. Security Considerations . . . . . . . . . . . . . . . . . . . 33
6. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 34 6. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 34
7. References . . . . . . . . . . . . . . . . . . . . . . . . . . 34 7. References . . . . . . . . . . . . . . . . . . . . . . . . . . 34
skipping to change at page 6, line 33 skipping to change at page 6, line 33
be a Registrar, Redirect Server, Stateless Proxy or Stateful be a Registrar, Redirect Server, Stateless Proxy or Stateful
Proxy. A DUT MAY also include a B2BUA, SBC functionality (this is Proxy. A DUT MAY also include a B2BUA, SBC functionality (this is
referred to as the "Signaling Server".) The DUT MAY be a multi- referred to as the "Signaling Server".) The DUT MAY be a multi-
port SIP-to-switched network gateway implemented as a SIP UAC or port SIP-to-switched network gateway implemented as a SIP UAC or
UAS. UAS.
o The DUT MAY have an internal SIP Application Level Gateway (ALG), o The DUT MAY have an internal SIP Application Level Gateway (ALG),
firewall, and/or a Network Address Translator (NAT). This is firewall, and/or a Network Address Translator (NAT). This is
referred to as the "SIP Aware Stateful Firewall." referred to as the "SIP Aware Stateful Firewall."
o The DUT or SUT MUST NOT be end user equipment, such as personal o The DUT or SUT MUST NOT be end user equipment, such as personal
digital assistant, a computer-based client, or a user terminal. digital assistant, a computer-based client, or a user terminal.
o The Tester acts as multiple "Emulated Agents" that initiate (or o The Tester acts as multiple "Emulated Agents" (EA) that initiate
respond to) SIP messages as session endpoints and source (or (or respond to) SIP messages as session endpoints and source (or
receive) associated media for established connections. receive) associated media for established connections.
o Control Signaling in presence of Media o Control Signaling in presence of Media
* The media performance is not benchmarked in this work item. * The media performance is not benchmarked in this work item.
* It is RECOMMENDED that control plane benchmarks are performed * It is RECOMMENDED that control plane benchmarks are performed
with media present, but this is optional. with media present, but this is optional.
* The SIP INVITE requests MUST include the SDP body. * The SIP INVITE requests MUST include the SDP body.
* The type of DUT dictates whether the associated media streams * The type of DUT dictates whether the associated media streams
traverse the DUT or SUT. Both scenarios are within the scope traverse the DUT or SUT. Both scenarios are within the scope
of this work item. of this work item.
* SIP is frequently used to create media streams; the control * SIP is frequently used to create media streams; the control
skipping to change at page 7, line 46 skipping to change at page 7, line 46
as the expected test outcome if it is different from the IETF as the expected test outcome if it is different from the IETF
recommendations. However, such behavior MUST be documented and recommendations. However, such behavior MUST be documented and
interpreted appropriately across multiple vendor implementations. interpreted appropriately across multiple vendor implementations.
This will make it more meaningful to compare the performance of This will make it more meaningful to compare the performance of
different SIP overload implementations. different SIP overload implementations.
o IMS-specific scenarios are not considered, but test cases can be o IMS-specific scenarios are not considered, but test cases can be
applied with 3GPP-specific SIP signaling and the P-CSCF as a DUT. applied with 3GPP-specific SIP signaling and the P-CSCF as a DUT.
2.2. Benchmarking Models 2.2. Benchmarking Models
This section shows the five models to be used when benchmarking SIP This section shows ten models to be used when benchmarking SIP
performance of a networking device. Figure 1 shows the DUT plays the performance of a networking device. Figure 1 shows shows the
role of a user agent client (UAC), initiating requests and absorbing configuration needed to benchmark the tester itself. This model will
responses. This model can be used as a baseline performance for the be used to establish the limitations of the test apparatus.
DUT acting as a UAC without associated media.
+--------+ Signaling request +--------+
| +----------------------------->| |
| Tester | | Tester |
| | Signaling response | EA |
| |<-----------------------------+ |
+--------+ +--------+
/|\ /|\
| Media |
+=========================================+
Figure 1: Baseline performance of the Emulated Agent without a DUT
present
Figure 2 shows the DUT playing the role of a user agent client (UAC),
initiating requests and absorbing responses. This model can be used
to baseline the performance of the DUT acting as an UAC without
associated media.
+--------+ Signaling request +--------+ +--------+ Signaling request +--------+
| +----------------------------->| | | +----------------------------->| |
| DUT | | Tester | | DUT | | Tester |
| | Signaling response | EA | | | Signaling response | EA |
| |<-----------------------------+ | | |<-----------------------------+ |
+--------+ +--------+ +--------+ +--------+
Figure 1: Baseline performance for DUT acting as a user agent client Figure 2: Baseline performance for DUT acting as a user agent client
without associated media without associated media
Figure 2 shows the DUT plays the role of a user agent server (UAS), Figure 3 shows the DUT plays the role of a user agent server (UAS),
absorbing the requests and sending responses. This model can be used absorbing the requests and sending responses. This model can be used
as a baseline performance for the DUT acting as a UAS without as a baseline performance for the DUT acting as a UAS without
associated media. associated media.
+--------+ Signaling request +--------+ +--------+ Signaling request +--------+
| +----------------------------->| | | +----------------------------->| |
| Tester | | DUT | | Tester | | DUT |
| EA | Response | | | EA | Response | |
| |<-----------------------------+ | | |<-----------------------------+ |
+--------+ +--------+ +--------+ +--------+
Figure 2: Baseline performance for DUT acting as a user agent server Figure 3: Baseline performance for DUT acting as a user agent server
without associated media without associated media
Figure 3 shows the DUT plays the role of a user agent client (UAC), Figure 4 shows the DUT plays the role of a user agent client (UAC),
initiating requests and absorbing responses. This model can be used initiating requests and absorbing responses. This model can be used
as a baseline performance for the DUT acting as a UAC with associated as a baseline performance for the DUT acting as a UAC with associated
media. media.
+--------+ Signaling request +--------+ +--------+ Signaling request +--------+
| +----------------------------->| | | +----------------------------->| |
| DUT | | Tester | | DUT | | Tester |
| | Signaling response | EA | | | Signaling response | EA |
| |<-----------------------------+ | | |<-----------------------------+ |
| |<============ Media =========>| | | |<============ Media =========>| |
+--------+ +--------+ +--------+ +--------+
Figure 3: Baseline performance for DUT acting as a user agent client Figure 4: Baseline performance for DUT acting as a user agent client
with associated media with associated media
Figure 4 shows the DUT plays the role of a user agent server (UAS), Figure 5 shows the DUT plays the role of a user agent server (UAS),
absorbing the requests and sending responses. This model can be used absorbing the requests and sending responses. This model can be used
as a baseline performance for the DUT acting as a UAS with associated as a baseline performance for the DUT acting as a UAS with associated
media. media.
+--------+ Signaling request +--------+ +--------+ Signaling request +--------+
| +----------------------------->| | | +----------------------------->| |
| Tester | | DUT | | Tester | | DUT |
| EA | Response | | | EA | Response | |
| |<-----------------------------+ | | |<-----------------------------+ |
| |<============ Media =========>| | | |<============ Media =========>| |
+--------+ +--------+ +--------+ +--------+
Figure 4: Baseline performance for DUT acting as a user agent server Figure 5: Baseline performance for DUT acting as a user agent server
with associated media with associated media
Figure 5 shows that the Tester acts as the initiating and responding Figure 6 shows that the Tester acts as the initiating and responding
Emulated Agents as the DUT/SUT forwards Session Attempts. EA as the DUT/SUT forwards Session Attempts.
+--------+ Session +--------+ Session +--------+ +--------+ Session +--------+ Session +--------+
| | Attempt | | Attempt | | | | Attempt | | Attempt | |
| |<------------+ |<------------+ | | |<------------+ |<------------+ |
| | | | | | | | | | | |
| | Response | | Response | | | | Response | | Response | |
| Tester +------------>| DUT +------------>| Tester | | Tester +------------>| DUT +------------>| Tester |
| (EA) | | | | (EA) | | (EA) | | | | (EA) |
| | | | | | | | | | | |
+--------+ +--------+ +--------+ +--------+ +--------+ +--------+
Figure 5: DUT/SUT performance benchmark for session establishment Figure 6: DUT/SUT performance benchmark for session establishment
without media without media
Figure 6 is used when performing those same benchmarks with Figure 7 is used when performing those same benchmarks with
Associated Media traversing the DUT/SUT. Associated Media traversing the DUT/SUT.
+--------+ Session +--------+ Session +--------+ +--------+ Session +--------+ Session +--------+
| | Attempt | | Attempt | | | | Attempt | | Attempt | |
| |<------------+ |<------------+ | | |<------------+ |<------------+ |
| | | | | | | | | | | |
| | Response | | Response | | | | Response | | Response | |
| Tester +------------>| DUT +------------>| Tester | | Tester +------------>| DUT +------------>| Tester |
| | | | | (EA) | | | | | | (EA) |
| | Media | | Media | | | | Media | | Media | |
| |<===========>| |<===========>| | | |<===========>| |<===========>| |
+--------+ +--------+ +--------+ +--------+ +--------+ +--------+
Figure 6: DUT/SUT performance benchmark for session establishment Figure 7: DUT/SUT performance benchmark for session establishment
with media traversing the DUT with media traversing the DUT
Figure 7 is to be used when performing those same benchmarks with Figure 8 is to be used when performing those same benchmarks with
Associated Media, but the media does not traverse the DUT/SUT. Associated Media, but the media does not traverse the DUT/SUT.
Again, the benchmarking of the media is not within the scope of this Again, the benchmarking of the media is not within the scope of this
work item. The SIP control signaling is benchmarked in the presence work item. The SIP control signaling is benchmarked in the presence
of Associated Media to determine if the SDP body of the signaling and of Associated Media to determine if the SDP body of the signaling and
the handling of media impacts the performance of the DUT/SUT. the handling of media impacts the performance of the DUT/SUT.
+--------+ Session +--------+ Session +--------+ +--------+ Session +--------+ Session +--------+
| | Attempt | | Attempt | | | | Attempt | | Attempt | |
| |<------------+ |<------------+ | | |<------------+ |<------------+ |
| | | | | | | | | | | |
| | Response | | Response | | | | Response | | Response | |
| Tester +------------>| DUT +------------>| Tester | | Tester +------------>| DUT +------------>| Tester |
| | | | | (EA) | | | | | | (EA) |
| | | | | | | | | | | |
+--------+ +--------+ +--------+ +--------+ +--------+ +--------+
/|\ /|\ /|\ /|\
| Media | | Media |
+=============================================+ +=============================================+
Figure 7: DUT/SUT performance benchmark for session establishment Figure 8: DUT/SUT performance benchmark for session establishment
with media external to the DUT with media external to the DUT
Figure 8 is used when performing benchmarks that require one or more Figure 9 is used when performing benchmarks that require one or more
intermediaries to be in the signaling path. The intent is to gather intermediaries to be in the signaling path. The intent is to gather
benchmarking statistics with a series of DUTs in place. In this benchmarking statistics with a series of DUTs in place. In this
topology, the media is delivered end-to-end and does not traverse the topology, the media is delivered end-to-end and does not traverse the
DUT. DUT.
SUT SUT
'--------------------------^^^^^^^^-----------------------` '--------------------------^^^^^^^^-----------------------`
/ \ / \
+------+ Session +---+ Session +---+ Session +------+ +------+ Session +---+ Session +---+ Session +------+
| | Attempt | | Attempt | | Attempt | | | | Attempt | | Attempt | | Attempt | |
skipping to change at page 11, line 21 skipping to change at page 11, line 43
| | | | | | | | | | | | | | | |
| | Response | | Response | | Response | | | | Response | | Response | | Response | |
|Tester+--------->|DUT+--------->|DUT|--------->|Tester| |Tester+--------->|DUT+--------->|DUT|--------->|Tester|
| | | | | | | | | | | | | | | |
| | | | | | | | | | | | | | | |
+------+ +---+ +---+ +------+ +------+ +---+ +---+ +------+
/|\ /|\ /|\ /|\
| Media | | Media |
+=============================================+ +=============================================+
Figure 8: DUT/SUT performance benchmark for session establishment Figure 9: DUT/SUT performance benchmark for session establishment
with multiple DUTs and end-to-end media with multiple DUTs and end-to-end media
Figure 9 is used when performing benchmarks that require one or more Figure 10 is used when performing benchmarks that require one or more
intermediaries to be in the signaling path. The intent is to gather intermediaries to be in the signaling path. The intent is to gather
benchmarking statistics with a series of DUTs in place. In this benchmarking statistics with a series of DUTs in place. In this
topology, the media is delivered hop-by-hop through each DUT. topology, the media is delivered hop-by-hop through each DUT.
SUT SUT
'--------------------------^^^^^^^^-----------------------` '--------------------------^^^^^^^^-----------------------`
/ \ / \
+------+ Session +---+ Session +---+ Session +------+ +------+ Session +---+ Session +---+ Session +------+
| | Attempt | | Attempt | | Attempt | | | | Attempt | | Attempt | | Attempt | |
| |<---------+ |<---------+ |<---------+ | | |<---------+ |<---------+ |<---------+ |
| | | | | | | | | | | | | | | |
| | Response | | Response | | Response | | | | Response | | Response | | Response | |
|Tester+--------->|DUT+--------->|DUT|--------->|Tester| |Tester+--------->|DUT+--------->|DUT|--------->|Tester|
| | | | | | | | | | | | | | | |
| | | | | | | | | | | | | | | |
| |<========>| |<========>| |<========>| | | |<========>| |<========>| |<========>| |
+------+ Media +---+ Media +---+ Media +------+ +------+ Media +---+ Media +---+ Media +------+
Figure 9: DUT/SUT performance benchmark for session establishment Figure 10: DUT/SUT performance benchmark for session establishment
with multiple DUTs and hop- by-hop media with multiple DUTs and hop- by-hop media
Figure 10 illustrates the SIP signaling for an Established Session. Figure 11 illustrates the SIP signaling for an Established Session.
The Tester acts as the Emulated Agent(s) and initiates a Session The Tester acts as the EAs and initiates a Session Attempt with the
Attempt with the DUT/SUT. When the Emulated Agent (EA) receives a DUT/SUT. When the Emulated Agent (EA) receives a 200 OK from the
200 OK from the DUT/SUT that session is considered to be an DUT/SUT that session is considered to be an Established Session. The
Established Session. The illustration indicates three states of the illustration indicates three states of the session bring created by
session bring created by the EA - Attempting, Established, and the EA - Attempting, Established, and Disconnecting. Sessions can be
Disconnecting. Sessions can be one of two type: Invite-Initiated one of two type: Invite-Initiated Session (IS) or Non-Invite
Session (IS) or Non-Invite Initiated Session (NS). Failure for the Initiated Session (NS). Failure for the DUT/SUT to successfully
DUT/SUT to successfully respond within the Establishment Threshold respond within the Establishment Threshold Time is considered a
Time is considered a Session Attempt Failure. SIP Invite messages Session Attempt Failure. SIP Invite messages MUST include the SDP
MUST include the SDP body to specify the Associated Media. Use of body to specify the Associated Media. Use of Associated Media, to be
Associated Media, to be sourced from the EA, is optional. When sourced from the EA, is optional. When Associated Media is used, it
Associated Media is used, it may traverse the DUT/SUT depending upon may traverse the DUT/SUT depending upon the type of DUT/SUT. The
the type of DUT/SUT. The Associated Media is shown in Figure 10 as Associated Media is shown in Figure 11 as "Media" connected to media
"Media" connected to media ports M1 and M2 on the EA. After the EA ports M1 and M2 on the EA. After the EA sends a BYE, the session
sends a BYE, the session disconnects. Performance test cases for disconnects. Performance test cases for session disconnects are not
session disconnects are not considered in this work item (the BYE considered in this work item (the BYE request is shown for
request is shown for completeness.) completeness.)
EA DUT/SUT M1 M2 EA DUT/SUT M1 M2
| | | | | | | |
| INVITE | | | | INVITE | | |
--------+--------------->| | | --------+--------------->| | |
| | | | | | | |
Attempting | | | Attempting | | |
| 200 OK | | | | 200 OK | | |
--------|<-------------- | | | --------|<-------------- | | |
| | | | | | | |
| | | | | | | |
skipping to change at page 12, line 40 skipping to change at page 13, line 25
Established | |<=====>| Established | |<=====>|
| | | | | | | |
| BYE | | | | BYE | | |
--------+--------------> | | | --------+--------------> | | |
| | | | | | | |
Disconnecting | | | Disconnecting | | |
| 200 OK | | | | 200 OK | | |
--------|<-------------- | | | --------|<-------------- | | |
| | | | | | | |
Figure 10: Basic SIP test topology Figure 11: Basic SIP test topology
3. Term Definitions 3. Term Definitions
3.1. Protocol Components 3.1. Protocol Components
3.1.1. Session 3.1.1. Session
Definition: Definition:
The combination of signaling and media messages and processes that The combination of signaling and media messages and processes that
enable two or more participants to communicate. enable two or more participants to communicate.
Discussion: Discussion:
SIP messages in the signaling plane can be used to create and SIP messages in the signaling plane can be used to create and
manage applications for one or more end users. SIP is often used manage applications for one or more end users. SIP is often used
to create and manage media streams in support of applications. A to create and manage media streams in support of applications. A
skipping to change at page 14, line 18 skipping to change at page 14, line 45
sessions are represented as an array session[x]. sessions are represented as an array session[x].
Sessions will be represented as a vector array with three Sessions will be represented as a vector array with three
components, as follows: components, as follows:
session-> session->
session[x].sig, the signaling component session[x].sig, the signaling component
session[x].medc, the media control component (e.g. RTCP) session[x].medc, the media control component (e.g. RTCP)
session[x].med[y], an array of associated media streams (e.g. session[x].med[y], an array of associated media streams (e.g.
RTP, SRTP, RTSP, MSRP). This media component may consist of zero RTP, SRTP, RTSP, MSRP). This media component may consist of zero
or more media streams. or more media streams.
Figure 11 models the vectors of the session. Figure 12 models the vectors of the session.
Measurement Units: Measurement Units:
N/A. N/A.
Issues: Issues:
None. None.
See Also: See Also:
Media Plane Media Plane
Signaling Plane Signaling Plane
Associated Media Associated Media
Invite-initiated Session (IS) Invite-initiated Session (IS)
skipping to change at page 15, line 31 skipping to change at page 15, line 44
/ / / /
/ | / |
/ / / /
sess.med / | sess.med / |
/_ _ _ _ _ _ _ _/ /_ _ _ _ _ _ _ _/
/ /
/ /
/ /
/ /
Figure 11: Application or session components Figure 12: Application or session components
3.1.2. Signaling Plane 3.1.2. Signaling Plane
Definition: Definition:
The control plane in which SIP messages [RFC3261] are exchanged The control plane in which SIP messages [RFC3261] are exchanged
between SIP Agents [RFC3261] to establish a connection for media between SIP Agents [RFC3261] to establish a connection for media
exchange. exchange.
Discussion: Discussion:
SIP messages are used to establish sessions in several ways: SIP messages are used to establish sessions in several ways:
skipping to change at page 16, line 13 skipping to change at page 16, line 28
session.sig. session.sig.
Measurement Units: Measurement Units:
N/A. N/A.
Issues: Issues:
None. None.
See Also: See Also:
Media Plane Media Plane
Emulated Agents EAs
3.1.3. Media Plane 3.1.3. Media Plane
Definition: Definition:
The data plane in which one or more media streams and their The data plane in which one or more media streams and their
associated media control protocols are exchanged after a media associated media control protocols are exchanged after a media
connection has been created by the exchange of signaling messages connection has been created by the exchange of signaling messages
in the Signaling Plane. in the Signaling Plane.
Discussion: Discussion:
skipping to change at page 18, line 9 skipping to change at page 18, line 27
stimulus -- sending a 503 response -- is considered inadequate and stimulus -- sending a 503 response -- is considered inadequate and
new response codes and behaviors may be specified in the future. new response codes and behaviors may be specified in the future.
From the perspective of this document, all these responses will be From the perspective of this document, all these responses will be
considered to be failures. There is thus no dependency between considered to be failures. There is thus no dependency between
this document and the ongoing work on the treatment of overload this document and the ongoing work on the treatment of overload
failure. failure.
3.1.6. Session Attempt 3.1.6. Session Attempt
Definition: Definition:
A SIP Session for which the Emulated Agent has sent the SIP INVITE A SIP Session for which the EA has sent the SIP INVITE or
or SUBSCRIBE NOTIFY and has not yet received a message response SUBSCRIBE NOTIFY and has not yet received a message response from
from the DUT/SUT. the DUT/SUT.
Discussion: Discussion:
The attempted session may be an IS or an NS. The Session Attempt The attempted session may be an IS or an NS. The Session Attempt
includes SIP INVITEs and SUBSCRIBE/NOTIFY messages. It also includes SIP INVITEs and SUBSCRIBE/NOTIFY messages. It also
includes all INVITEs that are rejected for lack of authentication includes all INVITEs that are rejected for lack of authentication
information. information.
Measurement Units: Measurement Units:
N/A. N/A.
skipping to change at page 18, line 32 skipping to change at page 19, line 4
Issues: Issues:
None. None.
See Also: See Also:
Session Session
Session Attempt Rate Session Attempt Rate
Invite-initiated Session Invite-initiated Session
Non-Invite initiated Session Non-Invite initiated Session
3.1.7. Established Session 3.1.7. Established Session
Definition: Definition:
A SIP session for which the Emulated Agent acting as the UE/UA has A SIP session for which the EA acting as the UE/UA has received a
received a 200 OK message from the DUT/SUT. 200 OK message from the DUT/SUT.
Discussion: Discussion:
An Established Session MAY be type INVITE-Session (IS) or Non- An Established Session MAY be type INVITE-Session (IS) or Non-
INVITE Session (NS). INVITE Session (NS).
Measurement Units: Measurement Units:
N/A. N/A.
Issues: Issues:
None. None.
skipping to change at page 20, line 22 skipping to change at page 20, line 41
Session Session
Invite-initiated Session Invite-initiated Session
3.1.10. Session Attempt Failure 3.1.10. Session Attempt Failure
Definition: Definition:
A session attempt that does not result in an Established Session. A session attempt that does not result in an Established Session.
Discussion: Discussion:
The session attempt failure may be indicated by the following The session attempt failure may be indicated by the following
observations at the Emulated Agent: observations at the EA:
1. Receipt of a SIP 4xx, 5xx, or 6xx class response to a Session 1. Receipt of a SIP 4xx, 5xx, or 6xx class response to a Session
Attempt. Attempt.
2. The lack of any received SIP response to a Session Attempt 2. The lack of any received SIP response to a Session Attempt
within the Establishment Threshold Time (c.f. Section 3.3.3). within the Establishment Threshold Time (c.f. Section 3.3.3).
Measurement Units: Measurement Units:
N/A. N/A.
Issues: Issues:
None. None.
skipping to change at page 20, line 47 skipping to change at page 21, line 24
3.1.11. Standing Sessions Count 3.1.11. Standing Sessions Count
Definition: Definition:
The number of Sessions currently established on the DUT/SUT at any The number of Sessions currently established on the DUT/SUT at any
instant. instant.
Discussion: Discussion:
The number of Standing Sessions is influenced by the Session The number of Standing Sessions is influenced by the Session
Duration and the Session Attempt Rate. Benchmarks MUST be Duration and the Session Attempt Rate. Benchmarks MUST be
reported with the maximum and average Standing Sessions for the reported with the maximum and average Standing Sessions for the
DUT/SUT. In order to determine the maximum and average Standing DUT/SUT for the duration of the test. In order to determine the
Sessions on the DUT/SUT for the duration of the test it is maximum and average Standing Sessions on the DUT/SUT for the
necessary to make periodic measurements of the number of Standing duration of the test it is necessary to make periodic measurements
Sessions on the DUT/SUT. The recommended value for the of the number of Standing Sessions on the DUT/SUT. The
measurement period is 1 second. recommended value for the measurement period is 1 second. Since
we cannot directly poll the DUT/SUT, we take the number of
standing sessions on the DUT/SUT to be the number of distinct
calls as measured by the number of distinct Call-IDs that the EA
is processing at the time of measurement.
Measurement Units: Measurement Units:
Number of sessions Number of sessions
Issues: Issues:
None. None.
See Also: See Also:
Session Duration Session Duration
Session Attempt Rate Session Attempt Rate
skipping to change at page 21, line 20 skipping to change at page 22, line 4
None. None.
See Also: See Also:
Session Duration Session Duration
Session Attempt Rate Session Attempt Rate
Session Attempt Rate Session Attempt Rate
3.2. Test Components 3.2. Test Components
3.2.1. Emulated Agent 3.2.1. Emulated Agent
Definition: Definition:
A device in test topology that initiates/responds to SIP messages A device in test topology that initiates/responds to SIP messages
as one or more session endpoints and, wherever applicable, as one or more session endpoints and, wherever applicable,
sources/receives Associated Media for Established Sessions. sources/receives Associated Media for Established Sessions.
Discussion: Discussion:
The Emulated Agent functions in the signaling and media planes. The EA functions in the signaling and media planes. The Tester
The Tester may act as multiple Emulated Agents. may act as multiple EAs.
Measurement Units: Measurement Units:
N/A N/A
Issues: Issues:
None. None.
See Also: See Also:
Media Plane Media Plane
Signaling Plane Signaling Plane
Established Session Established Session
Associated Media Associated Media
3.2.2. Signaling Server 3.2.2. Signaling Server
Definition: Definition:
Device in test topology that acts to create sessions between Device in test topology that acts to create sessions between EAs
Emulated Agents in the media plane. This device is either a DUT in the media plane. This device is either a DUT or component of a
or component of a SUT. SUT.
Discussion: Discussion:
The DUT MUST be a RFC 3261 capable network equipment such as a The DUT MUST be a RFC 3261 capable network equipment such as a
Registrar, Redirect Server, User Agent Server, Stateless Proxy, or Registrar, Redirect Server, User Agent Server, Stateless Proxy, or
Stateful Proxy. A DUT MAY also include B2BUA or SBC. Stateful Proxy. A DUT MAY also include B2BUA or SBC.
Measurement Units: Measurement Units:
NA NA
Issues: Issues:
skipping to change at page 22, line 20 skipping to change at page 23, line 4
Measurement Units: Measurement Units:
NA NA
Issues: Issues:
None. None.
See Also: See Also:
Signaling Plane Signaling Plane
3.2.3. SIP-Aware Stateful Firewall 3.2.3. SIP-Aware Stateful Firewall
Definition: Definition:
Device in test topology that provides Denial-of-Service (DoS) Device in test topology that provides Denial-of-Service (DoS)
Protection to the Signaling and Media Planes for the Emulated Protection to the Signaling and Media Planes for the EAs and
Agents and Signaling Server Signaling Server
Discussion: Discussion:
The SIP-Aware Stateful Firewall MAY be an internal component or The SIP-Aware Stateful Firewall MAY be an internal component or
function of the Session Server. The SIP-Aware Stateful Firewall function of the Session Server. The SIP-Aware Stateful Firewall
MAY be a standalone device. If it is a standalone device it MUST MAY be a standalone device. If it is a standalone device it MUST
be paired with a Signaling Server. If it is a standalone device be paired with a Signaling Server. If it is a standalone device
it MUST be benchmarked as part of a SUT. SIP-Aware Stateful it MUST be benchmarked as part of a SUT. SIP-Aware Stateful
Firewalls MAY include Network Address Translation (NAT) Firewalls MAY include Network Address Translation (NAT)
functionality. Ideally, the inclusion of the SIP-Aware Stateful functionality. Ideally, the inclusion of the SIP-Aware Stateful
Firewall as a SUT has no degradation to the measured performance Firewall as a SUT has no degradation to the measured performance
skipping to change at page 23, line 28 skipping to change at page 24, line 10
Issues: Issues:
None. None.
See Also: See Also:
3.3. Test Setup Parameters 3.3. Test Setup Parameters
3.3.1. Session Attempt Rate 3.3.1. Session Attempt Rate
Definition: Definition:
Configuration of the Emulated Agent for the number of sessions Configuration of the EA for the number of sessions that the EA
that the Emulated Agent attempts to establish with the DUT/SUT attempts to establish with the DUT/SUT over a specified time
over a specified time interval. interval.
Discussion: Discussion:
The Session Attempt Rate can cause variation in performance The Session Attempt Rate can cause variation in performance
benchmark measurements. Since this is the number of sessions benchmark measurements. Since this is the number of sessions
configured on the Tester, some sessions may not be successfully configured on the Tester, some sessions may not be successfully
established on the DUT. A session may be either an IS or an NS. established on the DUT. A session may be either an IS or an NS.
Measurement Units: Measurement Units:
Session attempts per second Session attempts per second
skipping to change at page 24, line 4 skipping to change at page 24, line 31
Session attempts per second Session attempts per second
Issues: Issues:
None. None.
See Also: See Also:
Session Session
Session Attempt Session Attempt
3.3.2. IS Media Attempt Rate 3.3.2. IS Media Attempt Rate
Definition: Definition:
Configuration on the Emulated Agent for number of ISs with Configuration on the EA for number of ISs with Associated Media to
Associated Media to be established at the DUT per continuous one- be established at the DUT per continuous one- second time
second time intervals. intervals.
Discussion: Discussion:
Note that a Media Session MUST be associated with an IS. In this Note that a Media Session MUST be associated with an IS. In this
document we assume that there is a one to one correspondence document we assume that there is a one to one correspondence
between IS session attempts and Media Session attempts. By between IS session attempts and Media Session attempts. By
including this definition we leave open the possibility that there including this definition we leave open the possibility that there
may be an IS that does not include a media description. Also note may be an IS that does not include a media description. Also note
that the IS Media Attempt Rate defines the number of media that the IS Media Attempt Rate defines the number of media
sessions we are trying to create, not the number of media sessions sessions we are trying to create, not the number of media sessions
that are actually created. Variations in the Media Session that are actually created. Variations in the Media Session
skipping to change at page 24, line 34 skipping to change at page 25, line 17
Issues: Issues:
None. None.
See Also: See Also:
IS IS
3.3.3. Establishment Threshold Time 3.3.3. Establishment Threshold Time
Definition: Definition:
Configuration of the Emulated Agent for representing the amount of Configuration of the EA for representing the amount of time that
time that an Emulated Agent will wait before declaring a Session an EA will wait before declaring a Session Attempt Failure.
Attempt Failure.
Discussion: Discussion:
This time duration is test dependent. This time duration is test dependent.
It is RECOMMENDED that the Establishment Threshold Time value be It is RECOMMENDED that the Establishment Threshold Time value be
set to Timer B (for ISs) or Timer F (for NSs) as specified in RFC set to Timer B (for ISs) or Timer F (for NSs) as specified in RFC
3261, Table 4 [RFC3261]. Following the default value of T1 3261, Table 4 [RFC3261]. Following the default value of T1
(500ms) specified in the table and a constant multiplier of 64 (500ms) specified in the table and a constant multiplier of 64
gives a value of 32 seconds for this timer (i.e., 500ms * 64 = gives a value of 32 seconds for this timer (i.e., 500ms * 64 =
32s). 32s).
skipping to change at page 25, line 16 skipping to change at page 25, line 41
Issues: Issues:
None. None.
See Also: See Also:
session establishment failure session establishment failure
3.3.4. Session Duration 3.3.4. Session Duration
Definition: Definition:
Configuration of the Emulated Agent that represents the amount of Configuration of the EA that represents the amount of time that
time that the SIP dialog is intended to exist between the two EAs the SIP dialog is intended to exist between the two EAs associated
associated with the test. with the test.
Discussion: Discussion:
The time at which the BYE is sent will control the Session The time at which the BYE is sent will control the Session
Duration Duration
Normally the Session Duration will be the same as the Media Normally the Session Duration will be the same as the Media
Session Hold Time. However, it is possible that the dialog Session Hold Time. However, it is possible that the dialog
established between the two EAs can support different media established between the two EAs can support different media
sessions at different points in time. Providing both parameters sessions at different points in time. Providing both parameters
allows the testing agency to explore this possibility. allows the testing agency to explore this possibility.
skipping to change at page 25, line 41 skipping to change at page 26, line 22
Issues: Issues:
None. None.
See Also: See Also:
Media Session Hold Time Media Session Hold Time
3.3.5. Media Packet Size 3.3.5. Media Packet Size
Definition: Definition:
Configuration on the Emulated Agent for a fixed size of packets Configuration on the EA for a fixed size of packets used for media
used for media streams. streams.
Discussion: Discussion:
For a single benchmark test, all sessions use the same size packet For a single benchmark test, all sessions use the same size packet
for media streams. The size of packets can cause variation in for media streams. The size of packets can cause variation in
performance benchmark measurements. performance benchmark measurements.
Measurement Units: Measurement Units:
bytes bytes
Issues: Issues:
None. None.
See Also: See Also:
3.3.6. Media Offered Load 3.3.6. Media Offered Load
Definition: Definition:
Configuration of the Emulated Agent for the constant rate of Configuration of the EA for the constant rate of Associated Media
Associated Media traffic offered by the Emulated Agent to the DUT/ traffic offered by the EA to the DUT/SUT for one or more
SUT for one or more Established Sessions of type IS. Established Sessions of type IS.
Discussion: Discussion:
The Media Offered Load to be used for a test MUST be reported with The Media Offered Load to be used for a test MUST be reported with
three components: three components:
1. per Associated Media stream; 1. per Associated Media stream;
2. per IS; 2. per IS;
3. aggregate. 3. aggregate.
For a single benchmark test, all sessions use the same Media For a single benchmark test, all sessions use the same Media
Offered Load per Media Stream. There may be multiple Associated Offered Load per Media Stream. There may be multiple Associated
Media streams per IS. The aggregate is the sum of all Associated Media streams per IS. The aggregate is the sum of all Associated
skipping to change at page 26, line 45 skipping to change at page 27, line 24
None. None.
See Also: See Also:
Established Session Established Session
Invite Initiated Session Invite Initiated Session
Associated Media Associated Media
3.3.7. Media Session Hold Time 3.3.7. Media Session Hold Time
Definition: Definition:
Parameter configured at the Emulated Agent, that represents the Parameter configured at the EA, that represents the amount of time
amount of time that the Associated Media for an Established that the Associated Media for an Established Session of type IS
Session of type IS will last. will last.
Discussion: Discussion:
The Associated Media streams may be bi-directional or uni- The Associated Media streams may be bi-directional or uni-
directional as indicated in the test methodology. directional as indicated in the test methodology.
Normally the Media Session Hold Time will be the same as the Normally the Media Session Hold Time will be the same as the
Session Duration. However, it is possible that the dialog Session Duration. However, it is possible that the dialog
established between the two EAs can support different media established between the two EAs can support different media
sessions at different points in time. Providing both parameters sessions at different points in time. Providing both parameters
allows the testing agency to explore this possibility. allows the testing agency to explore this possibility.
skipping to change at page 28, line 38 skipping to change at page 29, line 16
None. None.
See Also: See Also:
3.4. Benchmarks 3.4. Benchmarks
3.4.1. Registration Rate 3.4.1. Registration Rate
Definition: Definition:
The maximum number of registrations that can be successfully The maximum number of registrations that can be successfully
completed by the DUT/SUT in a given time period. completed by the DUT/SUT in a given time period without
registration failures in that time period.
Discussion: Discussion:
This benchmark is obtained with zero failure in which 100% of the This benchmark is obtained with zero failure in which 100% of the
registrations attempted by the Emulated Agent are successfully registrations attempted by the EA are successfully completed by
completed by the DUT/SUT. The maximum value is obtained by the DUT/SUT. The registration rate provisioned on the Emulated
testing to failure. This means that the registration rate Agent is raised and lowered as described in the algorithm in the
provisioned on the EA is raised progressively until a registration companion methodology draft [I-D.ietf-bmwg-sip-bench-meth] until a
attempt failure is observed. traffic load consisting of registrations at the given attempt rate
over the sustained period of time identified by T in the algorithm
completes without failure.
Measurement Units: Measurement Units:
registrations per second (rps) registrations per second (rps)
Issues: Issues:
None. None.
See Also: See Also:
3.4.2. Session Establishment Rate 3.4.2. Session Establishment Rate
Definition: Definition:
The average maximum rate at which the DUT/SUT can successfully The maximum number of sessions that can be successfully completed
establish sessions. by the DUT/SUT in a given time period without session
establishment failures in that time period.
Discussion: Discussion:
This metric is an average of maxima. Each maximum is measured in This benchmark is obtained with zero failure in which 100% of the
a separate sample. The Session Establishment Rate is the average sessions attempted by the emulated Agent are successfully
of the maximas established in each individual sample. In each completed by the DUT/SUT. The session attempt rate provisioned on
sample, the maximum in question is the number of sessions the EA is raised and lowered as described in the algorithm in the
successfully established in continuous one-second intervals with accompanying methodology document, until a traffic load at the
prior sessions remaining active. This maximum is designated in given attempt rate over the sustained period of time identified by
the equation below as "rate in sample i". The session T in the algorithm completes without any failed session attempts.
establishment rate is calculated using the following equation (n =
number of samples):
n
--
\ rate at sample i
/
--
i = 1
---------------------
(n)
In each sample, the maximum is obtained by testing to failure. Sessions may be IS or NS or a mix of both and will be defined in
With zero failure, 100% of the sessions introduced by the Emulated the particular test.
Agent are successfully established. The maximum value is obtained
by testing to failure. This means that the Session Attempt Rate
provisioned on the EA is raised progressively until a Session
Attempt Failure is observed. The maximum rate is the rate
acheived in the interval prior to the interval in which the
failure is observed. Sessions may be IS or NS or a a mix of both
and will be defined in the particular test.
Measurement Units: Measurement Units:
sessions per second (sps) sessions per second (sps)
Issues: Issues:
None. None.
See Also: See Also:
Invite-initiated Sessions Invite-initiated Sessions
Non-INVITE initiated Sessions Non-INVITE initiated Sessions
Session Attempt Rate Session Attempt Rate
3.4.3. Session Capacity 3.4.3. Session Capacity
Definition: Definition:
The maximum value of Standing Sessions Count achieved by the DUT/ The maximum value of Standing Sessions Count achieved by the DUT/
SUT during the process of steadily increasing the number of SUT during a time period T in which the EA is sending session
Session Attempts per unit time, before the first Session Attempt establishment messages at the Session Establishment Rate.
Failure occurs.
Discussion: Discussion:
When benchmarking Session Capacity for sessions with media it is Sessions may be IS or NS. If they are IS they can be with or
required that these sessions be permanently established (i.e., without media. When benchmarking Session Capacity for sessions
they remain active for the duration of the test.) This can be with media it is required that these sessions be permanently
achieved by causing the EA not to send a BYE for the duration of established (i.e., they remain active for the duration of the
the testing. In the signaling plane, this requirement means that test.) This can be achieved by causing the EA not to send a BYE
the dialog lasts as long as the test lasts. In order to test for the duration of the testing. In the signaling plane, this
Session Capacity for sessions with media, the Media Session Hold requirement means that the dialog lasts as long as the test lasts.
Time MUST be set to infinity so that sessions remain established When media is present, the Media Session Hold Time MUST be set to
for the duration of the test. If the DUT/SUT is dialog-stateful, infinity so that sessions remain established for the duration of
then we expect its performance will be impacted by setting Media the test. If the DUT/SUT is dialog-stateful, then we expect its
Session Hold Time to infinity, since the DUT/SUT will need to performance will be impacted by setting Media Session Hold Time to
allocate resources to process and store the state information. infinity, since the DUT/SUT will need to allocate resources to
The Session Capacity must be reported with the Session Attempt process and store the state information. The report of the
Rate used to reach the maximum. Since Session Attempt Rate is a Session Capacity must include the Session Establishment Rate at
zero-loss measurement, there must be zero failures to achieve the which it was measured.
Session Capacity. The maximum is indicated at the Emulated Agent
by arrival of a SIP 4xx, 5xx, or 6xx response from the DUT/SUT.
Sessions may be IS or NS.
Measurement Units: Measurement Units:
sessions sessions
Issues: Issues:
None. None.
See Also: See Also:
Established Session Established Session
Session Attempt Rate Session Attempt Rate
skipping to change at page 32, line 29 skipping to change at page 32, line 29
Issues: Issues:
None. None.
See Also: See Also:
Established Session Established Session
Session Attempt Session Attempt
3.4.6. Session Attempt Delay 3.4.6. Session Attempt Delay
Definition: Definition:
The average time measured at the Emulated Agent for a Session The average time measured at the EA for a Session Attempt to
Attempt to result in an Established Session. result in an Established Session.
Discussion: Discussion:
Time is measured from when the EA sends the first INVITE for the Time is measured from when the EA sends the first INVITE for the
call-ID in the case of an IS. Time is measured from when the EA call-ID in the case of an IS. Time is measured from when the EA
sends the first non-INVITE message in the case of an NS. Session sends the first non-INVITE message in the case of an NS. Session
Attempt Delay MUST be measured for every established session to Attempt Delay MUST be measured for every established session to
calculate the average. Session Attempt Delay MUST be measured at calculate the average. Session Attempt Delay MUST be measured at
the Maximum Session Establishment Rate. the Maximum Session Establishment Rate.
Measurement Units: Measurement Units:
skipping to change at page 34, line 51 skipping to change at page 34, line 51
Jacobson, "RTP: A Transport Protocol for Real-Time Jacobson, "RTP: A Transport Protocol for Real-Time
Applications", STD 64, RFC 3550, July 2003. Applications", STD 64, RFC 3550, July 2003.
[RFC3711] Baugher, M., McGrew, D., Naslund, M., Carrara, E., and K. [RFC3711] Baugher, M., McGrew, D., Naslund, M., Carrara, E., and K.
Norrman, "The Secure Real-time Transport Protocol (SRTP)", Norrman, "The Secure Real-time Transport Protocol (SRTP)",
RFC 3711, March 2004. RFC 3711, March 2004.
[I-D.ietf-soc-overload-design] [I-D.ietf-soc-overload-design]
Hilt, V., Noel, E., Shen, C., and A. Abdelal, "Design Hilt, V., Noel, E., Shen, C., and A. Abdelal, "Design
Considerations for Session Initiation Protocol (SIP) Considerations for Session Initiation Protocol (SIP)
Overload Control", draft-ietf-soc-overload-design-05 (work Overload Control", draft-ietf-soc-overload-design-08 (work
in progress), March 2011. in progress), July 2011.
[I-D.ietf-soc-overload-control] [I-D.ietf-soc-overload-control]
Gurbani, V., Hilt, V., and H. Schulzrinne, "Session Gurbani, V., Hilt, V., and H. Schulzrinne, "Session
Initiation Protocol (SIP) Overload Control", Initiation Protocol (SIP) Overload Control",
draft-ietf-soc-overload-control-02 (work in progress), draft-ietf-soc-overload-control-07 (work in progress),
February 2011. January 2012.
Appendix A. White Box Benchmarking Terminology Appendix A. White Box Benchmarking Terminology
Session Attempt Arrival Rate Session Attempt Arrival Rate
Definition: Definition:
The number of Session Attempts received at the DUT/SUT over a The number of Session Attempts received at the DUT/SUT over a
specified time period. specified time period.
Discussion: Discussion:
 End of changes. 69 change blocks. 
173 lines changed or deleted 174 lines changed or added

This html diff was produced by rfcdiff 1.41. The latest version is available from http://tools.ietf.org/tools/rfcdiff/