draft-ietf-bmwg-sip-bench-term-10.txt   draft-ietf-bmwg-sip-bench-term-11.txt 
Benchmarking Methodology Working Group C. Davids Benchmarking Methodology Working Group C. Davids
Internet-Draft Illinois Institute of Technology Internet-Draft Illinois Institute of Technology
Intended status: Informational V. Gurbani Intended status: Informational V. Gurbani
Expires: November 29, 2014 Bell Laboratories, Expires: January 3, 2015 Bell Laboratories,
Alcatel-Lucent Alcatel-Lucent
S. Poretsky S. Poretsky
Allot Communications Allot Communications
May 28, 2014 July 2, 2014
Terminology for Benchmarking Session Initiation Protocol (SIP) Devices: Terminology for Benchmarking Session Initiation Protocol (SIP) Devices:
Basic session setup and registration Basic session setup and registration
draft-ietf-bmwg-sip-bench-term-10 draft-ietf-bmwg-sip-bench-term-11
Abstract Abstract
This document provides a terminology for benchmarking the Session This document provides a terminology for benchmarking the Session
Initiation Protocol (SIP) performance of devices. Methodology Initiation Protocol (SIP) performance of devices. Methodology
related to benchmarking SIP devices is described in the companion related to benchmarking SIP devices is described in the companion
methodology document. Using these two documents, benchmarks can be methodology document. Using these two documents, benchmarks can be
obtained and compared for different types of devices such as SIP obtained and compared for different types of devices such as SIP
Proxy Servers, Registrars and Session Border Controllers. The term Proxy Servers, Registrars and Session Border Controllers. The term
"performance" in this context means the capacity of the device-under- "performance" in this context means the capacity of the device-under-
skipping to change at page 2, line 4 skipping to change at page 2, line 4
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/. Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on November 29, 2014. This Internet-Draft will expire on January 3, 2015.
Copyright Notice Copyright Notice
Copyright (c) 2014 IETF Trust and the persons identified as the Copyright (c) 2014 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License. described in the Simplified BSD License.
Table of Contents Table of Contents
1. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4 1. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4
2. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 5 2. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.1. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.1. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . 6
3. Term Definitions . . . . . . . . . . . . . . . . . . . . . . . 7 3. Term Definitions . . . . . . . . . . . . . . . . . . . . . . . 7
3.1. Protocol Components . . . . . . . . . . . . . . . . . . . 7 3.1. Protocol Components . . . . . . . . . . . . . . . . . . . 7
3.1.1. Session . . . . . . . . . . . . . . . . . . . . . . . 7 3.1.1. Session . . . . . . . . . . . . . . . . . . . . . . . 7
3.1.2. Signaling Plane . . . . . . . . . . . . . . . . . . . 10 3.1.2. Signaling Plane . . . . . . . . . . . . . . . . . . . 8
3.1.3. Media Plane . . . . . . . . . . . . . . . . . . . . . 11 3.1.3. Media Plane . . . . . . . . . . . . . . . . . . . . . 8
3.1.4. Associated Media . . . . . . . . . . . . . . . . . . . 11 3.1.4. Associated Media . . . . . . . . . . . . . . . . . . . 9
3.1.5. Overload . . . . . . . . . . . . . . . . . . . . . . . 12 3.1.5. Overload . . . . . . . . . . . . . . . . . . . . . . . 9
3.1.6. Session Attempt . . . . . . . . . . . . . . . . . . . 12 3.1.6. Session Attempt . . . . . . . . . . . . . . . . . . . 9
3.1.7. Established Session . . . . . . . . . . . . . . . . . 13 3.1.7. Established Session . . . . . . . . . . . . . . . . . 10
3.1.8. Invite-Initiated Session (IS) . . . . . . . . . . . . 14 3.1.8. Session Attempt Failure . . . . . . . . . . . . . . . 11
3.1.9. Non-INVITE-Initiated Session (NS) . . . . . . . . . . 14 3.2. Test Components . . . . . . . . . . . . . . . . . . . . . 11
3.1.10. Session Attempt Failure . . . . . . . . . . . . . . . 15 3.2.1. Emulated Agent . . . . . . . . . . . . . . . . . . . . 11
3.2. Test Components . . . . . . . . . . . . . . . . . . . . . 15 3.2.2. Signaling Server . . . . . . . . . . . . . . . . . . . 12
3.2.1. Emulated Agent . . . . . . . . . . . . . . . . . . . . 15 3.2.3. SIP Transport Protocol . . . . . . . . . . . . . . . . 12
3.2.2. Signaling Server . . . . . . . . . . . . . . . . . . . 16 3.3. Test Setup Parameters . . . . . . . . . . . . . . . . . . 13
3.2.3. SIP Transport Protocol . . . . . . . . . . . . . . . . 16 3.3.1. Session Attempt Rate . . . . . . . . . . . . . . . . . 13
3.3. Test Setup Parameters . . . . . . . . . . . . . . . . . . 17 3.3.2. Establishment Threshold Time . . . . . . . . . . . . . 13
3.3.1. Session Attempt Rate . . . . . . . . . . . . . . . . . 17 3.3.3. Session Duration . . . . . . . . . . . . . . . . . . . 14
3.3.2. Establishment Threshold Time . . . . . . . . . . . . . 17 3.3.4. Media Packet Size . . . . . . . . . . . . . . . . . . 14
3.3.3. Session Duration . . . . . . . . . . . . . . . . . . . 18 3.3.5. Codec Type . . . . . . . . . . . . . . . . . . . . . . 15
3.3.4. Media Packet Size . . . . . . . . . . . . . . . . . . 18 3.4. Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . 15
3.3.5. Codec Type . . . . . . . . . . . . . . . . . . . . . . 19 3.4.1. Session Establishment Rate . . . . . . . . . . . . . . 15
3.4. Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . 19 3.4.2. Registration Rate . . . . . . . . . . . . . . . . . . 16
3.4.1. Session Establishment Rate . . . . . . . . . . . . . . 19 3.4.3. Registration Attempt Rate . . . . . . . . . . . . . . 17
3.4.2. Registration Rate . . . . . . . . . . . . . . . . . . 20 4. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 17
3.4.3. Registration Attempt Rate . . . . . . . . . . . . . . 21 5. Security Considerations . . . . . . . . . . . . . . . . . . . 17
4. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 21 6. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 18
5. Security Considerations . . . . . . . . . . . . . . . . . . . 21 7. References . . . . . . . . . . . . . . . . . . . . . . . . . . 18
6. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 22 7.1. Normative References . . . . . . . . . . . . . . . . . . . 18
7. References . . . . . . . . . . . . . . . . . . . . . . . . . . 22 7.2. Informational References . . . . . . . . . . . . . . . . . 19
7.1. Normative References . . . . . . . . . . . . . . . . . . . 22 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 19
7.2. Informational References . . . . . . . . . . . . . . . . . 22
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 23
1. Terminology 1. Terminology
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in BCP 14, RFC2119 document are to be interpreted as described in BCP 14, RFC2119
[RFC2119]. RFC 2119 defines the use of these key words to help make [RFC2119]. RFC 2119 defines the use of these key words to help make
the intent of standards track documents as clear as possible. While the intent of standards track documents as clear as possible. While
this document uses these keywords, this document is not a standards this document uses these keywords, this document is not a standards
track document. The term Throughput is defined in RFC2544 [RFC2544]. track document. The term Throughput is defined in RFC2544 [RFC2544].
skipping to change at page 4, line 33 skipping to change at page 4, line 33
Many commonly used SIP terms in this document are defined in RFC 3261 Many commonly used SIP terms in this document are defined in RFC 3261
[RFC3261]. For convenience the most important of these are [RFC3261]. For convenience the most important of these are
reproduced below. Use of these terms in this document is consistent reproduced below. Use of these terms in this document is consistent
with their corresponding definition in the base SIP specification with their corresponding definition in the base SIP specification
[RFC3261] as amended by [RFC4320], [RFC5393] and [RFC6026]. [RFC3261] as amended by [RFC4320], [RFC5393] and [RFC6026].
o Call Stateful: A proxy is call stateful if it retains state for a o Call Stateful: A proxy is call stateful if it retains state for a
dialog from the initiating INVITE to the terminating BYE request. dialog from the initiating INVITE to the terminating BYE request.
A call stateful proxy is always transaction stateful, but the A call stateful proxy is always transaction stateful, but the
converse is not necessarily true. converse is not necessarily true.
o Stateful Proxy: A logical entity that maintains the client and o Stateful Proxy: A logical entity, as defined by [RFC3261], that
server transaction state machines defined by this specification maintains the client and server transaction state machines during
during the processing of a request, also known as a transaction the processing of a request. (Also known as a transaction
stateful proxy. The behavior of a stateful proxy is further stateful proxy.) The behavior of a stateful proxy is further
defined in Section 16 of RFC 3261 [RFC3261] . A transaction defined in Section 16 of RFC 3261 [RFC3261] . A transaction
stateful proxy is not the same as a call stateful proxy. stateful proxy is not the same as a call stateful proxy.
o Stateless Proxy: A logical entity that does not maintain the
client or server transaction state machines defined in this
specification when it processes requests. A stateless proxy
forwards every request it receives downstream and every response
it receives upstream.
o Back-to-back User Agent: A back-to-back user agent (B2BUA) is a o Back-to-back User Agent: A back-to-back user agent (B2BUA) is a
logical entity that receives a request and processes it as a user logical entity that receives a request and processes it as a user
agent server (UAS). In order to determine how the request should agent server (UAS). In order to determine how the request should
be answered, it acts as a user agent client (UAC) and generates be answered, it acts as a user agent client (UAC) and generates
requests. Unlike a proxy server, it maintains dialog state and requests. Unlike a proxy server, it maintains dialog state and
must participate in all requests sent on the dialogues it has must participate in all requests sent on the dialogues it has
established. Since it is a concatenation of a UAC and a UAS, no established. Since it is a concatenation of a UAC and a UAS, no
explicit definitions are needed for its behavior. explicit definitions are needed for its behavior.
2. Introduction 2. Introduction
skipping to change at page 6, line 17 skipping to change at page 6, line 13
defined in this document are intended for use with the companion defined in this document are intended for use with the companion
Methodology document. Methodology document.
2.1. Scope 2.1. Scope
The scope of this document is summarized as follows: The scope of this document is summarized as follows:
o This terminology document describes SIP signaling performance o This terminology document describes SIP signaling performance
benchmarks for black-box measurements of SIP networking devices. benchmarks for black-box measurements of SIP networking devices.
Stress and debug scenarios are not addressed in this document. Stress and debug scenarios are not addressed in this document.
o The DUT must be RFC 3261 capable network equipment. This may be a o The DUT must be RFC 3261 capable network equipment. This may be a
Registrar, Redirect Server, Stateless Proxy or Stateful Proxy. A Registrar, Redirect Server, or Stateful Proxy. This document does
DUT MAY also include a B2BUA, SBC functionality. not require the intermediary to assume the role of a stateless
o The DUT MUST NOT be end user equipment, such as personal digital proxy. A DUT MAY also include a B2BUA, SBC functionality.
assistant, a computer-based client, or a user terminal.
o The Tester acts as multiple "Emulated Agents" (EA) that initiate o The Tester acts as multiple "Emulated Agents" (EA) that initiate
(or respond to) SIP messages as session endpoints and source (or (or respond to) SIP messages as session endpoints and source (or
receive) associated media for established connections. receive) associated media for established connections.
o SIP Signaling in presence of media o SIP Signaling in presence of media
* The media performance is not benchmarked. * The media performance is not benchmarked.
* Some tests require media, but the use of media is limited to * Some tests require media, but the use of media is limited to
observing the performance of SIP signaling. Tests that require observing the performance of SIP signaling. Tests that require
media will annotate the media characteristics as a condition of media will annotate the media characteristics as a condition of
test. test.
* The type of DUT dictates whether the associated media streams * The type of DUT dictates whether the associated media streams
skipping to change at page 7, line 34 skipping to change at page 7, line 29
o IMS-specific scenarios are not considered, but test cases can be o IMS-specific scenarios are not considered, but test cases can be
applied with 3GPP-specific SIP signaling and the P-CSCF as a DUT. applied with 3GPP-specific SIP signaling and the P-CSCF as a DUT.
3. Term Definitions 3. Term Definitions
3.1. Protocol Components 3.1. Protocol Components
3.1.1. Session 3.1.1. Session
Definition: Definition:
The combination of signaling and media messages and processes that The combination of signaling and media messages and associated
support a SIP-based service. processing that enable a single SIP-based audio or video call, or
SIP registration.
Discussion: Discussion:
SIP messages are used to create and manage services for end users. The term "session" commonly implies a media session. In this
Often, these services include the creation of media streams that document the term is extended to cover the signaling and any media
are defined in the SDP body of a SIP message and carried in RTP specified and invoked by the corresponding signaling.
protocol data units. However, SIP messages can also be used to
create instant message services and subscription services, and
such services are not associated with media streams. SIP reserves
the term "session" to describe services that are analogous to
telephone calls on a circuit switched network. SIP reserves the
term "dialog" to refer to a signaling-only relationship between
User Agent peers. SIP reserves the term "transaction" to refer to
the brief communication between a client and a server that lasts
only until the final response to the SIP request. None of these
terms describes the entity whose performance we want to benchmark.
For example, the MESSAGE request does not create a dialog and can
be sent either within or outside of a dialog. It is not
associated with media, but it resembles a phone call in its
dependence on human rather than machine initiated responses. The
SUBSCRIBE method does create a dialog between the originating end-
user and the subscription service. It, too, is not associated
with a media session.
In light of the above observations we have extended the term
"session" to include SIP-based services that are not initiated by
INVITE requests and that do not have associated media. In this
extended definition, a session always has a signaling component
and may also have a media component. Thus, a session can be
defined as signaling-only or a combination of signaling and media.
We define the term "Associated Media", see Section 3.1.4, to
describe the situation in which media is associated with a SIP
dialog. The terminology "Invite-Initiated Session" (IS)
Section 3.1.8 and "Non-invite-Initiated Session" (NS)
Section 3.1.9 are used to distinguish between these two types of
session. An Invite-Initiated Session is a session as defined in
SIP. The performance of a device or system that supports Invite-
Initiated Sessions that do not create media sessions, "Invite-
Initiated Sessions without Associated Media", can be measured and
is of interest for comparison and as a limiting case. The
REGISTER request can be considered to be a "Non-invite-Initiated
Session without Associated Media." A separate set of benchmarks
is provided for REGISTER requests since most implementations of
SIP-based services require this request and since a registrar may
be a device under test.
A Session in the context of this document, can be considered to be
a vector with three components:
1. A component in the signaling plane (SIP messages), sess.sig;
2. A media component in the media plane (RTP and SRTP streams for
example), sess.med (which may be null);
3. A control component in the media plane (RTCP messages for
example), sess.medc (which may be null).
An IS is expected to have non-null sess.sig and sess.med
components. The use of control protocols in the media component
is media dependent, thus the expected presence or absence of
sess.medc is media dependent and test-case dependent. An NS is
expected to have a non-null sess.sig component, but null sess.med
and sess.medc components.
Packets in the Signaling Plane and Media Plane will be handled by
different processes within the DUT. They will take different
paths within a network. These different processes and paths may
produce variations in performance. The terminology and benchmarks
defined in this document and the methodology for their use are
designed to enable us to compare performance of the DUT with
reference to the type of SIP-supported application it is handling.
Note that one or more sessions can simultaneously exist between
any participants. This can be the case, for example, when the EA
sets up both an IM and a voice call through the DUT. These
sessions are represented as an array session[x].
Sessions will be represented as a vector array with three
components, as follows:
session->
session[x].sig, the signaling component
session[x].medc[y], the media control component (e.g. RTCP)
session[x].med[y], an array of associated media streams (e.g.
RTP, SRTP, RTSP, MSRP). This media component may consist of zero
or more media streams.
Figure 1 models the vectors of the session.
Measurement Units: Measurement Units:
N/A. N/A.
Issues: Issues:
None. None.
See Also: See Also:
Media Plane Media Plane
Signaling Plane Signaling Plane
Associated Media Associated Media
Invite-Initiated Session (IS)
Non-Invite-Initiated Session (NS)
|\
|
| \
sess.sig|
| \
|
| \
| o
| /
| / |
| /
| / |
| /
| / |
| /
| / | sess.medc
|/_____________________
/ /
/ |
/ /
sess.med / |
/_ _ _ _ _ _ _ _/
/
/
/
/
Figure 1: Session components
3.1.2. Signaling Plane 3.1.2. Signaling Plane
Definition: Definition:
The plane in which SIP messages [RFC3261] are exchanged between The plane in which SIP messages [RFC3261] are exchanged between
SIP Agents [RFC3261]. SIP Agents [RFC3261].
Discussion: Discussion:
SIP messages are used to establish sessions in several ways: SIP messages are used to establish sessions in several ways:
directly between two User Agents [RFC3261], through a Proxy Server directly between two User Agents [RFC3261], through a Proxy Server
[RFC3261], or through a series of Proxy Servers. The Session [RFC3261], or through a series of Proxy Servers. The Session
Description Protocol (SDP) is included in the Signaling Plane. Description Protocol (SDP) is included in the Signaling Plane.
The Signaling Plane for a single Session is represented by
session.sig.
Measurement Units: Measurement Units:
N/A. N/A.
Issues: Issues:
None. None.
See Also: See Also:
Media Plane Media Plane
EAs EAs
skipping to change at page 11, line 28 skipping to change at page 8, line 40
The data plane in which one or more media streams and their The data plane in which one or more media streams and their
associated media control protocols are exchanged between User associated media control protocols are exchanged between User
Agents after a media connection has been created by the exchange Agents after a media connection has been created by the exchange
of signaling messages in the Signaling Plane. of signaling messages in the Signaling Plane.
Discussion: Discussion:
Media may also be known as the "bearer channel". The Media Plane Media may also be known as the "bearer channel". The Media Plane
MUST include the media control protocol, if one is used, and the MUST include the media control protocol, if one is used, and the
media stream(s). Examples of media are audio and video. The media stream(s). Examples of media are audio and video. The
media streams are described in the SDP of the Signaling Plane. media streams are described in the SDP of the Signaling Plane.
The media for a single Session is represented by session.med. The
media control protocol for a single media description is
represented by session.medc.
Measurement Units: Measurement Units:
N/A. N/A.
Issues: Issues:
None. None.
See Also: See Also:
Signaling Plane Signaling Plane
skipping to change at page 12, line 4 skipping to change at page 9, line 12
See Also: See Also:
Signaling Plane Signaling Plane
3.1.4. Associated Media 3.1.4. Associated Media
Definition: Definition:
Media that corresponds to an 'm' line in the SDP payload of the Media that corresponds to an 'm' line in the SDP payload of the
Signaling Plane. Signaling Plane.
Discussion: Discussion:
Any media protocol MAY be used. Any media protocol MAY be used.
For any session's signaling component, session.sig, there may be
zero, one, or multiple associated media streams. When there are
multiple media streams, these are represented be a vector array
session.med[y]. When there are multiple media streams there will
be multiple media control protocol descriptions as well. They are
represented by a vector array session.medc[y].
Measurement Units: Measurement Units:
N/A. N/A.
Issues: Issues:
None. None.
3.1.5. Overload 3.1.5. Overload
Definition: Definition:
skipping to change at page 12, line 46 skipping to change at page 10, line 4
output queues, or disk resources. Any combination of resources output queues, or disk resources. Any combination of resources
may be fully utilized when a SIP server (the DUT) is in the may be fully utilized when a SIP server (the DUT) is in the
overload condition. For proxy-only (or intermediary) devices, it overload condition. For proxy-only (or intermediary) devices, it
is expected that the proxy will be driven into overload based on is expected that the proxy will be driven into overload based on
the delivery rate of signaling requests. the delivery rate of signaling requests.
Measurement Units: Measurement Units:
N/A. N/A.
3.1.6. Session Attempt 3.1.6. Session Attempt
Definition: Definition:
A SIP INVITE or REGISTER request sent by the EA that has not A SIP INVITE or REGISTER request sent by the EA that has not
received a final response. received a final response.
Discussion: Discussion:
The attempted session may be Invite Initiated or Non-invite- The attempted session may be either an invitation to an audio/
Initiated. When counting the number of session attempts we video communication or a registration attempt. When counting the
include all INVITEs that are rejected for lack of authentication number of session attempts we include all requests that are
information. The EA needs to record the total number of session rejected for lack of authentication information. The EA needs to
attempts including those attempts that are routinely rejected by a record the total number of session attempts including those
proxy that requires the UA to authenticate itself. The EA is attempts that are routinely rejected by a proxy that requires the
provisioned to deliver a specific number of session attempts per UA to authenticate itself. The EA is provisioned to deliver a
second. But the EA must also count the actual number of session specific number of session attempts per second. But the EA must
attempts per given tie interval. also count the actual number of session attempts per given time
interval.
Measurement Units: Measurement Units:
N/A. N/A.
Issues: Issues:
None. None.
See Also: See Also:
Session Session
Session Attempt Rate Session Attempt Rate
Invite-Initiated Session
Non-Invite-Initiated Session
3.1.7. Established Session 3.1.7. Established Session
Definition: Definition:
A SIP session for which the EA acting as the UE/UA has received a A SIP session for which the EA acting as the UE/UA has received a
200 OK message. 200 OK message.
Discussion: Discussion:
An Established Session MAY be Invite Initiated or Non-Invite- An Established Session may be either an invitation to an audio/
Initiated. Early dialogues for INVITE requests are out of scope video communication or a registration attempt. Early dialogues
for this work. for INVITE requests are out of scope for this work.
Measurement Units:
N/A.
Issues:
None.
See Also:
Invite-Initiated Session
3.1.8. Invite-Initiated Session (IS)
Definition:
A Session that is created by an exchange of messages in the
Signaling Plane, the first of which is a SIP INVITE request.
Discussion:
When an IS becomes an Established Session its signaling component
is identified by the SIP dialog parameter values, Call-ID, To-tag,
and From-tag (RFC3261 [RFC3261]). An IS may have zero, one or
multiple Associated Media descriptions in the SDP body. The
inclusion of media is test case dependent. An IS is successfully
established if the following two conditions are met:
1. Sess.sig is established by the end of Establishment Threshold
Time (c.f. Section 3.3.2), and
2. If the DUT is a device, such as an SBC or B2BUA, that may
receive media from a calling or called party before a
signaling dialog is established or confirmed , this fact needs
to be taken into account when the EA is built. Any additional
parameters that need to be configured in the EA must be added
to the list of test setup parameters in Section 5.1 of
[I-D.ietf-bmwg-sip-bench-meth]
Measurement Units: Measurement Units:
N/A. N/A.
Issues: Issues:
None. None.
See Also: See Also:
Session
Non-Invite-Initiated Session
Associated Media
3.1.9. Non-INVITE-Initiated Session (NS)
Definition:
A session that is created by an exchange of SIP messages in the
Signaling Plane the first of which is not a SIP INVITE message.
Discussion:
An NS is successfully established if the Session Attempt via a
non-INVITE request results in the EA receiving a 2xx reply before
the expiration of the Establishment Threshold timer (c.f.,
Section 3.3.2). For the purpose of this document, a NS is a
session created only by the REGISTER request and no other request.
Measurement Units:
N/A.
Issues:
None. None.
See Also: 3.1.8. Session Attempt Failure
Session
Invite-Initiated Session
3.1.10. Session Attempt Failure
Definition: Definition:
A session attempt that does not result in an Established Session. A session attempt that does not result in an Established Session.
Discussion: Discussion:
The session attempt failure may be indicated by the following The session attempt failure may be indicated by the following
observations at the EA: observations at the EA:
1. Receipt of a SIP 3xx-, 4xx-, 5xx-, or 6xx-class response to a 1. Receipt of a SIP 3xx-, 4xx-, 5xx-, or 6xx-class response to a
Session Attempt. Session Attempt.
2. The lack of any received SIP response to a Session Attempt 2. The lack of any received SIP response to a Session Attempt
skipping to change at page 16, line 29 skipping to change at page 12, line 19
Associated Media Associated Media
3.2.2. Signaling Server 3.2.2. Signaling Server
Definition: Definition:
Device in the test topology that facilitates the creation of Device in the test topology that facilitates the creation of
sessions between EAs. This device is the DUT. sessions between EAs. This device is the DUT.
Discussion: Discussion:
The DUT is a RFC3261-capable network intermediary such as a The DUT is a RFC3261-capable network intermediary such as a
Registrar, Redirect Server, User Agent Server, Stateless Proxy, Registrar, Redirect Server, Stateful Proxy, B2BUA or SBC.
Stateful Proxy, B2BUA or SBC.
Measurement Units: Measurement Units:
NA NA
Issues: Issues:
None. None.
See Also: See Also:
Signaling Plane Signaling Plane
skipping to change at page 17, line 17 skipping to change at page 13, line 9
While these are not units of measure, they are attributes that are While these are not units of measure, they are attributes that are
one of many factors that will contribute to the value of the one of many factors that will contribute to the value of the
measurements to be taken. TCP, UDP, SCTP, TLS over TCP, TLS over measurements to be taken. TCP, UDP, SCTP, TLS over TCP, TLS over
UDP, TLS over SCTP, and websockets are among the possible values UDP, TLS over SCTP, and websockets are among the possible values
to be recorded as part of the test. to be recorded as part of the test.
Issues: Issues:
None. None.
See Also: See Also:
None.
3.3. Test Setup Parameters 3.3. Test Setup Parameters
3.3.1. Session Attempt Rate 3.3.1. Session Attempt Rate
Definition: Definition:
Configuration of the EA for the number of sessions per second Configuration of the EA for the number of sessions per second
(sps) that the EA attempts to establish using the services of the (sps) that the EA attempts to establish using the services of the
DUT. DUT.
Discussion: Discussion:
The Session Attempt Rate is the number of sessions per second that The Session Attempt Rate is the number of sessions per second that
the EA sends toward the DUT. Some of the sessions attempted may the EA sends toward the DUT. Some of the sessions attempted may
not result in a session being established. A session in this case not result in a session being established.
may be either an IS or an NS.
Measurement Units: Measurement Units:
Session attempts per second Session attempts per second
Issues: Issues:
None. None.
See Also: See Also:
Session Session
Session Attempt Session Attempt
skipping to change at page 18, line 9 skipping to change at page 13, line 46
Definition: Definition:
Configuration of the EA that represents the amount of time that an Configuration of the EA that represents the amount of time that an
EA client will wait for a response from an EA server before EA client will wait for a response from an EA server before
declaring a Session Attempt Failure. declaring a Session Attempt Failure.
Discussion: Discussion:
This time duration is test dependent. This time duration is test dependent.
It is RECOMMENDED that the Establishment Threshold Time value be It is RECOMMENDED that the Establishment Threshold Time value be
set to Timer B (for ISs) or Timer F (for NSs) as specified in RFC set to Timer B or Timer F as specified in RFC 3261, Table 4
3261, Table 4 [RFC3261]. Following the default value of T1 [RFC3261]. Following the default value of T1 (500ms) specified in
(500ms) specified in the table and a constant multiplier of 64 the table and a constant multiplier of 64 gives a value of 32
gives a value of 32 seconds for this timer (i.e., 500ms * 64 = seconds for this timer (i.e., 500ms * 64 = 32s).
32s).
Measurement Units: Measurement Units:
Seconds Seconds
Issues: Issues:
None. None.
See Also:
None.
3.3.3. Session Duration 3.3.3. Session Duration
Definition: Definition:
Configuration of the EA that represents the amount of time that Configuration of the EA that represents the amount of time that
the SIP dialog is intended to exist between the two EAs associated the SIP dialog is intended to exist between the two EAs associated
with the test. with the test.
Discussion: Discussion:
The time at which the BYE is sent will control the Session The time at which the BYE is sent will control the Session
Duration. Duration.
Measurement Units: Measurement Units:
seconds seconds
Issues: Issues:
None. None.
See Also: See Also:
None.
3.3.4. Media Packet Size 3.3.4. Media Packet Size
Definition: Definition:
Configuration on the EA for a fixed number of frames or samples to Configuration on the EA for a fixed number of frames or samples to
be sent in each RTP packet of the media session. be sent in each RTP packet of the media stream when the test
involves Associated Media.
Discussion: Discussion:
This document describes a method to measure SIP performance. If
the DUT is processing media as well as SIP messages the media
processing will potentially slow down the SIP processing and lower
the SIP performance metric. The tests with associated media are
designed for audio codecs and the assumption was made that larger
media packets would require more processor time. This document
does not define parameters applicable to video codecs.
For a single benchmark test, media sessions use a defined number For a single benchmark test, media sessions use a defined number
of samples or frames per RTP packet. If two SBCs, for example, of samples or frames per RTP packet. If two SBCs, for example,
used the same codec but one puts more frames into the RTP packet, used the same codec but one puts more frames into the RTP packet,
this might cause variation in the performance benchmark results. this might cause variation in the performance benchmark results.
Measurement Units: Measurement Units:
An integer number of frames or samples, depending on whether An integer number of frames or samples, depending on whether
hybrid- or sample-based codec are used, respectively. hybrid- or sample-based codec are used, respectively.
Issues: Issues:
None. None.
See Also: See Also:
None.
3.3.5. Codec Type 3.3.5. Codec Type
Definition: Definition:
The name of the codec used to generate the media session. The name of the codec used to generate the media session.
Discussion Discussion
For a single benchmark text, all sessions use the same size packet For a single benchmark test, all sessions use the same size packet
for media streams. The size of packets can cause a variation in for media streams. The size of packets can cause a variation in
the performance benchmark measurements. the performance benchmark measurements.
Measurement Units: Measurement Units:
This is a textual name (alphanumeric) assigned to uniquely This is a textual name (alphanumeric) assigned to uniquely
identify the codec. identify the codec.
Issues: Issues:
None. None.
See Also: See Also:
None.
3.4. Benchmarks 3.4. Benchmarks
3.4.1. Session Establishment Rate 3.4.1. Session Establishment Rate
Definition: Definition:
The maximum value of the Session Attempt Rate that the DUT can The maximum value of the Session Attempt Rate that the DUT can
handle for an extended, pre-defined, period with zero failures. handle for an extended, pre-defined, period with zero failures.
Discussion: Discussion:
skipping to change at page 19, line 46 skipping to change at page 16, line 4
3.4. Benchmarks 3.4. Benchmarks
3.4.1. Session Establishment Rate 3.4.1. Session Establishment Rate
Definition: Definition:
The maximum value of the Session Attempt Rate that the DUT can The maximum value of the Session Attempt Rate that the DUT can
handle for an extended, pre-defined, period with zero failures. handle for an extended, pre-defined, period with zero failures.
Discussion: Discussion:
This benchmark is obtained with zero failure. The session attempt This benchmark is obtained with zero failure. The session attempt
rate provisioned on the EA is raised and lowered as described in rate provisioned on the EA is raised and lowered as described in
the algorithm in the accompanying methodology document the algorithm in the accompanying methodology document
[I-D.ietf-bmwg-sip-bench-meth], until a traffic load over the [I-D.ietf-bmwg-sip-bench-meth], until a traffic load over the
period of time necessary to attempt N sessions completes without period of time necessary to attempt N sessions completes without
failure, where N is a parameter specified in the algorithm and failure, where N is a parameter specified in the algorithm and
recorded in the Test Setup Report. Sessions may be IS or NS or a recorded in the Test Setup Report.
mix of both and will be defined in the particular test.
Measurement Units: Measurement Units:
sessions per second (sps) sessions per second (sps)
Issues: Issues:
None. None.
See Also: See Also:
Invite-Initiated Sessions Invite-Initiated Sessions
Non-Invite-Initiated Sessions Non-Invite-Initiated Sessions
skipping to change at page 20, line 47 skipping to change at page 17, line 4
request is a request for a Non-Invite-Initiated session. It is request is a request for a Non-Invite-Initiated session. It is
defined separately because it is a very important benchmark for defined separately because it is a very important benchmark for
most SIP installations. An example demonstrating its use is an most SIP installations. An example demonstrating its use is an
avalanche restart, where hundreds of thousands of end points avalanche restart, where hundreds of thousands of end points
register simultaneously following a power outage. In such a case, register simultaneously following a power outage. In such a case,
an authoritative measurement of the capacity of the device to an authoritative measurement of the capacity of the device to
register endpoints is useful to the network designer. register endpoints is useful to the network designer.
Additionally, in certain controlled networks, there appears to be Additionally, in certain controlled networks, there appears to be
a difference between the registration rate of new endpoints and a difference between the registration rate of new endpoints and
the registering rate of existing endpoints (register refreshes). the registering rate of existing endpoints (register refreshes).
This benchmark can capture these differences as well. This benchmark can capture these differences as well.
Measurement Units: Measurement Units:
registrations per second (rps) registrations per second (rps)
Issues: Issues:
None. None.
See Also: See Also:
None.
3.4.3. Registration Attempt Rate 3.4.3. Registration Attempt Rate
Definition: Definition:
Configuration of the EA for the number of registrations per second Configuration of the EA for the number of registrations per second
that the EA attempts to send to the DUT. that the EA attempts to send to the DUT.
Discussion: Discussion:
The Registration Attempt Rate is the number of registration The Registration Attempt Rate is the number of registration
requests per second that the EA sends toward the DUT. requests per second that the EA sends toward the DUT.
skipping to change at page 21, line 41 skipping to change at page 17, line 44
4. IANA Considerations 4. IANA Considerations
This document requires no IANA considerations. This document requires no IANA considerations.
5. Security Considerations 5. Security Considerations
Documents of this type do not directly affect the security of Documents of this type do not directly affect the security of
Internet or corporate networks as long as benchmarking is not Internet or corporate networks as long as benchmarking is not
performed on devices or systems connected to production networks. performed on devices or systems connected to production networks.
Security threats and how to counter these in SIP and the media layer Security threats and how to counter these in SIP and the media layer
is discussed in RFC3261 [RFC3261], RFC 3550 [RFC3550], RFC3711 is discussed in RFC3261 [RFC3261], RFC 3550 [RFC3550] and RFC3711
[RFC3711] and various other drafts. This document attempts to [RFC3711]. This document attempts to formalize a set of common
formalize a set of common terminology for benchmarking SIP networks. terminology for benchmarking SIP networks. Packets with unintended
Packets with unintended and/or unauthorized DSCP or IP precedence and/or unauthorized DSCP or IP precedence values may present security
values may present security issues. Determining the security issues. Determining the security consequences of such packets is out
consequences of such packets is out of scope for this document. of scope for this document.
6. Acknowledgments 6. Acknowledgments
The authors would like to thank Keith Drage, Cullen Jennings, Daryl The authors would like to thank Keith Drage, Cullen Jennings, Daryl
Malas, Al Morton, and Henning Schulzrinne for invaluable Malas, Al Morton, and Henning Schulzrinne for invaluable
contributions to this document. Dale Worley provided an extensive contributions to this document. Dale Worley provided an extensive
review that lead to improvements in the documents. We are grateful review that lead to improvements in the documents. We are grateful
to Barry Constantine for providing valuable comments during the to Barry Constantine, William Cerveny and Robert Sparks for providing
document's WGLC. valuable comments during the document's last calls and expert
reviews.
7. References 7. References
7.1. Normative References 7.1. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997. Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for
Network Interconnect Devices", RFC 2544, March 1999. Network Interconnect Devices", RFC 2544, March 1999.
 End of changes. 41 change blocks. 
268 lines changed or deleted 96 lines changed or added

This html diff was produced by rfcdiff 1.41. The latest version is available from http://tools.ietf.org/tools/rfcdiff/