Benchmarking Methodology Working Group                         C. Davids
Internet-Draft                          Illinois Institute of Technology
Intended status: Informational                                V. Gurbani
Expires: August 18, November 29, 2014                            Bell Laboratories,
                                                             S. Poretsky
                                                    Allot Communications
                                                       February 14,
                                                            May 28, 2014

Terminology for Benchmarking Session Initiation Protocol (SIP)  Devices:
                  Basic session setup and registration


   This document provides a terminology for benchmarking the Session
   Initiation Protocol (SIP) performance of devices.  Methodology
   related to benchmarking SIP devices is described in the companion
   methodology document.  Using these two documents, benchmarks can be
   obtained and compared for different types of devices such as SIP
   Proxy Servers, Registrars and Session Border Controllers.  The term
   "performance" in this context means the capacity of the device-under-
   test (DUT) to process SIP messages.  Media streams are used only to
   study how they impact the signaling behavior.  The intent of the two
   documents is to provide a normalized set of tests that will enable an
   objective comparison of the capacity of SIP devices.  Test setup
   parameters and a methodology is necessary because SIP allows a wide
   range of configuration and operational conditions that can influence
   performance benchmark measurements.  A standard terminology and
   methodology will ensure that benchmarks have consistent definition
   and were obtained following the same procedures.

Status of this Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."
   This Internet-Draft will expire on August 18, November 29, 2014.

Copyright Notice

   Copyright (c) 2014 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   ( in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

Table of Contents

   1.  Terminology  . . . . . . . . . . . . . . . . . . . . . . . . .  4
   2.  Introduction . . . . . . . . . . . . . . . . . . . . . . . . .  5
     2.1.  Scope  . . . . . . . . . . . . . . . . . . . . . . . . . .  6
   3.  Term Definitions . . . . . . . . . . . . . . . . . . . . . . .  7
     3.1.  Protocol Components  . . . . . . . . . . . . . . . . . . .  7
       3.1.1.  Session  . . . . . . . . . . . . . . . . . . . . . . .  7
       3.1.2.  Signaling Plane  . . . . . . . . . . . . . . . . . . . 10
       3.1.3.  Media Plane  . . . . . . . . . . . . . . . . . . . . . 11
       3.1.4.  Associated Media . . . . . . . . . . . . . . . . . . . 11
       3.1.5.  Overload . . . . . . . . . . . . . . . . . . . . . . . 12
       3.1.6.  Session Attempt  . . . . . . . . . . . . . . . . . . . 12
       3.1.7.  Established Session  . . . . . . . . . . . . . . . . . 13
       3.1.8.  Invite-Initiated Session (IS)  . . . . . . . . . . . . 14
       3.1.9.  Non-INVITE-Initiated Session (NS)  . . . . . . . . . . 14
       3.1.10. Session Attempt Failure  . . . . . . . . . . . . . . . 15
     3.2.  Test Components  . . . . . . . . . . . . . . . . . . . . . 15
       3.2.1.  Emulated Agent . . . . . . . . . . . . . . . . . . . . 15
       3.2.2.  Signaling Server . . . . . . . . . . . . . . . . . . . 16
       3.2.3.  SIP Transport Protocol . . . . . . . . . . . . . . . . 16
     3.3.  Test Setup Parameters  . . . . . . . . . . . . . . . . . . 17
       3.3.1.  Session Attempt Rate . . . . . . . . . . . . . . . . . 17
       3.3.2.  Establishment Threshold Time . . . . . . . . . . . . . 17
       3.3.3.  Session Duration . . . . . . . . . . . . . . . . . . . 18
       3.3.4.  Media Packet Size  . . . . . . . . . . . . . . . . . . 18
       3.3.5.  Codec Type . . . . . . . . . . . . . . . . . . . . . . 19
     3.4.  Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . 19
       3.4.1.  Session Establishment Rate . . . . . . . . . . . . . . 19
       3.4.2.  Registration Rate  . . . . . . . . . . . . . . . . . . 20
       3.4.3.  Registration Attempt Rate  . . . . . . . . . . . . . . 21
   4.  IANA Considerations  . . . . . . . . . . . . . . . . . . . . . 21
   5.  Security Considerations  . . . . . . . . . . . . . . . . . . . 21
   6.  Acknowledgments  . . . . . . . . . . . . . . . . . . . . . . . 22
   7.  References . . . . . . . . . . . . . . . . . . . . . . . . . . 22
     7.1.  Normative References . . . . . . . . . . . . . . . . . . . 22
     7.2.  Informational References . . . . . . . . . . . . . . . . . 22
   Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 23

1.  Terminology

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   document are to be interpreted as described in BCP 14, RFC2119
   [RFC2119].  RFC 2119 defines the use of these key words to help make
   the intent of standards track documents as clear as possible.  While
   this document uses these keywords, this document is not a standards
   track document.  The term Throughput is defined in RFC2544 [RFC2544].

   For the sake of clarity and continuity, this document adopts the
   template for definitions set out in Section 2 of RFC 1242 [RFC1242].

   The term Device Under Test (DUT) is defined in the following BMWG

      Device Under Test (DUT) (c.f., Section 3.1.1 RFC 2285 [RFC2285]).

   Many commonly used SIP terms in this document are defined in RFC 3261
   [RFC3261].  For convenience the most important of these are
   reproduced below.  Use of these terms in this document is consistent
   with their corresponding definition in the base SIP specification
   [RFC3261] as amended by [RFC4320], [RFC5393] and [RFC6026].

   o  Call Stateful: A proxy is call stateful if it retains state for a
      dialog from the initiating INVITE to the terminating BYE request.
      A call stateful proxy is always transaction stateful, but the
      converse is not necessarily true.
   o  Stateful Proxy: A logical entity that maintains the client and
      server transaction state machines defined by this specification
      during the processing of a request, also known as a transaction
      stateful proxy.  The behavior of a stateful proxy is further
      defined in Section 16 of RFC 3261 [RFC3261] .  A transaction
      stateful proxy is not the same as a call stateful proxy.
   o  Stateless Proxy: A logical entity that does not maintain the
      client or server transaction state machines defined in this
      specification when it processes requests.  A stateless proxy
      forwards every request it receives downstream and every response
      it receives upstream.
   o  Back-to-back User Agent: A back-to-back user agent (B2BUA) is a
      logical entity that receives a request and processes it as a user
      agent server (UAS).  In order to determine how the request should
      be answered, it acts as a user agent client (UAC) and generates
      requests.  Unlike a proxy server, it maintains dialog state and
      must participate in all requests sent on the dialogues it has
      established.  Since it is a concatenation of a UAC and a UAS, no
      explicit definitions are needed for its behavior.

2.  Introduction

   Service Providers and IT Organizations deliver Voice Over IP (VoIP)
   and Multimedia network services based on the IETF Session Initiation
   Protocol (SIP) [RFC3261].  SIP is a signaling protocol originally
   intended to be used to dynamically establish, disconnect and modify
   streams of media between end users.  As it has evolved it has been
   adopted for use in a growing number of services and applications.
   Many of these result in the creation of a media session, but some do
   not.  Examples of this latter group include text messaging and
   subscription services.  The set of benchmarking terms provided in
   this document is intended for use with any SIP-enabled device
   performing SIP functions in the interior of the network, whether or
   not these result in the creation of media sessions.  The performance
   of end-user devices is outside the scope of this document.

   A number of networking devices have been developed to support SIP-
   based VoIP services.  These include SIP Servers, Session Border
   Controllers (SBC) and Back-to-back User Agents (B2BUA).  These
   devices contain a mix of voice and IP functions whose performance may
   be reported using metrics defined by the equipment manufacturer or
   vendor.  The Service Provider or IT Organization seeking to compare
   the performance of such devices will not be able to do so using these
   vendor-specific metrics, whose conditions of test and algorithms for
   collection are often unspecified.

   SIP functional elements and the devices that include them can be
   configured many different ways and can be organized into various
   topologies.  These configuration and topological choices impact the
   value of any chosen signaling benchmark.  Unless these conditions-of-test conditions-of-
   test are defined, a true comparison of performance metrics across
   multiple vendor implementations will not be possible.

   Some SIP-enabled devices terminate or relay media as well as
   signaling.  The processing of media by the device impacts the
   signaling performance.  As a result, the conditions-of-test must
   include information as to whether or not the device under test
   processes media and if media.  If the device does process
   media, processes media during the test, a
   description of the media handled and the manner in which it
   is handled. must be provided.  This document and its
   companion methodology document [I-D.ietf-bmwg-sip-bench-meth] provide
   a set of black-box benchmarks for describing and comparing the
   performance of devices that incorporate the SIP User Agent Client and
   Server functions and that operate in the network's core.

   The definition of SIP performance benchmarks necessarily includes
   definitions of Test Setup Parameters and a test methodology.  These
   enable the Tester to perform benchmarking tests on different devices
   and to achieve comparable results.  This document provides a common
   set of definitions for Test Components, Test Setup Parameters, and
   Benchmarks.  All the benchmarks defined are black-box measurements of
   the SIP signaling plane.  The Test Setup Parameters and Benchmarks
   defined in this document are intended for use with the companion
   Methodology document.

2.1.  Scope

   The scope of this work item document is summarized as follows:
   o  This terminology document describes SIP signaling performance
      benchmarks for black-box measurements of SIP networking devices.
      Stress and debug scenarios are not addressed in this work item. document.
   o  The DUT must be an RFC 3261 capable network equipment.  This may be a
      Registrar, Redirect Server, Stateless Proxy or Stateful Proxy.  A
      DUT MAY also include a B2BUA, SBC functionality.
   o  The DUT MUST NOT be end user equipment, such as personal digital
      assistant, a computer-based client, or a user terminal.
   o  The Tester acts as multiple "Emulated Agents" (EA) that initiate
      (or respond to) SIP messages as session endpoints and source (or
      receive) associated media for established connections.
   o  SIP Signaling in presence of Media media
      *  The media performance is not benchmarked in this work item. benchmarked.
      *  Some tests require media, but the use of media is limited to
         observing the performance of SIP signaling.  Tests that require
         media will annotate the media characteristics as a condition of
      *  The type of DUT dictates whether the associated media streams
         traverse the DUT.  Both scenarios are within the scope of this
         work item.
      *  SIP is frequently used to create media streams; the signaling
         plane and media plane are treated as orthogonal to each other
         in this document.  While many devices support the creation of
         media streams, benchmarks that measure the performance of these
         streams are outside the scope of this document and its
         companion methodology document [I-D.ietf-bmwg-sip-bench-meth].
         Tests may be performed with or without the creation of media
         streams.  The presence or absence of media streams MUST be
         noted as a condition of the test as the performance of SIP
         devices may vary accordingly.  Even if the media is used during
         benchmarking, only the SIP performance will be benchmarked, not
         the media performance or quality.
   o  Both INVITE and non-INVITE scenarios (registrations) are addressed
      in this document.  However, benchmarking SIP presence or
      subscribe-notify extensions is not a part of this work item. document.
   o  Different transport -- such as UDP, TCP, SCTP, or TLS -- may be
      used.  The specific transport mechanism MUST be noted as a
      condition of the test as the performance of SIP devices may vary

   o  REGISTER and INVITE requests may be challenged or remain
      unchallenged for authentication purpose.  Whether or not the
      REGISTER and INVITE requests are challenged is a condition of test
      which will be recorded along with other such parameters which may
      impact the SIP performance of the device or system under test.
   o  Re-INVITE requests are not considered in scope of this work item document
      since the benchmarks for INVITEs are based on the dialog created
      by the INVITE and not on the transactions that take place within
      that dialog.
   o  Only session establishment is considered for the performance
      benchmarks.  Session disconnect is not considered in the scope of
      this work item. document.  This is because our goal is to determine the
      maximum capacity of the device or system under test, that is the
      number of simultaneous SIP sessions that the device or system can
      support.  It is true that there are BYE requests being created
      during the test process.  These transactions do contribute to the
      load on the device or system under test and thus are accounted for
      in the metric we derive.  We do not seek a separate metric for the
      number of BYE transactions a device or system can support.
   o  IMS-specific scenarios are not considered, but test cases can be
      applied with 3GPP-specific SIP signaling and the P-CSCF as a DUT.

3.  Term Definitions

3.1.  Protocol Components

3.1.1.  Session

      The combination of signaling and media messages and processes that
      support a SIP-based service.

      SIP messages are used to create and manage services for end users.
      Often, these services include the creation of media streams that
      are defined in the SDP body of a SIP message and carried in RTP
      protocol data units.  However, SIP messages can also be used to
      create Instant Message instant message services and subscription services, and
      such services are not associated with media streams.  SIP reserves
      the term "session" to describe services that are analogous to
      telephone calls on a circuit switched network.  SIP reserves the
      term "dialog" to refer to a signaling-only relationship between
      User Agent peers.  SIP reserves the term "transaction" to refer to
      the brief communication between a client and a server that lasts
      only until the final response to the SIP request.  None of these
      terms describes the entity whose performance we want to benchmark.
      For example, the MESSAGE request does not create a dialog and can
      be sent either within or outside of a dialog.  It is not
      associated with media, but it resembles a phone call in its
      dependence on human rather than machine initiated responses.  The
      SUBSCRIBE method does create a dialog between the originating end-
      user and the subscription service.  It, too, is not associated
      with a media session.

      In light of the above observations we have extended the term
      "session" to include SIP-based services that are not initiated by
      INVITE requests and that do not have associated media.  In this
      extended definition, a session always has a signaling component
      and may also have a media component.  Thus, a session can be
      defined as signaling-only or a combination of signaling and media.
      We define the term "Associated Media", see Section 3.1.4, to
      describe the situation in which media is associated with a SIP
      dialog.  The terminology "Invite-Initiated Session" (IS)
      Section 3.1.8 and "Non-invite-Initiated Session" (NS)
      Section 3.1.9 are used to distinguish between these two types of
      session.  An Invite-Initiated Session is a session as defined in
      SIP.  The performance of a device or system that supports Invite-
      Initiated Sessions that do not create media sessions, "Invite-
      Initiated Sessions without Associated Media", can be measured and
      is of interest for comparison and as a limiting case.  The
      REGISTER request can be considered to be a "Non-invite-Initiated
      Session without Associated Media."  A separate set of benchmarks
      is provided for REGISTER requests since most implementations of
      SIP-based services require this request and since a registrar may
      be a device under test.

      A Session in the context of this document, can be considered to be
      a vector with three components:

      1.  A component in the signaling plane (SIP messages), sess.sig;
      2.  A media component in the media plane (RTP and SRTP streams for
          example), (which may be null);
      3.  A control component in the media plane (RTCP messages for
          example), sess.medc (which may be null).

      An IS is expected to have non-null sess.sig and
      components.  The use of control protocols in the media component
      is media dependent, thus the expected presence or absence of
      sess.medc is media dependent and test-case dependent.  An NS is
      expected to have a non-null sess.sig component, but null
      and sess.medc components.

      Packets in the Signaling Plane and Media Plane will be handled by
      different processes within the DUT.  They will take different
      paths within a network.  These different processes and paths may
      produce variations in performance.  The terminology and benchmarks
      defined in this document and the methodology for their use are
      designed to enable us to compare performance of the DUT with
      reference to the type of SIP-supported application it is handling.

      Note that one or more sessions can simultaneously exist between
      any participants.  This can be the case, for example, when the EA
      sets up both an IM and a voice call through the DUT.  These
      sessions are represented as an array session[x].

      Sessions will be represented as a vector array with three
      components, as follows:
      session[x].sig, the signaling component
      session[x].medc[y], the media control component (e.g.  RTCP)
      session[x].med[y], an array of associated media streams (e.g.
      RTP, SRTP, RTSP, MSRP).  This media component may consist of zero
      or more media streams.
      Figure 1 models the vectors of the session.

   Measurement Units:


   See Also:
      Media Plane
      Signaling Plane
      Associated Media
      Invite-Initiated Session (IS)
      Non-Invite-Initiated Session (NS)
                    |   \
                    |     \
                    |       \
                    |         o
                    |        /
                    |       / |
                    |      /
                    |     /   |
                    |    /
                    |   /     |
                    |  /
                    | /       |   sess.medc
                   /               /
                  /           |
                 /               /   /             |
               /_ _ _ _ _ _ _ _/

                       Figure 1: Session components

3.1.2.  Signaling Plane

      The plane in which SIP messages [RFC3261] are exchanged between
      SIP Agents [RFC3261].

      SIP messages are used to establish sessions in several ways:
      directly between two User Agents [RFC3261], through a Proxy Server
      [RFC3261], or through a series of Proxy Servers.  The Session
      Description Protocol (SDP) is included in the Signaling Plane.
      The Signaling Plane for a single Session is represented by

   Measurement Units:


   See Also:
      Media Plane

3.1.3.  Media Plane

      The data plane in which one or more media streams and their
      associated media control protocols are exchanged between User
      Agents after a media connection has been created by the exchange
      of signaling messages in the Signaling Plane.

      Media may also be known as the "bearer channel".  The Media Plane
      MUST include the media control protocol, if one is used, and the
      media stream(s).  Examples of media are audio and video.  The
      media streams are described in the SDP of the Signaling Plane.
      The media for a single Session is represented by  The
      media control protocol for a single media description is
      represented by session.medc.

   Measurement Units:


   See Also:
      Signaling Plane

3.1.4.  Associated Media

      Media that corresponds to an 'm' line in the SDP payload of the
      Signaling Plane.


      Any media protocol MAY be used.
      For any session's signaling component, session.sig, there may be
      zero, one, or multiple associated media streams.  When there are
      multiple media streams, these are represented be a vector array[y].  When there are multiple media streams there will
      be multiple media control protocol descriptions as well.  They are
      represented by a vector array session.medc[y].

   Measurement Units:


3.1.5.  Overload

      Overload is defined as the state where a SIP server does not have
      sufficient resources to process all incoming SIP messages

      The distinction between an overload condition and other failure
      scenarios is outside the scope of black box testing and of this
      document.  Under overload conditions, all or a percentage of
      Session Attempts will fail due to lack of resources.  In black box
      testing the cause of the failure is not explored.  The fact that a
      failure occurred for whatever reason, will trigger the tester to
      reduce the offered load, as described in the companion methodology
      document, [I-D.ietf-bmwg-sip-bench-meth].  SIP server resources
      may include CPU processing capacity, network bandwidth, input/
      output queues, or disk resources.  Any combination of resources
      may be fully utilized when a SIP server (the DUT) is in the
      overload condition.  For proxy-only (or intermediary) devices, it
      is expected that the proxy will be driven into overload based on
      the delivery rate of signaling requests.

   Measurement Units:

3.1.6.  Session Attempt


      A SIP INVITE or REGISTER request sent by the EA that has not
      received a final response.

      The attempted session may be Invite Initiated or Non-invite-
      Initiated.  When counting the number of session attempts we
      include all INVITEs that are rejected for lack of authentication
      information.  The EA needs to record the total number of session
      attempts including those attempts that are routinely rejected by a
      proxy that requires the UA to authenticate itself.  The EA is
      provisioned to deliver a specific number of session attempts per
      second.  But the EA must also count the actual number of session
      attempts per given tie interval.

   Measurement Units:


   See Also:
      Session Attempt Rate
      Invite-Initiated Session
      Non-Invite-Initiated Session

3.1.7.  Established Session

      A SIP session for which the EA acting as the UE/UA has received a
      200 OK message.

      An Established Session MAY be Invite Initiated or Non-Invite-
      Initiated.  Early dialogues for INVITE requests are out of scope
      for this work.

   Measurement Units:


   See Also:

      Invite-Initiated Session

3.1.8.  Invite-Initiated Session (IS)

      A Session that is created by an exchange of messages in the
      Signaling Plane, the first of which is a SIP INVITE request.

      When an IS becomes an Established Session its signaling component
      is identified by the SIP dialog parameter values, Call-ID, To-tag,
      and From-tag (RFC3261 [RFC3261]).  An IS may have zero, one or
      multiple Associated Media descriptions in the SDP body.  The
      inclusion of media is test case dependent.  An IS is successfully
      established if the following two conditions are met:
      1.  Sess.sig is established by the end of Establishment Threshold
          Time (c.f.  Section 3.3.2), and
      2.  If the DUT is a device, such as an SBC or B2BUA, that may
          receive media from a calling or called party before a
          signaling dialog is established or confirmed , this fact needs
          to be taken into account when the EA is built.  Any additional
          parameters that need to be configured in the EA must be added
          to the list of test setup parameters in Section 5.1 of

   Measurement Units:


   See Also:
      Non-Invite-Initiated Session
      Associated Media

3.1.9.  Non-INVITE-Initiated Session (NS)

      A session that is created by an exchange of SIP messages in the
      Signaling Plane the first of which is not a SIP INVITE message.

      An NS is successfully established if the Session Attempt via a
      non- INVITE
      non-INVITE request results in the EA receiving a 2xx reply before
      the expiration of the Establishment Threshold timer (c.f.,
      Section 3.3.2).  For the purpose of this document, a NS is a
      session created only by the REGISTER request and no other request.

   Measurement Units:


   See Also:
      Invite-Initiated Session

3.1.10.  Session Attempt Failure

      A session attempt that does not result in an Established Session.

      The session attempt failure may be indicated by the following
      observations at the EA:
      1.  Receipt of a SIP 3xx-, 4xx-, 5xx-, or 6xx-class response to a
          Session Attempt.
      2.  The lack of any received SIP response to a Session Attempt
          within the Establishment Threshold Time (c.f.  Section 3.3.2).

   Measurement Units:


   See Also:
      Session Attempt

3.2.  Test Components

3.2.1.  Emulated Agent

      A device in the test topology that initiates/responds to SIP
      messages as one or more session endpoints and, wherever
      applicable, sources/receives Associated Media for Established

      The EA functions in the Signaling and Media Planes.  The Tester
      may act as multiple EAs.

   Measurement Units:


   See Also:
      Media Plane
      Signaling Plane
      Established Session
      Associated Media

3.2.2.  Signaling Server

      Device in the test topology that facilitates the creation of
      sessions between EAs.  This device is the DUT.

      The DUT is a RFC3261-capable network intermediary such as a
      Registrar, Redirect Server, User Agent Server, Stateless Proxy,
      Stateful Proxy, B2BUA or SBC.

   Measurement Units:


   See Also:
      Signaling Plane

3.2.3.  SIP Transport Protocol

      The protocol used for transport of the Signaling Plane messages.

      Performance benchmarks may vary for the same SIP networking device
      depending upon whether TCP, UDP, TLS, SCTP, websockets [RFC7118]
      or any future transport layer protocol is used.  For this reason
      it is necessary to measure the SIP Performance Benchmarks using
      these various transport protocols.  Performance Benchmarks MUST
      report the SIP Transport Protocol used to obtain the benchmark

   Measurement Units:
      While these are not units of measure, they are attributes that are
      one of many factors that will contribute to the value of the
      measurements to be taken.  TCP, UDP, SCTP, TLS over TCP, TLS over
      UDP, TLS over SCTP, and websockets are among the possible values
      to be recorded as part of the test.


   See Also:

3.3.  Test Setup Parameters

3.3.1.  Session Attempt Rate

      Configuration of the EA for the number of sessions per second
      (sps) that the EA attempts to establish using the services of the

      The Session Attempt Rate is the number of sessions per second that
      the EA sends toward the DUT.  Some of the sessions attempted may
      not result in a session being established.  A session in this case
      may be either an IS or an NS.

   Measurement Units:
      Session attempts per second


   See Also:
      Session Attempt

3.3.2.  Establishment Threshold Time

      Configuration of the EA that represents the amount of time that an
      EA client will wait for a response from an EA server before
      declaring a Session Attempt Failure.

      This time duration is test dependent.

      It is RECOMMENDED that the Establishment Threshold Time value be
      set to Timer B (for ISs) or Timer F (for NSs) as specified in RFC
      3261, Table 4 [RFC3261].  Following the default value of T1
      (500ms) specified in the table and a constant multiplier of 64
      gives a value of 32 seconds for this timer (i.e., 500ms * 64 =

   Measurement Units:


3.3.3.  Session Duration

      Configuration of the EA that represents the amount of time that
      the SIP dialog is intended to exist between the two EAs associated
      with the test.

      The time at which the BYE is sent will control the Session

   Measurement Units:


   See Also:

3.3.4.  Media Packet Size

      Configuration on the EA for a fixed number of frames or samples to
      be sent in each RTP packet of the media session.


      For a single benchmark test, media sessions use a defined number
      of samples or frames per RTP packet.  If two SBCs, for example,
      used the same codec but one puts more frames into the RTP packet,
      this might cause variation in the performance benchmark results.

   Measurement Units:
      An integer number of frames or samples, depending on whether
      hybrid- or sample-based codec are used, respectively.


   See Also:

3.3.5.  Codec Type

      The name of the codec used to generate the media session.

      For a single benchmark text, all sessions use the same size packet
      for media streams.  The size of packets can cause a variation in
      the performance benchmark measurements.

   Measurement Units:
      This is a textual name (alphanumeric) assigned to uniquely
      identify the codec.

   See Also:

3.4.  Benchmarks

3.4.1.  Session Establishment Rate

      The maximum value of the Session Attempt Rate that the DUT can
      handle for an extended, pre-defined, period with zero failures.

      This benchmark is obtained with zero failure in which 100% of the
      sessions attempted by the Emulated Agent are successfully
      completed by the DUT. failure.  The session attempt
      rate provisioned on the EA is raised and lowered as described in
      the algorithm in the accompanying methodology document, document
      [I-D.ietf-bmwg-sip-bench-meth], until a traffic load at the
      given attempt rate over the sustained
      period of time identified by
      T necessary to attempt N sessions completes without
      failure, where N is a parameter specified in the algorithm completes without any failed session attempts. and
      recorded in the Test Setup Report.  Sessions may be IS or NS or a
      mix of both and will be defined in the particular test.

   Measurement Units:
      sessions per second (sps)


   See Also:
      Invite-Initiated Sessions
      Non-Invite-Initiated Sessions
      Session Attempt Rate

3.4.2.  Registration Rate

      The maximum value of the Registration Attempt Rate that the DUT
      can handle for an extended, pre-defined, period with zero

      This benchmark is obtained with zero failures in which 100% of the
      registrations attempted by the EA are successfully completed by
      the DUT. failures.  The registration
      rate provisioned on the Emulated Agent is raised and lowered as
      described in the algorithm in the companion methodology draft [I-D.ietf-bmwg-sip-bench-meth]
      [I-D.ietf-bmwg-sip-bench-meth], until a traffic load consisting of
      registration attempts at the given attempt rate over the period of
      time necessary to attempt N registrations completes without
      failure, where N is a parameter specified in the algorithm and
      recorded in the Test Setup Report.
      This benchmark is described separately from the Session
      Establishment Rate (Section 3.4.1), although it could be
      considered a special case of that benchmark, since a REGISTER
      request is a request for a Non-Invite-Initiated session.  It is
      defined separately because it is a very important benchmark for
      most SIP installations.  An example demonstrating its use is an
      avalanche restart, where hundreds of thousands of end points
      register simultaneously following a power outage.  In such a case,
      an authoritative measurement of the capacity of the device to
      register endpoints is useful to the network designer.  Finally,
      Additionally, in certain controlled networks, there appears to be
      a difference in between the registration rate of new endpoints and
      the registering versus rate of existing endpoints refreshing their registrations. (register refreshes).
      This benchmark can capture these differences as well.

   Measurement Units:
      registrations per second (rps)


   See Also:

3.4.3.  Registration Attempt Rate

      Configuration of the EA for the number of registrations per second
      that the EA attempts to send to the DUT.

      The Registration Attempt Rate is the number of registration
      requests per second that the EA sends toward the DUT.

   Measurement Units:
      Registrations per second (rps)


   See Also:  Non-Invite-Initiated Session

4.  IANA Considerations

   This document requires no IANA considerations.

5.  Security Considerations

   Documents of this type do not directly affect the security of
   Internet or corporate networks as long as benchmarking is not
   performed on devices or systems connected to production networks.
   Security threats and how to counter these in SIP and the media layer
   is discussed in RFC3261 [RFC3261], RFC 3550 [RFC3550], RFC3711
   [RFC3711] and various other drafts.  This document attempts to
   formalize a set of common terminology for benchmarking SIP networks.
   Packets with unintended and/or unauthorized DSCP or IP precedence
   values may present security issues.  Determining the security
   consequences of such packets is out of scope for this document.

6.  Acknowledgments

   The authors would like to thank Keith Drage, Cullen Jennings, Daryl
   Malas, Al Morton, and Henning Schulzrinne for invaluable
   contributions to this document.  Dale Worley provided an extensive
   review that lead to improvements in the documents.  We are grateful
   to Barry Constantine for providing valuable comments during the
   document's WGLC.

7.  References

7.1.  Normative References

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119, March 1997.

   [RFC2544]  Bradner, S. and J. McQuaid, "Benchmarking Methodology for
              Network Interconnect Devices", RFC 2544, March 1999.

   [RFC3261]  Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston,
              A., Peterson, J., Sparks, R., Handley, M., and E.
              Schooler, "SIP: Session Initiation Protocol", RFC 3261,
              June 2002.

   [RFC5393]  Sparks, R., Lawrence, S., Hawrylyshen, A., and B. Campen,
              "Addressing an Amplification Vulnerability in Session
              Initiation Protocol (SIP) Forking Proxies", RFC 5393,
              December 2008.

   [RFC4320]  Sparks, R., "Actions Addressing Identified Issues with the
              Session Initiation Protocol's (SIP) Non-INVITE
              Transaction", RFC 4320, January 2006.

   [RFC6026]  Sparks, R. and T. Zourzouvillys, "Correct Transaction
              Handling for 2xx Responses to Session Initiation Protocol
              (SIP) INVITE Requests", RFC 6026, September 2010.

              Davids, C., Gurbani, V., and S. Poretsky, "Methodology for "SIP Performance
              Benchmarking SIP Networking Devices",
              draft-ietf-bmwg-sip-bench-meth-08 Methodology",
              draft-ietf-bmwg-sip-bench-meth-10 (work in progress),
              January 2013.
              May 2014.

7.2.  Informational References

   [RFC2285]  Mandeville, R., "Benchmarking Terminology for LAN
              Switching Devices", RFC 2285, February 1998.

   [RFC1242]  Bradner, S., "Benchmarking terminology for network
              interconnection devices", RFC 1242, July 1991.

   [RFC3550]  Schulzrinne, H., Casner, S., Frederick, R., and V.
              Jacobson, "RTP: A Transport Protocol for Real-Time
              Applications", STD 64, RFC 3550, July 2003.

   [RFC3711]  Baugher, M., McGrew, D., Naslund, M., Carrara, E., and K.
              Norrman, "The Secure Real-time Transport Protocol (SRTP)",
              RFC 3711, March 2004.

   [RFC6357]  Hilt, V., Noel, E., Shen, C., and A. Abdelal, "Design
              Considerations for Session Initiation Protocol (SIP)
              Overload Control", RFC 6357, August 2011.

   [RFC7118]  Baz Castillo, I., Millan Villegas, J., and V. Pascual,
              "The Websocket Protocol as a Transport for the Session
              Initiation Protocol (SIP)", RFC 7118, January 2014.

Authors' Addresses

   Carol Davids
   Illinois Institute of Technology
   201 East Loop Road
   Wheaton, IL  60187

   Phone: +1 630 682 6024

   Vijay K. Gurbani
   Bell Laboratories, Alcatel-Lucent
   1960 Lucent Lane
   Rm 9C-533
   Naperville, IL  60566

   Phone: +1 630 224 0216
   Scott Poretsky
   Allot Communications
   300 TradeCenter, Suite 4680
   Woburn, MA  08101

   Phone: +1 508 309 2179