Network Working Group                                       R. Craig                             J. H. Dunn
INTERNET-DRAFT                                         Cisco Systems                                   	Hewlett-Packard
Expires in six months                                     March 1997                             C. E. Martin
                                            		Hewlett-Packard

						 August 1998

Terminology for Cell/Call Call/Cell Benchmarking
                     <draft-ietf-bmwg-call-01.txt>
draft-ietf-bmwg-call-02.txt

Status of this Memo
This document is an Internet-Draft.  Internet-Drafts are working
documents of the Internet Engineering Task Force (IETF), its
areas, and its working groups.  Note that other groups may also
distribute working documents as Internet-Drafts.

Internet-Drafts are draft documents valid for a maximum of six
months and may be updated, replaced, or obsoleted by other
documents at any time.  It is inappropriate to use Internet-
Drafts as reference material or to cite them other than as ``work
in progress.''

To learn view the current status entire list of any Internet-Draft, current Internet-Drafts, please check
the
   ``1id-abstracts.txt'' "1id-abstracts.txt" listing contained in the Internet- Drafts Internet-Drafts
Shadow Directories on ftp.is.co.za (Africa), nic.nordu.net (Europe), ftp.nordu.net
(Northern Europe), ftp.nis.garr.it (Southern Europe), munnari.oz.au
(Pacific Rim), ds.internic.net ftp.ietf.org (US East Coast), or ftp.isi.edu
(US West Coast).

Abstract

   The purpose

This memo discusses and defines terms associated with performance
benchmarking tests and the results of this draft is to add terminology specific to these tests in the cell context
of cell-based and call-based switch environment switching devices.  The terms defined
in this memo will be used in addition to that terms defined by in RFCs
1242, 1944 and 2285.  This memo is a product of the Benchmarking
Methodology Working Group (BMWG) of the Internet Engineering Task
Force (IETF) (IETF).

I.  Background

1.  Introduction

This document provides terminology for benchmarking cell-based and
call-based switching devices.  It extends terminology already
defined for benchmarking network interconnect devices in RFC1242.

   While primarily directed towards wide area switches, portions RFCs
1242, 1944 and 2285.  Although some of the
   document definitions in this

memo may be useful for benchmarking other devices such as ADSU's.

1.  Introduction

   In light applicable to a broader group of network interconnect
devices, the increasing use primary focus of the terminology in this memo is on
cell-based and/or circuit-switched
   transport layers in building networks, it would and call-based switches.  Specifically, this includes
Asynchronous Transfer Mode (ATM) cell relay and signaling and
Frame Relay (FR) signaling.

This memo contains two major sections: Background and Definitions.
Within the definitions section are a formal definitions sub-
section, provided as a courtesy to the reader, and a measurement
definitions sub-section which contains performance metrics with
inherent units.

2.  Existing Definitions

   RFC 1242 "Benchmarking Terminology for Network Interconnect
Devices" should be useful consulted before attempting to develop make use of this
document.   RFC 1944 "Benchmarking Methodology for Network
Interconnect Devices" contains discussions of a set number of benchmarks with which terms
relevant to compare technologies,
   implementation strategies, the benchmarking of switching devices and products.

   1.1 should also
be consulted.  RFC 2285 "Benchmarking Terminology Brought Forward

      The terminology defined for LAN
Switching Devices" contains a number of terms pertaining to
traffic distributions and datagram interarrival.
   For the sake of clarity and continuity this RFC adopts the
template for definitions set out in Section 2 of RFC 1242 applies equally well 1242.
Definitions are indexed and grouped together in sections for ease
of reference.
   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL
NOT",   "SHOULD", "SHOULD NOT", "RECOMMENDED",  "MAY", and
"OPTIONAL" in this document are to be interpreted as described in
RFC 2119.

II. Definitions

The definitions presented in this
      memo.  There section have been divided into
two groups.  The first group is also a certain amount formal definitions, which are
required in the definitions of overlap with terms
      defined the performance metrics but are not
themselves strictly metrics.  These definitions are subsumed from
other work done in draft-ietf-bmwg-lanswitch-00.txt.

2. other working groups both inside and outside
the IETF.  They are provided as a courtesy to the reader.

1.  Formal Definitions

1.1.  Definition Format (from RFC1242)

Term to be defined.

Definition: The specific definition for the term.

Discussion: A brief discussion of the term, its application and
any restrictions on measurement procedures.

   Measurement units:
      Units used to record measurements of this term, if applicable.

3.  Term Definitions

3.1  Virtual Circuit

   This

Specification:  The working group applies to those switches that are connection-oriented.

   3.1.1  Call setup time

      Definition: and document in which the length of time for term
is specified.  Listed in the virtual circuit to be
      established.

      Discussion:  as measured from references.

1.2 Definitions

1.2.1 Available Bit Rate (ABR):

Definition:  ABR is an ATM layer service category for which the initiation of
limiting ATM layer transfer characteristics provided by the signalling
network may change subsequent to
      circuit establishment.

      Measurement units:  fractional seconds

      Issues:

      See also:

   3.1.2  Call setup rate (sustained)

      Definition:  the maximum sustained rate of successful connection establishment.

      Discussion:  without loss of existing calls.

      Measurement units:  calls per second

      Issues:

      See also:

   3.1.3  Call maintenance overhead
      Definition:  the amount A flow
control mechanism is specified which supports several types of work required
feedback to maintain control the calls
      that have been established. source rate in response to changing ATM
layer transfer characteristics.

Discussion:  It is expected that an end-system that adapts its
traffic in accordance with the feedback will experience a method to low cell
loss ratio and obtain a fair share of the desired result would be available bandwidth
according to
      benchmark with PVC's in place, then with SVC's.  The difference a network specific allocation policy. Cell delay
variation is not controlled in
      results would be the overhead.

      Measurement units:

      Issues:

      See also:

   3.1.4  Call teardown time this service, although admitted
cells are not delayed unnecessarily.

Specification: AF-TM4.0

1.2.2 Call-based

Definition:  An association between two or more users or between a
user and a network entity that is established by the length use of time for the virtual circuit to be torn
      down.
network capabilities.  This association may have zero or more
connections.

Discussion:  measured from the start  none

Specification: AF-UNI3.1

1.2.3 Cell

Definition: A unit of the signalling to the
      freeing transmission in ATM.  A fixed size frame
consisting of the resources associated with that call (end a 5 octet header and a 48 octet payload.

Discussion:  none

Specification: AF-UNI3.1

2. Performance Metrics

2.1.  Definition Format (from RFC1242)

Metric to end, if
      applicable).

      Measurement units:  fractional seconds

      Issues:

      See also:

   3.1.5  Call teardown rate (sustained) be defined.

Definition:  The specific definition for the maximum rate at which calls can be successfully
      torn down. metric.

Discussion:  without loss  A brief discussion of existing calls, the metric, its application and without failure
      to tear down
any calls that have been signalled to be destroyed.

      Measurement units:  teardowns per second

      Issues:

      See also:

   3.1.6  Impact of Signalling restrictions on Forwarding

      Definition:  cells per second versus calls per second

      Discussion:  some devices use the same engine for cell forwarding
      and call maintenance.  In this case, interaction between the two
      functions will be inevitable.  More interesting, however, would be
      the case where the two processing functions are clearly separate,
      yet still interact. measurement procedures.

Measurement units:  cells per second versus calls per second

      Issues:

      See also:

3.2  Cell/Packet Interaction

   This group applies Intrinsic units used to cell-based switches, connection-oriented or
   not.

   3.2.1  Packet disassembly/reassembly time (peak)

      Definition: quantify this metric.
This includes subsidiary units, e.g. microseconds are acceptable
if the length of intrinsic unit is seconds.

2.2 Definitions

2.2.1 Cell Transfer Delay (CTD)

Definition:  The elapsed time to disassemble a layer 3 packet
      into layer 2 cells, or reassemble cells into between a packet.

      Discussion:  with no packet or cell loss or corruption.  To arrive exit event at a baseline, one could measure the switching rate for cells
      derived from ~1440 byte frames which are flowing across
measurement point 1 (e.g., at the switch
      as cells, then forward those same frames into source UNI) and the switch from an
      interface which would require them to be disassembled.  For
      example, the baseline measurement is taken while switching cells
      OC3-OC3.  Then switch FDDI or POS-OC3 and take the delta in
      performance as the SAR overhead.

      Measurement units:  the appropriate fraction of a second

      Issues:

      See also:

   3.2.2  Packet disassembly/reassembly rate (sustained)

      Definition:  the maximum sustained rate at which packets can be
      disassembled/reassembled into/from cells.

      Discussion:  without loss or corruption.

      Measurement units:  packets per second

      Issues:

      See also:

   3.2.3  Full packet drop rate (on cell loss)

      Definition:  the rate at which
corresponding cell loss triggering full packet
      drop can be detected/sustained.

      Discussion:  When a packet is disassembled into cells, typically
      many cells result.  When these cells are transmitted, they are
      subject to loss or corruption. The device should recognize entry event at the
      cell/packet boundary that a cell or cells belonging to a given
      packet has been lost and should drop that packet, immediately
      freeing those resources.  A couple of things are of interest here:
      whether the switch is able to detect very small amounts of cell
      loss and correctly drop the associated packets and whether large
      amounts of cell loss perturb this ability in any way.

      Measurement units:  (dropped) packets per second

      Issues:

      See also:

   3.2.4  End to end data integrity

      Definition:  the percentage of packets (post-reassembly) that
      actually contain undetected data link layer corruption.

      Discussion:  some network devices have been known to regenerate
      CRC's over the re-assembled packet (i.e., measurement point 2 (e.g., the CRC is not carried
      end to end), resulting in undetected data link layer corruption or
      re-ordering of cells in a packet.

      Measurement units:  percentage

      Issues:  production of
destination UNI) for a stream of traffic containing internal
      checksums sufficiently strong to detect cell re-ordering (the IP
      checksum is not).  The ISIS LSP checksum is.

      See also:

3.3  Switch Fabric

   This group applies to all switches.

   3.3.2  Topology Table Size

      Definition:  number of network elements supported. particular connection.

Discussion:  switches may support a limited topology due to static
      table sizes or processing limitations.  This is true whether it's
      a "LAN" switch running spanning tree or a "WAN" switch running
      OSPF. The effect of a limited topology table on a switch in a
      real-world environment can be disastrous.

      A similar metric (2.14 Address handling) is mentioned in "draft-
      ietf-bmwg-lanswitch-00.txt".  Here, a more general metric is
      intended.

      Measurement units:  number

      Issues:  Measuring the effects of an overflow is probably
      meaningless, since in the multi-switch case, there is no longer
      any network to speak of, hence, nothing to measure.

      If a device handles table overflow gracefully, this should be
      noted.  Similarly, if a device crashes and burns on table
      overflow, this should be noted.

      See also:

   3.3.3  Topology Table Learning Rate

      Definition:  the rate at which the topology table can be filled or
      updated.

      Discussion:  a single switch in isolation learning MAC addresses
      will flood frames when the rate exceeds its learning capability.
      This metric is covered in "2.15 Address learning speed" of
      "draft-ietf-bmwg-lanswitch-00.txt".  We generalize the metric here
      to include the topological databases of routing protocols used in
      switched networks (among the switches themselves) as well as the
      spanning tree recalculation among multiple LAN switches.

      Measurement units:  frames per second 1) with maximum diversity of
      addresses, 2) with routing instability introduced.

      Issues:

      See also:

   3.3.5  Throughput (from RFC1242) (Cell forwarding rate)

      Definition:  The maximum rate at which none of the offered frames
      are dropped by the device.

      Discussion:  This metric probably overlaps work being done in the
      ATM Forum.

      Measurement units:  cells per second

      Issues:

      See also:

   3.3.6 Blocking Probability

      Definition:  likelihood of successful simultaneous communication
      amongst multiple ports.

      Discussion:  a switch is termed "non-blocking" if multiple ports
      are able to communicate across the switch fabric at the same time.
      We are interested in the probability of blocking occurring in the
      1:1, N:1, and N:M scenarios.

      One may calculate the "ideal" throughput in the absence of
      blocking, then take the delta with the experimental case and treat
      that as an empirical measurement of blocking probability, if
      enough samples are taken.

      Measurement units:  percentage likelihood of blocking.

      Issues:

      See also:

3.5  Congestion Control

   This group applies to all switches.

   3.5.1  Congestion avoidance

      Definition:  effectiveness of measures taken by the switch to
      avoid congestion.

      Discussion:  connections that are bursting above their committed
      rate may have cells buffered at the ingress, in order to avoid
      congestion in the trunks and impact on other connections, or they
      may simply be marked "discard-eligible" and forwarded into the
      network, hoping for the best.

      Distinguishing cell transfer delay between these two approaches should be relatively
      simple.  In the first case, latency for the bursting session
      increases, but there measurement points
is no cell loss.  Other sessions are
      unaffected.  In the second case, there may be cell loss across any
      of the sessions, and latency may increase across all.

      Measurement units:  dropped cells, latency

      Issues:

      See also:

   3.5.2  Congestion management

      Definition:  effectiveness sum of measures taken by the switch to deal
      with congestion.

      Discussion:  in the face of sustained traffic above committed rate
      on multiple sessions, a switch has little choice but to begin
      discarding cells, since buffering cannot be infinite.  This case
      might arise if one were wildly profligate in over-subscribing
      trunk bandwidth, or if one had neglected to analyze the network
      applications to be run over the network and they were found to be
      network-hostile (UDP, IPX, AT, NetBIOS, for example).

      The switch has some discretion in deciding which cells to drop.
      Presumably, the strategy should involve something resembling
      "fairness".

      The basic idea is that ill-behaved connections should not starve
      others for resources.

      Measurement units:  latency, cell drops

      Issues:

      See also:

3.6  Inter-switch protocols

   This group applies to all switches.

   3.6.1  Impact of Routing on Forwarding

      Definition:  interaction between routing protocol total inter-ATM node transmission delay and data
      forwarding operations.

      Discussion:  No amount of routing fluctuation should have an
      impact on data forwarding for unaffected destinations.  Similarly,
      no amount of data forwarding should cause the routing to become
      unstable.
total ATM node processing delay.

Measurement units:  route flaps per second versus cells per
      second, cells per second versus route stability (table fluctuation
      or peer loss).

      Issues:

      See also:

   3.6.2  Impact of Congestion Control seconds

2.2.2 Cell Delay Variation (CDV)

Definition:  interaction between congestion control  The variation in cell transfer delay associated with
a given traffic load, orientation and data
      forwarding operations.

      Discussion:  switches may share views of congestion in-band
      through distribution, as well as an
integration period.  CDV = max(CTD) - min(CTD) where max and min
indicate the network.  Should these feedback messages be delayed or
      lost, maximum  and minimum over the potential exists for an incorrect picture integration period,
respectively.

Discussion: CDV is a component of current
      network conditions, which may exacerbate congestion cell transfer delay, induced by
buffering and lead to cell loss.  Worse, it scheduling. Peak-to-peak CDV is possible to enter a stable oscillation
      state, where ever-increasing waves QoS delay
parameter associated with CBR and VBR services. The peak-to-peak
CDV is the ((1-a) quantile of congestion overwhelm the
      switches.

      Measurement units:

      Issues:

      See also:

3.7  Quality CTD) minus the fixed CTD that
could be experienced by any delivered cell on a connection during
the entire connection holding time. The parameter "a" is the
probability of Service

   This group applies to all switches.

   3.7.1  Traffic Management Policing a cell arriving late.

Measurement Units: seconds.

2.2.3 Cell Error Ratio (CER)

Definition:  impact  The ratio of misbehaving class on others, for example
      data forwarding on voice or video frames errored cells in a transmission in
relation to the total cells sent in a transmission associated with
a given traffic load, orientation and vice versa. distribution, as well as an
integration period.   CER = Errored Cells / Total Cells
Transmitted.

Discussion:  we wish  The measurement is taken over a time interval and is
desirable to be measured on an in-service circuit.

Measurement Units: dimensionless.

2.2.4  Cell Loss Ratio (CLR)

Definition:  The ratio of lost cells in a transmission in relation
to quantify the potential interaction amongst
      the various classes of service.  Constant bit rate (CBR), variable
      bit rate (VBR) (real total cells sent in a transmission associated with a given
traffic load, orientation and non-real time?), distribution, as well as an
integration period.  CLR = Lost Cells / Total Cells Transmitted.

Discussion:  CLR is a negotiated QoS parameter and available bit rate
      (ABR) streams acceptable
values are established, within their respective service
      levels, but sufficient network specific. The objective is to subscribe minimize CLR
provided the trunk end-system adapts the traffic to 90%. the changing ATM
layer transfer characteristics. The bit
      rate of each CLR parameter is increased until it has exceeded its allocation by
      a degree which should cause loss or delay in the other streams.

      Measurement units:  cells (lost) per second, latency

      Issues:  some switches perform compression and silence
      suppression.  Should these features be disabled?

      See also:

3.8  Multicast

   3.8.1  Cell replication

      Definition: value of
CLR that the device's ability network agrees to forward offer as an objective over the
lifetime of the connection. It is expressed as an order of
magnitude, having a cell range of 10-1 to multiple
      ports simultaneously (multicast).

      Discussion:

      Measurement units:  replication factor 1:N 10-15 and unspecified.

Measurement Units: dimensionless.

2.2.5 Cell Misinsertion Rate (CMR)

Definition: The ratio of cells per second
      measured at ingress versus cells per second measured received at an endpoint that were
not originally transmitted by the
      egresses.

      Issues:

      See also:

   3.8.2  Impact of multicast on unicast

      Definition:  switch's ability source end in relation to insulate unicast traffic from the
      effects
total number of multicast.

      Discussion: cells properly transmitted associated with a poorly-designed replication scheme could easily
      swamp unicast traffic.  Yet, multicast given
traffic often has QoS
      needs.  How does one reconcile the competing requirements? load, orientation and distribution, as well as an
integration period.  CMR = Mis-inserted Cells / Total Cells
Transmitted.

Discussion:  none

Measurement units:  cell loss, delay

      Issues:

      See also: Units: dimensionless.

3.  Security Considerations
   Security issues are not addressed in this memo.

4.  References

[AF-TM4.0] ATM Forum, Traffic Management Specification Version
4.0, af-tm-0056.00, April 1996.

[AF-UNI3.1] ATM Forum, User Network Interface Specification
Version 3.1, September 1994.

5. Editor's Address

   Robert Craig
   Cisco Systems
   7025 Kit Creek Road
   PO Box 14987
   Research Triangle Park, NC 27709
   (919) 472-2886
   rcraig@cisco.com Addresses

Jeffrey Dunn
Hewlett-Packard
3701 Koppers St.
Baltimore, MD 21227 USA

Phone: +1 (410) 362-7612
E-mail:  jeff_dunn@hp.com

Cynthia Martin
Hewlett-Packard
3701 Koppers St.
Baltimore, MD 21227

Phone +1 (410) 362-7631
E-mail:  cynthia_martin@hp.com