draft-ietf-bmwg-call-01.txt   draft-ietf-bmwg-call-02.txt 
Network Working Group R. Craig Network Working Group J. H. Dunn
INTERNET-DRAFT Cisco Systems INTERNET-DRAFT Hewlett-Packard
Expires in six months March 1997 Expires in six months C. E. Martin
Hewlett-Packard
Terminology for Cell/Call Benchmarking August 1998
<draft-ietf-bmwg-call-01.txt>
Status of this Memo Terminology for Call/Cell Benchmarking
draft-ietf-bmwg-call-02.txt
Status of this Memo
This document is an Internet-Draft. Internet-Drafts are working This document is an Internet-Draft. Internet-Drafts are working
documents of the Internet Engineering Task Force (IETF), its areas, documents of the Internet Engineering Task Force (IETF), its
and its working groups. Note that other groups may also distribute areas, and its working groups. Note that other groups may also
working documents as Internet-Drafts. distribute working documents as Internet-Drafts.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six
and may be updated, replaced, or obsoleted by other documents at any months and may be updated, replaced, or obsoleted by other
time. It is inappropriate to use Internet- Drafts as reference documents at any time. It is inappropriate to use Internet-
material or to cite them other than as ``work in progress.'' Drafts as reference material or to cite them other than as ``work
in progress.''
To learn the current status of any Internet-Draft, please check the To view the entire list of current Internet-Drafts, please check
``1id-abstracts.txt'' listing contained in the Internet- Drafts the "1id-abstracts.txt" listing contained in the Internet-Drafts
Shadow Directories on ftp.is.co.za (Africa), nic.nordu.net (Europe), Shadow Directories on ftp.is.co.za (Africa), ftp.nordu.net
munnari.oz.au (Pacific Rim), ds.internic.net (US East Coast), or (Northern Europe), ftp.nis.garr.it (Southern Europe), munnari.oz.au
ftp.isi.edu (US West Coast). (Pacific Rim), ftp.ietf.org (US East Coast), or ftp.isi.edu
(US West Coast).
Abstract Abstract
The purpose of this draft is to add terminology specific to the cell This memo discusses and defines terms associated with performance
and call-based switch environment to that defined by the Benchmarking benchmarking tests and the results of these tests in the context
of cell-based and call-based switching devices. The terms defined
in this memo will be used in addition to terms defined in RFCs
1242, 1944 and 2285. This memo is a product of the Benchmarking
Methodology Working Group (BMWG) of the Internet Engineering Task Methodology Working Group (BMWG) of the Internet Engineering Task
Force (IETF) in RFC1242. Force (IETF).
While primarily directed towards wide area switches, portions of the I. Background
document may be useful for benchmarking other devices such as ADSU's.
1. Introduction 1. Introduction
In light of the increasing use of cell-based and/or circuit-switched This document provides terminology for benchmarking cell-based and
transport layers in building networks, it would be useful to develop call-based switching devices. It extends terminology already
a set of benchmarks with which to compare technologies, defined for benchmarking network interconnect devices in RFCs
implementation strategies, and products. 1242, 1944 and 2285. Although some of the definitions in this
1.1 Terminology Brought Forward
The terminology defined in RFC 1242 applies equally well to this
memo. There is also a certain amount of overlap with terms
defined in draft-ietf-bmwg-lanswitch-00.txt.
2. Definition Format (from RFC1242)
Term to be defined.
Definition:
The specific definition for the term.
Discussion:
A brief discussion of the term, its application and any
restrictions on measurement procedures.
Measurement units:
Units used to record measurements of this term, if applicable.
3. Term Definitions
3.1 Virtual Circuit
This group applies to those switches that are connection-oriented.
3.1.1 Call setup time
Definition: the length of time for the virtual circuit to be
established.
Discussion: as measured from the initiation of the signalling to
circuit establishment.
Measurement units: fractional seconds
Issues:
See also:
3.1.2 Call setup rate (sustained)
Definition: the maximum sustained rate of successful connection
establishment.
Discussion: without loss of existing calls.
Measurement units: calls per second
Issues:
See also:
3.1.3 Call maintenance overhead
Definition: the amount of work required to maintain the calls
that have been established.
Discussion: a method to obtain the desired result would be to
benchmark with PVC's in place, then with SVC's. The difference in
results would be the overhead.
Measurement units:
Issues:
See also:
3.1.4 Call teardown time
Definition: the length of time for the virtual circuit to be torn
down.
Discussion: measured from the start of the signalling to the
freeing of the resources associated with that call (end to end, if
applicable).
Measurement units: fractional seconds
Issues:
See also:
3.1.5 Call teardown rate (sustained)
Definition: the maximum rate at which calls can be successfully
torn down.
Discussion: without loss of existing calls, and without failure
to tear down any calls that have been signalled to be destroyed.
Measurement units: teardowns per second
Issues:
See also:
3.1.6 Impact of Signalling on Forwarding
Definition: cells per second versus calls per second
Discussion: some devices use the same engine for cell forwarding
and call maintenance. In this case, interaction between the two
functions will be inevitable. More interesting, however, would be
the case where the two processing functions are clearly separate,
yet still interact.
Measurement units: cells per second versus calls per second
Issues:
See also:
3.2 Cell/Packet Interaction
This group applies to cell-based switches, connection-oriented or
not.
3.2.1 Packet disassembly/reassembly time (peak)
Definition: the length of time to disassemble a layer 3 packet
into layer 2 cells, or reassemble cells into a packet.
Discussion: with no packet or cell loss or corruption. To arrive
at a baseline, one could measure the switching rate for cells
derived from ~1440 byte frames which are flowing across the switch
as cells, then forward those same frames into the switch from an
interface which would require them to be disassembled. For
example, the baseline measurement is taken while switching cells
OC3-OC3. Then switch FDDI or POS-OC3 and take the delta in
performance as the SAR overhead.
Measurement units: the appropriate fraction of a second
Issues:
See also:
3.2.2 Packet disassembly/reassembly rate (sustained)
Definition: the maximum sustained rate at which packets can be
disassembled/reassembled into/from cells.
Discussion: without loss or corruption.
Measurement units: packets per second
Issues:
See also:
3.2.3 Full packet drop rate (on cell loss)
Definition: the rate at which cell loss triggering full packet
drop can be detected/sustained.
Discussion: When a packet is disassembled into cells, typically
many cells result. When these cells are transmitted, they are
subject to loss or corruption. The device should recognize at the
cell/packet boundary that a cell or cells belonging to a given
packet has been lost and should drop that packet, immediately
freeing those resources. A couple of things are of interest here:
whether the switch is able to detect very small amounts of cell
loss and correctly drop the associated packets and whether large
amounts of cell loss perturb this ability in any way.
Measurement units: (dropped) packets per second
Issues:
See also:
3.2.4 End to end data integrity
Definition: the percentage of packets (post-reassembly) that
actually contain undetected data link layer corruption.
Discussion: some network devices have been known to regenerate
CRC's over the re-assembled packet (i.e., the CRC is not carried
end to end), resulting in undetected data link layer corruption or
re-ordering of cells in a packet.
Measurement units: percentage
Issues: production of a stream of traffic containing internal
checksums sufficiently strong to detect cell re-ordering (the IP
checksum is not). The ISIS LSP checksum is.
See also:
3.3 Switch Fabric
This group applies to all switches.
3.3.2 Topology Table Size
Definition: number of network elements supported.
Discussion: switches may support a limited topology due to static
table sizes or processing limitations. This is true whether it's
a "LAN" switch running spanning tree or a "WAN" switch running
OSPF. The effect of a limited topology table on a switch in a
real-world environment can be disastrous.
A similar metric (2.14 Address handling) is mentioned in "draft-
ietf-bmwg-lanswitch-00.txt". Here, a more general metric is
intended.
Measurement units: number
Issues: Measuring the effects of an overflow is probably
meaningless, since in the multi-switch case, there is no longer
any network to speak of, hence, nothing to measure.
If a device handles table overflow gracefully, this should be
noted. Similarly, if a device crashes and burns on table
overflow, this should be noted.
See also:
3.3.3 Topology Table Learning Rate
Definition: the rate at which the topology table can be filled or
updated.
Discussion: a single switch in isolation learning MAC addresses
will flood frames when the rate exceeds its learning capability.
This metric is covered in "2.15 Address learning speed" of
"draft-ietf-bmwg-lanswitch-00.txt". We generalize the metric here
to include the topological databases of routing protocols used in
switched networks (among the switches themselves) as well as the
spanning tree recalculation among multiple LAN switches.
Measurement units: frames per second 1) with maximum diversity of
addresses, 2) with routing instability introduced.
Issues:
See also:
3.3.5 Throughput (from RFC1242) (Cell forwarding rate)
Definition: The maximum rate at which none of the offered frames
are dropped by the device.
Discussion: This metric probably overlaps work being done in the
ATM Forum.
Measurement units: cells per second
Issues:
See also:
3.3.6 Blocking Probability
Definition: likelihood of successful simultaneous communication
amongst multiple ports.
Discussion: a switch is termed "non-blocking" if multiple ports memo may be applicable to a broader group of network interconnect
are able to communicate across the switch fabric at the same time. devices, the primary focus of the terminology in this memo is on
We are interested in the probability of blocking occurring in the cell-based and call-based switches. Specifically, this includes
1:1, N:1, and N:M scenarios. Asynchronous Transfer Mode (ATM) cell relay and signaling and
Frame Relay (FR) signaling.
One may calculate the "ideal" throughput in the absence of This memo contains two major sections: Background and Definitions.
blocking, then take the delta with the experimental case and treat Within the definitions section are a formal definitions sub-
that as an empirical measurement of blocking probability, if section, provided as a courtesy to the reader, and a measurement
enough samples are taken. definitions sub-section which contains performance metrics with
inherent units.
Measurement units: percentage likelihood of blocking. 2. Existing Definitions
Issues: RFC 1242 "Benchmarking Terminology for Network Interconnect
Devices" should be consulted before attempting to make use of this
document. RFC 1944 "Benchmarking Methodology for Network
Interconnect Devices" contains discussions of a number of terms
relevant to the benchmarking of switching devices and should also
be consulted. RFC 2285 "Benchmarking Terminology for LAN
Switching Devices" contains a number of terms pertaining to
traffic distributions and datagram interarrival.
For the sake of clarity and continuity this RFC adopts the
template for definitions set out in Section 2 of RFC 1242.
Definitions are indexed and grouped together in sections for ease
of reference.
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL
NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described in
RFC 2119.
See also: II. Definitions
3.5 Congestion Control The definitions presented in this section have been divided into
two groups. The first group is formal definitions, which are
required in the definitions of the performance metrics but are not
themselves strictly metrics. These definitions are subsumed from
other work done in other working groups both inside and outside
the IETF. They are provided as a courtesy to the reader.
This group applies to all switches. 1. Formal Definitions
3.5.1 Congestion avoidance 1.1. Definition Format (from RFC1242)
Definition: effectiveness of measures taken by the switch to Term to be defined.
avoid congestion.
Discussion: connections that are bursting above their committed Definition: The specific definition for the term.
rate may have cells buffered at the ingress, in order to avoid
congestion in the trunks and impact on other connections, or they
may simply be marked "discard-eligible" and forwarded into the
network, hoping for the best.
Distinguishing between these two approaches should be relatively Discussion: A brief discussion of the term, its application and
simple. In the first case, latency for the bursting session any restrictions on measurement procedures.
increases, but there is no cell loss. Other sessions are
unaffected. In the second case, there may be cell loss across any
of the sessions, and latency may increase across all.
Measurement units: dropped cells, latency Specification: The working group and document in which the term
is specified. Listed in the references.
Issues: 1.2 Definitions
See also: 1.2.1 Available Bit Rate (ABR):
3.5.2 Congestion management Definition: ABR is an ATM layer service category for which the
limiting ATM layer transfer characteristics provided by the
network may change subsequent to connection establishment. A flow
control mechanism is specified which supports several types of
feedback to control the source rate in response to changing ATM
layer transfer characteristics.
Definition: effectiveness of measures taken by the switch to deal Discussion: It is expected that an end-system that adapts its
with congestion. traffic in accordance with the feedback will experience a low cell
loss ratio and obtain a fair share of the available bandwidth
according to a network specific allocation policy. Cell delay
variation is not controlled in this service, although admitted
cells are not delayed unnecessarily.
Discussion: in the face of sustained traffic above committed rate Specification: AF-TM4.0
on multiple sessions, a switch has little choice but to begin
discarding cells, since buffering cannot be infinite. This case
might arise if one were wildly profligate in over-subscribing
trunk bandwidth, or if one had neglected to analyze the network
applications to be run over the network and they were found to be
network-hostile (UDP, IPX, AT, NetBIOS, for example).
The switch has some discretion in deciding which cells to drop. 1.2.2 Call-based
Presumably, the strategy should involve something resembling
"fairness".
The basic idea is that ill-behaved connections should not starve Definition: An association between two or more users or between a
others for resources. user and a network entity that is established by the use of
network capabilities. This association may have zero or more
connections.
Measurement units: latency, cell drops Discussion: none
Issues: Specification: AF-UNI3.1
See also: 1.2.3 Cell
3.6 Inter-switch protocols Definition: A unit of transmission in ATM. A fixed size frame
consisting of a 5 octet header and a 48 octet payload.
This group applies to all switches. Discussion: none
3.6.1 Impact of Routing on Forwarding Specification: AF-UNI3.1
Definition: interaction between routing protocol and data 2. Performance Metrics
forwarding operations.
Discussion: No amount of routing fluctuation should have an 2.1. Definition Format (from RFC1242)
impact on data forwarding for unaffected destinations. Similarly,
no amount of data forwarding should cause the routing to become
unstable.
Measurement units: route flaps per second versus cells per Metric to be defined.
second, cells per second versus route stability (table fluctuation
or peer loss).
Issues: Definition: The specific definition for the metric.
See also: Discussion: A brief discussion of the metric, its application and
any restrictions on measurement procedures.
3.6.2 Impact of Congestion Control Measurement units: Intrinsic units used to quantify this metric.
This includes subsidiary units, e.g. microseconds are acceptable
if the intrinsic unit is seconds.
Definition: interaction between congestion control and data 2.2 Definitions
forwarding operations.
Discussion: switches may share views of congestion in-band 2.2.1 Cell Transfer Delay (CTD)
through the network. Should these feedback messages be delayed or
lost, the potential exists for an incorrect picture of current
network conditions, which may exacerbate congestion and lead to
cell loss. Worse, it is possible to enter a stable oscillation
state, where ever-increasing waves of congestion overwhelm the
switches.
Measurement units: Definition: The elapsed time between a cell exit event at the
measurement point 1 (e.g., at the source UNI) and the
corresponding cell entry event at a measurement point 2 (e.g., the
destination UNI) for a particular connection.
Issues: Discussion: The cell transfer delay between two measurement points
is the sum of the total inter-ATM node transmission delay and the
total ATM node processing delay.
See also: Measurement units: seconds
3.7 Quality of Service 2.2.2 Cell Delay Variation (CDV)
This group applies to all switches. Definition: The variation in cell transfer delay associated with
a given traffic load, orientation and distribution, as well as an
integration period. CDV = max(CTD) - min(CTD) where max and min
indicate the maximum and minimum over the integration period,
respectively.
3.7.1 Traffic Management Policing Discussion: CDV is a component of cell transfer delay, induced by
buffering and cell scheduling. Peak-to-peak CDV is a QoS delay
parameter associated with CBR and VBR services. The peak-to-peak
CDV is the ((1-a) quantile of the CTD) minus the fixed CTD that
could be experienced by any delivered cell on a connection during
the entire connection holding time. The parameter "a" is the
probability of a cell arriving late.
Definition: impact of misbehaving class on others, for example Measurement Units: seconds.
data forwarding on voice or video frames and vice versa.
Discussion: we wish to quantify the potential interaction amongst 2.2.3 Cell Error Ratio (CER)
the various classes of service. Constant bit rate (CBR), variable
bit rate (VBR) (real and non-real time?), and available bit rate
(ABR) streams are established, within their respective service
levels, but sufficient to subscribe the trunk to 90%. The bit
rate of each is increased until it has exceeded its allocation by
a degree which should cause loss or delay in the other streams.
Measurement units: cells (lost) per second, latency Definition: The ratio of errored cells in a transmission in
relation to the total cells sent in a transmission associated with
a given traffic load, orientation and distribution, as well as an
integration period. CER = Errored Cells / Total Cells
Transmitted.
Issues: some switches perform compression and silence Discussion: The measurement is taken over a time interval and is
suppression. Should these features be disabled? desirable to be measured on an in-service circuit.
See also: Measurement Units: dimensionless.
3.8 Multicast 2.2.4 Cell Loss Ratio (CLR)
3.8.1 Cell replication Definition: The ratio of lost cells in a transmission in relation
to the total cells sent in a transmission associated with a given
traffic load, orientation and distribution, as well as an
integration period. CLR = Lost Cells / Total Cells Transmitted.
Definition: the device's ability to forward a cell to multiple Discussion: CLR is a negotiated QoS parameter and acceptable
ports simultaneously (multicast). values are network specific. The objective is to minimize CLR
provided the end-system adapts the traffic to the changing ATM
layer transfer characteristics. The CLR parameter is the value of
CLR that the network agrees to offer as an objective over the
lifetime of the connection. It is expressed as an order of
magnitude, having a range of 10-1 to 10-15 and unspecified.
Discussion: Measurement Units: dimensionless.
Measurement units: replication factor 1:N and cells per second 2.2.5 Cell Misinsertion Rate (CMR)
measured at ingress versus cells per second measured at the
egresses.
Issues: Definition: The ratio of cells received at an endpoint that were
not originally transmitted by the source end in relation to the
total number of cells properly transmitted associated with a given
traffic load, orientation and distribution, as well as an
integration period. CMR = Mis-inserted Cells / Total Cells
Transmitted.
See also: Discussion: none
3.8.2 Impact of multicast on unicast Measurement Units: dimensionless.
Definition: switch's ability to insulate unicast traffic from the 3. Security Considerations
effects of multicast. Security issues are not addressed in this memo.
Discussion: a poorly-designed replication scheme could easily 4. References
swamp unicast traffic. Yet, multicast traffic often has QoS
needs. How does one reconcile the competing requirements?
Measurement units: cell loss, delay [AF-TM4.0] ATM Forum, Traffic Management Specification Version
4.0, af-tm-0056.00, April 1996.
Issues: [AF-UNI3.1] ATM Forum, User Network Interface Specification
Version 3.1, September 1994.
See also: 5. Editor's Addresses
Security Considerations Jeffrey Dunn
Hewlett-Packard
3701 Koppers St.
Baltimore, MD 21227 USA
Security issues are not addressed in this memo. Phone: +1 (410) 362-7612
E-mail: jeff_dunn@hp.com
Editor's Address Cynthia Martin
Hewlett-Packard
3701 Koppers St.
Baltimore, MD 21227
Robert Craig Phone +1 (410) 362-7631
Cisco Systems E-mail: cynthia_martin@hp.com
7025 Kit Creek Road
PO Box 14987
Research Triangle Park, NC 27709
(919) 472-2886
rcraig@cisco.com
 End of changes. 

This html diff was produced by rfcdiff 1.23, available from http://www.levkowetz.com/ietf/tools/rfcdiff/