draft-ietf-bmwg-reset-01.txt   draft-ietf-bmwg-reset-02.txt 
Benchmarking Methodology WG Rajiv Asati Benchmarking Methodology WG Rajiv Asati
Internet Draft Cisco Internet Draft Cisco
Updates: 2544 (if approved) Carlos Pignataro Updates: 2544 (if approved) Carlos Pignataro
Intended status: Informational Cisco Intended status: Informational Cisco
Expires: January 2011 Fernando Calabria Expires: April 2011 Fernando Calabria
Cisco Cisco
Cesar Olvera Cesar Olvera
Consulintel Consulintel
July 9, 2010 October 10, 2010
Device Reset Characterization Device Reset Characterization
draft-ietf-bmwg-reset-01 draft-ietf-bmwg-reset-02
Abstract Abstract
An operational forwarding device may need to be re-started An operational forwarding device may need to be re-started
(automatically or manually) for a variety of reasons, an event that (automatically or manually) for a variety of reasons, an event that
we call a "reset" in this document. Since there may be an we call a "reset" in this document. Since there may be an
interruption in the forwarding operation during a reset, it is interruption in the forwarding operation during a reset, it is
useful to know how long a device takes to begin forwarding packets useful to know how long a device takes to resume the forwarding
again. operation.
This document specifies a methodology for characterizing reset This document specifies a methodology for characterizing reset (and
during benchmarking of forwarding devices, and provides clarity and recovery time) during benchmarking of forwarding devices, and
consistency in reset test procedures beyond what's specified in provides clarity and consistency in reset test procedures beyond
RFC2544. It therefore updates RFC2544. what's specified in RFC2544. It therefore updates RFC2544.
Status of this Memo Status of this Memo
This Internet-Draft is submitted in full conformance with the This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79. provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet- other groups may also distribute working documents as Internet-
Drafts. Drafts.
skipping to change at page 2, line 10 skipping to change at page 2, line 10
Internet-Drafts are draft documents valid for a maximum of six Internet-Drafts are draft documents valid for a maximum of six
months and may be updated, replaced, or obsoleted by other documents months and may be updated, replaced, or obsoleted by other documents
at any time. It is inappropriate to use Internet-Drafts as at any time. It is inappropriate to use Internet-Drafts as
reference material or to cite them other than as "work in progress." reference material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt http://www.ietf.org/ietf/1id-abstracts.txt
The list of Internet-Draft Shadow Directories can be accessed at The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html http://www.ietf.org/shadow.html
This Internet-Draft will expire on January 9, 2011. This Internet-Draft will expire on April 10, 2011.
Copyright Notice Copyright Notice
Copyright (c) 2010 IETF Trust and the persons identified as the Copyright (c) 2010 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with carefully, as they describe your rights and restrictions with
respect to this document. Code Components extracted from this respect to this document. Code Components extracted from this
document must include Simplified BSD License text as described in document must include Simplified BSD License text as described in
Section 4.e of the Trust Legal Provisions and are provided without Section 4.e of the Trust Legal Provisions and are provided without
warranty as described in the Simplified BSD License. warranty as described in the Simplified BSD License.
Table of Contents Table of Contents
1. Introduction...................................................4 1. Introduction...................................................4
1.1. Scope.....................................................4 1.1. Scope.....................................................4
2. Key Words to Reflect Requirements..............................4 1.2. Recovery Time Measurement Methods.........................4
3. Test Requirements..............................................5 1.3. Reporting Format..........................................5
4. Reset Test.....................................................6 2. Key Words to Reflect Requirements..............................7
4.1. Hardware Reset............................................6 3. Test Requirements..............................................7
4.1.1. Routing Processor (RP) / Routing Engine reset........7 4. Reset Test.....................................................8
4.1.1.1. RP Reset for a single-RP device (REQUIRED)......7 4.1. Hardware Reset............................................8
4.1.1. Routing Processor (RP) / Routing Engine reset........9
4.1.1.1. RP Reset for a single-RP device (REQUIRED)......9
4.1.1.2. RP Switchover for a multiple-RP device (OPTIONAL) 4.1.1.2. RP Switchover for a multiple-RP device (OPTIONAL)
.........................................................9 ........................................................10
4.1.2. Line Card (LC) Removal and Insertion (REQUIRED).....11 4.1.2. Line Card (LC) Removal and Insertion (REQUIRED).....11
4.2. Software Reset...........................................13 4.2. Software Reset...........................................12
4.2.1. Operating System (OS) reset (REQUIRED)..............13 4.2.1. Operating System (OS) reset (REQUIRED)..............12
4.2.2. Process reset (OPTIONAL)............................15 4.2.2. Process reset (OPTIONAL)............................13
4.3. Power interruption.......................................17 4.3. Power interruption.......................................14
4.3.1. Power Interruption (REQUIRED).......................17 4.3.1. Power Interruption (REQUIRED).......................14
5. Security Considerations.......................................19 5. Security Considerations.......................................15
6. IANA Considerations...........................................20 6. IANA Considerations...........................................16
7. Acknowledgments...............................................20 7. Acknowledgments...............................................16
8. References....................................................21 8. References....................................................17
8.1. Normative References.....................................21 8.1. Normative References.....................................17
8.2. Informative References...................................21 8.2. Informative References...................................17
Authors' Addresses...............................................22 Authors' Addresses...............................................18
1. Introduction 1. Introduction
An operational forwarding device (or one of its components) may need An operational forwarding device (or one of its components) may need
to be re-started for a variety of reasons, an event that we call a to be re-started for a variety of reasons, an event that we call a
"reset" in this draft. Since there may be an interruption in the "reset" in this draft. Since there may be an interruption in the
forwarding operation during a reset, it is useful to know how long a forwarding operation during a reset, it is useful to know how long a
device takes to begin forwarding packets again. device takes to resume the forwarding operation. In other words, it
is desired to know how long the recovery time after the reset is.
However, the answer to this question is no longer simple and However, the answer to this question is no longer simple and
straight-forward as the modern forwarding devices employ many straight-forward as the modern forwarding devices employ many
hardware advancements (distributed forwarding, etc.) and software hardware advancements (distributed forwarding, etc.) and software
advancements (graceful restart, etc.) that influence the recovery advancements (graceful restart, etc.) that influence the recovery
time after the reset. time after the reset.
1.1. Scope 1.1. Scope
This document specifies a methodology for characterizing reset This document specifies a methodology for characterizing reset (and
during benchmarking of forwarding devices, and provides clarity and recovery time) during benchmarking of forwarding devices, and
consistency in reset procedures beyond what is specified in provides clarity and consistency in reset procedures beyond what is
[RFC2544]. These procedures may be used by other benchmarking specified in [RFC2544]. Software upgrades incur additional
documents such as [RFC2544], [RFC5180], [RFC5695], etc. benchmarking complexities and are outside the scope of this
document.
These procedures may be used by other benchmarking documents such as
[RFC2544], [RFC5180], [RFC5695], etc., and is expected that other
protocol-specific benchmarking documents would reference this
document for reset recovery time characterization.
This document updates Section 26.6 of [RFC2544]. This document updates Section 26.6 of [RFC2544].
This document focuses on only the reset criterion of benchmarking, This document focuses on only the reset criterion of benchmarking,
and presumes that it would be beneficial to [RFC5180], [RFC5695], and presumes that it would be beneficial to [RFC5180], [RFC5695],
and other BMWG benchmarking efforts. and other BMWG benchmarking efforts.
1.2. Recovery Time Measurement Methods
The 'recovery time' is the time during which the traffic forwarding
is temporarily interrupted following a reset event. Strictly
speaking, this is the time over which one or more frames are lost.
This definition is similar to that of 'Loss of connectivity period'
defined in [IGPConv] section 4.
There are two accepted methods to measure the 'recovery time':
1. Frame-Loss Method - This method requires testtool capability to
monitor the number of lost frames. In this method, the offered
stream rate (frames per second) must be known. The recovery
time is calculated per the below equation:
Recovery_time = Frames_lost (packets) / Offered_rate
(packets per second)
2. Time-Stamp Method - This method requires testtool capability to
timestamp each frame. In this method, the testtool timestamps
each transmitted frame and monitors the received frame's
timestamp. During the test, the test tool would record the
timestamp (Timestamp A) of the frame that was last received
prior to the reset interruption and the timestamp (Timestamp
B) of the first frame after the interruption stopped. The
difference between Timestamp B and Timestamp A is the recovery
time.
The tester/operator MAY use either method for recovery time
measurement depending on the testtool capability. However, the
Frame-loss method SHOULD be used if the test tool is capable of (a)
counting the number of lost frames per stream, and (b) transmitting
test frame despite the physical link status, whereas Time-stamp
method SHOULD be used if the test tool is capable of (a) time-
stamping each frame, (b) monitoring received frame's timestamp, and
(c) transmitting frames only if the physical link status is UP.
1.3. Reporting Format
All reset results are reported in a simple statement including the
frame loss and recovery times.
For each test case, it is RECOMMENDED that the following parameters
be reported in these units:
Parameter Units or Examples
Throughput Frames per second and bits per
second
Loss (average) Frames
Recovery Time (average) Milliseconds
Number of trials Integer count
Protocol IPv4, IPv6, MPLS, etc.
Frame Size Octets
Port Media Ethernet, GigE (Gigabit Ethernet),
POS (Packet over SONET), etc.
Port Speed 10 Gbps, 1 Gbps, 100 Mbps, etc.
Interface Encap. Ethernet, Ethernet VLAN,
PPP, HDLC, etc.
For mixed protocol environments, frames SHOULD be distributed
between all the different protocols. The distribution MAY
approximate the network conditions of deployment. IN all cases the
details of the mixed protocol distribution MUST be included in the
reporting.
Additionally, the DUT and test bed provisioning, configuration, and
deployed methodologies that may influence the overall recovery time
MUST be listed. (Refer to the additional factors listed in Section
3).
The reporting of results MUST regard repeatability considerations
from Section 4 of [RFC2544]. It is RECOMMENDED to perform multiple
trials and report average results.
2. Key Words to Reflect Requirements 2. Key Words to Reflect Requirements
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in BCP 14, RFC 2119 document are to be interpreted as described in BCP 14, RFC 2119
[RFC2119]. RFC 2119 defines the use of these key words to help make [RFC2119]. RFC 2119 defines the use of these key words to help make
the intent of standards track documents as clear as possible. While the intent of standards track documents as clear as possible. While
this document uses these keywords, this document is not a standards this document uses these keywords, this document is not a standards
track document. track document.
skipping to change at page 5, line 44 skipping to change at page 8, line 8
10. Performance - Redundancy strategy deployed for route 10. Performance - Redundancy strategy deployed for route
processors and line cards processors and line cards
11. Performance - Interface encapsulation as well as achievable 11. Performance - Interface encapsulation as well as achievable
Throughput [RFC2544] Throughput [RFC2544]
12. Any other internal or external factor that may influence 12. Any other internal or external factor that may influence
recovery time after a hardware or software reset recovery time after a hardware or software reset
After the tests are run, one of the characterization results The recovery time is one of the key characterization results
reported is the recovery time. While the recovery time during a reported after each test run. While the recovery time during a reset
reset test event may be zero, there may still be effects on traffic, test event may be zero, there may still be effects on traffic, such
such as transient delay variation or increased latency. However, as transient delay variation or increased latency. However, that is
that is not covered and deemed outside the scope of this document. not covered and deemed outside the scope of this document. In this
In this case, only "no loss" is reported. case, only "no loss" is reported.
4. Reset Test 4. Reset Test
This section contains the description of the tests that are related This section contains the description of the tests that are related
to the characterization of the time needed for DUTs (Device Under to the characterization of the time needed for DUTs (Device Under
Test) / SUTs (System Under Test) to recover from a reset. There are Test) / SUTs (System Under Test) to recover from a reset. There are
three types of reset considered in this document: three types of reset considered in this document:
1. Hardware resets 1. Hardware resets
skipping to change at page 7, line 35 skipping to change at page 9, line 46
available, or perform additional Operating System or hardware available, or perform additional Operating System or hardware
related task. related task.
Second, ensure that the DUT is able to forward the traffic for at Second, ensure that the DUT is able to forward the traffic for at
least 15 seconds before any test activities are performed. The least 15 seconds before any test activities are performed. The
traffic should use the minimum frame size possible on the media traffic should use the minimum frame size possible on the media
used in the testing and rate should be sufficient for the DUT to used in the testing and rate should be sufficient for the DUT to
attain the maximum forwarding throughput. This enables a finer attain the maximum forwarding throughput. This enables a finer
granularity in the recovery time measurement. granularity in the recovery time measurement.
Modern traffic generators support the feature of transmitting
packets while ignoring the link status. The operator / tester MUST
ensure that this feature is enabled.
Third, perform the Route Processor (RP) hardware reset at this Third, perform the Route Processor (RP) hardware reset at this
point. This entails for example physically removing the RP to point. This entails for example physically removing the RP to
later re-insert it, or triggering a hardware reset by other means later re-insert it, or triggering a hardware reset by other means
(e.g., command line interface, physical switch, etc.) (e.g., command line interface, physical switch, etc.)
Finally, the characterization is completed by recording the frame
Finally, the characterization is completed by measuring the loss (as reported by the test tool) and calculating the recovery
complete frame loss and calculating the recovery time from the time (by following the section 1.2).
moment the RP is re-initialized or reinserted until traffic is re-
established, by using the following equation:
Recovery_time = Packet_loss (packets)/Offered_rate (packets per
second).
Reporting format Reporting format
The reset results are reported in a simple statement including the The reporting format is defined in Section 1.3.
frame loss and recovery times.
For each test case, it is RECOMMENDED that the following
parameters be reported in these units:
Parameter Units or Examples
Throughput Frames per second and bits per
second
Loss (average) Frames
Time (average) Seconds, all significant digits and
at least two digits variable over
the trials
Number of trials Integer count
Protocol IPv4, IPv6, MPLS, etc.
Frame Size Octets
Port Media Ethernet, GigE (Gigabit Ethernet),
POS (Packet over SONET), etc.
Port Speed 10 Gbps, 1 Gbps, 100 Mbps, etc.
Interface Encap. Ethernet, Ethernet VLAN,
PPP, HDLC, etc.
Additionally, the DUT and test bed provisioning, configuration,
and deployed methodologies that may influence the overall recovery
time MUST be listed. (Refer to the additional factors listed in
Section 3).
The reporting of results MUST regard repeatability considerations
from Section 4 of [RFC2544]. It is RECOMMENDED to perform multiple
trials and report average results.
4.1.1.2. RP Switchover for a multiple-RP device (OPTIONAL) 4.1.1.2. RP Switchover for a multiple-RP device (OPTIONAL)
Objective Objective
To characterize time needed for "secondary" Route Processor To characterize time needed for "secondary" Route Processor
(sometimes referred to as "backup" RP) of a DUT to become active (sometimes referred to as "backup" RP) of a DUT to become active
after a "primary" (or "active") Route Processor hardware reset. after a "primary" (or "active") Route Processor hardware reset.
This process is often referred to as "RP Switchover". The This process is often referred to as "RP Switchover". The
characterization in this test should be done for the default DUT characterization in this test should be done for the default DUT
skipping to change at page 9, line 42 skipping to change at page 11, line 5
appropriate software files are available, or perform additional appropriate software files are available, or perform additional
Operating System or hardware related task. Operating System or hardware related task.
Second, ensure that the DUT is able to forward the traffic for at Second, ensure that the DUT is able to forward the traffic for at
least 15 seconds before any test activities are performed. The least 15 seconds before any test activities are performed. The
traffic should use the minimum frame size possible on the media traffic should use the minimum frame size possible on the media
used in the testing and rate should be sufficient for the DUT to used in the testing and rate should be sufficient for the DUT to
attain the maximum forwarding throughput. This enables a finer attain the maximum forwarding throughput. This enables a finer
granularity in the recovery time measurement. granularity in the recovery time measurement.
Modern traffic generators support the feature of transmitting
packets while ignoring the link status. The operator / tester MUST
ensure that this feature is enabled.
Third, perform the primary Route Processor (RP) hardware reset at Third, perform the primary Route Processor (RP) hardware reset at
this point. This entails for example physically removing the RP, this point. This entails for example physically removing the RP,
or triggering a hardware reset by other means (e.g., command line or triggering a hardware reset by other means (e.g., command line
interface, physical switch, etc.) It is up to the Operator to interface, physical switch, etc.) It is up to the Operator to
decide if the primary RP needs to be re-inserted after a grace decide if the primary RP needs to be re-inserted after a grace
period or not. period or not.
Finally, the characterization is completed by measuring the Finally, the characterization is completed by recording the frame
complete frame loss and calculating the recovery time from the loss (as reported by the test tool) and calculating the recovery
moment the active RP is hardware-reset until traffic is re- time (by following the section 1.2).
established, by using the following equation:
Recovery_time = Packet_loss (packets)/Offered_rate (packets per
second).
Reporting format Reporting format
The reset results are potentially reported twice, one for the The reset results are potentially reported twice, one for the
default switchover behavior (i.e., the DUT without any switchover- default switchover behavior (i.e., the DUT without any switchover-
specific enhanced configuration) and the other for the switchover- specific enhanced configuration) and the other for the switchover-
specific behavior if it exists (i.e., the DUT configured for specific behavior if it exists (i.e., the DUT configured for
optimized switchover capabilities that minimize the downtime optimized switchover capabilities that minimize the downtime
during the actual switchover). For each, the report consists of a during the actual switchover).
simple statement including the frame loss and recovery times, as
well as any specific redundancy scheme in place.
For each test case, it is RECOMMENDED that the following
parameters be reported in these units:
Parameter Units or Examples
Throughput Frames per second and bits per
second
Loss (average) Frames
Time (average) Seconds, all significant digits and
at least two digits variable over
the trials
Number of trials Integer count
Protocol IPv4, IPv6, MPLS, etc.
Frame Size Octets
Port Media Ethernet, GigE (Gigabit Ethernet),
POS (Packet over SONET), etc.
Port Speed 10 Gbps, 1 Gbps, 100 Mbps, etc.
Interface Encap. Ethernet, Ethernet VLAN,
PPP, HDLC, etc.
Additionally, the DUT and test bed provisioning, configuration,
and deployed methodologies that may influence the overall recovery
time MUST be listed. (Refer to the additional factors listed in
Section 3).
The reporting of results MUST regard repeatability considerations The reporting format is defined in Section 1.3, and also includes
from Section 4 of [RFC2544]. It is RECOMMENDED to perform multiple any specific redundancy scheme in place.
trials and report average results.
4.1.2. Line Card (LC) Removal and Insertion (REQUIRED) 4.1.2. Line Card (LC) Removal and Insertion (REQUIRED)
The Line Card (LC) is the DUT component that is responsible with The Line Card (LC) is the DUT component that is responsible with
packet forwarding. packet forwarding.
Objective Objective
To characterize time needed for a DUT to recover from a Line Card To characterize time needed for a DUT to recover from a Line Card
removal and insertion event. removal and insertion event.
skipping to change at page 11, line 42 skipping to change at page 12, line 12
ensure the appropriate software files are available, or perform ensure the appropriate software files are available, or perform
additional Operating System or hardware related task. additional Operating System or hardware related task.
Second, ensure that the DUT is able to forward the traffic for at Second, ensure that the DUT is able to forward the traffic for at
least 15 seconds before any test activities are performed. The least 15 seconds before any test activities are performed. The
traffic should use the minimum frame size possible on the media traffic should use the minimum frame size possible on the media
used in the testing and rate should be sufficient for the DUT to used in the testing and rate should be sufficient for the DUT to
attain the maximum forwarding throughput. This enables a finer attain the maximum forwarding throughput. This enables a finer
granularity in the recovery time measurement. granularity in the recovery time measurement.
Modern traffic generators support the feature of transmitting
packets while ignoring the link status. The operator / tester MUST
ensure that this feature is enabled.
Third, perform the Line Card (LC) hardware reset at this point. Third, perform the Line Card (LC) hardware reset at this point.
This entails for example physically removing the LC to later re- This entails for example physically removing the LC to later re-
insert it, or triggering a hardware reset by other means (e.g., insert it, or triggering a hardware reset by other means (e.g.,
command line interface (CLI), physical switch, etc.). However, command line interface (CLI), physical switch, etc.). However,
most accurate results will be obtained using the CLI or a physical most accurate results will be obtained using the CLI or a physical
switch, and therefore these are RECOMMENDED. Otherwise, the time switch, and therefore these are RECOMMENDED. Otherwise, the time
spent trying to physically seat the LC will get mixed into the spent trying to physically seat the LC will get mixed into the
results. results.
Finally, the characterization is completed by measuring the Finally, the characterization is completed by recording the frame
complete frame loss and calculating the recovery time from the loss (as reported by the test tool) and calculating the recovery
moment the LC is reinitialized or reinserted until traffic is re- time (by following the section 1.2).
established, by using the following equation:
Recovery_time = Packet_loss (packets)/Offered_rate (packets per
second).
Reporting Format Reporting Format
The reset results are reported in a simple statement including the The reporting format is defined in Section 1.3.
frame loss and recovery times.
For each test case, it is RECOMMENDED that the following
parameters be reported in these units:
Parameter Units or Examples
Throughput Frames per second and bits per
second
Loss (average) Frames
Time (average) Seconds, all significant digits and
at least two digits variable over
the trials
Number of trials Integer count
Protocol IPv4, IPv6, MPLS, etc.
Frame Size Octets
Port Media Ethernet, GigE (Gigabit Ethernet),
POS (Packet over SONET), etc.
Port Speed 10 Gbps, 1 Gbps, 100 Mbps, etc.
Interface Encap. Ethernet, Ethernet VLAN,
PPP, HDLC, etc.
Additionally, the DUT and test bed provisioning, configuration,
and deployed methodologies that may influence the overall recovery
time MUST be listed. (Refer to the additional factors listed in
Section 3).
The reporting of results MUST regard repeatability considerations
from Section 4 of [RFC2544]. It is RECOMMENDED to perform multiple
trials and report average results.
4.2. Software Reset 4.2. Software Reset
To characterize time needed for a DUT to recover from the software To characterize time needed for a DUT to recover from the software
reset. reset.
In contrast to a "hardware reset", a "software reset" involves only In contrast to a "hardware reset", a "software reset" involves only
the re-initialization of the execution, data structures, and partial the re-initialization of the execution, data structures, and partial
state within the software running on the DUT module(s). state within the software running on the DUT module(s).
skipping to change at page 14, line 5 skipping to change at page 13, line 16
parameters, ensure the appropriate software files are available, parameters, ensure the appropriate software files are available,
or perform additional Operating System task. or perform additional Operating System task.
Second, ensure that the DUT is able to forward the traffic for at Second, ensure that the DUT is able to forward the traffic for at
least 15 seconds before any test activities are performed. The least 15 seconds before any test activities are performed. The
traffic should use the minimum frame size possible on the media traffic should use the minimum frame size possible on the media
used in the testing and rate should be sufficient for the DUT to used in the testing and rate should be sufficient for the DUT to
attain the maximum forwarding throughput. This enables a finer attain the maximum forwarding throughput. This enables a finer
granularity in the recovery time measurement. granularity in the recovery time measurement.
Modern traffic generators support the feature of transmitting
packets while ignoring the link status. The operator / tester MUST
ensure that this feature is enabled.
Third, trigger an Operating System re-initialization in the DUT, Third, trigger an Operating System re-initialization in the DUT,
by operational means such as use of the DUT's Command Line by operational means such as use of the DUT's Command Line
Interface (CLI) or other management interface. Interface (CLI) or other management interface.
Finally, the characterization is completed by measuring the Finally, the characterization is completed by recording the frame
complete frame loss and calculating the recovery time from the loss (as reported by the test tool) and calculating the recovery
moment the reset instruction was given until the Operating System time (by following the section 1.2).
finished the reload and re-initialization, inferred by the re-
establishing of traffic, by using the following equation:
Recovery_time = Packet_loss (packets)/Offered_rate (packets per
second).
Reporting format Reporting format
The reset results are reported in a simple statement including the The reporting format is defined in Section 1.3.
frame loss and recovery times.
For each test case, it is RECOMMENDED that the following
parameters be reported in these units:
Parameter Units or Examples
Throughput Frames per second and bits per
second
Loss (average) Frames
Time (average) Seconds, all significant digits and
at least two digits variable over
the trials
Number of trials Integer count
Protocol IPv4, IPv6, MPLS, etc.
Frame Size Octets
Port Media Ethernet, GigE (Gigabit Ethernet),
POS (Packet over SONET), etc.
Port Speed 10 Gbps, 1 Gbps, 100 Mbps, etc.
Interface Encap. Ethernet, Ethernet VLAN,
PPP, HDLC, etc.
Additionally, the DUT and test bed provisioning, configuration,
and deployed methodologies that may influence the overall recovery
time MUST be listed. (Refer to the additional factors listed in
Section 3).
The reporting of results MUST regard repeatability considerations
from Section 4 of [RFC2544]. It is RECOMMENDED to perform multiple
trials and report average results.
4.2.2. Process reset (OPTIONAL) 4.2.2. Process reset (OPTIONAL)
Objective Objective
To characterize time needed for a DUT to recover from a software To characterize time needed for a DUT to recover from a software
process reset. process reset.
Such time period may depend upon the number and types of process Such time period may depend upon the number and types of process
running in the DUT and which ones are tested. Different running in the DUT and which ones are tested. Different
skipping to change at page 16, line 5 skipping to change at page 14, line 12
environmental variables, or perform additional Operating System environmental variables, or perform additional Operating System
task. task.
Second, ensure that the DUT is able to forward the traffic for at Second, ensure that the DUT is able to forward the traffic for at
least 15 seconds before any test activities are performed. The least 15 seconds before any test activities are performed. The
traffic should use the minimum frame size possible on the media traffic should use the minimum frame size possible on the media
used in the testing and rate should be sufficient for the DUT to used in the testing and rate should be sufficient for the DUT to
attain the maximum forwarding throughput. This enables a finer attain the maximum forwarding throughput. This enables a finer
granularity in the recovery time measurement. granularity in the recovery time measurement.
Modern traffic generators support the feature of transmitting
packets while ignoring the link status. The operator / tester MUST
ensure that this feature is enabled.
Third, trigger a process reset for each process running in the DUT Third, trigger a process reset for each process running in the DUT
and considered for testing from a management interface (e.g., by and considered for testing from a management interface (e.g., by
means of the Command Line Interface (CLI), etc.) means of the Command Line Interface (CLI), etc.)
Finally, the characterization for each individual process is Finally, the characterization is completed by recording the frame
completed by measuring the complete frame loss and calculating the loss (as reported by the test tool) and calculating the recovery
recovery time from the moment the reset instruction was given time (by following the section 1.2).
until the Operating System finished the reload and re-
initialization, inferred by the re-establishing of traffic, by
using the following equation:
Recovery_time = Packet_loss (packets)/Offered_rate (packets per
second).
Reporting format Reporting format
The reset results are reported in a simple statement including the The reporting format is defined in Section 1.3, and is used for
frame loss and recovery times for each process running in the DUT each process running in the DUT and tested. Given the
and tested. Given the implementation nature of this test, details implementation nature of this test, details of the actual process
of the actual process tested should be included along with the tested should be included along with the statement.
statement.
For each test case, it is RECOMMENDED that the following
parameters be reported in these units:
Parameter Units or Examples
Throughput Frames per second and bits per
second
Loss (average) Frames
Time (average) Seconds, all significant digits and
at least two digits variable over
the trials
Number of trials Integer count
Protocol IPv4, IPv6, MPLS, etc.
Frame Size Octets
Port Media Ethernet, GigE (Gigabit Ethernet),
POS (Packet over SONET), etc.
Port Speed 10 Gbps, 1 Gbps, 100 Mbps, etc.
Interface Encap. Ethernet, Ethernet VLAN,
PPP, HDLC, etc.
Additionally, the DUT and test bed provisioning, configuration,
and deployed methodologies that may influence the overall recovery
time MUST be listed. (Refer to the additional factors listed in
Section 3).
The reporting of results MUST regard repeatability considerations
from Section 4 of [RFC2544]. It is RECOMMENDED to perform multiple
trials and report average results.
4.3. Power interruption 4.3. Power interruption
"Power interruption" refers to the complete loss of power on the "Power interruption" refers to the complete loss of power on the
DUT. It can be viewed as a special case of a hardware reset, DUT. It can be viewed as a special case of a hardware reset,
triggered by the loss of the power supply to the DUT or its triggered by the loss of the power supply to the DUT or its
components, and is characterized by the re-initialization of all components, and is characterized by the re-initialization of all
hardware and software in the DUT. hardware and software in the DUT.
4.3.1. Power Interruption (REQUIRED) 4.3.1. Power Interruption (REQUIRED)
skipping to change at page 18, line 16 skipping to change at page 15, line 21
software files are available, or perform additional Operating software files are available, or perform additional Operating
System or hardware related task. System or hardware related task.
Second, ensure that the DUT is able to forward the traffic for at Second, ensure that the DUT is able to forward the traffic for at
least 15 seconds before any test activities are performed. The least 15 seconds before any test activities are performed. The
traffic should use the minimum frame size possible on the media traffic should use the minimum frame size possible on the media
used in the testing and rate should be sufficient for the DUT to used in the testing and rate should be sufficient for the DUT to
attain the maximum forwarding throughput. This enables a finer attain the maximum forwarding throughput. This enables a finer
granularity in the recovery time measurement. granularity in the recovery time measurement.
Modern traffic generators support the feature of transmitting
packets while ignoring the link status. The operator / tester MUST
ensure that this feature is enabled.
Third, interrupt the power (AC or DC) that feeds the corresponding Third, interrupt the power (AC or DC) that feeds the corresponding
DUTs power supplies at this point. This entails for example DUTs power supplies at this point. This entails for example
physically removing the power supplies in the DUT to later re- physically removing the power supplies in the DUT to later re-
insert them, or simply disconnecting or switching off their power insert them, or simply disconnecting or switching off their power
feeds (AC or DC as applicable). The actual power interruption feeds (AC or DC as applicable). The actual power interruption
should last at least 15 seconds. should last at least 15 seconds.
Finally, the characterization is completed by measuring the Finally, the characterization is completed by recording the frame
complete frame loss and calculating the recovery time from the loss (as reported by the test tool) and calculating the recovery
moment the power is restored or the power supplies reinserted in time (by following the section 1.2).
the DUT until traffic is re-established, by using the following
equation:
Recovery_time = Packet_loss (packets)/Offered_rate (packets per
second).
For easier comparison with other testing, the 15 seconds are For easier comparison with other testing, the 15 seconds are
removed from the reported recovery time. removed from the reported recovery time.
Reporting format Reporting format
The reset results are reported in a simple statement including the The reporting format is defined in Section 1.3.
frame loss and recovery times.
For each test case, it is RECOMMENDED that the following
parameters be reported in these units:
Parameter Units or Examples
Throughput Frames per second and bits per
second
Loss (average) Frames
Time (average) Seconds, all significant digits and
at least two digits variable over
the trials
Number of trials Integer count
Protocol IPv4, IPv6, MPLS, etc.
Frame Size Octets
Port Media Ethernet, GigE (Gigabit Ethernet),
POS (Packet over SONET), etc.
Port Speed 10 Gbps, 1 Gbps, 100 Mbps, etc.
Interface Encap. Ethernet, Ethernet VLAN,
PPP, HDLC, etc.
Additionally, the DUT and test bed provisioning, configuration,
and deployed methodologies that may influence the overall recovery
time MUST be listed. (Refer to the additional factors listed in
Section 3).
The reporting of results MUST regard repeatability considerations
from Section 4 of [RFC2544]. It is RECOMMENDED to perform multiple
trials and report average results.
5. Security Considerations 5. Security Considerations
Benchmarking activities, as described in this memo, are limited to Benchmarking activities, as described in this memo, are limited to
technology characterization using controlled stimuli in a laboratory technology characterization using controlled stimuli in a laboratory
environment, with dedicated address space and the constraints environment, with dedicated address space and the constraints
specified in the sections above. specified in the sections above.
The benchmarking network topology will be an independent test setup The benchmarking network topology will be an independent test setup
and MUST NOT be connected to devices that may forward the test and MUST NOT be connected to devices that may forward the test
skipping to change at page 20, line 24 skipping to change at page 16, line 24
this document. this document.
6. IANA Considerations 6. IANA Considerations
There is no IANA consideration for this document. There is no IANA consideration for this document.
7. Acknowledgments 7. Acknowledgments
The authors would like to thank Ron Bonica, who motivated us to The authors would like to thank Ron Bonica, who motivated us to
write this document. The authors would also like to thank Al Morton, write this document. The authors would also like to thank Al Morton,
Andrew Yourtchenko, David Newman, and John E Dawson for providing Andrew Yourtchenko, David Newman, John E. Dawson, Timmons C. Player,
thorough review, useful suggestions, and valuable input. Jan Novak, Steve Maxwell, and Ilya Varlashkin for providing thorough
review, useful suggestions, and valuable input.
This document was prepared using 2-Word-v2.0.template.dot. This document was prepared using 2-Word-v2.0.template.dot.
8. References 8. References
8.1. Normative References 8.1. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997. Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC2544] Bradner, S. and McQuaid, J., "Benchmarking Methodology for [RFC2544] Bradner, S. and McQuaid, J., "Benchmarking Methodology for
Network Interconnect Devices", RFC 2544, March 1999. Network Interconnect Devices", RFC 2544, March 1999.
8.2. Informative References 8.2. Informative References
[IGPConv] Poretsky, S., Imhoff, B., and K. Michielsen, "Benchmarking
Methodology for Link-State IGP Data Plane Route
Convergence", draft-ietf-bmwg-igp-dataplane-conv-meth-21
(work in progress), May 2010.
[RFC5180] Popoviciu, C., et al, "IPv6 Benchmarking Methodology for [RFC5180] Popoviciu, C., et al, "IPv6 Benchmarking Methodology for
Network Interconnect Devices", RFC 5180, May 2008. Network Interconnect Devices", RFC 5180, May 2008.
[RFC5695] Akhter, A., Asati, R., and C. Pignataro, "MPLS Forwarding [RFC5695] Akhter, A., Asati, R., and C. Pignataro, "MPLS Forwarding
Benchmarking Methodology for IP Flows", RFC 5695, November Benchmarking Methodology for IP Flows", RFC 5695, November
2009. 2009.
Authors' Addresses Authors' Addresses
Rajiv Asati Rajiv Asati
 End of changes. 34 change blocks. 
345 lines changed or deleted 170 lines changed or added

This html diff was produced by rfcdiff 1.40. The latest version is available from http://tools.ietf.org/tools/rfcdiff/