draft-ietf-bmwg-acc-bench-term-00.txt   draft-ietf-bmwg-acc-bench-term-01.txt 
Network Working Group Network Working Group
INTERNET-DRAFT INTERNET-DRAFT
Expires in: December 2003 Expires in: April 2004
Scott Poretsky Scott Poretsky
Avici Systems Quarry Technologies
Shankar Rao Shankar Rao
Qwest Communications Qwest Communications
Ray Piatt Ray Piatt
Cable and Wireless Cable and Wireless
June 2003 October 2003
Terminology for Benchmarking Terminology for Accelerated Stress Benchmarking
Core Router Software Accelerated Life Testing
<draft-ietf-bmwg-acc-bench-term-00.txt> <draft-ietf-bmwg-acc-bench-term-01.txt>
Status of this Memo Status of this Memo
This document is an Internet-Draft and is in full conformance with This document is an Internet-Draft and is in full conformance with
all provisions of Section 10 of RFC2026. all provisions of Section 10 of RFC2026.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet- other groups may also distribute working documents as Internet-
Drafts. Drafts.
skipping to change at page 1, line 45 skipping to change at page 1, line 44
progress." progress."
The list of current Internet-Drafts can be accessed at The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt http://www.ietf.org/ietf/1id-abstracts.txt
The list of Internet-Draft Shadow Directories can be accessed at The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html. http://www.ietf.org/shadow.html.
ABSTRACT ABSTRACT
This terminology document provides the terms to be used for This terminology document provides the terms to be used for
benchmarking router software under accelerated stress conditions. benchmarking a router or switch under Accelerated Stress conditions.
A framework is defined to configure routing protocols, security Terminology is provided for performing this benchmark using
policies, traffic forwarding, and management. Conditions to the Contol Plane, Data Plane, Management Plane, and Security
produce instability and accelerate operational conditions are Plane of the Device Under Test. Terms are defined for
also defined. Benchmarks for evaluating a router subjected to Configuration, Startup Conditions, Instability Conditions, Failure
the accelerated life test are introduced. The DUT configuration Conditions, and Benchmarks.
and accelerated stress conditions emulate those of Internet
Core routers.
Table of Contents Table of Contents
1. Introduction ............................................... 2 1. Introduction ............................................... 3
Router Software Accelerated Life Testing
2. Existing definitions ....................................... 3 2. Existing definitions ....................................... 3
3. Term definitions............................................ 3 3. Term definitions............................................ 3
3.1 Configuration Modules..................................... 3 3.1 Benchmark Planes.......................................... 3
3.1.1 Control Plane Configuration Module.................... 4 3.1.1 Control Plane........................................... 4
3.1.2 Data Plane Configuration Module....................... 5 3.1.2 Data Plane.............................................. 5
3.1.3 Management Configuration Module....................... 6 3.1.3 Management Plane........................................ 5
3.1.4 Security Configuration Module......................... 7 3.1.4 Security Plane.......................................... 6
3.2 Configuration Sets...................................... 7 3.2 Configuration Sets........................................ 6
3.2.1 Control Plane Configuration Sets...................... 7 3.2.1 Control Plane Configuration Set......................... 7
3.2.1.1 Routing Protocol Configuration Set.................. 7 3.2.2 Data Plane Configuration Set............................ 7
3.2.1.2 Multicast Protocol Configuration Set................ 8 3.2.2.1 Traffic Profile....................................... 8
3.2.1.3 MPLS Protocol Configuration Set..................... 9 3.2.3 Management Plane Configuration Set...................... 8
3.2.2 Data Plane Configuration Set.......................... 9 3.2.4 Security Plane Configuration Set........................ 9
3.2.3 Management Configuration Sets.........................10 3.3 Startup Conditions........................................10
3.2.3.1 User Access Configuration Set.......................10 3.3.1 Control Plane Startup Conditions........................10
3.2.3.2 SNMP Configuration Set..............................10 3.3.2 Security Plane Startup Conditions.......................11
3.2.3.3 Logging/Debug Configuration Set.....................11 3.4 Instability Conditions....................................11
3.2.3.4 Packet Statistics Collector Configuration...........11 3.4.1 Control Plane Instability Conditions....................12
3.2.4 Security Configuration Set............................11 3.4.2 Data Plane Instability Conditions.......................12
3.2.4.1 ACL Configuration Set.................................12 3.4.3 Management Plane Instability Conditions.................13
3.3 Instability Conditions..................................12 3.4.4 Security Plane Instability Conditions...................13
3.3.1 Interface Shutdown Cycling Rate.......................12 3.5 Failure Conditions........................................14
3.3.2 BGP Route Flap Rate...................................13 3.5.1 Unexpected Packet Loss..................................14
3.3.3 IGP Route Flap Rate...................................13 3.5.2 Unexpected Session Loss.................................14
3.3.4 Route Convergence Due to Better Next-Hop..............14 3.5.3 Misrouted Packets.......................................15
3.3.5 LSP Reroute Rate......................................14 3.5.4 Access Denial...........................................15
3.3.6 Intended Test Duration................................15 3.5.5 Errored Management Value................................16
3.4 Evaluation Benchmarks...................................15 3.5.6 Security Plane Failure..................................16
3.4.1 Run-Time without Error................................15 3.6 Benchmarks................................................17
3.4.2 Run-Time without Control Plane Error..................16 3.6.1 Success Threshold.......................................17
3.4.3 Run-Time without Data Plane Error.....................16 3.6.2 Accelerated-Life Test Duration..........................17
3.4.4 Run-Time without Management Plane Error...............17
3.4.5 Run-Time without Security Plane Error.................17
4. Security Considerations.....................................18 4. Security Considerations.....................................18
5. References..................................................18 5. References..................................................18
6. Author's Address............................................18 6. Author's Address............................................18
7. Full Copyright Statement....................................19 7. Full Copyright Statement....................................19
Appendix 1 - White Box Benchmarks..............................19 Appendix 1 - White Box Benchmarks..............................20
1. Introduction 1. Introduction
Router testing benchmarks have consistently been made in a Routers in an operational network are simultaneously configured with
monolithic fashion in which a single protocol or behavior is multiple protocols and security policies while forwarding traffic and
measured in an isolated environment. It is important to know the being managed. To accurately benchmark a router for deployment it is
limits for a router/switch's (hereby referred to as Router) behavior
for each protocol, however this does not produce a reliable benchmark
of the router's behavior in a deployed network. Routers in an
operational network are simultaneously configured with multiple
protocols and security policies while forwarding traffic and being
managed. To accurately benchmark a router for deployment it is
necessary to test that router in operational conditions by necessary to test that router in operational conditions by
simultaneously configuring the network protocols and security simultaneously configuring and scaling network protocols and security
policies, sourcing traffic, and managing the router. It is helpful policies, forwarding traffic, and managing the device. It is helpful
Router Software Accelerated Life Testing
to accelerate these network operational conditions so that the to accelerate these network operational conditions so that the
router under test can be benchmarked with faster test duration. router under test can be benchmarked with faster test duration.
Testing a router in accelerated network conditions is known as Testing a router in accelerated network conditions is known as
Accelerated Life Testing. White Box benchmarks are defined in Accelerated Stress Testing.
Appendix 1.
Accelerated Life Testing of Routers provides the following benefits:
1. Evaluation of multiple protocols enabled simultaneously as
configured in deployed networks
2. Evaluation of System and Software Stability
3. Evaluation of Manageability under stressful conditions
4. Identification of Software Coding bugs such as:
a. Memory Leaks
b. Suboptimal CPU Utilization
c. Coding Logic
These evaluations are externally observable to the DUT as control
plane or data plane errors.
Configuration and operational conditions vary for Core, Peering, This document defines the Benchmark Planes used for the Accelerated
Aggregate, and Edge Routers. This document focuses on defining Life Benchmarking. These are the Control Plane, Data Plane,
the terminology for Accelerated Life Testing of Core Routers. Management Plane, and Security Plane. For each plane, the
Terminology for configuring network conditions, accelerating Configuration Set, Startup Conditions, and Instability Conditions.
operational parameters, accelerating network instability, and Terms are also defined for observing Failure Conditions and reporting
evaluating results are provided. Benchmarks. White Box benchmarks are provided in Appendix 1 for
additional DUT behavior measurements.
2. Existing definitions 2. Existing definitions
RFC 1242 "Benchmarking Terminology for Network Interconnect RFC 1242 "Benchmarking Terminology for Network Interconnect
Devices" and RFC 2285 "Benchmarking Terminology for LAN Switching Devices" and RFC 2285 "Benchmarking Terminology for LAN Switching
Devices" should be consulted before attempting to make use of this Devices" should be consulted before attempting to make use of this
document. document.
For the sake of clarity and continuity this RFC adopts the template For the sake of clarity and continuity this RFC adopts the template
for definitions set out in Section 2 of RFC 1242. Definitions are for definitions set out in Section 2 of RFC 1242. Definitions are
indexed and grouped together in sections for ease of reference. indexed and grouped together in sections for ease of reference.
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in
this document are to be interpreted as described in RFC 2119. this document are to be interpreted as described in RFC 2119.
3. Term definitions 3. Term definitions
3.1 Configuration Modules 3.1 Benchmark Planes
Definition: Definition:
The features and protocols enabled for the Accelerated Life The features, conditions, and behavior for the Accelerated Stress
Test and conditions created during the test. Benchmarking.
Discussion: Discussion:
Features and Protocols are divided into four configuration There are four Benchmark Planes: Control Plane, Data Plane,
modules as shown in Figure 1. These are the Control Plane, Management Plane, and Security Plane as shown in Figure 1. The
Data Plane, Management Plane, and Security Plane. Benchmark Planes define the Configuration, Startup Conditions,
Instability Conditions, and Failure Conditions used for the test.
Router Software Accelerated Life Testing
Definitions for each of these modules are provided in the
following sections.
___________ ___________ ___________ ___________
| Control | | Management| | Control | | Management|
| Plane |___ ___| Plane | | Plane |___ ___| Plane |
| Module | | | | Module | | | | | | |
----------- | | ----------- ----------- | | -----------
\/ \/ ___________ \/ \/ ___________
___________ | Security | ___________ | Security |
| |<-----------| Plane | | |<-----------| Plane |
| DUT | | Module | | DUT | | |
|--->| |<---| ----------- |--->| |<---| -----------
| ----------- | | ----------- |
| | | |
| ___________ | | ___________ |
| | Data | | | | Data | |
|--->| Plane |<---| |--->| Plane |<---|
| Module | | |
----------- -----------
Figure 1. Configuration Modules Figure 1. Router Accelerated Stress Benchmarking Planes
Measurement units: N/A Measurement units: N/A
Issues: None Issues: None
See Also: See Also:
Control Plane Configuration Module Control Plane
Data Plane Configuration Module Data Plane
Management Configuration Module Management Plane
Security Configuration Module Security Plane
3.1.1 Control Plane Configuration Module
3.1.1 Control Plane
Definition: Definition:
The control protocols enabled for the Accelerated Life Test. The Description of the control protocols enabled for
the Accelerated Stress Benchmarking.
Discussion: Discussion:
Control Plane Configuration Module is divided into three The Control Plane defines the Configuration, Startup
Configuration Sets as shown in Figure 2. These are Routing Conditions, Instability Conditions, and Failure
Protocol, Multicast, and MPLS. These can be enabled or Conditions of the control protocols used for the test.
disabled for a benchmark test. For example, a network operator Control Plane protocols may include routing protocols,
that operates a network with BGP and an IGP without Multicast multicast protocols, and MPLS protocols. These can be
and MPLS protocols can execute a benchmark test with the enabled or disabled for a benchmark test.
Routing Protocol Configuration Set enabled and the Multicast
and MPLS Configuration Sets disabled. Configuration Sets are
further explained in the next section.
Measurement units: Enabled/Disabled Measurement units: N/A
Issues:None Issues:None
Router Software Accelerated Life Testing
____________ ____________ ____________
| Routing | | Multicast | | MPLS |
| Protocol |___ | Protocol | __| Protocol |
| Config Set | | | Config Set | | | Config Set |
------------ | ------------ | ------------
| | |
| | |
| \/ |
| ___________ |
| | | |
|------->| DUT |<------|
| |
-----------
Figure 2. Control Plane Configuration Module
See Also: See Also:
Routing Protocol Configuration Set Benchmark Planes
Multicast Protocol Configuration Set Control Plane Configuration Set
MPLS Protocol Configuration Set Control Plane Startup Conditions
Control Plane Instability Conditions
3.1.2 Data Plane Configuration Module Control Plane Failure Conditions
3.1.2 Data Plane
Definition: Definition:
The data traffic and interfaces enabled for the Accelerated The data traffic profile used for the Accelerated Stress
Life Test. Benchmarking.
Discussion: Discussion:
Accelerated Life Testing includes protocols, as defined by the The Data Plane defines the Configuration, Startup
Control Plane Configuration Module, and data forwarding, as Conditions, Instability Conditions, and Failure
defined by the Data Plane Configuration Module. As shown in Conditions of the data traffic used for the test.
Figure 3, the Data Plane Configuration Module has a The Data Plane includes the traffic and interface
corresponding Traffic Profile Configuration Set. profile.
___________
| Traffic |
| Profile |___
| Config Set| |
----------- |
\/
____________
| |
| DUT |
| |
------------
Measurement Units: Measurement Units:
Enabled/Disabled N/A
See Also: See Also:
Traffic Profile Configuration Set Benchmark Planes
System Configuration Set Data Plane Configuration Set
Router Software Accelerated Life Testing Data Plane Startup Conditions
Data Plane Instability Conditions
Data Plane Failure Conditions
3.1.3 Management Configuration Module 3.1.3 Management Plane
Definition: Definition:
The router management features enabled for the The Management features and tools used for the
Accelerated Life Test. Accelerated Stress Benchmarking.
Discussion: Discussion:
A key component of the Accelerated Life Test is the A key component of the Accelerated Stress Benchmarking is the
Management Configuration Module to assess manageability Management Plane to assess manageability of the router
of the router under stress. The Management Configuration under stress. The Management Plane defines the Configuration,
Module can be divided into User Access Configuration Set, Startup Conditions, Instability Conditions, and Failure
SNMP Configuration Set, Logging/Debug Configuration Set, Conditions of the management protocols and features used for
and Packet Statistics Collector Configuration Set as the test. The Management Plane includes SNMP, Logging/Debug,
shown in Figure 4. and Statistics Collection.
____________ ____________ ____________
| User | | | | Logging/ |
| Access | | SNMP | __| Debug |
| Config Set | | Config Set | | | Config Set |
------------ ------------ | ------------
| | |
| | |
| \/ |
| ___________ |
| | | |
|------->| DUT |<-----|
| |
-----------
|
|
\/
___________
| Packet |
| Statistics|
| Collector |
| Config Set|
-----------
Figure 4. Management Configuration Module
Measurement units: Measurement units:
Enabled/Disabled N/A
Issues: Issues:
None None
See Also: See Also:
User Access Configuration Set Benchmark Planes
SNMP Configuration Set Management Plane Configuration Set
Logging/Debug Configuration Set Management Plane Startup Conditions
Packet Statistics Collector Configuration Set Management Plane Instability Conditions
Router Software Accelerated Life Testing Management Plane Failure Conditions
3.1.4 Security Plane
3.1.4 Security Configuration Module
Definition: Definition:
Security features enabled for the Accelerated Life Test. The Security features used during the Accelerated Stress
Benchmarking.
Discussion: Discussion:
Accelerated Life Testing of Core Router can include The Control Plane defines the Configuration, Startup
configuration of Security features. The Security Conditions, Instability Conditions, and Failure
Configuration Module includes the ACL Configuration Conditions of the security features and protocols used for
Set, Secure Protocol Configuration Set, and User Login the test. The Security Plane includes the ACLs, Firewall,
Configuration Set as shown in Figure 5. Secure Protocols, and User Login.
____________ ____________ ____________
| | | Secure | | User |
| ACL | | Protocol | __| Login |
| Config Set | | Config Set | | | Config Set |
------------ ------------ | ------------
| | |
| | |
| \/ |
| ___________ |
| | | |
|-------> | DUT |<--------|
| |
-----------
Figure 5. Security Configuration Module
Measurement units: Measurement units:
Enabled/Disabled N/A
Issues: Issues:
None None
See Also: See Also:
ACL Configuration Set Benchmark Planes
Secure Protocol Configuration Set Security Plane Configuration Set
Password Login Configuration Set Security Plane Startup Conditions
Security Plane Instability Conditions
Security Plane Failure Conditions
3.2 Configuration Sets 3.2 Configuration Sets
3.2.1 Control Plane Configuration Sets
3.2.1.1 Routing Protocol Configuration Set
Definition: Definition:
Table of Routing Protocols configuration for Accelerated The features and scaling limits used during the Accelerated Stress
Life Test. Benchmarking.
Discussion: Discussion:
The Routing Protocol Configuration Set is one of the There are four Configuration Sets: Control Plane Configuration Set,
Control Plane Configuration Sets BGP, MBGP, ISIS, OSPF are Data Plane Configuration Set, Management Plane Configuration Set,
the routing protocols that can be enabled or disabled in and Security Plane Configuration Set.
any combination. The specific protocol parameters used to
Router Software Accelerated Life Testing
establish the test conditions are also provided in the table. Measurement units: N/A
Measurement units: Issues: None
PARAMETER UNITS
BGP Enabled/Disabled
Number of EBGP Peers Peers
Number of IBGP Peers Peers
Number of BGP Route Instances Routes
Number of BGP Installed Routes Routes
MBGP Enabled/Disabled See Also:
Number of MBGP Route Instances Routes Control Plane Configuration Set
Number of MBGP Installed Routes Routes Data Plane Configuration Set
Management Plane Configuration Set
Security Plane Configuration Set
ISIS Enabled/Disabled 3.2.1 Control Plane Configuration Set
ISIS-TE Enabled/Disabled Definition:
Number of ISIS Adjacencies Adjacencies The routing protocols and scaling values used for the Accelerated
Number of ISIS Routes Routes Life Benchmarking.
Number of Nodes per Area Nodes
OSPF Enabled/Disabled Discussion:
OSPF-TE Enabled/Disabled Control Plane Configuration Set is shown in Figure 2 and specifies
Number of OSPF Adjacencies Adjacencies the Routing Protocols, Multicast, and MPLS configuration. Specific
Number of OSPF Routes Routes protocols can be enabled or disabled for a benchmark test.
Number of Nodes per Area Nodes
Issues: Measurement units: None
None
Issues: None
____________ ____________ ____________
| Routing | | Multicast | | MPLS |
| Protocols |___ | Protocols | __| Protocols |
| | | | | | | |
------------ | ------------ | ------------
| | |
| | |
| \/ |
| ___________ |
| | | |
|------->| DUT |<------|
| |
-----------
Figure 2. Control Plane Configuration Module
See Also: See Also:
Control Plane Configuration Module Data Plane Configuration Set
Multicast Protocol Configuration Set Management Configuration Set
MPLS Protocol Configuration Set Security Configuration Set
3.2.1.2 Multicast Protocol Configuration Set 3.2.2 Data Plane Configuration Set
Definition: Definition:
Table of Multicast Protocols configuration for Accelerated The data traffic profile enabled for the Accelerated Stress
Life Test. Benchmarking.
Discussion: Discussion:
The Multicast Protocol Configuration Set is one of the Data Plane Configuration Set includes the Traffic Profile and
Control Plane Configuration Sets PIM-SM and MSDP are interfaces used for the Accelerated Stress Benchmarking.
the routing protocols that can be enabled or disabled in
any combination. The specific protocol parameters used to
establish the test conditions are also provided in the table.
Measurement units:
PARAMETER UNITS
PIM-SM Enabled/Disabled
RP Enabled/Disabled
Number of Multicast Groups Groups
Router Software Accelerated Life Testing
MSDP Enabled/Disabled Measurement Units:
N/A
Issues: Issues:
None None
See Also: See Also:
Control Plane Configuration Module Traffic Profile
Routing Protocol Configuration Set 3.2.2.1 Traffic Profile
MPLS Protocol Configuration Set Definition
The characteristics of the Offered Load to the DUT used for
3.2.1.3 MPLS Protocol Configuration Set the Accelerated Stress Benchmarking.
Definition:
Table of MPLS Protocols configuration for Accelerated Life Test.
Discussion: Discussion
The MPLS Protocol Configuration Set is one of the Control Plane The Traffic Profile specifies the number of packet size(s), packet
Configuration Sets MPLS-TE (RSVP-TE, ISIS-TE, OSPF-TE) and LDP rate per interface, number of flows, and encapsulation used for the offered load to the DUT.
are the routing protocols that can be enabled or disabled in any
combination. The specific protocol parameters used to establish
the test conditions are also provided in the table.
Measurement units: Measurement Units:
PARAMETER UNITS Traffic Profile is reported as follows:
MPLS-TE
Number of Ingress Tunnels Tunnels
Number of Mid-Point Tunnels Tunnels
Number of Egress Tunnels Tunnels
LDP Parameter Units
Number of Sessions Sessions --------- ------
Number of FECs FECs Packet Size(s) bytes
Packet Rate(interface) array of packets per second
Number of Flows number
Encapsulation(flow) array of encapsulation type
Issues: Issues:
None None
See Also: See Also:
Control Plane Configuration Module Data Plane Configuration Set
Routing Protocol Configuration Set
Multicast Protocol Configuration Set
3.2.2 Data Plane Configuration Sets
3.2.2.1 Data Plane Configuration Set 3.2.3 Management Plane Configuration Set
Definition: Definition:
Table of Forwarded Traffic configuration for Accelerated Life The router management features enabled for the
Test. Accelerated Stress Test.
Discussion: Discussion:
A key component of the Accelerated Stress Test is the Management Configuration Set to assess manageability of the router under
stress. The Management Configuration Set defines the management configuration of the DUT. Features that are part of the
Management Configuration Set include SNMP, Logging/Debug, and
Statistics Collection, and services such as FTP, as shown in
Figure 3.
Measurement units: Measurement units:
N/A
Router Software Accelerated Life Testing
PARAMETER UNITS
Traffic Forwarding Enabled/Disabled
Packet Size Byte Size Distribution
Number of Flows Flows
Offered Load bps (or pps) Aggregate
Number of Interfaces Interfaces
Issues: Issues:
None None
See Also: See Also:
Data Plane Configuration Module Control Plane Configuration Set
Data Plane Configuration Set
Security Plane Configuration Set
____________ ____________
| | | Logging/ |
| SNMP | __| Debug |
| | | | |
------------ | ------------
| |
| |
\/ |
___________ |
| | |
| DUT |<---|
| |
-----------
|
|
\/
___________
| Packet |
| Statistics|
| Collector |
| |
-----------
3.2.3 Management Configuration Sets Figure 3. Management Plane Configuration Set
3.2.3.1 User Access Configuration Set
3.2.4 Security Plane Configuration Set
Definition: Definition:
Table of User Access methods during Accelerated Life Test. Security features and scaling enabled for the Accelerated Stress
Test.
Discussion: Discussion:
The Security Plane Configuration Set includes the configuration
and scaling of ACLs, Firewall, IPsec, and User Access, as shown
in Figure 4.
____________ ____________ ____________
| | | Secure | | User |
|ACL/Firewall| | Protocol | __| Access |
| | | | | | |
------------ ------------ | ------------
| | |
| | |
| \/ |
| ___________ |
| | | |
|-------> | DUT |<--------|
| |
-----------
Figure 4. Security Configuration Module
Measurement units: Measurement units:
PARAMETER UNITS N/A
Telnet Rate Sessions/Hour
FTP Rate Sessions/Hour
Concurrent Telnet Sessions Sessions
Concurrent FTP Session Sessions
SSH Enabled/Disabled
RADIUS Enabled/Disabled
TACACS Enabled/Disabled
Issues: Issues:
None None
See Also: See Also:
Management Configuration Module ACL Configuration Set
SNMP Configuration Set Secure Protocol Configuration Set
Logging/Debug Configuration Set Password Login Configuration Set
Packet Statistics Collector Configuration Set
3.2.3.2 SNMP Configuration Set 3.3 Startup Conditions
Definition: Definition:
Table of SNMP Configuration for the Accelerated Life Test. Test conditions that occur at the start of the Accelerated
Life Benchmark to establish conditions for the remainder of
the test.
Discussion: Discussion:
Startup Conditions may cause stress on the DUT and produce
failure. Startup Conditions are defined for the Control
Plane and Security Plane.
Measurement units: Measurement units:
SNMP GET Rate SNMP Gets/minute N/A
Issues: Issues:
None None
Router Software Accelerated Life Testing
See Also: See Also:
Management Configuration Module Control Plane Startup Conditions
User Access Configuration Set Data Plane Startup Conditions
Logging/Debug Configuration Set Management Plane Startup Conditions
Packet Statistics Collector Configuration Set Security Plane Startup Conditions
3.2.3.3 Logging/Debug Configuration Set 3.3.1 Control Plane Startup Conditions
Definition: Definition:
Table of Logging and Debug configuration for the Control Plane conditions that occur at the start
Accelerated Life Test. of the Accelerated Stress Benchmarking to establish conditions
for the remainder of the test.
Discussion: Discussion:
Startup Conditions may cause stress on the DUT and produce
failure. Startup Conditions for the Control Plane include
session establishment rate, number of sessions established
and number of routes learned.
Measurement units: Measurement units:
Logging Enabled/Disabled N/A
Protocol Debug Enabled/Disabled
Issues: Issues:
None None
See Also: See Also:
Management Configuration Module Startup Conditions
User Access Configuration Set Security Plane Startup Conditions
SNMP Configuration Set Control Plane Configuration Set
Packet Statistics Collector Configuration Set
3.2.3.4 Packet Statistics Collector Configuration Set 3.3.2 Security Plane Startup Conditions
Definition: Definition:
Table of Packet Statistics Collection for the Security Plane conditions that occur at the start
Accelerated Life Test. of the Accelerated Stress Benchmarking to establish conditions
for the remainder of the test.
Discussion: Discussion:
Startup Conditions may cause stress on the DUT and produce
failure. Startup Conditions for the Security Plane include
session establishment rate, number of sessions established
and number of policies learned, and number of user access
sessions opened.
Measurement units: Measurement units:
Packet Statistics Collector Enabled/Disabled N/A
Sampling Rate X:1 packets
Issues: Issues:
None None
See Also: See Also:
Management Configuration Module Startup Conditions
User Access Configuration Set Data Plane Startup Conditions
SNMP Configuration Set Management Plane Startup Conditions
Logging/Debug Configuration Set Security Plane Startup Conditions
3.2.4 Security Configuration Sets
Router Software Accelerated Life Testing
3.2.4.1 ACL Configuration Set 3.4 Instability Conditions
Definition: Definition:
Table of ACL configuration for the Accelerated Life Test Test conditions that occur during the Accelerated Stress
Benchmark to produce instability and stress the DUT.
Discussion: Discussion:
Instability Conditions are experienced by the DUT after the
Startup Conditions have completed. Instability Conditions
occur for the Control Plane, Data Plane, Management Plane,
and Security Plane.
Measurement units: Measurement units:
Access-Control Lists Enabled/Disabled N/A
Number of ACLs ACLs
ACL Depth ACEs
Issues: Issues:
None None
See Also: See Also:
Security Configuration Module Control Plane Instability Conditions
Data Plane Instability Conditions
Management Plane Instability Conditions
Security Plane Instability Conditions
3.3 Instability Conditions 3.4.1 Control Plane Instability Conditions
Definition: Definition:
Test conditions that occur during the Accelerated Life Test Control Plane conditions that occur during the Accelerated Stress
that are typical of instability in an operational network. Benchmark to produce instability and stress the DUT.
These conditions occur at a much greater rate during the
test.
Discussion: Discussion:
Configuration Modules and Configuration Sets establish the Control Plane Instability Conditions are experienced by the DUT
configuration of the DUT for the Accelerated Life Test. after the Startup Conditions have completed. Control Plane
Instability Conditions are events that occur during the Instability Conditions experienced by the DUT include session
Accelerated Life Test to stress the router. loss, route withdrawal, and route cost changes.
Measurement units: Measurement units:
N/A N/A
Issues: Issues:
None None
See Also: See Also:
Interface Shutdown Cycling Rate Instability Conditions
BGP Route Flap Rate Data Plane Instability Conditions
IGP Route Flap Rate Management Plane Instability Conditions
Route Convergence Due to Better Next-Hop Security Plane Instability Conditions
LSP Reroute Rate
Target Run Time
3.3.1 Interface Shutdown Rate
3.4.2 Data Plane Instability Conditions
Definition: Definition:
The rate at which physical interfaces are shutdown on the DUT. Data Plane conditions that occur during the Accelerated Stress
Benchmark to produce instability and stress the DUT.
Discussion: Discussion:
Data Plane Instability Conditions are experienced by the DUT
Router Software Accelerated Life Testing after the Startup Conditions have completed. Data Plane
This instability condition is an administrative shutdown or remote Instability Conditions experienced by the DUT include interface
loss of carrier for every interface of the DUT. This produces great shutdown, link loss, and overloaded links.
instability on the DUT due to continuous protocol session loss and
convergence.
Measurement units: Measurement units:
number of interface shutdowns per minute N/A
Issues: Issues:
None None
See Also: See Also:
Instability Conditions Instability Conditions
Route Flap Rate Control Plane Instability Conditions
Target Run Time Management Plane Instability Conditions
Security Plane Instability Conditions
3.3.2 BGP Route Flap Rate 3.4.3 Management Plane Instability Conditions
Definition: Definition:
The rate at which BGP routes installed in the route table are Management Plane conditions that occur during the Accelerated
flapped. Life Benchmark to produce instability and stress the DUT.
Discussion: Discussion:
This instability condition defines the rate of repeated Management Plane Instability Conditions are experienced by the DUT
installation, withdrawal,and installation of routes installed in after the Startup Conditions have completed. Management Plane
the route table. This produces great instability on the DUT due Instability Conditions experienced by the DUT include repeated
to continuous changes to the forwarding table. FTP of large files.
Measurement units: Measurement units:
routes flapped per second N/A
Issues: Issues:
None None
See Also: See Also:
Instability Conditions Instability Conditions
Interface Shutdown Cycling Rate Control Plane Instability Conditions
Target Run Time Data Plane Instability Conditions
Security Plane Instability Conditions
3.3.3 IGP Route Flap Rate 3.4.4 Security Plane Instability Conditions
Definition: Definition:
The rate at which IGP routes installed in the route table are Security Plane conditions that occur during the Accelerated
flapped. Life Benchmark to produce instability and stress the DUT.
Discussion: Discussion:
This instability condition defines the rate of repeated Security Plane Instability Conditions are experienced by the DUT
installation, withdrawal,and installation of routes installed in after the Startup Conditions have completed. Security Plane
the route table. This produces great instability on the DUT due Instability Conditions experienced by the DUT include session
to continuous changes to the forwarding table. loss and policy changes.
Router Software Accelerated Life Testing
Measurement units: Measurement units:
routes flapped per second N/A
Issues: Issues:
None None
See Also: See Also:
Instability Conditions Instability Conditions
Interface Shutdown Cycling Rate Control Plane Instability Conditions
Target Run Time Data Plane Instability Conditions
Management Plane Instability Conditions
3.5 Failure Conditions
3.3.4 Route Convergence Due to Better Next-Hop 3.5.1 Unexpected Session Loss
Definition: Definition:
The rate at which routes in the FIB are replaced by a route Session Loss due to Control Plane or Data Plane failure during the
instance in the RIB with a different next-hop becoming more Accelerated Stress Benchmarking.
preferred.
Discussion: Discussion:
The route changes are due to attribute and TLV changes. RIB Session loss will be observed due to Convergence Events produced
contains routes matching those in the FIB. Convergence due by the Instability Conditions. This is not considered a failure
to better path can occur for BGP and IGP routes. during the Accelerated Stress Benchmarking. It is expected behavior.
Unexpected packet loss occurs due to Data Plane or Control Plane
failure.
Measurement units: Measurement units:
more preferred routes per second sessions
Issues: Issues:
None None
See Also: See Also:
Instability Conditions Unexpected Packet Loss
Interface Shutdown Cycling Rate Misrouted Packets
Target Run Time Access Denial
Errored Management Values
3.3.5 LSP Reroute Rate Security Plane Failure
3.5.2 Unexpected Packet Loss
Definition: Definition:
The rate at which established LSPs experience a failure condition Packet Loss due to Control Plane or Data Plane failure during the
and are forced to reroute. Accelerated Stress Benchmarking.
Discussion: Discussion:
This instability condition defines the rate of repeated LSP Packet loss will be observed due to Convergence Events produced
reroutes. This produces great instability on the DUT due to by the Instability Conditions. This is not considered a failure
continuous tunnel path calculations and changes to the during the Accelerated Stress Benchmarking. It is expected behavior.
forwarding table. Rerouting can be performed with Headend Unexpected packet loss occurs due to Data Plane or Control Plane
Reroute, Standby LSP, or Fast Reroute. failure.
Measurement units: Measurement units:
routes flapped per second packets
Issues: Issues:
None None
Router Software Accelerated Life Testing
See Also: See Also:
Instability Conditions Unexpected Session Loss
Interface Shutdown Cycling Rate Misrouted Packets
Target Run Time Access Denial
Errored Management Values
3.3.6 Intended Test Duration Security Plane Failure
3.5.3 Misrouted Packets
Definition: Definition:
The maximum targetted run time for Accelerated Life Test. Packets routed to the incorrect next-hop or LSP due to Control
Plane failure during the Accelerated Stress Benchmarking.
Discussion: Discussion:
This is an instability condition because the protocol sessions Packets will be rerouted to new next-hops and LSPs due to Convergence
and scaling values must be maintained for this specified time with Events produced by the Instability Conditions. This is not considered
interfaceshutdowns and route flapping. The test may be halted a failure during the Accelerated Stress Benchmarking. It is expected
upon reaching the Intended Test Duration. behavior. Unexpected Misrouted packets occur due to Control Plane
failure.
Measurement units: Measurement units:
Hours packets
Issues: Issues:
None None
See Also: See Also:
Instability Conditions Unexpected Packet Loss
Interface Shutdown Cycling Rate Unexpected Session Loss
Route Flap Rate Access Denial
Errored Management Values
3.4 Evaluation Benchmarks Security Plane Failure
3.4.1 Run-Time without Error 3.5.4 Access Denial
Definition: Definition:
The run-time for the Accelerated Life Test without the DUT Inability to management the DUT due to the inability
exhibiting an error in the control plane, data plane, management to gain remote access to the DUT.
plane, or security plane.
Discussion: Discussion:
For a successful test, the Run-Time Without Error will equal the Handling of Instability Conditions may cause a Control Plane failure
Intended Test Duration. In the event of any failure, the Run-Time of the DUT externally observable as the inability to remotely manage
Without Error is less than the Intended Test Duration and will equal the DUT. Whether using SNMP, Telnet, SSH, or FTP, failure to manage
the maximum of the Run-Time without Control Plane Error, Run-Time the device constitues a failure.
without Data Plane Error, Run-Time without Management Plane Error,
and Run-Time without Security Plane Error.
Measurement units: Measurement units:
Hours and Minutes Failed session
Issues: Issues:
None None
See Also: See Also:
Unexpected Packet Loss
Router Software Accelerated Life Testing Unexpected Session Loss
Misrouted Packets
Intended Test Duration Errored Management Value
Run-Time without Control Plane Error Security Plane Failure
Run-Time without Data Plane Error 3.5.5 Errored Management Value
Run-Time without Management Plane Error
Run-Time without Security Plane Error
3.4.2 Run-Time without Control Plane Error
Definition: Definition:
The run-time for the Accelerated Life Test without the DUT Incorrect SNMP object value or statistic during the Accelerated
exhibiting an error in the control plane to Routing, Multicast, Life Benchmark.
and MPLS Protocol.
Discussion: Discussion:
For a successful test, the Run-Time Without Control Plane Error Management of a router relies upon accurate reporting of object
will equal the Run-Time without Error. In the event of a Control values and statistics. The Accelerated Stress Benchmarkinging validates
Plane failure, the Run-Time Without Control Plane Error is less that the DUT is able to report correct values when experiencing
than the Run-Time without Error. Instability Conditions.
Measurement units: Measurement units:
Hours and Minutes Name of incorrect object or statistic
Issues: Issues:
None None
See Also: See Also:
Intended Test Duration Unexpected Packet Loss
Run-Time without Error Unexpected Session Loss
Run-Time without Data Plane Error Misrouted Packets
Run-Time without Management Plane Error Access Denial
Run-Time without Security Plane Error Security Plane Failure
Success Threshold
3.4.3 Run-Time without Data Plane Error
3.5.6 Security Plane Failure
Definition: Definition:
The run-time for the Accelerated Life Test without the DUT Violation of Security Policy for packets directed to the DUT
exhibiting an error in the Data Plane for traffic forwarding or routed through the DUT.
or DUT interfaces.
Discussion: Discussion:
For a successful test, the Run-Time Without Data Plane Error Security Policy is configured to permit and deny access to packets
will equal the Run-Time without Error. In the event of a Data destined for the DUT and packets to be routed through the DUT.
Plane failure, the Run-Time Without Data Plane Error is less A violation of this security policy is when either a packet to
than the Run-Time without Error. be denied is permitted or a packet to be permitted is denied.
Measurement units: Measurement units:
Hours and Minutes packets
Issues: Issues:
None None
See Also: See Also:
Unexpected Packet Loss
Unexpected Session Loss
Misrouted Packets
Access Denial
Errored Management Value
3.6 Benchmarks
Router Software Accelerated Life Testing 3.6.1 Success Threshold
Intended Test Duration
Run-Time without Error
Run-Time without Control Plane Error
Run-Time without Management Plane Error
Run-Time without Security Plane Error
3.4.4 Run-Time without Management Plane Error
Definition: Definition:
The run-time for the Accelerated Life Test without the DUT The intended number of hours for the DUT to operate without failure
exhibiting an error in the Management Plane to User Access, during the Accelerated Stress Benchmarking.
SNMP, or Logging/Debug.
Discussion: Discussion:
For a successful test, the Run-Time Without Management Plane Success Threshold is a user determined value of time. It is
Error will equal the Run-Time without Error. In the event of necessary to define the Success Threshold because it is possible
a Management Plane failure, the Run-Time Without Management for the DUT to experience zero failures for given Configuration
Plane Error is less than the Run-Time without Error. Sets, Startup Conditions, and Instability Conditions
Measurement units: Measurement units:
Hours and Minutes Hours
Issues: Issues:
None None
See Also: See Also:
Intended Test Duration Accelerated-Life Test Duration
Run-Time without Error
Run-Time without Control Plane Error
Run-Time without Data Plane Error
Run-Time without Security Plane Error
3.4.5 Run-Time without Security Plane Error 3.6.2 Accelerated-Life Test Duration
Definition: Definition:
The run-time for the Accelerated Life Test without the DUT The run-time for the Accelerated Stress Test without the DUT
exhibiting an error in the Security Plane to ACLs. exhibiting an error in the control plane, data plane, management
plane, or security plane.
Discussion: Discussion:
For a successful test, the Run-Time Without Security Plane For a successful test, the Test Duration will equal the
Error will equal the Run-Time without Error. In the event Success Threshold. In the event of any failure, the Test Duration
of a Security Plane failure, the Run-Time Without Control is less than the Success Threshold. Reporting of the Test
Plane Error is less than the Run-Time without Error. Duration MUST inlcude the Failure Condition from those defined
in this document.
Measurement units: Measurement units:
Hours and Minutes Hours
Issues: Issues:
None None
See Also: See Also:
Intended Test Duration Success Threshold
Router Software Accelerated Life Testing
Run-Time without Error
Run-Time without Control Plane Error
Run-Time without Data Plane Error
Run-Time without Management Plane Error
4. Security Considerations 4. Security Considerations
Documents of this type do not directly effect the security of Documents of this type do not directly effect the security of
the Internet or of corporate networks as long as benchmarking the Internet or of corporate networks as long as benchmarking
is not performed on devices or systems connected to operating is not performed on devices or systems connected to operating
networks. networks.
5. References 5. References
[1] Bradner, S., Editor, "Benchmarking Terminology for Network [1] Bradner, S., Editor, "Benchmarking Terminology for Network
Interconnection Devices", RFC 1242, July 1991. Interconnection Devices", RFC 1242, July 1991.
skipping to change at page 18, line 37 skipping to change at page 18, line 30
[4] "Core Router Evaluation for Higher Availability", Scott [4] "Core Router Evaluation for Higher Availability", Scott
Poretsky, NANOG 25, June 8, 2002, Toronto, CA. Poretsky, NANOG 25, June 8, 2002, Toronto, CA.
[5] "Router Stress Testing to Validate Readiness for Network [5] "Router Stress Testing to Validate Readiness for Network
Deployment", Scott Poretsky, IEEE CQR 2003. Deployment", Scott Poretsky, IEEE CQR 2003.
6. Author's Address 6. Author's Address
Scott Poretsky Scott Poretsky
Avici Systems Quarry Technologies
101 Billerica Avenue 8 New England Executive Park
N. Billerica, MA 01862 Burlington, MA 01803
USA USA
Phone: + 1 781 395 5090
Phone: + 1 978 964 2287 EMail: sporetsky@quarrytech.com
EMail: sporetsky@avici.com
Shankar Rao Shankar Rao
950 17th Street
Suite 1900
Qwest Communications Qwest Communications
Denver, CO Denver, CO 80210
USA USA
Phone: + 1 303 437 6643 Phone: + 1 303 437 6643
Email: srao@qwest.net Email: srao@qwest.net
Ray Piatt Ray Piatt
Cable and Wireless Cable and Wireless
11700 Plaza America Drive 11700 Plaza America Drive
Reston, VA 20190 Reston, VA 20190
USA USA
Router Software Accelerated Life Testing
Phone: + 1 703 292 2113 Phone: + 1 703 292 2113
Email: rpiatt@cw.net Email: rpiatt@cw.net
7. Full Copyright Statement 7. Full Copyright Statement
Copyright (C) The Internet Society (1998). All Rights Copyright (C) The Internet Society (1998). All Rights
Reserved. Reserved.
This document and translations of it may be copied and This document and translations of it may be copied and
furnished to others, and derivative works that comment on or furnished to others, and derivative works that comment on or
skipping to change at page 19, line 43 skipping to change at page 20, line 10
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY
THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY
RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS
FOR A PARTICULAR PURPOSE. FOR A PARTICULAR PURPOSE.
Appendix 1. White Box Benchmarking Terminology Appendix 1. White Box Benchmarking Terminology
Appendix 1.1 Minimum Available Memory Appendix 1.1 Minimum Available Memory
Definition: Definition:
Minimum DUT Available Memory during the duration of the Minimum DUT Available Memory during the duration of the
Accelerated Life Test. Accelerated Stress Test.
Discussion: Discussion:
It is necessary to monitor DUT memory to measure this It is necessary to monitor DUT memory to measure this
benchmark. benchmark.
Measurement units: Measurement units:
bytes bytes
Issues: Issues:
None None
See Also: See Also:
Maximum CPU Utilization Maximum CPU Utilization
Router Software Accelerated Life Testing
Appendix 1.2 Maximum CPU Utilization Appendix 1.2 Maximum CPU Utilization
Definition: Definition:
Maximum DUT CPU utilization during the duration of the Maximum DUT CPU utilization during the duration of the
Accelerated Life Test. Accelerated Stress Test.
Discussion: Discussion:
It is necessary to monitor DUT CPU Utilization to measure It is necessary to monitor DUT CPU Utilization to measure
this benchmark. this benchmark.
Measurement units: Measurement units:
% %
Issues: Issues:
None None
 End of changes. 

This html diff was produced by rfcdiff 1.23, available from http://www.levkowetz.com/ietf/tools/rfcdiff/