draft-ietf-bmwg-acc-bench-term-08.txt   draft-ietf-bmwg-acc-bench-term-09.txt 
Network Working Group Network Working Group
INTERNET-DRAFT INTERNET-DRAFT
Expires in: September 2006 Expires in: December 2006
Scott Poretsky Scott Poretsky
Reef Point Systems Reef Point Systems
Shankar Rao Shankar Rao
Qwest Communications Qwest Communications
Terminology for Accelerated Stress Benchmarking Terminology for Accelerated Stress Benchmarking
<draft-ietf-bmwg-acc-bench-term-08.txt> <draft-ietf-bmwg-acc-bench-term-09.txt>
Intellectual Property Rights (IPR) statement: Intellectual Property Rights (IPR) statement:
By submitting this Internet-Draft, each author represents that any By submitting this Internet-Draft, each author represents that any
applicable patent or other IPR claims of which he or she is aware applicable patent or other IPR claims of which he or she is aware
have been or will be disclosed, and any of which he or she becomes have been or will be disclosed, and any of which he or she becomes
aware will be disclosed, in accordance with Section 6 of BCP 79. aware will be disclosed, in accordance with Section 6 of BCP 79.
Status of this Memo Status of this Memo
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
skipping to change at page 1, line 38 skipping to change at page 1, line 38
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt. http://www.ietf.org/ietf/1id-abstracts.txt.
The list of Internet-Draft Shadow Directories can be accessed at The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html. http://www.ietf.org/shadow.html.
Copyright Notice Copyright Notice
Copyright (C) The Internet Society (2006). All Rights Reserved. Copyright (C) The Internet Society (2006).
ABSTRACT ABSTRACT
This document provides the Terminology for performing Stress This document provides the Terminology for performing Accelerated
Benchmarking of networking devices. The three phases of the Stress Stress Benchmarking of networking devices. The three phases of
Test: Startup, Instability and Recovery are defined along with the the Stress Test: Startup, Instability and Recovery are defined
benchmarks and configuration terms associated with the each phase. along with the benchmarks and configuration terms associated with
Also defined are the Benchmark Planes fundamental to stress testing the each phase. Also defined are the Benchmark Planes fundamental
configuration, setup and measurement. The terminology is to be to stress testing configuration, setup and measurement. The
used with the companion framework and methodology documents. terminology is to be used with the companion framework and
methodology documents.
Table of Contents Table of Contents
1. Introduction ............................................... 3 1. Introduction ............................................... 3
2. Existing definitions ....................................... 3 2. Existing definitions ....................................... 3
3. Term definitions............................................ 4 3. Term definitions............................................ 4
3.1 General Terms............................................. 4 3.1 General Terms............................................. 4
Stress Benchmarking Stress Benchmarking
3.1.1 Benchmark Planes...................................... 4 3.1.1 Benchmark Planes...................................... 4
3.1.2 Configuration Sets.................................... 5 3.1.2 Configuration Sets.................................... 5
skipping to change at page 3, line 22 skipping to change at page 3, line 22
Routers in an operational network are simultaneously configured with Routers in an operational network are simultaneously configured with
multiple protocols and security policies while forwarding traffic and multiple protocols and security policies while forwarding traffic and
being managed. To accurately benchmark a router for deployment it is being managed. To accurately benchmark a router for deployment it is
necessary to test that router in operational conditions by necessary to test that router in operational conditions by
simultaneously configuring and scaling network protocols and security simultaneously configuring and scaling network protocols and security
policies, forwarding traffic, and managing the device. It is helpful policies, forwarding traffic, and managing the device. It is helpful
to accelerate these network operational conditions so that the to accelerate these network operational conditions so that the
router under test can be benchmarked with faster test duration. router under test can be benchmarked with faster test duration.
Testing a router in accelerated network conditions is known as Testing a router in accelerated network conditions is known as
Accelerated Stress Testing. Accelerated Stress Benchmarking.
This document provides the Terminology for performing Stress This document provides the Terminology for performing Stress
Benchmarking of networking devices. The three phases of the Stress Benchmarking of networking devices. The three phases of the Stress
Test: Startup, Instability and Recovery are defined along with the Test: Startup, Instability and Recovery are defined along with the
benchmark and configuration terms associated with the each phase. benchmark and configuration terms associated with the each phase.
Benchmarks for stress testing are defined using the Aggregate Benchmarks for stress testing are defined using the Aggregate
Forwarding Rate and control plane Session Count during each phase Forwarding Rate and control plane Session Count during each phase
of the test. For each plane, the Configuration Set, Startup of the test. For each plane, the Configuration Set, Startup
Conditions, and Instability Conditions are defined. Also defined are Conditions, and Instability Conditions are defined. Also defined are
the Benchmark Planes fundamental to stress testing configuration, setup the Benchmark Planes fundamental to stress testing configuration,
and measurement. These are the Control Plane, Data Plane, Management setup and measurement. These are the Control Plane, Data Plane,
Plane and Security Plane Multiple benchmarks are made for each Management Plane and Security Plane Multiple benchmarks are made
Benchmark Plane during each Phase. Benchmarks can be compared across for each Benchmark Plane during each Phase. Benchmarks can be
multiple planes for the same DUT or at the same plane for 2 or more compared across multiple planes for the same DUT or at the same
DUTS. These benchmarks White Box benchmarks are provided in Appendix 1 plane for 2 or more DUTS. These benchmarks White Box benchmarks
for additional DUT behavior measurements. The terminology is to be are provided in Appendix 1 for additional DUT behavior
used with the companion methodology document [4]. The sequence of measurements. The terminology is to be used with the companion
phases, actions, and benchmarks are shown in Table 1. methodology document [4]. The sequence of phases, actions, and
benchmarks are shown in Table 1.
2. Existing definitions 2. Existing definitions
RFC 1242 "Benchmarking Terminology for Network Interconnect RFC 1242 "Benchmarking Terminology for Network Interconnect
Devices" and RFC 2285 "Benchmarking Terminology for LAN Switching Devices" and RFC 2285 "Benchmarking Terminology for LAN Switching
Devices" should be consulted before attempting to make use of this Devices" should be consulted before attempting to make use of this
document. For the sake of clarity and continuity this RFC adopts document. For the sake of clarity and continuity this RFC adopts
the template for definitions set out in Section 2 of RFC 1242. the template for definitions set out in Section 2 of RFC 1242.
Definitions are indexed and grouped together in sections for ease Definitions are indexed and grouped together in sections for ease
of reference. of reference.
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in BCP 14, RFC 2119 document are to be interpreted as described in BCP 14, RFC 2119
[5]. RFC 2119 defines the use of these key words to help make the [5]. RFC 2119 defines the use of these key words to help make the
intent of standards track documents as clear as possible. While this intent of standards track documents as clear as possible. While this
document uses these keywords, this document is not a standards track document uses these keywords, it is not a standards track document.
document.
Stress Benchmarking Stress Benchmarking
Table 1. Phase Sequence and Benchmarks Table 1. Phase Sequence and Benchmarks
III. Recovery Phase II. Instability Phase I. Startup Phase III. Recovery Phase II. Instability Phase I. Startup Phase
<-----------------<---<-------------------<----<--------------< <-----------------<---<-------------------<----<--------------<
Remove Instability Achieve Configuration Apply Startup Remove Instability Achieve Configuration Apply Startup
Conditions Set Conditions Conditions Set Conditions
Benchmark: Benchmark: Benchmark: Benchmark: Benchmark: Benchmark:
skipping to change at page 5, line 53 skipping to change at page 5, line 53
See Also: See Also:
Control Plane Configuration Set Control Plane Configuration Set
Data Plane Configuration Set Data Plane Configuration Set
Management Plane Configuration Set Management Plane Configuration Set
Security Plane Configuration Set Security Plane Configuration Set
3.1.3 Startup Conditions 3.1.3 Startup Conditions
Definition: Definition:
Test conditions that occur at the start of the Accelerated Test conditions that occur at the start of the Accelerated
Life Benchmark to establish conditions for the remainder of Stress Benchmark to establish conditions for the remainder of
the test. the test.
Stress Benchmarking Stress Benchmarking
Discussion: Discussion:
Startup Conditions may cause stress on the DUT and produce Startup Conditions may cause stress on the DUT and produce
failure. Startup Conditions are defined for the Control failure. Startup Conditions are defined for the Control
Plane and Security Plane. Plane and Security Plane.
Measurement units: Measurement units:
skipping to change at page 12, line 21 skipping to change at page 12, line 21
None None
See Also: See Also:
Startup Phase Startup Phase
3.3.3 Control Plane 3.3.3 Control Plane
3.3.3.1 Control Plane Configuration Set 3.3.3.1 Control Plane Configuration Set
Definition: Definition:
The routing protocols and scaling values used for the Accelerated The routing protocols and scaling values used for the Accelerated
Life Benchmarking. Stress Benchmarking.
Discussion: Discussion:
Control Plane Configuration Set is shown in Figure 2 and specifies Control Plane Configuration Set is shown in Figure 2 and specifies
the Routing Protocols, Multicast, and MPLS configuration. Specific the Routing Protocols, Multicast, and MPLS configuration. Specific
protocols can be enabled or disabled for a benchmark test. protocols can be enabled or disabled for a benchmark test.
Measurement units: Measurement units:
N/A N/A
Issues: Issues:
skipping to change at page 13, line 55 skipping to change at page 13, line 55
See Also: See Also:
Traffic Profile Traffic Profile
3.3.4.2 Traffic Profile 3.3.4.2 Traffic Profile
Definition Definition
The characteristics of the Offered Load to the DUT used for The characteristics of the Offered Load to the DUT used for
the Accelerated Stress Benchmarking. the Accelerated Stress Benchmarking.
Discussion Discussion
The Traffic Profile specifies the number of packet size(s), packet The Traffic Profile specifies the number of packet size(s),
rate per interface, number of flows, and encapsulation used for the packet rate per interface, number of flows, and encapsulation
offered load to the DUT. used for the offered load to the DUT.
Stress Benchmarking Stress Benchmarking
Measurement Units: Measurement Units:
Traffic Profile is reported as follows: Traffic Profile is reported as follows:
Parameter Units Parameter Units
--------- ------ --------- ------
Packet Size(s) bytes Packet Size(s) bytes
Packet Rate(interface) array of packets per second Packet Rate(interface) array of packets per second
skipping to change at page 14, line 28 skipping to change at page 14, line 28
None None
See Also: See Also:
Data Plane Configuration Set Data Plane Configuration Set
3.3.5 Management Plane 3.3.5 Management Plane
3.3.5.1 Management Plane Configuration Set 3.3.5.1 Management Plane Configuration Set
Definition: Definition:
The router management features enabled for the The router management features enabled for the
Accelerated Stress Test. Accelerated Stress Benchmark.
Discussion: Discussion:
A key component of the Accelerated Stress Test is the Management A key component of the Accelerated Stress Benchmark is the
Configuration Set to assess manageability of the router under Management Configuration Set to assess manageability of the
stress. The Management Configuration Set defines the management router under stress. The Management Configuration Set defines
configuration of the DUT. Features that are part of the the management configuration of the DUT. Features that are
Management Configuration Set include Telnet access, SNMP, part of the Management Configuration Set include access, SNMP,
Logging/Debug, and Statistics Collection, and services such as Logging/Debug, and Statistics Collection, and services such as
FTP, as shown in Figure 3. These features should be enabled FTP, as shown in Figure 3. These features should be enabled
throughout the Stress test. SNMP Gets should be made continuously throughout the Stress test. SNMP Gets should be made
with multiple FTP and Telnet sessions operating simultaneously. continuously with multiple FTP and Telnet sessions operating
FTP sessions should be opened and closed at varying intervals simultaneously. FTP sessions should be opened and closed at
and get and put files while open. Telnet sessions should be varying intervals and get and put files while open. Telnet
opened and closed at varying intervals and enter valid and invalid sessions should be opened and closed at varying intervals and
show and configuration commands while open. enter valid and invalid show and configuration commands while
open.
Measurement units: Measurement units:
N/A N/A
Issues: Issues:
None None
See Also: See Also:
Control Plane Configuration Set Control Plane Configuration Set
Data Plane Configuration Set Data Plane Configuration Set
skipping to change at page 20, line 32 skipping to change at page 20, line 32
See Also: See Also:
Instability Conditions Instability Conditions
Control Plane Instability Conditions Control Plane Instability Conditions
Management Plane Instability Conditions Management Plane Instability Conditions
Security Plane Instability Conditions Security Plane Instability Conditions
3.4.3.3 Management Plane Instability Conditions 3.4.3.3 Management Plane Instability Conditions
Definition: Definition:
Management Plane conditions that occur during the Accelerated Management Plane conditions that occur during the Accelerated
Life Benchmark to produce instability and stress the DUT. Stress Benchmark to produce instability and stress the DUT.
Discussion: Discussion:
Management Plane Instability Conditions are experienced by the DUT Management Plane Instability Conditions are experienced by the DUT
after the Startup Conditions have completed. Management Plane after the Startup Conditions have completed. Management Plane
Instability Conditions experienced by the DUT include repeated Instability Conditions experienced by the DUT include repeated
FTP of large files. FTP of large files.
Measurement units: Measurement units:
N/A N/A
skipping to change at page 20, line 56 skipping to change at page 20, line 56
See Also: See Also:
Instability Conditions Instability Conditions
Control Plane Instability Conditions Control Plane Instability Conditions
Data Plane Instability Conditions Data Plane Instability Conditions
Security Plane Instability Conditions Security Plane Instability Conditions
3.4.3.4 Security Plane Instability Conditions 3.4.3.4 Security Plane Instability Conditions
Definition: Definition:
Security Plane conditions that occur during the Accelerated Security Plane conditions that occur during the Accelerated
Life Benchmark to produce instability and stress the DUT. Stress Benchmark to produce instability and stress the DUT.
Stress Benchmarking Stress Benchmarking
Discussion: Discussion:
Security Plane Instability Conditions are experienced by the DUT Security Plane Instability Conditions are experienced by the DUT
after the Startup Conditions have completed. Security Plane after the Startup Conditions have completed. Security Plane
Instability Conditions experienced by the DUT include session Instability Conditions experienced by the DUT include session
loss and uninitiated policy changes. loss and uninitiated policy changes.
Measurement units: Measurement units:
skipping to change at page 23, line 55 skipping to change at page 23, line 55
Uncontrolled Session Loss Uncontrolled Session Loss
3.5.2.5 Variability Benchmarks 3.5.2.5 Variability Benchmarks
Definition: Definition:
The difference between the measured Benchmarks of the The difference between the measured Benchmarks of the
same DUT over multiple iterations. same DUT over multiple iterations.
Discussion: Discussion:
Ideally, the measured benchmarks should be the same for multiple Ideally, the measured benchmarks should be the same for multiple
iterations with the same DUT. Configuration Sets and Instability iterations with the same DUT. Configuration Sets and
Conditions SHOULD be held constant for this benchmark. Whether the Instability Conditions SHOULD be held constant for this
DUT can exhibit such predictable and repeatable behavior is an benchmark. Whether the DUT can exhibit such predictable and
important benchmark in itself. repeatable behavior is an important benchmark in itself.
Stress Benchmarking Stress Benchmarking
Measurement units: Measurement units:
As applicable to each Benchmark. The results are to be As applicable to each Benchmark. The results are to be
presented in a table format for successive Iterations. presented in a table format for successive Iterations.
Ideally, the differences should be zero. Ideally, the differences should be zero.
Issues: Issues:
None None
skipping to change at page 24, line 41 skipping to change at page 24, line 41
[1] Bradner, S., Editor, "Benchmarking Terminology for Network [1] Bradner, S., Editor, "Benchmarking Terminology for Network
Interconnection Devices", RFC 1242, March 1991. Interconnection Devices", RFC 1242, March 1991.
[2] Mandeville, R., "Benchmarking Terminology for LAN Switching [2] Mandeville, R., "Benchmarking Terminology for LAN Switching
Devices", RFC 2285, June 1998. Devices", RFC 2285, June 1998.
[3] Bradner, S. and McQuaid, J., "Benchmarking Methodology for [3] Bradner, S. and McQuaid, J., "Benchmarking Methodology for
Network Interconnect Devices", RFC 2544, March 1999. Network Interconnect Devices", RFC 2544, March 1999.
[4] Poretsky, S. and Rao, S., "Methodology for Accelerated [4] Poretsky, S. and Rao, S., "Methodology for Accelerated
Stress Benchmarking", draft-ietf-bmwg-acc-bench-meth-04, Stress Benchmarking", draft-ietf-bmwg-acc-bench-meth-05,
work in progress, March 2006. work in progress, June 2006.
[5] Bradner, S., "Key words for use in RFCs to Indicate Requirement [5] Bradner, S., "Key words for use in RFCs to Indicate
Levels", RFC 2119, March 1997. Requirement Levels", RFC 2119, March 1997.
6.2 Informative References 6.2 Informative References
[RFC3871] RFC 3871 "Operational Security Requirements for Large [RFC3871] RFC 3871 "Operational Security Requirements for Large
Internet Service Provider (ISP) IP Network Infrastructure. Internet Service Provider (ISP) IP Network Infrastructure.
G. Jones, Ed.. IETF, September 2004. G. Jones, Ed.. IETF, September 2004.
[NANOG25] "Core Router Evaluation for Higher Availability", Scott [NANOG25] Poretsky, S., "Core Router Evaluation for Higher
Poretsky, NANOG 25, June 8, 2002, Toronto, CA. Availability", NANOG 25, June 8, 2002, Toronto, CA.
[IEEECQR] "Router Stress Testing to Validate Readiness for Network [IEEECQR] Poretsky, S., "Router Stress Testing to Validate
Deployment", Scott Poretsky, IEEE CQR 2003. Readiness for Network Deployment", IEEE CQR 2003.
Stress Benchmarking Stress Benchmarking
7. Author's Address 7. Author's Address
Scott Poretsky
Reef Point Systems Reef Point Systems
8 New England Executive Park 8 New England Executive Park
Burlington, MA 01803 Burlington, MA 01803
USA USA
Phone: + 1 781 395 5090 Phone: + 1 781 395 5090
EMail: sporetsky@reefpoint.com EMail: sporetsky@reefpoint.com
Shankar Rao Shankar Rao
1801 California Street 1801 California Street
8th Floor 8th Floor
Qwest Communications Qwest Communications
Denver, CO 80202 Denver, CO 80202
USA USA
Phone: + 1 303 437 6643 Phone: + 1 303 437 6643
Email: shankar.rao@qwest.com Email: shankar.rao@qwest.com
Appendix 1. White Box Benchmarking Terminology Appendix 1. White Box Benchmarking Terminology
Minimum Available Memory Minimum Available Memory
Definition: Definition:
Minimum DUT Available Memory during the duration of the Minimum DUT Available Memory during the duration of the
Accelerated Stress Test. Accelerated Stress Benchmark.
Discussion: Discussion:
It is necessary to monitor DUT memory to measure this It is necessary to monitor DUT memory to measure this
benchmark. benchmark.
Measurement units: Measurement units:
bytes bytes
Issues: None Issues: None
See Also: See Also:
Maximum CPU Utilization Maximum CPU Utilization
Maximum CPU Utilization Maximum CPU Utilization
Definition: Definition:
Maximum DUT CPU utilization during the duration of the Maximum DUT CPU utilization during the duration of the
Accelerated Stress Test. Accelerated Stress Benchmark.
Discussion: Discussion:
It is necessary to monitor DUT CPU Utilization to measure It is necessary to monitor DUT CPU Utilization to measure
this benchmark. this benchmark.
Measurement units: % Measurement units: %
Issues: None Issues: None
See Also: See Also:
 End of changes. 23 change blocks. 
55 lines changed or deleted 58 lines changed or added

This html diff was produced by rfcdiff 1.32. The latest version is available from http://www.levkowetz.com/ietf/tools/rfcdiff/