draft-ietf-bmwg-acc-bench-term-11.txt   draft-ietf-bmwg-acc-bench-term-12.txt 
Network Working Group Network Working Group
INTERNET-DRAFT INTERNET-DRAFT
Expires in: September 2007 Expires in: January 2008
Intended Status: Informational Intended Status: Informational
Scott Poretsky Scott Poretsky
Reef Point Systems Reef Point Systems
Shankar Rao Shankar Rao
Qwest Communications Qwest Communications
Terminology for Accelerated Stress Benchmarking Terminology for Accelerated Stress Benchmarking
<draft-ietf-bmwg-acc-bench-term-11.txt> <draft-ietf-bmwg-acc-bench-term-12.txt>
Intellectual Property Rights (IPR) statement: Intellectual Property Rights (IPR) statement:
By submitting this Internet-Draft, each author represents that any By submitting this Internet-Draft, each author represents that any
applicable patent or other IPR claims of which he or she is aware applicable patent or other IPR claims of which he or she is aware
have been or will be disclosed, and any of which he or she becomes have been or will be disclosed, and any of which he or she becomes
aware will be disclosed, in accordance with Section 6 of BCP 79. aware will be disclosed, in accordance with Section 6 of BCP 79.
Status of this Memo Status of this Memo
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
skipping to change at page 2, line 30 skipping to change at page 2, line 30
3.3.1 Startup Phase.........................................10 3.3.1 Startup Phase.........................................10
3.3.2 Benchmarks............................................10 3.3.2 Benchmarks............................................10
3.3.2.1 Stable Aggregate Forwarding Rate..................10 3.3.2.1 Stable Aggregate Forwarding Rate..................10
3.3.2.2 Stable Latency....................................11 3.3.2.2 Stable Latency....................................11
3.3.2.3 Stable Session Count..............................11 3.3.2.3 Stable Session Count..............................11
3.3.3 Control Plane.........................................12 3.3.3 Control Plane.........................................12
3.3.3.1 Control Plane Configuration Set...................12 3.3.3.1 Control Plane Configuration Set...................12
3.3.3.2 Control Plane Startup Conditions..................13 3.3.3.2 Control Plane Startup Conditions..................13
3.3.4 Data Plane............................................13 3.3.4 Data Plane............................................13
3.3.4.1 Data Plane Configuration Set......................13 3.3.4.1 Data Plane Configuration Set......................13
3.3.4.2 Traffic Profile...................................13 3.3.4.2 Traffic Profile...................................14
3.3.5 Management Plane......................................14 3.3.5 Management Plane......................................14
3.3.5.1 Management Plane Configuration Set................14 3.3.5.1 Management Plane Configuration Set................14
3.3.6 Security Plane........................................15 3.3.6 Security Plane........................................15
3.3.6.1 Security Plane Configuration Set..................15 3.3.6.1 Security Plane Configuration Set..................15
3.3.6.2 Security Plane Startup Conditions.................16 3.3.6.2 Security Plane Startup Conditions.................16
3.4 Instability...............................................16 3.4 Instability...............................................17
3.4.1 Instability Phase.....................................16 3.4.1 Instability Phase.....................................17
3.4.2 Benchmarks............................................17 3.4.2 Benchmarks............................................17
3.4.2.1 Unstable Aggregate Forwarding Rate................17 3.4.2.1 Unstable Aggregate Forwarding Rate................17
3.4.2.2 Aggregate Forwarding Rate Degradation.............17 3.4.2.2 Aggregate Forwarding Rate Degradation.............18
3.4.2.3 Average Aggregate Forwarding Rate Degradation.....18 3.4.2.3 Average Aggregate Forwarding Rate Degradation.....18
3.4.2.4 Unstable Latency..................................18 3.4.2.4 Unstable Latency..................................19
3.4.2.5 Unstable Uncontrolled Sessions Lost...............19 3.4.2.5 Unstable Uncontrolled Sessions Lost...............19
3.4.3 Instability Conditions................................19 3.4.3 Instability Conditions................................20
3.4.3.1 Control Plane Instability Conditions..............19 3.4.3.1 Control Plane Instability Conditions..............20
3.4.3.2 Data Plane Instability Conditions.................20 3.4.3.2 Data Plane Instability Conditions.................20
3.4.3.3 Management Plane Instability Conditions...........20 3.4.3.3 Management Plane Instability Conditions...........21
3.4.3.4 Security Plane Instability Conditions.............20 3.4.3.4 Security Plane Instability Conditions.............21
3.5 Recovery..................................................21 3.5 Recovery..................................................22
3.5.1 Recovery Phase........................................21 3.5.1 Recovery Phase........................................22
3.5.2 Benchmarks............................................21 3.5.2 Benchmarks............................................22
3.5.2.1 Recovered Aggregate Forwarding Rate...............21 3.5.2.1 Recovered Aggregate Forwarding Rate...............22
3.5.2.2 Recovered Latency.................................22 3.5.2.2 Recovered Latency.................................23
3.5.2.3 Recovery Time.....................................22 3.5.2.3 Recovery Time.....................................23
3.5.2.4 Recovered Uncontrolled Sessions Lost..............23 3.5.2.4 Recovered Uncontrolled Sessions Lost..............24
3.5.2.5 Variability Benchmarks............................23 3.5.2.5 Variability Benchmarks............................24
4. IANA Considerations.........................................24 4. IANA Considerations.........................................25
Stress Benchmarking Stress Benchmarking
5. Security Considerations.....................................24 5. Security Considerations.....................................25
6. References..................................................24 6. Acknowledgements............................................25
7. Author's Address............................................25 7. References..................................................25
Appendix 1 - White Box Benchmarks..............................25 8. Author's Address............................................26
Appendix 1 - White Box Benchmarks..............................26
1. Introduction 1. Introduction
Routers in an operational network are configured with multiple Routers in an operational network are configured with multiple
protocols and security policies while simultaneously forwarding protocols and security policies while simultaneously forwarding
traffic and being managed. To accurately benchmark a router for traffic and being managed. To accurately benchmark a router for
deployment, it is necessary to test that router under operational deployment, it is necessary to test that router under operational
conditions by simultaneously configuring and scaling network conditions by simultaneously configuring and scaling network
protocols and security policies, forwarding traffic, and managing protocols and security policies, forwarding traffic, and managing
the device. It is useful to accelerate these network operational the device in a lab environment. It is useful to accelerate these
conditions so that the router under test can be benchmarked with network operational conditions so that the router under test can
a shorter test duration. Testing a router in accelerated network be benchmarked in a lab environment with a shorter test duration.
conditions is known as Accelerated Stress Benchmarking. Testing a router in accelerated network conditions is known as
Accelerated Stress Benchmarking.
This document provides the Terminology for performing Stress This document provides the Terminology for performing Stress
Benchmarking of networking devices. The three phases of the Stress Benchmarking of networking devices. The three phases of the Stress
Test: Startup, Instability and Recovery are defined along with the Test: Startup, Instability and Recovery are defined along with the
benchmark and configuration terms associated with the each phase. benchmark and configuration terms associated with the each phase.
Benchmarks for stress testing are defined using the Aggregate Benchmarks for stress testing are defined using the Aggregate
Forwarding Rate and control plane Session Count during each phase Forwarding Rate and control plane Session Count during each phase
of the test. For each plane, the Configuration Set, Startup of the test. For each plane, the Configuration Set, Startup
Conditions, and Instability Conditions are defined. Also defined are Conditions, and Instability Conditions are defined. Also defined are
the Benchmark Planes fundamental to stress testing configuration, the Benchmark Planes fundamental to stress testing configuration,
skipping to change at page 12, line 19 skipping to change at page 12, line 19
Issues: Issues:
None None
See Also: See Also:
Startup Phase Startup Phase
3.3.3 Control Plane 3.3.3 Control Plane
3.3.3.1 Control Plane Configuration Set 3.3.3.1 Control Plane Configuration Set
Definition: Definition:
The routing protocols and scaling values used for the Accelerated The routing protocols and scaling values used for the
Stress Benchmarking. Accelerated Stress Benchmarking.
Discussion: Discussion:
Control Plane Configuration Set is shown in Figure 2 and specifies Control Plane Configuration Set is represented in Figure 2 and
the Routing Protocols, Multicast, and MPLS configuration. Specific specifies protocol configurations for protocols such as, but not
limitied to, Routing, Multicast, and MPLS. Specific
protocols can be enabled or disabled for a benchmark test. protocols can be enabled or disabled for a benchmark test.
Measurement units: Measurement units:
N/A N/A
Issues: Issues:
None None
See Also: See Also:
Data Plane Configuration Set Data Plane Configuration Set
skipping to change at page 12, line 50 skipping to change at page 12, line 52
| Routing | | Multicast | | MPLS | | Routing | | Multicast | | MPLS |
| Protocols |___ | Protocols | __| Protocols | | Protocols |___ | Protocols | __| Protocols |
| | | | | | | | | | | | | | | |
------------ | ------------ | ------------ ------------ | ------------ | ------------
| | | | | |
| | | | | |
| \/ | | \/ |
| ___________ | | ___________ |
| | | | | | | |
|------->| DUT |<------| |------->| DUT |<------|
``| | | |
----------- -----------
Figure 2. Control Plane Configuration Module Figure 2. Control Plane Configuration Module
Stress Benchmarking Stress Benchmarking
3.3.3.2 Control Plane Startup Conditions 3.3.3.2 Control Plane Startup Conditions
Definition: Definition:
Control Plane conditions that occur at the start Control Plane conditions that occur at the start
of the Accelerated Stress Benchmarking to establish conditions of the Accelerated Stress Benchmarking to establish conditions
for the remainder of the test. for the remainder of the test.
skipping to change at page 13, line 34 skipping to change at page 13, line 34
See Also: See Also:
Startup Conditions Startup Conditions
Security Plane Startup Conditions Security Plane Startup Conditions
Control Plane Configuration Set Control Plane Configuration Set
3.3.4 Data Plane 3.3.4 Data Plane
3.3.4.1 Data Plane Configuration Set 3.3.4.1 Data Plane Configuration Set
Definition: Definition:
The data traffic profile enabled for the Accelerated Stress The data traffic profile and interfaces that are enabled for
Benchmarking. the Accelerated Stress Benchmarking.
Discussion: Discussion:
Data Plane Configuration Set includes the Traffic Profile and Data Plane Configuration Set includes the Traffic Profile and
interfaces used for the Accelerated Stress Benchmarking. interfaces used for the Accelerated Stress Benchmarking.
The interface type(s) and number of interfaces for each
interface type MUST be reported.
Measurement Units: Measurement Units:
N/A N/A
Issues: None Issues: None
See Also: See Also:
Traffic Profile Traffic Profile
Stress Benchmarking
3.3.4.2 Traffic Profile 3.3.4.2 Traffic Profile
Definition Definition
The characteristics of the Offered Load to the DUT used for The characteristics of the Offered Load to the DUT on each
the Accelerated Stress Benchmarking. interface for the Accelerated Stress Benchmarking.
Discussion Discussion
The Traffic Profile specifies the number of packet size(s), The Traffic Profile specifies the number of packet size(s),
packet rate per interface, number of flows, and encapsulation packet rate, number of flows, and encapsulation on a
used for the offered load to the DUT. per-interface basis used for the offered load to the DUT.
Stress Benchmarking
Measurement Units: Measurement Units:
Traffic Profile is reported as follows: Traffic Profile is reported as follows:
Parameter Units Parameter Units
--------- ------ --------- ------
Packet Size(s) bytes Packet Size(s) bytes
Packet Rate(interface) array of packets per second Packet Rate(interface) array of packets per second
Number of Flows number Number of Flows number of flows
Encapsulation(flow) array of encapsulation type Encapsulation(flow) array of encapsulation type
Mirroring enabled/disabled
Issues: Issues:
None None
See Also: See Also:
Data Plane Configuration Set Data Plane Configuration Set
3.3.5 Management Plane 3.3.5 Management Plane
3.3.5.1 Management Plane Configuration Set 3.3.5.1 Management Plane Configuration Set
skipping to change at page 14, line 40 skipping to change at page 15, line 5
Discussion: Discussion:
A key component of the Accelerated Stress Benchmark is the A key component of the Accelerated Stress Benchmark is the
Management Configuration Set to assess manageability of the Management Configuration Set to assess manageability of the
router under stress. The Management Configuration Set defines router under stress. The Management Configuration Set defines
the management configuration of the DUT. Features that are the management configuration of the DUT. Features that are
part of the Management Configuration Set include access, SNMP, part of the Management Configuration Set include access, SNMP,
Logging/Debug, and Statistics Collection, and services such as Logging/Debug, and Statistics Collection, and services such as
FTP, as shown in Figure 3. These features SHOULD be enabled FTP, as shown in Figure 3. These features SHOULD be enabled
throughout the Stress test. throughout the Stress test.
Stress Benchmarking
Measurement units: Measurement units:
N/A N/A
Issues: Issues:
None None
See Also: See Also:
Control Plane Configuration Set Control Plane Configuration Set
Data Plane Configuration Set Data Plane Configuration Set
Security Plane Configuration Set Security Plane Configuration Set
Stress Benchmarking
____________ ____________ ____________ ____________
| | | Logging/ | | | | Logging/ |
| SNMP | __| Debug | | SNMP | __| Debug |
| | | | | | | | | |
------------ | ------------ ------------ | ------------
| | | |
| | | |
\/ | \/ |
___________ | ___________ |
| | | | | |
skipping to change at page 15, line 44 skipping to change at page 16, line 5
Security features and scaling enabled for the Accelerated Stress Security features and scaling enabled for the Accelerated Stress
Test. Test.
Discussion: Discussion:
The Security Plane Configuration Set includes the configuration The Security Plane Configuration Set includes the configuration
and scaling of ACLs, Firewall, IPsec, and User Access, as shown and scaling of ACLs, Firewall, IPsec, and User Access, as shown
in Figure 4. Tunnels SHOULD be established and policies in Figure 4. Tunnels SHOULD be established and policies
configured. Instability is introduced by flapping tunnels and configured. Instability is introduced by flapping tunnels and
configuring and removing policies. configuring and removing policies.
Stress Benchmarking
____________ ____________ ____________ ____________ ____________ ____________
| | | Secure | | User | | | | Secure | | User |
|ACL/Firewall| | Protocol | __| Access | |ACL/Firewall| | Protocol | __| Access |
| | | | | | | | | | | | | |
------------ ------------ | ------------ ------------ ------------ | ------------
| | | | | |
| | | | | |
| \/ | | \/ |
| ___________ | | ___________ |
| | | | | | | |
|------->| DUT |<--------| |------->| DUT |<--------|
| | | |
----------- -----------
Figure 4. Security Configuration Module Figure 4. Security Configuration Module
Stress Benchmarking
Measurement units: Measurement units:
N/A N/A
Issues: Issues:
None None
See Also: See Also:
ACL Configuration Set ACL Configuration Set
Secure Protocol Configuration Set Secure Protocol Configuration Set
skipping to change at page 16, line 41 skipping to change at page 17, line 4
N/A N/A
Issues: Issues:
None None
See Also: See Also:
Startup Conditions Startup Conditions
Data Plane Startup Conditions Data Plane Startup Conditions
Management Plane Startup Conditions Management Plane Startup Conditions
Security Plane Startup Conditions Security Plane Startup Conditions
Stress Benchmarking
3.4 Instability 3.4 Instability
3.4.1 Instability Phase 3.4.1 Instability Phase
Definition: Definition:
The portion of the benchmarking test in which the The portion of the benchmarking test in which the
Instability Conditions are offered to the DUT. Instability Conditions are offered to the DUT.
Discussion: Discussion:
The Instability Phase is the middle Phase of The Instability Phase is the middle Phase of
of the benchmarking test following the Startup of the benchmarking test following the Startup
Phase and preceding the Recovery Phase. Phase and preceding the Recovery Phase.
Measurement Units: Measurement Units:
None None
Stress Benchmarking
Issues: Issues:
None None
See Also: See Also:
Instability Conditions Instability Conditions
Startup Phase Startup Phase
Recovery Phase Recovery Phase
3.4.2 Benchmarks 3.4.2 Benchmarks
skipping to change at page 17, line 35 skipping to change at page 18, line 4
Measurement units: Measurement units:
pps pps
Issues: Issues:
None None
See Also: See Also:
Instability Conditions Instability Conditions
Aggregate Forwarding Rate Aggregate Forwarding Rate
Stress Benchmarking
3.4.2.2 Aggregate Forwarding Rate Degradation 3.4.2.2 Aggregate Forwarding Rate Degradation
Definition: Definition:
The reduction in Aggregate Forwarding Rate during the The reduction in Aggregate Forwarding Rate during the
Instability Phase. Instability Phase.
Discussion: Discussion:
The Aggregate Forwarding Rate Degradation is calculated The Aggregate Forwarding Rate Degradation is calculated
for each measurement of the Unstable Aggregate Forwarding for each measurement of the Unstable Aggregate Forwarding
skipping to change at page 18, line 4 skipping to change at page 18, line 28
Forwarding Rate, such that Forwarding Rate, such that
Aggregate Forwarding Rate Degradation= Aggregate Forwarding Rate Degradation=
Stable Aggregate Forwarding Rate - Stable Aggregate Forwarding Rate -
Unstable Aggregate Forwarding Rate Unstable Aggregate Forwarding Rate
Ideally, the Aggregate Forwarding Rate Degradation is zero. Ideally, the Aggregate Forwarding Rate Degradation is zero.
Measurement Units: Measurement Units:
pps pps
Stress Benchmarking
Issues: Issues:
None None
See Also: See Also:
Instability Phase Instability Phase
Unstable Aggregate Forwarding Rate Unstable Aggregate Forwarding Rate
3.4.2.3 Average Aggregate Forwarding Rate Degradation 3.4.2.3 Average Aggregate Forwarding Rate Degradation
skipping to change at page 18, line 32 skipping to change at page 19, line 4
Sum (Unstable Aggregate Forwarding Rate)) / Number of Samples Sum (Unstable Aggregate Forwarding Rate)) / Number of Samples
Measurement Units: Measurement Units:
pps pps
Issues: Issues:
None None
See Also: See Also:
Aggregate Forwarding Rate Degradation Aggregate Forwarding Rate Degradation
Stress Benchmarking
3.4.2.4 Unstable Latency 3.4.2.4 Unstable Latency
Definition: Definition:
The average increase in measured packet latency during The average increase in measured packet latency during
the Instability Phase compared to the Startup Phase. the Instability Phase compared to the Startup Phase.
Discussion: Discussion:
Latency SHOULD be measured at a fixed interval during the Latency SHOULD be measured at a fixed interval during the
Instability Phase. Unstable Latency is the difference Instability Phase. Unstable Latency is the difference
skipping to change at page 19, line 4 skipping to change at page 19, line 31
Measurement units: Measurement units:
seconds seconds
Issues: Issues:
None None
See Also: See Also:
Instability Phase Instability Phase
Stable Latency Stable Latency
Stress Benchmarking
3.4.2.5 Unstable Uncontrolled Sessions Lost 3.4.2.5 Unstable Uncontrolled Sessions Lost
Definition: Definition:
Control Plane sessions that are in the down state Control Plane sessions that are in the down state
but were not intentionally brought down during the but were not intentionally brought down during the
Instability Phase. Instability Phase.
Discussion: Discussion:
The test equipment is able to control protocol The test equipment is able to control protocol
skipping to change at page 19, line 29 skipping to change at page 20, line 4
Measurement units: Measurement units:
sessions sessions
Issues: Issues:
None None
See Also: See Also:
Controlled Session Loss Controlled Session Loss
Uncontrolled Session Loss Uncontrolled Session Loss
Stress Benchmarking
3.4.3 Instability Conditions 3.4.3 Instability Conditions
3.4.3.1 Control Plane Instability Conditions 3.4.3.1 Control Plane Instability Conditions
Definition: Definition:
Control Plane conditions that occur during the Accelerated Stress Control Plane conditions that occur during the Accelerated
Benchmark to produce instability and stress the DUT. Stress Benchmark to produce instability and stress the DUT.
Discussion: Discussion:
Control Plane Instability Conditions are experienced by the DUT Control Plane Instability Conditions are experienced by the DUT
after the Startup Conditions have completed. Control Plane after the Startup Conditions have completed. Control Plane
Instability Conditions experienced by the DUT include session Instability Conditions experienced by the DUT include session
loss, route withdrawal, and route cost changes. loss, route withdrawal, and route cost changes.
Measurement units: Measurement units:
N/A N/A
Issues: Issues:
None None
See Also: See Also:
Instability Conditions Instability Conditions
Data Plane Instability Conditions Data Plane Instability Conditions
Management Plane Instability Conditions Management Plane Instability Conditions
Security Plane Instability Conditions Security Plane Instability Conditions
Stress Benchmarking
3.4.3.2 Data Plane Instability Conditions 3.4.3.2 Data Plane Instability Conditions
Definition: Definition:
Data Plane conditions that occur during the Accelerated Stress Data Plane conditions that occur during the Accelerated Stress
Benchmark to produce instability and stress the DUT. Benchmark to produce instability and stress the DUT.
Discussion: Discussion:
Data Plane Instability Conditions are experienced by the DUT Data Plane Instability Conditions are experienced by the DUT
after the Startup Conditions have completed. Data Plane after the Startup Conditions have completed. Data Plane
Instability Conditions experienced by the DUT include interface Instability Conditions experienced by the DUT include interface
skipping to change at page 20, line 28 skipping to change at page 21, line 4
N/A N/A
Issues: Issues:
None None
See Also: See Also:
Instability Conditions Instability Conditions
Control Plane Instability Conditions Control Plane Instability Conditions
Management Plane Instability Conditions Management Plane Instability Conditions
Security Plane Instability Conditions Security Plane Instability Conditions
Stress Benchmarking
3.4.3.3 Management Plane Instability Conditions 3.4.3.3 Management Plane Instability Conditions
Definition: Definition:
Management Plane conditions that occur during the Accelerated Management Plane conditions that occur during the Accelerated
Stress Benchmark to produce instability and stress the DUT. Stress Benchmark to produce instability and stress the DUT.
Discussion: Discussion:
Management Plane Instability Conditions are experienced by the DUT Management Plane Instability Conditions are experienced by the
after the Startup Conditions have completed. Management Plane DUT after the Startup Conditions have completed. Management
Instability Conditions experienced by the DUT include repeated Plane Instability Conditions experienced by the DUT include
FTP of large files. repeated FTP of large files.
Measurement units: Measurement units:
N/A N/A
Issues: Issues:
None None
See Also: See Also:
Instability Conditions Instability Conditions
Control Plane Instability Conditions Control Plane Instability Conditions
Data Plane Instability Conditions Data Plane Instability Conditions
Security Plane Instability Conditions Security Plane Instability Conditions
3.4.3.4 Security Plane Instability Conditions 3.4.3.4 Security Plane Instability Conditions
Definition: Definition:
Security Plane conditions that occur during the Accelerated Security Plane conditions that occur during the Accelerated
Stress Benchmark to produce instability and stress the DUT. Stress Benchmark to produce instability and stress the DUT.
Stress Benchmarking
Discussion: Discussion:
Security Plane Instability Conditions are experienced by the DUT Security Plane Instability Conditions are experienced by the DUT
after the Startup Conditions have completed. Security Plane after the Startup Conditions have completed. Security Plane
Instability Conditions experienced by the DUT include session Instability Conditions experienced by the DUT include session
loss and uninitiated policy changes. loss and uninitiated policy changes.
Measurement units: Measurement units:
N/A N/A
Issues: Issues:
None None
See Also: See Also:
Instability Conditions Instability Conditions
Control Plane Instability Conditions Control Plane Instability Conditions
Data Plane Instability Conditions Data Plane Instability Conditions
Management Plane Instability Conditions Management Plane Instability Conditions
Stress Benchmarking
3.5 Recovery 3.5 Recovery
3.5.1 Recovery Phase 3.5.1 Recovery Phase
Definition: Definition:
The portion of the benchmarking test in which the The portion of the benchmarking test in which the
Startup Conditions are generated with the DUT, but Startup Conditions are generated with the DUT, but
the Instability Conditions are no longer offered to the Instability Conditions are no longer offered to
the DUT. the DUT.
Discussion: Discussion:
The Recovery Phase is the final Phase of the The Recovery Phase is the final Phase of the
skipping to change at page 21, line 41 skipping to change at page 22, line 21
Startup Conditions are generated with the DUT, but Startup Conditions are generated with the DUT, but
the Instability Conditions are no longer offered to the Instability Conditions are no longer offered to
the DUT. the DUT.
Discussion: Discussion:
The Recovery Phase is the final Phase of the The Recovery Phase is the final Phase of the
benchmarking test following the Startup Phase and benchmarking test following the Startup Phase and
Instability Phase. Startup Conditions MUST NOT be Instability Phase. Startup Conditions MUST NOT be
Restarted. Restarted.
Measurement Units: Measurement Units: None
None
Issues: Issues: None
None
See Also: See Also:
Startup Conditions Startup Conditions
Startup Phase Startup Phase
Instability Conditions Instability Conditions
Instability Phase Instability Phase
3.5.2 Benchmarks 3.5.2 Benchmarks
3.5.2.1 Recovered Aggregate Forwarding Rate 3.5.2.1 Recovered Aggregate Forwarding Rate
Definition Definition
Rate of traffic forwarded by the DUT during the Recovery Rate of traffic forwarded by the DUT during the Recovery
Phase. Phase.
Stress Benchmarking
Discussion: Discussion:
Recovered Aggregate Forwarding Rate is an instantaneous Recovered Aggregate Forwarding Rate is an instantaneous
measurement of the Aggregate Forwarding Rate during the measurement of the Aggregate Forwarding Rate during the
Recovery Phase. Ideally, each measurement of the Recovered Recovery Phase. Ideally, each measurement of the Recovered
Aggregate Forwarding Rate equals the Stable Aggregate Aggregate Forwarding Rate equals the Stable Aggregate
Forwarding Rate because the Instability Conditions Forwarding Rate because the Instability Conditions
do not exist in both the Startup and Recovery Phases. do not exist in both the Startup and Recovery Phases.
Measurement Units: Measurement Units:
pps pps
Issues: Issues: None
None
See Also: See Also:
Aggregate Forwarding Rate Aggregate Forwarding Rate
Recovery Phase Recovery Phase
Recovered Aggregate Forwarding Rate Recovered Aggregate Forwarding Rate
Startup Phase Startup Phase
Stable Aggregate Forwarding Rate Stable Aggregate Forwarding Rate
Stress Benchmarking
3.5.2.2 Recovered Latency 3.5.2.2 Recovered Latency
Definition: Definition:
The average increase in measured packet latency during The average increase in measured packet latency during
the Recovery Phase compared to the Startup Phase. the Recovery Phase compared to the Startup Phase.
Discussion: Discussion:
Latency SHOULD be measured at a fixed interval during the Latency SHOULD be measured at a fixed interval during the
Recovery Phase. Unstable Latency is the difference Recovery Phase. Unstable Latency is the difference
skipping to change at page 23, line 5 skipping to change at page 23, line 36
See Also: See Also:
Recovery Phase Recovery Phase
Stable Latency Stable Latency
3.5.2.3 Recovery Time 3.5.2.3 Recovery Time
Definition Definition
The amount of time for the Recovered Aggregate Forwarding The amount of time for the Recovered Aggregate Forwarding
Rate to become equal to the Stable Aggregate Forwarding Rate. Rate to become equal to the Stable Aggregate Forwarding Rate.
Stress Benchmarking
Discussion Discussion
Recovery Time is measured beginning at the instant the Recovery Time is measured beginning at the instant the
Instability Phase ends until the Recovered Aggregate Instability Phase ends until the Recovered Aggregate
Forwarding Rate equals the Stable Aggregate Forwarding Forwarding Rate equals the Stable Aggregate Forwarding
Rate for a minimum duration of 180 consecutive seconds. Rate for a minimum duration of 180 consecutive seconds.
Measurement Units: Measurement Units:
milliseconds milliseconds
Issues: Issues:
None None
See Also: See Also:
Recovered Aggregate Forwarding Rate Recovered Aggregate Forwarding Rate
Stable Aggregate Forwarding Rate Stable Aggregate Forwarding Rate
Stress Benchmarking
3.5.2.4 Recovered Uncontrolled Control Plane Sessions Lost 3.5.2.4 Recovered Uncontrolled Control Plane Sessions Lost
Definition: Definition:
Control Plane sessions that are in the down state Control Plane sessions that are in the down state
but were not intentionally brought down during the but were not intentionally brought down during the
Recovery Phase. Recovery Phase.
Discussion: Discussion:
The test equipment is able to control protocol The test equipment is able to control protocol
skipping to change at page 24, line 5 skipping to change at page 24, line 43
The difference between the measured Benchmarks of the The difference between the measured Benchmarks of the
same DUT over multiple iterations. same DUT over multiple iterations.
Discussion: Discussion:
Ideally, the measured benchmarks should be the same for multiple Ideally, the measured benchmarks should be the same for multiple
iterations with the same DUT. Configuration Sets and iterations with the same DUT. Configuration Sets and
Instability Conditions MUST be held constant for this Instability Conditions MUST be held constant for this
benchmark. Whether the DUT can exhibit such predictable and benchmark. Whether the DUT can exhibit such predictable and
repeatable behavior is an important benchmark in itself. repeatable behavior is an important benchmark in itself.
Stress Benchmarking
Measurement units: Measurement units:
As applicable to each Benchmark. The results are to be As applicable to each Benchmark. The results are to be
presented in a table format for successive Iterations. presented in a table format for successive Iterations.
Ideally, the differences should be zero. Ideally, the differences should be zero.
Issues: Issues:
None None
See Also: See Also:
Startup Period Startup Period
Instability Period Instability Period
Recovery Period Recovery Period
Stress Benchmarking
4. IANA Considerations 4. IANA Considerations
This document requires no IANA considerations. This document requires no IANA considerations.
5. Security Considerations 5. Security Considerations
Documents of this type do not directly affect the security of Documents of this type do not directly affect the security of
the Internet or of corporate networks as long as benchmarking the Internet or of corporate networks as long as benchmarking
is not performed on devices or systems connected to operating is not performed on devices or systems connected to operating
networks. networks.
6. References 6. Acknowledgements
6.1 Normative References The authors would like to thank the BMWG and particularly
Al Morton, Jay Karthik, and George Jones for their contributions.
7. References
7.1 Normative References
[1] Bradner, S., Editor, "Benchmarking Terminology for Network [1] Bradner, S., Editor, "Benchmarking Terminology for Network
Interconnection Devices", RFC 1242, March 1991. Interconnection Devices", RFC 1242, March 1991.
[2] Mandeville, R., "Benchmarking Terminology for LAN Switching [2] Mandeville, R., "Benchmarking Terminology for LAN Switching
Devices", RFC 2285, June 1998. Devices", RFC 2285, June 1998.
[3] Bradner, S. and McQuaid, J., "Benchmarking Methodology for [3] Bradner, S. and McQuaid, J., "Benchmarking Methodology for
Network Interconnect Devices", RFC 2544, March 1999. Network Interconnect Devices", RFC 2544, March 1999.
[4] Poretsky, S. and Rao, S., "Methodology Guidelines for [4] Poretsky, S. and Rao, S., "Methodology Guidelines for
Accelerated Stress Benchmarking", Accelerated Stress Benchmarking",
draft-ietf-bmwg-acc-bench-meth-07, work in progress, draft-ietf-bmwg-acc-bench-meth-07, work in progress,
March 2007. March 2007.
[5] Bradner, S., "Key words for use in RFCs to Indicate [5] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", RFC 2119, March 1997. Requirement Levels", RFC 2119, March 1997.
6.2 Informative References 7.2 Informative References
[RFC3871] Jones, G., "Operational Security Requirements for Large [RFC3871] Jones, G., "Operational Security Requirements for Large
Internet Service Provider (ISP) IP Network Infrastructure.", Internet Service Provider (ISP) IP Network Infrastructure.",
IETF RFC 3871 , September 2004. IETF RFC 3871 , September 2004.
[NANOG25] Poretsky, S., "Core Router Evaluation for Higher [NANOG25] Poretsky, S., "Core Router Evaluation for Higher
Availability", NANOG 25, June 8, 2002, Toronto, CA. Availability", NANOG 25, June 8, 2002, Toronto, CA.
[IEEECQR] Poretsky, S., "Router Stress Testing to Validate [IEEECQR] Poretsky, S., "Router Stress Testing to Validate
Readiness for Network Deployment", IEEE CQR 2003. Readiness for Network Deployment", IEEE CQR 2003.
Stress Benchmarking Stress Benchmarking
7. Author's Address 8. Author's Address
Scott Poretsky Scott Poretsky
Reef Point Systems Reef Point Systems
8 New England Executive Park 8 New England Executive Park
Burlington, MA 01803 Burlington, MA 01803 USA
USA
Phone: + 1 781 395 5090 Phone: + 1 781 395 5090
EMail: sporetsky@reefpoint.com EMail: sporetsky@reefpoint.com
Shankar Rao Shankar Rao
1801 California Street 1801 California Street
8th Floor 8th Floor
Qwest Communications Qwest Communications
Denver, CO 80202 Denver, CO 80202 USA
USA
Phone: + 1 303 437 6643 Phone: + 1 303 437 6643
Email: shankar.rao@qwest.com Email: shankar.rao@qwest.com
Appendix 1. White Box Benchmarking Terminology Appendix 1. White Box Benchmarking Terminology
Minimum Available Memory Minimum Available Memory
Definition: Definition:
Minimum DUT Available Memory during the duration of the Minimum DUT Available Memory during the duration of the
Accelerated Stress Benchmark. Accelerated Stress Benchmark.
Discussion: Discussion:
 End of changes. 55 change blocks. 
79 lines changed or deleted 83 lines changed or added

This html diff was produced by rfcdiff 1.33. The latest version is available from http://tools.ietf.org/tools/rfcdiff/