draft-ietf-bmwg-acc-bench-term-12.txt   draft-ietf-bmwg-acc-bench-term-13.txt 
Network Working Group Network Working Group S. Poretsky
INTERNET-DRAFT Internet Draft NextPoint Networks
Expires in: January 2008 Expires: August 2008
Intended Status: Informational Intended Status: Informational Shankar Rao
Scott Poretsky
Reef Point Systems
Shankar Rao
Qwest Communications Qwest Communications
February 25, 2008
Terminology for Accelerated Stress Benchmarking Terminology for Accelerated Stress Benchmarking
<draft-ietf-bmwg-acc-bench-term-12.txt> <draft-ietf-bmwg-acc-bench-term-13.txt>
Intellectual Property Rights (IPR) statement: Intellectual Property Rights (IPR) statement:
By submitting this Internet-Draft, each author represents that any By submitting this Internet-Draft, each author represents that any
applicable patent or other IPR claims of which he or she is aware applicable patent or other IPR claims of which he or she is aware
have been or will be disclosed, and any of which he or she becomes have been or will be disclosed, and any of which he or she becomes
aware will be disclosed, in accordance with Section 6 of BCP 79. aware will be disclosed, in accordance with Section 6 of BCP 79.
Status of this Memo Status of this Memo
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
skipping to change at page 1, line 39 skipping to change at page 1, line 38
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt. http://www.ietf.org/ietf/1id-abstracts.txt.
The list of Internet-Draft Shadow Directories can be accessed at The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html. http://www.ietf.org/shadow.html.
Copyright Notice Copyright Notice
Copyright (C) The IETF Trust (2007). Copyright (C) The IETF Trust (2008).
ABSTRACT ABSTRACT
This document provides the Terminology for performing Accelerated This document provides the Terminology for performing Accelerated
Stress Benchmarking of networking devices. The three phases of Stress Benchmarking of networking devices. The three phases of
the Stress Test: Startup, Instability and Recovery are defined the Stress Test: Startup, Instability and Recovery are defined
along with the benchmarks and configuration terms associated with along with the benchmarks and configuration terms associated with
the each phase. Also defined are the Benchmark Planes fundamental the each phase. Also defined are the Benchmark Planes fundamental
to stress testing configuration, setup and measurement. The to stress testing configuration, setup and measurement. The
terminology is to be used with the companion framework and terminology is to be used with the companion framework and
methodology documents. methodology documents.
skipping to change at page 2, line 12 skipping to change at page 2, line 12
2. Existing definitions ....................................... 3 2. Existing definitions ....................................... 3
3. Term definitions............................................ 4 3. Term definitions............................................ 4
Stress Benchmarking Stress Benchmarking
3.1 General Terms............................................. 4 3.1 General Terms............................................. 4
3.1.1 Benchmark Planes...................................... 4 3.1.1 Benchmark Planes...................................... 4
3.1.2 Configuration Sets.................................... 5 3.1.2 Configuration Sets.................................... 5
3.1.3 Startup Conditions.................................... 5 3.1.3 Startup Conditions.................................... 5
3.1.4 Instability Conditions................................ 6 3.1.4 Instability Conditions................................ 6
3.1.5 Aggregate Forwarding Rate............................. 6 3.1.5 Aggregate Forwarding Rate............................. 6
3.1.6 Controlled Session Loss............................... 7 3.1.6 Discontinued Sessions................................. 7
3.1.7 Uncontrolled Session Loss............................. 7 3.1.7 Uncontrolled Session Loss............................. 7
3.2 Benchmark Planes.......................................... 8 3.2 Benchmark Planes.......................................... 8
3.2.1 Control Plane......................................... 8 3.2.1 Control Plane......................................... 8
3.2.2 Data Plane............................................ 8 3.2.2 Data Plane............................................ 8
3.2.3 Management Plane...................................... 8 3.2.3 Management Plane...................................... 8
3.2.4 Security Plane........................................ 9 3.2.4 Security Plane........................................ 9
3.3 Startup...................................................10 3.3 Startup...................................................10
3.3.1 Startup Phase.........................................10 3.3.1 Startup Phase.........................................10
3.3.2 Benchmarks............................................10 3.3.2 Benchmarks............................................10
3.3.2.1 Stable Aggregate Forwarding Rate..................10 3.3.2.1 Stable Aggregate Forwarding Rate..................10
skipping to change at page 3, line 36 skipping to change at page 3, line 36
This document provides the Terminology for performing Stress This document provides the Terminology for performing Stress
Benchmarking of networking devices. The three phases of the Stress Benchmarking of networking devices. The three phases of the Stress
Test: Startup, Instability and Recovery are defined along with the Test: Startup, Instability and Recovery are defined along with the
benchmark and configuration terms associated with the each phase. benchmark and configuration terms associated with the each phase.
Benchmarks for stress testing are defined using the Aggregate Benchmarks for stress testing are defined using the Aggregate
Forwarding Rate and control plane Session Count during each phase Forwarding Rate and control plane Session Count during each phase
of the test. For each plane, the Configuration Set, Startup of the test. For each plane, the Configuration Set, Startup
Conditions, and Instability Conditions are defined. Also defined are Conditions, and Instability Conditions are defined. Also defined are
the Benchmark Planes fundamental to stress testing configuration, the Benchmark Planes fundamental to stress testing configuration,
setup and measurement. These are the Control Plane, Data Plane, setup and measurement. These are the Control Plane, Data Plane,
Management Plane and Security Plane Multiple benchmarks are measured Management Plane and Security Plane. Multiple benchmarks are measured
for each Benchmark Plane during each Phase. Benchmarks can be for each Benchmark Plane during each Phase. Benchmarks can be
compared across multiple planes for the same DUT or at the same compared across multiple planes for the same DUT or at the same
plane for 2 or more DUTS. Benchmarks of internal DUT characteristics plane for 2 or more DUTS. Benchmarks of internal DUT characteristics
such as memory and CPU utilization (also known as White Box such as memory and CPU utilization (also known as White Box
benchmarks) are described in Appendix 1, to allow additional benchmarks) are described in Appendix 1, to allow additional
characterization of DUT behavior. The terminology is to be used with characterization of DUT behavior. The terminology is to be used with
the companion methodology document [4]. The sequence of phases, the companion methodology document [4]. The sequence of phases,
actions, and benchmarks are shown in Table 1. actions, and benchmarks are shown in Table 1.
2. Existing definitions 2. Existing definitions
skipping to change at page 4, line 11 skipping to change at page 4, line 11
[5]. RFC 2119 defines the use of these key words to help make the [5]. RFC 2119 defines the use of these key words to help make the
intent of standards track documents as clear as possible. While this intent of standards track documents as clear as possible. While this
document uses these keywords, it is not a standards track document. document uses these keywords, it is not a standards track document.
Stress Benchmarking Stress Benchmarking
Table 1. Phase Sequence and Benchmarks Table 1. Phase Sequence and Benchmarks
III. Recovery Phase II. Instability Phase I. Startup Phase III. Recovery Phase II. Instability Phase I. Startup Phase
<-----------------<---<-------------------<----<--------------< <-----------------<---<-------------------<----<--------------<
Remove Instability Achieve Configuration Apply Startup Remove Instability Achieve Configuration Apply Startup
Conditions Set Conditions Conditions Set and Apply Conditions
Instability Conditions
Benchmark: Benchmark: Benchmark: Benchmark: Benchmark: Benchmark:
Recovered Aggregate Unstable Aggregate Stable Aggregate Recovered Aggregate Unstable Aggregate Stable Aggregate
Forwarding Rate Forwarding Rate Forwarding Rate Forwarding Rate Forwarding Rate Forwarding Rate
Degraded Aggregate Degraded Aggregate
Forwarding Rate Forwarding Rate
Average Degraded Average Degraded
Forwarding Rate Forwarding Rate
skipping to change at page 5, line 54 skipping to change at page 5, line 54
See Also: See Also:
Control Plane Configuration Set Control Plane Configuration Set
Data Plane Configuration Set Data Plane Configuration Set
Management Plane Configuration Set Management Plane Configuration Set
Security Plane Configuration Set Security Plane Configuration Set
3.1.3 Startup Conditions 3.1.3 Startup Conditions
Definition: Definition:
Test conditions that occur at the start of the Accelerated Test conditions applied at the start of the Accelerated
Stress Benchmark to establish conditions for the remainder of Stress Benchmark to establish conditions for the remainder of
the test. the test.
Stress Benchmarking Stress Benchmarking
Discussion: Discussion:
Startup Conditions may cause stress on the DUT and produce Startup Conditions may cause stress on the DUT and produce
failure. Startup Conditions are defined for the Control failure. Startup Conditions are defined for the Control
Plane and Security Plane. Plane and Security Plane.
skipping to change at page 6, line 27 skipping to change at page 6, line 27
See Also: See Also:
Control Plane Startup Conditions Control Plane Startup Conditions
Data Plane Startup Conditions Data Plane Startup Conditions
Management Plane Startup Conditions Management Plane Startup Conditions
Security Plane Startup Conditions Security Plane Startup Conditions
3.1.4 Instability Conditions 3.1.4 Instability Conditions
Definition: Definition:
Test conditions that occur during the Accelerated Stress Test conditions applied during the Accelerated Stress
Benchmark to produce instability and stress the DUT. Benchmark to produce instability and stress the DUT.
Discussion: Discussion:
Instability Conditions are applied to the DUT after the Instability Conditions are applied to the DUT after the
Startup Conditions have completed. Instability Conditions Startup Conditions have completed. Instability Conditions
occur for the Control Plane, Data Plane, Management Plane, occur for the Control Plane, Data Plane, Management Plane,
and Security Plane. and Security Plane.
Measurement units: Measurement units:
N/A N/A
skipping to change at page 6, line 51 skipping to change at page 6, line 51
See Also: See Also:
Control Plane Instability Conditions Control Plane Instability Conditions
Data Plane Instability Conditions Data Plane Instability Conditions
Management Plane Instability Conditions Management Plane Instability Conditions
Security Plane Instability Conditions Security Plane Instability Conditions
3.1.5 Aggregate Forwarding Rate 3.1.5 Aggregate Forwarding Rate
Definition: Definition:
Sum of forwarding rates for all interfaces on the Sum of forwarding rates for all interfaces on the
DUT during the Startup Phase. DUT.
Discussion: Discussion:
Each interface of the DUT forwards traffic at some Each interface of the DUT forwards traffic at some
measured rate. The Aggregate Forwarding Rate is the measured rate. The Aggregate Forwarding Rate is the
sum of forwarding rates for all interfaces on the DUT. sum of forwarding rates for all interfaces on the DUT.
Stress Benchmarking Stress Benchmarking
Measurement units: Measurement units:
pps pps
Issues: Issues:
None None
See Also: See Also:
Startup Phase Startup Phase
3.1.6 Controlled Session Loss 3.1.6 Discontinued Sessions
Definition: Definition:
Control Plane sessions that are intentionally brought Control Plane sessions that are intentionally brought
down during the Stress test. down during the Stress test.
Discussion: Discussion:
Controlled Session Loss is performed during the test in Discontinued Sessions is performed during the test in
order to stress the DUT by forcing it to tear down Control order to stress the DUT by forcing it to tear down Control
Plane sessions while handling traffic. It is assumed that Plane sessions while handling traffic. It is assumed that
the test equipment is able to control protocol session the test equipment is able to control protocol session
state with the DUT and is therefore able to introduce state with the DUT and is therefore able to introduce
Controlled Session Loss. Discontinued Sessions.
Measurement units: Measurement units:
None None
Issues: Issues:
None None
See Also: See Also:
Uncontrolled Session Loss Uncontrolled Session Loss
skipping to change at page 7, line 59 skipping to change at page 7, line 59
DUT which the test equipment itself did not DUT which the test equipment itself did not
intentionally bring down. intentionally bring down.
Measurement units: Measurement units:
N/A N/A
Issues: Issues:
None None
See Also: See Also:
Controlled Session Loss Discontinued Sessions
Stress Benchmarking Stress Benchmarking
3.2 Benchmark Planes 3.2 Benchmark Planes
3.2.1 Control Plane 3.2.1 Control Plane
Definition: Definition:
The Description of the control protocols enabled for The Description of the control protocols enabled for
the Accelerated Stress Benchmarking. the Accelerated Stress Benchmarking.
Discussion: Discussion:
skipping to change at page 8, line 43 skipping to change at page 8, line 43
Definition: Definition:
The data traffic profile used for the Accelerated Stress The data traffic profile used for the Accelerated Stress
Benchmarking. Benchmarking.
Discussion: Discussion:
The Data Plane defines the Configuration, Startup The Data Plane defines the Configuration, Startup
Conditions, and Instability Conditions of the data Conditions, and Instability Conditions of the data
traffic. The Data Plane includes the traffic and traffic. The Data Plane includes the traffic and
interface profile. interface profile.
Issues:
None
Measurement Units: Measurement Units:
N/A N/A
See Also: See Also:
Benchmark Planes Benchmark Planes
Data Plane Configuration Set Data Plane Configuration Set
Data Plane Startup Conditions Data Plane Startup Conditions
Data Plane Instability Conditions Data Plane Instability Conditions
3.2.3 Management Plane 3.2.3 Management Plane
skipping to change at page 9, line 44 skipping to change at page 9, line 44
Discussion: Discussion:
The Security Plane defines the Configuration, Startup The Security Plane defines the Configuration, Startup
Conditions, and Instability Conditions of the security Conditions, and Instability Conditions of the security
features and protocols. The Security Plane includes the features and protocols. The Security Plane includes the
ACLs, Firewall, Secure Protocols, and User Login. ACLs, Firewall, Secure Protocols, and User Login.
Measurement units: Measurement units:
N/A N/A
Issues: None Issues:
None
See Also: See Also:
Benchmark Planes Benchmark Planes
Security Plane Configuration Set Security Plane Configuration Set
Security Plane Startup Conditions Security Plane Startup Conditions
Security Plane Instability Conditions Security Plane Instability Conditions
Stress Benchmarking Stress Benchmarking
3.3 Startup 3.3 Startup
3.3.1 Startup Phase 3.3.1 Startup Phase
Definition Definition
The portion of the benchmarking test in which the The step of the benchmarking test in which the
Startup Conditions are generated with the DUT. This Startup Conditions are generated with the DUT. This
begins with the attempt to establish the first session begins with the attempt to establish the first session
and ends when the last Control Plane session is and ends when the last Control Plane session is
established. established.
Discussion: Discussion:
The Startup Phase is the first Phase of the benchmarking The Startup Phase is the first Phase of the benchmarking
test preceding the Instability Phase and Recovery Phase. test preceding the Instability Phase and Recovery Phase.
It is specified by the Configuration Sets and Startup It is specified by the Configuration Sets and Startup
Conditions for each Benchmark Plane. The Startup Phase ends Conditions for each Benchmark Plane. The Startup Phase ends
and Instability Phase MUST begin when the Configuration Sets and Instability Phase MUST begin when the Configuration Sets
are achieved with the DUT. are achieved with the DUT. The DUT MUST be stable and
without failure during the Startup Phase to continue to the
Instability Phase. If there is failure during the Startup
Phase then the test MUST be restarted with new Startup
Conditions.
Measurement Units: Measurement Units:
None None
Issues: Issues:
The 'last control plane session is established' may not None
be a sufficient indicator that steady-state is achieved
and Instability Conditions can be applied to begin the
Instability Phase.
See Also: See Also:
Benchmark Plane Benchmark Plane
Configuration Sets Configuration Sets
Startup Conditions Startup Conditions
Instability Phase Instability Phase
Recovery Phase Recovery Phase
3.3.2 Benchmarks 3.3.2 Benchmarks
3.3.2.1 Stable Aggregate Forwarding Rate 3.3.2.1 Stable Aggregate Forwarding Rate
Definition: Definition:
Average rate of traffic forwarded by the DUT during the Sum of forwarding rates for all interfaces on the
Startup Phase. DUT during the Startup Phase.
Discussion: Discussion:
Stable Aggregate Forwarding Rate is the calculated The Stable Aggregate Forwarding Rate is calculated from
average of the Aggregate Forwarding Rates measured during measurement samples throughout the entire Startup Phase.
the Startup Phase. Stable Aggregate Forwarding Rate is the calculated average
of the samples measured during the Startup Phase. It is
RECOMMENDED that the sample measurements be made on every
DUT interface every 1 second.
Measurement units: Measurement units:
pps pps
Stress Benchmarking Stress Benchmarking
Issues: Issues:
The act of the DUT establishing the Startup Conditions The act of the DUT establishing the Startup Conditions
could influence the forwarding rate in certain could influence the forwarding rate in certain
implementations so that this "baseline" for the implementations so that this "baseline" for the
remainder of the test is lowered. The alternative is remainder of the test is lowered. The alternative is
to change the definition of Stable Aggregate to change the definition of Stable Aggregate
Forwarding Rate so that it is measured during the Startup Forwarding Rate so that it is measured after Startup
Phase, but after Startup Conditions are achieved. Conditions are achieved. The disadvantage of that
The disadvantage of this definition would be that it definition would be that it loses measurement of any
loses measurement of any impact that establishing impact that establishing Startup Conditions would have
Startup Conditions would have on forwarding rate. When on forwarding rate. When comparing the Startup Aggregate
comparing the Startup Aggregate Forwarding Rate benchmark Forwarding Rate benchmark of two devices it is preferred
of two devices it is preferred to know the impact to know the impact establishing Startup Conditions has
establishing Startup Conditions has on Forwarding Rate. on Forwarding Rate.
The definition was therefore selected so that Stable
Aggregate Forwarding Rate is calculated from measurement
samples throughout the entire Startup Phase.
See Also: See Also:
Startup Phase Startup Phase
Aggregate Forwarding Rate Aggregate Forwarding Rate
3.3.2.2 Stable Latency 3.3.2.2 Stable Latency
Definition: Definition:
Average measured latency of traffic forwarded by the DUT Average measured latency of traffic forwarded by the DUT
during the Startup Phase. during the Startup Phase.
skipping to change at page 12, line 21 skipping to change at page 12, line 21
None None
See Also: See Also:
Startup Phase Startup Phase
3.3.3 Control Plane 3.3.3 Control Plane
3.3.3.1 Control Plane Configuration Set 3.3.3.1 Control Plane Configuration Set
Definition: Definition:
The routing protocols and scaling values used for the The control protocols and scaling values used for the
Accelerated Stress Benchmarking. Accelerated Stress Benchmarking.
Discussion: Discussion:
Control Plane Configuration Set is represented in Figure 2 and Control Plane Configuration Set is represented in Figure 2 and
specifies protocol configurations for protocols such as, but not specifies protocol configurations for protocols such as, but not
limitied to, Routing, Multicast, and MPLS. Specific limitied to, Routing, Multicast, SIP and MPLS. Specific
protocols can be enabled or disabled for a benchmark test. protocols can be enabled or disabled for a benchmark test.
Measurement units: Measurement units:
N/A N/A
Issues: Issues:
None None
See Also: See Also:
Data Plane Configuration Set Data Plane Configuration Set
skipping to change at page 12, line 51 skipping to change at page 12, line 51
____________ ____________ ____________ ____________ ____________ ____________
| Routing | | Multicast | | MPLS | | Routing | | Multicast | | MPLS |
| Protocols |___ | Protocols | __| Protocols | | Protocols |___ | Protocols | __| Protocols |
| | | | | | | | | | | | | | | |
------------ | ------------ | ------------ ------------ | ------------ | ------------
| | | | | |
| | | | | |
| \/ | | \/ |
| ___________ | | ___________ |
| | | | | | | |
|------->| DUT |<------| ____________ +------->| DUT |<------+
| | | SIP |----------->| |
----------- | Sessions | -----------
------------
Figure 2. Control Plane Configuration Module Figure 2. Control Plane Configuration Module
Stress Benchmarking Stress Benchmarking
3.3.3.2 Control Plane Startup Conditions 3.3.3.2 Control Plane Startup Conditions
Definition: Definition:
Control Plane conditions that occur at the start Control Plane conditions that occur at the start
of the Accelerated Stress Benchmarking to establish conditions of the Accelerated Stress Benchmarking to establish conditions
for the remainder of the test. for the remainder of the test.
skipping to change at page 14, line 23 skipping to change at page 14, line 23
packet rate, number of flows, and encapsulation on a packet rate, number of flows, and encapsulation on a
per-interface basis used for the offered load to the DUT. per-interface basis used for the offered load to the DUT.
Measurement Units: Measurement Units:
Traffic Profile is reported as follows: Traffic Profile is reported as follows:
Parameter Units Parameter Units
--------- ------ --------- ------
Packet Size(s) bytes Packet Size(s) bytes
Packet Rate(interface) array of packets per second Packet Rate(interface) array of packets per second
Aggregate Offered Load pps
Number of Flows number of flows Number of Flows number of flows
Traffic Type array of (RTP, UDP, TCP, other)
Encapsulation(flow) array of encapsulation type Encapsulation(flow) array of encapsulation type
Mirroring enabled/disabled Mirroring enabled/disabled
Issues: Issues:
None None
See Also: See Also:
Data Plane Configuration Set Data Plane Configuration Set
3.3.5 Management Plane 3.3.5 Management Plane
skipping to change at page 17, line 11 skipping to change at page 17, line 11
Data Plane Startup Conditions Data Plane Startup Conditions
Management Plane Startup Conditions Management Plane Startup Conditions
Security Plane Startup Conditions Security Plane Startup Conditions
Stress Benchmarking Stress Benchmarking
3.4 Instability 3.4 Instability
3.4.1 Instability Phase 3.4.1 Instability Phase
Definition: Definition:
The portion of the benchmarking test in which the The step of the benchmarking test in which the Instability
Instability Conditions are offered to the DUT. Conditions are offered to the DUT.
Discussion: Discussion:
The Instability Phase is the middle Phase of The Instability Phase is the middle phase of the benchmarking
of the benchmarking test following the Startup test following the Startup Phase and preceding the Recovery
Phase and preceding the Recovery Phase. Phase. The Startup Phase MUST complete without failure to
begin the Instability Phase.
Measurement Units: Measurement Units:
None None
Issues: Issues:
None None
See Also: See Also:
Instability Conditions Instability Conditions
Startup Phase Startup Phase
skipping to change at page 19, line 53 skipping to change at page 19, line 53
DUT which the test equipment itself did not DUT which the test equipment itself did not
intentionally bring down. intentionally bring down.
Measurement units: Measurement units:
sessions sessions
Issues: Issues:
None None
See Also: See Also:
Controlled Session Loss Discontinued Sessions
Uncontrolled Session Loss Uncontrolled Session Loss
Stress Benchmarking Stress Benchmarking
3.4.3 Instability Conditions 3.4.3 Instability Conditions
3.4.3.1 Control Plane Instability Conditions 3.4.3.1 Control Plane Instability Conditions
Definition: Definition:
Control Plane conditions that occur during the Accelerated Control Plane conditions that occur during the Accelerated
Stress Benchmark to produce instability and stress the DUT. Stress Benchmark to produce instability and stress the DUT.
skipping to change at page 22, line 10 skipping to change at page 22, line 10
Instability Conditions Instability Conditions
Control Plane Instability Conditions Control Plane Instability Conditions
Data Plane Instability Conditions Data Plane Instability Conditions
Management Plane Instability Conditions Management Plane Instability Conditions
Stress Benchmarking Stress Benchmarking
3.5 Recovery 3.5 Recovery
3.5.1 Recovery Phase 3.5.1 Recovery Phase
Definition: Definition:
The portion of the benchmarking test in which the The step of the benchmarking test in which the
Startup Conditions are generated with the DUT, but Startup Conditions are generated with the DUT, but
the Instability Conditions are no longer offered to the Instability Conditions are no longer offered to
the DUT. the DUT.
Discussion: Discussion:
The Recovery Phase is the final Phase of the The Recovery Phase is the final Phase of the
benchmarking test following the Startup Phase and benchmarking test following the Startup Phase and
Instability Phase. Startup Conditions MUST NOT be Instability Phase. Startup Conditions MUST NOT be
Restarted. Restarted.
skipping to change at page 24, line 27 skipping to change at page 24, line 27
DUT which the test equipment itself did not DUT which the test equipment itself did not
intentionally bring down. intentionally bring down.
Measurement units: Measurement units:
sessions sessions
Issues: Issues:
None None
See Also: See Also:
Controlled Session Loss Discontinued Sessions
Uncontrolled Session Loss Uncontrolled Session Loss
3.5.2.5 Variability Benchmarks 3.5.2.5 Variability Benchmarks
Definition: Definition:
The difference between the measured Benchmarks of the The difference between the measured Benchmarks of the
same DUT over multiple iterations. same DUT over multiple iterations.
Discussion: Discussion:
Ideally, the measured benchmarks should be the same for multiple Ideally, the measured benchmarks should be the same for multiple
skipping to change at page 25, line 33 skipping to change at page 25, line 33
Interconnection Devices", RFC 1242, March 1991. Interconnection Devices", RFC 1242, March 1991.
[2] Mandeville, R., "Benchmarking Terminology for LAN Switching [2] Mandeville, R., "Benchmarking Terminology for LAN Switching
Devices", RFC 2285, June 1998. Devices", RFC 2285, June 1998.
[3] Bradner, S. and McQuaid, J., "Benchmarking Methodology for [3] Bradner, S. and McQuaid, J., "Benchmarking Methodology for
Network Interconnect Devices", RFC 2544, March 1999. Network Interconnect Devices", RFC 2544, March 1999.
[4] Poretsky, S. and Rao, S., "Methodology Guidelines for [4] Poretsky, S. and Rao, S., "Methodology Guidelines for
Accelerated Stress Benchmarking", Accelerated Stress Benchmarking",
draft-ietf-bmwg-acc-bench-meth-07, work in progress, draft-ietf-bmwg-acc-bench-meth-09, work in progress,
March 2007. February 2008.
[5] Bradner, S., "Key words for use in RFCs to Indicate [5] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", RFC 2119, March 1997. Requirement Levels", RFC 2119, March 1997.
7.2 Informative References 7.2 Informative References
[RFC3871] Jones, G., "Operational Security Requirements for Large [RFC3871] Jones, G., "Operational Security Requirements for Large
Internet Service Provider (ISP) IP Network Infrastructure.", Internet Service Provider (ISP) IP Network Infrastructure.",
IETF RFC 3871 , September 2004. IETF RFC 3871 , September 2004.
[NANOG25] Poretsky, S., "Core Router Evaluation for Higher [NANOG25] Poretsky, S., "Core Router Evaluation for Higher
skipping to change at page 26, line 8 skipping to change at page 26, line 8
[NANOG25] Poretsky, S., "Core Router Evaluation for Higher [NANOG25] Poretsky, S., "Core Router Evaluation for Higher
Availability", NANOG 25, June 8, 2002, Toronto, CA. Availability", NANOG 25, June 8, 2002, Toronto, CA.
[IEEECQR] Poretsky, S., "Router Stress Testing to Validate [IEEECQR] Poretsky, S., "Router Stress Testing to Validate
Readiness for Network Deployment", IEEE CQR 2003. Readiness for Network Deployment", IEEE CQR 2003.
Stress Benchmarking Stress Benchmarking
8. Author's Address 8. Author's Address
Scott Poretsky Scott Poretsky
Reef Point Systems NextPoint Networks
8 New England Executive Park 3 Federal Street
Burlington, MA 01803 USA Billerica, MA 01821
Phone: + 1 781 395 5090 USA
EMail: sporetsky@reefpoint.com Phone: + 1 508 439 9008
EMail: sporetsky@nextpointnetworks.com
Shankar Rao Shankar Rao
1801 California Street 1801 California Street
8th Floor 8th Floor
Qwest Communications Qwest Communications
Denver, CO 80202 USA Denver, CO 80202 USA
Phone: + 1 303 437 6643 Phone: + 1 303 437 6643
Email: shankar.rao@qwest.com Email: shankar.rao@qwest.com
Appendix 1. White Box Benchmarking Terminology Appendix 1. White Box Benchmarking Terminology
skipping to change at page 27, line 8 skipping to change at page 27, line 8
Measurement units: % Measurement units: %
Issues: None Issues: None
See Also: See Also:
Minimum Available Memory Minimum Available Memory
Stress Benchmarking Stress Benchmarking
Full Copyright Statement Full Copyright Statement
Copyright (C) The IETF Trust (2007). Copyright (C) The IETF Trust (2008).
This document is subject to the rights, licenses and restrictions This document is subject to the rights, licenses and restrictions
contained in BCP 78, and except as set forth therein, the authors contained in BCP 78, and except as set forth therein, the authors
retain all their rights. retain all their rights.
This document and the information contained herein are provided This document and the information contained herein are provided
on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE
REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE
IETF TRUST AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL IETF TRUST AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL
WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY
 End of changes. 36 change blocks. 
64 lines changed or deleted 76 lines changed or added

This html diff was produced by rfcdiff 1.34. The latest version is available from http://tools.ietf.org/tools/rfcdiff/