Network Working Group
   INTERNET-DRAFT
   Expires in: April October 2005
                                       		Scott Poretsky
                                                Quarry Technologies

						Shankar Rao
						Qwest Communications

                                                October 2004

                                                February 2005

             Terminology for Accelerated Stress Benchmarking
              <draft-ietf-bmwg-acc-bench-term-04.txt>
              <draft-ietf-bmwg-acc-bench-term-05.txt>

Intellectual Property Rights (IPR) statement:
By submitting this Internet-Draft, I certify that any applicable
patent or other IPR claims of which I am aware have been disclosed, or
will be disclosed, and any of which I become aware will be disclosed,
in accordance with RFC 3668.

   Status of this Memo

   This document is an Internet-Draft and is in full conformance with
   all provisions of Section 10 of RFC2026.
   Internet-Drafts are working documents of the Internet Engineering
   Task Force  (IETF), its areas, and its working groups.  Note that
   other groups may also distribute working documents as Internet-
   Drafts.

   Internet-Drafts are draft documents valid for a maximum of six
   months and may be updated, replaced, or obsoleted by other
   documents at any time.  It is inappropriate to use Internet-Drafts
   as reference material or to cite them other than as "work in
   progress."

   The list of current Internet-Drafts can be accessed at
   http://www.ietf.org/ietf/1id-abstracts.txt
   The list of Internet-Draft Shadow Directories can be accessed at
   http://www.ietf.org/shadow.html.

   ABSTRACT
   This document provides the Terminology for performing Stress
   Benchmarking of networking devices.  The three phases of the Stress
   Test: Startup, Instability and Recovery are defined along with the
   benchmarks and configuration terms associated with the each phase.
   Also defined are the Benchmark Planes fundamental to stress testing
   configuration, setup and measurement.  The terminology is to be
   used with the companion framework and methodology documents.

   Table of Contents
     1. Introduction ............................................... 3
     2. Existing definitions ....................................... 3
     3. Term definitions............................................ 3 4
      3.1 General Terms............................................. 3 4
	3.1.1 Benchmark Planes...................................... 3 4
        3.1.2 Configuration Sets.................................... 4 5
        3.1.3 Startup Conditions.................................... 4 5
        3.1.4 Instability Conditions................................ 5 6
        3.1.5 Aggregate Forwarding Rate............................. 6
   	3.1.6 Controlled Session Loss............................... 6 7
   	3.1.7 Uncontrolled Session Loss............................. 6 7
      3.2 Benchmark Planes.......................................... 7 8
        3.2.1 Control Plane......................................... 7 8
        3.2.2 Data Plane............................................ 7 8
        3.2.3 Management Plane...................................... 8
        3.2.4 Security Plane........................................ 8
      3.3 Startup................................................... 9
      3.3 Startup...................................................10
        3.3.1 Startup Phase......................................... 9 Phase.........................................10
	3.3.2 Benchmarks............................................10
          3.3.2.1 Stable Aggregate Forwarding Rate..................10
          3.3.2.2 Stable Latency....................................10 Latency....................................11
          3.3.2.3 Stable Session Count..............................11
        3.3.3 Control Plane.........................................11 Plane.........................................12
          3.3.3.1 Control Plane Configuration Set...................11 Set...................12
          3.3.3.2 Control Plane Startup Conditions..................12 Conditions..................13
   	3.3.4 Data Plane............................................12 Plane............................................13
          3.3.4.1 Data Plane Configuration Set......................12 Set......................13
  	  3.3.4.2 Traffic Profile...................................13
  	3.3.5 Management Plane......................................13 Plane......................................14
  	  3.3.5.1 Management Plane Configuration Set................13 Set................14
        3.3.6 Security Plane........................................14 Plane........................................15
          3.3.6.1 Security Plane Configuration Set..................14 Set..................15
   	  3.3.6.2 Security Plane Startup Conditions.................15 Conditions.................16
      3.4 Instability...............................................15 Instability...............................................16
        3.4.1 Instability Phase.....................................15 Phase.....................................16
        3.4.2 Benchmarks............................................16 Benchmarks............................................17
          3.4.2.1 Unstable Aggregate Forwarding Rate................16 Rate................17
          3.4.2.2 Degraded Aggregate Forwarding Rate................17
          3.4.2.3 Average Degraded Aggregate Forwarding Rate........17 Rate........18
          3.4.2.4 Unstable Latency..................................17 Latency..................................18
   	  3.4.2.5 Unstable Uncontrolled Sessions Lost...............18 Lost...............19
        3.4.3 Instability Conditions................................18 Conditions................................19
          3.4.3.1 Control Plane Instability Conditions..............18 Conditions..............19
   	  3.4.3.2 Data Plane Instability Conditions.................19 Conditions.................20
          3.4.3.3 Management Plane Instability Conditions...........19 Conditions...........20
          3.4.3.4 Security Plane Instability Conditions.............20
      3.5 Recovery..................................................20 Recovery..................................................21
        3.5.1 Recovery Phase........................................20 Phase........................................21
        3.5.2 Benchmarks............................................21
          3.5.2.1 Recovered Aggregate Forwarding Rate...............21
          3.5.2.2 Recovered Latency.................................21 Latency.................................22
          3.5.2.3 Recovery Time.....................................22
          3.5.2.4 Recovered Uncontrolled Sessions Lost..............22 Lost..............23
          3.5.2.5 Variability Benchmarks............................23
     4. Security Considerations.....................................23 Considerations.....................................24
     5. References..................................................23 Normative References........................................24
     6. Informative References......................................24
     7. Author's Address............................................24 Address............................................25
     Appendix 1 - White Box Benchmarks..............................24 Benchmarks..............................25

 1. Introduction

   Routers in an operational network are simultaneously configured with
   multiple protocols and security policies while forwarding traffic and
   being managed.  To accurately benchmark a router for deployment it is
   necessary to test that router in operational conditions by
   simultaneously configuring and scaling network protocols and security
   policies, forwarding traffic, and managing the device.  It is helpful
   to accelerate these network operational conditions so that the
   router under test can be benchmarked with faster test duration.
   Testing a router in accelerated network conditions is known as
   Accelerated Stress Testing.

   This document provides the Terminology for performing Stress
   Benchmarking of networking devices.  The three phases of the Stress
   Test: Startup, Instability and Recovery are defined along with the
   benchmark and configuration terms associated with the each phase.
   Benchmarks for stress testing are defined using the Aggregate
   Forwarding Rate and control plane Session Count during each phase
   of the test.  Also defined are the Benchmark Planes fundamental to
   stress testing configuration, setup and measurement.  These are
   the Control Plane, Data Plane, Management Plane and Security Plane
   For each plane, the Configuration Set, Startup Conditions, and
   Instability Conditions are defined.  White Box benchmarks are
   provided in Appendix 1 for additional DUT behavior measurements.
   The terminology is to be used with the companion methodology
   document [6]. [4].  The sequence of phases, actions, and benchmarks
   are shown in Table 1.

   2.  Existing definitions

   RFC 1242 "Benchmarking Terminology for Network Interconnect
   Devices" and RFC 2285 "Benchmarking Terminology for LAN Switching
   Devices" should be consulted before attempting to make use of this
   document. For the sake of clarity and continuity this RFC adopts
   the template for definitions set out in Section 2 of RFC 1242.
   Definitions are indexed and grouped together in sections for ease
   of reference.

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in BCP 14, RFC 2119
   [Br97].  RFC 2119. 2119 defines the use of these key words to help make the
   intent of standards track documents as clear as possible.  While this
   document uses these keywords, this document is not a standards track
   document.

   Table 1. Phase Sequence and Benchmarks
   III. Recovery Phase   II. Instability Phase    I. Startup Phase
   <-----------------<---<-------------------<----<--------------<
    Remove Instability   Achieve Configuration    Apply Startup
    Conditions           Set                      Conditions

    Benchmark:           Benchmark:               Benchmark:
    Recovered Aggregate  Unstable Aggregate       Stable Aggregate
    Forwarding Rate      Forwarding Rate          Forwarding Rate

                         Degraded Aggregate
                         Forwarding Rate

                         Average Degraded
                         Forwarding Rate

    Recovered Latency    Unstable Latency         Startup Latency

    Recovered Uncontrolled Recovered Uncontrolled Stable Session Count
    Sessions Lost          Sessions Lost

    Recovery Time

   3. Term definitions
   3.1 General Terms
   3.1.1 Benchmark Planes

      Definition:
	The features, conditions, and behavior for the Accelerated Stress
	Benchmarking.

      Discussion:
	There are four Benchmark Planes: Control Plane, Data Plane,
	Management Plane, and Security Plane as shown in Figure 1. The
	Benchmark Planes define the Configuration, Startup Conditions,
	Instability Conditions, and Failure Conditions used for the test.

      Measurement units:
	N/A

      Issues:
	None

      See Also:
	Control Plane
 	Data Plane
 	Management Plane
	Security Plane
	 ___________		 ___________
	|  Control  |		| Management|
	|   Plane   |___     ___|   Plane   |
	|   	    |   |   |	|   	    |
	 -----------    |   |	 -----------
		       \/  \/		       ___________
		      ___________	      | Security  |
		     |	     	 |<-----------|   Plane	  |
		     |    DUT    |	      |           |
		|--->|	     	 |<---|        -----------
	 	|     -----------     |
		|		      |
		|     ___________     |
		|    |   Data    |    |
		|--->|   Plane   |<---|
		     |   	 |
	 	      -----------

	Figure 1.  Router Accelerated Stress Benchmarking Planes

      Measurement units:
	N/A

      Issues:
	None

      See Also:
	Control Plane
 	Data Plane
 	Management Plane
	Security Plane

    3.1.2 Configuration Sets

      Definition:
	The features and scaling limits used during the Accelerated Stress
	Benchmarking.

      Discussion:
	There are four Configuration Sets: Control Plane Configuration Set,
	Data Plane Configuration Set, Management Plane Configuration Set,
	and Security Plane Configuration Set.

      Measurement units:
	N/A

      Issues:
	None

      See Also:
	Control Plane Configuration Set
 	Data Plane Configuration Set
 	Management Plane Configuration Set
	Security Plane Configuration Set

   3.1.3 Startup Conditions

      Definition:
	Test conditions that occur at the start of the Accelerated
	Life Benchmark to establish conditions for the remainder of
	the test.

      Discussion:
	Startup Conditions may cause stress on the DUT and produce
	failure.  Startup Conditions are defined for the Control
	Plane and Security Plane.

      Measurement units:
	N/A

      Issues:
	None

      See Also:
	Control Plane Startup Conditions
	Data Plane Startup Conditions
	Management Plane Startup Conditions
	Security Plane Startup Conditions

   3.1.4 Instability Conditions

      Definition:
	Test conditions that occur during the Accelerated Stress
	Benchmark to produce instability and stress the DUT.

      Discussion:
	Instability Conditions are applied to the DUT after the
	Startup Conditions have completed.  Instability Conditions
	occur for the Control Plane, Data Plane, Management Plane,
	and Security Plane.

      Measurement units:
      N/A

      Issues:  None

     See Also:
   	Control Plane Instability Conditions
   	Data Plane Instability Conditions
 	Management Plane Instability Conditions
	Security Plane Instability Conditions

   3.1.5 Aggregate Forwarding Rate

     Definition:
	Sum of forwarding rates for all interfaces on the
	DUT during the Startup Phase.

      Discussion:
	Each interface of the DUT forwards traffic at some
	measured rate.  The Aggregate Forwarding Rate is the
	sum of forwarding rates for all interfaces on the DUT.

      Measurement units:
	pps

      Issues:
	None

      See Also:
   	Startup Phase

   3.1.6 Controlled Session Loss

     Definition:
	Control Plane sessions that are intentionally brought
	down during the Stress test.

      Discussion:
	The test equipment is able to control protocol
	session state with the DUT.

      Measurement units:
	None

      Issues:
	None

      See Also:
	Uncontrolled Session Loss

   3.1.7 Uncontrolled Session Loss

     Definition:
	Control Plane sessions that are in the down state
	but were not intentionally brought down during the
	Stress test.

      Discussion:
	The test equipment is able to control protocol
	session state with the DUT.  The test equipment
	is also to monitor for sessions lost with the
	DUT which the test equipment itself did not
	intentionally bring down.

      Measurement units:
	N/A

      Issues:
	None

      See Also:
	Controlled Session Loss
   3.2 Benchmark Planes

   3.2.1 Control Plane
      Definition:
	The Description of the control protocols enabled for
	the Accelerated Stress Benchmarking.

      Discussion:
	The Control Plane defines the Configuration, Startup
	Conditions, and Instability Conditions of the control
        protocols.  Control Plane protocols may include routing
        protocols, multicast protocols, and MPLS protocols.
        These can be enabled or disabled for a benchmark test.

      Measurement units:
	N/A

      Issues:
	None

      See Also:
	Benchmark Planes
	Control Plane Configuration Set
	Control Plane Startup Conditions
	Control Plane Instability Conditions

  3.2.2 Data Plane
      Definition:
	The data traffic profile used for the Accelerated Stress
	Benchmarking.

      Discussion:
	The Data Plane defines the Configuration, Startup
	Conditions, and Instability Conditions of the data
        traffic.  The Data Plane includes the traffic and
        interface profile.

      Measurement Units:
        N/A

      See Also:
	Benchmark Planes
	Data Plane Configuration Set
	Data Plane Startup Conditions
	Data Plane Instability Conditions

   3.2.3 Management Plane

      Definition:
  	The Management features and tools used for the
	Accelerated Stress Benchmarking.

      Discussion:
	A key component of the Accelerated Stress Benchmarking is the
	Management Plane to assess manageability of the router
	under stress.  The Management Plane defines the Configuration,
	Startup Conditions, and Instability Conditions of the
        management protocols and features.  The Management Plane
        includes SNMP, Logging/Debug, Statistics Collection, and
        management configuration sessions such as telnet, SSH, and
        serial console.  SNMP Gets SHOULD be performed continuously.
        Management configuration sessions should be open
        simultaneously and be repeatedly open and closed.  Open
        management sessions should have valid and invalid
        configuration and show commands entered.

      Measurement units:
        N/A

      Issues:
	None

      See Also:
	Benchmark Planes
	Management Plane Configuration Set
	Management Plane Startup Conditions
	Management Plane Instability Conditions

   3.2.4 Security Plane

      Definition:
	The Security features used during the Accelerated Stress
	Benchmarking.

      Discussion:
        The Security Plane defines the Configuration, Startup
        Conditions, and Instability Conditions of the security
        features and protocols.  The Security Plane includes the
        ACLs, Firewall, Secure Protocols, and User Login.  Tunnels
        for those such as IPsec should be established and flapped.
        Policies for Firewalls and ACLs should be repeatedly added
        and removed from the configuration via telnet, SSH, or
        serial management sessions.

      Measurement units:
        N/A

      Issues: None

      See Also:
	Benchmark Planes
	Security Plane Configuration Set
	Security Plane Startup Conditions
	Security Plane Instability Conditions
   3.3 Startup

   3.3.1 Startup Phase

     Definition
	The portion of the benchmarking test in which the
	Startup Conditions are generated with the DUT.  This
	begins with the attempt to establish the first session
	and ends when the last Control Plane session is
	established.

     Discussion:
        The Startup Phase is the first Phase of the benchmarking
        test preceding the Instability Phase and Recovery Phase.
        It is specified by the Configuration Sets and Startup
        Conditions for each Benchmark Plane.  The Startup Phase ends
        and Instability Phase may begin when the Configuration Sets
        are achieved with the DUT.

     Measurement Units:
	None

     Issues:
	The 'last control plane session is established' may not
	be a sufficient indicator that steady-state is achieved
	and Instability Conditions can be applied to begin the
	Instability Phase.

     See Also:
	Benchmark Plane
	Configuration Sets
	Startup Conditions
	Instability Phase
	Recovery Phase

   3.3.2 Benchmarks
   3.3.2.1 Stable Aggregate Forwarding Rate

     Definition:
	Average rate of traffic forwarded by the DUT during the
	Startup Phase.

      Discussion:
	Stable Aggregate Forwarding Rate is the calculated
	average of the Aggregate Forwarding Rates measured during
	the Startup Phase.

      Measurement units:
	pps
      Issues:
	The act of the DUT establishing the Startup Conditions
	could influence the forwarding rate in certain
	implementations so that this "baseline" for the
	remainder of the test is lowered.  The alternative is
	to change the definition of Stable Aggregate
	Forwarding Rate so that it measured during the Startup
	Phase, but after Startup Conditions are achieved.
	The disadvantage of this definition would be that it
	loses measurement of any impact that establishing
        Startup  Conditions would have on forwarding rate. When
	comparing the Startup Aggregate Forwarding Rate benchmark
	of two devices it is preferred to know the impact
	establishing Startup Conditions has on Forwarding Rate.
	The definition was therefore selected so that Stable
	Aggregate Forwarding Rate is calculated from measurement
	samples throughout the entire Startup Phase.

      See Also:
        Startup Phase
        Aggregate Forwarding Rate

   3.3.2.2 Stable Latency

      Definition:
        Average measured latency of traffic forwarded by the DUT
        during the Startup Phase.

      Discussion:
        Stable Latency is the calculated average Latency during
	the Startup Phase.

      Measurement units:
	seconds

      Issues:
	None

      See Also:
        Startup Phase
	Stable Aggregate Forwarding Rate

   3.3.2.3 Stable Session Count

     Definition:
        Total number of control plane sessions/adjacencies
        established and maintained by the DUT during the Startup
	Phase and prior to Instability Conditions being initiated.

      Discussion:
	This measurement should be made after the Control
	Plane Startup Conditions are applied to the DUT.

      Measurement units:
	sessions

      Issues:
	None

      See Also:
        Startup Phase

   3.3.3 Control Plane

   3.3.3.1 Control Plane Configuration Set
      Definition:
	The routing protocols and scaling values used for the Accelerated
	Life Benchmarking.

      Discussion:
	Control Plane Configuration Set is shown in Figure 2 and specifies
	the Routing Protocols, Multicast, and MPLS configuration.  Specific
	protocols can be enabled or disabled for a benchmark test.

      Measurement units:
	N/A

      Issues:
	None

      See Also:
	Data Plane Configuration Set
      Management Configuration Set
      Security Configuration Set

	 ____________ 		 ____________		 ____________
	|  Routing   |	 	|  Multicast |		|    MPLS    |
	|  Protocols |___	|  Protocols | 	      __|  Protocols |
	| 	     |   |  	| 	     |        |	| 	     |
	 ------------    |   	 ------------         |	 ------------
			 |		|	      |
			 |		|	      |
			 |  	       \/	      |
	    		 |	   ___________	      |
			 |	  |	      |	      |
			 |------->|    DUT    |<------|
				``|	      |
	 		 	   -----------
	Figure 2.  Control Plane Configuration Module
   3.3.3.2 Control Plane Startup Conditions

      Definition:
	Control Plane conditions that occur at the start
	of the Accelerated Stress Benchmarking to establish conditions
	for the remainder of the test.

      Discussion:
	Startup Conditions may cause stress on the DUT and produce
	failure.  Startup Conditions for the Control Plane include
	session establishment rate, number of sessions established
	and number of routes learned.

      Measurement units:
	N/A

      Issues:
	None

      See Also:
	Startup Conditions
	Security Plane Startup Conditions
	Control Plane Configuration Set

   3.3.4 Data Plane
   3.3.4.1 Data Plane Configuration Set

      Definition:
	The data traffic profile enabled for the Accelerated Stress
	Benchmarking.

      Discussion:
	Data Plane Configuration Set includes the Traffic Profile and
	interfaces used for the Accelerated Stress Benchmarking.

      Measurement Units:
        N/A

      Issues: None

      See Also:
	Traffic Profile

 3.3.4.2 Traffic Profile
       Definition
	The characteristics of the Offered Load to the DUT used for
	the Accelerated Stress Benchmarking.

      Discussion
	The Traffic Profile specifies the number of packet size(s), packet
	rate per interface, number of flows, and encapsulation used for the
 	offered load to the DUT.

      Measurement Units:
      Traffic Profile is reported as follows:

	Parameter			Units
	---------			------
	Packet Size(s)			bytes
	Packet Rate(interface)		array of packets per second
	Number of Flows			number
	Encapsulation(flow)		array of encapsulation type

      Issues:
	None

      See Also:
	Data Plane Configuration Set

  3.3.5 Management Plane
  3.3.5.1 Management Plane Configuration Set

      Definition:
  	The router management features enabled for the
	Accelerated Stress Test.

      Discussion:
        A key component of the Accelerated Stress Test is the Management
        Configuration Set to assess manageability of the router under
        stress.  The Management Configuration Set defines the management
        configuration of the DUT.  Features that are part of the
        Management Configuration Set include Telnet access, SNMP,
        Logging/Debug, and Statistics Collection, and services such as
        FTP, as shown in Figure 3. These features should be enabled
        throughout the Stress test. SNMP Gets should be made continuously
        with multiple FTP and Telnet sessions operating simultaneously.
        FTP sessions should be opened and closed at varying intervals
        and get and put files while open.  Telnet sessions should be
        opened and closed at varying intervals and enter valid and invalid
        show and configuration commands while open.

      Measurement units:
      N/A

      Issues:
	None

      See Also:
	Control Plane Configuration Set
	Data Plane Configuration Set
      Security Plane Configuration Set
   	 	 	  ____________	      ____________
			 |  	      |	     |  Logging/  |
			 |    SNMP    |    __|   Debug    |
		   	 | 	      |   |  |      	  |
	 	          ------------    |   ------------
	      		 	|	  |
	      		 	|	  |
	       	       		\/	  |
	      	  	  ___________	  |
	      		 |	     | 	  |
	      	 	 |    DUT    |<---|
		         |	     |
	 		  -----------
			      |
			      |
			      \/
			___________
		       |   Packet  |
		       | Statistics|
		       | Collector |
		       | 	   |
	 	        -----------

		Figure 3.  Management Plane Configuration Set

   3.3.6 Security Plane
   3.3.6.1 Security Plane Configuration Set

      Definition:
	Security features and scaling enabled for the Accelerated Stress
	Test.

      Discussion:
        The Security Plane Configuration Set includes the configuration
        and scaling of ACLs, Firewall, IPsec, and User Access, as shown
        in Figure 4.  Tunnels should be established and policies
        configured.  Instability is introduced by flapping tunnels and
        configuring and removing policies.

	 ____________ 	   ____________	        ____________
	|            |	  |   Secure   |       |    User    |
	|ACL/Firewall|    |  Protocol  |     __|   Access   |
	| 	     |    | 	       |    |  |            |
	 ------------      ------------     |	------------
	     |			|	    |
	     |			|	    |
	     |  	        \/	    |
	     |	       ___________	    |
	     |	      |	    	  | 	    |
	     |------->|    DUT    |<--------|
		      |	          |
	 	       -----------
		Figure 4.  Security Configuration Module
      Measurement units:
        N/A

      Issues:
	None

      See Also:
  	ACL Configuration Set
  	Secure Protocol Configuration Set
  	Password Login Configuration Set

   3.3.6.2 Security Plane Startup Conditions
      Definition:
	Security Plane conditions that occur at the start
	of the Accelerated Stress Benchmarking to establish conditions
	for the remainder of the test.

      Discussion:
	Startup Conditions may cause stress on the DUT and produce
	failure.  Startup Conditions for the Security Plane include
	session establishment rate, number of sessions established
	and number of policies learned, and number of user access
	sessions opened.

      Measurement units:
	N/A

      Issues:
	None

      See Also:
	Startup Conditions
	Data Plane Startup Conditions
	Management Plane Startup Conditions
	Security Plane Startup Conditions

   3.4 Instability

   3.4.1 Instability Phase

     Definition:
	The portion of the benchmarking test in which the
	Instability Conditions are offered to the DUT.

     Discussion:
	The Instability Phase is the middle Phase of
	of the benchmarking test following the Startup
	Phase and preceding the Recovery Phase.

     Measurement Units:
	None
    Issues:
	None

     See Also:
	Instability Conditions
	Startup Phase
	Recovery Phase

   3.4.2 Benchmarks
   3.4.2.1 Unstable Aggregate Forwarding Rate

     Definition:
	Rate of traffic forwarded by the DUT during the
	Instability Phase.

      Discussion:
	Unstable Aggregated Forwarding Rate is an instantaneous
	measurement of the Aggregate Forwarding Rate during the
	Instability Phase.

      Measurement units:
	pps

      Issues:
	None

      See Also:
   	Instability Conditions
	Aggregate Forwarding Rate

   3.4.2.2 Degraded Aggregate Forwarding Rate

      Definition:
	The reduction in Aggregate Forwarding Rate during the
	Instability Phase.

      Discussion:
	The Degraded Aggregate Forwarding Rate is calculated
	for each measurement of the Unstable Aggregate
	Forwarding Rate.  The Degraded Aggregate Forwarding
	Rate is calculated by subtracting each measurement
	of the Unstable Aggregate Forwarding Rate from the
	Stable Aggregate Forwarding Rate, such that

	Degraded Forwarding Rate =
	Stable Aggregate Forwarding Rate -
		Unstable Aggregate Forwarding Rate

	Ideally, the Degraded Aggregate Forwarding Rate is zero.

      Measurement Units:
	pps
      Issues:
	None

      See Also:
	Instability Phase
	Unstable Aggregate Forwarding Rate

   3.4.2.3 Average Degraded Aggregate Forwarding Rate

      Definition
	DUT Benchmark that is the calculated average of the
	obtained Degraded Forwarding Rates.

      Discussion:

      Measurement Units:
	pps

      Issues:
	None

      See Also:
	Degraded Aggregate Forwarding Rate

   3.4.2.4 Unstable Latency

      Definition:
        The average increase in measured packet latency during
        the Instability Phase compared to the Startup Phase.

      Discussion:
        Latency SHOULD be measured at a fixed interval during the
        Instability Phase.  Unstable Latency is the difference
        between Stable Latency and the average Latency measured
        during the Instability Phase.  It is expected that there
        be an increase in average latency from the Startup Phase
        to the Instability phase, but it is possible that the
        difference be zero.  The Unstable Latency cannot be a
        negative number.

      Measurement units:
	seconds

      Issues:
	None

      See Also:
        Instability Phase
	Stable Latency
   3.4.2.5 Unstable Uncontrolled Sessions Lost

     Definition:
	Control Plane sessions that are in the down state
	but were not intentionally brought down during the
	Instability Phase.

      Discussion:
	The test equipment is able to control protocol
	session state with the DUT.  The test equipment
	is also to monitor for sessions lost with the
	DUT which the test equipment itself did not
	intentionally bring down.

      Measurement units:
	sessions

      Issues:
	None

      See Also:
	Controlled Session Loss
	Uncontrolled Session Loss

   3.4.3 Instability Conditions

   3.4.3.1 Control Plane Instability Conditions

      Definition:
	Control Plane conditions that occur during the Accelerated Stress
	Benchmark to produce instability and stress the DUT.

      Discussion:
	Control Plane Instability Conditions are experienced by the DUT
	after the Startup Conditions have completed.  Control Plane
	Instability Conditions experienced by the DUT include session
	loss, route withdrawal, and route cost changes.

      Measurement units:
	N/A

      Issues:
	None

      See Also:
   	Instability Conditions
   	Data Plane Instability Conditions
 	Management Plane Instability Conditions
	Security Plane Instability Conditions
   3.4.3.2 Data Plane Instability Conditions
     Definition:
	Data Plane conditions that occur during the Accelerated Stress
	Benchmark to produce instability and stress the DUT.

      Discussion:
	Data Plane Instability Conditions are experienced by the DUT
	after the Startup Conditions have completed.  Data Plane
	Instability Conditions experienced by the DUT include interface
	shutdown, link loss, and overloaded links.

      Measurement units:
	N/A

      Issues:
	None

      See Also:
   	Instability Conditions
   	Control Plane Instability Conditions
 	Management Plane Instability Conditions
	Security Plane Instability Conditions

   3.4.3.3 Management Plane Instability Conditions
      Definition:
	Management Plane conditions that occur during the Accelerated
	Life Benchmark to produce instability and stress the DUT.

      Discussion:
	Management Plane Instability Conditions are experienced by the DUT
	after the Startup Conditions have completed.  Management Plane
	Instability Conditions experienced by the DUT include repeated
	FTP of large files.

      Measurement units:
	N/A

      Issues:
	None

      See Also:
   	Instability Conditions
 	Control Plane Instability Conditions
   	Data Plane Instability Conditions
	Security Plane Instability Conditions

   3.4.3.4 Security Plane Instability Conditions

     Definition:
	Security Plane conditions that occur during the Accelerated
	Life Benchmark to produce instability and stress the DUT.

      Discussion:
	Security Plane Instability Conditions are experienced by the DUT
	after the Startup Conditions have completed.  Security Plane
	Instability Conditions experienced by the DUT include session
	loss and uninitiated policy changes.

      Measurement units:
	N/A

      Issues:
	None

      See Also:
   	Instability Conditions
 	Control Plane Instability Conditions
   	Data Plane Instability Conditions
	Management Plane Instability Conditions

   3.5 Recovery

   3.5.1 Recovery Phase

     Definition:
	The portion of the benchmarking test in which the
	Startup Conditions are generated with the DUT, but
	the Instability Conditions are no longer offered to
	the DUT.

     Discussion:
	The Recovery Phase is the final Phase of the
	benchmarking test following the Startup Phase and
	Instability Phase. Startup Conditions must not be
	Restarted.

     Measurement Units:
	None

     Issues:
	None

     See Also:
	Startup Conditions
	Startup Phase
	Instability Conditions
	Instability Phase

   3.5.2 Benchmarks
   3.5.2.1 Recovered Aggregate Forwarding Rate

      Definition
	Rate of traffic forwarded by the DUT during the Recovery
	Phase.

      Discussion:
	Recovered Aggregate Forwarding Rate is an instantaneous
	measurement of the Aggregate Forwarding Rate during the
	Recovery Phase.  Ideally, each measurement of the Recovered
	Aggregate Forwarding Rate equals the Stable Aggregate
	Forwarding Rate because the Instability Conditions
	do not exist in both the Startup and Recovery Phases.

      Measurement Units:
	pps

      Issues:
	None

      See Also:
	Aggregate Forwarding Rate
	Recovery Phase
	Recovered Aggregate Forwarding Rate
	Startup Phase
	Stable Aggregate Forwarding Rate

   3.5.2.2 Recovered Latency

      Definition:
        The average increase in measured packet latency during
        the Recovery Phase compared to the Startup Phase.

      Discussion:
        Latency SHOULD be measured at a fixed interval during the
        Recovery Phase.  Unstable Latency is the difference
        between Stable Latency and the average Latency measured
        during the Recovery Phase.  It is expected that there
        be no increase in average latency from the Startup Phase
        to the Recovery Phase.  The Recovered Latency cannot be a
        negative number.

      Measurement units:
	seconds

      Issues: None

      See Also:
        Recovery Phase
	Stable Latency

   3.5.2.3 Recovery Time

      Definition
	The amount of time for the Recovered Aggregate Forwarding
	Rate to become equal to the Stable Aggregate Forwarding Rate.

      Discussion
	Recovery Time is measured beginning at the instant the
	Instability Phase ends until the Recovered Aggregate
	Forwarding Rate equals the Stable Aggregate Forwarding
	Rate for a minimum duration of 180 consecutive seconds.

      Measurement Units:
	seconds

      Issues:
	None

      See Also:
	Recovered Aggregate Forwarding  Rate
	Stable Aggregate Forwarding Rate

   3.5.2.4  Recovered Uncontrolled Control Plane Sessions Lost

     Definition:
	Control Plane sessions that are in the down state
	but were not intentionally brought down during the
	Recovery Phase.

      Discussion:
	The test equipment is able to control protocol
	session state with the DUT.  The test equipment
	is also to monitor for sessions lost with the
	DUT which the test equipment itself did not
	intentionally bring down.

      Measurement units:
	sessions

      Issues:
	None

      See Also:
	Controlled Session Loss
	Uncontrolled Session Loss

   3.5.2.5 Variability Benchmarks

      Definition:
        The difference between the measured Benchmarks of the
        same DUT over multiple iterations.

      Discussion:
        Ideally, the benchmarks measured should be the same for
        multiple iterations with the same DUT.  Configuration
        Sets Instability conditions SHOULD be held constant for
        this benchmark. Whether the DUT can exhibit such predictable
        and repeatable behavior is an important benchmark in itself.

      Measurement units:
        As applicable to each Benchmark. The results are to be
        presented in a table format for successive Iterations.
        Ideally, the differences should be zero.

      Issues:
        None

      See Also:
        Startup Period
        Instability Period
        Recovery Period

   4. Security Considerations
        Documents of this type do not directly effect the security of
        the Internet or of corporate networks as long as benchmarking
        is not performed on devices or systems connected to operating
        networks.

   5. Normative References
      [1]   Bradner, S., Editor, "Benchmarking Terminology for Network
            Interconnection Devices", RFC 1242, October 1991.

      [2]   Mandeville, R., "Benchmarking Terminology for LAN Switching
            Devices", RFC 2285, June 1998.

      [3]   Bradner, S. and McQuaid, J., "Benchmarking Methodology for
	    Network Interconnect Devices", RFC 2544, March 1999.

      [4]   Poretsky, S. and Rao, S., "Methodology for Accelerated
	    Stress Benchmarking", draft-ietf-bmwg-acc-bench-meth-01,
	    work in progress, February 2005.

   6. Informative References
      [RFC3871]  RFC 3871 "Operational Security Requirements for Large
            Internet Service Provider (ISP) IP Network Infrastructure.
            G. Jones, Ed.. IETF, September 2004.

      [NANOG25]   "Core Router Evaluation for Higher Availability", Scott
	    Poretsky, NANOG 25, June 8, 2002, Toronto, CA.

      [5]

      [IEEECQR]   "Router Stress Testing to Validate Readiness for Network
	    Deployment", Scott Poretsky, IEEE CQR 2003.

      [6]

      [CONVMETH]   Poretsky, S. and Rao, S., "Methodology "Benchmarking Methodology for Accelerated
	    Stress Benchmarking", draft-ietf-bmwg-acc-bench-meth-01, IGP Data Plane
	    Route Convergence", draft-ietf-bmwg-igp-dataplane-conv-meth-05,
	    work in progress, October 2004.

   6. February 2005.

      [CONVTERM]   Poretsky, S., "Benchmarking Terminology for IGP Data Plane
	    Route Convergence", draft-ietf-bmwg-igp-dataplane-conv-term-05,
	    work in progress, February 2005.

   7. Author's Address

     	Scott Poretsky
   	Quarry Technologies
  	8 New England Executive Park
   	Burlington, MA 01803
    	USA
    	Phone: + 1 781 395 5090
   	EMail: sporetsky@quarrytech.com

	Shankar Rao
	Qwest Communications
	Denver, CO
	USA
	Phone: + 1 303 437 6643
	Email: shankar.rao@qwest.com

Appendix 1.  White Box Benchmarking Terminology
   Minimum Available Memory
      Definition:
	Minimum DUT Available Memory during the duration of the
	Accelerated Stress Test.

      Discussion:
	It is necessary to monitor DUT memory to measure this
	benchmark.

      Measurement units:
	bytes

      Issues: None

      See Also:
	Maximum CPU Utilization

   Maximum CPU Utilization
      Definition:
	Maximum DUT CPU utilization during the duration of the
	Accelerated Stress Test.

      Discussion:
	It is necessary to monitor DUT CPU Utilization to measure
	this benchmark.

      Measurement units:  %

      Issues: None

      See Also:
	Minimum Available Memory

Intellectual Property Statement

The IETF takes no position regarding the validity or scope of any Intel-
lectual Property Rights or other rights that might be claimed to pertain
to the implementation or use of the technology described in this docu-
ment or the extent to which any license under such rights might or might
not be available; nor does it represent that it has made any independent
effort to identify any such rights.  Information on the procedures with
respect to rights in RFC documents can be found in BCP 78 and BCP 79.

Copies of IPR disclosures made to the IETF Secretariat and any
assurances of licenses to be made available, or the result of an attempt
made to obtain a general license or permission for the use of such
proprietary rights by implementers or users of this specification can be
obtained from the IETF on-line IPR repository at
http://www.ietf.org/ipr.

The IETF invites any interested party to bring to its attention any
copyrights, patents or patent applications, or other proprietary rights
that may cover technology that may be required to implement this stan-
dard.  Please address the information to the IETF at ietf-ipr@ietf.org.

Disclaimer of Warranty

This document and the information contained herein are provided on an
"AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR
IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET
ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED,
INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMA-
TION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF
MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Copyright Statement

Copyright (C) The Internet Society (2004). (2005).  This document is subject to
the rights, licenses and restrictions contained in BCP 78, and except as
set forth therein, the authors retain all their rights.