draft-ietf-bmwg-issu-meth-01.txt   draft-ietf-bmwg-issu-meth-02.txt 
Benchmarking Working Group Sarah Banks Benchmarking Working Group Sarah Banks
Internet Draft VSS Monitoring Internet Draft VSS Monitoring
Intended status: Informational Fernando Calabria Intended status: Informational Fernando Calabria
Expires: November 30, 2015 Cisco Systems Expires: February 10, 2016 Cisco Systems
Gery Czirjak Gery Czirjak
Ramdas Machat Ramdas Machat
Juniper Networks Juniper Networks
May 30, 2015 Aug 8, 2015
ISSU Benchmarking Methodology ISSU Benchmarking Methodology
draft-ietf-bmwg-issu-meth-01 draft-ietf-bmwg-issu-meth-02
Abstract Abstract
Modern forwarding devices attempt to minimize any control and data Modern forwarding devices attempt to minimize any control and data
plane disruptions while performing planned software changes, by plane disruptions while performing planned software changes, by
implementing a technique commonly known as In Service Software implementing a technique commonly known as In Service Software
Upgrade (ISSU) This document specifies a set of common methodologies Upgrade (ISSU) This document specifies a set of common methodologies
and procedures designed to characterize the overall behavior of a and procedures designed to characterize the overall behavior of a
Device Under Test (DUT), subject to an ISSU event. Device Under Test (DUT), subject to an ISSU event.
skipping to change at page 1, line 45 skipping to change at page 1, line 45
documents at any time. It is inappropriate to use Internet-Drafts documents at any time. It is inappropriate to use Internet-Drafts
as reference material or to cite them other than as as reference material or to cite them other than as
"work in progress." "work in progress."
The list of current Internet-Drafts can be accessed at The list of current Internet-Drafts can be accessed at
http://www.ietf.org/1id-abstracts.html http://www.ietf.org/1id-abstracts.html
The list of Internet-Draft Shadow Directories can be accessed at The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html http://www.ietf.org/shadow.html
This Internet-Draft will expire on September 6, 2015. This Internet-Draft will expire on February 10, 2016.
Copyright Notice Copyright Notice
Copyright (c) 2015 IETF Trust and the persons identified as the Copyright (c) 2015 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
skipping to change at page 3, line 18 skipping to change at page 3, line 18
7. Final Report - Data Presentation - Analysis...................13 7. Final Report - Data Presentation - Analysis...................13
7.1. Data collection considerations...........................15 7.1. Data collection considerations...........................15
8. Security Considerations.......................................15 8. Security Considerations.......................................15
9. IANA Considerations...........................................16 9. IANA Considerations...........................................16
10. References...................................................16 10. References...................................................16
10.1. Normative References....................................16 10.1. Normative References....................................16
10.2. Informative References..................................16 10.2. Informative References..................................16
11. Acknowledgments..............................................16 11. Acknowledgments..............................................16
1. Introduction 1. Introduction
As required by most Service Provider (SP) network operators, ISSU
As required by most Service Provider (SP) network operators, ISSU
functionality has been implemented by modern forwarding devices to functionality has been implemented by modern forwarding devices to
upgrade or downgrade from one software version to another with a upgrade or downgrade from one software version to another with a
goal of eliminating the downtime of the router and/or the outage goal of eliminating the downtime of the router and/or the outage
of service. However, it is noted that while most operators desire of service. However, it is noted that while most operators desire
complete elimination of downtime, minimization of downtime and complete elimination of downtime, minimization of downtime and
service degradation is often the expectation. service degradation is often the expectation.
The ISSU operation may apply in terms of an atomic version change of The ISSU operation may apply in terms of an atomic version change of
the entire system software or it may be applied in a more modular the entire system software or it may be applied in a more modular
sense such as for a patch or maintenance upgrade. The procedure sense such as for a patch or maintenance upgrade. The procedure
skipping to change at page 5, line 15 skipping to change at page 5, line 15
3. Generic ISSU Process, phased approach 3. Generic ISSU Process, phased approach
ISSU may be viewed as the behavior of a device when exposed to a ISSU may be viewed as the behavior of a device when exposed to a
planned change in its software functionality. This may mean changes planned change in its software functionality. This may mean changes
to the core operating system, separate processes or daemons or even to the core operating system, separate processes or daemons or even
of firmware logic in programmable hardware devices (e.g. CPLD/FPGA). of firmware logic in programmable hardware devices (e.g. CPLD/FPGA).
The goal of an ISSU implementation is to permit such actions with The goal of an ISSU implementation is to permit such actions with
minimal or no disruption to the primary operation of the device in minimal or no disruption to the primary operation of the device in
question. question.
ISSU may be user initiated through direct interaction with the ISSU may be user initiated through direct interaction with the
device or activated through some automated process on a management device or activated through some automated process on a management
system or even on the device itself. For the purposes of this system or even on the device itself. For the purposes of this
document, we will focus on the model where the ISSU action is document, we will focus on the model where the ISSU action is
initiated by direct user intervention. initiated by direct user intervention.
The ISSU process can be viewed as a series of different phases or The ISSU process can be viewed as a series of different phases or
activities, as defined below. For each of these phases, the test activities, as defined below. For each of these phases, the test
operator MUST record the outcome as well as any relevant operator must record the outcome as well as any relevant
observations (defined further in the present document). Note that, a observations (defined further in the present document). Note that, a
given vendor implementation may or may not permit the abortion of given vendor implementation may or may not permit the abortion of
the in-progress ISSU at particular stages. There may also be certain the in-progress ISSU at particular stages. There may also be certain
restrictions as to ISSU availability given certain functional restrictions as to ISSU availability given certain functional
configurations (for example, ISSU in the presence of Bidirectional configurations (for example, ISSU in the presence of Bidirectional
Failure Detection (BFD) [RFC 5880] may not be supported). It is Failure Detection (BFD) [RFC 5880] may not be supported). It is
incumbent upon the test operator to ensure that the DUT is incumbent upon the test operator to ensure that the DUT is
appropriately configured to provide the appropriate test appropriately configured to provide the appropriate test
environment. As with any properly orchestrated test effort, the test environment. As with any properly orchestrated test effort, the test
plan document should reflect these and other relevant details and plan document should reflect these and other relevant details and
SHOULD be written with close attention to the expected production- should be written with close attention to the expected production-
operating environment. The combined analysis of the results of each operating environment. The combined analysis of the results of each
phase will characterize the overall ISSU process with the main goal phase will characterize the overall ISSU process with the main goal
of being able to identify and quantify any disruption in service of being able to identify and quantify any disruption in service
(from the data and control plane perspective) allowing operators to (from the data and control plane perspective) allowing operators to
plan their maintenance activities with greater precision. plan their maintenance activities with greater precision.
3.1. Software Download 3.1. Software Download
In this first phase, the requested software package may be In this first phase, the requested software package may be
downloaded to the router and is typically stored onto a device. The downloaded to the router and is typically stored onto a device. The
downloading of software may be performed automatically by the device downloading of software may be performed automatically by the device
as part of the upgrade process, or it may be initiated separately. as part of the upgrade process, or it may be initiated separately.
Such separation allows an administrator to download the new code Such separation allows an administrator to download the new code
inside or outside of a maintenance window; it is anticipated that inside or outside of a maintenance window; it is anticipated that
downloading new code and saving it to disk on the router will not downloading new code and saving it to disk on the router will not
impact operations. In the case where the software can be downloaded impact operations. In the case where the software can be downloaded
outside of the actual upgrade process, the administrator SHOULD do outside of the actual upgrade process, the administrator should do
so; downloading software can skew timing results based on factors so; downloading software can skew timing results based on factors
that are often not comparative in nature. Internal compatibility that are often not comparative in nature. Internal compatibility
verification may be performed by the software running on the DUT, to verification may be performed by the software running on the DUT, to
verify the checksum of the files downloaded as well as any other verify the checksum of the files downloaded as well as any other
pertinent checks. Depending upon vendor implementation, these pertinent checks. Depending upon vendor implementation, these
mechanisms may extend to include verification that the downloaded mechanisms may extend to include verification that the downloaded
module(s) meet a set of identified pre-requisites such as hardware module(s) meet a set of identified pre-requisites such as (but not
or firmware compatibility or minimum software requirements. Where limited to) hardware or firmware compatibility, minimum software
such mechanisms are made available by the product, they should be requirements or even ensure that device is "authorized" to run the
verified, by the tester, with the perspective of avoiding target software.
Where such mechanisms are made available by the product, they
should be verified, by the tester, with the perspective of avoiding
operational issues in production. Verification should include both operational issues in production. Verification should include both
positive verification (ensuring that an ISSU action should be positive verification (ensuring that an ISSU action should be
permitted) as well as negative tests (creation of scenarios where permitted) as well as negative tests (creation of scenarios where
the verification mechanisms would report exceptions). the verification mechanisms would report exceptions).
3.2. Software Staging 3.2. Software Staging
In this second phase, the requested software package is loaded in In this second phase, the requested software package is loaded in
the pertinent components of a given forwarding device (typically the the pertinent components of a given forwarding device (typically the
RP in standby state). Internal compatibility verification may be RP in standby state). Internal compatibility verification may be
performed by the software running on the DUT, as part of the upgrade performed by the software running on the DUT, as part of the upgrade
process itself, to verify the checksum of the files downloaded as process itself, to verify the checksum of the files downloaded as
well as any other pertinent checks. Depending upon vendor well as any other pertinent checks. Depending upon vendor
implementation, these mechanisms may extend to include verification implementation, these mechanisms may extend to include verification
that the downloaded module(s) meet a set of identified pre- that the downloaded module(s) meet a set of identified pre-
requisites such as hardware or firmware compatibility or minimum requisites such as hardware or firmware compatibility or minimum
software requirements. Where such mechanisms are made available by software requirements. Where such mechanisms are made available by
the product, they should be verified, by the tester (again with the the product, they should be verified, by the tester (again with the
perspective of avoiding operational issues in production). In this perspective of avoiding operational issues in production). In this
case, the execution of these checks is within scope of the upgrade case, the execution of these checks is within scope of the upgrade
time, and SHOULD be included in the testing results. Once the new time, and should be included in the testing results. Once the new
software is downloaded to the pertinent components of the DUT, the software is downloaded to the pertinent components of the DUT, the
upgrade begins and the DUT begins to prepare itself for upgrade. upgrade begins and the DUT begins to prepare itself for upgrade.
Depending on the vendor implementation, it is expected that Depending on the vendor implementation, it is expected that
redundant hardware pieces within the DUT are upgraded, including the redundant hardware pieces within the DUT are upgraded, including the
backup or secondary RP. backup or secondary RP.
3.3. Upgrade Run 3.3. Upgrade Run
In this phase, a switchover of RPs may take place, where one RP is In this phase, a switchover of RPs may take place, where one RP is
now upgraded with the new version of software. More importantly, the now upgraded with the new version of software. More importantly, the
skipping to change at page 7, line 16 skipping to change at page 7, line 16
interruptions to the forwarding plane should be minimal to none. For interruptions to the forwarding plane should be minimal to none. For
some implementations, the above two steps may be concatenated into some implementations, the above two steps may be concatenated into
one monolithic operation. In such case, the calculation of the one monolithic operation. In such case, the calculation of the
respective ISSU time intervals may need to be adapted accordingly. respective ISSU time intervals may need to be adapted accordingly.
If any control or data plane interruptions are observed within this If any control or data plane interruptions are observed within this
stage, they should be recorded as part of the results document. stage, they should be recorded as part of the results document.
3.4. Upgrade Acceptance 3.4. Upgrade Acceptance
In this phase, the new version of software MUST be running in all In this phase, the new version of software must be running in all
the physical nodes of the logical forwarding device. (RP's and LC's the physical nodes of the logical forwarding device. (RP's and LC's
as applicable). At this point, configuration control is returned to as applicable). At this point, configuration control is returned to
the operator and normal device operation i.e. outside of ISSU- the operator and normal device operation i.e. outside of ISSU-
oriented operation, is resumed. oriented operation, is resumed.
4. Test Methodology 4. Test Methodology
As stated by [RFC 6815], the Test Topology Setup must be part As stated by [RFC 6815], the Test Topology Setup must be part
of an ITE (Isolated Test Environment) of an ITE (Isolated Test Environment)
The reporting of results MUST take into account the repeatability The reporting of results must take into account the repeatability
considerations from Section 4 of [RFC 2544]. It is RECOMMENDED to considerations from Section 4 of [RFC 2544]. It is RECOMMENDED to
perform multiple trials and report average results. The results are perform multiple trials and report average results. The results are
reported in a simple statement including the measured frame loss and reported in a simple statement including the measured frame loss and
ISSU impact times. ISSU impact times.
4.1. Test Topology 4.1. Test Topology
The hardware configuration of the DUT (Device Under test) SHOULD be The hardware configuration of the DUT (Device Under test) should be
identical to the one expected to be or currently deployed in identical to the one expected to be or currently deployed in
production in order for the benchmark to have relevance. This would production in order for the benchmark to have relevance. This would
include the number of RP's, hardware version, memory and initial include the number of RP's, hardware version, memory and initial
software release, any common chassis components, such as fabric software release, any common chassis components, such as fabric
hardware in the case of a fabric-switching platform and the specific hardware in the case of a fabric-switching platform and the specific
LC's (version, memory, interfaces type, rate etc.) LC's (version, memory, interfaces type, rate etc.)
For the Control and Data plane, differing configuration approached For the Control and Data plane, differing configuration approached
MAY be utilized. The recommended approach relies on "mimicking" the may be utilized. The recommended approach relies on "mimicking" the
existing production data and control plane information, in order to existing production data and control plane information, in order to
emulate all the necessary Layer1 through Layer3 communications and, emulate all the necessary Layer1 through Layer3 communications and,
if appropriate, the upper layer characteristics of the network, as if appropriate, the upper layer characteristics of the network, as
well as end to end traffic/communication pairs. In other words, well as end to end traffic/communication pairs. In other words,
design a representative load model of the production environment and design a representative load model of the production environment and
deploy a collapsed topology utilizing test tools and/or external deploy a collapsed topology utilizing test tools and/or external
devices, where the DUT will be tested. Note that, the negative devices, where the DUT will be tested. Note that, the negative
impact of ISSU operations is likely to impact scaled, dynamic impact of ISSU operations is likely to impact scaled, dynamic
topologies to a greater extent than simpler, static environments. As topologies to a greater extent than simpler, static environments. As
such, this methodology (based upon production configuration) is such, this methodology (based upon production configuration) is
skipping to change at page 8, line 43 skipping to change at page 8, line 43
timing extrapolations at a minimum granularity of 100 milliseconds timing extrapolations at a minimum granularity of 100 milliseconds
e.g. 100Mbps for a 10Gbps interface. The use of steady traffic e.g. 100Mbps for a 10Gbps interface. The use of steady traffic
streams rather than bursty loads is preferred to simplify analysis. streams rather than bursty loads is preferred to simplify analysis.
The traffic should be patterned to provide a broad range of source The traffic should be patterned to provide a broad range of source
and destination pairs, which resolve to a variety of FIB (forwarding and destination pairs, which resolve to a variety of FIB (forwarding
information base) prefix lengths. If the production network information base) prefix lengths. If the production network
environment includes multicast traffic or VPN's (L2, L3 or IPSec) it environment includes multicast traffic or VPN's (L2, L3 or IPSec) it
is critical to include these in the model. is critical to include these in the model.
For mixed protocol environments (e.g. IPv4 and IPv6), frames SHOULD For mixed protocol environments (e.g. IPv4 and IPv6), frames should
be distributed between the different protocols. The distribution be distributed between the different protocols. The distribution
SHOULD approximate the network conditions of deployment. In all should approximate the network conditions of deployment. In all
cases, the details of the mixed protocol distribution MUST be cases, the details of the mixed protocol distribution must be
included in the reporting. included in the reporting.
The feature, protocol timing and other relevant configurations The feature, protocol timing and other relevant configurations
should be matched to the expected production environment. Deviations should be matched to the expected production environment. Deviations
from the production templates may be deemed necessary by the test from the production templates may be deemed necessary by the test
operator (for example, certain features may not support ISSU or the operator (for example, certain features may not support ISSU or the
test bed may not be able to accommodate such). However, the impact test bed may not be able to accommodate such). However, the impact
of any such divergence should be clearly understood and the of any such divergence should be clearly understood and the
differences MUST be recorded in the results documentation. It is differences must be recorded in the results documentation. It is
recommended that an NMS system be deployed, preferably similar to recommended that an NMS system be deployed, preferably similar to
that utilized in production. This will allow for monitoring of the that utilized in production. This will allow for monitoring of the
DUT while it is being tested both in terms of supporting the system DUT while it is being tested both in terms of supporting the system
resource impact analysis as well as from the perspective of resource impact analysis as well as from the perspective of
detecting interference with non-transit (management) traffic as a detecting interference with non-transit (management) traffic as a
result of the ISSU operation. Additionally, a DUT management session result of the ISSU operation. Additionally, a DUT management session
other than snmp-based, typical of usage in production, should be other than snmp-based, typical of usage in production, should be
established to the DUT and monitored for any disruption. It is established to the DUT and monitored for any disruption. It is
suggested that the actual test exercise be managed utilizing direct suggested that the actual test exercise be managed utilizing direct
console access to the DUT, if at all possible to avoid the console access to the DUT, if at all possible to avoid the
skipping to change at page 9, line 39 skipping to change at page 9, line 39
applied for each of the above phases follows: applied for each of the above phases follows:
5.1. Pre-ISSU recommended verifications 5.1. Pre-ISSU recommended verifications
1. Verify that enough hardware and software resources are available 1. Verify that enough hardware and software resources are available
to complete the Load operation (enough disk space). to complete the Load operation (enough disk space).
2. Verify that the redundancy states between RPs and other nodes are 2. Verify that the redundancy states between RPs and other nodes are
as expected (e.g. redundancy on, RP's synchronized). as expected (e.g. redundancy on, RP's synchronized).
3. Verify that the device, if running NSR capable routing protocols, 3. Verify that the device, if running NSR (Non Stop Routing)
is in a "ready" state; that is, that the sync between RPs is capable routing protocols, is in a "ready" state; that is,
complete and the system is ready for failover, if necessary. that the sync between RPs is complete and the system is ready
for failover, if necessary.
4. Gather a configuration snapshot of the device and all of its 4. Gather a configuration snapshot of the device and all of its
applicable components. applicable components.
5. Verify that the node is operating in a "steady" state (that is, 5. Verify that the node is operating in a "steady" state (that is,
no critical or maintenance function is being currently performed). no critical or maintenance function is being currently performed).
6. Note any other operational characteristics that the tester may 6. Note any other operational characteristics that the tester may
deem applicable to the specific implementation deployed. deem applicable to the specific implementation deployed.
skipping to change at page 12, line 35 skipping to change at page 12, line 35
information from the production environment and determine a specific information from the production environment and determine a specific
number of routes to flap every 'fixed' or 'variable' interval. number of routes to flap every 'fixed' or 'variable' interval.
Alternatively, the operator may wish to simply pre-select a fixed Alternatively, the operator may wish to simply pre-select a fixed
number of prefixes to flap. As an example, an operator may decide to number of prefixes to flap. As an example, an operator may decide to
flap 1% of all the BGP routes every minute and restore them 1 minute flap 1% of all the BGP routes every minute and restore them 1 minute
afterwards. The tester may wish to apply this negative stimulus afterwards. The tester may wish to apply this negative stimulus
throughout the entire ISSU process or most importantly, during the throughout the entire ISSU process or most importantly, during the
run phase. It is important to ensure that these routes, which are run phase. It is important to ensure that these routes, which are
introduced solely for stress proposes, must not overlap the ones introduced solely for stress proposes, must not overlap the ones
(per the Load Model) specifically leveraged to calculate the TP (per the Load Model) specifically leveraged to calculate the TP
(recorded outage). Furthermore, there SHOULD NOT be 'operator (recorded outage). Furthermore, there should NOT be 'operator
induced' control plane - protocol adjacency flaps for the duration induced' control plane - protocol adjacency flaps for the duration
of the test process as it may adversely affect the characterization of the test process as it may adversely affect the characterization
of the entire test exercise. For example, triggering IGP adjacency of the entire test exercise. For example, triggering IGP adjacency
events may force re-computation of underlying routing tables with events may force re-computation of underlying routing tables with
attendant impact to the perceived ISSU timings. While not attendant impact to the perceived ISSU timings. While not
recommended, if such trigger events are desired by the test recommended, if such trigger events are desired by the test
operator, care should be taken to avoid the introduction of operator, care should be taken to avoid the introduction of
unexpected anomalies within the test harness. unexpected anomalies within the test harness.
6. ISSU Abort and Rollback 6. ISSU Abort and Rollback
skipping to change at page 13, line 41 skipping to change at page 13, line 41
ISSU Traffic Impact Time (milliseconds) TP Time ISSU Traffic Impact Time (milliseconds) TP Time
ISSU Housekeeping Interval T ISSU Housekeeping Interval T
(Time for both RP's up on new code and fully synced - Redundancy (Time for both RP's up on new code and fully synced - Redundancy
restored) restored)
Total ISSU Maintenance Window T4 (sum of T1+T2+T3) Total ISSU Maintenance Window T4 (sum of T1+T2+T3)
The results reporting MUST provide the following information: The results reporting must provide the following information:
DUT hardware and software detail DUT hardware and software detail
Test Topology definition and diagram (especially as related to the Test Topology definition and diagram (especially as related to the
ISSU operation) ISSU operation)
Load Model description including protocol mixes and any divergence Load Model description including protocol mixes and any divergence
from the production environment from the production environment
Time Results as per above Time Results as per above
skipping to change at page 15, line 15 skipping to change at page 15, line 15
cases, any unexpected behavioral changes should be analyzed and a cases, any unexpected behavioral changes should be analyzed and a
determination made as to the impact of the change (be it determination made as to the impact of the change (be it
functional variances or operational impacts to existing scripts or functional variances or operational impacts to existing scripts or
management mechanisms. management mechanisms.
7.1. Data collection considerations 7.1. Data collection considerations
When a DUT is undergoing an ISSU operation, it's worth noting that When a DUT is undergoing an ISSU operation, it's worth noting that
the DUT's data collection and reporting of data, such as counters, the DUT's data collection and reporting of data, such as counters,
interface statistics, log messages, etc., may not be accurate. As interface statistics, log messages, etc., may not be accurate. As
such, one SHOULD NOT rely on the DUTs data collection methods, but such, one should NOT rely on the DUTs data collection methods, but
rather, SHOULD use the test tools and equipment to collect data used rather, should use the test tools and equipment to collect data used
for reporting in Section 7. Care and consideration should be paid in for reporting in Section 7. Care and consideration should be paid in
testing or adding new test cases, such that the desired data can be testing or adding new test cases, such that the desired data can be
collected from the test tools themselves, or other external collected from the test tools themselves, or other external
equipment, outside of the DUT itself. equipment, outside of the DUT itself.
8. Security Considerations 8. Security Considerations
All BMWG memos are limited to testing in a laboratory Isolated Test All BMWG memos are limited to testing in a laboratory Isolated Test
Environment (ITE), thus avoiding accidental interruption to Environment (ITE), thus avoiding accidental interruption to
production networks due to test activities. production networks due to test activities.
All benchmarking activities are limited to technology All benchmarking activities are limited to technology
characterization using controlled stimuli in a laboratory characterization using controlled stimuli in a laboratory
environment with dedicated address space and the other constraints environment with dedicated address space and the other constraints
[RFC 2544] [RFC 2544]
The benchmarking network topology will be an independent test setup The benchmarking network topology will be an independent test setup
and MUST NOT be connected to devices that may forward the test and must NOT be connected to devices that may forward the test
traffic into a production network or misroute traffic to the test traffic into a production network or misroute traffic to the test
management network. management network.
Further, benchmarking is performed on a "black-box" basis, relying Further, benchmarking is performed on a "black-box" basis, relying
solely on measurements observable external to the device under test/ solely on measurements observable external to the device under test/
system under test (DUT/SUT). system under test (DUT/SUT).
Special capabilities SHOULD NOT exist in the DUT/SUT specifically Special capabilities should NOT exist in the DUT/SUT specifically
for benchmarking purposes. Any implications for network security for benchmarking purposes. Any implications for network security
arising from the DUT/SUT SHOULD be identical in the lab and in arising from the DUT/SUT should be identical in the lab and in
production networks. production networks.
9. IANA Considerations 9. IANA Considerations
There are no IANA actions required by this memo. There are no IANA actions required by this memo.
10. References 10. References
10.1. Normative References 10.1. Normative References
 End of changes. 25 change blocks. 
32 lines changed or deleted 36 lines changed or added

This html diff was produced by rfcdiff 1.42. The latest version is available from http://tools.ietf.org/tools/rfcdiff/