draft-ietf-ippm-capacity-metric-method-00.txt   draft-ietf-ippm-capacity-metric-method-01.txt 
Network Working Group A. Morton Network Working Group A. Morton
Internet-Draft AT&T Labs Internet-Draft AT&T Labs
Updates: ???? (if approved) R. Geib Updates: ???? (if approved) R. Geib
Intended status: Standards Track Deutsche Telekom Intended status: Standards Track Deutsche Telekom
Expires: August 1, 2020 L. Ciavattone Expires: September 10, 2020 L. Ciavattone
AT&T Labs AT&T Labs
January 29, 2020 March 9, 2020
Metrics and Methods for IP Capacity Metrics and Methods for IP Capacity
draft-ietf-ippm-capacity-metric-method-00 draft-ietf-ippm-capacity-metric-method-01
Abstract Abstract
This memo revisits the problem of Network Capacity metrics first This memo revisits the problem of Network Capacity metrics first
examined in RFC 5136. The memo specifies a more practical Maximum examined in RFC 5136. The memo specifies a more practical Maximum
IP-layer Capacity metric definition catering for measurement IP-layer Capacity metric definition catering for measurement
purposes, and outlines the corresponding methods of measurement. purposes, and outlines the corresponding methods of measurement.
Requirements Language Requirements Language
skipping to change at page 1, line 44 skipping to change at page 1, line 44
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/. Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on August 1, 2020. This Internet-Draft will expire on September 10, 2020.
Copyright Notice Copyright Notice
Copyright (c) 2020 IETF Trust and the persons identified as the Copyright (c) 2020 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(https://trustee.ietf.org/license-info) in effect on the date of (https://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
skipping to change at page 2, line 42 skipping to change at page 2, line 42
5.6. Reporting the Metric . . . . . . . . . . . . . . . . . . 8 5.6. Reporting the Metric . . . . . . . . . . . . . . . . . . 8
6. Maximum IP-Layer Capacity Metric Definitions (Statistic) . . 8 6. Maximum IP-Layer Capacity Metric Definitions (Statistic) . . 8
6.1. Formal Name . . . . . . . . . . . . . . . . . . . . . . . 8 6.1. Formal Name . . . . . . . . . . . . . . . . . . . . . . . 8
6.2. Parameters . . . . . . . . . . . . . . . . . . . . . . . 8 6.2. Parameters . . . . . . . . . . . . . . . . . . . . . . . 8
6.3. Metric Definitions . . . . . . . . . . . . . . . . . . . 9 6.3. Metric Definitions . . . . . . . . . . . . . . . . . . . 9
6.4. Related Round-Trip Delay and Loss Definitions . . . . . . 10 6.4. Related Round-Trip Delay and Loss Definitions . . . . . . 10
6.5. Discussion . . . . . . . . . . . . . . . . . . . . . . . 10 6.5. Discussion . . . . . . . . . . . . . . . . . . . . . . . 10
6.6. Reporting the Metric . . . . . . . . . . . . . . . . . . 11 6.6. Reporting the Metric . . . . . . . . . . . . . . . . . . 11
7. IP-Layer Sender Bit Rate Singleton Metric Definitions . . . . 11 7. IP-Layer Sender Bit Rate Singleton Metric Definitions . . . . 11
7.1. Formal Name . . . . . . . . . . . . . . . . . . . . . . . 11 7.1. Formal Name . . . . . . . . . . . . . . . . . . . . . . . 11
7.2. Parameters . . . . . . . . . . . . . . . . . . . . . . . 11 7.2. Parameters . . . . . . . . . . . . . . . . . . . . . . . 12
7.3. Metric Definition . . . . . . . . . . . . . . . . . . . . 12 7.3. Metric Definition . . . . . . . . . . . . . . . . . . . . 12
7.4. Discussion . . . . . . . . . . . . . . . . . . . . . . . 12 7.4. Discussion . . . . . . . . . . . . . . . . . . . . . . . 12
7.5. Reporting the Metric . . . . . . . . . . . . . . . . . . 12 7.5. Reporting the Metric . . . . . . . . . . . . . . . . . . 12
8. Method of Measurement . . . . . . . . . . . . . . . . . . . . 12 8. Method of Measurement . . . . . . . . . . . . . . . . . . . . 13
8.1. Load Rate Adjustment Algorithm (from udpst) . . . . . . . 13 8.1. Load Rate Adjustment Algorithm . . . . . . . . . . . . . 13
8.2. Measurement Qualification or Verification . . . . . . . . 14 8.2. Measurement Qualification or Verification . . . . . . . . 14
8.3. Measurement Considerations . . . . . . . . . . . . . . . 15 8.3. Measurement Considerations . . . . . . . . . . . . . . . 15
9. Reporting . . . . . . . . . . . . . . . . . . . . . . . . . . 16 9. Reporting Formats . . . . . . . . . . . . . . . . . . . . . . 16
10. Security Considerations . . . . . . . . . . . . . . . . . . . 17 10. Security Considerations . . . . . . . . . . . . . . . . . . . 18
11. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 17 11. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 18
12. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 17 12. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 18
13. References . . . . . . . . . . . . . . . . . . . . . . . . . 17 13. References . . . . . . . . . . . . . . . . . . . . . . . . . 18
13.1. Normative References . . . . . . . . . . . . . . . . . . 17 13.1. Normative References . . . . . . . . . . . . . . . . . . 18
13.2. Informative References . . . . . . . . . . . . . . . . . 19 13.2. Informative References . . . . . . . . . . . . . . . . . 20
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 20 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 21
1. Introduction 1. Introduction
The IETF's efforts to define Network and Bulk Transport Capacity have The IETF's efforts to define Network and Bulk Transport Capacity have
been chartered and progressed for over twenty years. Over that time, been chartered and progressed for over twenty years. Over that time,
the performance community has seen development of Informative the performance community has seen development of Informative
definitions in [RFC3148] for Framework for Bulk Transport Capacity definitions in [RFC3148] for Framework for Bulk Transport Capacity
(BTC), RFC 5136 for Network Capacity and Maximum IP-layer Capacity, (BTC), RFC 5136 for Network Capacity and Maximum IP-layer Capacity,
and the Experimental metric definitions and methods in [RFC8337], and the Experimental metric definitions and methods in [RFC8337],
Model-Based Metrics for BTC. Model-Based Metrics for BTC.
skipping to change at page 4, line 5 skipping to change at page 4, line 5
support UDP transport. A number of liaisons have been exchanged on support UDP transport. A number of liaisons have been exchanged on
this topic [ refs to ITU-T SG12, ETSI STQ, BBF liaisons ], discussing this topic [ refs to ITU-T SG12, ETSI STQ, BBF liaisons ], discussing
the laboratory and field tests that support the UDP-based approach to the laboratory and field tests that support the UDP-based approach to
IP-layer Capacity measurement. IP-layer Capacity measurement.
This memo also recognizes the many updates to the IP Performance This memo also recognizes the many updates to the IP Performance
Metrics Framework [RFC2330] published over twenty years, and makes Metrics Framework [RFC2330] published over twenty years, and makes
use of [RFC7312] for Advanced Stream and Sampling Framework, and RFC use of [RFC7312] for Advanced Stream and Sampling Framework, and RFC
8468 [RFC8468]IPv4, IPv6, and IPv4-IPv6 Coexistence Updates. 8468 [RFC8468]IPv4, IPv6, and IPv4-IPv6 Coexistence Updates.
NOTE: The text contains a few Author comments, in brackets [RG: ,
acm: ]
2. Scope and Goals 2. Scope and Goals
The scope of this memo is to define a metric and corresponding method The scope of this memo is to define a metric and corresponding method
to unambiguously perform Active measurements of Maximum IP-Layer to unambiguously perform Active measurements of Maximum IP-Layer
Capacity, along with related metrics and methods. Capacity, along with related metrics and methods.
The main goal is to harmonize the specified metric and method across The main goal is to harmonize the specified metric and method across
the industry, and this memo is the vehicle through which working the industry, and this memo is the vehicle through which working
group (and eventually, IETF) consensus will be captured and group (and eventually, IETF) consensus will be captured and
communicated to achieve broad agreement, and possibly result in communicated to achieve broad agreement, and possibly result in
skipping to change at page 6, line 49 skipping to change at page 6, line 49
This section defines the REQUIRED aspects of the measureable IP-layer This section defines the REQUIRED aspects of the measureable IP-layer
Capacity metric (unless otherwise indicated) for measurements between Capacity metric (unless otherwise indicated) for measurements between
specified Source and Destination hosts: specified Source and Destination hosts:
Define the IP-layer capacity, C(T,I,PM), to be the number of IP-layer Define the IP-layer capacity, C(T,I,PM), to be the number of IP-layer
bits (including header and data fields) in packets that can be bits (including header and data fields) in packets that can be
transmitted from the Src host and correctly received by the Dst host transmitted from the Src host and correctly received by the Dst host
during one contiguous sub-interval, dt. during one contiguous sub-interval, dt.
The number of these IP-layer bits is designated n0[dtn-1,dtn] for a The number of these IP-layer bits is designated n0[dtn,dtn+1] for a
specific dtn. specific dt.
When the packet size is known and of fixed size, the packet count When the packet size is known and of fixed size, the packet count
during a single sub-interval dt multiplied by the total bits in IP during a single sub-interval dt multiplied by the total bits in IP
header and data fields is equal to n0[dtn-1,dtn]. header and data fields is equal to n0[dtn,dtn+1].
Anticipating a Sample of Singletons, the interval dt SHOULD be set to Anticipating a Sample of Singletons, the interval dt SHOULD be set to
a natural number m so that T+I = T + m*dt with dtn - dtn-1 = dt and a natural number m so that T+I = T + m*dt with dtn+1 - dtn = dt and
with 0 < n <= m. with 1 <= n <= m.
Parameter PM represents other performance metrics [see section Parameter PM represents other performance metrics [see section
Related Round-Trip Delay and Loss Definitions below]; their Related Round-Trip Delay and Loss Definitions below]; their
measurement results SHALL be collected during measurement of IP-layer measurement results SHALL be collected during measurement of IP-layer
Capacity and associated with the corresponding dtn for further Capacity and associated with the corresponding dtn for further
evaluation and reporting. evaluation and reporting.
Mathematically, this definition can be represented as: Mathematically, this definition can be represented as:
( n0[dtn-1,dtn] ) ( n0[dtn,dtn+1] )
C(T,I,PM) = ------------------------- C(T,I,PM) = -------------------------
dt dt
Equation for IP-Layer Capacity Equation for IP-Layer Capacity
and: and:
o n0 is the total number of IP-layer header and payload bits that o n0 is the total number of IP-layer header and payload bits that
can be transmitted in Standard Formed packets from the Src host can be transmitted in Standard Formed packets from the Src host
and correctly received by the Dst host during one contiguous sub- and correctly received by the Dst host during one contiguous sub-
skipping to change at page 9, line 12 skipping to change at page 9, line 12
No additional Parameters or definitions are needed. No additional Parameters or definitions are needed.
6.3. Metric Definitions 6.3. Metric Definitions
This section defines the REQUIRED aspects of the Maximum IP-layer This section defines the REQUIRED aspects of the Maximum IP-layer
Capacity metric (unless otherwise indicated) for measurements between Capacity metric (unless otherwise indicated) for measurements between
specified Source and Destination hosts: specified Source and Destination hosts:
Define the Maximum IP-layer capacity, Maximum_C(T,I,PM), to be the Define the Maximum IP-layer capacity, Maximum_C(T,I,PM), to be the
maximum number of IP-layer bits n0[dtn-1,dtn] that can be transmitted maximum number of IP-layer bits n0[dtn,dtn+1] that can be transmitted
in packets from the Src host and correctly received by the Dst host, in packets from the Src host and correctly received by the Dst host,
over all dt length intervals in [T, T+I], and meeting the PM over all dt length intervals in [T, T+I], and meeting the PM
criteria. Equivalently the Maximum of a Sample of size m of criteria. Equivalently the Maximum of a Sample of size m of
C(T,I,PM) collected during the interval [T, T+I] and meeting the PM C(T,I,PM) collected during the interval [T, T+I] and meeting the PM
criteria. criteria.
The interval dt SHOULD be set to a natural number m so that T+I = T + The interval dt SHOULD be set to a natural number m so that T+I = T +
m*dt with dtn - dtn-1 = dt and with 0 < n <= m. m*dt with dtn+1 - dtn = dt and with 1 <= n <= m.
Parameter PM represents the other performance metrics [see section Parameter PM represents the other performance metrics [see section
Related Round-Trip Delay and Loss Definitions below] and their Related Round-Trip Delay and Loss Definitions below] and their
measurement results for the maximum IP-layer capacity. At least one measurement results for the maximum IP-layer capacity. At least one
target performance threshold (PM criterion) MUST be defined. If more target performance threshold (PM criterion) MUST be defined. If more
than one target performance threshold is defined, then the sub- than one target performance threshold is defined, then the sub-
interval with maximum number of bits transmitted MUST meet all the interval with maximum number of bits transmitted MUST meet all the
target performance thresholds. target performance thresholds.
Mathematically, this definition can be represented as: Mathematically, this definition can be represented as:
max ( n0[dtn-1,dtn] ) max ( n0[dtn,dtn+1] )
[T,T+I] [T,T+I]
Maximum_C(T,I,PM) = ------------------------- Maximum_C(T,I,PM) = -------------------------
dt dt
where: where:
T T+I T T+I
_________________________________________ _________________________________________
| | | | | | | | | | | | | | | | | | | | | |
dtn=0 1 2 3 4 5 6 7 8 9 m=10 dtn=1 2 3 4 5 6 7 8 9 10 n+1
n=m
Equation for Maximum Capacity Equation for Maximum Capacity
and: and:
o n0 is the total number of IP-layer header and payload bits that o n0 is the total number of IP-layer header and payload bits that
can be transmitted in Standard Formed packets from the Src host can be transmitted in Standard Formed packets from the Src host
and correctly received by the Dst host during one contiguous sub- and correctly received by the Dst host during one contiguous sub-
interval, dt in length, during the interval [T, T+I], interval, dt in length, during the interval [T, T+I],
skipping to change at page 10, line 31 skipping to change at page 10, line 31
test of duration, I. When the transmitted packet rate is held test of duration, I. When the transmitted packet rate is held
constant at the Src host, the m sub-intervals may also be viewed as constant at the Src host, the m sub-intervals may also be viewed as
trials to evaluate the stability of n0 and metric(s) in the PM list trials to evaluate the stability of n0 and metric(s) in the PM list
over all dt-length intervals in I. over all dt-length intervals in I.
Measurements according to these definitions SHALL use UDP transport Measurements according to these definitions SHALL use UDP transport
layer. layer.
6.4. Related Round-Trip Delay and Loss Definitions 6.4. Related Round-Trip Delay and Loss Definitions
RTD[dtn-1,dtn] is defined as a sample of the [RFC2681] Round-trip RTD[dtn,dtn+1] is defined as a sample of the [RFC2681] Round-trip
Delay between the Src host and the Dst host over the interval Delay between the Src host and the Dst host over the interval
[T,T+I], and corresponds to the dt interval containing [T,T+I], and corresponds to the dt interval containing
Maximum_C(T,I,PM). The statistics used to to summarize RTD[dtn- Maximum_C(T,I,PM). The statistics used to to summarize
1,dtn] MAY include the minimum, maximum, and mean. RTD[dtn,dtn+1] MAY include the minimum, maximum, and mean.
RTL[dtn-1,dtn] is defined as a sample of the [RFC6673] Round-trip RTL[dtn,dtn+1] is defined as a sample of the [RFC6673] Round-trip
Loss between the Src host and the Dst host over the interval [T,T+I] Loss between the Src host and the Dst host over the interval [T,T+I]
and corresponds to the dt interval containing Maximum_C(T,I,PM). The and corresponds to the dt interval containing Maximum_C(T,I,PM). The
statistics used to to summarize RTL[dtn-1,dtn] MAY include the lost statistics used to to summarize RTL[dtn-1,dtn] MAY include the lost
packet count and the lost packet ratio. packet count and the lost packet ratio.
6.5. Discussion 6.5. Discussion
If traffic conditioning applies along a path for which Maximum If traffic conditioning applies along a path for which Maximum
_C(T,I,PM) is to be determined, different values for dt SHOULD be _C(T,I,PM) is to be determined, different values for dt SHOULD be
picked and measurements be executed during multiple intervals [T, picked and measurements be executed during multiple intervals [T,
skipping to change at page 11, line 18 skipping to change at page 11, line 18
6.6. Reporting the Metric 6.6. Reporting the Metric
The Maximum IP-Layer Capacity SHALL be reported with meaningful The Maximum IP-Layer Capacity SHALL be reported with meaningful
resolution, in units of Megabits per second. resolution, in units of Megabits per second.
The Related Round Trip Delay and/or Loss metric measurements for the The Related Round Trip Delay and/or Loss metric measurements for the
same Singleton SHALL be reported, also with meaningful resolution for same Singleton SHALL be reported, also with meaningful resolution for
the values measured. the values measured.
When there are demonstrated and repeatable modes in the Sample, then When there are demonstrated and repeatable Capacity modes in the
the Maximum IP-Layer Capacity SHALL be reported for each mode, along Sample, then the Maximum IP-Layer Capacity SHALL be reported for each
with the relative time from the beginning of the stream that the mode mode, along with the relative time from the beginning of the stream
was observed to be present. Bimodal Maxima have been observed with that the mode was observed to be present. Bimodal Maxima have been
some services, sometimes called a "turbo" mode" intending to deliver observed with some services, sometimes called a "turbo mode"
short transfers more quickly, or reduce the initial buffering time intending to deliver short transfers more quickly, or reduce the
for some video streams. initial buffering time for some video streams. Note that modes
lasting less than dt duration will not be detected.
Some transmission technologies have multiple methods of operation
that may be activated when channel conditions degrade or improve, and
these transmission methods may determine the Maximum IP-Layer
Capacity. Examples include line-of-sight microwave modulator
constellations, or cellular modem technologies where the changes may
be initiated by a user moving from one coverage area to another.
Operation in the different transmission methods may be observed over
time, but the modes of Maximum IP-Layer Capacity will not be
activated deterministically as with the "turbo mode" described in the
paragraph above.
7. IP-Layer Sender Bit Rate Singleton Metric Definitions 7. IP-Layer Sender Bit Rate Singleton Metric Definitions
This section sets requirements for the following components to This section sets requirements for the following components to
support the IP-layer Sender Bitrate Metric. support the IP-layer Sender Bitrate Metric.
7.1. Formal Name 7.1. Formal Name
Type-P-IP-Sender-Bit-Rate, or informally called IP-layer Sender Type-P-IP-Sender-Bit-Rate, or informally called IP-layer Sender
Bitrate. Bitrate.
skipping to change at page 13, line 5 skipping to change at page 13, line 14
8. Method of Measurement 8. Method of Measurement
The duration of a test, I, MUST be constrained in a production The duration of a test, I, MUST be constrained in a production
network, since this is an active test method and it will likely cause network, since this is an active test method and it will likely cause
congestion on the Src to Dst host path during a test. congestion on the Src to Dst host path during a test.
Additional Test methods and configurations may be provided in this Additional Test methods and configurations may be provided in this
section, after review and further testing. section, after review and further testing.
8.1. Load Rate Adjustment Algorithm (from udpst) 8.1. Load Rate Adjustment Algorithm
A table is pre-built defining all the offered load rates that will be A table is pre-built defining all the offered load rates that will be
supported (R1 - Rn, in ascending order). Each rate is defined as supported (R1 - Rn, in ascending order). Each rate is defined as
datagrams of size S, sent as a burst of count C, every time interval datagrams of size S, sent as a burst of count C, every time interval
T. While it is advantageous to use datagrams of as large a size as T. While it is advantageous to use datagrams of as large a size as
possible, it may be prudent to use a slightly smaller maximum that possible, it may be prudent to use a slightly smaller maximum that
allows for secondary protocol headers and/or tunneling without allows for secondary protocol headers and/or tunneling without
resulting in IP-layer fragmentation. resulting in IP-layer fragmentation.
At the beginning of a test, the sender begins sending at rate R1 and At the beginning of a test, the sender begins sending at rate R1 and
skipping to change at page 14, line 7 skipping to change at page 14, line 14
If the feedback indicates that there were no sequence number If the feedback indicates that there were no sequence number
anomalies AND the delay variation was above the lower threshold, but anomalies AND the delay variation was above the lower threshold, but
below the upper threshold, the offered load rate is not changed. below the upper threshold, the offered load rate is not changed.
This allows time for recent changes in the offered load rate to This allows time for recent changes in the offered load rate to
stabilize, and the feedback to represent current conditions more stabilize, and the feedback to represent current conditions more
accurately. accurately.
Lastly, the method for confirming congestion is that there were Lastly, the method for confirming congestion is that there were
sequence number anomalies OR the delay variation was above the upper sequence number anomalies OR the delay variation was above the upper
threshold for two consecutive feedback intervals. threshold for two consecutive feedback intervals. The algorithm
described above is also presented in ITU-T Rec. Y.1540, 2020
version[Y.1540], in Annexes A and B.
8.2. Measurement Qualification or Verification 8.2. Measurement Qualification or Verification
When assessing a Maximum rate as the metric specifies, artificially When assessing a Maximum rate as the metric specifies, artificially
high (optimistic) values might be measured until some buffer on the high (optimistic) values might be measured until some buffer on the
path is filled. Other causes include bursts of back-to-back packets path is filled. Other causes include bursts of back-to-back packets
with idle intervals delivered by a path, while the measurement with idle intervals delivered by a path, while the measurement
interval (dt) is small and aligned with the bursts. The artificial interval (dt) is small and aligned with the bursts. The artificial
values might result in an un-sustainable Maximum Capacity observed values might result in an un-sustainable Maximum Capacity observed
when the method of measurement is searching for the Maximum, and that when the method of measurement is searching for the Maximum, and that
skipping to change at page 15, line 14 skipping to change at page 15, line 23
network coordination, the concerns raised in RFC 6815[RFC6815] are network coordination, the concerns raised in RFC 6815[RFC6815] are
alleviated. The method for assessing Max IP Capacity is different alleviated. The method for assessing Max IP Capacity is different
from classic [RFC2544] methods: they use short term load adjustment from classic [RFC2544] methods: they use short term load adjustment
and are sensitive to loss and delay, like other congestion control and are sensitive to loss and delay, like other congestion control
algorithms used on the Internet every day. algorithms used on the Internet every day.
8.3. Measurement Considerations 8.3. Measurement Considerations
In general, the wide-spread measurements that this memo encourages In general, the wide-spread measurements that this memo encourages
will encounter wide-spread behaviors. The bimodal IP Capacity will encounter wide-spread behaviors. The bimodal IP Capacity
behavior is a good example. behaviors already discussed in Section 6.6 are good examples.
In general, it is RECOMMENDED to locate test endpoints as close to
the intended measured link(s) as practical (this is not always
possible for reasons of scale; there is a limit on number of test
endpoints coming from many perspecitves, management and measurement
traffic for example).
The path measured may be state-full based on many factors, and the The path measured may be state-full based on many factors, and the
Parameter "Time of day" when a test starts may not be enough enough Parameter "Time of day" when a test starts may not be enough
information. Repeatable testing may require the time from the information. Repeatable testing may require the time from the
beginning of a measured flow, and how the flow is constructed beginning of a measured flow, and how the flow is constructed
including how much traffic has already been sent on that flow when a including how much traffic has already been sent on that flow when a
state-change is observed, because the state-change may be based on state-change is observed, because the state-change may be based on
time or bytes sent or both. time or bytes sent or both.
Many different traffic shapers and on-demand access technology may be Many different traffic shapers and on-demand access technologies may
encountered, as anticipated in [RFC7312], and play a key role in be encountered, as anticipated in [RFC7312], and play a key role in
measurement results. Methods MUST be prepared to provide a short measurement results. Methods MUST be prepared to provide a short
preamble transmission to activate on-demand access, and to discard preamble transmission to activate on-demand access, and to discard
the preamble from subsequent test results. the preamble from subsequent test results.
Conditions which might be encountered during measurement, where
packet losses may occur independently from the measurement sending
rate:
1. Congestion of an interconnection or backbone interface may appear
as packet losses distributed over time in the test stream, due to
much higher rate interfaces in the backbone.
2. Packet loss due to use of Random Early Detection (RED) or other
active queue management.
3. There may be only small delay variation independent of sending
rate under these conditions, too.
4. Persistent competing traffic on measurement paths that include
shared media may cause random packet losses in the test stream.
It is possible to mitigate these conditions using the flexibility of
the load-rate adjusting algorithm described in Section 8.1 above
(tuning specific parameters).
In general, results depend on the sending stream characteristics; the In general, results depend on the sending stream characteristics; the
measurement community has known this for a long time, and to keep it measurement community has known this for a long time, and needs to
front of mind. Although the default is a single flow (F=1) for keep it front of mind. Although the default is a single flow (F=1)
testing, use of multiple flows may be advantageous for the following for testing, use of multiple flows may be advantageous for the
reasons: following reasons:
1. the test hosts may be able to create higher load than with a 1. the test hosts may be able to create higher load than with a
single flow, or parallel test hosts may be used to generate 1 single flow, or parallel test hosts may be used to generate 1
flow each. flow each.
2. there may be link aggregation present (flow-based load balancing) 2. there may be link aggregation present (flow-based load balancing)
and multiple flows are need to occupy each member of the and multiple flows are needed to occupy each member of the
aggregate. aggregate.
Each flow would be controlled using its own implementation of the Each flow would be controlled using its own implementation of the
Load Adjustment (Search) Algorithm. Load Adjustment (Search) Algorithm.
As testing continues, implementers should expect some evolution in As testing continues, implementers should expect some evolution in
the methods. the methods.
9. Reporting 9. Reporting Formats
The singleton IP-Layer Capacity results SHOULD be accompanied by the
context under which they were measured.
o timestamp (especially the time when the maximum was observed in
dtn)
o source and destination (by IP or other meaningful ID)
o other inner parameters of the test case (Section 4)
o outer parameters, such as "done in motion" or other factors
belonging to the context of the measurement
o result validity (indicating cases where the process was somehow
interrupted or the attempt failed)
o a field where unusual circumstances could be documented, and
another one for "ignore/mask out" purposes in further processing
The Maximum IP-Layer Capacity results SHOULD be reported in the The Maximum IP-Layer Capacity results SHOULD be reported in the
format of a table with a row for each of the test Phases and Number format of a table with a row for each of the test Phases and Number
of Flows. There SHOULD be columns for the phases with number of of Flows. There SHOULD be columns for the phases with number of
flows, and for the resultant Maximum IP-Layer Capacity results for flows, and for the resultant Maximum IP-Layer Capacity results for
the aggregate and each flow tested. the aggregate and each flow tested.
The PM list metrics corresponding to the sub-interval where the As mentioned in Section 6.6, bi-modal (or multi-modal) maxima SHALL
Maximum Capacity occurred MUST accompany a report of Maximum IP-Layer be reported for each mode separately.
Capacity results, for each test phase.
+--------------+----------------------+-----------+-----------------+ +--------------+----------------------+-----------+-----------------+
| Phase, # | Max IP-Layer | Loss | RTT min, max, | | Phase, # | Max IP-Layer | Loss | RTT min, max, |
| Flows | Capacity, Mbps | Ratio | msec | | Flows | Capacity, Mbps | Ratio | msec |
+--------------+----------------------+-----------+-----------------+ +--------------+----------------------+-----------+-----------------+
| Search,1 | 967.31 | 0.0002 | 30, 58 | | Search,1 | 967.31 | 0.0002 | 30, 58 |
| Verify,1 | 966.00 | 0.0000 | 30, 38 | | Verify,1 | 966.00 | 0.0000 | 30, 38 |
+--------------+----------------------+-----------+-----------------+ +--------------+----------------------+-----------+-----------------+
Maximum IP-layer Capacity Results Maximum IP-layer Capacity Results
Static and configuration parameters: Static and configuration parameters:
The sub-interval time, dt, MUST accompany a report of Maximum IP- The sub-interval time, dt, MUST accompany a report of Maximum IP-
Layer Capacity results, and the remaining Parameters from Section 4, Layer Capacity results, and the remaining Parameters from Section 4,
General Parameters. General Parameters.
The PM list metrics corresponding to the sub-interval where the
Maximum Capacity occurred MUST accompany a report of Maximum IP-Layer
Capacity results, for each test phase.
The IP-Layer Sender Bit rate results SHOULD be reported in the format The IP-Layer Sender Bit rate results SHOULD be reported in the format
of a table with a row for each of the test Phases, sub-intervals (st) of a table with a row for each of the test Phases, sub-intervals (st)
and Number of Flows. There SHOULD be columns for the phases with and Number of Flows. There SHOULD be columns for the phases with
number of flows, and for the resultant IP-Layer Sender Bit rate number of flows, and for the resultant IP-Layer Sender Bit rate
results for the aggregate and each flow tested. results for the aggregate and each flow tested.
+------------------------+-------------+-----------------------+----+ +------------------------+-------------+-----------------------+----+
| Phase, Flow or | st, sec | Sender Bit Rate, Mbps | ?? | | Phase, Flow or | st, sec | Sender Bit Rate, Mbps | ?? |
| Aggregate | | | | | Aggregate | | | |
+------------------------+-------------+-----------------------+----+ +------------------------+-------------+-----------------------+----+
skipping to change at page 17, line 24 skipping to change at page 18, line 37
considerations [add references to LMAP Framework, etc.]. considerations [add references to LMAP Framework, etc.].
<There are certainly some new ones for Capacity testing> <There are certainly some new ones for Capacity testing>
11. IANA Considerations 11. IANA Considerations
This memo makes no requests of IANA. This memo makes no requests of IANA.
12. Acknowledgements 12. Acknowledgements
Thanks to Joachim Fabini, Matt Mathis, and Ignacio Alvarez-Hamelin Thanks to Joachim Fabini, Matt Mathis, Ignacio Alvarez-Hamelin, and
for their extensive comments on the memo and related topics. Wolfgang Balzer for their extensive comments on the memo and related
topics.
13. References 13. References
13.1. Normative References 13.1. Normative References
[RFC1242] Bradner, S., "Benchmarking Terminology for Network [RFC1242] Bradner, S., "Benchmarking Terminology for Network
Interconnection Devices", RFC 1242, DOI 10.17487/RFC1242, Interconnection Devices", RFC 1242, DOI 10.17487/RFC1242,
July 1991, <https://www.rfc-editor.org/info/rfc1242>. July 1991, <https://www.rfc-editor.org/info/rfc1242>.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
skipping to change at page 20, line 13 skipping to change at page 21, line 26
AppendixB:Back2BackTestingTimeSeries(fromCI)>. AppendixB:Back2BackTestingTimeSeries(fromCI)>.
[VSPERF-BSLV] [VSPERF-BSLV]
Morton, A. and S. Rao, "Evolution of Repeatability in Morton, A. and S. Rao, "Evolution of Repeatability in
Benchmarking: Fraser Plugfest (Summary for IETF BMWG)", Benchmarking: Fraser Plugfest (Summary for IETF BMWG)",
July 2018, July 2018,
<https://datatracker.ietf.org/meeting/102/materials/ <https://datatracker.ietf.org/meeting/102/materials/
slides-102-bmwg-evolution-of-repeatability-in- slides-102-bmwg-evolution-of-repeatability-in-
benchmarking-fraser-plugfest-summary-for-ietf-bmwg-00>. benchmarking-fraser-plugfest-summary-for-ietf-bmwg-00>.
[Y.1540] Y.1540, I. R., "Internet protocol data communication
service - IP packet transfer and availability performance
parameters", January 2020,
<https://www.itu.int/rec/T-REC-Y.1540-201103-I/en>.
Authors' Addresses Authors' Addresses
Al Morton Al Morton
AT&T Labs AT&T Labs
200 Laurel Avenue South 200 Laurel Avenue South
Middletown,, NJ 07748 Middletown,, NJ 07748
USA USA
Phone: +1 732 420 1571 Phone: +1 732 420 1571
Fax: +1 732 368 1192 Fax: +1 732 368 1192
 End of changes. 32 change blocks. 
63 lines changed or deleted 130 lines changed or added

This html diff was produced by rfcdiff 1.47. The latest version is available from http://tools.ietf.org/tools/rfcdiff/