draft-ietf-bmwg-dcbench-terminology-09.txt   draft-ietf-bmwg-dcbench-terminology-10.txt 
Internet Engineering Task Force L. Avramov Internet Engineering Task Force L. Avramov
INTERNET-DRAFT, Intended status: Informational Google INTERNET-DRAFT, Intended status: Informational Google
Expires: November 26,2017 J. Rapp Expires: November 26,2017 J. Rapp
May 25, 2017 VMware June 9, 2017 VMware
Data Center Benchmarking Terminology Data Center Benchmarking Terminology
draft-ietf-bmwg-dcbench-terminology-09 draft-ietf-bmwg-dcbench-terminology-10
Abstract Abstract
The purpose of this informational document is to establish definitions The purpose of this informational document is to establish definitions
and describe measurement techniques for data center benchmarking, as and describe measurement techniques for data center benchmarking, as
well as it is to introduce new terminologies applicable to data center well as it is to introduce new terminologies applicable to data center
performance evaluations. The purpose of this document is not to define performance evaluations. The purpose of this document is not to define
the test methodology, but rather establish the important concepts for the test methodology, but rather establish the important concepts for
benchmarking network switches and routers in the data center. benchmarking network switches and routers in the data center.
skipping to change at page 2, line 15 skipping to change at page 2, line 15
described in the Simplified BSD License. described in the Simplified BSD License.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 2 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1. Requirements Language . . . . . . . . . . . . . . . . . . 4 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 4
1.2. Definition format . . . . . . . . . . . . . . . . . . . . . 4 1.2. Definition format . . . . . . . . . . . . . . . . . . . . . 4
2. Latency . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2. Latency . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.1. Definition . . . . . . . . . . . . . . . . . . . . . . . . 4 2.1. Definition . . . . . . . . . . . . . . . . . . . . . . . . 4
2.2 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . 5 2.2 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.3 Measurement Units . . . . . . . . . . . . . . . . . . . . . 5 2.3 Measurement Units . . . . . . . . . . . . . . . . . . . . . 6
3 Jitter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 3 Jitter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
3.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . 6 3.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . 6
3.2 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . 6 3.2 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . 6
3.3 Measurement Units . . . . . . . . . . . . . . . . . . . . . 6 3.3 Measurement Units . . . . . . . . . . . . . . . . . . . . . 7
4 Physical Layer Calibration . . . . . . . . . . . . . . . . . . . 7 4 Physical Layer Calibration . . . . . . . . . . . . . . . . . . . 7
4.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . 7 4.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . 7
4.2 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . 7 4.2 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . 7
4.3 Measurement Units . . . . . . . . . . . . . . . . . . . . . 7 4.3 Measurement Units . . . . . . . . . . . . . . . . . . . . . 7
5 Line rate . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 5 Line rate . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
5.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . 8 5.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . 8
5.2 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . 8 5.2 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . 8
5.3 Measurement Units . . . . . . . . . . . . . . . . . . . . . 9 5.3 Measurement Units . . . . . . . . . . . . . . . . . . . . . 9
6 Buffering . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 6 Buffering . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
6.1 Buffer . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 6.1 Buffer . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
skipping to change at page 2, line 45 skipping to change at page 2, line 45
6.2.2 Discussion . . . . . . . . . . . . . . . . . . . . . . . 13 6.2.2 Discussion . . . . . . . . . . . . . . . . . . . . . . . 13
6.2.3 Measurement Units . . . . . . . . . . . . . . . . . . . 13 6.2.3 Measurement Units . . . . . . . . . . . . . . . . . . . 13
7 Application Throughput: Data Center Goodput . . . . . . . . . . 14 7 Application Throughput: Data Center Goodput . . . . . . . . . . 14
7.1. Definition . . . . . . . . . . . . . . . . . . . . . . . . 14 7.1. Definition . . . . . . . . . . . . . . . . . . . . . . . . 14
7.2. Discussion . . . . . . . . . . . . . . . . . . . . . . . . 14 7.2. Discussion . . . . . . . . . . . . . . . . . . . . . . . . 14
7.3. Measurement Units . . . . . . . . . . . . . . . . . . . . . 14 7.3. Measurement Units . . . . . . . . . . . . . . . . . . . . . 14
8. Security Considerations . . . . . . . . . . . . . . . . . . . 15 8. Security Considerations . . . . . . . . . . . . . . . . . . . 15
9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 15 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 15
10. References . . . . . . . . . . . . . . . . . . . . . . . . . 15 10. References . . . . . . . . . . . . . . . . . . . . . . . . . 15
10.1. Normative References . . . . . . . . . . . . . . . . . . 15 10.1. Normative References . . . . . . . . . . . . . . . . . . 15
10.2. Informative References . . . . . . . . . . . . . . . . . 15 10.2. Informative References . . . . . . . . . . . . . . . . . 16
10.3. Acknowledgments . . . . . . . . . . . . . . . . . . . . . 16 10.3. Acknowledgments . . . . . . . . . . . . . . . . . . . . . 16
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 16 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 16
1. Introduction 1. Introduction
Traffic patterns in the data center are not uniform and are Traffic patterns in the data center are not uniform and are
constantly changing. They are dictated by the nature and variety of constantly changing. They are dictated by the nature and variety of
applications utilized in the data center. It can be largely east-west applications utilized in the data center. It can be largely east-west
traffic flows in one data center and north-south in another, while traffic flows in one data center and north-south in another, while
some may combine both. Traffic patterns can be bursty in nature and some may combine both. Traffic patterns can be bursty in nature and
contain many-to-one, many-to-many, or one-to-many flows. Each flow contain many-to-one, many-to-many, or one-to-many flows. Each flow
skipping to change at page 4, line 28 skipping to change at page 4, line 28
any restrictions on measurement procedures. any restrictions on measurement procedures.
Measurement Units: Methodology for the measure and units used to Measurement Units: Methodology for the measure and units used to
report measurements of this term, if applicable. report measurements of this term, if applicable.
2. Latency 2. Latency
2.1. Definition 2.1. Definition
Latency is a the amount of time it takes a frame to transit the DUT. Latency is a the amount of time it takes a frame to transit the DUT.
Latency is measured in unit of time (seconds, milliseconds, Latency is measured in units of time (seconds, milliseconds,
microseconds and so on). The purpose of measuring latency is to microseconds and so on). The purpose of measuring latency is to
understand the impact of adding a device in the communication path. understand the impact of adding a device in the communication path.
The Latency interval can be assessed between different combinations The Latency interval can be assessed between different combinations
of events, regardless of the type of switching device (bit forwarding of events, regardless of the type of switching device (bit forwarding
aka cut-through or store forward type of device) aka cut-through or store-and-forward type of device)
Traditionally the latency measurement definitions are: Traditionally the latency measurement definitions are:
FILO (First In Last Out) The time interval starting when the end of FILO (First In Last Out) The time interval starting when the end of
the first bit of the input frame reaches the input port and ending the first bit of the input frame reaches the input port and ending
when the last bit of the output frame is seen on the output port when the last bit of the output frame is seen on the output port
FIFO (First In First Out) The time interval starting when the end of FIFO (First In First Out) The time interval starting when the end of
the first bit of the input frame reaches the input port and ending the first bit of the input frame reaches the input port and ending
when the start of the first bit of the output frame is seen on the when the start of the first bit of the output frame is seen on the
output port output port
LILO (Last In Last Out) The time interval starting when the last bit LILO (Last In Last Out) The time interval starting when the last bit
of the input frame reaches the input port and the last bit of the of the input frame reaches the input port and the last bit of the
output frame is seen on the output port output frame is seen on the output port
LIFO (Last In First Out) The time interval starting when the last bit LIFO (Last In First Out) The time interval starting when the last bit
of the input frame reaches the input port and ending when the first of the input frame reaches the input port and ending when the first
bit of the output frame is seen on the output port. bit of the output frame is seen on the output port.
Another possibility to summarize the four different definitions above Another possibility to summarize the four different definitions above
is to refer to the bit position as they normally occur: input to is to refer to the bit position as they normally occur: Input to
output. output.
FILO is FL (First bit Last bit). FIFO is FF (First bit First bit). FILO is FL (First bit Last bit). FIFO is FF (First bit First bit).
LILO is LL (Last bit Last bit). LIFO is LF (Last bit First bit). LILO is LL (Last bit Last bit). LIFO is LF (Last bit First bit).
This definition explained in this section in context of data center This definition explained in this section in context of data center
switching benchmarking is in lieu of the previous definition of switching benchmarking is in lieu of the previous definition of
Latency defined in RFC 1242, section 3.8 and is quoted here: Latency defined in RFC 1242, section 3.8 and is quoted here:
For store and forward devices: The time interval starting when the For store and forward devices: The time interval starting when the
skipping to change at page 5, line 40 skipping to change at page 5, line 40
FILO is the most important measuring definition. All type of switches FILO is the most important measuring definition. All type of switches
MUST be measured with the FILO mechanism: FILO will include the MUST be measured with the FILO mechanism: FILO will include the
latency of the switch and the latency of the frame as well as the latency of the switch and the latency of the frame as well as the
serialization delay. It is a picture of the 'whole' latency going serialization delay. It is a picture of the 'whole' latency going
through the DUT. For applications, which are latency sensitive and through the DUT. For applications, which are latency sensitive and
can function with initial bytes of the frame, FIFO MAY be an can function with initial bytes of the frame, FIFO MAY be an
additional type of measuring to supplement FILO. additional type of measuring to supplement FILO.
Not all DUTs are exclusively cut-through or store-and-forward. Data Not all DUTs are exclusively cut-through or store-and-forward. Data
Center DUTs are frequently store-and-forward for smaller packet sizes Center DUTs are frequently store-and-forward for smaller packet sizes
and then adopting a cut-through behavior. FILO covers all scenarios. and then adopting a cut-through behavior. The change of behavior
happens for at specific larger packet sizes. The value of the packet
size for the behavior to change MAY be configurable depending on the
DUT manufacturer. FILO covers all scenarios: Store-and-forward or
cut-through. The threshold of behavior change does not matter for
benchmarking since FILO covers both possible scenarios.
LIFO mechanism can be used with store forward type of switches but LIFO mechanism can be used with store forward type of switches but
not with cut-through type of switches, as it will provide negative not with cut-through type of switches, as it will provide negative
latency values for larger packet sizes because LIFO removes the latency values for larger packet sizes because LIFO removes the
serialization delay. Therefore, this mechanism MUST NOT be used when serialization delay. Therefore, this mechanism MUST NOT be used when
comparing latencies of two different DUTs. comparing latencies of two different DUTs.
2.3 Measurement Units 2.3 Measurement Units
The measuring methods to use for benchmarking purposes are as follow:
The measuring methods to use for benchmarking purposes are as
follows:
1) FILO MUST be used as a measuring method, as this will include the 1) FILO MUST be used as a measuring method, as this will include the
latency of the packet; and today the application commonly need to latency of the packet; and today the application commonly need to
read the whole packet to process the information and take an action. read the whole packet to process the information and take an action.
2) FIFO MAY be used for certain applications able to proceed data as 2) FIFO MAY be used for certain applications able to proceed the data
the first bits arrive (FPGA for example) as the first bits arrive (FPGA for example)
3) LIFO MUST NOT be used, because it subtracts the latency of the 3) LIFO MUST NOT be used, because it subtracts the latency of the
packet; unlike all the other methods. packet; unlike all the other methods.
3 Jitter 3 Jitter
3.1 Definition 3.1 Definition
Jitter in the data center context is synonymous with the common term Jitter in the data center context is synonymous with the common term
Delay variation. It is derived from multiple measurements of one-way Delay variation. It is derived from multiple measurements of one-way
skipping to change at page 7, line 11 skipping to change at page 7, line 16
The measurement of delay variation is expressed in units of seconds. The measurement of delay variation is expressed in units of seconds.
A PDV histogram MAY be provided for the population of packets A PDV histogram MAY be provided for the population of packets
measured. measured.
4 Physical Layer Calibration 4 Physical Layer Calibration
4.1 Definition 4.1 Definition
The calibration of the physical layer consists of defining and The calibration of the physical layer consists of defining and
measuring the latency of the physical devices used to perform test on measuring the latency of the physical devices used to perform tests
the DUT. on the DUT.
It includes the list of all physical layer components used as listed It includes the list of all physical layer components used as listed
here after: here after:
-type of device used to generate traffic / measure traffic -Type of device used to generate traffic / measure traffic
-type of line cards used on the traffic generator -Type of line cards used on the traffic generator
-type of transceivers on traffic generator -Type of transceivers on traffic generator
-type of transceivers on DUT -Type of transceivers on DUT
-type of cables -Type of cables
-length of cables -Length of cables
-software name, and version of traffic generator and DUT -Software name, and version of traffic generator and DUT
-list of enabled features on DUT MAY be provided and is recommended -List of enabled features on DUT MAY be provided and is recommended
[especially the control plane protocols such as LLDP, Spanning-Tree [especially the control plane protocols such as LLDP, Spanning-Tree
etc.]. A comprehensive configuration file MAY be provided to this etc.]. A comprehensive configuration file MAY be provided to this
effect. effect.
4.2 Discussion 4.2 Discussion
Physical layer calibration is part of the end to end latency, which Physical layer calibration is part of the end to end latency, which
should be taken into acknowledgment while evaluating the DUT. Small should be taken into acknowledgment while evaluating the DUT. Small
variations of the physical components of the test may impact the variations of the physical components of the test may impact the
latency being measured, therefore they MUST be described when latency being measured, therefore they MUST be described when
presenting results. presenting results.
4.3 Measurement Units 4.3 Measurement Units
It is RECOMMENDED to use all cables of: The same type, the same
It is RECOMMENDED to use all cables of : the same type, the same
length, when possible using the same vendor. It is a MUST to document length, when possible using the same vendor. It is a MUST to document
the cables specifications on section [4.1s] along with the test the cables specifications on section 4.1 along with the test results.
results. The test report MUST specify if the cable latency has been The test report MUST specify if the cable latency has been removed
removed from the test measures or not. The accuracy of the traffic from the test measures or not. The accuracy of the traffic generator
generator measure MUST be provided [this is usually a value in the measure MUST be provided [this is usually a value in the 20ns range
20ns range for current test equipment]. for current test equipment].
5 Line rate 5 Line rate
5.1 Definition 5.1 Definition
The transmit timing, or maximum transmitted data rate is controlled The transmit timing, or maximum transmitted data rate is controlled
by the "transmit clock" in the DUT. The receive timing (maximum by the "transmit clock" in the DUT. The receive timing (maximum
ingress data rate) is derived from the transmit clock of the ingress data rate) is derived from the transmit clock of the
connected interface. connected interface.
skipping to change at page 9, line 45 skipping to change at page 9, line 48
Test set equipment manufacturers are well-aware of the standards, and Test set equipment manufacturers are well-aware of the standards, and
allow a software-controlled +/- 100 PPM "offset" (clock-rate allow a software-controlled +/- 100 PPM "offset" (clock-rate
adjustment) to compensate for normal variations in the clock speed of adjustment) to compensate for normal variations in the clock speed of
"devices under test". This offset adjustment allows engineers to "devices under test". This offset adjustment allows engineers to
determine the approximate speed the connected device is operating, determine the approximate speed the connected device is operating,
and verify that it is within parameters allowed by standards. and verify that it is within parameters allowed by standards.
5.3 Measurement Units 5.3 Measurement Units
"Line Rate" CAN be measured in terms of "Frame Rate": "Line Rate" can be measured in terms of "Frame Rate":
Frame Rate = Transmit-Clock-Frequency / (Frame-Length*8 + Minimum_Gap Frame Rate = Transmit-Clock-Frequency / (Frame-Length*8 + Minimum_Gap
+ Preamble + Start-Frame Delimiter) + Preamble + Start-Frame Delimiter)
Minimum_Gap represents the inter frame gap. This formula "scales up" Minimum_Gap represents the inter frame gap. This formula "scales up"
or "scales down" to represent 1 GB Ethernet, or 10 GB Ethernet and so or "scales down" to represent 1 GB Ethernet, or 10 GB Ethernet and so
on. on.
Example for 1 GB Ethernet speed with 64-byte frames: Frame Rate = Example for 1 GB Ethernet speed with 64-byte frames: Frame Rate =
1,000,000,000 /(64*8 + 96 + 56 + 8) Frame Rate = 1,000,000,000 / 672 1,000,000,000 /(64*8 + 96 + 56 + 8) Frame Rate = 1,000,000,000 / 672
Frame Rate = 1,488,095.2 frames per second. Frame Rate = 1,488,095.2 frames per second.
Considering the allowance of +/- 100 PPM, a switch may "legally" Considering the allowance of +/- 100 PPM, a switch may "legally"
transmit traffic at a frame rate between 1,487,946.4 FPS and transmit traffic at a frame rate between 1,487,946.4 FPS and
skipping to change at page 10, line 21 skipping to change at page 10, line 25
Considering the allowance of +/- 100 PPM, a switch may "legally" Considering the allowance of +/- 100 PPM, a switch may "legally"
transmit traffic at a frame rate between 1,487,946.4 FPS and transmit traffic at a frame rate between 1,487,946.4 FPS and
1,488,244 FPS. Each 1 PPM variation in clock rate will translate to 1,488,244 FPS. Each 1 PPM variation in clock rate will translate to
a 1.488 frame-per-second frame rate increase or decrease. a 1.488 frame-per-second frame rate increase or decrease.
In a production network, it is very unlikely to see precise line rate In a production network, it is very unlikely to see precise line rate
over a very brief period. There is no observable difference between over a very brief period. There is no observable difference between
dropping packets at 99% of line rate and 100% of line rate. dropping packets at 99% of line rate and 100% of line rate.
Line rate CAN measured at 100% of line rate with a -100PPM Line rate can be measured at 100% of line rate with a -100PPM
adjustment. adjustment.
Line rate SHOULD be measured at 99,98% with 0 PPM adjustment. Line rate SHOULD be measured at 99,98% with 0 PPM adjustment.
The PPM adjustment SHOULD only be used for a line rate type of The PPM adjustment SHOULD only be used for a line rate type of
measurement. measurement.
6 Buffering 6 Buffering
6.1 Buffer 6.1 Buffer
6.1.1 Definition 6.1.1 Definition
Buffer Size: the term buffer size represents the total amount of Buffer Size: The term buffer size represents the total amount of
frame buffering memory available on a DUT. This size is expressed in frame buffering memory available on a DUT. This size is expressed in
Byte; KB (kilobytes), MB (megabytes) or GB (gigabyte). When the B (bytes); KB (kilobytes), MB (megabytes) or GB (gigabyte). When the
buffer size is expressed it SHOULD be defined by a size metric stated buffer size is expressed it SHOULD be defined by a size metric stated
above. When the buffer size is expressed, an indication of the frame above. When the buffer size is expressed, an indication of the frame
MTU used for that measurement is also necessary as well as the cos or MTU used for that measurement is also necessary as well as the cos or
dscp value set; as often times the buffers are carved by quality of dscp value set; as often times the buffers are carved by quality of
service implementation. Please refer to the buffer efficiency section service implementation. Please refer to the buffer efficiency section
for further details. for further details.
Example: Buffer Size of DUT when sending 1518 bytes frames is 18 Mb. Example: Buffer Size of DUT when sending 1518 bytes frames is 18 Mb.
Port Buffer Size: the port buffer size is the amount of buffer a Port Buffer Size: The port buffer size is the amount of buffer for a
single ingress port, egress port or combination of ingress and egress single ingress port, egress port or combination of ingress and egress
buffering location for a single port. The reason for mentioning the buffering location for a single port. The reason for mentioning the
three locations for the port buffer is because the DUT buffering three locations for the port buffer is because the DUT buffering
scheme can be unknown or untested, and so knowing the buffer location scheme can be unknown or untested, and so knowing the buffer location
helps clarify the buffer architecture and consequently the total helps clarify the buffer architecture and consequently the total
buffer size. The Port Buffer Size is an informational value that MAY buffer size. The Port Buffer Size is an informational value that MAY
be provided from the DUT vendor. It is not a value that is tested by be provided from the DUT vendor. It is not a value that is tested by
benchmarking. Benchmarking will be done using the Maximum Port Buffer benchmarking. Benchmarking will be done using the Maximum Port Buffer
Size or Maximum Buffer Size methodology. Size or Maximum Buffer Size methodology.
Maximum Port Buffer Size: in most cases, this is the same as the Port Maximum Port Buffer Size: In most cases, this is the same as the Port
Buffer Size. In certain switch architecture called SoC (switch on Buffer Size. In certain switch architecture called SoC (switch on
chip), there is a port buffer and a shared buffer pool available for chip), there is a port buffer and a shared buffer pool available for
all ports. The Maximum Port Buffer Size , in terms of an SoC buffer, all ports. The Maximum Port Buffer Size , in terms of an SoC buffer,
represents the sum of the port buffer and the maximum value of shared represents the sum of the port buffer and the maximum value of shared
buffer allowed for this port, defined in terms of B (byte), KB buffer allowed for this port, defined in terms of B (byte), KB
(kilobyte), MB (megabyte), or GB (gigabyte). The Maximum Port Buffer (kilobyte), MB (megabyte), or GB (gigabyte). The Maximum Port Buffer
Size needs to be expressed along with the frame MTU used for the Size needs to be expressed along with the frame MTU used for the
measurement and the cos or dscp bit value set for the test. measurement and the cos or dscp bit value set for the test.
Example: a DUT has been measured to have 3KB of port buffer for 1518 Example: A DUT has been measured to have 3KB of port buffer for 1518
frame size packets and a total of 4.7 MB of maximum port buffer for frame size packets and a total of 4.7 MB of maximum port buffer for
1518 frame size packets and a cos of 0. 1518 frame size packets and a cos of 0.
Maximum DUT Buffer Size: this is the total size of Buffer a DUT can Maximum DUT Buffer Size: This is the total size of Buffer a DUT can
be measured to have. It is, most likely, different than than the be measured to have. It is, most likely, different than than the
Maximum Port Buffer Size. It CAN also be different from the sum of Maximum Port Buffer Size. It can also be different from the sum of
Maximum Port Buffer Size. The Maximum Buffer Size needs to be Maximum Port Buffer Size. The Maximum Buffer Size needs to be
expressed along with the frame MTU used for the measurement and along expressed along with the frame MTU used for the measurement and along
with the cos or dscp value set during the test. with the cos or dscp value set during the test.
Example: a DUT has been measured to have 3KB of port buffer for 1518 Example: A DUT has been measured to have 3KB of port buffer for 1518
frame size packets and a total of 4.7 MB of maximum port buffer for frame size packets and a total of 4.7 MB of maximum port buffer for
1518 frame size packets. The DUT has a Maximum Buffer Size of 18 MB 1518 B frame size packets. The DUT has a Maximum Buffer Size of 18 MB
at 1500 bytes and a cos of 0. at 1500 B and a cos of 0.
Burst: The burst is a fixed number of packets sent over a percentage Burst: The burst is a fixed number of packets sent over a percentage
of linerate of a defined port speed. The amount of frames sent are of linerate of a defined port speed. The amount of frames sent are
evenly distributed across the interval, T. A constant, C, can be evenly distributed across the interval, T. A constant, C, can be
defined to provide the average time between two consecutive packets defined to provide the average time between two consecutive packets
evenly spaced. evenly spaced.
Microburst: it is a burst. A microburst is when packet drops occur Microburst: It is a burst. A microburst is when packet drops occur
when there is not sustained or noticeable congestion upon a link or when there is not sustained or noticeable congestion upon a link or
device. A characterization of microburst is when the Burst is not device. A characterization of microburst is when the Burst is not
evenly distributed over T, and is less than the constant C [C= evenly distributed over T, and is less than the constant C [C=
average time between two consecutive packets evenly spaced out]. average time between two consecutive packets evenly spaced out].
Intensity of Microburst: this is a percentage, representing the level Intensity of Microburst: This is a percentage, representing the level
of microburst between 1 and 100%. The higher the number the higher of microburst between 1 and 100%. The higher the number the higher
the microburst is. I=[1-[ (TP2-Tp1)+(Tp3-Tp2)+....(TpN-Tp(n-1) ] / the microburst is. I=[1-[ (TP2-Tp1)+(Tp3-Tp2)+....(TpN-Tp(n-1) ] /
Sum(packets)]]*100 Sum(packets)]]*100
The above definitions are not meant to comment on the ideal sizing of The above definitions are not meant to comment on the ideal sizing of
a buffer, rather on how to measure it. A larger buffer is not a buffer, rather on how to measure it. A larger buffer is not
necessarily better and CAN cause issues with buffer bloat. necessarily better and can cause issues with buffer bloat.
6.1.2 Discussion 6.1.2 Discussion
When measuring buffering on a DUT, it is important to understand the When measuring buffering on a DUT, it is important to understand the
behavior for each and all ports. This provides data for the total behavior for each and all ports. This provides data for the total
amount of buffering available on the switch. The terms of buffer amount of buffering available on the switch. The terms of buffer
efficiency here helps one understand the optimum packet size for the efficiency here helps one understand the optimum packet size for the
buffer, or the real volume of the buffer available for a specific buffer, or the real volume of the buffer available for a specific
packet size. This section does not discuss how to conduct the test packet size. This section does not discuss how to conduct the test
methodology; instead, it explains the buffer definitions and what methodology; instead, it explains the buffer definitions and what
metrics should be provided for a comprehensive data center device metrics should be provided for a comprehensive data center device
buffering benchmarking. buffering benchmarking.
6.1.3 Measurement Units 6.1.3 Measurement Units
When Buffer is measured: When Buffer is measured:
-the buffer size MUST be measured -The buffer size MUST be measured
-the port buffer size MAY be provided for each port -The port buffer size MAY be provided for each port
-the maximum port buffer size MUST be measured -The maximum port buffer size MUST be measured
-the maximum DUT buffer size MUST be measured -The maximum DUT buffer size MUST be measured
-the intensity of microburst MAY be mentioned when a microburst test -The intensity of microburst MAY be mentioned when a microburst test
is performed is performed
-the cos or dscp value set during the test SHOULD be provided -The cos or dscp value set during the test SHOULD be provided
6.2 Incast 6.2 Incast
6.2.1 Definition 6.2.1 Definition
The term Incast, very commonly utilized in the data center, refers to The term Incast, very commonly utilized in the data center, refers to
the traffic pattern of many-to-one or many-to-many conversations. the traffic pattern of many-to-one or many-to-many conversations.
Typically in the data center it would refer to many different ingress Typically in the data center it would refer to many different ingress
server ports (many), sending traffic to a common uplink (one), or server ports (many), sending traffic to a common uplink (one), or
multiple uplinks (many). This pattern is generalized for any network multiple uplinks (many). This pattern is generalized for any network
as many incoming ports sending traffic to one or few uplinks. It can as many incoming ports sending traffic to one or few uplinks. It can
also be found in many-to-many traffic patterns. also be found in many-to-many traffic patterns.
Synchronous arrival time: When two, or more, frames of respective Synchronous arrival time: When two, or more, frames of respective
sizes L1 and L2 arrive at their respective one or multiple ingress sizes L1 and L2 arrive at their respective one or multiple ingress
ports, and there is an overlap of the arrival time for any of the ports, and there is an overlap of the arrival time for any of the
bits on the DUT, then the frames L1 and L2 have a synchronous arrival bits on the DUT, then the frames L1 and L2 have a synchronous arrival
times. This is called incast. times. This is called incast.
Asynchronous arrival time: Any condition not defined by synchronous. Asynchronous arrival time: Any condition not defined by synchronous
arrival time.
Percentage of synchronization: this defines the level of overlap Percentage of synchronization: This defines the level of overlap
[amount of bits] between the frames L1,L2..Ln. [amount of bits] between the frames L1,L2..Ln.
Example: two 64 bytes frames, of length L1 and L2, arrive to ingress Example: Two 64 bytes frames, of length L1 and L2, arrive to ingress
port 1 and port 2 of the DUT. There is an overlap of 6.4 bytes port 1 and port 2 of the DUT. There is an overlap of 6.4 bytes
between the two where L1 and L2 were at the same time on the between the two where L1 and L2 were at the same time on the
respective ingress ports. Therefore the percentage of synchronization respective ingress ports. Therefore the percentage of synchronization
is 10%. is 10%.
Stateful type traffic defines packets exchanged with a stateful Stateful type traffic defines packets exchanged with a stateful
protocol such as for example TCP. protocol such as for example TCP.
Stateless type traffic defines packets exchanged with a stateless Stateless type traffic defines packets exchanged with a stateless
protocol such as for example UDP. protocol such as for example UDP.
6.2.2 Discussion 6.2.2 Discussion
In this scenario, buffers are solicited on the DUT. In a ingress In this scenario, buffers are solicited on the DUT. In an ingress
buffering mechanism, the ingress port buffers would be solicited buffering mechanism, the ingress port buffers would be solicited
along with Virtual Output Queues, when available; whereas in an along with Virtual Output Queues, when available; whereas in an
egress buffer mechanism, the egress buffer of the one outgoing port egress buffer mechanism, the egress buffer of the one outgoing port
would be used. would be used.
In either cases, regardless of where the buffer memory is located on In either case, regardless of where the buffer memory is located on
the switch architecture, the Incast creates buffer utilization. the switch architecture, the Incast creates buffer utilization.
When one or more frames having synchronous arrival times at the DUT When one or more frames having synchronous arrival times at the DUT
they are considered forming an Incast. they are considered forming an Incast.
6.2.3 Measurement Units 6.2.3 Measurement Units
It is a MUST to measure the number of ingress and egress ports. It is It is a MUST to measure the number of ingress and egress ports. It is
a MUST to have a non null percentage of synchronization, which MUST a MUST to have a non-null percentage of synchronization, which MUST
be specified. be specified.
7 Application Throughput: Data Center Goodput 7 Application Throughput: Data Center Goodput
7.1. Definition 7.1. Definition
In Data Center Networking, a balanced network is a function of In Data Center Networking, a balanced network is a function of
maximal throughput 'and' minimal loss at any given time. This is maximal throughput 'and' minimal loss at any given time. This is
defined by the Goodput [4]. Goodput is the application-level defined by the Goodput [4]. Goodput is the application-level
throughput. The definition used is a variance of the definition in throughput. The definition used is a variance of the definition in
skipping to change at page 14, line 28 skipping to change at page 14, line 29
7.2. Discussion 7.2. Discussion
In data center benchmarking, the goodput is a value that SHOULD be In data center benchmarking, the goodput is a value that SHOULD be
measured. It provides a realistic idea of the usage of the available measured. It provides a realistic idea of the usage of the available
bandwidth. A goal in data center environments is to maximize the bandwidth. A goal in data center environments is to maximize the
goodput while minimizing the loss. goodput while minimizing the loss.
7.3. Measurement Units 7.3. Measurement Units
When S is the total bytes received from all senders [not inclusive of When S represents the payload bytes, which does not include packet or
packet headers or TCP headers - it's the payload] and Ft is the TCP headers, and Ft is the Finishing Time of the last sender, the
Finishing Time of the last sender; the Goodput G is then measured by Goodput, G, is then measured by the following formula:
the following formula: G=S/Ft bytes per second
Example: a TCP file transfer over HTTP protocol on a 10Gb/s media. G=S/Ft bytes per second
Example: A TCP file transfer over HTTP protocol on a 10Gb/s media.
The file cannot be transferred over Ethernet as a single continuous The file cannot be transferred over Ethernet as a single continuous
stream. It must be broken down into individual frames of 1500 bytes stream. It must be broken down into individual frames of 1500 bytes
when the standard MTU [Maximum Transmission Unit] is used. Each when the standard MTU [Maximum Transmission Unit] is used. Each
packet requires 20 bytes of IP header information and 20 bytes of TCP packet requires 20 bytes of IP header information and 20 bytes of TCP
header information; therefore 1460 byte are available per packet for header information; therefore 1460 byte are available per packet for
the file transfer. Linux based systems are further limited to 1448 the file transfer. Linux based systems are further limited to 1448
bytes as they also carry a 12 byte timestamp. Finally, the date is bytes as they also carry a 12 byte timestamp. Finally, the date is
transmitted in this example over Ethernet which adds a 26 byte transmitted in this example over Ethernet which adds a 26 byte
overhead per packet. overhead per packet.
G= 1460/1526 x 10 Gbit/s which is 9.567 Gbit/s or 1.196 Gigabytes per G= 1460/1526 x 10 Gbit/s which is 9.567 Gbit/s or 1.196 Gigabytes per
second. second.
Please note: this example does not take into consideration additional Please note: This example does not take into consideration additional
Ethernet overhead, such as the interframe gap (a minimum of 96 bit Ethernet overhead, such as the interframe gap (a minimum of 96 bit
times), nor collisions (which have a variable impact, depending on times), nor collisions (which have a variable impact, depending on
the network load). the network load).
When conducting Goodput measurements please document in addition to When conducting Goodput measurements please document in addition to
the 4.1 section: the 4.1 section:
-the TCP Stack used -The TCP Stack used
-OS Versions -OS Versions
-NIC firmware version and model -NIC firmware version and model
For example, Windows TCP stacks and different Linux versions can For example, Windows TCP stacks and different Linux versions can
influence TCP based tests results. influence TCP based tests results.
8. Security Considerations 8. Security Considerations
 End of changes. 53 change blocks. 
63 lines changed or deleted 72 lines changed or added

This html diff was produced by rfcdiff 1.45. The latest version is available from http://tools.ietf.org/tools/rfcdiff/