draft-ietf-bmwg-ngfw-performance-02.txt   draft-ietf-bmwg-ngfw-performance-03.txt 
Benchmarking Methodology Working Group B. Balarajah Benchmarking Methodology Working Group B. Balarajah
Internet-Draft Internet-Draft
Intended status: Informational C. Rossenhoevel Intended status: Informational C. Rossenhoevel
Expires: May 22, 2020 EANTC AG Expires: September 10, 2020 EANTC AG
B. Monkman B. Monkman
NetSecOPEN NetSecOPEN
November 19, 2019 March 9, 2020
Benchmarking Methodology for Network Security Device Performance Benchmarking Methodology for Network Security Device Performance
draft-ietf-bmwg-ngfw-performance-02 draft-ietf-bmwg-ngfw-performance-03
Abstract Abstract
This document provides benchmarking terminology and methodology for This document provides benchmarking terminology and methodology for
next-generation network security devices including next-generation next-generation network security devices including next-generation
firewalls (NGFW), intrusion detection and prevention solutions (IDS/ firewalls (NGFW), intrusion detection and prevention solutions (IDS/
IPS) and unified threat management (UTM) implementations. This IPS) and unified threat management (UTM) implementations. This
document aims to strongly improve the applicability, reproducibility, document aims to strongly improve the applicability, reproducibility,
and transparency of benchmarks and to align the test methodology with and transparency of benchmarks and to align the test methodology with
today's increasingly complex layer 7 application use cases. The main today's increasingly complex layer 7 application use cases. The main
skipping to change at page 1, line 41 skipping to change at page 1, line 41
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/. Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on May 22, 2020. This Internet-Draft will expire on September 10, 2020.
Copyright Notice Copyright Notice
Copyright (c) 2019 IETF Trust and the persons identified as the Copyright (c) 2020 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(https://trustee.ietf.org/license-info) in effect on the date of (https://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as the Trust Legal Provisions and are provided without warranty as
skipping to change at page 2, line 21 skipping to change at page 2, line 21
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3
2. Requirements . . . . . . . . . . . . . . . . . . . . . . . . 4 2. Requirements . . . . . . . . . . . . . . . . . . . . . . . . 4
3. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 3. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
4. Test Setup . . . . . . . . . . . . . . . . . . . . . . . . . 4 4. Test Setup . . . . . . . . . . . . . . . . . . . . . . . . . 4
4.1. Testbed Configuration . . . . . . . . . . . . . . . . . . 4 4.1. Testbed Configuration . . . . . . . . . . . . . . . . . . 4
4.2. DUT/SUT Configuration . . . . . . . . . . . . . . . . . . 5 4.2. DUT/SUT Configuration . . . . . . . . . . . . . . . . . . 5
4.3. Test Equipment Configuration . . . . . . . . . . . . . . 9 4.3. Test Equipment Configuration . . . . . . . . . . . . . . 9
4.3.1. Client Configuration . . . . . . . . . . . . . . . . 9 4.3.1. Client Configuration . . . . . . . . . . . . . . . . 10
4.3.2. Backend Server Configuration . . . . . . . . . . . . 11 4.3.2. Backend Server Configuration . . . . . . . . . . . . 11
4.3.3. Traffic Flow Definition . . . . . . . . . . . . . . . 11 4.3.3. Traffic Flow Definition . . . . . . . . . . . . . . . 12
4.3.4. Traffic Load Profile . . . . . . . . . . . . . . . . 12 4.3.4. Traffic Load Profile . . . . . . . . . . . . . . . . 13
5. Test Bed Considerations . . . . . . . . . . . . . . . . . . . 13 5. Test Bed Considerations . . . . . . . . . . . . . . . . . . . 14
6. Reporting . . . . . . . . . . . . . . . . . . . . . . . . . . 14 6. Reporting . . . . . . . . . . . . . . . . . . . . . . . . . . 15
6.1. Key Performance Indicators . . . . . . . . . . . . . . . 15 6.1. Key Performance Indicators . . . . . . . . . . . . . . . 16
7. Benchmarking Tests . . . . . . . . . . . . . . . . . . . . . 16 7. Benchmarking Tests . . . . . . . . . . . . . . . . . . . . . 17
7.1. Throughput Performance With NetSecOPEN Traffic Mix . . . 16 7.1. Throughput Performance With NetSecOPEN Traffic Mix . . . 17
7.1.1. Objective . . . . . . . . . . . . . . . . . . . . . . 16 7.1.1. Objective . . . . . . . . . . . . . . . . . . . . . . 17
7.1.2. Test Setup . . . . . . . . . . . . . . . . . . . . . 17 7.1.2. Test Setup . . . . . . . . . . . . . . . . . . . . . 18
7.1.3. Test Parameters . . . . . . . . . . . . . . . . . . . 17 7.1.3. Test Parameters . . . . . . . . . . . . . . . . . . . 18
7.1.4. Test Procedures and expected Results . . . . . . . . 19 7.1.4. Test Procedures and expected Results . . . . . . . . 20
7.2. TCP/HTTP Connections Per Second . . . . . . . . . . . . . 20 7.2. TCP/HTTP Connections Per Second . . . . . . . . . . . . . 21
7.2.1. Objective . . . . . . . . . . . . . . . . . . . . . . 20 7.2.1. Objective . . . . . . . . . . . . . . . . . . . . . . 21
7.2.2. Test Setup . . . . . . . . . . . . . . . . . . . . . 20 7.2.2. Test Setup . . . . . . . . . . . . . . . . . . . . . 21
7.2.3. Test Parameters . . . . . . . . . . . . . . . . . . . 20 7.2.3. Test Parameters . . . . . . . . . . . . . . . . . . . 21
7.2.4. Test Procedures and Expected Results . . . . . . . . 22 7.2.4. Test Procedures and Expected Results . . . . . . . . 22
7.3. HTTP Throughput . . . . . . . . . . . . . . . . . . . . . 23 7.3. HTTP Throughput . . . . . . . . . . . . . . . . . . . . . 24
7.3.1. Objective . . . . . . . . . . . . . . . . . . . . . . 23 7.3.1. Objective . . . . . . . . . . . . . . . . . . . . . . 24
7.3.2. Test Setup . . . . . . . . . . . . . . . . . . . . . 23 7.3.2. Test Setup . . . . . . . . . . . . . . . . . . . . . 24
7.3.3. Test Parameters . . . . . . . . . . . . . . . . . . . 23 7.3.3. Test Parameters . . . . . . . . . . . . . . . . . . . 24
7.3.4. Test Procedures and Expected Results . . . . . . . . 25 7.3.4. Test Procedures and Expected Results . . . . . . . . 26
7.4. TCP/HTTP Transaction Latency . . . . . . . . . . . . . . 26 7.4. TCP/HTTP Transaction Latency . . . . . . . . . . . . . . 27
7.4.1. Objective . . . . . . . . . . . . . . . . . . . . . . 26 7.4.1. Objective . . . . . . . . . . . . . . . . . . . . . . 27
7.4.2. Test Setup . . . . . . . . . . . . . . . . . . . . . 26 7.4.2. Test Setup . . . . . . . . . . . . . . . . . . . . . 27
7.4.3. Test Parameters . . . . . . . . . . . . . . . . . . . 26 7.4.3. Test Parameters . . . . . . . . . . . . . . . . . . . 27
7.4.4. Test Procedures and Expected Results . . . . . . . . 28 7.4.4. Test Procedures and Expected Results . . . . . . . . 29
7.5. Concurrent TCP/HTTP Connection Capacity . . . . . . . . . 29 7.5. Concurrent TCP/HTTP Connection Capacity . . . . . . . . . 30
7.5.1. Objective . . . . . . . . . . . . . . . . . . . . . . 29 7.5.1. Objective . . . . . . . . . . . . . . . . . . . . . . 30
7.5.2. Test Setup . . . . . . . . . . . . . . . . . . . . . 30 7.5.2. Test Setup . . . . . . . . . . . . . . . . . . . . . 31
7.5.3. Test Parameters . . . . . . . . . . . . . . . . . . . 30 7.5.3. Test Parameters . . . . . . . . . . . . . . . . . . . 31
7.5.4. Test Procedures and expected Results . . . . . . . . 31 7.5.4. Test Procedures and expected Results . . . . . . . . 32
7.6. TCP/HTTPS Connections per second . . . . . . . . . . . . 32 7.6. TCP/HTTPS Connections per second . . . . . . . . . . . . 33
7.6.1. Objective . . . . . . . . . . . . . . . . . . . . . . 32 7.6.1. Objective . . . . . . . . . . . . . . . . . . . . . . 33
7.6.2. Test Setup . . . . . . . . . . . . . . . . . . . . . 33 7.6.2. Test Setup . . . . . . . . . . . . . . . . . . . . . 34
7.6.3. Test Parameters . . . . . . . . . . . . . . . . . . . 33 7.6.3. Test Parameters . . . . . . . . . . . . . . . . . . . 34
7.6.4. Test Procedures and expected Results . . . . . . . . 35 7.6.4. Test Procedures and expected Results . . . . . . . . 36
7.7. HTTPS Throughput . . . . . . . . . . . . . . . . . . . . 36 7.7. HTTPS Throughput . . . . . . . . . . . . . . . . . . . . 37
7.7.1. Objective . . . . . . . . . . . . . . . . . . . . . . 36 7.7.1. Objective . . . . . . . . . . . . . . . . . . . . . . 37
7.7.2. Test Setup . . . . . . . . . . . . . . . . . . . . . 36 7.7.2. Test Setup . . . . . . . . . . . . . . . . . . . . . 37
7.7.3. Test Parameters . . . . . . . . . . . . . . . . . . . 36 7.7.3. Test Parameters . . . . . . . . . . . . . . . . . . . 37
7.7.4. Test Procedures and Expected Results . . . . . . . . 39 7.7.4. Test Procedures and Expected Results . . . . . . . . 40
7.8. HTTPS Transaction Latency . . . . . . . . . . . . . . . . 40 7.8. HTTPS Transaction Latency . . . . . . . . . . . . . . . . 41
7.8.1. Objective . . . . . . . . . . . . . . . . . . . . . . 40 7.8.1. Objective . . . . . . . . . . . . . . . . . . . . . . 41
7.8.2. Test Setup . . . . . . . . . . . . . . . . . . . . . 40 7.8.2. Test Setup . . . . . . . . . . . . . . . . . . . . . 41
7.8.3. Test Parameters . . . . . . . . . . . . . . . . . . . 40 7.8.3. Test Parameters . . . . . . . . . . . . . . . . . . . 41
7.8.4. Test Procedures and Expected Results . . . . . . . . 42 7.8.4. Test Procedures and Expected Results . . . . . . . . 43
7.9. Concurrent TCP/HTTPS Connection Capacity . . . . . . . . 43 7.9. Concurrent TCP/HTTPS Connection Capacity . . . . . . . . 44
7.9.1. Objective . . . . . . . . . . . . . . . . . . . . . . 43 7.9.1. Objective . . . . . . . . . . . . . . . . . . . . . . 44
7.9.2. Test Setup . . . . . . . . . . . . . . . . . . . . . 43 7.9.2. Test Setup . . . . . . . . . . . . . . . . . . . . . 44
7.9.3. Test Parameters . . . . . . . . . . . . . . . . . . . 43 7.9.3. Test Parameters . . . . . . . . . . . . . . . . . . . 45
7.9.4. Test Procedures and expected Results . . . . . . . . 45 7.9.4. Test Procedures and expected Results . . . . . . . . 46
8. Formal Syntax . . . . . . . . . . . . . . . . . . . . . . . . 46 8. Formal Syntax . . . . . . . . . . . . . . . . . . . . . . . . 47
9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 46 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 47
10. Security Considerations . . . . . . . . . . . . . . . . . . . 46 10. Security Considerations . . . . . . . . . . . . . . . . . . . 48
11. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 47 11. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 48
12. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 47 12. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 48
13. References . . . . . . . . . . . . . . . . . . . . . . . . . 47 13. References . . . . . . . . . . . . . . . . . . . . . . . . . 48
13.1. Normative References . . . . . . . . . . . . . . . . . . 47 13.1. Normative References . . . . . . . . . . . . . . . . . . 48
13.2. Informative References . . . . . . . . . . . . . . . . . 47 13.2. Informative References . . . . . . . . . . . . . . . . . 49
Appendix A. NetSecOPEN Basic Traffic Mix . . . . . . . . . . . . 48 Appendix A. NetSecOPEN Basic Traffic Mix . . . . . . . . . . . . 49
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 56 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 58
1. Introduction 1. Introduction
15 years have passed since IETF recommended test methodology and 15 years have passed since IETF recommended test methodology and
terminology for firewalls initially ([RFC2647], [RFC3511]). The terminology for firewalls initially ([RFC2647], [RFC3511]). The
requirements for network security element performance and requirements for network security element performance and
effectiveness have increased tremendously since then. Security effectiveness have increased tremendously since then. Security
function implementations have evolved to more advanced areas and have function implementations have evolved to more advanced areas and have
diversified into intrusion detection and prevention, threat diversified into intrusion detection and prevention, threat
management, analysis of encrypted traffic, etc. In an industry of management, analysis of encrypted traffic, etc. In an industry of
growing importance, well-defined and reproducible key performance growing importance, well-defined and reproducible key performance
indicators (KPIs) are increasingly needed: They enable fair and indicators (KPIs) are increasingly needed as they enable fair and
reasonable comparison of network security functions. All these reasonable comparison of network security functions. All these
reasons have led to the creation of a new next-generation firewall reasons have led to the creation of a new next-generation firewall
benchmarking document. benchmarking document.
2. Requirements 2. Requirements
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described in BCP "OPTIONAL" in this document are to be interpreted as described in BCP
14 [RFC2119], [RFC8174] when, and only when, they appear in all 14 [RFC2119], [RFC8174] when, and only when, they appear in all
capitals, as shown here. capitals, as shown here.
3. Scope 3. Scope
This document provides testing terminology and testing methodology This document provides testing terminology and testing methodology
for next-generation firewalls and related security functions. It for next-generation firewalls security devices. It covers security
covers two main areas: security effectiveness configurations, effectiveness configurations, followed by performance benchmark
followed by performance benchmark testing. This document focuses on testing. This document focuses on advanced, realistic, and
advanced, realistic, and reproducible testing methods. Additionally, reproducible testing methods. Additionally, it describes test bed
it describes test bed environments, test tool requirements and test environments, test tool requirements and test result formats.
result formats.
4. Test Setup 4. Test Setup
Test setup defined in this document is applicable to all benchmarking Test setup defined in this document is applicable to all benchmarking
test scenarios described in Section 7. test scenarios described in Section 7.
4.1. Testbed Configuration 4.1. Testbed Configuration
Testbed configuration MUST ensure that any performance implications Testbed configuration MUST ensure that any performance implications
that are discovered during the benchmark testing aren't due to the that are discovered during the benchmark testing aren't due to the
inherent physical network limitations such as number of physical inherent physical network limitations such as number of physical
links and forwarding performance capabilities (throughput and links and forwarding performance capabilities (throughput and
latency) of the network devise in the testbed. For this reason, this latency) of the network devise in the testbed. For this reason, this
document recommends avoiding external devices such as switches and document recommends avoiding external devices such as switches and
routers in the testbed wherever possible. routers in the testbed wherever possible.
However, in the typical deployment, the security devices ( Device However, in the typical deployment, the security devices (Device
Under Test/System Under Test) are connected to routers and switches Under Test/System Under Test) are connected to routers and switches
which will reduce the number of entries in MAC or ARP tables of the which will reduce the number of entries in MAC or ARP tables of the
Device Under Test/System Under Test (DUT/SUT). If MAC or ARP tables Device Under Test/System Under Test (DUT/SUT). If MAC or ARP tables
have many entries, this may impact the actual DUT/SUT performance due have many entries, this may impact the actual DUT/SUT performance due
to MAC and ARP/ND table lookup processes. Therefore, it is to MAC and ARP/ND table lookup processes. Therefore, it is
RECOMMENDED to connect aggregation switches or routers between test RECOMMENDED to connect aggregation switches or routers between test
equipment and DUT/SUT as shown in Figure 1. The aggregation switches equipment and DUT/SUT as shown in Figure 1. The aggregation switches
or routers can be also used to aggregate the test equipment or DUT/ or routers can be also used to aggregate the test equipment or DUT/
SUT ports, if the numbers of used ports are mismatched between test SUT ports, if the numbers of used ports are mismatched between test
equipment and DUT/SUT. equipment and DUT/SUT.
skipping to change at page 6, line 7 skipping to change at page 6, line 4
A unique DUT/SUT configuration MUST be used for all benchmarking A unique DUT/SUT configuration MUST be used for all benchmarking
tests described in Section 7. Since each DUT/SUT will have their own tests described in Section 7. Since each DUT/SUT will have their own
unique configuration, users SHOULD configure their device with the unique configuration, users SHOULD configure their device with the
same parameters and security features that would be used in the same parameters and security features that would be used in the
actual deployment of the device or a typical deployment in order to actual deployment of the device or a typical deployment in order to
achieve maximum security coverage. achieve maximum security coverage.
This document attempts to define the recommended security features This document attempts to define the recommended security features
which SHOULD be consistently enabled for all the benchmarking tests which SHOULD be consistently enabled for all the benchmarking tests
described in Section 7. Table 1 below describes the RECOMMENDED sets described in Section 7. Table 1 below describes the sets of security
of feature list which SHOULD be configured on the DUT/SUT. feature list which SHOULD be configured on the DUT/SUT.
Based on customer use case, users MAY enable or disable SSL Based on customer use case, users MAY enable or disable SSL
inspection feature for "Throughput Performance with NetSecOPEN inspection feature for "Throughput Performance with NetSecOPEN
Traffic Mix" test scenario described in Section 7.1 Traffic Mix" test scenario described in Section 7.1
To improve repeatability, a summary of the DUT configuration To improve repeatability, a summary of the DUT configuration
including description of all enabled DUT/SUT features MUST be including description of all enabled DUT/SUT features MUST be
published with the benchmarking results. published with the benchmarking results.
+------------------------+ +------------------------+
skipping to change at page 7, line 32 skipping to change at page 7, line 30
* CVSS V2 Severity: High (7-10) * CVSS V2 Severity: High (7-10)
* If doing a group test the published start date and published * If doing a group test the published start date and published
end date SHOULD be the same end date SHOULD be the same
o Geographical location filtering and Application Identification and o Geographical location filtering and Application Identification and
Control configured to be triggered based on a site or application Control configured to be triggered based on a site or application
from the defined traffic mix from the defined traffic mix
In addition, it is also RECOMMENDED to configure a realistic number In addition, a realistic number of access control rules (ACL) MUST be
of access policy rules on the DUT/SUT. This document determines the configured on the DUT/SUT. However, this is applicable only for the
number of access policy rules for three different classes of DUT/SUT. security devices where ACL's are configurable. This document
The classification of the DUT/SUT MAY be based on its maximum determines the number of access policy rules for four different
supported firewall throughput performance number defined in the classes of DUT/SUT. The classification of the DUT/SUT MAY be based
vendor data sheet. This document classifies the DUT/SUT in four on its maximum supported firewall throughput performance number
different categories; namely Extra Small, Small, Medium, and Large. defined in the vendor data sheet. This document classifies the DUT/
SUT in four different categories; namely Extra Small, Small, Medium,
and Large.
The RECOMMENDED throughput values for the following classes are: The RECOMMENDED throughput values for the following classes are:
Extra Small (XS) - supported throughput less than 1Gbit/s Extra Small (XS) - supported throughput less than 1Gbit/s
Small (S) - supported throughput less than 5Gbit/s Small (S) - supported throughput less than 5Gbit/s
Medium (M) - supported throughput greater than 5Gbit/s and less than Medium (M) - supported throughput greater than 5Gbit/s and less than
10Gbit/s 10Gbit/s
Large (L) - supported throughput greater than 10Gbit/s Large (L) - supported throughput greater than 10Gbit/s
The Access Conrol Rules (ACL) defined in Table 2 MUST be configured The Access Control Rules (ACL) defined in Table 2 MUST be configured
from top to bottom in the correct order as shown in the table. from top to bottom in the correct order as shown in the table. The
(Note: There will be differences between how security vendors ACL entries MUST be configured in Forward Information Base (FIB)
implement ACL decision making.) The configured ACL MUST NOT block table of the DUT/SUT. (Note: There will be differences between how
the test traffic used for the benchmarking test scenarios. security vendors implement ACL decision making.) The configured ACL
MUST NOT block the security and performance test traffic used for the
benchmarking test scenarios.
+---------------------------------------------------+---------------+ +---------------------------------------------------+---------------+
| | DUD/SUT | | | DUT/SUT |
| | Classification| | | Classification|
| | #rules | | | #rules |
+-----------+-----------+------------------+------------+---+---+---+ +-----------+-----------+------------------+------------+---+---+---+
| | Match | | | | | | | | | Match | | | | | | |
| Rules Type| Criteria | Description | Action | XS| S | M | L | | Rules Type| Criteria | Description | Action | XS| S | M | L |
+-------------------------------------------------------------------+ +-------------------------------------------------------------------+
|Application|Application| Any application | block | 5 | 10| 20| 50| |Application|Application| Any application | block | 5 | 10| 20| 50|
|layer | | traffic NOT | | | | | | |layer | | traffic NOT | | | | | |
| | | included in the | | | | | | | | | included in the | | | | | |
| | | test traffic | | | | | | | | | test traffic | | | | | |
skipping to change at page 9, line 11 skipping to change at page 10, line 5
+-----------+-----------+------------------+--------+---+---+---+---+ +-----------+-----------+------------------+--------+---+---+---+---+
Table 2: DUT/SUT Access List Table 2: DUT/SUT Access List
4.3. Test Equipment Configuration 4.3. Test Equipment Configuration
In general, test equipment allows configuring parameters in different In general, test equipment allows configuring parameters in different
protocol layers. These parameters thereby influence the traffic protocol layers. These parameters thereby influence the traffic
flows which will be offered and impact performance measurements. flows which will be offered and impact performance measurements.
This document specifies common test equipment configuration This section specifies common test equipment configuration parameters
parameters applicable for all test scenarios defined in Section 7. applicable for all test scenarios defined in Section 7. Any test
Any test scenario specific parameters are described under the test scenario specific parameters are described under the test setup
setup section of each test scenario individually. section of each test scenario individually.
4.3.1. Client Configuration 4.3.1. Client Configuration
This section specifies which parameters SHOULD be considered while This section specifies which parameters SHOULD be considered while
configuring clients using test equipment. Also, this section configuring clients using test equipment. Also, this section
specifies the recommended values for certain parameters. specifies the RECOMMENDED values for certain parameters.
4.3.1.1. TCP Stack Attributes 4.3.1.1. TCP Stack Attributes
The TCP stack SHOULD use a TCP Reno [RFC5681] variant, which include The TCP stack SHOULD use a TCP Reno [RFC5681] variant, which include
congestion avoidance, back off and windowing, fast retransmission, congestion avoidance, back off and windowing, fast retransmission,
and fast recovery on every TCP connection between client and server and fast recovery on every TCP connection between client and server
endpoints. The default IPv4 and IPv6 MSS segments size MUST be set endpoints. The default IPv4 and IPv6 MSS segments size MUST be set
to 1460 bytes and 1440 bytes respectively and a TX and RX receive to 1460 bytes and 1440 bytes respectively and a TX and RX receive
windows of 64 KByte. Client initial congestion window MUST NOT windows of 64 KByte. Client initial congestion window MUST NOT
exceed 10 times the MSS. Delayed ACKs are permitted and the maximum exceed 10 times the MSS. Delayed ACKs are permitted and the maximum
client delayed Ack MUST NOT exceed 10 times the MSS before a forced client delayed Ack MUST NOT exceed 10 times the MSS before a forced
ACK. Up to 3 retries SHOULD be allowed before a timeout event is ACK. Up to 3 retries SHOULD be allowed before a timeout event is
declared. All traffic MUST set the TCP PSH flag to high. The source declared. All traffic MUST set the TCP PSH flag to high. The source
port range SHOULD be in the range of 1024 - 65535. Internal timeout port range SHOULD be in the range of 1024 - 65535. Internal timeout
SHOULD be dynamically scalable per RFC 793. Client SHOULD initiate SHOULD be dynamically scalable per RFC 793. Client SHOULD initiate
and close TCP connections. TCP connections MUST be closed via FIN. and close TCP connections. TCP connections MUST be closed via FIN.
4.3.1.2. Client IP Address Space 4.3.1.2. Client IP Address Space
The sum of the client IP space SHOULD contain the following The sum of the client IP space SHOULD contain the following
attributes. The traffic blocks SHOULD consist of multiple unique, attributes. The IP blocks SHOULD consist of multiple unique,
discontinuous static address blocks. A default gateway is permitted. discontinuous static address blocks. A default gateway is permitted.
The IPv4 ToS byte or IPv6 traffic class should be set to '00' or The IPv4 Type of Service (ToS) byte or IPv6 traffic class should be
'000000' respectively. set to '00' or '000000' respectively.
The following equation can be used to determine the required total The following equation can be used to determine the required total
number of client IP addresses. number of client IP addresses.
Desired total number of client IP = Target throughput [Mbit/s] / Desired total number of client IP = Target throughput [Mbit/s] /
Throughput per IP address [Mbit/s] Throughput per IP address [Mbit/s]
Based on deployment and use case scenario, the value for "Throughput Based on deployment and use case scenario, the value for "Throughput
per IP address" can be varied. per IP address" can be varied.
(Option 1) DUT/SUT deployment scenario 1 : 6-7 Mbit/s per IP (e.g. (Option 1) DUT/SUT deployment scenario 1 : 6-7 Mbit/s per IP (e.g.
1,400-1,700 IPs per 10Gbit/s throughput) 1,400-1,700 IPs per 10Gbit/s throughput)
(Option 2) DUT/SUT deployment scenario 2 : 0.1-0.2 Mbit/s per IP (Option 2) DUT/SUT deployment scenario 2 : 0.1-0.2 Mbit/s per IP
(e.g. 50,000-100,000 IPs per 10Gbit/s throughput) (e.g. 50,000-100,000 IPs per 10Gbit/s throughput)
Based on deployment and use case scenario, client IP addresses SHOULD Based on deployment and use case scenario, client IP addresses SHOULD
skipping to change at page 10, line 49 skipping to change at page 11, line 44
browser SHOULD advertise a User-Agent header. Headers MUST be sent browser SHOULD advertise a User-Agent header. Headers MUST be sent
uncompressed. The browser SHOULD enforce content length validation. uncompressed. The browser SHOULD enforce content length validation.
For encrypted traffic, the following attributes SHALL define the For encrypted traffic, the following attributes SHALL define the
negotiated encryption parameters. The test clients MUST use TLSv1.2 negotiated encryption parameters. The test clients MUST use TLSv1.2
or higher. TLS record size MAY be optimized for the HTTPS response or higher. TLS record size MAY be optimized for the HTTPS response
object size up to a record size of 16 KByte. The client endpoint object size up to a record size of 16 KByte. The client endpoint
MUST send TLS Extension Server Name Indication (SNI) information when MUST send TLS Extension Server Name Indication (SNI) information when
opening a security tunnel. Each client connection MUST perform a opening a security tunnel. Each client connection MUST perform a
full handshake with server certificate and MUST NOT use session reuse full handshake with server certificate and MUST NOT use session reuse
or resumption. Cipher suite and key size should be defined in the or resumption. Cipher suite and key size are defined in the
parameter session of each test scenario. parameter section of the specific test scenarios.
4.3.2. Backend Server Configuration 4.3.2. Backend Server Configuration
This document specifies which parameters should be considered while This section specifies which parameters should be considered while
configuring emulated backend servers using test equipment. configuring emulated backend servers using test equipment.
4.3.2.1. TCP Stack Attributes 4.3.2.1. TCP Stack Attributes
The TCP stack on the server side SHOULD be configured similar to the The TCP stack on the server side SHOULD be configured similar to the
client side configuration described in Section 4.3.1.1. In addition, client side configuration described in Section 4.3.1.1. In addition,
server initial congestion window MUST NOT exceed 10 times the MSS. server initial congestion window MUST NOT exceed 10 times the MSS.
Delayed ACKs are permitted and the maximum server delayed ACK MUST Delayed ACKs are permitted and the maximum server delayed ACK MUST
NOT exceed 10 times the MSS before a forced ACK. NOT exceed 10 times the MSS before a forced ACK.
skipping to change at page 11, line 34 skipping to change at page 12, line 29
4.3.2.3. HTTP / HTTPS Server Pool Endpoint Attributes 4.3.2.3. HTTP / HTTPS Server Pool Endpoint Attributes
The server pool for HTTP SHOULD listen on TCP port 80 and emulate The server pool for HTTP SHOULD listen on TCP port 80 and emulate
HTTP version 1.1 with persistence. The Server MUST advertise server HTTP version 1.1 with persistence. The Server MUST advertise server
type in the Server response header [RFC2616]. For HTTPS server, TLS type in the Server response header [RFC2616]. For HTTPS server, TLS
1.2 or higher MUST be used with a maximum record size of 16 KByte and 1.2 or higher MUST be used with a maximum record size of 16 KByte and
MUST NOT use ticket resumption or Session ID reuse . The server MUST MUST NOT use ticket resumption or Session ID reuse . The server MUST
listen on port TCP 443. The server SHALL serve a certificate to the listen on port TCP 443. The server SHALL serve a certificate to the
client. It is REQUIRED that the HTTPS server also check Host SNI client. It is REQUIRED that the HTTPS server also check Host SNI
information with the FQDN. Cipher suite and key size should be information with the FQDN. Cipher suite and key size are defined in
defined in the parameter section of each test scenario. the parameter section of the specific test scenarios.
4.3.3. Traffic Flow Definition 4.3.3. Traffic Flow Definition
This section describes the traffic pattern between client and server This section describes the traffic pattern between client and server
endpoints. At the beginning of the test, the server endpoint endpoints. At the beginning of the test, the server endpoint
initializes and will be ready to accept connection states including initializes and will be ready to accept connection states including
initialization of the TCP stack as well as bound HTTP and HTTPS initialization of the TCP stack as well as bound HTTP and HTTPS
servers. When a client endpoint is needed, it will initialize and be servers. When a client endpoint is needed, it will initialize and be
given attributes such as a MAC and IP address. The behavior of the given attributes such as a MAC and IP address. The behavior of the
client is to sweep though the given server IP space, sequentially client is to sweep though the given server IP space, sequentially
skipping to change at page 12, line 50 skipping to change at page 13, line 50
SUT, in contrast, the emulated servers should be ready to accept SUT, in contrast, the emulated servers should be ready to accept
requests from DUT/SUT or from emulated clients. requests from DUT/SUT or from emulated clients.
2. In the ramp up phase, the test equipment SHOULD start to generate 2. In the ramp up phase, the test equipment SHOULD start to generate
the test traffic. It SHOULD use a set approximate number of the test traffic. It SHOULD use a set approximate number of
unique client IP addresses actively to generate traffic. The unique client IP addresses actively to generate traffic. The
traffic SHOULD ramp from zero to desired target objective. The traffic SHOULD ramp from zero to desired target objective. The
target objective will be defined for each benchmarking test. The target objective will be defined for each benchmarking test. The
duration for the ramp up phase MUST be configured long enough, so duration for the ramp up phase MUST be configured long enough, so
that the test equipment does not overwhelm DUT/SUT's supported that the test equipment does not overwhelm DUT/SUT's supported
performance metrics namely; connections per second, concurrent performance metrics namely; connections per second, throughput,
TCP connections, and application transactions per second. The concurrent TCP connections, and application transactions per
RECOMMENDED time duration for the ramp up phase is 180-300 second. No measurements are made in this phase.
seconds. No measurements are made in this phase.
3. In the sustain phase, the test equipment SHOULD continue 3. In the sustain phase, the test equipment SHOULD continue
generating traffic to constant target value for a constant number generating traffic to constant target value for a constant number
of active client IPs. The mininum RECOMMENDED time duration for of active client IPs. The mininum RECOMMENDED time duration for
sustain phase is 300 seconds. This is the phase where sustain phase is 300 seconds. This is the phase where
measurements occur. measurements occur.
4. In the ramp down/close phase, no new connections are established, 4. In the ramp down/close phase, no new connections are established,
and no measurements are made. The time duration for ramp up and and no measurements are made. The time duration for ramp up and
ramp down phase SHOULD be same. The RECOMMENDED duration of this ramp down phase SHOULD be same.
phase is between 180 to 300 seconds.
5. The last phase is administrative and will occur when the test 5. The last phase is administrative and will occur when the test
equipment merges and collates the report data. equipment merges and collates the report data.
5. Test Bed Considerations 5. Test Bed Considerations
This section recommends steps to control the test environment and This section recommends steps to control the test environment and
test equipment, specifically focusing on virtualized environments and test equipment, specifically focusing on virtualized environments and
virtualized test equipment. virtualized test equipment.
skipping to change at page 13, line 38 skipping to change at page 14, line 35
the system under test and the test equipment do not limit the the system under test and the test equipment do not limit the
performance of the traffic generator. This is specifically performance of the traffic generator. This is specifically
important for virtualized components (vSwitches, vRouters). important for virtualized components (vSwitches, vRouters).
2. Verify that the performance of the test equipment matches and 2. Verify that the performance of the test equipment matches and
reasonably exceeds the expected maximum performance of the system reasonably exceeds the expected maximum performance of the system
under test. under test.
3. Assert that the test bed characteristics are stable during the 3. Assert that the test bed characteristics are stable during the
entire test session. Several factors might influence stability entire test session. Several factors might influence stability
specifically for virtualized test beds, for example additional specifically for virtualized test beds. For example additional
workloads in a virtualized system, load balancing and movement of workloads in a virtualized system, load balancing and movement of
virtual machines during the test, or simple issues such as virtual machines during the test, or simple issues such as
additional heat created by high workloads leading to an emergency additional heat created by high workloads leading to an emergency
CPU performance reduction. CPU performance reduction.
Test bed reference pre-tests help to ensure that the maximum desired Test bed reference pre-tests help to ensure that the maximum desired
traffic generator aspects such as throughput, transaction per second, traffic generator aspects such as throughput, transaction per second,
connection per second, concurrent connection and latency. connection per second, concurrent connection and latency.
Once the desired maximum performance goals for the system under test Once the desired maximum performance goals for the system under test
skipping to change at page 21, line 4 skipping to change at page 21, line 47
7.2.3.2. Test Equipment Configuration Parameters 7.2.3.2. Test Equipment Configuration Parameters
Test equipment configuration parameters MUST conform to the Test equipment configuration parameters MUST conform to the
requirements defined in Section 4.3. Following parameters MUST be requirements defined in Section 4.3. Following parameters MUST be
documented for this test scenario: documented for this test scenario:
Client IP address range defined in Section 4.3.1.2 Client IP address range defined in Section 4.3.1.2
Server IP address range defined in Section 4.3.2.2 Server IP address range defined in Section 4.3.2.2
Traffic distribution ratio between IPv4 and IPv6 defined in Traffic distribution ratio between IPv4 and IPv6 defined in
Section 4.3.1.2 Section 4.3.1.2
Target connections per second: Initial value from product data sheet Target connections per second: Initial value from product data sheet
(if known) (if known)
Initial connections per second: 10% of "Target connections per Initial connections per second: 10% of "Target connections per
second" second" (an optional parameter for documentation)
The client SHOULD negotiate HTTP 1.1 and close the connection with The client SHOULD negotiate HTTP 1.1 and close the connection with
FIN immediately after completion of one transaction. In each test FIN immediately after completion of one transaction. In each test
iteration, client MUST send GET command requesting a fixed HTTP iteration, client MUST send GET command requesting a fixed HTTP
response object size. response object size.
The RECOMMENDED response object sizes are 1, 2, 4, 16, 64 KByte The RECOMMENDED response object sizes are 1, 2, 4, 16, 64 KByte
7.2.3.3. Test Results Validation Criteria 7.2.3.3. Test Results Validation Criteria
skipping to change at page 21, line 37 skipping to change at page 22, line 31
response code other than 200 OK) MUST be less than 0.001% (1 out response code other than 200 OK) MUST be less than 0.001% (1 out
of 100,000 transactions) of total attempt transactions of 100,000 transactions) of total attempt transactions
b. Number of Terminated TCP connections due to unexpected TCP RST b. Number of Terminated TCP connections due to unexpected TCP RST
sent by DUT/SUT MUST be less than 0.001% (1 out of 100,000 sent by DUT/SUT MUST be less than 0.001% (1 out of 100,000
connections) of total initiated TCP connections connections) of total initiated TCP connections
c. During the sustain phase, traffic should be forwarded at a c. During the sustain phase, traffic should be forwarded at a
constant rate constant rate
d. Concurrent TCP connections SHOULD be constant during steady d. Concurrent TCP connections MUST be constant during steady state
state. Any deviation of concurrent TCP connections MUST be less and any deviation of concurrent TCP connections SHOULD be less
than 10%. This confirms the DUT opens and closes TCP connections than 10%. This confirms the DUT opens and closes TCP connections
almost at the same rate almost at the same rate
7.2.3.4. Measurement 7.2.3.4. Measurement
Following KPI metric MUST be reported for each test iteration. Following KPI metric MUST be reported for each test iteration.
average TCP Connections Per Second average TCP Connections Per Second
7.2.4. Test Procedures and Expected Results 7.2.4. Test Procedures and Expected Results
skipping to change at page 24, line 4 skipping to change at page 24, line 45
Test equipment configuration parameters MUST conform to the Test equipment configuration parameters MUST conform to the
requirements defined in Section 4.3. Following parameters MUST be requirements defined in Section 4.3. Following parameters MUST be
documented for this test scenario: documented for this test scenario:
Client IP address range defined in Section 4.3.1.2 Client IP address range defined in Section 4.3.1.2
Server IP address range defined in Section 4.3.2.2 Server IP address range defined in Section 4.3.2.2
Traffic distribution ratio between IPv4 and IPv6 defined in Traffic distribution ratio between IPv4 and IPv6 defined in
Section 4.3.1.2 Section 4.3.1.2
Target Throughput: Initial value from product data sheet (if known) Target Throughput: Initial value from product data sheet (if known)
Initial Throughput: 10% of "Target Throughput" Initial Throughput: 10% of "Target Throughput" (an optional parameter
for documentation)
Number of HTTP response object requests (transactions) per Number of HTTP response object requests (transactions) per
connection: 10 connection: 10
RECOMMENDED HTTP response object size: 1 KByte, 16 KByte, 64 KByte, RECOMMENDED HTTP response object size: 1 KByte, 16 KByte, 64 KByte,
256 KByte and mixed objects defined in the table 256 KByte and mixed objects defined in the table
+---------------------+---------------------+ +---------------------+---------------------+
| Object size (KByte) | Number of requests/ | | Object size (KByte) | Number of requests/ |
| | Weight | | | Weight |
+---------------------+---------------------+ +---------------------+---------------------+
| 0.2 | 1 | | 0.2 | 1 |
+---------------------+---------------------+ +---------------------+---------------------+
| 6 | 1 | | 6 | 1 |
skipping to change at page 25, line 5 skipping to change at page 25, line 46
The following test Criteria is defined as test results validation The following test Criteria is defined as test results validation
criteria. Test results validation criteria MUST be monitored during criteria. Test results validation criteria MUST be monitored during
the whole sustain phase of the traffic load profile the whole sustain phase of the traffic load profile
a. Number of failed Application transactions (receiving any HTTP a. Number of failed Application transactions (receiving any HTTP
response code other than 200 OK) MUST be less than 0.001% (1 out response code other than 200 OK) MUST be less than 0.001% (1 out
of 100,000 transactions) of attempt transactions. of 100,000 transactions) of attempt transactions.
b. Traffic should be forwarded constantly. b. Traffic should be forwarded constantly.
c. Concurrent connetions MUST be constant. The deviation of c. Concurrent TCP connections MUST be constant during steady state
concurrent TCP connection MUST NOT increase more than 10% and any deviation of concurrent TCP connections SHOULD be less
than 10%. This confirms the DUT opens and closes TCP connections
almost at the same rate
7.3.3.4. Measurement 7.3.3.4. Measurement
The KPI metrics MUST be reported for this test scenario: The KPI metrics MUST be reported for this test scenario:
average Throughput and average HTTP Transactions per Second average Throughput and average HTTP Transactions per Second
7.3.4. Test Procedures and Expected Results 7.3.4. Test Procedures and Expected Results
The test procedure is designed to measure HTTP throughput of the DUT/ The test procedure is designed to measure HTTP throughput of the DUT/
skipping to change at page 27, line 26 skipping to change at page 28, line 26
Section 4.3.1.2 Section 4.3.1.2
Target objective for scenario 1: 50% of the maximum connection per Target objective for scenario 1: 50% of the maximum connection per
second measured in test scenario TCP/HTTP Connections Per Second second measured in test scenario TCP/HTTP Connections Per Second
(Section 7.2) (Section 7.2)
Target objective for scenario 2: 50% of the maximum throughput Target objective for scenario 2: 50% of the maximum throughput
measured in test scenario HTTP Throughput (Section 7.3) measured in test scenario HTTP Throughput (Section 7.3)
Initial objective for scenario 1: 10% of Target objective for Initial objective for scenario 1: 10% of Target objective for
scenario 1" scenario 1" (an optional parameter for documentation)
Initial objective for scenario 2: 10% of "Target objective for Initial objective for scenario 2: 10% of "Target objective for
scenario 2" scenario 2" (an optional parameter for documentation)
HTTP transaction per TCP connection: test scenario 1 with single HTTP transaction per TCP connection: test scenario 1 with single
transaction and the second scenario with 10 transactions transaction and the second scenario with 10 transactions
HTTP 1.1 with GET command requesting a single object. The HTTP 1.1 with GET command requesting a single object. The
RECOMMENDED object sizes are 1, 16 or 64 KByte. For each test RECOMMENDED object sizes are 1, 16 or 64 KByte. For each test
iteration, client MUST request a single HTTP response object size. iteration, client MUST request a single HTTP response object size.
7.4.3.3. Test Results Validation Criteria 7.4.3.3. Test Results Validation Criteria
skipping to change at page 28, line 12 skipping to change at page 29, line 12
response code other than 200 OK) MUST be less than 0.001% (1 out response code other than 200 OK) MUST be less than 0.001% (1 out
of 100,000 transactions) of attempt transactions. of 100,000 transactions) of attempt transactions.
b. Number of Terminated TCP connections due to unexpected TCP RST b. Number of Terminated TCP connections due to unexpected TCP RST
sent by DUT/SUT MUST be less than 0.001% (1 out of 100,000 sent by DUT/SUT MUST be less than 0.001% (1 out of 100,000
connections) of total initiated TCP connections connections) of total initiated TCP connections
c. During the sustain phase, traffic should be forwarded at a c. During the sustain phase, traffic should be forwarded at a
constant rate. constant rate.
d. Concurrent TCP connections should be constant during steady d. Concurrent TCP connections MUST be constant during steady state
state. This confirms the DUT opens and closes TCP connections at and any deviation of concurrent TCP connections SHOULD be less
the same rate. than 10%. This confirms the DUT opens and closes TCP connections
almost at the same rate
e. After ramp up the DUT MUST achieve the "Target objective" defined e. After ramp up the DUT MUST achieve the "Target objective" defined
in the parameter Section 7.4.3.2 and remain in that state for the in the parameter Section 7.4.3.2 and remain in that state for the
entire test duration (sustain phase). entire test duration (sustain phase).
7.4.3.4. Measurement 7.4.3.4. Measurement
Following KPI metrics MUST be reported for each test scenario and Following KPI metrics MUST be reported for each test scenario and
HTTP response object sizes separately: HTTP response object sizes separately:
skipping to change at page 30, line 38 skipping to change at page 31, line 38
Server IP address range defined in Section 4.3.2.2 Server IP address range defined in Section 4.3.2.2
Traffic distribution ratio between IPv4 and IPv6 defined in Traffic distribution ratio between IPv4 and IPv6 defined in
Section 4.3.1.2 Section 4.3.1.2
Target concurrent connection: Initial value from product data Target concurrent connection: Initial value from product data
sheet (if known) sheet (if known)
Initial concurrent connection: 10% of "Target concurrent Initial concurrent connection: 10% of "Target concurrent
connection" connection" (an optional parameter for documentation)
Maximum connections per second during ramp up phase: 50% of Maximum connections per second during ramp up phase: 50% of
maximum connections per second measured in test scenario TCP/HTTP maximum connections per second measured in test scenario TCP/HTTP
Connections per second (Section 7.2) Connections per second (Section 7.2)
Ramp up time (in traffic load profile for "Target concurrent Ramp up time (in traffic load profile for "Target concurrent
connection"): "Target concurrent connection" / "Maximum connection"): "Target concurrent connection" / "Maximum
connections per second during ramp up phase" connections per second during ramp up phase"
Ramp up time (in traffic load profile for "Initial concurrent Ramp up time (in traffic load profile for "Initial concurrent
skipping to change at page 31, line 32 skipping to change at page 32, line 32
the whole sustain phase of the traffic load profile. the whole sustain phase of the traffic load profile.
a. Number of failed Application transactions (receiving any HTTP a. Number of failed Application transactions (receiving any HTTP
response code other than 200 OK) MUST be less than 0.001% (1 out response code other than 200 OK) MUST be less than 0.001% (1 out
of 100,000 transaction) of total attempted transactions of 100,000 transaction) of total attempted transactions
b. Number of Terminated TCP connections due to unexpected TCP RST b. Number of Terminated TCP connections due to unexpected TCP RST
sent by DUT/SUT MUST be less than 0.001% (1 out of 100,000 sent by DUT/SUT MUST be less than 0.001% (1 out of 100,000
connections) of total initiated TCP connections connections) of total initiated TCP connections
c. During the sustain phase, traffic should be forwarded constantly c. During the sustain phase, traffic SHOULD be forwarded constantly
7.5.3.4. Measurement 7.5.3.4. Measurement
Following KPI metric MUST be reported for this test scenario: Following KPI metric MUST be reported for this test scenario:
average Concurrent TCP Connections average Concurrent TCP Connections
7.5.4. Test Procedures and expected Results 7.5.4. Test Procedures and expected Results
The test procedure is designed to measure the concurrent TCP The test procedure is designed to measure the concurrent TCP
skipping to change at page 33, line 48 skipping to change at page 34, line 48
Server IP address range defined in Section 4.3.2.2 Server IP address range defined in Section 4.3.2.2
Traffic distribution ratio between IPv4 and IPv6 defined in Traffic distribution ratio between IPv4 and IPv6 defined in
Section 4.3.1.2 Section 4.3.1.2
Target connections per second: Initial value from product data sheet Target connections per second: Initial value from product data sheet
(if known) (if known)
Initial connections per second: 10% of "Target connections per Initial connections per second: 10% of "Target connections per
second" second" (an optional parameter for documentation)
RECOMMENDED ciphers and keys: RECOMMENDED ciphers and keys:
1. ECHDE-ECDSA-AES128-GCM-SHA256 with Prime256v1 (Signature Hash 1. ECHDE-ECDSA-AES128-GCM-SHA256 with Prime256v1 (Signature Hash
Algorithm: ecdsa_secp256r1_sha256 and Supported group: sepc256r1) Algorithm: ecdsa_secp256r1_sha256 and Supported group: sepc256r1)
2. ECDHE-RSA-AES128-GCM-SHA256 with RSA 2048 (Signature Hash 2. ECDHE-RSA-AES128-GCM-SHA256 with RSA 2048 (Signature Hash
Algorithm: rsa_pkcs1_sha256 and Supported group: sepc256) Algorithm: rsa_pkcs1_sha256 and Supported group: sepc256)
3. ECDHE-ECDSA-AES256-GCM-SHA384 with Secp521 (Signature Hash 3. ECDHE-ECDSA-AES256-GCM-SHA384 with Secp521 (Signature Hash
skipping to change at page 34, line 39 skipping to change at page 35, line 39
response code other than 200 OK) MUST be less than 0.001% (1 out response code other than 200 OK) MUST be less than 0.001% (1 out
of 100,000 transactions) of attempt transactions of 100,000 transactions) of attempt transactions
b. Number of Terminated TCP connections due to unexpected TCP RST b. Number of Terminated TCP connections due to unexpected TCP RST
sent by DUT/SUT MUST be less than 0.001% (1 out of 100,000 sent by DUT/SUT MUST be less than 0.001% (1 out of 100,000
connections) of total initiated TCP connections connections) of total initiated TCP connections
c. During the sustain phase, traffic should be forwarded at a c. During the sustain phase, traffic should be forwarded at a
constant rate constant rate
d. Concurrent TCP connections SHOULD be constant during steady d. Concurrent TCP connections MUST be constant during steady state
state. This confirms that the DUT open and close the TCP and any deviation of concurrent TCP connections SHOULD be less
connections at the same rate than 10%. This confirms the DUT opens and closes TCP connections
almost at the same rate
7.6.3.4. Measurement 7.6.3.4. Measurement
Following KPI metrics MUST be reported for this test scenario: Following KPI metrics MUST be reported for this test scenario:
average TCP Connections Per Second, average TLS Handshake Rate (TLS average TCP Connections Per Second, average TLS Handshake Rate (TLS
Handshake Rate can be measured in the test scenario using 1KB object Handshake Rate can be measured in the test scenario using 1KB object
size) size)
7.6.4. Test Procedures and expected Results 7.6.4. Test Procedures and expected Results
skipping to change at page 37, line 6 skipping to change at page 38, line 6
documented for this test scenario: documented for this test scenario:
Client IP address range defined in Section 4.3.1.2 Client IP address range defined in Section 4.3.1.2
Server IP address range defined in Section 4.3.2.2 Server IP address range defined in Section 4.3.2.2
Traffic distribution ratio between IPv4 and IPv6 defined in Traffic distribution ratio between IPv4 and IPv6 defined in
Section 4.3.1.2 Section 4.3.1.2
Target Throughput: Initial value from product data sheet (if known) Target Throughput: Initial value from product data sheet (if known)
Initial Throughput: 10% of "Target Throughput" Initial Throughput: 10% of "Target Throughput" (an optional parameter
for documentation)
Number of HTTPS response object requests (transactions) per Number of HTTPS response object requests (transactions) per
connection: 10 connection: 10
RECOMMENDED ciphers and keys: RECOMMENDED ciphers and keys:
1. ECHDE-ECDSA-AES128-GCM-SHA256 with Prime256v1 (Signature Hash 1. ECHDE-ECDSA-AES128-GCM-SHA256 with Prime256v1 (Signature Hash
Algorithm: ecdsa_secp256r1_sha256 and Supported group: sepc256r1) Algorithm: ecdsa_secp256r1_sha256 and Supported group: sepc256r1)
2. ECDHE-RSA-AES128-GCM-SHA256 with RSA 2048 (Signature Hash 2. ECDHE-RSA-AES128-GCM-SHA256 with RSA 2048 (Signature Hash
skipping to change at page 38, line 44 skipping to change at page 39, line 44
The following test Criteria is defined as test results validation The following test Criteria is defined as test results validation
criteria. Test results validation criteria MUST be monitored during criteria. Test results validation criteria MUST be monitored during
the whole sustain phase of the traffic load profile. the whole sustain phase of the traffic load profile.
a. Number of failed Application transactions (receiving any HTTP a. Number of failed Application transactions (receiving any HTTP
response code other than 200 OK) MUST be less than 0.001% (1 out response code other than 200 OK) MUST be less than 0.001% (1 out
of 100,000 transactions) of attempt transactions. of 100,000 transactions) of attempt transactions.
b. Traffic should be forwarded constantly. b. Traffic should be forwarded constantly.
c. The deviation of concurrent TCP connections MUST be less than 10% c. Concurrent TCP connections MUST be constant during steady state
and any deviation of concurrent TCP connections SHOULD be less
than 10%. This confirms the DUT opens and closes TCP connections
almost at the same rate
7.7.3.4. Measurement 7.7.3.4. Measurement
The KPI metrics MUST be reported for this test scenario: The KPI metrics MUST be reported for this test scenario:
average Throughput and average HTTPS Transactions Per Second average Throughput and average HTTPS Transactions Per Second
7.7.4. Test Procedures and Expected Results 7.7.4. Test Procedures and Expected Results
The test procedure consists of three major steps. This test The test procedure consists of three major steps. This test
skipping to change at page 40, line 46 skipping to change at page 42, line 4
7.8.3.2. Test Equipment Configuration Parameters 7.8.3.2. Test Equipment Configuration Parameters
Test equipment configuration parameters MUST conform to the Test equipment configuration parameters MUST conform to the
requirements defined in Section 4.3. Following parameters MUST be requirements defined in Section 4.3. Following parameters MUST be
documented for this test scenario: documented for this test scenario:
Client IP address range defined in Section 4.3.1.2 Client IP address range defined in Section 4.3.1.2
Server IP address range defined in Section 4.3.2.2 Server IP address range defined in Section 4.3.2.2
Traffic distribution ratio between IPv4 and IPv6 defined in Traffic distribution ratio between IPv4 and IPv6 defined in
Section 4.3.1.2 Section 4.3.1.2
RECOMMENDED cipher suites and key size: ECDHE-ECDSA-AES256-GCM-SHA384 RECOMMENDED cipher suites and key size: ECDHE-ECDSA-AES256-GCM-SHA384
with Secp521 bits key size (Signature Hash Algorithm: with Secp521 bits key size (Signature Hash Algorithm:
ecdsa_secp384r1_sha384 and Supported group: sepc521r1) ecdsa_secp384r1_sha384 and Supported group: sepc521r1)
Target objective for scenario 1: 50% of the maximum connections per Target objective for scenario 1: 50% of the maximum connections per
second measured in test scenario TCP/HTTPS Connections per second second measured in test scenario TCP/HTTPS Connections per second
(Section 7.6) (Section 7.6)
Target objective for scenario 2: 50% of the maximum throughput Target objective for scenario 2: 50% of the maximum throughput
measured in test scenario HTTPS Throughput (Section 7.7) measured in test scenario HTTPS Throughput (Section 7.7)
Initial objective for scenario 1: 10% of Target objective for Initial objective for scenario 1: 10% of Target objective for
scenario 1" scenario 1" (an optional parameter for documentation)
Initial objective for scenario 2: 10% of "Target objective for Initial objective for scenario 2: 10% of "Target objective for
scenario 2" scenario 2" (an optional parameter for documentation)
HTTPS transaction per TCP connection: test scenario 1 with single HTTPS transaction per TCP connection: test scenario 1 with single
transaction and the second scenario with 10 transactions transaction and the second scenario with 10 transactions
HTTPS 1.1 with GET command requesting a single 1, 16 or 64 KByte HTTPS 1.1 with GET command requesting a single 1, 16 or 64 KByte
object. For each test iteration, client MUST request a single HTTPS object. For each test iteration, client MUST request a single HTTPS
response object size. response object size.
7.8.3.3. Test Results Validation Criteria 7.8.3.3. Test Results Validation Criteria
skipping to change at page 41, line 44 skipping to change at page 43, line 5
response code other than 200 OK) MUST be less than 0.001% (1 out response code other than 200 OK) MUST be less than 0.001% (1 out
of 100,000 transactions) of attempt transactions. of 100,000 transactions) of attempt transactions.
b. Number of Terminated TCP connections due to unexpected TCP RST b. Number of Terminated TCP connections due to unexpected TCP RST
sent by DUT/SUT MUST be less than 0.001% (1 out of 100,000 sent by DUT/SUT MUST be less than 0.001% (1 out of 100,000
connections) of total initiated TCP connections connections) of total initiated TCP connections
c. During the sustain phase, traffic should be forwarded at a c. During the sustain phase, traffic should be forwarded at a
constant rate. constant rate.
d. Concurrent TCP connections should be constant during steady d. Concurrent TCP connections MUST be constant during steady state
state. This confirms the DUT opens and closes TCP connections at and any deviation of concurrent TCP connections SHOULD be less
the same rate. than 10%. This confirms the DUT opens and closes TCP connections
almost at the same rate
e. After ramp up the DUT MUST achieve the "Target objective" defined e. After ramp up the DUT MUST achieve the "Target objective" defined
in the parameter Section 7.8.3.2 and remain in that state for the in the parameter Section 7.8.3.2 and remain in that state for the
entire test duration (sustain phase). entire test duration (sustain phase).
7.8.3.4. Measurement 7.8.3.4. Measurement
Following KPI metrics MUST be reported for each test scenario and Following KPI metrics MUST be reported for each test scenario and
HTTPS response object sizes separately: HTTPS response object sizes separately:
skipping to change at page 44, line 20 skipping to change at page 45, line 36
Section 4.3.1.2 Section 4.3.1.2
RECOMMENDED cipher suites and key size: ECDHE-ECDSA-AES256-GCM- RECOMMENDED cipher suites and key size: ECDHE-ECDSA-AES256-GCM-
SHA384 with Secp521 bits key size (Signature Hash Algorithm: SHA384 with Secp521 bits key size (Signature Hash Algorithm:
ecdsa_secp384r1_sha384 and Supported group: sepc521r1) ecdsa_secp384r1_sha384 and Supported group: sepc521r1)
Target concurrent connections: Initial value from product data Target concurrent connections: Initial value from product data
sheet (if known) sheet (if known)
Initial concurrent connections: 10% of "Target concurrent Initial concurrent connections: 10% of "Target concurrent
connections" connections" (an optional parameter for documentation)
Connections per second during ramp up phase: 50% of maximum Connections per second during ramp up phase: 50% of maximum
connections per second measured in test scenario TCP/HTTPS connections per second measured in test scenario TCP/HTTPS
Connections per second (Section 7.6) Connections per second (Section 7.6)
Ramp up time (in traffic load profile for "Target concurrent Ramp up time (in traffic load profile for "Target concurrent
connections"): "Target concurrent connections" / "Maximum connections"): "Target concurrent connections" / "Maximum
connections per second during ramp up phase" connections per second during ramp up phase"
Ramp up time (in traffic load profile for "Initial concurrent Ramp up time (in traffic load profile for "Initial concurrent
skipping to change at page 46, line 8 skipping to change at page 47, line 20
concurrent TCP connections". The measured KPIs during the sustain concurrent TCP connections". The measured KPIs during the sustain
phase MUST meet the validation criteria "a" and "b" defined in phase MUST meet the validation criteria "a" and "b" defined in
Section 7.9.3.3. Section 7.9.3.3.
If the KPI metrics do not meet the validation criteria, the test If the KPI metrics do not meet the validation criteria, the test
procedure MUST NOT be continued to "Step 2". procedure MUST NOT be continued to "Step 2".
7.9.4.2. Step 2: Test Run with Target Objective 7.9.4.2. Step 2: Test Run with Target Objective
Configure test equipment to establish "Target concurrent TCP Configure test equipment to establish "Target concurrent TCP
connections".The test equipment SHOULD follow the traffic load connections". The test equipment SHOULD follow the traffic load
profile definition (except ramp up time) as described in profile definition (except ramp up time) as described in
Section 4.3.4. Section 4.3.4.
During the ramp up and sustain phase, the other KPIs such as During the ramp up and sustain phase, the other KPIs such as
throughput, TCP connections per second and application transactions throughput, TCP connections per second and application transactions
per second MUST NOT reach to the maximum value that the DUT/SUT can per second MUST NOT reach to the maximum value that the DUT/SUT can
support. support.
The test equipment SHOULD start to measure and record KPIs defined in The test equipment SHOULD start to measure and record KPIs defined in
Section 7.9.3.4. The frequency of measurement SHOULD be 2 seconds. Section 7.9.3.4. The frequency of measurement SHOULD be 2 seconds.
skipping to change at page 46, line 49 skipping to change at page 48, line 12
Note to RFC Editor: this section may be removed on publication as an Note to RFC Editor: this section may be removed on publication as an
RFC. RFC.
10. Security Considerations 10. Security Considerations
The primary goal of this document is to provide benchmarking The primary goal of this document is to provide benchmarking
terminology and methodology for next-generation network security terminology and methodology for next-generation network security
devices. However, readers should be aware that there is some overlap devices. However, readers should be aware that there is some overlap
between performance and security issues. Specifically, the optimal between performance and security issues. Specifically, the optimal
configuration for network security device performance may not be the configuration for network security device performance may not be the
most secure, and vice-versa. The Cipher suites are recommended in most secure, and vice-versa. The Cipher suites recommended in this
this document are just for test purpose only. The Cipher suite document are just for test purpose only. The Cipher suite
recommendation for a real deployment is outside the scope of this recommendation for a real deployment is outside the scope of this
document. document.
11. Acknowledgements 11. Acknowledgements
Acknowledgements will be added in the future release. Acknowledgements will be added in the future release.
12. Contributors 12. Contributors
The authors would like to thank the many people that contributed The authors would like to thank the many people that contributed
skipping to change at page 48, line 35 skipping to change at page 49, line 47
o Multiple unique certificates for HTTPS/TLS o Multiple unique certificates for HTTPS/TLS
o A wide variety of different object sizes o A wide variety of different object sizes
o Different URL paths o Different URL paths
o Mix of HTTP and HTTPS o Mix of HTTP and HTTPS
A traffic mix for testing performance of next generation firewalls A traffic mix for testing performance of next generation firewalls
MUST also facility application identification using different MUST also facilitate application identification using different
detection methods with and without decryption of the traffic. Such detection methods with and without decryption of the traffic. Such
as: as:
o HTTP HOST based application detection o HTTP HOST based application detection
o HTTPS/TLS Server Name Indication (SNI) o HTTPS/TLS Server Name Indication (SNI)
o Certificate Subject Common Name (CN) o Certificate Subject Common Name (CN)
The mix MUST be of sufficient complexity and volume to render The mix MUST be of sufficient complexity and volume to render
differences in individual apps as statistically insignificant. For differences in individual apps as statistically insignificant. For
example, changes in like to like apps - such as one type of video example, changes in like to like apps - such as one type of video
service vs. another both consist of larger objects whereas one news service vs. another both consist of larger objects whereas one news
site vs. another both typically have more connections then other apps site vs. another both typically have more connections then other apps
because of trackers and embedded advertising content. To achieve because of trackers and embedded advertising content. To achieve
skipping to change at page 49, line 18 skipping to change at page 50, line 29
o Hundreds of unique certificates for HTTPS/TLS o Hundreds of unique certificates for HTTPS/TLS
o Thousands of different object sizes per client in orders matching o Thousands of different object sizes per client in orders matching
applications applications
The following is a description of what a popular application in an The following is a description of what a popular application in an
enterprise traffic mix contains. enterprise traffic mix contains.
Table 5 lists the FQDNs, number of transactions and bytes transferred Table 5 lists the FQDNs, number of transactions and bytes transferred
as an example client interacts with Office 365 Outlook, Word, Excel, as an example, client interactions with Office 365 Outlook, Word,
PowerPoint, SharePoint and Skype. Excel, PowerPoint, SharePoint and Skype.
+---------------------------------+------------+-------------+ +---------------------------------+------------+-------------+
| Office365 FQDN | Bytes | Transaction | | Office365 FQDN | Bytes | Transaction |
+============================================================+ +============================================================+
| r1.res.office365.com | 14,056,960 | 192 | | r1.res.office365.com | 14,056,960 | 192 |
+---------------------------------+------------+-------------+ +---------------------------------+------------+-------------+
| s1-word-edit-15.cdn.office.net | 6,731,019 | 22 | | s1-word-edit-15.cdn.office.net | 6,731,019 | 22 |
+---------------------------------+------------+-------------+ +---------------------------------+------------+-------------+
| company1-my.sharepoint.com | 6,269,492 | 42 | | company1-my.sharepoint.com | 6,269,492 | 42 |
+---------------------------------+------------+-------------+ +---------------------------------+------------+-------------+
skipping to change at page 56, line 48 skipping to change at page 58, line 33
Carsten Rossenhoevel Carsten Rossenhoevel
EANTC AG EANTC AG
Salzufer 14 Salzufer 14
Berlin 10587 Berlin 10587
Germany Germany
Email: cross@eantc.de Email: cross@eantc.de
Brian Monkman Brian Monkman
NetSecOPEN NetSecOPEN
417 Independence Court
Mechanicsburg, PA 17050
USA
Email: bmonkman@netsecopen.org Email: bmonkman@netsecopen.org
 End of changes. 55 change blocks. 
144 lines changed or deleted 158 lines changed or added

This html diff was produced by rfcdiff 1.47. The latest version is available from http://tools.ietf.org/tools/rfcdiff/