draft-ietf-tcpm-generalized-ecn-02.txt   draft-ietf-tcpm-generalized-ecn-03.txt 
Network Working Group M. Bagnulo Network Working Group M. Bagnulo
Internet-Draft UC3M Internet-Draft UC3M
Obsoletes: 5562 (if approved) B. Briscoe Obsoletes: 5562 (if approved) B. Briscoe
Intended status: Experimental CableLabs Intended status: Experimental CableLabs
Expires: May 3, 2018 October 30, 2017 Expires: April 25, 2019 October 22, 2018
ECN++: Adding Explicit Congestion Notification (ECN) to TCP Control ECN++: Adding Explicit Congestion Notification (ECN) to TCP Control
Packets Packets
draft-ietf-tcpm-generalized-ecn-02 draft-ietf-tcpm-generalized-ecn-03
Abstract Abstract
This document describes an experimental modification to ECN when used This document describes an experimental modification to ECN when used
with TCP. It allows the use of ECN on the following TCP packets: with TCP. It allows the use of ECN on the following TCP packets:
SYNs, pure ACKs, Window probes, FINs, RSTs and retransmissions. SYNs, pure ACKs, Window probes, FINs, RSTs and retransmissions.
Status of This Memo Status of This Memo
This Internet-Draft is submitted in full conformance with the This Internet-Draft is submitted in full conformance with the
skipping to change at page 1, line 34 skipping to change at page 1, line 34
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/. Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on May 3, 2018. This Internet-Draft will expire on April 25, 2019.
Copyright Notice Copyright Notice
Copyright (c) 2017 IETF Trust and the persons identified as the Copyright (c) 2018 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(https://trustee.ietf.org/license-info) in effect on the date of (https://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License. described in the Simplified BSD License.
This document may contain material from IETF Documents or IETF
Contributions published or made publicly available before November
10, 2008. The person(s) controlling the copyright in some of this
material may not have granted the IETF Trust the right to allow
modifications of such material outside the IETF Standards Process.
Without obtaining an adequate license from the person(s) controlling
the copyright in such materials, this document may not be modified
outside the IETF Standards Process, and derivative works of it may
not be created outside the IETF Standards Process, except to format
it for publication as an RFC or to translate it into languages other
than English.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . 3 1.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . 4
1.2. Experiment Goals . . . . . . . . . . . . . . . . . . . . 4 1.2. Experiment Goals . . . . . . . . . . . . . . . . . . . . 5
1.3. Document Structure . . . . . . . . . . . . . . . . . . . 5 1.3. Document Structure . . . . . . . . . . . . . . . . . . . 6
2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 5 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 6
3. Specification . . . . . . . . . . . . . . . . . . . . . . . . 6 3. Specification . . . . . . . . . . . . . . . . . . . . . . . . 7
3.1. Network (e.g. Firewall) Behaviour . . . . . . . . . . . . 6 3.1. Network (e.g. Firewall) Behaviour . . . . . . . . . . . . 7
3.2. Endpoint Behaviour . . . . . . . . . . . . . . . . . . . 7 3.2. Sender Behaviour . . . . . . . . . . . . . . . . . . . . 8
3.2.1. SYN . . . . . . . . . . . . . . . . . . . . . . . . . 8 3.2.1. SYN (Send) . . . . . . . . . . . . . . . . . . . . . 9
3.2.2. SYN-ACK . . . . . . . . . . . . . . . . . . . . . . . 11 3.2.2. SYN-ACK (Send) . . . . . . . . . . . . . . . . . . . 12
3.2.3. Pure ACK . . . . . . . . . . . . . . . . . . . . . . 13 3.2.3. Pure ACK (Send) . . . . . . . . . . . . . . . . . . . 13
3.2.4. Window Probe . . . . . . . . . . . . . . . . . . . . 14 3.2.4. Window Probe (Send) . . . . . . . . . . . . . . . . . 14
3.2.5. FIN . . . . . . . . . . . . . . . . . . . . . . . . . 14 3.2.5. FIN (Send) . . . . . . . . . . . . . . . . . . . . . 15
3.2.6. RST . . . . . . . . . . . . . . . . . . . . . . . . . 15 3.2.6. RST (Send) . . . . . . . . . . . . . . . . . . . . . 15
3.2.7. Retransmissions . . . . . . . . . . . . . . . . . . . 15 3.2.7. Retransmissions (Send) . . . . . . . . . . . . . . . 16
3.2.8. General Fall-back for any Control Packet or 3.2.8. General Fall-back for any Control Packet or
Retransmission . . . . . . . . . . . . . . . . . . . 16 Retransmission . . . . . . . . . . . . . . . . . . . 16
4. Rationale . . . . . . . . . . . . . . . . . . . . . . . . . . 16 3.3. Receiver Behaviour . . . . . . . . . . . . . . . . . . . 16
4.1. The Reliability Argument . . . . . . . . . . . . . . . . 16 3.3.1. Receiver Behaviour for Any TCP Control Packet or
4.2. SYNs . . . . . . . . . . . . . . . . . . . . . . . . . . 17 Retransmission . . . . . . . . . . . . . . . . . . . 17
4.2.1. Argument 1a: Unrecognized CE on the SYN . . . . . . . 17 3.3.2. SYN (Receive) . . . . . . . . . . . . . . . . . . . . 17
4.2.2. Argument 1b: Unrecognized ECT on the SYN . . . . . . 19 3.3.3. Pure ACK (Receive) . . . . . . . . . . . . . . . . . 18
4.2.3. Argument 2: DoS Attacks . . . . . . . . . . . . . . . 21 3.3.4. FIN (Receive) . . . . . . . . . . . . . . . . . . . . 18
4.3. SYN-ACKs . . . . . . . . . . . . . . . . . . . . . . . . 22 3.3.5. RST (Receive) . . . . . . . . . . . . . . . . . . . . 18
4.3.1. Response to Congestion on a SYN-ACK . . . . . . . . . 22 3.3.6. Retransmissions (Receive) . . . . . . . . . . . . . . 19
4.3.2. Fall-Back if ECT SYN-ACK Fails . . . . . . . . . . . 23 4. Rationale . . . . . . . . . . . . . . . . . . . . . . . . . . 19
4.4. Pure ACKs . . . . . . . . . . . . . . . . . . . . . . . . 23 4.1. The Reliability Argument . . . . . . . . . . . . . . . . 19
4.4.1. Cwnd Response to CE-Marked Pure ACKs . . . . . . . . 25 4.2. SYNs . . . . . . . . . . . . . . . . . . . . . . . . . . 20
4.4.2. ACK Rate Response to CE-Marked Pure ACKs . . . . . . 26 4.2.1. Argument 1a: Unrecognized CE on the SYN . . . . . . . 20
4.4.3. Summary: Enabling ECN on Pure ACKs . . . . . . . . . 26 4.2.2. Argument 1b: ECT Considered Invalid on the SYN . . . 21
4.5. Window Probes . . . . . . . . . . . . . . . . . . . . . . 27 4.2.3. Caching Strategies for ECT on SYNs . . . . . . . . . 23
4.6. FINs . . . . . . . . . . . . . . . . . . . . . . . . . . 28 4.2.4. Argument 2: DoS Attacks . . . . . . . . . . . . . . . 25
4.7. RSTs . . . . . . . . . . . . . . . . . . . . . . . . . . 28 4.3. SYN-ACKs . . . . . . . . . . . . . . . . . . . . . . . . 26
4.8. Retransmitted Packets. . . . . . . . . . . . . . . . . . 29 4.3.1. Possibility of Unrecognized CE on the SYN-ACK . . . . 26
4.9. General Fall-back for any Control Packet . . . . . . . . 30 4.3.2. Response to Congestion on a SYN-ACK . . . . . . . . . 26
5. Interaction with popular variants or derivatives of TCP . . . 31 4.3.3. Fall-Back if ECT SYN-ACK Fails . . . . . . . . . . . 28
5.1. IW10 . . . . . . . . . . . . . . . . . . . . . . . . . . 31 4.4. Pure ACKs . . . . . . . . . . . . . . . . . . . . . . . . 28
5.2. TFO . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 4.4.1. Mechanisms to Respond to CE-Marked Pure ACKs . . . . 29
5.3. TCP Derivatives . . . . . . . . . . . . . . . . . . . . . 33 4.4.2. Summary: Enabling ECN on Pure ACKs . . . . . . . . . 32
6. Security Considerations . . . . . . . . . . . . . . . . . . . 33 4.5. Window Probes . . . . . . . . . . . . . . . . . . . . . . 33
7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 33 4.6. FINs . . . . . . . . . . . . . . . . . . . . . . . . . . 34
8. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 33 4.7. RSTs . . . . . . . . . . . . . . . . . . . . . . . . . . 34
9. References . . . . . . . . . . . . . . . . . . . . . . . . . 34 4.8. Retransmitted Packets. . . . . . . . . . . . . . . . . . 35
9.1. Normative References . . . . . . . . . . . . . . . . . . 34 4.9. General Fall-back for any Control Packet . . . . . . . . 36
9.2. Informative References . . . . . . . . . . . . . . . . . 34 5. Interaction with popular variants or derivatives of TCP . . . 37
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 37 5.1. IW10 . . . . . . . . . . . . . . . . . . . . . . . . . . 37
5.2. TFO . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
5.3. TCP Derivatives . . . . . . . . . . . . . . . . . . . . . 39
6. Security Considerations . . . . . . . . . . . . . . . . . . . 39
7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 39
8. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 39
9. References . . . . . . . . . . . . . . . . . . . . . . . . . 40
9.1. Normative References . . . . . . . . . . . . . . . . . . 40
9.2. Informative References . . . . . . . . . . . . . . . . . 40
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 43
1. Introduction 1. Introduction
RFC 3168 [RFC3168] specifies support of Explicit Congestion RFC 3168 [RFC3168] specifies support of Explicit Congestion
Notification (ECN) in IP (v4 and v6). By using the ECN capability, Notification (ECN) in IP (v4 and v6). By using the ECN capability,
network elements (e.g. routers, switches) performing Active Queue network elements (e.g. routers, switches) performing Active Queue
Management (AQM) can use ECN marks instead of packet drops to signal Management (AQM) can use ECN marks instead of packet drops to signal
congestion to the endpoints of a communication. This results in congestion to the endpoints of a communication. This results in
lower packet loss and increased performance. RFC 3168 also specifies lower packet loss and increased performance. RFC 3168 also specifies
support for ECN in TCP, but solely on data packets. For various support for ECN in TCP, but solely on data packets. For various
reasons it precludes the use of ECN on TCP control packets (TCP SYN, reasons it precludes the use of ECN on TCP control packets (TCP SYN,
TCP SYN-ACK, pure ACKs, Window probes) and on retransmitted packets. TCP SYN-ACK, pure ACKs, Window probes) and on retransmitted packets.
RFC 3168 is silent about the use of ECN on RST and FIN packets. RFC RFC 3168 is silent about the use of ECN on RST and FIN packets. RFC
5562 [RFC5562] is an experimental modification to ECN that enables 5562 [RFC5562] is an experimental modification to ECN that enables
ECN support for TCP SYN-ACK packets. ECN support for TCP SYN-ACK packets.
This document defines an experimental modification to ECN [RFC3168] This document defines an experimental modification to ECN [RFC3168]
that shall be called ECN++. It enables ECN support on all the that shall be called ECN++. It enables ECN support on all the
aforementioned types of TCP packet. aforementioned types of TCP packet.
ECN++ is a sender-side change. It works whether the two ends of the ECN++ uses a sender-only deployment model. It works whether the two
TCP connection use classic ECN feedback [RFC3168] or experimental ends of the TCP connection use classic ECN feedback [RFC3168] or
Accurate ECN feedback (AccECN [I-D.ietf-tcpm-accurate-ecn]). experimental Accurate ECN feedback (AccECN
Nonetheless, if the client does not implement AccECN, it cannot use [I-D.ietf-tcpm-accurate-ecn]). Nonetheless, if the client does not
ECN++ on the one packet that offers most benefit from it - the implement AccECN, it cannot use ECN++ on the one packet that offers
initial SYN. Therefore, implementers of ECN++ are RECOMMENDED to most benefit from it - the initial SYN. Therefore, implementers of
also implement AccECN. ECN++ are RECOMMENDED to also implement AccECN.
ECN++ is designed for compatibility with a number of latency ECN++ is designed for compatibility with a number of latency
improvements to TCP such as TCP Fast Open (TFO [RFC7413]), initial improvements to TCP such as TCP Fast Open (TFO [RFC7413]), initial
window of 10 SMSS (IW10 [RFC6928]) and Low latency Low Loss Scalable window of 10 SMSS (IW10 [RFC6928]) and Low latency Low Loss Scalable
Transport (L4S [I-D.ietf-tsvwg-l4s-arch]), but they can all be Transport (L4S [I-D.ietf-tsvwg-l4s-arch]), but they can all be
implemented and deployed independently. implemented and deployed independently. [RFC8311] is a standards
[I-D.ietf-tsvwg-ecn-experimentation] is a standards track procedural track procedural device that relaxes requirements in RFC 3168 and
device that relaxes requirements in RFC 3168 and other standards other standards track RFCs that would otherwise preclude the
track RFCs that would otherwise preclude the experimental experimental modifications needed for ECN++ and other ECN
modifications needed for ECN++ and other ECN experiments. experiments.
1.1. Motivation 1.1. Motivation
The absence of ECN support on TCP control packets and retransmissions The absence of ECN support on TCP control packets and retransmissions
has a potential harmful effect. In any ECN deployment, non-ECN- has a potential harmful effect. In any ECN deployment, non-ECN-
capable packets suffer a penalty when they traverse a congested capable packets suffer a penalty when they traverse a congested
bottleneck. For instance, with a drop probability of 1%, 1% of bottleneck. For instance, with a drop probability of 1%, 1% of
connection attempts suffer a timeout of about 1 second before the SYN connection attempts suffer a timeout of about 1 second before the SYN
is retransmitted, which is highly detrimental to the performance of is retransmitted, which is highly detrimental to the performance of
short flows. TCP control packets, particularly TCP SYNs and SYN- short flows. TCP control packets, particularly TCP SYNs and SYN-
skipping to change at page 4, line 25 skipping to change at page 4, line 47
probability for non-ECN packets. Therefore non-ECN SYNs are dropped probability for non-ECN packets. Therefore non-ECN SYNs are dropped
aggressively, rendering it nearly impossible to establish a new aggressively, rendering it nearly impossible to establish a new
connection in the presence of even mild traffic load. connection in the presence of even mild traffic load.
Finally, there are ongoing experimental efforts to promote the Finally, there are ongoing experimental efforts to promote the
adoption of a slightly modified variant of DCTCP (and similar adoption of a slightly modified variant of DCTCP (and similar
congestion controls) over the Internet to achieve low latency, low congestion controls) over the Internet to achieve low latency, low
loss and scalable throughput (L4S) for all communications loss and scalable throughput (L4S) for all communications
[I-D.ietf-tsvwg-l4s-arch]. In such an approach, L4S packets identify [I-D.ietf-tsvwg-l4s-arch]. In such an approach, L4S packets identify
themselves using an ECN codepoint [I-D.ietf-tsvwg-ecn-l4s-id]. With themselves using an ECN codepoint [I-D.ietf-tsvwg-ecn-l4s-id]. With
L4S and potentially other similar cases, preventing TCP control L4S, preventing TCP control packets from obtaining the benefits of
packets from obtaining the benefits of ECN would not only expose them ECN would not only expose them to the prevailing level of congestion
to the prevailing level of congestion loss, but it would also loss, but it would also classify them into a different queue. Then
classify control packets into a different queue with different only L4S data packets would enjoy ultra-low latency forwarding, while
network treatment, which may also lead to reordering, further the packets controlling and retransmitting these data packets would
degrading TCP performance. still get stuck behind the queue induced by legacy ('Classic') TCP
traffic.
1.2. Experiment Goals 1.2. Experiment Goals
The goal of the experimental modifications defined in this document The goal of the experimental modifications defined in this document
is to allow the use of ECN on all TCP packets. Experiments are is to allow the use of ECN on all TCP packets. Experiments are
expected in the public Internet as well as in controlled environments expected in the public Internet as well as in controlled environments
to understand the following issues: to understand the following issues:
o How SYNs, Window probes, pure ACKs, FINs, RSTs and retransmissions o How SYNs, Window probes, pure ACKs, FINs, RSTs and retransmissions
that carry the ECT(0), ECT(1) or CE codepoints are processed by that carry the ECT(0), ECT(1) or CE codepoints are processed by
skipping to change at page 5, line 9 skipping to change at page 5, line 32
including [RFC3168], [RFC5562], [RFC3540] and including [RFC3168], [RFC5562], [RFC3540] and
[I-D.ietf-tcpm-accurate-ecn]. [I-D.ietf-tcpm-accurate-ecn].
o How much the performance of TCP communications is improved by o How much the performance of TCP communications is improved by
allowing ECN marking of each packet type. allowing ECN marking of each packet type.
o To identify any issues (including security issues) raised by o To identify any issues (including security issues) raised by
enabling ECN marking of these packets. enabling ECN marking of these packets.
The data gathered through the experiments described in this document, The data gathered through the experiments described in this document,
particularly under the first 2 bullets above, will help in the design particularly under the first 2 bullets above, will help in the
of the final mechanism (if any) for adding ECN support to the redesign of the final mechanism (if needed) for adding ECN support to
different packet types considered in this document. Whenever data the different packet types considered in this document. Whenever
input is needed to assist in a design choice, it is spelled out data input is needed to assist in a design choice, it is spelled out
throughout the document. throughout the document.
Success criteria: The experiment will be a success if we obtain Success criteria: The experiment will be a success if we obtain
enough data to have a clearer view of the deployability and benefits enough data to have a clearer view of the deployability and benefits
of enabling ECN on all TCP packets, as well as any issues. If the of enabling ECN on all TCP packets, as well as any issues. If the
results of the experiment show that it is feasible to deploy such results of the experiment show that it is feasible to deploy such
changes; that there are gains to be achieved through the changes changes; that there are gains to be achieved through the changes
described in this specification; and that no other major issues may described in this specification; and that no other major issues may
interfere with the deployment of the proposed changes; then it would interfere with the deployment of the proposed changes; then it would
be reasonable to adopt the proposed changes in a standards track be reasonable to adopt the proposed changes in a standards track
skipping to change at page 5, line 43 skipping to change at page 6, line 23
specification that will be necessary to interwork with a number of specification that will be necessary to interwork with a number of
popular variants or derivatives of TCP. RFC 3168 provides a number popular variants or derivatives of TCP. RFC 3168 provides a number
of specific reasons why ECN support is not appropriate for each of specific reasons why ECN support is not appropriate for each
packet type. In Section 4, we revisit each of these arguments for packet type. In Section 4, we revisit each of these arguments for
each packet type to justify why it is reasonable to conduct this each packet type to justify why it is reasonable to conduct this
experiment. experiment.
2. Terminology 2. Terminology
The keywords MUST, MUST NOT, REQUIRED, SHALL, SHALL NOT, SHOULD, The keywords MUST, MUST NOT, REQUIRED, SHALL, SHALL NOT, SHOULD,
SHOULD NOT, RECOMMENDED, MAY, and OPTIONAL, when they appear in this SHOULD NOT, RECOMMENDED, NOT RECOMMENDED, MAY, and OPTIONAL in this
document, are to be interpreted as described in [RFC2119]. document, are to be interpreted as described in BCP 14 [RFC2119] when
and only when they appear in all capitals.
Pure ACK: A TCP segment with the ACK flag set and no data payload. Pure ACK: A TCP segment with the ACK flag set and no data payload.
SYN: A TCP segment with the SYN (synchronize) flag set. SYN: A TCP segment with the SYN (synchronize) flag set.
Window probe: Defined in [RFC0793], a window probe is a TCP segment Window probe: Defined in [RFC0793], a window probe is a TCP segment
with only one byte of data sent to learn if the receive window is with only one byte of data sent to learn if the receive window is
still zero. still zero.
FIN: A TCP segment with the FIN (finish) flag set. FIN: A TCP segment with the FIN (finish) flag set.
RST: A TCP segment with the RST (reset) flag set. RST: A TCP segment with the RST (reset) flag set.
Retransmission: A TCP segment that has been retransmitted by the TCP Retransmission: A TCP segment that has been retransmitted by the TCP
sender. sender.
TCP client: The initiating end of a TCP connection. Also called the
initiator.
TCP server: The responding end of a TCP connection. Also called the
responder.
ECT: ECN-Capable Transport. One of the two codepoints ECT(0) or ECT: ECN-Capable Transport. One of the two codepoints ECT(0) or
ECT(1) in the ECN field [RFC3168] of the IP header (v4 or v6). An ECT(1) in the ECN field [RFC3168] of the IP header (v4 or v6). An
ECN-capable sender sets one of these to indicate that both transport ECN-capable sender sets one of these to indicate that both transport
end-points support ECN. When this specification says the sender sets end-points support ECN. When this specification says the sender sets
an ECT codepoint, by default it means ECT(0). Optionally, it could an ECT codepoint, by default it means ECT(0). Optionally, it could
mean ECT(1), which is in the process of being redefined for use by mean ECT(1), which is in the process of being redefined for use by
L4S experiments [I-D.ietf-tsvwg-ecn-experimentation] L4S experiments [RFC8311] [I-D.ietf-tsvwg-ecn-l4s-id].
[I-D.ietf-tsvwg-ecn-l4s-id].
Not-ECT: The ECN codepoint set by senders that indicates that the Not-ECT: The ECN codepoint set by senders that indicates that the
transport is not ECN-capable. transport is not ECN-capable.
CE: Congestion Experienced. The ECN codepoint that an intermediate CE: Congestion Experienced. The ECN codepoint that an intermediate
node sets to indicate congestion [RFC3168]. A node sets an node sets to indicate congestion [RFC3168]. A node sets an
increasing proportion of ECT packets to CE as the level of congestion increasing proportion of ECT packets to CE as the level of congestion
increases. increases.
3. Specification 3. Specification
The experimental ECN++ changes to the specification of TCP over ECN
[RFC3168] defined here primarily alter the behaviour of the sending
host for each half-connection. However, there are subsections for
forwarding elements and receivers below, which recommend that they
accept the new packets - they should do already, but might not. This
will allow implementers to check the receive side code while they are
altering the send-side code. All changes can be deployed at each
end-point independently of others and independent of any network
behaviour.
The feedback behaviour at the receiver depends on whether classic ECN
TCP feedback [RFC3168] or Accurate ECN (AccECN) TCP feedback
[I-D.ietf-tcpm-accurate-ecn] has been negotiated. Nonetheless,
neither receiver feedback behaviour is altered by the present
specification.
3.1. Network (e.g. Firewall) Behaviour 3.1. Network (e.g. Firewall) Behaviour
Previously the specification of ECN for TCP [RFC3168] required the Previously the specification of ECN for TCP [RFC3168] required the
sender to set not-ECT on TCP control packets and retransmissions. sender to set not-ECT on TCP control packets and retransmissions.
Some readers of RFC 3168 might have erroneously interpreted this as a Some readers of RFC 3168 might have erroneously interpreted this as a
requirement for firewalls, intrusion detection systems, etc. to check requirement for firewalls, intrusion detection systems, etc. to check
and enforce this behaviour. Section 4.3 of and enforce this behaviour. Section 4.3 of [RFC8311] updates RFC
[I-D.ietf-tsvwg-ecn-experimentation] updates RFC 3168 to remove this 3168 to remove this ambiguity. It requires firewalls or any
ambiguity. It require firewalls or any intermediate nodes not to intermediate nodes not to treat certain types of ECN-capable TCP
treat certain types of ECN-capable TCP segment differently (except segment differently (except potentially in one attack scenario).
potentially in one attack scenario). This is likely to only involve This is likely to only involve a firewall rule change in a fraction
a firewall rule change in a fraction of cases (at most 0.4% of paths of cases (at most 0.4% of paths according to the tests reported in
according to the tests reported in Section 4.2.2). Section 4.2.2).
In case a TCP sender encounters a middlebox blocking ECT on certain In case a TCP sender encounters a middlebox blocking ECT on certain
TCP segments, the specification below includes behaviour to fall back TCP segments, the specification below includes behaviour to fall back
to non-ECN. However, this loses the benefit of ECN on control to non-ECN. However, this loses the benefit of ECN on control
packets. So operators are RECOMMENDED to alter their firewall rules packets. So operators are RECOMMENDED to alter their firewall rules
to comply with the requirement referred to above (section 4.3 of to comply with the requirement referred to above (section 4.3 of
[I-D.ietf-tsvwg-ecn-experimentation]). [RFC8311]).
3.2. Endpoint Behaviour
The changes to the specification of TCP over ECN [RFC3168] defined
here solely alter the behaviour of the sending host for each half-
connection. All changes can be deployed at each end-point
independently of others and independent of any network behaviour.
The feedback behaviour at the receiver depends on whether classic ECN 3.2. Sender Behaviour
TCP feedback [RFC3168] or Accurate ECN (AccECN) TCP feedback
[I-D.ietf-tcpm-accurate-ecn] has been negotiated. Nonetheless,
neither receiver feedback behaviour is altered by the present
specification.
For each type of control packet or retransmission, the following For each type of control packet or retransmission, the following
sections detail changes to the sender's behaviour in two respects: i) sections detail changes to the sender's behaviour in two respects: i)
whether it sets ECT; and ii) its response to congestion feedback. whether it sets ECT; and ii) its response to congestion feedback.
Table 1 summarises these two behaviours for each type of packet, but Table 1 summarises these two behaviours for each type of packet, but
the relevant subsection below should be referred to for the detailed the relevant subsection below should be referred to for the detailed
behaviour. The subsection on the SYN is more complex than the behaviour. The subsection on the SYN is more complex than the
others, because it has to include fall-back behaviour if the ECT others, because it has to include fall-back behaviour if the ECT
packet appears not to have got through, and caching of the outcome to packet appears not to have got through, and caching of the outcome to
detect persistent failures. detect persistent failures.
+---------+-----------------+------------------+--------------------+ +---------+----------------+-----------------+----------------------+
| TCP | ECN field if | ECN field if | Congestion | | TCP | ECN field if | ECN field if | Congestion Response |
| packet | AccECN f/b | RFC3168 f/b | Response | | packet | AccECN f/b | RFC3168 f/b | |
| type | negotiated* | negotiated* | | | type | negotiated* | negotiated* | |
+---------+-----------------+------------------+--------------------+ +---------+----------------+-----------------+----------------------+
| SYN | ECT | not-ECT | Reduce IW | | SYN | ECT | not-ECT | If AccECN, reduce IW |
| | | | | | | | | |
| SYN-ACK | ECT | ECT | Reduce IW | | SYN-ACK | ECT | ECT | Reduce IW |
| | | | | | | | | |
| Pure | ECT | ECT | Usual cwnd | | Pure | ECT | not-ECT | If AccECN, usual |
| ACK | | | response and | | ACK | | | cwnd response and |
| | | | optionally | | | | | optionally [RFC5690] |
| | | | [RFC5690] | | | | | |
| | | | | | W Probe | ECT | ECT | Usual cwnd response |
| W Probe | ECT | ECT | Usual cwnd | | | | | |
| | | | response | | FIN | ECT | ECT | None or optionally |
| | | | | | | | | [RFC5690] |
| FIN | ECT | ECT | None or optionally | | | | | |
| | | | [RFC5690] | | RST | ECT | ECT | N/A |
| | | | | | | | | |
| RST | ECT | ECT | N/A | | Re-XMT | ECT | ECT | Usual cwnd response |
| | | | | +---------+----------------+-----------------+----------------------+
| Re-XMT | ECT | ECT | Usual cwnd |
| | | | response |
+---------+-----------------+------------------+--------------------+
Window probe and retransmission are abbreviated to W Probe an Re-XMT. Window probe and retransmission are abbreviated to W Probe an Re-XMT.
* For a SYN, "negotiated" means "requested". * For a SYN, "negotiated" means "requested".
Table 1: Summary of sender behaviour. In each case the relevant Table 1: Summary of sender behaviour. In each case the relevant
section below should be referred to for the detailed behaviour section below should be referred to for the detailed behaviour
It can be seen that the sender can set ECT in all cases, except if it It can be seen that the sender cannot set ECT on the SYN if it is not
is not requesting AccECN feedback on the SYN. Therefore it is requesting AccECN feedback. Therefore it is RECOMMENDED that the
RECOMMENDED that the experimental AccECN specification experimental AccECN specification [I-D.ietf-tcpm-accurate-ecn] is
[I-D.ietf-tcpm-accurate-ecn] is implemented (as well as the present implemented (as well as the present specification), because it is
specification), because it is expected that ECT on the SYN will give expected that ECT on the SYN will give the most significant
the most significant performance gain, particularly for short flows. performance gain, particularly for short flows.
Nonetheless, this specification also caters for the case where AccECN
feedback is not implemented.
3.2.1. SYN Nonetheless, this specification also caters for the case where an
ECN++ TCP sender is not using AccECN. This could be because it does
not support AccECN or because the other end of the TCP connection
does not (AccECN can only be used for a connection if both ends
support it).
3.2.1. SYN (Send)
3.2.1.1. Setting ECT on the SYN 3.2.1.1. Setting ECT on the SYN
With classic [RFC3168] ECN feedback, the SYN was never expected to be With classic [RFC3168] ECN feedback, the SYN was not expected to be
ECN-capable, so the flag provided to feed back congestion was put to ECN-capable, so the flag provided to feed back congestion was put to
another use (it is used in combination with other flags to indicate another use (it is used in combination with other flags to indicate
that the responder supports ECN). In contrast, Accurate ECN (AccECN) that the responder supports ECN). In contrast, Accurate ECN (AccECN)
feedback [I-D.ietf-tcpm-accurate-ecn] provides two codepoints in the feedback [I-D.ietf-tcpm-accurate-ecn] provides a codepoint in the
SYN-ACK for the responder to feed back whether or not the SYN arrived SYN-ACK for the responder to feed back whether the SYN arrived marked
marked CE. CE. Therefore the setting of the IP/ECN field on the SYN is
specified separately for each case in the following two subsections.
Therefore, a TCP initiator MUST NOT set ECT on a SYN unless it also 3.2.1.1.1. ECN++ TCP Client also Supports AccECN
attempts to negotiate Accurate ECN feedback in the same SYN.
For the experiments proposed here, if the SYN is requesting AccECN For the ECN++ experiment, if the SYN is requesting AccECN feedback,
feedback, the TCP sender will also set ECT on the SYN. It can ignore the TCP sender will also set ECT on the SYN. It can ignore the
the prohibition in section 6.1.1 of RFC 3168 against setting ECT on prohibition in section 6.1.1 of RFC 3168 against setting ECT on such
such a SYN. a SYN, as per Section 4.3 of [RFC8311].
The following subsections about the SYN solely apply to this case 3.2.1.1.2. ECN++ TCP Client does not Support AccECN
where the initiator sent an ECT SYN.
3.2.1.2. Caching Lack of AccECN Support for ECT on SYNs A TCP initiator MUST NOT set ECT on a SYN if it does not also attempt
to negotiate Accurate ECN feedback in the same SYN.
If the TCP initiator does not support AccECN, the rest of
Section 3.2.1 does not apply. It solely applies to the case where
the TCP initiator supports AccECN as well as ECN++.
3.2.1.2. Caching where to use ECT on SYNs
As explained above, this subsection only applies if the ECN++ TCP
client also supports AccECN.
Until AccECN servers become widely deployed, a TCP initiator that Until AccECN servers become widely deployed, a TCP initiator that
sets ECT on a SYN (which implies the same SYN also requests AccECN, sets ECT on a SYN (which implies the same SYN also requests AccECN,
as required above) SHOULD also maintain a cache entry per server to as required above) SHOULD also maintain a cache entry per server to
record that the server does not support AccECN and therefore has no record servers that it is not worth sending an ECT SYN to, e.g.
logic for congestion markings on the SYN. Mobile hosts MAY maintain because they do not support AccECN and therefore have no logic for
a cache entry per access network to record lack of AccECN support by congestion markings on the SYN. Mobile hosts MAY maintain a cache
entry per access network to record 'non-ECT SYN' entries against
proxies (see Section 4.2.1). proxies (see Section 4.2.1).
The initiator will record any server's SYN-ACK response that does not Subsequently the initiator will not set ECT on a SYN to such a server
support AccECN. Subsequently the initiator will not set ECT on a SYN or proxy, but it can still always request AccECN support (because the
to such a server, but it can still always request AccECN support response will state any earlier stage of ECN evolution that the
(because the response will state any earlier stage of ECN evolution server supports with no performance penalty). The initiator will
that the server supports with no performance penalty). The initiator discover a server that has upgraded to support AccECN as soon as it
will discover a server that has upgraded to support AccECN as soon as next connects, then it can remove the server from its cache and
it next connects, then it can remove the server from its cache and
subsequently always set ECT for that server. subsequently always set ECT for that server.
If the initiator times out without seeing a SYN-ACK, it will also The client can limit the size of its cache of 'non-ECT SYN' servers.
cache this fact (see fall-back in Section 3.2.1.4 for details). Then, while AccECN is not widely deployed, it will only cache the
'non-ECT SYN' servers that are most used and most recently used by
the client. As the client accesses servers that have been expelled
from its cache, it will simply use ECT on the SYN by default.
There is no need to cache successful attempts, because the default Servers that do not support ECN as a whole do not need to be recorded
ECT SYN behaviour performs optimally on success anyway. Servers that
do not support ECN as a whole probably do not need to be recorded
separately from non-support of AccECN because the response to a separately from non-support of AccECN because the response to a
request for AccECN immediately states which stage in the evolution of request for AccECN immediately states which stage in the evolution of
ECN the server supports (AccECN [I-D.ietf-tcpm-accurate-ecn], classic ECN the server supports (AccECN [I-D.ietf-tcpm-accurate-ecn], classic
ECN [RFC3168] or no ECN). ECN [RFC3168] or no ECN).
The above strategy is named "optimistic ECT and cache failures". It The above strategy is named "optimistic ECT and cache failures". It
is believed to be sufficient based on initial measurements and is believed to be sufficient based on three measurement studies and
assumptions detailed in Section 4.2.1, which also gives alternative assumptions detailed in Section 4.2.1. However, Section 4.2.1 gives
strategies in case larger scale measurements uncover different two other strategies and the choice between them depends on the
scenarios. implementer's goals and the deployment prevalence of ECN variants in
the network and on servers, not to mention the prevalence of some
significant bugs.
If the initiator times out without seeing a SYN-ACK, it will
separately cache this fact (see fall-back in Section 3.2.1.4 for
details).
3.2.1.3. SYN Congestion Response 3.2.1.3. SYN Congestion Response
As explained above, this subsection only applies if the ECN++ TCP
client also supports AccECN.
If the SYN-ACK returned to the TCP initiator confirms that the server If the SYN-ACK returned to the TCP initiator confirms that the server
supports AccECN, it will also indicate whether or not the SYN was CE- supports AccECN, it will also indicate whether or not the SYN was CE-
marked. If the SYN was CE-marked, the initiator MUST reduce its marked. If the SYN was CE-marked, the initiator MUST reduce its
Initial Window (IW) and SHOULD reduce it to 1 SMSS (sender maximum Initial Window (IW) and SHOULD reduce it to 1 SMSS (sender maximum
segment size). segment size).
If ECT has been set on the SYN and if the SYN-ACK shows that the If the initiator has set ECT on the SYN and if the SYN-ACK shows that
server does not support AccECN, the TCP initiator MUST conservatively the server does not support AccECN, the TCP initiator MUST
reduce its Initial Window and SHOULD reduce it to 1 SMSS. A conservatively reduce its Initial Window and SHOULD reduce it to 1
reduction to greater than 1 SMSS MAY be appropriate (see SMSS. A reduction to greater than 1 SMSS MAY be appropriate (see
Section 4.2.1). Conservatism is necessary because a non-AccECN SYN- Section 4.2.1). Conservatism is necessary because a non-AccECN SYN-
ACK cannot show whether the SYN was CE-marked. ACK cannot show whether the SYN was CE-marked.
If the TCP initiator (host A) receives a SYN from the remote end If the TCP initiator (host A) receives a SYN from the remote end
(host B) after it has sent a SYN to B, it indicates the (unusual) (host B) after it has sent a SYN to B, it indicates the (unusual)
case of a simultaneous open. Host A will respond with a SYN-ACK. case of a simultaneous open. Host A will respond with a SYN-ACK.
Host A will probably then receive a SYN-ACK in response to its own Host A will probably then receive a SYN-ACK in response to its own
SYN, after which it can follow the appropriate one of the two SYN, after which it can follow the appropriate one of the two
paragraphs above. paragraphs above.
skipping to change at page 10, line 42 skipping to change at page 11, line 31
no response to its SYN [RFC6298], because both the SYN and the SYN- no response to its SYN [RFC6298], because both the SYN and the SYN-
ACK have been successfully delivered through the network. Also, the ACK have been successfully delivered through the network. Also, the
initiator does not need to exit slow start or reduce ssthresh, which initiator does not need to exit slow start or reduce ssthresh, which
is not even required when a SYN is lost [RFC5681]. is not even required when a SYN is lost [RFC5681].
If an initial window of 10 (IW10 [RFC6928]) is implemented, Section 5 If an initial window of 10 (IW10 [RFC6928]) is implemented, Section 5
gives additional recommendations. gives additional recommendations.
3.2.1.4. Fall-Back Following No Response to an ECT SYN 3.2.1.4. Fall-Back Following No Response to an ECT SYN
As explained above, this subsection only applies if the ECN++ TCP
client also supports AccECN.
An ECT SYN might be lost due to an over-zealous path element (or An ECT SYN might be lost due to an over-zealous path element (or
server) blocking ECT packets that do not conform to RFC 3168. Some server) blocking ECT packets that do not conform to RFC 3168. Some
evidence of this was found in a 2014 study [ecn-pam], but in a more evidence of this was found in a 2014 study [ecn-pam], but in a more
recent study using 2017 data [Mandalari18] extensive measurements recent study using 2017 data [Mandalari18] extensive measurements
found no case where ECT on TCP control packets was treated any found no case where ECT on TCP control packets was treated any
differently from ECT on TCP data packets. Loss is commonplace for differently from ECT on TCP data packets. Loss is commonplace for
numerous other reasons, e.g. congestion loss at a non-ECN queue on numerous other reasons, e.g. congestion loss at a non-ECN queue on
the forward or reverse path, transmission errors, etc. the forward or reverse path, transmission errors, etc.
Alternatively, the cause of the loss might be the attempt to Alternatively, the cause of the loss might be the attempt to
negotiate AccECN, or possibly other unrelated options on the SYN. negotiate AccECN, or possibly other unrelated options on the SYN.
skipping to change at page 11, line 29 skipping to change at page 12, line 21
commonplace for other reasons. Even if it does eventually decide to commonplace for other reasons. Even if it does eventually decide to
give up setting ECT on the SYN, it will probably not need to give up give up setting ECT on the SYN, it will probably not need to give up
on AccECN on the SYN. In any case, if a cache is used, it SHOULD be on AccECN on the SYN. In any case, if a cache is used, it SHOULD be
arranged to expire so that the initiator will infrequently attempt to arranged to expire so that the initiator will infrequently attempt to
check whether the problem has been resolved. check whether the problem has been resolved.
Other fall-back strategies MAY be adopted where applicable (see Other fall-back strategies MAY be adopted where applicable (see
Section 4.2.2 for suggestions, and the conditions under which they Section 4.2.2 for suggestions, and the conditions under which they
would apply). would apply).
3.2.2. SYN-ACK 3.2.2. SYN-ACK (Send)
3.2.2.1. Setting ECT on the SYN-ACK 3.2.2.1. Setting ECT on the SYN-ACK
For the experiments proposed here, the TCP implementation will set For the ECN++ experiment, the TCP implementation will set ECT on SYN-
ECT on SYN-ACKs. It can ignore the requirement in section 6.1.1 of ACKs. It can ignore the requirement in section 6.1.1 of RFC 3168 to
RFC 3168 to set not-ECT on a SYN-ACK. set not-ECT on a SYN-ACK, as per Section 4.3 of [RFC8311].
The feedback behaviour by the initiator in response to a CE-marked
SYN-ACK from the responder depends on whether classic ECN feedback
[RFC3168] or AccECN feedback [I-D.ietf-tcpm-accurate-ecn] has been
negotiated. In either case no change is required to RFC 3168 or the
AccECN specification.
Some classic ECN implementations might ignore a CE-mark on a SYN-ACK,
or even ignore a SYN-ACK packet entirely if it is set to ECT or CE.
This is a possibility because an RFC 3168 implementation would not
necessarily expect a SYN-ACK to be ECN-capable.
FOR DISCUSSION: To eliminate this problem, the WG could decide to
prohibit setting ECT on SYN-ACKs unless AccECN has been
negotiated. However, this issue already came up when the IETF
first decided to experiment with ECN on SYN-ACKs [RFC5562] and it
was decided to go ahead without any extra precautionary measures
because the risk was low. This was because the probability of
encountering the problem was believed to be low and the harm if
the problem arose was also low (see Appendix B of RFC 5562).
MEASUREMENTS NEEDED: Server-side experiments could determine
whether this specific problem is indeed rare across the current
installed base of clients that support ECN.
3.2.2.2. SYN-ACK Congestion Response 3.2.2.2. SYN-ACK Congestion Response
A host that sets ECT on SYN-ACKs MUST reduce its initial window in A host that sets ECT on SYN-ACKs MUST reduce its initial window in
response to any congestion feedback, whether using classic ECN or response to any congestion feedback, whether using classic ECN or
AccECN. It SHOULD reduce it to 1 SMSS. This is different to the AccECN (see Section 4.3.1). It SHOULD reduce it to 1 SMSS. This is
behaviour specified in an earlier experiment that set ECT on the SYN- different to the behaviour specified in an earlier experiment that
ACK [RFC5562]. This is justified in Section 4.3. set ECT on the SYN-ACK [RFC5562]. This is justified in Section 4.3.
The responder does not have to back off its retransmission timer The responder does not have to back off its retransmission timer
because the ECN feedback proves that the network is delivering because the ECN feedback proves that the network is delivering
packets successfully and is not severely overloaded. Also the packets successfully and is not severely overloaded. Also the
responder does not have to leave slow start or reduce ssthresh, which responder does not have to leave slow start or reduce ssthresh, which
is not even required when a SYN-ACK has been lost. is not even required when a SYN-ACK has been lost.
The congestion response to CE-marking on a SYN-ACK for a server that The congestion response to CE-marking on a SYN-ACK for a server that
implements either the TCP Fast Open experiment (TFO [RFC7413]) or the implements either the TCP Fast Open experiment (TFO [RFC7413]) or the
initial window of 10 experiment (IW10 [RFC6928]) is discussed in initial window of 10 experiment (IW10 [RFC6928]) is discussed in
skipping to change at page 12, line 49 skipping to change at page 13, line 23
fall-back. It would make sense to co-ordinate all the strategies for fall-back. It would make sense to co-ordinate all the strategies for
fall-back in order to isolate the specific cause of the problem. fall-back in order to isolate the specific cause of the problem.
This fall-back strategy attempts to use ECT one more time than the This fall-back strategy attempts to use ECT one more time than the
strategy for ECT SYN-ACKs in [RFC5562] (which is made obsolete, being strategy for ECT SYN-ACKs in [RFC5562] (which is made obsolete, being
superseded by the present specification). Other fall-back strategies superseded by the present specification). Other fall-back strategies
MAY be adopted if found to be more effective, e.g. fall-back to not- MAY be adopted if found to be more effective, e.g. fall-back to not-
ECT on the first retransmission attempt. ECT on the first retransmission attempt.
The server MAY cache failed connection attempts, e.g. per client The server MAY cache failed connection attempts, e.g. per client
access network. An client-based alternative to caching at the server access network. A client-based alternative to caching at the server
is given in Section 4.3.2. If the TCP server is caching failed is given in Section 4.3.3. If the TCP server is caching failed
connection attempts, it SHOULD NOT give up using ECT on the first connection attempts, it SHOULD NOT give up using ECT on the first
SYN-ACK of subsequent connection attempts until it is clear that the SYN-ACK of subsequent connection attempts until it is clear that the
blockage persistently and specifically affects ECT on SYN-ACKs. This blockage persistently and specifically affects ECT on SYN-ACKs. This
is because loss is so commonplace for other reasons (see is because loss is so commonplace for other reasons (see
Section 3.2.1.4). If a cache is used, it SHOULD be arranged to Section 3.2.1.4). If a cache is used, it SHOULD be arranged to
expire so that the server will infrequently attempt to check whether expire so that the server will infrequently attempt to check whether
the problem has been resolved. the problem has been resolved.
3.2.3. Pure ACK 3.2.3. Pure ACK (Send)
For the experiments proposed here, the TCP implementation will set A Pure ACK is an ACK packet that does not carry data, which includes
ECT on pure ACKs. It can ignore the requirement in section 6.1.4 of the Pure ACK at the end of TCP's 3-way handshake.
RFC 3168 to set not-ECT on a pure ACK.
A host that sets ECT on pure ACKs MUST reduce its congestion window For the ECN++ experiment, whether a TCP implementation sets ECT on a
in response to any congestion feedback, in order to regulate any data Pure ACK depends on whether or not Accurate ECN TCP feedback
segments it might be sending amongst the pure ACKs. {ToDo: Write-up [I-D.ietf-tcpm-accurate-ecn] has been successfully negotiated for a
reconsideration of this requirement in the light of WG comments.} It particular TCP connection, as specified in the following two
MAY also implement AckCC [RFC5690] to regulate the pure ACK rate, but subsections.
this is not required. Note that, in comparison, TCP Congestion
Control [RFC5681] does not require a TCP to detect or respond to loss
of pure ACKs at all; it requires no reduction in congestion window or
ACK rate.
The question of whether the receiver of pure ACKs is required to feed 3.2.3.1. Pure ACK without AccECN Feedback
back any CE marks on them is a matter for the relevant feedback
specification ([RFC3168] or [I-D.ietf-tcpm-accurate-ecn]). It is
outside the scope of the present specification. Currently AccECN
feedback is required to count CE marking of any control packet
including pure ACKs. Whereas RFC 3168 is silent on this point, so
feedback of CE-markings might be implementation specific (see
Section 4.4.1).
DISCUSSION: An AccECN deployment or an implementation of RFC 3168 If AccECN has not been successfully negotiated for a connection, ECT
that feeds back CE on pure ACKs will be at a disadvantage compared MUST NOT be set on Pure ACKs by either end.
to an RFC 3168 implementation that does not. To solve this, the
WG could decide to prohibit setting ECT on pure ACKs unless AccECN 3.2.3.2. Pure ACK with AccECN Feedback
has been negotiated. If it does, the penultimate sentence of the
Introduction will need to be modified. For the ECN++ experiment, if AccECN has been successfully negotiated,
either end of the connection will set ECT on Pure ACKs. They can
ignore the requirement in section 6.1.4 of RFC 3168 to set not-ECT on
a pure ACK, as per Section 4.3 of [RFC8311].
MEASUREMENTS NEEDED: Measurements are needed to learn how the MEASUREMENTS NEEDED: Measurements are needed to learn how the
deployed base of network elements and RFC 3168 servers react to deployed base of network elements and RFC 3168 servers react to
pure ACKs marked with the ECT(0)/ECT(1)/CE codepoints, i.e. pure ACKs marked with the ECT(0)/ECT(1)/CE codepoints, i.e.
whether they are dropped, codepoint cleared or processed and the whether they are dropped, codepoint cleared or processed and the
congestion indication fed back on a subsequent packet. congestion indication fed back on a subsequent packet.
3.2.4. Window Probe See Section 3.3.3 for the implications if a host receives a CE-marked
Pure ACK.
For the experiments proposed here, the TCP sender will set ECT on 3.2.3.2.1. Pure ACK Congestion Response
window probes. It can ignore the prohibition in section 6.1.6 of RFC
3168 against setting ECT on a window probe. As explained above, this subsection only applies if AccECN has been
successfully negotiated for the TCP connection.
A host that sets ECT on pure ACKs SHOULD respond to the congestion
signal resulting from pure ACKs being marked with the CE codepoint.
The specific response will need to be defined as an update to each
congestion control specification. Possible responses to congestion
feedback include reducing the congestion window (CWND) and/or
regulating the pure ACK rate (see Section 4.4.1.1).
Note that, in comparison, TCP Congestion Control [RFC5681] does not
require a TCP to detect or respond to loss of pure ACKs at all; it
requires no reduction in congestion window or ACK rate.
3.2.4. Window Probe (Send)
For the ECN++ experiment, the TCP sender will set ECT on window
probes. It can ignore the prohibition in section 6.1.6 of RFC 3168
against setting ECT on a window probe, as per Section 4.3 of
[RFC8311].
A window probe contains a single octet, so it is no different from a A window probe contains a single octet, so it is no different from a
regular TCP data segment. Therefore a TCP receiver will feed back regular TCP data segment. Therefore a TCP receiver will feed back
any CE marking on a window probe as normal (either using classic ECN any CE marking on a window probe as normal (either using classic ECN
feedback or AccECN feedback). The sender of the probe will then feedback or AccECN feedback). The sender of the probe will then
reduce its congestion window as normal. reduce its congestion window as normal.
A receive window of zero indicates that the application is not A receive window of zero indicates that the application is not
consuming data fast enough and does not imply anything about network consuming data fast enough and does not imply anything about network
congestion. Once the receive window opens, the congestion window congestion. Once the receive window opens, the congestion window
skipping to change at page 14, line 35 skipping to change at page 15, line 19
to present a problem, given the duration between window probes to present a problem, given the duration between window probes
doubles [RFC1122] as long as the receiver is advertising a zero doubles [RFC1122] as long as the receiver is advertising a zero
window (currently minimum 1 second, maximum at least 1 minute window (currently minimum 1 second, maximum at least 1 minute
[RFC6298]). [RFC6298]).
MEASUREMENTS NEEDED: Measurements are needed to learn how the MEASUREMENTS NEEDED: Measurements are needed to learn how the
deployed base of network elements and servers react to Window deployed base of network elements and servers react to Window
probes marked with the ECT(0)/ECT(1)/CE codepoints, i.e. whether probes marked with the ECT(0)/ECT(1)/CE codepoints, i.e. whether
they are dropped, codepoint cleared or processed. they are dropped, codepoint cleared or processed.
3.2.5. FIN 3.2.5. FIN (Send)
A TCP implementation can set ECT on a FIN. A TCP implementation can set ECT on a FIN.
The TCP data receiver MUST ignore the CE codepoint on incoming FINs See Section 3.3.4 for the implications if a host receives a CE-marked
that fail any validity check. The validity check in section 5.2 of FIN.
[RFC5961] is RECOMMENDED.
A congestion response to a CE-marking on a FIN is not required. A congestion response to a CE-marking on a FIN is not required.
After sending a FIN, the endpoint will not send any more data in the After sending a FIN, the endpoint will not send any more data in the
connection. Therefore, even if the FIN-ACK indicates that the FIN connection. Therefore, even if the FIN-ACK indicates that the FIN
was CE-marked (whether using classic or AccECN feedback), reducing was CE-marked (whether using classic or AccECN feedback), reducing
the congestion window will not affect anything. the congestion window will not affect anything.
After sending a FIN, a host might send one or more pure ACKs. If it After sending a FIN, a host might send one or more pure ACKs. If it
is using one of the techniques in Section 3.2.3 to regulate the is using one of the techniques in Section 3.2.3 to regulate the
delayed ACK ratio for pure ACKs, it could equally be applied after a delayed ACK ratio for pure ACKs, it could equally be applied after a
FIN. But this is not required. FIN. But this is not required.
MEASUREMENTS NEEDED: Measurements are needed to learn how the MEASUREMENTS NEEDED: Measurements are needed to learn how the
deployed base of network elements and servers react to FIN packets deployed base of network elements and servers react to FIN packets
marked with the ECT(0)/ECT(1)/CE codepoints, i.e. whether they marked with the ECT(0)/ECT(1)/CE codepoints, i.e. whether they
are dropped, codepoint cleared or processed. are dropped, codepoint cleared or processed.
3.2.6. RST 3.2.6. RST (Send)
A TCP implementation can set ECT on a RST. A TCP implementation can set ECT on a RST.
The "challenge ACK" approach to checking the validity of RSTs See Section 3.3.5 for the implications if a host receives a CE-marked
(section 3.2 of [RFC5961] is RECOMMENDED at the data receiver. RST.
A congestion response to a CE-marking on a RST is not required (and A congestion response to a CE-marking on a RST is not required (and
actually not possible). actually not possible).
MEASUREMENTS NEEDED: Measurements are needed to learn how the MEASUREMENTS NEEDED: Measurements are needed to learn how the
deployed base of network elements and servers react to RST packets deployed base of network elements and servers react to RST packets
marked with the ECT(0)/ECT(1)/CE codepoints, i.e. whether they marked with the ECT(0)/ECT(1)/CE codepoints, i.e. whether they
are dropped, codepoint cleared or processed. are dropped, codepoint cleared or processed.
3.2.7. Retransmissions 3.2.7. Retransmissions (Send)
For the experiments proposed here, the TCP sender will set ECT on For the ECN++ experiment, the TCP sender will set ECT on
retransmitted segments. It can ignore the prohibition in section retransmitted segments. It can ignore the prohibition in section
6.1.5 of RFC 3168 against setting ECT on retransmissions. 6.1.5 of RFC 3168 against setting ECT on retransmissions, as per
Section 4.3 of [RFC8311].
Nonetheless, the TCP data receiver MUST ignore the CE codepoint on See Section 3.3.6 for the implications if a host receives a CE-marked
incoming segments that fail any validity check. The validity check retransmission.
in section 5.2 of [RFC5961] is RECOMMENDED. This will effectively
mitigate an attack that uses spoofed data packets to fool the
receiver into feeding back spoofed congestion indications to the
sender, which in turn would be fooled into continually halving its
congestion window.
If the TCP sender receives feedback that a retransmitted packet was If the TCP sender receives feedback that a retransmitted packet was
CE-marked, it will react as it would to any feedback of CE-marking on CE-marked, it will react as it would to any feedback of CE-marking on
a data packet. a data packet.
MEASUREMENTS NEEDED: Measurements are needed to learn how the MEASUREMENTS NEEDED: Measurements are needed to learn how the
deployed base of network elements and servers react to deployed base of network elements and servers react to
retransmissions marked with the ECT(0)/ECT(1)/CE codepoints, i.e. retransmissions marked with the ECT(0)/ECT(1)/CE codepoints, i.e.
whether they are dropped, codepoint cleared or processed. whether they are dropped, codepoint cleared or processed.
skipping to change at page 16, line 21 skipping to change at page 16, line 45
In case traversal problems arise in future, fall-back measures have In case traversal problems arise in future, fall-back measures have
been specified above, but only for the cases where ECT on the initial been specified above, but only for the cases where ECT on the initial
packet of a half-connection (SYN or SYN-ACK) is persistently failing packet of a half-connection (SYN or SYN-ACK) is persistently failing
to get through. to get through.
Fall-back measures for blockage of ECT on other TCP control packets Fall-back measures for blockage of ECT on other TCP control packets
MAY be implemented. However they are not specified here given the MAY be implemented. However they are not specified here given the
lack of any evidence they will be needed. Section 4.9 justifies this lack of any evidence they will be needed. Section 4.9 justifies this
advice in more detail. advice in more detail.
3.3. Receiver Behaviour
The present ECN++ specification primarily concerns the behaviour for
sending TCP control packets or retransmissions. Below are a few
changes to the receive side of an implementation that are recommended
while updating its send side. Nonetheless, where deployment is
concerned, ECN++ is still a sender-only deployment, because it does
not depend on receivers complying with any of these recommendations.
3.3.1. Receiver Behaviour for Any TCP Control Packet or Retransmission
RFC8311 is a standards track update to RFC 3168 in order to (amongst
other things) "...allow the use of ECT codepoints on SYN packets,
pure acknowledgement packets, window probe packets, and
retransmissions of packets..., provided that the changes from RFC
3168 are documented in an Experimental RFC in the IETF document
stream."
Section 4.3 of RFC 8311 amends every statement in RFC 3168 that
precludes the use of ECT on control packets and retransmissions to
add "unless otherwise specified by an Experimental RFC in the IETF
document stream". The present specification is such an Experimental
RFC. Therefore, In order for this experiment to be useful, the
following requirements follow from RFC8311:
o Any TCP implementation SHOULD accept receipt of any valid TCP
control packet or retransmission irrespective of its IP/ECN field.
If any existing implementation does not, it SHOULD be updated to
do so.
o A TCP implementation taking part in the experiments proposed here
MUST accept receipt of any valid TCP control packet or
retransmission irrespective of its IP/ECN field.
These measures are derived from the robustness principle of "... be
liberal in what you accept from others", in order to ensure
compatibility with any future protocol changes that allow ECT on any
TCP packet.
3.3.2. SYN (Receive)
RFC 3168 negotiates the use of ECN for the connection end-to-end
using the ECN flags in the TCP header. When RFC3168 says that "A
host MUST NOT set ECT on SYN ... packets." it is silent as to what a
TCP server ought to do if it receives a SYN packet with a non-zero
IP/ECN field.
Some implementations of TCP servers (e.g. current Linux) assume that,
if a host receives a SYN with a non-zero IP/ECN field, it must be due
to network mangling, and they disable ECN for the rest of the
connection. Section 4.2.2.2 finds that this type of network mangling
seems to be virtually non-existent so it would be preferable to
report any such mangling so it can be fixed.
For the avoidance of doubt, the normative statements for all TCP
control packets in Section 3.3.1 are interpreted for the case when a
SYN is received as follows:
o Any TCP server implementation SHOULD accept receipt of a valid SYN
that requests ECN support for the connection, irrespective of the
IP/ECN field of the SYN. If any existing implementation does not,
it SHOULD be updated to do so.
o A TCP implementation taking part in the ECN++ experiment MUST
accept receipt of a valid SYN, irrespective of its IP/ECN field.
o If the SYN is CE-marked and the server has no logic to feed back a
CE mark on a SYN-ACK (e.g. it does not support AccECN), it has to
ignore the CE-mark (the client detects this case and behaves
conservatively in mitigation - see Section 3.2.1.3).
3.3.3. Pure ACK (Receive)
For the avoidance of doubt, the normative statements for all TCP
control packets in Section 3.3.1 are interpreted for the case when a
Pure ACK is received as follows:
o Any TCP implementation SHOULD accept receipt of a pure ACK with a
non-zero ECN field, despite current RFCs precluding the sending of
such packets.
o A TCP implementation taking part in the ECN++ experiment MUST
accept receipt of a pure ACK with a non-zero ECN field.
The question of whether and how the receiver of pure ACKs is required
to feed back any CE marks on them is outside the scope of the present
specification because it is a matter for the relevant feedback
specification ([RFC3168] or [I-D.ietf-tcpm-accurate-ecn]). Currently
AccECN feedback is required to count CE marking of any control packet
including pure ACKs. Whereas RFC 3168 is silent on this point, so
feedback of CE-markings might be implementation specific (see
Section 4.4.1.1).
3.3.4. FIN (Receive)
The TCP data receiver MUST ignore the CE codepoint on incoming FINs
that fail any validity check. The validity check in section 5.2 of
[RFC5961] is RECOMMENDED.
3.3.5. RST (Receive)
The "challenge ACK" approach to checking the validity of RSTs
(section 3.2 of [RFC5961] is RECOMMENDED at the data receiver.
3.3.6. Retransmissions (Receive)
The TCP data receiver MUST ignore the CE codepoint on incoming
segments that fail any validity check. The validity check in section
5.2 of [RFC5961] is RECOMMENDED. This will effectively mitigate an
attack that uses spoofed data packets to fool the receiver into
feeding back spoofed congestion indications to the sender, which in
turn would be fooled into continually reducing its congestion window.
4. Rationale 4. Rationale
This section is informative, not normative. It presents counter- This section is informative, not normative. It presents counter-
arguments against the justifications in the RFC series for disabling arguments against the justifications in the RFC series for disabling
ECN on TCP control segments and retransmissions. It also gives ECN on TCP control segments and retransmissions. It also gives
rationale for why ECT is safe on control segments that have not, so rationale for why ECT is safe on control segments that have not, so
far, been mentioned in the RFC series. First it addresses over- far, been mentioned in the RFC series. First it addresses over-
arching arguments used for most packet types, then it addresses the arching arguments used for most packet types, then it addresses the
specific arguments for each packet type in turn. specific arguments for each packet type in turn.
skipping to change at page 17, line 45 skipping to change at page 20, line 36
malicious host might be able to inject a large number of TCP SYN malicious host might be able to inject a large number of TCP SYN
packets through a potentially congested ECN-enabled router, packets through a potentially congested ECN-enabled router,
congesting it even further." congesting it even further."
The first point actually describes two subtly different issues. So The first point actually describes two subtly different issues. So
below three arguments are countered in turn. below three arguments are countered in turn.
4.2.1. Argument 1a: Unrecognized CE on the SYN 4.2.1. Argument 1a: Unrecognized CE on the SYN
This argument certainly applied at the time RFC 5562 was written, This argument certainly applied at the time RFC 5562 was written,
when no ECN responder mechanism had any logic to recognize or feed when no ECN responder mechanism had any logic to recognize a CE
back a CE marking on a SYN. The problem was that, during the 3WHS, marking on a SYN and, even if logic were added, there was no field in
the SYN-ACK to feed it back. The problem was that, during the 3WHS,
the flag in the TCP header for ECN feedback (called Echo Congestion the flag in the TCP header for ECN feedback (called Echo Congestion
Experienced) had been overloaded to negotiate the use of ECN itself. Experienced) had been overloaded to negotiate the use of ECN itself.
So there was no space for feedback in a SYN-ACK.
The accurate ECN (AccECN) protocol [I-D.ietf-tcpm-accurate-ecn] has The accurate ECN (AccECN) protocol [I-D.ietf-tcpm-accurate-ecn] has
since been designed to solve this problem, using a two-pronged since been designed to solve this problem. Two features are
approach. First AccECN uses the 3 ECN bits in the TCP header as 8 important here:
codepoints, so there is space for the responder to feed back whether
there was CE on the SYN. Second a TCP initiator can always request
AccECN support on every SYN, and any responder reveals its level of
ECN support: AccECN, classic ECN, or no ECN. Therefore, if a
responder does indicate that it supports AccECN, the initiator can be
sure that, if there is no CE feedback on the SYN-ACK, then there
really was no CE on the SYN.
An initiator can combine AccECN with three possible strategies for 1. An AccECN server uses the 3 'ECN' bits in the TCP header of the
setting ECT on a SYN: SYN-ACK to respond to the client. 4 of the possible 8 codepoints
provide enough space for the server to feed back which of the 4
IP/ECN codepoints was on the incoming SYN (including CE of
course).
2. If any of these 4 codepoints are in the SYN-ACK, it confirms that
the server supports AccECN and, if another codepoint is returned,
it confirms that the server doesn't support AccECN.
This still does not seem to allow a client set ECT on a SYN, it only
finds out whether the server would have supported it afterwards. The
trick the client uses for ECN++ is to set ECT on the SYN
optimistically then, if the SYN-ACK reveals that the server wouldn't
have understood CE on the SYN, the client responds conservatively as
if the SYN was marked with CE.
Happily, the appropriate conservative congestion response is to
reduce the initial window, and it is extremely rare for a TCP client
to send more than one packet as its initial request anyway. Any
clients that do frequently use a larger initial window for their
first message to the server can cache which servers will not
understand ECT on a SYN (see Section 4.2.3 below).
4.2.2. Argument 1b: ECT Considered Invalid on the SYN
Given, until now, ECT-marked SYN packets have been prohibited, it
cannot be assumed they will be accepted, by TCP middleboxes or
servers.
4.2.2.1. ECT on SYN Considered Invalid by Middleboxes
According to a study using 2014 data [ecn-pam] from a limited range
of fixed vantage points, for the top 1M Alexa web sites, adding the
ECN capability to SYNs was increasing connection establishment
failures by about 0.4%.
From a wider range of fixed and mobile vantage points, a more recent
study in Jan-May 2017 [Mandalari18] found no occurrences of blocking
of ECT on SYNs. However, in more than half the mobile networks
tested it found wiping of the ECN codepoint at the first hop.
MEASUREMENTS NEEDED: As wiping at the first hop is remedied,
measurements will be needed to check whether SYNs with ECT are
sometimes blocked deeper into the path.
Silent failures introduce a retransmission timeout delay (default 1
second) at the initiator before it attempts any fall back strategy
(whereas explicit RSTs can be dealt with immediately). Ironically,
making SYNs ECN-capable is intended to avoid the timeout when a SYN
is lost due to congestion. Fortunately, if there is any discard of
ECN-capable SYNs due to policy, it will occur predictably, not
randomly like congestion. So the initiator should be able to avoid
it by caching those sites that do not support ECN-capable SYNs (see
the last paragraph of Section 3.2.1.2).
4.2.2.2. ECT on SYN Considered Invalid by Servers
A study conducted in Nov 2017 [Kuehlewind18] found that, of the 82%
of the Alexa top 50k web servers that supported ECN, 84% disabled ECN
if the IP/ECN field on the SYN was ECT0, CE or either. Given most
web servers use Linux, this behaviour can most likely be traced to a
patch contributed in May 2012 that was first distributed in v3.5 of
the Linux kernel [strict-ecn]. The comment says "RFC3168 : 6.1.1 SYN
packets must not have ECT/ECN bits set. If we receive a SYN packet
with these bits set, it means a network is playing bad games with TOS
bits. In order to avoid possible false congestion notifications, we
disable TCP ECN negociation." Of course, some of the 84% might be
due to similar code in other OSs.
For brevity we shall call this the "contra-Postel" ECN test, because
it is conservative with what it accepts, contrary to Postel's
robustness principle. A robust protocol will not usually assume
network mangling without comparing with the value originally sent,
and one packet is not sufficient to make an assumption with such
irreversible consequences anyway.
Ironically, networks rarely seem to alter the IP/ECN field on a SYN
from zero to non-zero anyway. In a study conducted in Jan-May 2017
over millions of paths from vantage points in a few dozen mobile and
fixed networks [Mandalari18], no such transition was observed. With
such a small or non-existent incidence of this sort of network
mangling, it would be preferable to report any residual problem paths
so that they can be fixed.
Whatever, the widespread presence of this 'contra-Postel' test proves
that RFC 5562 was correct to expect that ECT would be considered
invalid on SYNs. Nonetheless, it is not an insurmountable problem -
caching can work round it. The prevalence of these "contra-Postel"
ECN servers makes it challenging to cache them all. However,
Section 4.2.3 below explains how a cache of limited size can
alleviate this problem for a client's most popular sites.
For the future, [RFC8311] updates RFC 3168 to clarify that the IP/ECN
field does not have to be zero on a SYN if documented in an
experimental RFC such as the present ECN++ specification.
4.2.3. Caching Strategies for ECT on SYNs
Given the server handling of ECN on SYNs outlined in Section 4.2.2.2
above, an initiator might combine AccECN with three candidate caching
strategies for setting ECT on a SYN:
(S1): Pessimistic ECT and cache successes: The initiator always (S1): Pessimistic ECT and cache successes: The initiator always
requests AccECN in the SYN, but without setting ECT. Then it requests AccECN, but by default without ECT on the SYN. Then
records those servers that confirm that they support AccECN in it caches those servers that confirm that they support AccECN
a cache. On a subsequent connection to any server that as 'ECT SYN OK'. On a subsequent connection to any server
supports AccECN, the initiator can then set ECT on the SYN. that supports AccECN, the initiator can then set ECT on the
SYN. When connecting to other servers (non-ECN or classic
ECN) it will not set ECT on the SYN, so it will not fail the
'contra-Postel' test.
(S2): Optimistic ECT: The initiator always sets ECT optimistically Longer term, as servers upgrade to AccECN, the initiator is
on the initial SYN and it always requests AccECN support. still requesting AccECN, so it will add them to the cache and
Then, if the server response shows it has no AccECN logic (so use ECT on subsequent SYNs to those servers. However,
it cannot feed back a CE mark), the initiator conservatively assuming it has to cap the size of the cache, the client will
behaves as if the SYN was CE-marked, by reducing its initial not have the benefit of ECT SYNs to those less frequently used
window. AccECN servers expelled from its cache.
A. No cache: The optimistic ECT strategy ought to work fairly (S2): Optimistic ECT: The initiator always requests AccECN and by
well without caching any responses. default sets ECT on the SYN. Then, if the server response
shows it has no AccECN logic (so it cannot feed back a CE
mark), the initiator conservatively behaves as if the SYN was
CE-marked, by reducing its initial window.
A. No cache.
B. Cache failures: The optimistic ECT strategy can be B. Cache failures: The optimistic ECT strategy can be
improved by recording solely those servers that do not improved by caching solely those servers that do not
support AccECN. On subsequent connections to these non- support AccECN as 'ECT SYN NOK'. This would include non-
ECN servers and all Classic ECN servers whether 'contra-
Postel' or not. On subsequent connections to these non-
AccECN servers, the initiator will still request AccECN AccECN servers, the initiator will still request AccECN
but not set ECT on the SYN. Then, the initiator can use but not set ECT on the SYN. Then, the connection can
its full initial window (if it has enough request data to still fall back to Classic ECN, if the server supports it,
need it). Longer term, as servers upgrade to AccECN, the and the initiator can use its full initial window (if it
initiator will remove them from the cache and use ECT on has enough request data to need it).
subsequent SYNs to that server.
Longer term, as servers upgrade to AccECN, the initiator
will remove them from the cache and use ECT on subsequent
SYNs to that server.
Where an access network operator mediates Internet access Where an access network operator mediates Internet access
via a proxy that does not support AccECN, the optimistic via a proxy that does not support AccECN, the optimistic
ECT strategy will always fail. This scenario is more ECT strategy will always fail. This scenario is more
likely in mobile networks. Therefore, a mobile host could likely in mobile networks. Therefore, a mobile host could
cache lack of AccECN support per attached access network cache lack of AccECN support per attached access network
operator. Whenever it attached to a new operator, it operator. Whenever it attached to a new operator, it
could check a well-known AccECN test server and, if it could check a well-known AccECN test server and, if it
found no AccECN support, it would add a cache entry for found no AccECN support, it would add a cache entry for
the attached operator. It would only use ECT when neither the attached operator. It would only use ECT when neither
skipping to change at page 19, line 14 skipping to change at page 24, line 21
its per server cache when not attached to a non-AccECN its per server cache when not attached to a non-AccECN
proxy. proxy.
(S3): ECT by configuration: In a controlled environment, the (S3): ECT by configuration: In a controlled environment, the
administrator can make sure that servers support ECN-capable administrator can make sure that servers support ECN-capable
SYN packets. Examples of controlled environments are single- SYN packets. Examples of controlled environments are single-
tenant DCs, and possibly multi-tenant DCs if it is assumed tenant DCs, and possibly multi-tenant DCs if it is assumed
that each tenant mostly communicates with its own VMs. that each tenant mostly communicates with its own VMs.
For unmanaged environments like the public Internet, pragmatically For unmanaged environments like the public Internet, pragmatically
the choice is between strategies (S1) and (S2B): the choice is between strategies (S1), (S2A) and (S2B). The
normative specification for ECT on a SYN in Section 3.2.1 recommends
the "optimistic ECT and cache failures" strategy (S2B) but the choice
depends on the implementer's motivation for using ECN++, and the
deployment prevalence of different technologies and bug-fixes. For
instance, if a user's Internet access bottleneck supported L4S ECN
but not Classic ECN, strategy (S2A) would make most sense and there
would be no point trying to avoid the 'contra-Postel' test and
negotiate Classic ECN.
o The "pessimistic ECT and cache successes" strategy (S1) suffers o The "pessimistic ECT and cache successes" strategy (S1) suffers
from exposing the initial SYN to the prevailing loss level, even from exposing the initial SYN to the prevailing loss level, even
if the server supports ECT on SYNs, but only on the first if the server supports ECT on SYNs, but only on the first
connection to each AccECN server. connection to each AccECN server. If AccECN becomes widely
deployed on servers, SYNs to those AccECN servers that are less
o The "optimistic ECT and cache failures" strategy (S2B) exploits a frequently used by the client and therefore don't fit in the cache
server's support for ECT on SYNs from the very first attempt. But will not benefit from ECN protection at all.
if the server turns out not to support AccECN, the initiator has
to conservatively limit its initial window - usually
unnecessarily. Nonetheless, initiator request data (as opposed to
server response data) is rarely larger than 1 SMSS anyway {ToDo:
reference? (this information was given informally by Yuchung
Cheng)}.
The normative specification for ECT on a SYN in Section 3.2.1 uses
the "optimistic ECT and cache failures" strategy (S2B) on the
assumption that an initial window of 1 SMSS is usually sufficient for
client requests anyway. Clients that often initially send more than
1 SMSS of data could use strategy (S1) during initial deployment, and
strategy (S2B) later (when the probability of servers supporting
AccECN and the likelihood of seeing some CE marking is higher).
Also, as deployment proceeds, caching successes (S1) starts off small
then grows, while caching failures (S2B) becomes large at first, then
shrinks.
MEASUREMENTS NEEDED: Measurements are needed to determine whether
one or the other strategy would be sufficient for any particular
client, or whether a particular client would need both strategies
in different circumstances.
4.2.2. Argument 1b: Unrecognized ECT on the SYN
Given, until now, ECT-marked SYN packets have been prohibited, it
cannot be assumed they will be accepted.
According to a study using 2014 data [ecn-pam] from a limited range
of vantage points, out of the top 1M Alexa web sites, 4791 (0.82%)
IPv4 sites and 104 (0.61%) IPv6 sites failed to establish a
connection when they received a TCP SYN with any ECN codepoint set in
the IP header and the appropriate ECN flags in the TCP header. Of
these, about 41% failed to establish a connection due to the ECN
flags in the TCP header even with a Not-ECT ECN field in the IP
header (i.e. despite full compliance with RFC 3168). Therefore
adding the ECN capability to SYNs was increasing connection
establishment failures by about 0.4%.
In a study using 2017 data from a wider range of fixed and mobile
vantage points to the top 500k Alexa servers, no case was found where
adding the ECN capability to a SYN increased the likelihood of
connection establishment failure [Mandalari18].
MEASUREMENTS NEEDED: More investigation is needed to understand
the different outcomes of the 2014 and 2017 studies.
RFC 3168 says "a host MUST NOT set ECT on SYN [...] packets", but it
does not say what the responder should do if an ECN-capable SYN
arrives. So, in the 2014 study, perhaps some responder
implementations were checking that the SYN complied with RFC 3168,
then silently ignoring non-compliant SYNs (or perhaps returning a
RST). Also some middleboxes (e.g. firewalls) might have been
discarding non-compliant SYNs. For the future,
[I-D.ietf-tsvwg-ecn-experimentation] updates RFC 3168 to clarify that
middleboxes "SHOULD NOT" do this, but that does not alter the past.
Whereas RSTs can be dealt with immediately, silent failures introduce
a retransmission timeout delay (default 1 second) at the initiator
before it attempts any fall back strategy. Ironically, making SYNs
ECN-capable is intended to avoid the timeout when a SYN is lost due
to congestion. Fortunately, if there is any discard of ECN-capable
SYNs due to policy, it will occur predictably, not randomly like
congestion. So the initiator can avoid it by caching those sites
that do not support ECN-capable SYNs. This further justifies the use
of the "optimistic ECT and cache failures" strategy in Section 3.2.1.
MEASUREMENTS NEEDED: Experiments are needed to determine whether
blocking of ECT on SYNs is widespread, and how many occurrences of
problems would be masked by how few cache entries.
If blocking is too widespread for the "optimistic ECT and cache o The "optimistic ECT without a cache" strategy (S2A) is the
failures" strategy (S2B), the "pessimistic ECT and cache successes" simplest. It would satisfy the goal of an implementer who is
strategy (Section 4.2.1) would be better. solely interested in ultra-low latency using AccECN and ECN++
(e.g. accessing L4S servers) and is not concerned about fall-back
to Classic ECN (e.g. when accessing other servers).
MEASUREMENTS NEEDED: Then measurements would be needed on whether o The "optimistic ECT and cache failures" strategy (S2B) exploits
failures were still widespread on the third connection attempt ECT on SYNs from the very first attempt. But if the server turns
after the more careful ("pessimistic") first and second attempts. out to be 'contra-Postel' it will disable ECN for the connection,
but only for the first connection if it's one of the client's more
popular servers that fits in the cache. If the server turns out
not to support AccECN, the initiator has to conservatively limit
its initial window, but again only for the first connection if
it's one of the client's more popular servers (and anyway this
rarely makes any difference when most client requests fit in a
single packet).
If so, it might be necessary to send a not-ECT SYN a short delay Note that, if AccECN deployment grows, caching successes (S1) starts
after an ECT SYN and only accept the non-ECT connection if it off small then grows, while caching failures (S2B) becomes large at
returned first. This would reduce the performance penalty for those first, then shrinks. At half-way, the size of the cache has to be
deploying ECT SYN support. capped with either approach, so the default behaviour for all the
servers that do not fit in the cache is as important as the behaviour
for the popular servers that do fit.
FOR DISCUSSION: If this becomes necessary, how much delay ought to MEASUREMENTS NEEDED: Measurements are needed to determine which
be required before the second SYN? Certainly less than the strategy would be sufficient for any particular client, whether a
standard RTO (1 second). But more or less than the maximum RTT particular client would need different strategies in different
expected over the surface of the earth (roughly 250ms)? Or even circumstances and how many occurrences of problems would be masked
back-to-back? by how few cache entries.
However, based on the data above from [ecn-pam], even a cache of a Another strategy would be to send a not-ECT SYN a short delay (below
dozen or so sites ought to avoid all ECN-related performance problems the typical lowest RTT) after an ECT SYN and only accept the non-ECT
with roughly the Alexa top thousand. So it is questionable whether connection if it returned first. This would reduce the performance
sending two SYNs will be necessary, particularly given failures at penalty for those deploying ECT SYN support. However, this 'happy
well-maintained sites could reduce further once ECT SYNs are eyeballs' approach becomes complex when multiple optional features
standardized. are all tried on the first SYN (or on multiple SYNs), so it is not
recommended.
4.2.3. Argument 2: DoS Attacks 4.2.4. Argument 2: DoS Attacks
[RFC5562] says that ECT SYN packets could be misused by malicious [RFC5562] says that ECT SYN packets could be misused by malicious
clients to augment "the well-known TCP SYN attack". It goes on to clients to augment "the well-known TCP SYN attack". It goes on to
say "a malicious host might be able to inject a large number of TCP say "a malicious host might be able to inject a large number of TCP
SYN packets through a potentially congested ECN-enabled router, SYN packets through a potentially congested ECN-enabled router,
congesting it even further." congesting it even further."
We assume this is a reference to the TCP SYN flood attack (see We assume this is a reference to the TCP SYN flood attack (see
https://en.wikipedia.org/wiki/SYN_flood), which is an attack against https://en.wikipedia.org/wiki/SYN_flood), which is an attack against
a responder end point. We assume the idea of this attack is to use a responder end point. We assume the idea of this attack is to use
skipping to change at page 22, line 8 skipping to change at page 26, line 11
ECT it would make it legitimate for firewalls to discard them. ECT it would make it legitimate for firewalls to discard them.
However this would negate the considerable benefit of ECT SYNs for However this would negate the considerable benefit of ECT SYNs for
compliant transports and seems unnecessary because RFC 3168 already compliant transports and seems unnecessary because RFC 3168 already
provides the means to address this concern. In section 7, RFC 3168 provides the means to address this concern. In section 7, RFC 3168
says "During periods where ... the potential packet marking rate says "During periods where ... the potential packet marking rate
would be high, our recommendation is that routers drop packets rather would be high, our recommendation is that routers drop packets rather
then set the CE codepoint..." and this advice is repeated in then set the CE codepoint..." and this advice is repeated in
[RFC7567] (section 4.2.1). This makes it harder for flooding packets [RFC7567] (section 4.2.1). This makes it harder for flooding packets
to gain from ECT. to gain from ECT.
Further experiments are needed to test how much malicious hosts can [ecn-overload] showed that ECT can only slightly augment flooding
use ECT to augment flooding attacks without triggering AQMs to turn attacks relative to a non-ECT attack. It was hard to overload the
off ECN support (flying "just under the radar"). If it is found that link without causing the queue to grow, which in turn caused the AQM
ECT can only slightly augment flooding attacks, the risk of such to disable ECN and switch to drop, thus negating any advantage of
attacks will need to be weighed against the performance benefits of using ECT. This was true even with the switch-over point set to 25%
ECT SYNs. drop probability (i.e. the arrival rate was 133% of the link rate).
4.3. SYN-ACKs 4.3. SYN-ACKs
The proposed approach in Section 3.2.2 for experimenting with ECN- The proposed approach in Section 3.2.2 for experimenting with ECN-
capable SYN-ACKs is effectively identical to the scheme called ECN+ capable SYN-ACKs is effectively identical to the scheme called ECN+
[ECN-PLUS]. In 2005, the ECN+ paper demonstrated that it could [ECN-PLUS]. In 2005, the ECN+ paper demonstrated that it could
reduce the average Web response time by an order of magnitude. It reduce the average Web response time by an order of magnitude. It
also argued that adding ECT to SYN-ACKs did not raise any new also argued that adding ECT to SYN-ACKs did not raise any new
security vulnerabilities. security vulnerabilities.
4.3.1. Response to Congestion on a SYN-ACK 4.3.1. Possibility of Unrecognized CE on the SYN-ACK
The feedback behaviour by the initiator in response to a CE-marked
SYN-ACK from the responder depends on whether classic ECN feedback
[RFC3168] or AccECN feedback [I-D.ietf-tcpm-accurate-ecn] has been
negotiated. In either case no change is required to RFC 3168 or the
AccECN specification.
Some classic ECN client implementations might ignore a CE-mark on a
SYN-ACK, or even ignore a SYN-ACK packet entirely if it is set to ECT
or CE. This is a possibility because an RFC 3168 implementation
would not necessarily expect a SYN-ACK to be ECN-capable. This issue
already came up when the IETF first decided to experiment with ECN on
SYN-ACKs [RFC5562] and it was decided to go ahead without any extra
precautionary measures. This was because the probability of
encountering the problem was believed to be low and the harm if the
problem arose was also low (see Appendix B of RFC 5562).
4.3.2. Response to Congestion on a SYN-ACK
The IETF has already specified an experiment with ECN-capable SYN-ACK The IETF has already specified an experiment with ECN-capable SYN-ACK
packets [RFC5562]. It was inspired by the ECN+ paper, but it packets [RFC5562]. It was inspired by the ECN+ paper, but it
specified a much more conservative congestion response to a CE-marked specified a much more conservative congestion response to a CE-marked
SYN-ACK, called ECN+/TryOnce. This required the server to reduce its SYN-ACK, called ECN+/TryOnce. This required the server to reduce its
initial window to 1 segment (like ECN+), but then the server had to initial window to 1 segment (like ECN+), but then the server had to
send a second SYN-ACK and wait for its ACK before it could continue send a second SYN-ACK and wait for its ACK before it could continue
with its initial window of 1 SMSS. The second SYN-ACK of this 5-way with its initial window of 1 SMSS. The second SYN-ACK of this 5-way
handshake had to carry no data, and had to disable ECN, but no handshake had to carry no data, and had to disable ECN, but no
justification was given for these last two aspects. justification was given for these last two aspects.
The present ECN experiment obsoletes RFC 5562 because it uses the The present ECN++ experimental specification obsoletes RFC 5562
ECN+ congestion response, not ECN+/TryOnce. First we argue against because it uses the ECN+ congestion response, not ECN+/TryOnce.
the rationale for ECN+/TryOnce given in sections 4.4 and 6.2 of First we argue against the rationale for ECN+/TryOnce given in
[RFC5562]. It starts with a rather too literal interpretation of the sections 4.4 and 6.2 of [RFC5562]. It starts with a rather too
requirement in RFC 3168 that says TCP's response to a single CE mark literal interpretation of the requirement in RFC 3168 that says TCP's
has to be "essentially the same as the congestion control response to response to a single CE mark has to be "essentially the same as the
a *single* dropped packet." TCP's response to a dropped initial (SYN congestion control response to a *single* dropped packet." TCP's
or SYN-ACK) packet is to wait for the retransmission timer to expire response to a dropped initial (SYN or SYN-ACK) packet is to wait for
(currently 1s). However, this long delay assumes the worst case the retransmission timer to expire (currently 1s). However, this
between two possible causes of the loss: a) heavy overload; or b) the long delay assumes the worst case between two possible causes of the
normal capacity-seeking behaviour of other TCP flows. When the loss: a) heavy overload; or b) the normal capacity-seeking behaviour
network is still delivering CE-marked packets, it implies that there of other TCP flows. When the network is still delivering CE-marked
is an AQM at the bottleneck and that it is not overloaded. This is packets, it implies that there is an AQM at the bottleneck and that
because an AQM under overload will disable ECN (as recommended in it is not overloaded. This is because an AQM under overload will
section 7 of RFC 3168 and repeated in section 4.2.1 of RFC 7567). So disable ECN (as recommended in section 7 of RFC 3168 and repeated in
scenario (a) can be ruled out. Therefore, TCP's response to a CE- section 4.2.1 of RFC 7567). So scenario (a) can be ruled out.
marked SYN-ACK can be similar to its response to the loss of _any_ Therefore, TCP's response to a CE-marked SYN-ACK can be similar to
packet, rather than backing off as if the special _initial_ packet of its response to the loss of _any_ packet, rather than backing off as
a flow has been lost. if the special _initial_ packet of a flow has been lost.
How TCP responds to the loss of any single packet depends what it has How TCP responds to the loss of any single packet depends what it has
just been doing. But there is not really a precedent for TCP's just been doing. But there is not really a precedent for TCP's
response when it experiences a CE mark having sent only one (small) response when it experiences a CE mark having sent only one (small)
packet. If TCP had been adding one segment per RTT, it would have packet. If TCP had been adding one segment per RTT, it would have
halved its congestion window, but it hasn't established a congestion halved its congestion window, but it hasn't established a congestion
window yet. If it had been exponentially increasing it would have window yet. If it had been exponentially increasing it would have
exited slow start, but it hasn't started exponentially increasing yet exited slow start, but it hasn't started exponentially increasing yet
so it hasn't established a slow-start threshold. so it hasn't established a slow-start threshold.
skipping to change at page 23, line 29 skipping to change at page 28, line 5
and it is probably already somewhere around the AQM's operating point and it is probably already somewhere around the AQM's operating point
- it is unlikely to be well below and it might be well above. So, it - it is unlikely to be well below and it might be well above. So, it
does not seem sensible to add a number of packets at once. On the does not seem sensible to add a number of packets at once. On the
other hand, it is highly unlikely that the SYN-ACK itself pushed the other hand, it is highly unlikely that the SYN-ACK itself pushed the
AQM into congestion, so it will be safe to introduce another single AQM into congestion, so it will be safe to introduce another single
segment immediately (1 RTT after the SYN-ACK). Therefore, starting segment immediately (1 RTT after the SYN-ACK). Therefore, starting
to probe for capacity with a slow start from an initial window of 1 to probe for capacity with a slow start from an initial window of 1
segment seems appropriate to the circumstances. This is the approach segment seems appropriate to the circumstances. This is the approach
adopted in Section 3.2.2. adopted in Section 3.2.2.
4.3.2. Fall-Back if ECT SYN-ACK Fails 4.3.3. Fall-Back if ECT SYN-ACK Fails
An alternative to the server caching failed connection attempts would An alternative to the server caching failed connection attempts would
be for the server to rely on the client caching failed attempts (on be for the server to rely on the client caching failed attempts (on
the basis that the client would cache a failure whether ECT was the basis that the client would cache a failure whether ECT was
blocked on the SYN or the SYN-ACK). This strategy cannot be used if blocked on the SYN or the SYN-ACK). This strategy cannot be used if
the SYN does not request AccECN support. It works as follows: if the the SYN does not request AccECN support. It works as follows: if the
server receives a SYN that requests AccECN support but is set to not- server receives a SYN that requests AccECN support but is set to not-
ECT, it replies with a SYN-ACK also set to not-ECT. If a middlebox ECT, it replies with a SYN-ACK also set to not-ECT. If a middlebox
only blocks ECT on SYNs, not SYN-ACKs, this strategy might disable only blocks ECT on SYNs, not SYN-ACKs, this strategy might disable
ECN on a SYN-ACK when it did not need to, but at least it saves the ECN on a SYN-ACK when it did not need to, but at least it saves the
skipping to change at page 24, line 37 skipping to change at page 29, line 12
implementations, a single dropped ACK generally has only a very implementations, a single dropped ACK generally has only a very
small effect on the TCP's sending rate." small effect on the TCP's sending rate."
We next address each of the arguments presented above. We next address each of the arguments presented above.
The first argument is a specific instance of the reliability argument The first argument is a specific instance of the reliability argument
for the case of pure ACKs. This has already been addressed by for the case of pure ACKs. This has already been addressed by
countering the general reliability argument in Section 4.1. countering the general reliability argument in Section 4.1.
The second argument says that ECN ought not to be enabled unless The second argument says that ECN ought not to be enabled unless
there is a mechanism to respond to it. However, actually there _is_ there is a mechanism to respond to it. This argument actually
a mechanism to respond to congestion on a pure ACK that RFC 3168 has comprises three sub-arguments:
overlooked - the congestion window mechanism. When data segments and
pure ACKs are interspersed, congestion notifications ought to
regulate the congestion window, whether they are on data segments or
on pure ACKs. Otherwise, if ECN is disabled on Pure ACKs, and if
(say) 70% of the segments in one direction are Pure ACKs, about 70%
of the congestion notifications will be missed and the data segments
will not be correctly regulated.
So RFC 3168 ought to have considered two congestion response Mechanism feasibility: If ECN is enabled on Pure ACKs, are there, or
mechanisms - reducing the congestion window (cwnd) and reducing the could there be, suitable mechanisms to detect, feed back and
ACK rate - and only the latter was missing. Further, RFC 3168 was respond to ECN-marked Pure ACKs?
incorrect to assume that, if one ACK was a pure ACK, all segments in
the same direction would be pure ACKs. Admittedly a continual stream
of pure ACKs in one direction is quite a common case (e.g. a file
download). However, it is also common for the pure ACKs to be
interspersed with data segments (e.g. HTTP/2 browser requests
controlling a web application). Indeed, it is more likely that any
congestion experienced by pure ACKs will be due to mixing with data
segments, either within the same flow, or within competing flows.
This insight swings the argument towards enabling ECN on pure ACKs so Do no extra harm: There has never been a mechanism to respond to
that CE marks can drive the cwnd response to congestion (whenever loss of non-ECN Pure ACKs. So it seems that adding ECN without a
data segments are interspersed with the pure ACKs). Then to response mechanism will do no extra harm to others, while
separately decide whether an ACK rate response is also required (when improving a connection's own performance (because loss of an ACK
they are ECN-enabled). The two types of response are addressed holds back new data). However, if the end systems have no
separately in the following two subsections, then a final subsection response mechanism, ECN Pure ACKS do slightly more harm than non-
draws conclusions. ECN, because the AQM doesn't immediately clear ECT packets from
the queue until it reaches overload and disables ECN.
4.4.1. Cwnd Response to CE-Marked Pure ACKs Standards policy: Even if there were no harm to others, does it set
an undesirable precedent to allow a flow to use ECN to protect its
Pure ACKs from loss, when there is no mechanism to respond to ECN-
marking?
If the sender of pure ACKs sets them to ECT, the bullets below assess The last two arguments involve value judgements, but they both depend
whether the three stages of the congestion response mechanism will on the concrete technical question of mechanism feasibility, which
all work for each type of congestion feedback (classic ECN [RFC3168] will therefore be addressed first in Section 4.4.1 below. Then
and AccECN [I-D.ietf-tcpm-accurate-ecn]): Section 4.4.2 draws conclusions by addressing the value judgements in
the other two questions.
Detection: The receiver of a pure ACK can detect a CE marking on it: 4.4.1. Mechanisms to Respond to CE-Marked Pure ACKs
* Classic feedback: the receiver will not expect CE marks on pure The question of whether the receiver of pure ACKs is required to
ACKs, so it will be implementation-dependent whether it happens detect and feed back any CE-marking is outside the scope of the
to check for CE marks on all packets. present specification - it is a matter for the relevant feedback
specification (classic ECN [RFC3168] and AccECN
[I-D.ietf-tcpm-accurate-ecn]). The response to congestion feedback
is also out of scope, because it would be defined in the base TCP
congestion control specification [RFC5681] or its variants.
Nonetheless, in order to decide whether the present ECN++
experimental specification should require a host to set ECT on pure
ACKs, we only need to know whether a response mechanism would be
feasible - we do not have to standardize it. So the bullets below
assess, for each type of feedback, whether the three stages of the
congestion response mechanism could all work.
Detection: Can the receiver of a pure ACK detect a CE marking on
it?:
* Classic feedback: RFC 3168 is silent on this point. The
implementer of the receiver would not expect CE marks on pure
ACKs, but the implementation might happen to check for CE marks
before it looks for the data. So detection will be
implementation-dependent.
* AccECN feedback: the AccECN specification requires the receiver * AccECN feedback: the AccECN specification requires the receiver
of any TCP packets to count any CE marks on them (whether or of any TCP packets to count any CE marks on them (whether or
not control packets are ECN-capable). not it sends ECN-capable control packets itself).
Feedback: TCP never ACKs a pure ACK, but the receiver of a CE-mark Feedback: TCP never ACKs a pure ACK, but the receiver of a CE-mark
on a pure ACK can feed it back when it sends a subsequent data on a pure ACK could feed it back when it sends a subsequent data
segment (if it ever does): segment (if it ever does):
* Classic feedback: the receiver (of the pure ACKs) would set the * Classic feedback: RFC 3168 is silent on this point, so feedback
echo congestion experienced (ECE) flag in the TCP header as of CE-markings might be implementation specific. If the
normal. receiver (of the pure ACKs) did generate feedback, it would set
the echo congestion experienced (ECE) flag in the TCP header of
subsequent packets in the round, as it would to feed back CE on
data packets.
* AccECN feedback: the receiver continually feeds back a count of * AccECN feedback: the receiver continually feeds back a count of
the number of CE-marked packets that it has received (and, if the number of CE-marked packets that it has received and,
possible, a count of CE-marked bytes). optionally, a count of CE-marked bytes. For either metric,
AccECN includes pure ACKs and indeed all types of packets.
Congestion response: In either case (classic or AccECN feedback), if Congestion response: In either case (classic or AccECN feedback), if
the TCP sender does receive feedback about CE-markings on pure the TCP sender does receive feedback about CE-markings on pure
ACKs, it will react in the usual way by reducing its congestion ACKs, it will be able to reduce the congestion window (cwnd) and/
window accordingly. This will regulate the rate of any data or the ACK rate.
packets it is sending amongst the pure ACKs. Note that, while a
host has no application data to send, any congestion window it has
attained might also be reduced by the congestion window validation
mechanism [RFC7661].
4.4.2. ACK Rate Response to CE-Marked Pure ACKs Therefore a congestion response mechanism is clearly feasible if
AccECN has been negotiated, but the position is unknown for the
installed base of classic ECN feedback.
4.4.1.1. Congestion Window Response to CE-Marked Pure ACKs
This subsection explores issues that congestion control designers
will need to consider when defining a cwnd response to CE-marked Pure
ACKs.
A CE-mark on a Pure ACK does not mean that only Pure ACKs are causing
congestion. It only means that the marked Pure ACK is part of an
aggregate that is collectively causing a bottleneck queue to randomly
CE-mark a fraction of the packets. A CE-mark on a Pure ACK might be
due to data packets in other flows through the same bottleneck, due
to data packets interspersed between Pure ACKs in the same half-
connection, or just due to the rate of Pure ACKs alone. (RFC 3168
only considered the last possibility, which led to the argument that
ECN-enabled Pure ACKs had to be deferred, because ACK congestion
control was a research issue.)
If a host has been sending a mix of Pure ACKs and data, it doesn't
need to work out whether a particular CE mark was on a Pure ACK or
not; it just needs to respond to congestion feedback as a whole by
reducing its congestion window (cwnd), which limits the data it can
launch into flight through the congested bottleneck. If it is purely
receiving data and sending only Pure ACKs, reducing cwnd will have
caused it no harm, having no effect on its ACK rate (the next
subsection addresses that).
However, when a host is sending data as well as Pure ACKs, it would
not be right for CE-marks on Pure ACKs and on data packets to induce
the same reduction in cwnd. A possible way to address this issue
would be to weight the response by the size of the marked packets.
For instance, one could calculate the fraction of CE-marked bytes
(headers and data) over each round trip (say) as follows:
(CE-marked header bytes + CE-marked data bytes) / (all header
bytes + all data bytes)
Header bytes can be calculated by multiplying a packet count by a
nominal header size, which is possible with AccECN feedback, because
it gives a count of CE-marked packets (as well as CE-marked bytes).
The above simple aggregate calculation caters for the full range of
scenarios; from all Pure ACKs to just a few interspersed with data
packets.
Note that any mechanism that reduces cwnd due to CE-marked Pure ACKs
would need to be integrated with the congestion window validation
mechanism [RFC7661], which already conservatively reduces cwnd over
time because cwnd becomes stale if it is not used to fill the pipe.
4.4.1.2. ACK Rate Response to CE-Marked Pure ACKs
Reducing the congestion window will have no effect on the rate of Reducing the congestion window will have no effect on the rate of
pure ACKs. The worst case here is if the bottleneck is congested pure ACKs. The worst case here is if the bottleneck is congested
solely with pure ACKs, but it could also be problematic if a large solely with pure ACKs, but it could also be problematic if a large
fraction of the load was from unresponsive ACKs, leaving little or no fraction of the load was from unresponsive ACKs, leaving little or no
capacity for the load from responsive data. capacity for the load from responsive data.
Since RFC 3168 was published, Acknowledgement Congestion Control Since RFC 3168 was published, experimental Acknowledgement Congestion
(AckCC) techniques have been documented in [RFC5690] (informational). Control (AckCC) techniques have been documented in [RFC5690]
So any pair of TCP end-points can choose to agree to regulate the (informational). So any pair of TCP end-points can choose to agree
delayed ACK ratio in response to lost or CE-marked pure ACKs. to regulate the delayed ACK ratio in response to lost or CE-marked
However, the protocol has a number of open deployment issues (e.g. it pure ACKs. However, the protocol has a number of open issues
concerning deployment (e.g. it requires support from both ends, it
relies on two new TCP options, one of which is required on the SYN relies on two new TCP options, one of which is required on the SYN
where option space is at a premium and, if either option is blocked where option space is at a premium and, if either option is blocked
by a middlebox, no fall-back behaviour is specified). The new TCP by a middlebox, no fall-back behaviour is specified).
options addressed two problems, namely that TCP had: i) no mechanism
to allow ECT to be set on pure ACKs; and ii) no mechanism to feed The new TCP options address two problems, namely that TCP had: i) no
back loss or CE-marking of pure ACKs. A combination of the present mechanism to allow ECT to be set on pure ACKs; and ii) no mechanism
specification and AccECN addresses both these problems, at least for to feed back loss or CE-marking of pure ACKs. A combination of the
ECN marking. So it might now be possible to design an ECN-specific present specification and AccECN addresses both these problems, at
ACK congestion control scheme without the extra TCP options proposed least for CE-marking. So it might now be possible to design an ECN-
in RFC 5690. However, such a mechanism is out of scope of the specific ACK congestion control scheme without the extra TCP options
present document. proposed in RFC 5690. However, such a mechanism is out of scope of
the present document.
Setting aside the practicality of RFC 5690, the need for AckCC has Setting aside the practicality of RFC 5690, the need for AckCC has
not been conclusively demonstrated. It has been argued that the not been conclusively demonstrated. It has been argued that the
Internet has survived so far with no mechanism to even detect loss of Internet has survived so far with no mechanism to even detect loss of
pure ACKs. However, it has also been argued that ECN is not the same pure ACKs. However, it has also been argued that ECN is not the same
as loss. Packet discard can naturally thin the ACK load to whatever as loss. Packet discard can naturally thin the ACK load to whatever
the bottleneck can support, whereas ECN marking does not (it queues the bottleneck can support, whereas ECN marking does not (it queues
the ACKs instead). Nonetheless, RFC 3168 (section 7) recommends that the ACKs instead). Nonetheless, RFC 3168 (section 7) recommends that
an AQM switches over from ECN marking to discard when the marking an AQM switches over from ECN marking to discard when the marking
probability becomes high. Therefore discard can still be relied on probability becomes high. Therefore discard can still be relied on
to thin out ECN-enabled pure ACKs as a last resort. to thin out ECN-enabled pure ACKs as a last resort.
4.4.3. Summary: Enabling ECN on Pure ACKs 4.4.2. Summary: Enabling ECN on Pure ACKs
In the case when AccECN has been negotiated, the arguments for ECT In the case when AccECN has been negotiated, it provides a feasible
(and CE) on pure ACKs heavily outweigh those against. ECN is always congestion response mechanism, so the arguments for ECT on pure ACKs
more and never less reliable for delivery of congestion notification. heavily outweigh those against. ECN is always more and never less
The cwnd response has been overlooked as a mechanism for responding reliable for delivery of congestion notification. A cwnd reduction
to congestion on pure ACKs, so it is incorrect not to set ECT on pure needs to be considered by congestion control designers as a response
ACKs when they are interspersed with data segments. And when they to congestion on pure ACKs. Separately, AckCC (or an improved
are not, packet discard still acts as the "congestion response of variant exploiting AccECN) could optionally be used to regulate the
last resort". In contrast, not setting ECT on pure ACKs is certainly spacing between pure ACKs. However, it is not clear whether AckCC is
detrimental to performance, because when a pure ACK is lost it can justified. If it is not, packet discard will still act as the
prevent the release of new data. Separately, AckCC (or perhaps an "congestion response of last resort" by thinning out the traffic. In
improved variant exploiting AccECN) could optionally be used to contrast, not setting ECT on pure ACKs is certainly detrimental to
regulate the spacing between pure ACKs. However, it is not clear performance, because when a pure ACK is lost it can prevent the
whether AckCC is justified. release of new data.
In the case when Classic ECN has been negotiated, there is still an In the case when Classic ECN has been negotiated, the argument for
argument for ECT (and CE) on pure ACKs, but it is less clear-cut. ECT on pure ACKs is less clear-cut. Some of the installed base of
Some existing RFC 3168 implementations might happen to RFC 3168 implementations might happen to (unintentionally) provide a
(unintentionally) provide the correct feedback to support a cwnd feedback mechanism to support a cwnd response. For those that did
response. Even for those that did not, setting ECT on pure ACKs not, setting ECT on pure ACKs would be better for the flow's own
would still be better for performance than not setting it and do no performance than not setting it. However, where there was no
extra harm. If AckCC was required, it is designed to work with RFC feedback mechanism, setting ECT could do slightly more harm than not
3168 ECN. setting it. AckCC could provide a complementary response mechanism,
because it is designed to work with RFC 3168 ECN, but it has
deployment challenges. In summary, a congestion response mechanism
is unlikely to be feasible with the installed base of classic ECN.
During review of this specification, it was decided that allowing
hosts to set ECT on Pure ACKs without a feasible response mechanism
would set an undesirable precedent. It would certainly improve the
flow's own performance, but it would slightly increase potential harm
to others. Therefore, Section 3.2.3 allows ECT on Pure ACKs if
AccECN feedback has been negotiated, but not with classic RFC 3168
ECN feedback.
4.5. Window Probes 4.5. Window Probes
Section 6.1.6 of RFC 3168 presents only the reliability argument for Section 6.1.6 of RFC 3168 presents only the reliability argument for
prohibiting ECT on Window probes: prohibiting ECT on Window probes:
"If a window probe packet is dropped in the network, this loss is "If a window probe packet is dropped in the network, this loss is
not detected by the receiver. Therefore, the TCP data sender MUST not detected by the receiver. Therefore, the TCP data sender MUST
NOT set either an ECT codepoint or the CWR bit on window probe NOT set either an ECT codepoint or the CWR bit on window probe
packets. packets.
skipping to change at page 32, line 22 skipping to change at page 38, line 22
If the initiator implements IW10, it seems rather over-conservative If the initiator implements IW10, it seems rather over-conservative
to reduce IW from 10 to 1 just in case a congestion marking was to reduce IW from 10 to 1 just in case a congestion marking was
missed. Nonetheless, the reduction to 1 SMSS will rarely harm missed. Nonetheless, the reduction to 1 SMSS will rarely harm
performance, because: performance, because:
o as long as the initiator is caching failures to negotiate AccECN, o as long as the initiator is caching failures to negotiate AccECN,
subsequent attempts to access the same server will not use ECT on subsequent attempts to access the same server will not use ECT on
the SYN anyway, so there will no longer be any need to the SYN anyway, so there will no longer be any need to
conservatively reduce IW; conservatively reduce IW;
o currently it is not common for a TCP initiator (client) to have o currently, at least for web sessions, it is extremely rare for a
more than one data segment to send {ToDo: evidence/reference?} - TCP initiator (client) to have more than one data segment to send
IW10 is primarily exploited by TCP servers. at the start of a TCP connection [28; Fig 3] - IW10 is primarily
exploited by TCP servers.
If a responder receives feedback that the SYN-ACK was CE-marked, If a responder receives feedback that the SYN-ACK was CE-marked,
Section 3.2.2.2 mandates that it reduces its initial window to 1 Section 3.2.2.2 mandates that it reduces its initial window to 1
SMSS. When the responder also implements IW10, it is particularly SMSS. When the responder also implements IW10, it is particularly
important to adhere to this requirement in order to avoid overflowing important to adhere to this requirement in order to avoid overflowing
a queue that is clearly already congested. a queue that is clearly already congested.
5.2. TFO 5.2. TFO
TCP Fast Open (TFO [RFC7413]) is an experiment to remove the round TCP Fast Open (TFO [RFC7413]) is an experiment to remove the round
skipping to change at page 33, line 7 skipping to change at page 39, line 11
o The handling of ECN marking on a SYN is no different whether or o The handling of ECN marking on a SYN is no different whether or
not it carries data. not it carries data.
o In response to any CE-marking on the SYN-ACK, the responder adopts o In response to any CE-marking on the SYN-ACK, the responder adopts
the normal response to congestion, as discussed in Section 7.2 of the normal response to congestion, as discussed in Section 7.2 of
[RFC7413]. [RFC7413].
5.3. TCP Derivatives 5.3. TCP Derivatives
Experience from experiments on adding ECN support to all TCP packets
ought to be directly transferable between TCP and derivatives of TCP,
like SCTP or QUIC.
Stream Control Transmission Protocol (SCTP [RFC4960]) is a standards Stream Control Transmission Protocol (SCTP [RFC4960]) is a standards
track transport protocol derived from TCP. SCTP currently does not track transport protocol derived from TCP. SCTP currently does not
include ECN support, but Appendix A of RFC 4960 broadly describes how include ECN support, but Appendix A of RFC 4960 broadly describes how
it would be supported and a (long-expired) draft on the addition of it would be supported and a (long-expired) draft on the addition of
ECN to SCTP has been produced [I-D.stewart-tsvwg-sctpecn]. This ECN to SCTP has been produced [I-D.stewart-tsvwg-sctpecn]. This
draft avoided setting ECT on control packets and retransmissions, draft avoided setting ECT on control packets and retransmissions,
closely following the arguments in RFC 3168. closely following the arguments in RFC 3168.
QUIC [I-D.ietf-quic-transport] is another standards track transport QUIC [I-D.ietf-quic-transport] is another standards track transport
protocol offering similar services to TCP but intended to exploit protocol offering similar services to TCP but intended to exploit
some of the benefits of running over UDP. A way to add ECN support some of the benefits of running over UDP. Building on the arguments
to QUIC has been proposed [I-D.johansson-quic-ecn]. in the current draft, a QUIC sender sets ECT(0) on all packets.
Experience from experiments on adding ECN support to all TCP packets
ought to be directly transferable to derivatives of TCP, like SCTP or
QUIC.
6. Security Considerations 6. Security Considerations
Section 3.2.6 considers the question of whether ECT on RSTs will Section 3.2.6 considers the question of whether ECT on RSTs will
allow RST attacks to be intensified. There are several security allow RST attacks to be intensified. There are several security
arguments presented in RFC 3168 for preventing the ECN marking of TCP arguments presented in RFC 3168 for preventing the ECN marking of TCP
control packets and retransmitted segments. We believe all of them control packets and retransmitted segments. We believe all of them
have been properly addressed in Section 4, particularly Section 4.2.3 have been properly addressed in Section 4, particularly Section 4.2.4
and Section 4.8 on DoS attacks using spoofed ECT-marked SYNs and and Section 4.8 on DoS attacks using spoofed ECT-marked SYNs and
spoofed CE-marked retransmissions. spoofed CE-marked retransmissions.
7. IANA Considerations 7. IANA Considerations
There are no IANA considerations in this memo. There are no IANA considerations in this memo.
8. Acknowledgments 8. Acknowledgments
Thanks to Mirja Kuehlewind, David Black, Padma Bhooma and Gorry Thanks to Mirja Kuehlewind, David Black, Padma Bhooma, Gorry
Fairhurst for their useful reviews. Fairhurst, Michael Scharf, Yuchung Cheng and Christophe Paasch for
their useful reviews.
The work of Marcelo Bagnulo has been performed in the framework of The work of Marcelo Bagnulo has been performed in the framework of
the H2020-ICT-2014-2 project 5G NORMA. His contribution reflects the the H2020-ICT-2014-2 project 5G NORMA. His contribution reflects the
consortium's view, but the consortium is not liable for any use that consortium's view, but the consortium is not liable for any use that
may be made of any of the information contained therein. may be made of any of the information contained therein.
Bob Briscoe's contribution was partly funded by the Research Council Bob Briscoe's contribution was partly funded by the Research Council
of Norway through the TimeIn project. The views expressed here are of Norway through the TimeIn project. The views expressed here are
solely those of the authors. solely those of the authors.
9. References 9. References
9.1. Normative References 9.1. Normative References
[I-D.ietf-tcpm-accurate-ecn] [I-D.ietf-tcpm-accurate-ecn]
Briscoe, B., Kuehlewind, M., and R. Scheffenegger, "More Briscoe, B., Kuehlewind, M., and R. Scheffenegger, "More
Accurate ECN Feedback in TCP", draft-ietf-tcpm-accurate- Accurate ECN Feedback in TCP", draft-ietf-tcpm-accurate-
ecn-03 (work in progress), May 2017. ecn-07 (work in progress), July 2018.
[I-D.ietf-tsvwg-ecn-experimentation]
Black, D., "Relaxing Restrictions on Explicit Congestion
Notification (ECN) Experimentation", draft-ietf-tsvwg-ecn-
experimentation-07 (work in progress), October 2017.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997, DOI 10.17487/RFC2119, March 1997,
<https://www.rfc-editor.org/info/rfc2119>. <https://www.rfc-editor.org/info/rfc2119>.
[RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition [RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition
of Explicit Congestion Notification (ECN) to IP", of Explicit Congestion Notification (ECN) to IP",
RFC 3168, DOI 10.17487/RFC3168, September 2001, RFC 3168, DOI 10.17487/RFC3168, September 2001,
<https://www.rfc-editor.org/info/rfc3168>. <https://www.rfc-editor.org/info/rfc3168>.
[RFC5961] Ramaiah, A., Stewart, R., and M. Dalal, "Improving TCP's [RFC5961] Ramaiah, A., Stewart, R., and M. Dalal, "Improving TCP's
Robustness to Blind In-Window Attacks", RFC 5961, Robustness to Blind In-Window Attacks", RFC 5961,
DOI 10.17487/RFC5961, August 2010, DOI 10.17487/RFC5961, August 2010,
<https://www.rfc-editor.org/info/rfc5961>. <https://www.rfc-editor.org/info/rfc5961>.
[RFC8311] Black, D., "Relaxing Restrictions on Explicit Congestion
Notification (ECN) Experimentation", RFC 8311,
DOI 10.17487/RFC8311, January 2018,
<https://www.rfc-editor.org/info/rfc8311>.
9.2. Informative References 9.2. Informative References
[ecn-overload]
Steen, H., "Destruction Testing: Ultra-Low Delay using
Dual Queue Coupled Active Queue Management", Masters
Thesis, Uni Oslo , May 2017,
<https://www.duo.uio.no/bitstream/handle/10852/57424/
thesis-henrste.pdf?sequence=1>.
[ecn-pam] Trammell, B., Kuehlewind, M., Boppart, D., Learmonth, I., [ecn-pam] Trammell, B., Kuehlewind, M., Boppart, D., Learmonth, I.,
Fairhurst, G., and R. Scheffenegger, "Enabling Internet- Fairhurst, G., and R. Scheffenegger, "Enabling Internet-
Wide Deployment of Explicit Congestion Notification", Wide Deployment of Explicit Congestion Notification",
Int'l Conf. on Passive and Active Network Measurement Int'l Conf. on Passive and Active Network Measurement
(PAM'15) pp193-205, 2015. (PAM'15) pp193-205, 2015, <https://link.springer.com/
chapter/10.1007/978-3-319-15509-8_15>.
[ECN-PLUS] [ECN-PLUS]
Kuzmanovic, A., "The Power of Explicit Congestion Kuzmanovic, A., "The Power of Explicit Congestion
Notification", ACM SIGCOMM 35(4):61--72, 2005. Notification", ACM SIGCOMM 35(4):61--72, 2005,
<http://dl.acm.org/citation.cfm?id=1080100>.
[I-D.ietf-quic-transport] [I-D.ietf-quic-transport]
Iyengar, J. and M. Thomson, "QUIC: A UDP-Based Multiplexed Iyengar, J. and M. Thomson, "QUIC: A UDP-Based Multiplexed
and Secure Transport", draft-ietf-quic-transport-07 (work and Secure Transport", draft-ietf-quic-transport-15 (work
in progress), October 2017. in progress), October 2018.
[I-D.ietf-tsvwg-ecn-l4s-id] [I-D.ietf-tsvwg-ecn-l4s-id]
Schepper, K. and B. Briscoe, "Identifying Modified Schepper, K. and B. Briscoe, "Identifying Modified
Explicit Congestion Notification (ECN) Semantics for Explicit Congestion Notification (ECN) Semantics for
Ultra-Low Queuing Delay", draft-ietf-tsvwg-ecn-l4s-id-00 Ultra-Low Queuing Delay (L4S)", draft-ietf-tsvwg-ecn-l4s-
(work in progress), May 2017. id-03 (work in progress), July 2018.
[I-D.ietf-tsvwg-l4s-arch] [I-D.ietf-tsvwg-l4s-arch]
Briscoe, B., Schepper, K., and M. Bagnulo, "Low Latency, Briscoe, B., Schepper, K., and M. Bagnulo, "Low Latency,
Low Loss, Scalable Throughput (L4S) Internet Service: Low Loss, Scalable Throughput (L4S) Internet Service:
Architecture", draft-ietf-tsvwg-l4s-arch-00 (work in Architecture", draft-ietf-tsvwg-l4s-arch-02 (work in
progress), May 2017. progress), March 2018.
[I-D.johansson-quic-ecn]
Johansson, I., "ECN support in QUIC", draft-johansson-
quic-ecn-03 (work in progress), May 2017.
[I-D.stewart-tsvwg-sctpecn] [I-D.stewart-tsvwg-sctpecn]
Stewart, R., Tuexen, M., and X. Dong, "ECN for Stream Stewart, R., Tuexen, M., and X. Dong, "ECN for Stream
Control Transmission Protocol (SCTP)", draft-stewart- Control Transmission Protocol (SCTP)", draft-stewart-
tsvwg-sctpecn-05 (work in progress), January 2014. tsvwg-sctpecn-05 (work in progress), January 2014.
[judd-nsdi] [judd-nsdi]
Judd, G., "Attaining the promise and avoiding the pitfalls Judd, G., "Attaining the promise and avoiding the pitfalls
of TCP in the Datacenter", USENIX Symposium on Networked of TCP in the Datacenter", USENIX Symposium on Networked
Systems Design and Implementation (NSDI'15) pp.145-157, Systems Design and Implementation (NSDI'15) pp.145-157,
May 2015. May 2015, <https://www.usenix.org/node/188966>.
[Kuehlewind18]
Kuehlewind, M., Walter, M., Learmonth, I., and B.
Trammell, "Tracing Internet Path Transparency", In Proc:
Network Traffic Measurement and Analysis Conference (TMA)
2018 , June 2018, <http://tma.ifip.org/2018/wp-
content/uploads/sites/3/2018/06/tma2018_paper12.pdf>.
[Mandalari18] [Mandalari18]
Mandalari, A., Lutu, A., Briscoe, B., Bagnulo, M., and Oe. Mandalari, A., Lutu, A., Briscoe, B., Bagnulo, M., and Oe.
Alay, "Measuring ECN++: Good News for ++, Bad News for ECN Alay, "Measuring ECN++: Good News for ++, Bad News for ECN
over Mobile", IEEE Communications Magazine , March 2018. over Mobile", IEEE Communications Magazine , March 2018,
<https://ieeexplore.ieee.org/document/8316790>.
(to appear) [Manzoor17]
Manzoor, J., Drago, I., and R. Sadre, "How HTTP/2 is
changing Web traffic and how to detect it", In Proc:
Network Traffic Measurement and Analysis Conference (TMA)
2017 pp.1-9, June 2017,
<https://ieeexplore.ieee.org/document/8002899>.
[RFC0793] Postel, J., "Transmission Control Protocol", STD 7, [RFC0793] Postel, J., "Transmission Control Protocol", STD 7,
RFC 793, DOI 10.17487/RFC0793, September 1981, RFC 793, DOI 10.17487/RFC0793, September 1981,
<https://www.rfc-editor.org/info/rfc793>. <https://www.rfc-editor.org/info/rfc793>.
[RFC1122] Braden, R., Ed., "Requirements for Internet Hosts - [RFC1122] Braden, R., Ed., "Requirements for Internet Hosts -
Communication Layers", STD 3, RFC 1122, Communication Layers", STD 3, RFC 1122,
DOI 10.17487/RFC1122, October 1989, DOI 10.17487/RFC1122, October 1989,
<https://www.rfc-editor.org/info/rfc1122>. <https://www.rfc-editor.org/info/rfc1122>.
skipping to change at page 37, line 10 skipping to change at page 43, line 29
[RFC7661] Fairhurst, G., Sathiaseelan, A., and R. Secchi, "Updating [RFC7661] Fairhurst, G., Sathiaseelan, A., and R. Secchi, "Updating
TCP to Support Rate-Limited Traffic", RFC 7661, TCP to Support Rate-Limited Traffic", RFC 7661,
DOI 10.17487/RFC7661, October 2015, DOI 10.17487/RFC7661, October 2015,
<https://www.rfc-editor.org/info/rfc7661>. <https://www.rfc-editor.org/info/rfc7661>.
[RFC8257] Bensley, S., Thaler, D., Balasubramanian, P., Eggert, L., [RFC8257] Bensley, S., Thaler, D., Balasubramanian, P., Eggert, L.,
and G. Judd, "Data Center TCP (DCTCP): TCP Congestion and G. Judd, "Data Center TCP (DCTCP): TCP Congestion
Control for Data Centers", RFC 8257, DOI 10.17487/RFC8257, Control for Data Centers", RFC 8257, DOI 10.17487/RFC8257,
October 2017, <https://www.rfc-editor.org/info/rfc8257>. October 2017, <https://www.rfc-editor.org/info/rfc8257>.
[strict-ecn]
Dumazet, E., "tcp: be more strict before accepting ECN
negociation", Linux netdev patch list , May 2012,
<https://github.com/torvalds/linux/commit/
bd14b1b2e29bd6812597f896dde06eaf7c6d2f24#diff-
5c7c60ed5f9efb6bce97ff5233f17282>.
Authors' Addresses Authors' Addresses
Marcelo Bagnulo Marcelo Bagnulo
Universidad Carlos III de Madrid Universidad Carlos III de Madrid
Av. Universidad 30 Av. Universidad 30
Leganes, Madrid 28911 Leganes, Madrid 28911
SPAIN SPAIN
Phone: 34 91 6249500 Phone: 34 91 6249500
Email: marcelo@it.uc3m.es Email: marcelo@it.uc3m.es
 End of changes. 111 change blocks. 
520 lines changed or deleted 847 lines changed or added

This html diff was produced by rfcdiff 1.47. The latest version is available from http://tools.ietf.org/tools/rfcdiff/