draft-ietf-tsvwg-l4s-arch-04.txt   draft-ietf-tsvwg-l4s-arch-05.txt 
Transport Area Working Group B. Briscoe, Ed. Transport Area Working Group B. Briscoe, Ed.
Internet-Draft CableLabs Internet-Draft Independent
Intended status: Informational K. De Schepper Intended status: Informational K. De Schepper
Expires: January 9, 2020 Nokia Bell Labs Expires: August 23, 2020 Nokia Bell Labs
M. Bagnulo Braun M. Bagnulo Braun
Universidad Carlos III de Madrid Universidad Carlos III de Madrid
G. White G. White
CableLabs CableLabs
July 8, 2019 February 20, 2020
Low Latency, Low Loss, Scalable Throughput (L4S) Internet Service: Low Latency, Low Loss, Scalable Throughput (L4S) Internet Service:
Architecture Architecture
draft-ietf-tsvwg-l4s-arch-04 draft-ietf-tsvwg-l4s-arch-05
Abstract Abstract
This document describes the L4S architecture for the provision of a This document describes the L4S architecture, which enables Internet
new Internet service that could eventually replace best efforts for applications to achieve Low Latency, Low Loss, and Scalable
all traffic: Low Latency, Low Loss, Scalable throughput (L4S). It is throughput (L4S), while coexisting on shared network bottlenecks with
becoming common for _all_ (or most) applications being run by a user existing Internet applications that are not built to take advantage
at any one time to require low latency. However, the only solution of this new technology.
the IETF can offer for ultra-low queuing delay is Diffserv, which
only favours a minority of packets at the expense of others. In
extensive testing the new L4S service keeps average queuing delay
under a millisecond for _all_ applications even under very heavy
load, without sacrificing utilization; and it keeps congestion loss
to zero. It is becoming widely recognized that adding more access
capacity gives diminishing returns, because latency is becoming the
critical problem. Even with a high capacity broadband access, the
reduced latency of L4S remarkably and consistently improves
performance under load for applications such as interactive video,
conversational video, voice, Web, gaming, instant messaging, remote
desktop and cloud-based apps (even when all being used at once over
the same access link). The insight is that the root cause of queuing
delay is in TCP, not in the queue. By fixing the sending TCP (and
other transports) queuing latency becomes so much better than today
that operators will want to deploy the network part of L4S to enable
new products and services. Further, the network part is simple to
deploy - incrementally with zero-config. Both parts, sender and
network, ensure coexistence with other legacy traffic. At the same
time L4S solves the long-recognized problem with the future
scalability of TCP throughput.
This document describes the L4S architecture, briefly describing the In traditional bottleneck links that utilize a single, shared egress
different components and how the work together to provide the queue, a variety of application traffic flows can share the
aforementioned enhanced Internet service. bottleneck queue simultaneously. As a result, each sender's behavior
and its response to the congestion signals (delay, packet drop, ECN
marking) provided by the queue can impact the performance of all
other applications that share the link. Furthermore, it is
considered important that new protocols coexist in a reasonably fair
manner with existing protocols (most notably TCP and QUIC). As a
result, senders tend to normalize on behaviors that are not
significantly different than those in use by the majority of the
existing senders. For many years, the majority of traffic on the
Internet has used either the Reno AIMD congestion controller or the
Cubic algorithm, and as a result any newly proposed congestion
controller needs to demonstrate that it provides reasonable fairness
when sharing a bottleneck with flows that use Reno or Cubic. This
has led to an ossification in congestion control, where improved
congestion controllers cannot easily be deployed on the Internet.
It is well known that the common existing congestion controllers
(e.g. Reno and Cubic) increase their congestion window (the amount
of data in flight) until they induce congestion, and they respond to
the congestion signals of packet loss (or equivalently ECN marks) by
significantly reducing their congestion window. This leads to a
large sawtooth of the congestion window that manifests itself as a
combination of queue delay and/or link underutilization.
Meanwhile, in closed network environments, such as data centres, new
congestion controllers (e.g. DCTCP [RFC8257]) have been deployed
that significantly outperform Reno and Cubic in terms of queue delay
and link utilization across a much wider range of network conditions.
The L4S architecture provides an approach that allows for the
deployment of next generation congestion controllers while preserving
reasonably fair coexistence with Reno and Cubic.
The L4S architecture consists of three components: network support to
isolate L4S traffic from other traffic and to provide appropriate
congestion signaling to both types; protocol features that allow
network elements to identify L4S traffic and allow for communication
of congestion signaling; and host support for immediate congestion
signaling and an appropriate congestion response that enables
scalable performance.
Status of This Memo Status of This Memo
This Internet-Draft is submitted in full conformance with the This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79. provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/. Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on January 9, 2020. This Internet-Draft will expire on August 23, 2020.
Copyright Notice Copyright Notice
Copyright (c) 2019 IETF Trust and the persons identified as the Copyright (c) 2020 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(https://trustee.ietf.org/license-info) in effect on the date of (https://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License. described in the Simplified BSD License.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3
2. L4S Architecture Overview . . . . . . . . . . . . . . . . . . 4 2. L4S Architecture Overview . . . . . . . . . . . . . . . . . . 5
3. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 6 3. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 6
4. L4S Architecture Components . . . . . . . . . . . . . . . . . 7 4. L4S Architecture Components . . . . . . . . . . . . . . . . . 8
5. Rationale . . . . . . . . . . . . . . . . . . . . . . . . . . 10 5. Rationale . . . . . . . . . . . . . . . . . . . . . . . . . . 11
5.1. Why These Primary Components? . . . . . . . . . . . . . . 10 5.1. Why These Primary Components? . . . . . . . . . . . . . . 11
5.2. Why Not Alternative Approaches? . . . . . . . . . . . . . 12 5.2. Why Not Alternative Approaches? . . . . . . . . . . . . . 13
6. Applicability . . . . . . . . . . . . . . . . . . . . . . . . 15 6. Applicability . . . . . . . . . . . . . . . . . . . . . . . . 15
6.1. Applications . . . . . . . . . . . . . . . . . . . . . . 15 6.1. Applications . . . . . . . . . . . . . . . . . . . . . . 15
6.2. Use Cases . . . . . . . . . . . . . . . . . . . . . . . . 16 6.2. Use Cases . . . . . . . . . . . . . . . . . . . . . . . . 17
6.3. Deployment Considerations . . . . . . . . . . . . . . . . 17 6.3. Deployment Considerations . . . . . . . . . . . . . . . . 18
6.3.1. Deployment Topology . . . . . . . . . . . . . . . . . 18 6.3.1. Deployment Topology . . . . . . . . . . . . . . . . . 19
6.3.2. Deployment Sequences . . . . . . . . . . . . . . . . 19 6.3.2. Deployment Sequences . . . . . . . . . . . . . . . . 20
6.3.3. L4S Flow but Non-L4S Bottleneck . . . . . . . . . . . 21 6.3.3. L4S Flow but Non-L4S Bottleneck . . . . . . . . . . . 22
6.3.4. Other Potential Deployment Issues . . . . . . . . . . 23 6.3.4. Other Potential Deployment Issues . . . . . . . . . . 23
7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 23 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 23
8. Security Considerations . . . . . . . . . . . . . . . . . . . 23 8. Security Considerations . . . . . . . . . . . . . . . . . . . 23
8.1. Traffic (Non-)Policing . . . . . . . . . . . . . . . . . 23 8.1. Traffic (Non-)Policing . . . . . . . . . . . . . . . . . 23
8.2. 'Latency Friendliness' . . . . . . . . . . . . . . . . . 24 8.2. 'Latency Friendliness' . . . . . . . . . . . . . . . . . 24
8.3. Interaction between Rate Policing and L4S . . . . . . . . 24 8.3. Interaction between Rate Policing and L4S . . . . . . . . 25
8.4. ECN Integrity . . . . . . . . . . . . . . . . . . . . . . 25 8.4. ECN Integrity . . . . . . . . . . . . . . . . . . . . . . 26
9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 26 9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 26
10. References . . . . . . . . . . . . . . . . . . . . . . . . . 26 10. References . . . . . . . . . . . . . . . . . . . . . . . . . 26
10.1. Normative References . . . . . . . . . . . . . . . . . . 26 10.1. Normative References . . . . . . . . . . . . . . . . . . 26
10.2. Informative References . . . . . . . . . . . . . . . . . 26 10.2. Informative References . . . . . . . . . . . . . . . . . 27
Appendix A. Standardization items . . . . . . . . . . . . . . . 32 Appendix A. Standardization items . . . . . . . . . . . . . . . 33
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 34 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 35
1. Introduction 1. Introduction
It is increasingly common for _all_ of a user's applications at any It is increasingly common for _all_ of a user's applications at any
one time to require low delay: interactive Web, Web services, voice, one time to require low delay: interactive Web, Web services, voice,
conversational video, interactive video, interactive remote presence, conversational video, interactive video, interactive remote presence,
instant messaging, online gaming, remote desktop, cloud-based instant messaging, online gaming, remote desktop, cloud-based
applications and video-assisted remote control of machinery and applications and video-assisted remote control of machinery and
industrial processes. In the last decade or so, much has been done industrial processes. In the last decade or so, much has been done
to reduce propagation delay by placing caches or servers closer to to reduce propagation delay by placing caches or servers closer to
skipping to change at page 3, line 44 skipping to change at page 4, line 14
It has been demonstrated that, once access network bit rates reach It has been demonstrated that, once access network bit rates reach
levels now common in the developed world, increasing capacity offers levels now common in the developed world, increasing capacity offers
diminishing returns if latency (delay) is not addressed. diminishing returns if latency (delay) is not addressed.
Differentiated services (Diffserv) offers Expedited Forwarding (EF Differentiated services (Diffserv) offers Expedited Forwarding (EF
[RFC3246]) for some packets at the expense of others, but this is not [RFC3246]) for some packets at the expense of others, but this is not
sufficient when all (or most) of a user's applications require low sufficient when all (or most) of a user's applications require low
latency. latency.
Therefore, the goal is an Internet service with ultra-Low queueing Therefore, the goal is an Internet service with ultra-Low queueing
Latency, ultra-Low Loss and Scalable throughput (L4S) - for _all_ Latency, ultra-Low Loss and Scalable throughput (L4S). Ultra-low
traffic. A service for all traffic will need none of the queuing latency means less than 1 millisecond (ms) on average and
less than about 2 ms at the 99th percentile. L4S is potentially for
_all_ traffic - a service for all traffic needs none of the
configuration or management baggage (traffic policing, traffic configuration or management baggage (traffic policing, traffic
contracts) associated with favouring some packets over others. This contracts) associated with favouring some traffic over others. This
document describes the L4S architecture for achieving that goal. document describes the L4S architecture for achieving these goals.
It must be said that queuing delay only degrades performance It must be said that queuing delay only degrades performance
infrequently [Hohlfeld14]. It only occurs when a large enough infrequently [Hohlfeld14]. It only occurs when a large enough
capacity-seeking (e.g. TCP) flow is running alongside the user's capacity-seeking (e.g. TCP) flow is running alongside the user's
traffic in the bottleneck link, which is typically in the access traffic in the bottleneck link, which is typically in the access
network. Or when the low latency application is itself a large network. Or when the low latency application is itself a large
capacity-seeking flow (e.g. interactive video). At these times, the capacity-seeking flow (e.g. interactive video). At these times, the
performance improvement from L4S must be so remarkable that network performance improvement from L4S must be sufficient that network
operators will be motivated to deploy it. operators will be motivated to deploy it.
Active Queue Management (AQM) is part of the solution to queuing Active Queue Management (AQM) is part of the solution to queuing
under load. AQM improves performance for all traffic, but there is a under load. AQM improves performance for all traffic, but there is a
limit to how much queuing delay can be reduced by solely changing the limit to how much queuing delay can be reduced by solely changing the
network; without addressing the root of the problem. network; without addressing the root of the problem.
The root of the problem is the presence of standard TCP congestion The root of the problem is the presence of standard TCP congestion
control (Reno [RFC5681]) or compatible variants (e.g. TCP Cubic control (Reno [RFC5681]) or compatible variants (e.g. TCP Cubic
[RFC8312]). We shall call this family of congestion controls [RFC8312]). We shall use the term 'Classic' for these Reno-Friendly
'Classic' TCP. It has been demonstrated that if the sending host congestion controls. It has been demonstrated that if the sending
replaces Classic TCP with a 'Scalable' alternative, when a suitable host replaces a Classic congestion control with a 'Scalable'
AQM is deployed in the network the performance under load of all the alternative, when a suitable AQM is deployed in the network the
above interactive applications can be stunningly improved. For performance under load of all the above interactive applications can
instance, queuing delay under heavy load with the example DCTCP/DualQ be significantly improved. For instance, queuing delay under heavy
solution cited below is roughly 1 millisecond (1 to 2 ms) at the 99th load with the example DCTCP/DualQ solution cited below is roughly 1
percentile without losing link utilization. This compares with 5 to millisecond (1 to 2 ms) at the 99th percentile without losing link
20 ms on _average_ with a Classic TCP and current state-of-the-art utilization. This compares with 5 to 20 ms on _average_ with a
AQMs such as fq_CoDel [RFC8290] or PIE [RFC8033] and about 20-30 ms Classic congestion control and current state-of-the-art AQMs such as
at the 99th percentile. Also, with a Classic TCP, 5 ms of queuing is fq_CoDel [RFC8290] or PIE [RFC8033] and about 20-30 ms at the 99th
usually only possible by losing some utilization. percentile. Also, with a Classic congestion control, reducing
queueing to even 5 ms is typically only possible by losing some
utilization.
It has been convincingly demonstrated [DCttH15] that it is possible It has been demonstrated [DCttH15] that it is possible to deploy such
to deploy such an L4S service alongside the existing best efforts an L4S service alongside the existing best efforts service so that
service so that all of a user's applications can shift to it when all of a user's applications can shift to it when their stack is
their stack is updated. Access networks are typically designed with updated. Access networks are typically designed with one link as the
one link as the bottleneck for each site (which might be a home, bottleneck for each site (which might be a home, small enterprise or
small enterprise or mobile device), so deployment at a single network mobile device), so deployment at a single network node should give
node should give nearly all the benefit. The L4S approach also nearly all the benefit. The L4S approach also requires component
requires component mechanisms at the endpoints to fulfill its goal. mechanisms at the endpoints to fulfill its goal. This document
This document presents the L4S architecture, by describing the presents the L4S architecture, by describing the different components
different components and how they interact to provide the scalable and how they interact to provide the scalable low-latency, low-loss,
low-latency, low-loss, Internet service. Internet service.
2. L4S Architecture Overview 2. L4S Architecture Overview
There are three main components to the L4S architecture (illustrated There are three main components to the L4S architecture (illustrated
in Figure 1): in Figure 1):
1) Network: L4S traffic needs to be isolated from the queuing 1) Network: L4S traffic needs to be isolated from the queuing
latency of Classic traffic. However, the two should be able to latency of Classic traffic. One queue per application flow (FQ)
freely share a common pool of capacity. This is because there is is one way to achieve this, e.g. [RFC8290]. However, just two
no way to predict how many flows at any one time might use each queues is sufficient and does not require inspection of transport
service and capacity in access networks is too scarce to partition layer headers in the network, which is not always possible (see
into two. The Dual Queue Coupled AQM Section 5.2). With just two queues, it might seem impossible to
[I-D.ietf-tsvwg-aqm-dualq-coupled] was developed as a minimal know how much capacity to schedule for each queue without
complexity solution to this problem. The two queues appear to be inspecting how many flows at any one time are using each. And
separated by a 'semi-permeable' membrane that partitions latency capacity in access networks is too costly to arbitrarily partition
but not bandwidth (explained later). into two. The Dual Queue Coupled AQM was developed as a minimal
complexity solution to this problem. It acts like a 'semi-
Per-flow queuing such as in [RFC8290] could be used (see permeable' membrane that partitions latency but not bandwidth.
Section 4), but it partitions both latency and bandwidth between Note that there is no bandwidth priority between the two queues
every end-to-end flow. So it is rather overkill, which brings because they are for transition from Classic to L4S behaviour, not
disadvantages (see Section 5.2), not least that large number of prioritization. Section 4 gives a high level explanation of how
queues are needed when two are sufficient. FQ and DualQ solutions work, and
[I-D.ietf-tsvwg-aqm-dualq-coupled] gives a full explanation of the
Coupled DualQ.
2) Protocol: A host needs to distinguish L4S and Classic packets 2) Protocol: A host needs to distinguish L4S and Classic packets
with an identifier so that the network can classify them into with an identifier so that the network can classify them into
their separate treatments. [I-D.ietf-tsvwg-ecn-l4s-id] considers their separate treatments. [I-D.ietf-tsvwg-ecn-l4s-id] considers
various alternative identifiers, and concludes that all various alternative identifiers, and concludes that all
alternatives involve compromises, but the ECT(1) and CE codepoints alternatives involve compromises, but the ECT(1) and CE codepoints
of the ECN field represent a workable solution. of the ECN field represent a workable solution.
3) Host: Scalable congestion controls already exist. They solve the 3) Host: Scalable congestion controls already exist. They solve the
scaling problem with TCP that was first pointed out in [RFC3649]. scaling problem with Reno congestion control that was explained in
The one used most widely (in controlled environments) is Data [RFC3649]. The one used most widely (in controlled environments)
Center TCP (DCTCP [RFC8257]), which has been implemented and is Data Center TCP (DCTCP [RFC8257]), which has been implemented
deployed in Windows Server Editions (since 2012), in Linux and in and deployed in Windows Server Editions (since 2012), in Linux and
FreeBSD. Although DCTCP as-is 'works' well over the public in FreeBSD. Although DCTCP as-is 'works' well over the public
Internet, most implementations lack certain safety features that Internet, most implementations lack certain safety features that
will be necessary once it is used outside controlled environments will be necessary once it is used outside controlled environments
like data centres (see later). A similar scalable congestion like data centres (see Section 6.3.3 and Appendix A). A similar
control will also need to be transplanted into protocols other scalable congestion control will also need to be transplanted into
than TCP (QUIC, SCTP, RTP/RTCP, RMCAT, etc.) Indeed, between the protocols other than TCP (QUIC, SCTP, RTP/RTCP, RMCAT, etc.)
present document being drafted and published, the following Indeed, between the present document being drafted and published,
scalable congestion controls were implemented: TCP Prague the following scalable congestion controls were implemented: TCP
[PragueLinux], QUIC Prague and an L4S variant of the RMCAT SCReAM Prague [PragueLinux], QUIC Prague, an L4S variant of the RMCAT
controller [RFC8298]. SCReAM controller [RFC8298] and the L4S ECN part of BBRv2
[I-D.cardwell-iccrg-bbr-congestion-control] intended for TCP and
QUIC.
(2) (1) (2) (1)
.-------^------. .--------------^-------------------. .-------^------. .--------------^-------------------.
,-(3)-----. ______ ,-(3)-----. ______
; ________ : L4S --------. | | ; ________ : L4S --------. | |
:|Scalable| : _\ ||___\_| mark | :|Scalable| : _\ ||___\_| mark |
:| sender | : __________ / / || / |______|\ _________ :| sender | : __________ / / || / |______|\ _________
:|________|\; | |/ --------' ^ \1|condit'nl| :|________|\; | |/ --------' ^ \1|condit'nl|
`---------'\_| IP-ECN | Coupling : \|priority |_\ `---------'\_| IP-ECN | Coupling : \|priority |_\
________ / |Classifier| : /|scheduler| / ________ / |Classifier| : /|scheduler| /
skipping to change at page 6, line 33 skipping to change at page 6, line 45
3. Terminology 3. Terminology
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in [RFC2119]. In this document are to be interpreted as described in [RFC2119]. In this
document, these words will appear with that interpretation only when document, these words will appear with that interpretation only when
in ALL CAPS. Lower case uses of these words are not to be in ALL CAPS. Lower case uses of these words are not to be
interpreted as carrying RFC-2119 significance. COMMENT: Since this interpreted as carrying RFC-2119 significance. COMMENT: Since this
will be an information document, This should be removed. will be an information document, This should be removed.
Classic service: The 'Classic' service is intended for all the Classic Congestion Control: A congestion control behaviour that can
congestion control behaviours that currently co-exist with TCP co-exist with standard TCP Reno [RFC5681] without causing flow
Reno (e.g. TCP Cubic, Compound, SCTP, etc). rate starvation. With Classic congestion controls, as flow rate
scales, the number of round trips between congestion signals
(losses or ECN marks) rises with the flow rate. So it takes
longer and longer to recover after each congestion event.
Low-Latency, Low-Loss and Scalable (L4S) service: The 'L4S' service Therefore control of queuing and utilization becomes very slack,
is intended for traffic from scalable TCP algorithms such as Data and the slightest disturbance prevents a high rate from being
Center TCP. But it is also more general--it will allow a set of attained [RFC3649].
congestion controls with similar scaling properties to DCTCP (e.g.
Relentless [Mathis09]) to evolve.
Both Classic and L4S services can cope with a proportion of For instance, with 1500 byte packets and an end-to-end round trip
unresponsive or less-responsive traffic as well (e.g. DNS, VoIP, time (RTT) of 36 ms, over the years, as Reno flow rate scales from
etc). 2 to 100 Mb/s the number of round trips taken to recover from a
congestion event rises proportionately, from 4 to 200. Cubic was
developed to be less unscalable, but it is approaching its scaling
limit; with the same RTT of 36ms, at 100Mb/s it takes over 300
round trips to recover, and at 800 Mb/s its recovery time doubles
to over 600 round trips, or more than 20 seconds.
Scalable Congestion Control: A congestion control where the packet Scalable Congestion Control: A congestion control where the average
flow rate per round trip (the window) is inversely proportional to time from one congestion signal to the next (the recovery time)
the level (probability) of congestion signals. Then, as flow rate remains invariant as the flow rate scales, all other factors being
scales, the number of congestion signals per round trip remains equal. This maintains the same degree of control over queueing
invariant, maintaining the same degree of control. For instance, and utilization whatever the flow rate, as well as ensuring that
DCTCP averages 2 congestion signals per round-trip whatever the high throughput is robust to disturbances. For instance, DCTCP
flow rate. averages 2 congestion signals per round-trip whatever the flow
rate. See Section 4.3 of [I-D.ietf-tsvwg-ecn-l4s-id] for more
explanation.
Classic Congestion Control: A congestion control with a flow rate Classic service: The Classic service is intended for all the
that can co-exist with standard TCP Reno [RFC5681] without congestion control behaviours that co-exist with Reno [RFC5681]
starvation. With Classic congestion controls, as capacity (e.g. Reno itself, Cubic [RFC8312], Compound
increases enabling higher flow rates, the number of round trips [I-D.sridharan-tcpm-ctcp], TFRC [RFC5348]).
between congestion signals (losses or ECN marks) rises in
proportion to the flow rate. So control of queuing and/or
utilization becomes very slack. For instance, with 1500 B packets
and an RTT of 18 ms, as TCP Reno flow rate increases from 2 to 100
Mb/s the number of round trips between congestion signals rises
proportionately, from 2 to 100.
The default congestion control in Linux (TCP Cubic) is Reno- Low-Latency, Low-Loss Scalable throughput (L4S) service: The 'L4S'
compatible for most Internet access scenarios expected for some service is intended for traffic from scalable congestion control
years. For instance, with a typical domestic round-trip time algorithms, such as Data Center TCP [RFC8257]. The L4S service is
(RTT) of 18ms, TCP Cubic only switches out of Reno-compatibility for more general traffic than just DCTCP--it allows the set of
mode once the flow rate approaches 1 Gb/s. For a typical data congestion controls with similar scaling properties to DCTCP to
centre RTT of 1 ms, the switch-over point is theoretically 1.3 Tb/ evolve (e.g. Relentless TCP [Mathis09], TCP Prague [LinuxPrague]
s. However, with a less common transcontinental RTT of 100 ms, it and the L4S variant of SCREAM for real-time media [RFC8298]).
only remains Reno-compatible up to 13 Mb/s. All examples assume
1,500 B packets.
Classic ECN: The original proposed standard Explicit Congestion Both Classic and L4S services can cope with a proportion of
Notification (ECN) protocol [RFC3168], which requires ECN signals unresponsive or less-responsive traffic as well, as long as it
to be treated the same as drops, both when generated in the does not build a queue (e.g. DNS, VoIP, game sync datagrams,
network and when responded to by the sender. etc).
The terms Classic or L4S can also qualify other nouns, such
'queue', 'codepoint', 'identifier', 'classification', 'packet',
'flow'. For example: a 'Classic queue', means a queue providing
the Classic service; an L4S packet means a packet with an L4S
identifier sent from an L4S congestion control.
Classic ECN: The original Explicit Congestion Notification (ECN)
protocol [RFC3168], which requires ECN signals to be treated the
same as drops, both when generated in the network and when
responded to by the sender. The names used for the four
codepoints of the 2-bit IP-ECN field are as defined in [RFC3168]:
Not ECT, ECT(0), ECT(1) and CE, where ECT stands for ECN-Capable
Transport and CE stands for Congestion Experienced.
Site: A home, mobile device, small enterprise or campus, where the Site: A home, mobile device, small enterprise or campus, where the
network bottleneck is typically the access link to the site. Not network bottleneck is typically the access link to the site. Not
all network arrangements fit this model but it is a useful, widely all network arrangements fit this model but it is a useful, widely
applicable generalisation. applicable generalisation.
4. L4S Architecture Components 4. L4S Architecture Components
The L4S architecture is composed of the following elements. The L4S architecture is composed of the following elements.
skipping to change at page 8, line 46 skipping to change at page 9, line 27
instance, VoIP, low rate datagrams to sync online games, instance, VoIP, low rate datagrams to sync online games,
relatively low rate application-limited traffic, DNS, LDAP, etc. relatively low rate application-limited traffic, DNS, LDAP, etc.
This traffic would need to be tagged with specific identifiers, This traffic would need to be tagged with specific identifiers,
e.g. a low latency Diffserv Codepoint such as Expedited e.g. a low latency Diffserv Codepoint such as Expedited
Forwarding (EF [RFC3246]), Non-Queue-Building (NQB Forwarding (EF [RFC3246]), Non-Queue-Building (NQB
[I-D.white-tsvwg-nqb]), or operator-specific identifiers. [I-D.white-tsvwg-nqb]), or operator-specific identifiers.
Network components: The L4S architecture encompasses either dual- Network components: The L4S architecture encompasses either dual-
queue or per-flow queue solutions: queue or per-flow queue solutions:
a. The Dual Queue Coupled AQM has been specified as generically as a. The Coupled Dual Queue AQM achieves the 'semi-permeable' membrane
possible [I-D.ietf-tsvwg-aqm-dualq-coupled] as a 'semi-permeable' property mentioned earlier as follows. The obvious part is that
membrane without specifying the particular AQMs to use in the two using two separate queues isolates the queuing delay of one from
queues. Informational appendices of the draft are provided for the other. The less obvious part is how the two queues act as if
pseudocode examples of different possible AQM approaches. The they are a single pool of bandwidth without the scheduler needing
aim is for designers to be free to implement diverse ideas. So to decide between them. This is achieved by making the Classic
the brief normative body of the draft only specifies the minimum traffic appear as if it is an equivalent amount of traffic in the
constraints an AQM needs to comply with to ensure that the L4S L4S queue, by coupling across the drop probability of the Classic
and Classic services will coexist. The core idea is the tension AQM to drive the ECN marking level applied to L4S traffic. This
between the scheduler's prioritization of L4S over Classic and makes the L4S flows slow down to leave just enough capacity for
the coupling from the Classic to the L4S AQM. The L4S AQM the Classic traffic (as they would if they were the same type of
derives its level of ECN marking from the maximum of the traffic sharing the same queue). Then the scheduler can serve
congestion levels in both queues. So L4S flows leave enough the L4S queue with priority, because the L4S traffic isn't
space between their packets for Classic flows, as if they were offering up enough traffic to use all the priority that it is
all the same type of TCP, all sharing one FIFO queue. given. Therefore, on short time-scales (sub-round-trip) the
prioritization of the L4S queue protects its low latency by
allowing bursts to dissipate quickly; but on longer time-scales
(round-trip and longer) the Classic queue creates an equal and
opposite pressure against the L4S traffic to ensure that neither
has priority when it comes to bandwidth. The tension between
prioritizing L4S and coupling marking from Classic results in
per-flow fairness. To protect against unresponsive traffic in
the L4S queue taking advantage of the prioritization and starving
the Classic queue, it is advisable not to use strict priority,
but instead to use a weighted scheduler.
Initially a zero-config variant of RED called Curvy RED was When there is no Classic traffic, the L4S queue's AQM comes into
implemented, tested and documented. Then, a variant of PIE play, and it sets an appropriate marking rate to maintain ultra-
called DualPI2 (pronounced Dual PI Squared) [PI2] was implemented low queuing delay.
and found to perform better than Curvy RED over a wide range of
conditions, so it was documented in another appendix of
[I-D.ietf-tsvwg-aqm-dualq-coupled].
b. A scheduler with per-flow queues can be used for L4S. It would The Coupled Dual Queue AQM has been specified as generically as
be simple to modify an existing design such as FQ-CoDel or FQ- possible [I-D.ietf-tsvwg-aqm-dualq-coupled] without specifying
PIE, although this has not been implemented and evaluated because the particular AQMs to use in the two queues so that designers
the goal of the original proponents of L4S was to avoid per-flow are free to implement diverse ideas. Then informational
scheduling. appendices give pseudocode examples of different specific AQM
approaches. Initially a zero-config variant of RED called Curvy
RED was implemented, tested and documented. Then, a variant of
PIE called DualPI2 (pronounced Dual PI Squared) [DualPI2Linux]
was implemented and found to perform better than Curvy RED over a
wide range of conditions, so it was documented in another
appendix of [I-D.ietf-tsvwg-aqm-dualq-coupled]. A Coupled DualQ
variant based on PIE has also been specified and implemented for
Low Latency DOCSIS [DOCSIS3.1].
The idea would be to implement two AQMs (Classic and Scalable) b. A scheduler with per-flow queues can be used for L4S. It is
and switch each per-flow queue to use an instance of the simple to modify an existing design such as FQ-CoDel or FQ-PIE.
appropriate AQM for the flow, based on the ECN codepoints of the For instance within each queue of an FQ_CoDel system, as well as
packets. Flows of non-ECN or ECT(0) packets would use a Classic a CoDel AQM, immediate (unsmoothed) shallow threshold ECN marking
AQM such as CoDel or PIE, while flows of ECT(1) packets without has been added. Then the Classic AQM such as CoDel or PIE is
any ECT(0) packets would use a simple shallow threshold AQM with applied to non-ECN or ECT(0) packets, while the shallow threshold
immediate (unsmoothed) marking. The FQ scheduler might work as is applied to ECT(1) packets, to give sub-millisecond average
is, because it is likely that L4S flows would be continually queue delay.
categorized as 'new' flows. However, this presumption has not
been tested under a wide range of conditions. A variant of FQ-
CoDel already exists that adapts to a shallower threshold AQM for
ECN-capable packets.
Host mechanisms: The L4S architecture includes a number of mechanisms Host mechanisms: The L4S architecture includes a number of mechanisms
in the end host that we enumerate next: in the end host that we enumerate next:
a. Data Center TCP is the most widely used example of a scalable a. Data Center TCP is the most widely used example of a scalable
congestion control. It has been documented as an informational congestion control. It has been documented as an informational
record of the protocol currently in use [RFC8257]. It will be record of the protocol currently in use [RFC8257]. It will be
necessary to define a number of safety features for a variant necessary to define a number of safety features for a variant
usable on the public Internet. A draft list of these, known as usable on the public Internet. A draft list of these, known as
the TCP Prague requirements, has been drawn up (see Appendix A of the Prague L4S requirements, has been drawn up (see Appendix A of
[I-D.ietf-tsvwg-ecn-l4s-id]). The list also includes some [I-D.ietf-tsvwg-ecn-l4s-id]). The list also includes some
optional performance improvements. optional performance improvements.
b. Transport protocols other than TCP use various congestion b. Transport protocols other than TCP use various congestion
controls designed to be friendly with Classic TCP. Before they controls designed to be friendly with Reno. Before they can use
can use the L4S service, it will be necessary to implement the L4S service, it will be necessary to implement scalable
scalable variants of each of these congestion control behaviours. variants of each of these congestion control behaviours. The
The following standards track RFCs currently define these following standards track RFCs currently define these protocols:
protocols: ECN in TCP [RFC3168], in SCTP [RFC4960], in RTP ECN in TCP [RFC3168], in SCTP [RFC4960], in RTP [RFC6679], and in
[RFC6679], and in DCCP [RFC4340]. Not all are in widespread use, DCCP [RFC4340]. Not all are in widespread use, but those that
but those that are will eventually need to be updated to allow a are will eventually need to be updated to allow a different
different congestion response, which they will have to indicate congestion response, which they will have to indicate by using
by using the ECT(1) codepoint. Scalable variants are under the ECT(1) codepoint. Scalable variants are under consideration
consideration for some new transport protocols that are for some new transport protocols that are themselves under
themselves under development, e.g. QUIC development, e.g. QUIC [I-D.ietf-quic-transport] and certain
[I-D.ietf-quic-transport] and certain real-time media congestion real-time media congestion avoidance techniques (RMCAT)
avoidance techniques (RMCAT) protocols. protocols.
c. ECN feedback is sufficient for L4S in some transport protocols c. ECN feedback is sufficient for L4S in some transport protocols
(RTCP, DCCP) but not others: (RTCP, DCCP) but not others:
* For the case of TCP, the feedback protocol for ECN embeds the * For the case of TCP, the feedback protocol for ECN embeds the
assumption from Classic ECN that an ECN mark is the same as a assumption from Classic ECN that an ECN mark is the same as a
drop, making it unusable for a scalable TCP. Therefore, the drop, making it unusable for a scalable TCP. Therefore, the
implementation of TCP receivers will have to be upgraded implementation of TCP receivers will have to be upgraded
[RFC7560]. Work to standardize and implement more accurate [RFC7560]. Work to standardize and implement more accurate
ECN feedback for TCP (AccECN) is in progress ECN feedback for TCP (AccECN) is in progress
skipping to change at page 10, line 51 skipping to change at page 11, line 41
signalling is a key part of the L4S approach. In contrast, use of signalling is a key part of the L4S approach. In contrast, use of
drop as a congestion signal creates a tension because drop is both drop as a congestion signal creates a tension because drop is both
a useful signal (more would reduce delay) and an impairment (less a useful signal (more would reduce delay) and an impairment (less
would reduce delay): would reduce delay):
* Explicit congestion signals can be used many times per round * Explicit congestion signals can be used many times per round
trip, to keep tight control, without any impairment. Under trip, to keep tight control, without any impairment. Under
heavy load, even more explicit signals can be applied so the heavy load, even more explicit signals can be applied so the
queue can be kept short whatever the load. Whereas state-of- queue can be kept short whatever the load. Whereas state-of-
the-art AQMs have to introduce very high packet drop at high the-art AQMs have to introduce very high packet drop at high
load to keep the queue short. Further, when using ECN, TCP's load to keep the queue short. Further, when using ECN, the
sawtooth reduction can be smaller and therefore return to the congestion control's sawtooth reduction can be smaller and
operating point more often, without worrying that this causes therefore return to the operating point more often, without
more signals (one at the top of each smaller sawtooth). The worrying that this causes more signals (one at the top of each
consequent smaller amplitude sawteeth fit between a very smaller sawtooth). The consequent smaller amplitude sawteeth
shallow marking threshold and an empty queue, so delay fit between a very shallow marking threshold and an empty
variation can be very low, without risk of under-utilization. queue, so delay variation can be very low, without risk of
under-utilization.
* Explicit congestion signals can be sent immediately to track * Explicit congestion signals can be sent immediately to track
fluctuations of the queue. L4S shifts smoothing from the fluctuations of the queue. L4S shifts smoothing from the
network (which doesn't know the round trip times of all the network (which doesn't know the round trip times of all the
flows) to the host (which knows its own round trip time). flows) to the host (which knows its own round trip time).
Previously, the network had to smooth to keep a worst-case Previously, the network had to smooth to keep a worst-case
round trip stable, delaying congestion signals by 100-200ms. round trip stable, delaying congestion signals by 100-200ms.
All the above makes it clear that explicit congestion signalling All the above makes it clear that explicit congestion signalling
is only advantageous for latency if it does not have to be is only advantageous for latency if it does not have to be
skipping to change at page 12, line 24 skipping to change at page 13, line 14
Low loss: Latency is not the only concern of L4S. The 'Low Loss" Low loss: Latency is not the only concern of L4S. The 'Low Loss"
part of the name denotes that L4S generally achieves zero part of the name denotes that L4S generally achieves zero
congestion loss due to its use of ECN. Otherwise, loss would congestion loss due to its use of ECN. Otherwise, loss would
itself cause delay, particularly for short flows, due to itself cause delay, particularly for short flows, due to
retransmission delay [RFC2884]. retransmission delay [RFC2884].
Scalable throughput: The "Scalable throughput" part of the name Scalable throughput: The "Scalable throughput" part of the name
denotes that the per-flow throughput of scalable congestion denotes that the per-flow throughput of scalable congestion
controls should scale indefinitely, avoiding the imminent scaling controls should scale indefinitely, avoiding the imminent scaling
problems with TCP-Friendly congestion control algorithms problems with Reno-Friendly congestion control algorithms
[RFC3649]. It was known when TCP was first developed that it [RFC3649]. It was known when TCP congestion avoidance was first
would not scale to high bandwidth-delay products (see footnote 6 developed that it would not scale to high bandwidth-delay products
in [TCP-CA]). Today, regular broadband bit-rates over WAN (see footnote 6 in [TCP-CA]). Today, regular broadband bit-rates
distances are already beyond the scaling range of `classic' TCP over WAN distances are already beyond the scaling range of Classic
Reno. So `less unscalable' Cubic [RFC8312] and Reno congestion control. So `less unscalable' Cubic [RFC8312] and
Compound [I-D.sridharan-tcpm-ctcp] variants of TCP have been Compound [I-D.sridharan-tcpm-ctcp] variants of TCP have been
successfully deployed. However, these are now approaching their successfully deployed. However, these are now approaching their
scaling limits. For instance, at 800Mb/s with a 20ms round trip, scaling limits. For instance, at 800Mb/s with a 20ms round trip,
Cubic induces a congestion signal only every 500 round trips or 10 Cubic induces a congestion signal only every 500 round trips or 10
seconds, which makes its dynamic control very sloppy. In contrast seconds, which makes its dynamic control very sloppy. In contrast
on average a scalable congestion control like DCTCP or TCP Prague on average a scalable congestion control like DCTCP or TCP Prague
induces 2 congestion signals per round trip, which remains induces 2 congestion signals per round trip, which remains
invariant for any flow rate, keeping dynamic control very tight. invariant for any flow rate, keeping dynamic control very tight.
5.2. Why Not Alternative Approaches? 5.2. Why Not Alternative Approaches?
skipping to change at page 13, line 19 skipping to change at page 14, line 9
of the traffic on a link. It is not applicable when all the of the traffic on a link. It is not applicable when all the
applications in use at one time at a single site (home, small applications in use at one time at a single site (home, small
business or mobile device) require low latency. Also, because L4S business or mobile device) require low latency. Also, because L4S
is for all traffic, it needs none of the management baggage is for all traffic, it needs none of the management baggage
(traffic policing, traffic contracts) associated with favouring (traffic policing, traffic contracts) associated with favouring
some packets over others. This baggage has held Diffserv back some packets over others. This baggage has held Diffserv back
from widespread end-to-end deployment. from widespread end-to-end deployment.
State-of-the-art AQMs: AQMs such as PIE and fq_CoDel give a State-of-the-art AQMs: AQMs such as PIE and fq_CoDel give a
significant reduction in queuing delay relative to no AQM at all. significant reduction in queuing delay relative to no AQM at all.
The L4S work is intended to complement these AQMs, and we L4S is intended to complement these AQMs, and should not distract
definitely do not want to distract from the need to deploy them as from the need to deploy them as widely as possible. Nonetheless,
widely as possible. Nonetheless, without addressing the large without addressing the large saw-toothing rate variations of
saw-toothing rate variations of Classic congestion controls, AQMs Classic congestion controls, AQMs alone cannot reduce queuing
alone cannot reduce queuing delay too far without significantly delay too far without significantly reducing link utilization.
reducing link utilization. The L4S approach resolves this tension The L4S approach resolves this tension by ensuring hosts can
by ensuring hosts can minimize the size of their sawteeth without minimize the size of their sawteeth without appearing so
appearing so aggressive to legacy flows that they starve them. aggressive to legacy flows that they starve them.
Per-flow queuing: Similarly per-flow queuing is not incompatible Per-flow queuing: Similarly, per-flow queuing is not incompatible
with the L4S approach. However, one queue for every flow can be with the L4S approach. However, one queue for every flow can be
thought of as overkill compared to the minimum of two queues for thought of as overkill compared to the minimum of two queues for
all traffic needed for the L4S approach. The overkill of per-flow all traffic needed for the L4S approach. The overkill of per-flow
queuing has side-effects: queuing has side-effects:
A. fq makes high performance networking equipment costly A. fq makes high performance networking equipment costly
(processing and memory) - in contrast dual queue code can be (processing and memory) - in contrast dual queue code can be
very simple; very simple;
B. fq requires packet inspection into the end-to-end transport B. fq requires packet inspection into the end-to-end transport
layer, which doesn't sit well alongside encryption for privacy layer, which doesn't sit well alongside encryption for privacy
- in contrast the use of ECN as the classifier for L4S - in contrast the use of ECN as the classifier for L4S
requires no deeper inspection than the IP layer; requires no deeper inspection than the IP layer;
C. fq isolates the queuing of each flow from the others but not C. fq isolates the queuing of each flow from the others but not
from itself so existing FQ implementations still needs to have from itself so existing FQ implementations still need to have
support for scalable congestion control added. support for scalable congestion control added.
It might seem that self-inflicted queuing delay should not It might seem that self-inflicted queuing delay should not
count, because if the delay wasn't in the network it would count, because if the delay wasn't in the network it would
just shift to the sender. However, modern adaptive just shift to the sender. However, modern adaptive
applications, e.g. HTTP/2 [RFC7540] or the interactive media applications, e.g. HTTP/2 [RFC7540] or the interactive media
applications described in Section 6, can keep low latency applications described in Section 6, can keep low latency
objects at the front of their local send queue by shuffling objects at the front of their local send queue by shuffling
priorities of other objects dependent on the progress of other priorities of other objects dependent on the progress of other
transfers. They cannot shuffle packets once they have transfers. They cannot shuffle packets once they have
skipping to change at page 14, line 30 skipping to change at page 15, line 21
than requiring one specific policy choice as the mechanism than requiring one specific policy choice as the mechanism
itself. A scheduler (like fq) has to decide packet-by-packet itself. A scheduler (like fq) has to decide packet-by-packet
which flow to schedule without knowing application intent. which flow to schedule without knowing application intent.
Whereas a separate policing function can be configured less Whereas a separate policing function can be configured less
strictly, so that senders can still control the instantaneous strictly, so that senders can still control the instantaneous
rate of each flow dependent on the needs of each application rate of each flow dependent on the needs of each application
(e.g. variable rate video), giving more wriggle-room before a (e.g. variable rate video), giving more wriggle-room before a
flow is deemed non-compliant. Also policing of queuing and of flow is deemed non-compliant. Also policing of queuing and of
flow-rates can be applied independently. flow-rates can be applied independently.
Alternative Back-off ECN (ABE): Yet again, L4S is not an alternative Alternative Back-off ECN (ABE): Here again, L4S is not an
to ABE but a complement that introduces much lower queuing delay. alternative to ABE but a complement that introduces much lower
ABE [RFC8511] alters the host behaviour in response to ECN marking queuing delay. ABE [RFC8511] alters the host behaviour in
to utilize a link better and give ECN flows faster throughput. It response to ECN marking to utilize a link better and give ECN
uses ECT(0) and assumes the network still treats ECN and drop the flows faster throughput. It uses ECT(0) and assumes the network
same. Therefore ABE exploits any lower queuing delay that AQMs still treats ECN and drop the same. Therefore ABE exploits any
can provide. But as explained above, AQMs still cannot reduce lower queuing delay that AQMs can provide. But as explained
queuing delay too far without losing link utilization (to allow above, AQMs still cannot reduce queuing delay too far without
for other, non-ABE, flows). losing link utilization (to allow for other, non-ABE, flows).
BBRv1: v1 of Bottleneck Bandwidth and Round-trip propagation time BBRv1: v1 of Bottleneck Bandwidth and Round-trip propagation time
(BBR [I-D.cardwell-iccrg-bbr-congestion-control]) controls queuing (BBR [I-D.cardwell-iccrg-bbr-congestion-control]) controls queuing
delay end-to-end without needing any special logic in the network, delay end-to-end without needing any special logic in the network,
such as an AQM - so it works pretty-much on any path. Setting such as an AQM - so it works pretty-much on any path. Setting
some problems with capacity sharing aside, queuing delay is good some problems with capacity sharing aside, queuing delay is good
with BBRv1, but perhaps not quite as low as with state-of-the-art with BBRv1, but perhaps not quite as low as with state-of-the-art
AQMs such as PIE or fq_CoDel, and certainly nowhere near as low as AQMs such as PIE or fq_CoDel, and certainly nowhere near as low as
with L4S. Queuing delay is also not consistently low, due to its with L4S. Queuing delay is also not consistently low, due to its
regular bandwidth probes and the aggressive flow start-up phase. regular bandwidth probes and the aggressive flow start-up phase.
L4S is a complement to BBRv1. Indeed BBRv2 (not released at the L4S is a complement to BBRv1. Indeed BBRv2 uses L4S ECN and a
time of writing) is likely to use L4S ECN and a TCP-Prague-like scalable L4S congestion control behaviour in response to any ECN
behaviour if it discovers a compatible path. Otherwise it will signalling from the path.
use an evolution of BBRv1.
6. Applicability 6. Applicability
6.1. Applications 6.1. Applications
A transport layer that solves the current latency issues will provide A transport layer that solves the current latency issues will provide
new service, product and application opportunities. new service, product and application opportunities.
With the L4S approach, the following existing applications will With the L4S approach, the following existing applications will
immediately experience significantly better quality of experience experience significantly better quality of experience under load:
under load:
o Gaming; o Gaming, including cloud based gaming;
o VoIP; o VoIP;
o Video conferencing; o Video conferencing;
o Web browsing; o Web browsing;
o (Adaptive) video streaming; o (Adaptive) video streaming;
o Instant messaging. o Instant messaging.
skipping to change at page 17, line 6 skipping to change at page 17, line 42
capacity; capacity;
* cellular networks are further complicated by a perceived need * cellular networks are further complicated by a perceived need
to buffer in order to make hand-overs imperceptible; to buffer in order to make hand-overs imperceptible;
* Satellite networks generally have a very large base RTT, so * Satellite networks generally have a very large base RTT, so
even with minimal queuing, overall delay can never be extremely even with minimal queuing, overall delay can never be extremely
low; low;
* Nonetheless, it is certainly desirable not to hold a buffer * Nonetheless, it is certainly desirable not to hold a buffer
purely because of the sawteeth of Classic TCP, when it is more purely because of the sawteeth of Classic congestion controls,
than is needed for all the above reasons. when it is more than is needed for all the above reasons.
o Private networks of heterogeneous data centres, where there is no o Private networks of heterogeneous data centres, where there is no
single administrator that can arrange for all the simultaneous single administrator that can arrange for all the simultaneous
changes to senders, receivers and network needed to deploy DCTCP: changes to senders, receivers and network needed to deploy DCTCP:
* a set of private data centres interconnected over a wide area * a set of private data centres interconnected over a wide area
with separate administrations, but within the same company with separate administrations, but within the same company
* a set of data centres operated by separate companies * a set of data centres operated by separate companies
interconnected by a community of interest network (e.g. for the interconnected by a community of interest network (e.g. for the
skipping to change at page 17, line 49 skipping to change at page 18, line 38
traffic to favour for queuing, when L4S gives favourable traffic to favour for queuing, when L4S gives favourable
queuing to all traffic. queuing to all traffic.
o If queuing delay is minimized, applications with a fixed delay o If queuing delay is minimized, applications with a fixed delay
budget can communicate over longer distances, or via a longer budget can communicate over longer distances, or via a longer
chain of service functions [RFC7665] or onion routers. chain of service functions [RFC7665] or onion routers.
6.3. Deployment Considerations 6.3. Deployment Considerations
The DualQ is, in itself, an incremental deployment framework for L4S The DualQ is, in itself, an incremental deployment framework for L4S
AQMs so that L4S traffic can coexist with existing Classic "TCP- AQMs so that L4S traffic can coexist with existing Classic (Reno-
friendly" traffic. Section 6.3.1 explains why only deploying a DualQ friendly) traffic. Section 6.3.1 explains why only deploying a DualQ
AQM [I-D.ietf-tsvwg-aqm-dualq-coupled] in one node at each end of the AQM [I-D.ietf-tsvwg-aqm-dualq-coupled] in one node at each end of the
access link will realize nearly all the benefit of L4S. access link will realize nearly all the benefit of L4S.
L4S involves both end systems and the network, so Section 6.3.2 L4S involves both end systems and the network, so Section 6.3.2
suggests some typical sequences to deploy each part, and why there suggests some typical sequences to deploy each part, and why there
will be an immediate and significant benefit after deploying just one will be an immediate and significant benefit after deploying just one
part. part.
If an ECN-enabled DualQ AQM has not been deployed at a bottleneck, an If an ECN-enabled DualQ AQM has not been deployed at a bottleneck, an
L4S flow is required to include a fall-back strategy to Classic L4S flow is required to include a fall-back strategy to Classic
skipping to change at page 18, line 23 skipping to change at page 19, line 13
how to minimize the effect of false negative detection. how to minimize the effect of false negative detection.
6.3.1. Deployment Topology 6.3.1. Deployment Topology
DualQ AQMs will not have to be deployed throughout the Internet DualQ AQMs will not have to be deployed throughout the Internet
before L4S will work for anyone. Operators of public Internet access before L4S will work for anyone. Operators of public Internet access
networks typically design their networks so that the bottleneck will networks typically design their networks so that the bottleneck will
nearly always occur at one known (logical) link. This confines the nearly always occur at one known (logical) link. This confines the
cost of queue management technology to one place. cost of queue management technology to one place.
The case of mesh networks is different and will be discussed later. The case of mesh networks is different and will be discussed later in
But the known bottleneck case is generally true for Internet access this section. But the known bottleneck case is generally true for
to all sorts of different 'sites', where the word 'site' includes Internet access to all sorts of different 'sites', where the word
home networks, small-to-medium sized campus or enterprise networks 'site' includes home networks, small-to-medium sized campus or
and even cellular devices (Figure 2). Also, this known-bottleneck enterprise networks and even cellular devices (Figure 2). Also, this
case tends to be applicable whatever the access link technology; known-bottleneck case tends to be applicable whatever the access link
whether xDSL, cable, cellular, line-of-sight wireless or satellite. technology; whether xDSL, cable, cellular, line-of-sight wireless or
satellite.
Therefore, the full benefit of the L4S service should be available in Therefore, the full benefit of the L4S service should be available in
the downstream direction when the DualQ AQM is deployed at the the downstream direction when the DualQ AQM is deployed at the
ingress to this bottleneck link (or links for multihomed sites). And ingress to this bottleneck link (or links for multihomed sites). And
similarly, the full upstream service will be available once the DualQ similarly, the full upstream service will be available once the DualQ
is deployed at the upstream ingress. is deployed at the upstream ingress.
______ ______
( ) ( )
__ __ ( ) __ __ ( )
skipping to change at page 21, line 12 skipping to change at page 21, line 33
Figure 3 illustrates some example sequences in which the parts of L4S Figure 3 illustrates some example sequences in which the parts of L4S
might be deployed. It consists of the following stages: might be deployed. It consists of the following stages:
1. Here, the immediate benefit of a single AQM deployment can be 1. Here, the immediate benefit of a single AQM deployment can be
seen, but limited to a controlled trial or controlled deployment. seen, but limited to a controlled trial or controlled deployment.
In this example downstream deployment is first, but in other In this example downstream deployment is first, but in other
scenarios the upstream might be deployed first. If no AQM at all scenarios the upstream might be deployed first. If no AQM at all
was previously deployed for the downstream access, the DualQ AQM was previously deployed for the downstream access, the DualQ AQM
greatly improves the Classic service (as well as adding the L4S greatly improves the Classic service (as well as adding the L4S
service). If an AQM was already deployed, the Classic service service). If an AQM was already deployed, the Classic service
will be unchanged (and L4S will still be added). will be unchanged (and L4S will add an improvement on top).
2. In this stage, the name 'TCP Prague' is used to represent a 2. In this stage, the name 'TCP Prague' is used to represent a
variant of DCTCP that is safe to use in a production environment. variant of DCTCP that is safe to use in a production environment.
If the application is primarily unidirectional, 'TCP Prague' at If the application is primarily unidirectional, 'TCP Prague' at
one end will provide all the benefit needed. Accurate ECN one end will provide all the benefit needed. Accurate ECN
feedback (AccECN) [I-D.ietf-tcpm-accurate-ecn] is needed at the feedback (AccECN) [I-D.ietf-tcpm-accurate-ecn] is needed at the
other end, but it is a generic ECN feedback facility that is other end, but it is a generic ECN feedback facility that is
already planned to be deployed for other purposes, e.g. DCTCP, already planned to be deployed for other purposes, e.g. DCTCP,
BBR [I-D.cardwell-iccrg-bbr-congestion-control]. The two ends BBR [I-D.cardwell-iccrg-bbr-congestion-control]. The two ends
can be deployed in either order, because TCP Prague only enables can be deployed in either order, because, in TCP, an L4S
itself if it has negotiated the use of AccECN feedback with the congestion control only enables itself if it has negotiated the
other end during the connection handshake. Thus, deployment of use of AccECN feedback with the other end during the connection
TCP Prague on a server enables L4S trials to move to a production handshake. Thus, deployment of TCP Prague on a server enables
service in one direction, wherever AccECN is deployed at the L4S trials to move to a production service in one direction,
other end. This stage might be further motivated by the wherever AccECN is deployed at the other end. This stage might
performance improvements of TCP Prague relative to DCTCP (see be further motivated by the performance improvements of TCP
Appendix A.2 of [I-D.ietf-tsvwg-ecn-l4s-id]). Prague relative to DCTCP (see Appendix A.2 of
[I-D.ietf-tsvwg-ecn-l4s-id]).
3. This is a two-move stage to enable L4S upstream. The DualQ or 3. This is a two-move stage to enable L4S upstream. The DualQ or
TCP Prague can be deployed in either order as already explained. TCP Prague can be deployed in either order as already explained.
To motivate the first of two independent moves, the deferred To motivate the first of two independent moves, the deferred
benefit of enabling new services after the second move has to be benefit of enabling new services after the second move has to be
worth it to cover the first mover's investment risk. As worth it to cover the first mover's investment risk. As
explained already, the potential for new interactive services explained already, the potential for new interactive services
provides this motivation. The DualQ AQM also greatly improves provides this motivation. The DualQ AQM also greatly improves
the upstream Classic service, assuming no other AQM has already the upstream Classic service, assuming no other AQM has already
been deployed. been deployed.
skipping to change at page 21, line 51 skipping to change at page 22, line 25
Note that other deployment sequences might occur. For instance: the Note that other deployment sequences might occur. For instance: the
upstream might be deployed first; a non-TCP protocol might be used upstream might be deployed first; a non-TCP protocol might be used
end-to-end, e.g. QUIC, RMCAT; a body such as the 3GPP might require end-to-end, e.g. QUIC, RMCAT; a body such as the 3GPP might require
L4S to be implemented in 5G user equipment, or other random acts of L4S to be implemented in 5G user equipment, or other random acts of
kindness. kindness.
6.3.3. L4S Flow but Non-L4S Bottleneck 6.3.3. L4S Flow but Non-L4S Bottleneck
If L4S is enabled between two hosts but there is no L4S AQM at the If L4S is enabled between two hosts but there is no L4S AQM at the
bottleneck, any drop from the bottleneck will trigger the L4S sender bottleneck, any drop from the bottleneck will trigger the L4S sender
to fall back to a classic ('TCP-Friendly') behaviour (see to fall back to a classic ('Reno-Friendly') behaviour (see
Appendix A.1.3 of [I-D.ietf-tsvwg-ecn-l4s-id]). Appendix A.1.3 of [I-D.ietf-tsvwg-ecn-l4s-id]).
Unfortunately, as well as protecting legacy traffic, this rule Unfortunately, as well as protecting legacy traffic, this rule
degrades the L4S service whenever there is a loss, even if the loss degrades the L4S service whenever there is a loss, even if the loss
was not from a non-DualQ bottleneck (false negative). And was not from a non-DualQ bottleneck (false negative). And
unfortunately, prevalent drop can be due to other causes, e.g.: unfortunately, prevalent drop can be due to other causes, e.g.:
o congestion loss at other transient bottlenecks, e.g. due to bursts o congestion loss at other transient bottlenecks, e.g. due to bursts
in shallower queues; in shallower queues;
o transmission errors, e.g. due to electrical interference; o transmission errors, e.g. due to electrical interference;
o rate policing. o rate policing.
Three complementary approaches are in progress to address this issue, Three complementary approaches are in progress to address this issue,
but they are all currently research: but they are all currently research:
o In TCP Prague, ignore certain losses deemed unlikely to be due to o In Prague congestion control, ignore certain losses deemed
congestion (using some ideas from BBR unlikely to be due to congestion (using some ideas from BBR
[I-D.cardwell-iccrg-bbr-congestion-control] but with no need to [I-D.cardwell-iccrg-bbr-congestion-control] but with no need to
ignore nearly all losses). This could mask any of the above types ignore nearly all losses). This could mask any of the above types
of loss (requires consensus on how to safely interoperate with of loss (requires consensus on how to safely interoperate with
drop-based congestion controls). drop-based congestion controls).
o A combination of RACK, reconfigured link retransmission and L4S o A combination of RACK, reconfigured link retransmission and L4S
could address transmission errors [UnorderedLTE], could address transmission errors [UnorderedLTE],
[I-D.ietf-tsvwg-ecn-l4s-id]; [I-D.ietf-tsvwg-ecn-l4s-id];
o Hybrid ECN/drop policers (see Section 8.3). o Hybrid ECN/drop policers (see Section 8.3).
L4S deployment scenarios that minimize these issues (e.g. over L4S deployment scenarios that minimize these issues (e.g. over
wireline networks) can proceed in parallel to this research, in the wireline networks) can proceed in parallel to this research, in the
expectation that research success could continually widen L4S expectation that research success could continually widen L4S
applicability. applicability.
Classic ECN support is starting to materialize on the Internet as an Classic ECN support is starting to materialize on the Internet as an
increased level of CE marking. Given some of this Classic ECN might increased level of CE marking. Given some of this Classic ECN might
be due to single-queue ECN deployment, an L4S sender will have to be due to single-queue ECN deployment, an L4S sender will have to
fall back to a classic ('TCP-Friendly') behaviour if it detects that fall back to a classic ('Reno-Friendly') behaviour if it detects that
ECN marking is accompanied by greater queuing delay or greater delay ECN marking is accompanied by greater queuing delay or greater delay
variation than would be expected with L4S (see Appendix A.1.4 of variation than would be expected with L4S (see Appendix A.1.4 of
[I-D.ietf-tsvwg-ecn-l4s-id]). It is hard to detect whether this is [I-D.ietf-tsvwg-ecn-l4s-id]). It is hard to detect whether this is
all due to the addition of support for ECN in the Linux all due to the addition of support for ECN in the Linux
implementation of FQ-CoDel, which would not require fall-back to implementation of FQ-CoDel, which would not require fall-back to
Classic behaviour, because FQ inherently forces the throughput of Classic behaviour, because FQ inherently forces the throughput of
each flow to be equal irrespective of its aggressiveness. each flow to be equal irrespective of its aggressiveness.
6.3.4. Other Potential Deployment Issues 6.3.4. Other Potential Deployment Issues
skipping to change at page 24, line 15 skipping to change at page 24, line 34
be supported at every hop. Such arrangements would only require be supported at every hop. Such arrangements would only require
simple registered/not-registered packet classification, rather than simple registered/not-registered packet classification, rather than
the managed, application-specific traffic policing against customer- the managed, application-specific traffic policing against customer-
specific traffic contracts that Diffserv uses. specific traffic contracts that Diffserv uses.
8.2. 'Latency Friendliness' 8.2. 'Latency Friendliness'
The L4S service does rely on self-constraint - not in terms of The L4S service does rely on self-constraint - not in terms of
limiting rate, but in terms of limiting latency (burstiness). It is limiting rate, but in terms of limiting latency (burstiness). It is
hoped that self-interest and standardisation of dynamic behaviour hoped that self-interest and standardisation of dynamic behaviour
(cf. TCP slow-start) will be sufficient to prevent transports from (especially flow start-up) will be sufficient to prevent transports
sending excessive bursts of L4S traffic, given the application's own from sending excessive bursts of L4S traffic, given the application's
latency will suffer most from such behaviour. own latency will suffer most from such behaviour.
Whether burst policing becomes necessary remains to be seen. Without Whether burst policing becomes necessary remains to be seen. Without
it, there will be potential for attacks on the low latency of the L4S it, there will be potential for attacks on the low latency of the L4S
service. However it may only be necessary to apply such policing service. However it may only be necessary to apply such policing
reactively, e.g. punitively targeted at any deployments of new bursty reactively, e.g. punitively targeted at any deployments of new bursty
malware. malware.
A per-flow (5-tuple) queue protection function A per-flow (5-tuple) queue protection function
[I-D.briscoe-docsis-q-protection] has been developed for the low [I-D.briscoe-docsis-q-protection] has been developed for the low
latency queue in DOCSIS, which has adopted the DualQ L4S latency queue in DOCSIS, which has adopted the DualQ L4S
skipping to change at page 25, line 38 skipping to change at page 26, line 10
The design of L4S-friendly rate policers will require a separate The design of L4S-friendly rate policers will require a separate
dedicated document. For further discussion of the interaction dedicated document. For further discussion of the interaction
between L4S and Diffserv, see [I-D.briscoe-tsvwg-l4s-diffserv]. between L4S and Diffserv, see [I-D.briscoe-tsvwg-l4s-diffserv].
8.4. ECN Integrity 8.4. ECN Integrity
Receiving hosts can fool a sender into downloading faster by Receiving hosts can fool a sender into downloading faster by
suppressing feedback of ECN marks (or of losses if retransmissions suppressing feedback of ECN marks (or of losses if retransmissions
are not necessary or available otherwise). Various ways to protect are not necessary or available otherwise). Various ways to protect
TCP feedback integrity have been developed. For instance: transport feedback integrity have been developed. For instance:
o The sender can test the integrity of the receiver's feedback by o The sender can test the integrity of the receiver's feedback by
occasionally setting the IP-ECN field to the congestion occasionally setting the IP-ECN field to the congestion
experienced (CE) codepoint, which is normally only set by a experienced (CE) codepoint, which is normally only set by a
congested link. Then the sender can test whether the receiver's congested link. Then the sender can test whether the receiver's
feedback faithfully reports what it expects (see 2nd para of feedback faithfully reports what it expects (see 2nd para of
Section 20.2 of [RFC3168]). Section 20.2 of [RFC3168]).
o A network can enforce a congestion response to its ECN markings o A network can enforce a congestion response to its ECN markings
(or packet losses) by auditing congestion exposure (ConEx) (or packet losses) by auditing congestion exposure (ConEx)
skipping to change at page 26, line 17 skipping to change at page 26, line 35
o The ECN Nonce [RFC3540] was proposed to detect tampering with o The ECN Nonce [RFC3540] was proposed to detect tampering with
congestion feedback, but it has been reclassified as historic congestion feedback, but it has been reclassified as historic
[RFC8311]. [RFC8311].
Appendix C.1 of [I-D.ietf-tsvwg-ecn-l4s-id] gives more details of Appendix C.1 of [I-D.ietf-tsvwg-ecn-l4s-id] gives more details of
these techniques including their applicability and pros and cons. these techniques including their applicability and pros and cons.
9. Acknowledgements 9. Acknowledgements
Thanks to Richard Scheffenegger, Wes Eddy, Karen Nielsen and David Thanks to Richard Scheffenegger, Wes Eddy, Karen Nielsen, David Black
Black for their useful review comments. and Jake Holland for their useful review comments.
Bob Briscoe and Koen De Schepper were part-funded by the European Bob Briscoe and Koen De Schepper were part-funded by the European
Community under its Seventh Framework Programme through the Reducing Community under its Seventh Framework Programme through the Reducing
Internet Transport Latency (RITE) project (ICT-317700). Bob Briscoe Internet Transport Latency (RITE) project (ICT-317700). Bob Briscoe
was also part-funded by the Research Council of Norway through the was also part-funded by the Research Council of Norway through the
TimeIn project. The views expressed here are solely those of the TimeIn project, partly by CableLabs and partly by the Comcast
Innovation Fund. The views expressed here are solely those of the
authors. authors.
10. References 10. References
10.1. Normative References 10.1. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997, DOI 10.17487/RFC2119, March 1997,
<https://www.rfc-editor.org/info/rfc2119>. <https://www.rfc-editor.org/info/rfc2119>.
10.2. Informative References 10.2. Informative References
[DCttH15] De Schepper, K., Bondarenko, O., Briscoe, B., and I. [DCttH15] De Schepper, K., Bondarenko, O., Briscoe, B., and I.
Tsang, "`Data Centre to the Home': Ultra-Low Latency for Tsang, "`Data Centre to the Home': Ultra-Low Latency for
All", RITE project Technical Report , 2015, All", RITE project Technical Report , 2015,
<http://riteproject.eu/publications/>. <http://riteproject.eu/publications/>.
[DOCSIS3.1]
CableLabs, "MAC and Upper Layer Protocols Interface
(MULPI) Specification, CM-SP-MULPIv3.1", Data-Over-Cable
Service Interface Specifications DOCSIS(R) 3.1 Version i17
or later, January 2019, <https://specification-
search.cablelabs.com/CM-SP-MULPIv3.1>.
[DualPI2Linux]
Albisser, O., De Schepper, K., Briscoe, B., Tilmans, O.,
and H. Steen, "DUALPI2 - Low Latency, Low Loss and
Scalable (L4S) AQM", Proc. Linux Netdev 0x13 , March 2019,
<https://www.netdevconf.org/0x13/session.html?talk-
DUALPI2-AQM>.
[Hohlfeld14] [Hohlfeld14]
Hohlfeld , O., Pujol, E., Ciucu, F., Feldmann, A., and P. Hohlfeld , O., Pujol, E., Ciucu, F., Feldmann, A., and P.
Barford, "A QoE Perspective on Sizing Network Buffers", Barford, "A QoE Perspective on Sizing Network Buffers",
Proc. ACM Internet Measurement Conf (IMC'14) hmm, November Proc. ACM Internet Measurement Conf (IMC'14) hmm, November
2014. 2014.
[I-D.briscoe-conex-policing] [I-D.briscoe-conex-policing]
Briscoe, B., "Network Performance Isolation using Briscoe, B., "Network Performance Isolation using
Congestion Policing", draft-briscoe-conex-policing-01 Congestion Policing", draft-briscoe-conex-policing-01
(work in progress), February 2014. (work in progress), February 2014.
skipping to change at page 27, line 23 skipping to change at page 28, line 12
draft-briscoe-tsvwg-l4s-diffserv-02 (work in progress), draft-briscoe-tsvwg-l4s-diffserv-02 (work in progress),
November 2018. November 2018.
[I-D.cardwell-iccrg-bbr-congestion-control] [I-D.cardwell-iccrg-bbr-congestion-control]
Cardwell, N., Cheng, Y., Yeganeh, S., and V. Jacobson, Cardwell, N., Cheng, Y., Yeganeh, S., and V. Jacobson,
"BBR Congestion Control", draft-cardwell-iccrg-bbr- "BBR Congestion Control", draft-cardwell-iccrg-bbr-
congestion-control-00 (work in progress), July 2017. congestion-control-00 (work in progress), July 2017.
[I-D.ietf-quic-transport] [I-D.ietf-quic-transport]
Iyengar, J. and M. Thomson, "QUIC: A UDP-Based Multiplexed Iyengar, J. and M. Thomson, "QUIC: A UDP-Based Multiplexed
and Secure Transport", draft-ietf-quic-transport-20 (work and Secure Transport", draft-ietf-quic-transport-25 (work
in progress), April 2019. in progress), January 2020.
[I-D.ietf-tcpm-accurate-ecn] [I-D.ietf-tcpm-accurate-ecn]
Briscoe, B., Kuehlewind, M., and R. Scheffenegger, "More Briscoe, B., Kuehlewind, M., and R. Scheffenegger, "More
Accurate ECN Feedback in TCP", draft-ietf-tcpm-accurate- Accurate ECN Feedback in TCP", draft-ietf-tcpm-accurate-
ecn-08 (work in progress), March 2019. ecn-09 (work in progress), July 2019.
[I-D.ietf-tcpm-generalized-ecn] [I-D.ietf-tcpm-generalized-ecn]
Bagnulo, M. and B. Briscoe, "ECN++: Adding Explicit Bagnulo, M. and B. Briscoe, "ECN++: Adding Explicit
Congestion Notification (ECN) to TCP Control Packets", Congestion Notification (ECN) to TCP Control Packets",
draft-ietf-tcpm-generalized-ecn-03 (work in progress), draft-ietf-tcpm-generalized-ecn-05 (work in progress),
October 2018. November 2019.
[I-D.ietf-tsvwg-aqm-dualq-coupled] [I-D.ietf-tsvwg-aqm-dualq-coupled]
Schepper, K., Briscoe, B., and G. White, "DualQ Coupled Schepper, K., Briscoe, B., and G. White, "DualQ Coupled
AQMs for Low Latency, Low Loss and Scalable Throughput AQMs for Low Latency, Low Loss and Scalable Throughput
(L4S)", draft-ietf-tsvwg-aqm-dualq-coupled-09 (work in (L4S)", draft-ietf-tsvwg-aqm-dualq-coupled-10 (work in
progress), July 2019. progress), July 2019.
[I-D.ietf-tsvwg-ecn-encap-guidelines] [I-D.ietf-tsvwg-ecn-encap-guidelines]
Briscoe, B., Kaippallimalil, J., and P. Thaler, Briscoe, B., Kaippallimalil, J., and P. Thaler,
"Guidelines for Adding Congestion Notification to "Guidelines for Adding Congestion Notification to
Protocols that Encapsulate IP", draft-ietf-tsvwg-ecn- Protocols that Encapsulate IP", draft-ietf-tsvwg-ecn-
encap-guidelines-13 (work in progress), May 2019. encap-guidelines-13 (work in progress), May 2019.
[I-D.ietf-tsvwg-ecn-l4s-id] [I-D.ietf-tsvwg-ecn-l4s-id]
Schepper, K. and B. Briscoe, "Identifying Modified Schepper, K. and B. Briscoe, "Identifying Modified
Explicit Congestion Notification (ECN) Semantics for Explicit Congestion Notification (ECN) Semantics for
Ultra-Low Queuing Delay (L4S)", draft-ietf-tsvwg-ecn-l4s- Ultra-Low Queuing Delay (L4S)", draft-ietf-tsvwg-ecn-l4s-
id-06 (work in progress), March 2019. id-08 (work in progress), November 2019.
[I-D.smith-encrypted-traffic-management] [I-D.smith-encrypted-traffic-management]
Smith, K., "Network management of encrypted traffic", Smith, K., "Network management of encrypted traffic",
draft-smith-encrypted-traffic-management-05 (work in draft-smith-encrypted-traffic-management-05 (work in
progress), May 2016. progress), May 2016.
[I-D.sridharan-tcpm-ctcp] [I-D.sridharan-tcpm-ctcp]
Sridharan, M., Tan, K., Bansal, D., and D. Thaler, Sridharan, M., Tan, K., Bansal, D., and D. Thaler,
"Compound TCP: A New TCP Congestion Control for High-Speed "Compound TCP: A New TCP Congestion Control for High-Speed
and Long Distance Networks", draft-sridharan-tcpm-ctcp-02 and Long Distance Networks", draft-sridharan-tcpm-ctcp-02
skipping to change at page 28, line 37 skipping to change at page 29, line 26
[I-D.white-tsvwg-nqb] [I-D.white-tsvwg-nqb]
White, G. and T. Fossati, "Identifying and Handling Non White, G. and T. Fossati, "Identifying and Handling Non
Queue Building Flows in a Bottleneck Link", draft-white- Queue Building Flows in a Bottleneck Link", draft-white-
tsvwg-nqb-02 (work in progress), June 2019. tsvwg-nqb-02 (work in progress), June 2019.
[L4Sdemo16] [L4Sdemo16]
Bondarenko, O., De Schepper, K., Tsang, I., and B. Bondarenko, O., De Schepper, K., Tsang, I., and B.
Briscoe, "orderedUltra-Low Delay for All: Live Experience, Briscoe, "orderedUltra-Low Delay for All: Live Experience,
Live Analysis", Proc. MMSYS'16 pp33:1--33:4, May 2016, Live Analysis", Proc. MMSYS'16 pp33:1--33:4, May 2016,
<http://dl.acm.org/citation.cfm?doid=2910017.2910633 <http://dl.acm.org/citation.cfm?doid=2910017.2910633
(videos of demos: https://riteproject.eu/ (videos of demos:
dctth/#1511dispatchwg )>. https://riteproject.eu/dctth/#1511dispatchwg )>.
[LinuxPrague]
Briscoe, B., De Schepper, K., Albisser, O., Misund, J.,
Tilmans, O., Kuehlewind, M., and A. Ahmed, "Implementing
the `TCP Prague' Requirements for Low Latency Low Loss
Scalable Throughput (L4S)", Proc. Linux Netdev 0x13 ,
March 2019, <https://www.netdevconf.org/0x13/
session.html?talk-tcp-prague-l4s>.
[Mathis09] [Mathis09]
Mathis, M., "Relentless Congestion Control", PFLDNeT'09 , Mathis, M., "Relentless Congestion Control", PFLDNeT'09 ,
May 2009, <https://www.gdt.id.au/~gdt/ May 2009, <https://www.gdt.id.au/~gdt/
presentations/2010-07-06-questnet-tcp/reference- presentations/2010-07-06-questnet-tcp/reference-
materials/papers/ materials/papers/mathis-relentless-congestion-
mathis-relentless-congestion-control.pdf>. control.pdf>.
[NewCC_Proc] [NewCC_Proc]
Eggert, L., "Experimental Specification of New Congestion Eggert, L., "Experimental Specification of New Congestion
Control Algorithms", IETF Operational Note ion-tsv-alt-cc, Control Algorithms", IETF Operational Note ion-tsv-alt-cc,
July 2007. July 2007.
[PI2] De Schepper, K., Bondarenko, O., Tsang, I., and B.
Briscoe, "PI^2 : A Linearized AQM for both Classic and
Scalable TCP", Proc. ACM CoNEXT 2016 pp.105-119, December
2016,
<http://dl.acm.org/citation.cfm?doid=2999572.2999578>.
[PragueLinux] [PragueLinux]
Briscoe, B., De Schepper, K., Albisser, O., Misund, J., Briscoe, B., De Schepper, K., Albisser, O., Misund, J.,
Tilmans, O., Kuehlewind, M., and A. Ahmed, "Implementing Tilmans, O., Kuehlewind, M., and A. Ahmed, "Implementing
the `TCP Prague' Requirements for Low Latency Low Loss the `TCP Prague' Requirements for Low Latency Low Loss
Scalable Throughput (L4S)", Proc. Linux Netdev 0x13 , Scalable Throughput (L4S)", Proc. Linux Netdev 0x13 ,
March 2019, <https://www.netdevconf.org/0x13/ March 2019, <https://www.netdevconf.org/0x13/
session.html?talk-tcp-prague-l4s>. session.html?talk-tcp-prague-l4s>.
[RFC2697] Heinanen, J. and R. Guerin, "A Single Rate Three Color [RFC2697] Heinanen, J. and R. Guerin, "A Single Rate Three Color
Marker", RFC 2697, DOI 10.17487/RFC2697, September 1999, Marker", RFC 2697, DOI 10.17487/RFC2697, September 1999,
skipping to change at page 30, line 19 skipping to change at page 31, line 14
[RFC4774] Floyd, S., "Specifying Alternate Semantics for the [RFC4774] Floyd, S., "Specifying Alternate Semantics for the
Explicit Congestion Notification (ECN) Field", BCP 124, Explicit Congestion Notification (ECN) Field", BCP 124,
RFC 4774, DOI 10.17487/RFC4774, November 2006, RFC 4774, DOI 10.17487/RFC4774, November 2006,
<https://www.rfc-editor.org/info/rfc4774>. <https://www.rfc-editor.org/info/rfc4774>.
[RFC4960] Stewart, R., Ed., "Stream Control Transmission Protocol", [RFC4960] Stewart, R., Ed., "Stream Control Transmission Protocol",
RFC 4960, DOI 10.17487/RFC4960, September 2007, RFC 4960, DOI 10.17487/RFC4960, September 2007,
<https://www.rfc-editor.org/info/rfc4960>. <https://www.rfc-editor.org/info/rfc4960>.
[RFC5348] Floyd, S., Handley, M., Padhye, J., and J. Widmer, "TCP
Friendly Rate Control (TFRC): Protocol Specification",
RFC 5348, DOI 10.17487/RFC5348, September 2008,
<https://www.rfc-editor.org/info/rfc5348>.
[RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion [RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion
Control", RFC 5681, DOI 10.17487/RFC5681, September 2009, Control", RFC 5681, DOI 10.17487/RFC5681, September 2009,
<https://www.rfc-editor.org/info/rfc5681>. <https://www.rfc-editor.org/info/rfc5681>.
[RFC5925] Touch, J., Mankin, A., and R. Bonica, "The TCP [RFC5925] Touch, J., Mankin, A., and R. Bonica, "The TCP
Authentication Option", RFC 5925, DOI 10.17487/RFC5925, Authentication Option", RFC 5925, DOI 10.17487/RFC5925,
June 2010, <https://www.rfc-editor.org/info/rfc5925>. June 2010, <https://www.rfc-editor.org/info/rfc5925>.
[RFC6040] Briscoe, B., "Tunnelling of Explicit Congestion [RFC6040] Briscoe, B., "Tunnelling of Explicit Congestion
Notification", RFC 6040, DOI 10.17487/RFC6040, November Notification", RFC 6040, DOI 10.17487/RFC6040, November
skipping to change at page 32, line 9 skipping to change at page 33, line 9
<https://www.rfc-editor.org/info/rfc8511>. <https://www.rfc-editor.org/info/rfc8511>.
[TCP-CA] Jacobson, V. and M. Karels, "Congestion Avoidance and [TCP-CA] Jacobson, V. and M. Karels, "Congestion Avoidance and
Control", Laurence Berkeley Labs Technical Report , Control", Laurence Berkeley Labs Technical Report ,
November 1988, <http://ee.lbl.gov/papers/congavoid.pdf>. November 1988, <http://ee.lbl.gov/papers/congavoid.pdf>.
[TCP-sub-mss-w] [TCP-sub-mss-w]
Briscoe, B. and K. De Schepper, "Scaling TCP's Congestion Briscoe, B. and K. De Schepper, "Scaling TCP's Congestion
Window for Small Round Trip Times", BT Technical Report Window for Small Round Trip Times", BT Technical Report
TR-TUB8-2015-002, May 2015, TR-TUB8-2015-002, May 2015,
<http://www.bobbriscoe.net/projects/latency/ <http://www.bobbriscoe.net/projects/latency/sub-mss-
sub-mss-w.pdf>. w.pdf>.
[UnorderedLTE] [UnorderedLTE]
Austrheim, M., "Implementing immediate forwarding for 4G Austrheim, M., "Implementing immediate forwarding for 4G
in a network simulator", Masters Thesis, Uni Oslo , June in a network simulator", Masters Thesis, Uni Oslo , June
2019. 2019.
Appendix A. Standardization items Appendix A. Standardization items
The following table includes all the items that will need to be The following table includes all the items that will need to be
standardized to provide a full L4S architecture. standardized to provide a full L4S architecture.
skipping to change at page 34, line 40 skipping to change at page 35, line 40
| | | | | | | | | | | | | | | | | |
| 5-2 | tcpm/ | | | Y | Y | Y | ? | | 5-2 | tcpm/ | | | Y | Y | Y | ? |
| | iccrg? | | | | | | | | | iccrg? | | | | | | |
| 5-3 | tcpm/ | | | Y | Y | Y | ? | | 5-3 | tcpm/ | | | Y | Y | Y | ? |
| | iccrg? | | | | | | | | | iccrg? | | | | | | |
+-----+--------+-----+-------+-----------+--------+--------+--------+ +-----+--------+-----+-------+-----------+--------+--------+--------+
Authors' Addresses Authors' Addresses
Bob Briscoe (editor) Bob Briscoe (editor)
CableLabs Independent
UK UK
Email: ietf@bobbriscoe.net Email: ietf@bobbriscoe.net
URI: http://bobbriscoe.net/ URI: http://bobbriscoe.net/
Koen De Schepper Koen De Schepper
Nokia Bell Labs Nokia Bell Labs
Antwerp Antwerp
Belgium Belgium
Email: koen.de_schepper@nokia.com Email: koen.de_schepper@nokia.com
 End of changes. 67 change blocks. 
288 lines changed or deleted 362 lines changed or added

This html diff was produced by rfcdiff 1.47. The latest version is available from http://tools.ietf.org/tools/rfcdiff/