draft-ietf-rmcat-nada-00.txt | draft-ietf-rmcat-nada-01.txt | |||
---|---|---|---|---|
Network Working Group X. Zhu, R. Pan | Network Working Group X. Zhu | |||
Internet Draft M. A. Ramalho, S. Mena | Internet-Draft R. Pan | |||
Intended Status: Informational C. Ganzhorn, P. E. Jones | Intended status: Experimental M. Ramalho | |||
Expires: October 29, 2015 Cisco Systems | Expires: April 11, 2016 S. Mena | |||
S. De Aronco | P. Jones | |||
Ecole Polytechnique Federale de Lausanne | J. Fu | |||
April 28, 2015 | Cisco Systems | |||
S. D'Aronco | ||||
EPFL | ||||
C. Ganzhorn | ||||
October 9, 2015 | ||||
NADA: A Unified Congestion Control Scheme for Real-Time Media | NADA: A Unified Congestion Control Scheme for Real-Time Media | |||
draft-ietf-rmcat-nada-00 | draft-ietf-rmcat-nada-01 | |||
Abstract | Abstract | |||
Network-Assisted Dynamic Adaptation (NADA) is a novel congestion | This document describes NADA (network-assisted dynamic adaptation), a | |||
control scheme for interactive real-time media applications, such as | novel congestion control scheme for interactive real-time media | |||
video conferencing. In NADA, the sender regulates its sending rate | applications, such as video conferencing. In the proposed scheme, | |||
based on either implicit or explicit congestion signaling in a | the sender regulates its sending rate based on either implicit or | |||
consistent manner. As one example of explicit signaling, NADA can | explicit congestion signaling, in a unified approach. The scheme can | |||
benefit from explicit congestion notification (ECN) markings from | benefit from explicit congestion notification (ECN) markings from | |||
network nodes. It also maintains consistent sender behavior in the | network nodes. It also maintains consistent sender behavior in the | |||
absence of explicit signaling by reacting to queuing delay and packet | absence of such markings, by reacting to queuing delays and packet | |||
loss. | losses instead. | |||
This document describes the overall system architecture for NADA, as | ||||
well as recommended behavior at the sender and the receiver. | ||||
Status of this Memo | Status of This Memo | |||
This Internet-Draft is submitted to IETF in full conformance with the | This Internet-Draft is submitted in full conformance with the | |||
provisions of BCP 78 and BCP 79. | provisions of BCP 78 and BCP 79. | |||
Internet-Drafts are working documents of the Internet Engineering | Internet-Drafts are working documents of the Internet Engineering | |||
Task Force (IETF), its areas, and its working groups. Note that | Task Force (IETF). Note that other groups may also distribute | |||
other groups may also distribute working documents as | working documents as Internet-Drafts. The list of current Internet- | |||
Internet-Drafts. | Drafts is at http://datatracker.ietf.org/drafts/current/. | |||
Internet-Drafts are draft documents valid for a maximum of six months | Internet-Drafts are draft documents valid for a maximum of six months | |||
and may be updated, replaced, or obsoleted by other documents at any | and may be updated, replaced, or obsoleted by other documents at any | |||
time. It is inappropriate to use Internet-Drafts as reference | time. It is inappropriate to use Internet-Drafts as reference | |||
material or to cite them other than as "work in progress." | material or to cite them other than as "work in progress." | |||
The list of current Internet-Drafts can be accessed at | This Internet-Draft will expire on April 11, 2016. | |||
http://www.ietf.org/1id-abstracts.html | ||||
The list of Internet-Draft Shadow Directories can be accessed at | ||||
http://www.ietf.org/shadow.html | ||||
Copyright and License Notice | Copyright Notice | |||
Copyright (c) 2012 IETF Trust and the persons identified as the | Copyright (c) 2015 IETF Trust and the persons identified as the | |||
document authors. All rights reserved. | document authors. All rights reserved. | |||
This document is subject to BCP 78 and the IETF Trust's Legal | This document is subject to BCP 78 and the IETF Trust's Legal | |||
Provisions Relating to IETF Documents | Provisions Relating to IETF Documents | |||
(http://trustee.ietf.org/license-info) in effect on the date of | (http://trustee.ietf.org/license-info) in effect on the date of | |||
publication of this document. Please review these documents | publication of this document. Please review these documents | |||
carefully, as they describe your rights and restrictions with respect | carefully, as they describe your rights and restrictions with respect | |||
to this document. Code Components extracted from this document must | to this document. Code Components extracted from this document must | |||
include Simplified BSD License text as described in Section 4.e of | include Simplified BSD License text as described in Section 4.e of | |||
the Trust Legal Provisions and are provided without warranty as | the Trust Legal Provisions and are provided without warranty as | |||
described in the Simplified BSD License. | described in the Simplified BSD License. | |||
Table of Contents | Table of Contents | |||
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 | 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 | |||
2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . 3 | 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3 | |||
3. System Model . . . . . . . . . . . . . . . . . . . . . . . . . 3 | 3. System Overview . . . . . . . . . . . . . . . . . . . . . . . 3 | |||
4. NADA Receiver Behavior . . . . . . . . . . . . . . . . . . . . 4 | 4. Core Congestion Control Algorithm . . . . . . . . . . . . . . 4 | |||
4.1 Estimation of one-way delay and queuing delay . . . . . . . 4 | 4.1. Mathematical Notations . . . . . . . . . . . . . . . . . 5 | |||
4.2 Estimation of packet loss/marking ratio . . . . . . . . . . 5 | 4.2. Receiver-Side Algorithm . . . . . . . . . . . . . . . . . 7 | |||
4.3 Non-linear warping of delay . . . . . . . . . . . . . . . . 6 | 4.3. Sender-Side Algorithm . . . . . . . . . . . . . . . . . . 9 | |||
4.4 Aggregating congestion signals . . . . . . . . . . . . . . . 7 | 5. Practical Implementation of NADA . . . . . . . . . . . . . . 10 | |||
4.5 Estimating receiving rate . . . . . . . . . . . . . . . . . 7 | 5.1. Receiver-Side Operation . . . . . . . . . . . . . . . . . 10 | |||
4.6 Sending periodic feedback . . . . . . . . . . . . . . . . . 7 | 5.1.1. Estimation of one-way delay and queuing delay . . . . 11 | |||
4.7 Discussions on delay metrics . . . . . . . . . . . . . . . . 8 | 5.1.2. Estimation of packet loss/marking ratio . . . . . . . 11 | |||
5. NADA Sender Behavior . . . . . . . . . . . . . . . . . . . . . 9 | 5.1.3. Estimation of receiving rate . . . . . . . . . . . . 11 | |||
5.1 Reference rate calculation . . . . . . . . . . . . . . . . . 10 | 5.2. Sender-Side Operation . . . . . . . . . . . . . . . . . . 12 | |||
5.1.1 Accelerated ramp up . . . . . . . . . . . . . . . . . . 10 | 5.2.1. Rate shaping buffer . . . . . . . . . . . . . . . . . 12 | |||
5.1.2. Gradual rate update . . . . . . . . . . . . . . . . . . 11 | 5.2.2. Adjusting video target rate and sending rate . . . . 13 | |||
5.2 Video encoder rate control . . . . . . . . . . . . . . . . . 12 | 6. Discussions and Further Investigations . . . . . . . . . . . 13 | |||
5.3 Rate shaping buffer . . . . . . . . . . . . . . . . . . . . 12 | 6.1. Choice of delay metrics . . . . . . . . . . . . . . . . . 13 | |||
5.4 Adjusting video target rate and sending rate . . . . . . . . 12 | 6.2. Method for delay, loss, and marking ratio estimation . . 14 | |||
6. Incremental Deployment . . . . . . . . . . . . . . . . . . . . 13 | 6.3. Impact of parameter values . . . . . . . . . . . . . . . 14 | |||
7. Implementation Status . . . . . . . . . . . . . . . . . . . . . 13 | 6.4. Sender-based vs. receiver-based calculation . . . . . . . 15 | |||
8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . . 14 | 6.5. Incremental deployment . . . . . . . . . . . . . . . . . 16 | |||
9. References . . . . . . . . . . . . . . . . . . . . . . . . . . 14 | 7. Implementation Status . . . . . . . . . . . . . . . . . . . . 16 | |||
9.1 Normative References . . . . . . . . . . . . . . . . . . . 14 | 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 16 | |||
9.2 Informative References . . . . . . . . . . . . . . . . . . 14 | 9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 17 | |||
Appendix A. Network Node Operations . . . . . . . . . . . . . . . 15 | 10. References . . . . . . . . . . . . . . . . . . . . . . . . . 17 | |||
A.1 Default behavior of drop tail . . . . . . . . . . . . . . . 16 | 10.1. Normative References . . . . . . . . . . . . . . . . . . 17 | |||
A.2 ECN marking . . . . . . . . . . . . . . . . . . . . . . . . 16 | 10.2. Informative References . . . . . . . . . . . . . . . . . 17 | |||
A.3 PCN marking . . . . . . . . . . . . . . . . . . . . . . . . 16 | Appendix A. Network Node Operations . . . . . . . . . . . . . . 19 | |||
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 17 | A.1. Default behavior of drop tail queues . . . . . . . . . . 19 | |||
A.2. RED-based ECN marking . . . . . . . . . . . . . . . . . . 19 | ||||
A.3. Random Early Marking with Virtual Queues . . . . . . . . 20 | ||||
1. Introduction | Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 21 | |||
1. Introduction | ||||
Interactive real-time media applications introduce a unique set of | Interactive real-time media applications introduce a unique set of | |||
challenges for congestion control. Unlike TCP, the mechanism used for | challenges for congestion control. Unlike TCP, the mechanism used | |||
real-time media needs to adapt quickly to instantaneous bandwidth | for real-time media needs to adapt quickly to instantaneous bandwidth | |||
changes, accommodate fluctuations in the output of video encoder rate | changes, accommodate fluctuations in the output of video encoder rate | |||
control, and cause low queuing delay over the network. An ideal | control, and cause low queuing delay over the network. An ideal | |||
scheme should also make effective use of all types of congestion | scheme should also make effective use of all types of congestion | |||
signals, including packet loss, queuing delay, and explicit | signals, including packet loss, queuing delay, and explicit | |||
congestion notification (ECN) [RFC3168] markings. | congestion notification (ECN) [RFC3168] markings. The requirements | |||
for the congestion control algorithm are outlined in | ||||
[I-D.ietf-rmcat-cc-requirements]. | ||||
Based on the above considerations, this document describes a scheme | This document describes an experimental congestion control scheme | |||
called network-assisted dynamic adaptation (NADA). The NADA design | called network-assisted dynamic adaptation (NADA). The NADA design | |||
benefits from explicit congestion control signals (e.g., ECN | benefits from explicit congestion control signals (e.g., ECN | |||
markings) from the network, yet also operates when only implicit | markings) from the network, yet also operates when only implicit | |||
congestion indicators (delay and/or loss) are available. In addition, | congestion indicators (delay and/or loss) are available. In | |||
it supports weighted bandwidth sharing among competing video flows. | addition, it supports weighted bandwidth sharing among competing | |||
video flows. The signaling mechanism consists of standard RTP | ||||
This documentation describes the overall system architecture, | timestamp [RFC3550] and standard RTCP feedback reports. | |||
recommended designs at the sender and receiver, as well as expected | ||||
network node operations. The signaling mechanism consists of standard | ||||
RTP timestamp [RFC3550] and standard RTCP feedback reports. | ||||
2. Terminology | 2. Terminology | |||
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", | The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", | |||
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this | "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this | |||
document are to be interpreted as described in RFC 2119 [RFC2119]. | document are to be interpreted as described [RFC2119]. | |||
3. System Model | 3. System Overview | |||
The overall system consists of the following elements: | Figure 1 shows the end-to-end system for real-time media transport | |||
that NADA operates in. | ||||
* Source media stream, in the form of consecutive raw video | +---------+ r_vin +--------+ +--------+ +----------+ | |||
frames and audio samples; | | Media |<--------| RTP | |Network | | RTP | | |||
| Encoder |========>| Sender |=======>| Node |====>| Receiver | | ||||
+---------+ r_vout +--------+ r_send +--------+ +----------+ | ||||
/|\ | | ||||
| | | ||||
+---------------------------------+ | ||||
RTCP Feedback Report | ||||
* Media encoder with rate control capabilities. It takes the | Figure 1: System Overview | |||
source media stream and encodes it to an RTP stream at a target | ||||
bit rate R_v. Note that the actual output rate from the encoder | ||||
R_o may fluctuate around the target R_v. Also, the encoder can | ||||
only change its rate at rather coarse time intervals, e.g., once | ||||
every 0.5 seconds. | ||||
* RTP sender, responsible for calculating the target bit rate | o Media encoder with rate control capabilities. It encodes the | |||
R_n based on network congestion indicators (delay, loss, or ECN | source media stream into an RTP stream with target bit rate r_vin. | |||
marking reports from the receiver), for updating the video | The actual output rate from the encoder r_vout may fluctuate | |||
encoder with a new target rate R_v, and for regulating the | around the target r_vin. In addition, the encoder can only change | |||
actual sending rate R_s accordingly. A rate shaping buffer is | its bit rate at rather coarse time intervals, e.g., once every 0.5 | |||
employed to absorb the instantaneous difference between video | seconds. | |||
encoder output rate R_v and sending rate R_s. The buffer size | ||||
L_s, together with R_n, influences the calculation of actual | ||||
sending rate R_s and video encoder target rate R_v. The RTP | ||||
sender also generates RTP timestamp in outgoing packets. | ||||
* RTP receiver, responsible for measuring and estimating end-to- | o RTP sender: responsible for calculating the NADA reference rate | |||
end delay based on sender RTP timestamp. In the presence of | based on network congestion indicators (delay, loss, or ECN | |||
packet loss and ECN markings, it keeps track of packet loss and | marking reports from the receiver), for updating the video encoder | |||
ECN marking ratios. It calculates the equivalent delay x_n that | with a new target rate r_vin, and for regulating the actual | |||
accounts for queuing delay, ECN marking, and packet loss, as | sending rate r_send accordingly. The RTP sender also provides an | |||
well as the derivative (i.e., rate of change) of this congestion | RTP timestamp for each outgoing packet. | |||
signal as x'_n. The receiver feeds both pieces of information | ||||
(x_n and x'_n) back to the sender via periodic RTCP reports. | ||||
* Network node, with several modes of operation. The system can | o RTP receiver: responsible for measuring and estimating end-to-end | |||
work with the default behavior of a simple drop tail queue. It | delay based on sender RTP timestamp, packet loss and ECN marking | |||
can also benefit from advanced AQM features such as RED-based | ratios, as well as receiving rate (r_recv) of the flow. It | |||
ECN marking, and PCN marking using a token bucket algorithm. | calculates the aggregated congestion signal (x_n) that accounts | |||
Note that network node operation is out of scope for the design | for queuing delay, ECN marking, and packet losses, and determines | |||
of NADA. | the mode for sender rate adaptation (rmode) based on whether the | |||
flow has encountered any standing non-zero congestion. The | ||||
receiver sends periodic RTCP reports back to the sender, | ||||
containing values of x_n, rmode, and r_recv. | ||||
In the following, we will elaborate on the respective operations at the | o Network node with several modes of operation. The system can work | |||
NADA receiver and sender. | with the default behavior of a simple drop tail queue. It can | |||
also benefit from advanced AQM features such as PIE, FQ-CoDel, | ||||
RED-based ECN marking, and PCN marking using a token bucket | ||||
algorithm. Note that network node operation is out of scope for | ||||
the design of NADA. | ||||
4. NADA Receiver Behavior | 4. Core Congestion Control Algorithm | |||
The receiver continuously monitors end-to-end per-packet statistics in | Like TCP-Friendly Rate Control (TFRC) [Floyd-CCR00] [RFC5348], NADA | |||
terms of delay, loss, and/or ECN marking ratios. It then aggregates all | is a rate-based congestion control algorithm. In its simplest form, | |||
forms of congestion indicators into the form of an equivalent delay and | the sender reacts to the collection of network congestion indicators | |||
periodically reports this back to the sender. In addition, the receiver | in the form of an aggregated congestion signal, and operates in one | |||
tracks the receiving rate of the flow and includes that in the feedback | of two modes: | |||
message. | ||||
4.1 Estimation of one-way delay and queuing delay | o Accelerated ramp-up: when the bottleneck is deemed to be | |||
underutilized, the rate increases multiplicatively with respect to | ||||
the rate of previously successful transmissions. The rate | ||||
increase mutliplier (gamma) is calculated based on observed round- | ||||
trip-time and target feedback interval, so as to limit self- | ||||
inflicted queuing delay. | ||||
The delay estimation process in NADA follows a similar approach as in | o Gradual rate update: in the presence of non-zero aggregate | |||
earlier delay-based congestion control schemes, such as LEDBAT | congestion signal, the sending rate is adjusted in reaction to | |||
[RFC6817]. NADA estimates the forward delay as having a constant base | both its value (x_n) and its change in value (x_diff). | |||
delay component plus a time varying queuing delay component. The base | ||||
delay is estimated as the minimum value of one-way delay observed over a | ||||
relatively long period (e.g., tens of minutes), whereas the individual | ||||
queuing delay value is taken to be the difference between one-way delay | ||||
and base delay. | ||||
In mathematical terms, for packet n arriving at the receiver, one-way | This section introduces the list of mathematical notations and | |||
delay is calculated as: | describes the core congestion control algorithm at the sender and | |||
receiver, respectively. Additional details on recommended practical | ||||
implementations are described in Section 5.1 and Section 5.2. | ||||
z_n = t_r,n - t_s,n, | 4.1. Mathematical Notations | |||
where t_s,n and t_r,n are sender and receiver timestamps, respectively. | This section summarizes the list of variables and parameters used in | |||
A real-world implementation should also properly handle practical issues | the NADA algorithm. | |||
such as wrap-around in the value of z_n, which are omitted from the | ||||
above simple expression for brevity. | ||||
The base delay, d_f, is estimated as the minimum value of previously | +--------------+-------------------------------------------------+ | |||
observed z_n's over a relatively long period. This assumes that the | | Notation | Variable Name | | |||
drift between sending and receiving clocks remains bounded by a small | +--------------+-------------------------------------------------+ | |||
value. | | t_curr | Current timestamp | | |||
| t_last | Last time sending/receiving a feedback | | ||||
| delta | Observed interval between current and previous | | ||||
| | feedback reports: delta = t_curr-t_last | | ||||
| r_n | Reference rate based on network congestion | | ||||
| r_send | Sending rate | | ||||
| r_recv | Receiving rate | | ||||
| r_vin | Target rate for video encoder | | ||||
| r_vout | Output rate from video encoder | | ||||
| d_base | Estimated baseline delay | | ||||
| d_fwd | Measured and filtered one-way delay | | ||||
| d_n | Estimated queueing delay | | ||||
| d_tilde | Equivalent delay after non-linear warping | | ||||
| p_mark | Estimated packet ECN marking ratio | | ||||
| p_loss | Estimated packet loss ratio | | ||||
| x_n | Aggregate congestion signal | | ||||
| x_prev | Previous value of aggregate congestion signal | | ||||
| x_diff | Change in aggregate congestion signal w.r.t. | | ||||
| | its previous value: x_diff = x_n - x_prev | | ||||
| rmode | Rate update mode: (0 = accelerated ramp-up; | | ||||
| | 1 = gradual update) | | ||||
| gamma | Rate increase multiplier in accelerated ramp-up | | ||||
| | mode | | ||||
| rtt | Estimated round-trip-time at sender | | ||||
| buffer_len | Rate shaping buffer occupancy measured in bytes | | ||||
+--------------+-------------------------------------------------+ | ||||
Correspondingly, the queuing delay experienced by the packet n is | Figure 2: List of variables. | |||
estimated as: | ||||
d_n = z_n - d_f. | +---------------+---------------------------------+----------------+ | |||
| Notation | Parameter Name | Default Value | | ||||
+--------------+----------------------------------+----------------+ | ||||
| PRIO | Weight of priority of the flow | 1.0 | ||||
| RMIN | Minimum rate of application | 150 Kbps | | ||||
| | supported by media encoder | | | ||||
| RMAX | Maximum rate of application | 1.5 Mbps | | ||||
| | supported by media encoder | | | ||||
| X_REF | Reference congestion level | 20ms | | ||||
| KAPPA | Scaling parameter for gradual | 0.5 | | ||||
| | rate update calculation | | | ||||
| ETA | Scaling parameter for gradual | 2.0 | | ||||
| | rate update calculation | | | ||||
| TAU | Upper bound of RTT in gradual | 500ms | | ||||
| | rate update calculation | | | ||||
| DELTA | Target feedback interval | 100ms | | ||||
| LOGWIN | Observation window in time for | 500ms | | ||||
| | calculating packet summary | | | ||||
| | statistics at receiver | | | ||||
| QEPS | Threshold for determining queuing| 10ms | | ||||
| | delay build up at receiver | | | ||||
+..............+..................................+................+ | ||||
| QTH | Delay threshold for non-linear | 100ms | | ||||
| | warping | | | ||||
| QMAX | Delay upper bound for non-linear | 400ms | | ||||
| | warping | | | ||||
| DLOSS | Delay penalty for loss | 1.0s | | ||||
| DMARK | Delay penalty for ECN marking | 200ms | | ||||
+..............+..................................+................+ | ||||
| GAMMA_MAX | Upper bound on rate increase | 20% | | ||||
| | ratio for accelerated ramp-up | | | ||||
| QBOUND | Upper bound on self-inflicted | 50ms | | ||||
| | queuing delay during ramp up | | | ||||
+..............+..................................+................+ | ||||
| FPS | Frame rate of incoming video | 30 | | ||||
| BETA_S | Scaling parameter for modulating | 0.1 | | ||||
| | outgoing sending rate | | | ||||
| BETA_V | Scaling parameter for modulating | 0.1 | | ||||
| | video encoder target rate | | | ||||
| ALPHA | Smoothing factor in exponential | 0.1 | | ||||
| | smoothing of packet loss and | | | ||||
| | marking ratios | | ||||
+--------------+----------------------------------+----------------+ | ||||
The individual sample values of queuing delay should be further filtered | Figure 3: List of algorithm parameters. | |||
against various non-congestion-induced noise, such as spikes due to | ||||
processing "hiccup" at the network nodes. We denote the resulting | ||||
queuing delay value as d_hat_n. | ||||
Our current implementation employs a simple 5-point median filter over | 4.2. Receiver-Side Algorithm | |||
per-packet queuing delay estimates, followed by an exponential smoothing | ||||
filter. We have found such relatively simple treatment to suffice in | ||||
guarding against processing delay outliers observed in wired | ||||
connections. For wireless connections with a higher packet delay | ||||
variation (PDV), more sophisticated techniques on de-noising, outlier | ||||
rejection, and trend analysis may be needed. | ||||
Like other delay-based congestion control schemes, performance of NADA | The receiver-side algorithm can be outlined as below: | |||
depends on the accuracy of its delay measurement and estimation module. | ||||
Appendix A in [RFC6817] provides an extensive discussion on this aspect. | ||||
4.2 Estimation of packet loss/marking ratio | On initialization: | |||
set d_base = +INFINITY | ||||
set p_loss = 0 | ||||
set p_mark = 0 | ||||
set r_recv = 0 | ||||
set both t_last and t_curr as current time | ||||
The receiver detects packet losses via gaps in the RTP sequence numbers | On receiving a media packet: | |||
of received packets. It then calculates instantaneous packet loss ratio | obtain current timestamp t_curr | |||
as the ratio between the number of missing packets over the number of | obtain from packet header sending time stamp t_sent | |||
total transmitted packets in the given time window (e.g., during the | obtain one-way delay measurement: d_fwd = t_curr - t_sent | |||
most recent 500ms). This instantaneous value is passed over an | update baseline delay: d_base = min(d_base, d_fwd) | |||
exponential smoothing filter, and the filtered result is reported back | update queuing delay: d_n = d_fwd - d_base | |||
to the sender as the observed packet loss ratio p_L. | update packet loss ratio estimate p_loss | |||
update packet marking ratio estimate p_mark | ||||
update measurement of receiving rate r_recv | ||||
We note that more sophisticated methods in packet loss ratio | On time to send a new feedback report (t_curr - t_last > DELTA): | |||
calculation, such as that adopted by TFRC [Floyd-CCR00], will likely be | calculate non-linear warping of delay d_tilde if packet loss exists | |||
beneficial. These alternatives are currently under investigation. | calculate aggregate congestion signal x_n | |||
determine mode of rate adaptation for sender: rmode | ||||
send RTCP feedback report containing values of: rmode, x_n, and r_recv | ||||
update t_last = t_curr | ||||
Estimation of packet marking ratio p_M, when ECN is enabled at | In order for a delay-based flow to hold its ground when competing | |||
bottleneck network nodes along the path, will follow the same procedure | against loss-based flows (e.g., loss-based TCP), it is important to | |||
as above. Here it is assumed that ECN marking information at the IP | distinguish between different levels of observed queuing delay. For | |||
header are somehow passed along to the transport layer by the receiving | instance, a moderate queuing delay value below 100ms is likely self- | |||
endpoint. | inflicted or induced by other delay-based flows, whereas a high | |||
queuing delay value of several hundreds of milliseconds may indicate | ||||
the presence of a loss-based flow that does not refrain from | ||||
increased delay. | ||||
4.3 Non-linear warping of delay | When packet losses are observed, the estimated queuing delay follows | |||
a non-linear warping inspired by the delay-adaptive congestion window | ||||
backoff policy in [Budzisz-TON11]: | ||||
In order for a delay-based flow to hold its ground and sustain a | / d_n, if d_n<QTH; | |||
reasonable share of bandwidth in the presence of a loss-based flow | | | |||
(e.g., loss-based TCP), it is important to distinguish between different | | (QMAX - d_n)^4 | |||
levels of observed queuing delay. For instance, a moderate queuing delay | d_tilde = < QTH ----------------, if QTH<d_n<QMAX (1) | |||
value below 100ms is likely self-inflicted or induced by other delay- | | (QMAX - QTH)^4 | |||
based flows, whereas a high queuing delay value of several hundreds of | | | |||
milliseconds may indicate the presence of a loss-based flow that does | \ 0, otherwise. | |||
not refrain from increased delay. | ||||
Inspired by the delay-adaptive congestion window backoff policy in | Here, the queuing delay value is unchanged when it is below the first | |||
[Budzisz-TON11] -- the work by itself is a window-based congestion | threshold QTH; it is scaled down following a non-linear curve when | |||
control scheme with fair coexistence with TCP -- we devise the following | its value falls between QTH and QMAX; above QMAX, the high queuing | |||
non-linear warping of estimated queuing delay value: | delay value no longer counts toward congestion control. | |||
d_tilde_n = (d_hat_n), if d_hat_n < d_th; | The aggregate congestion signal is: | |||
(d_max - d_hat_n)^4 | x_n = d_tilde + p_mark*DMARK + p_loss*DLOSS. (2) | |||
d_tilde_n = d_th --------------------, if d_th<d_hat_n<d_max; | ||||
(d_max - d_th)^4 | ||||
d_tilde_n = 0, if d_hat_n > d_max. | Here, DMARK is prescribed delay penalty associated with ECN markings | |||
and DLOSS is prescribed delay penalty associated with packet losses. | ||||
The value of DLOSS and DMARK does not depend on configurations at the | ||||
network node, but does assume that ECN markings, when available, | ||||
occur before losses. Furthermore, the values of DLOSS and DMARK need | ||||
to be set consistently across all NADA flows for them to compete | ||||
fairly. | ||||
Here, the queuing delay value is unchanged when it is below the first | In the absence of packet marking and losses, the value of x_n reduces | |||
threshold d_th; it is discounted following a non-linear curve when its | to the observed queuing delay d_n. In that case the NADA algorithm | |||
value falls between d_th and d_max; above d_max, the high queuing delay | operates in the regime of delay-based adaptation. | |||
value no longer counts toward congestion control. | ||||
When queuing delay is in the range (0, d_th), NADA operates in pure | Given observed per-packet delay and loss information, the receiver is | |||
delay-based mode if no losses/markings are present. When queuing delay | also in a good position to determine whether the network is | |||
exceeds d_max, NADA reacts to loss/marking only. In between d_th and | underutilized and recommend the corresponding rate adaptation mode | |||
d_max, the sending rate will converge and stabilize at an operating | for the sender. The criteria for operating in accelerated ramp-up | |||
point with a fairly high queuing delay and non-zero packet loss ratio. | mode are: | |||
In our current implementation d_th is chosen as 50ms and d_max is chosen | o No recent packet losses within the observation window LOGWIN; and | |||
as 400ms. The impact of the choice of d_th and d_max will be | ||||
investigated in future work. | ||||
4.4 Aggregating congestion signals | o No build-up of queuing delay: d_fwd-d_base < QEPS for all previous | |||
delay samples within the observation window LOGWIN. | ||||
The receiver aggregates all three forms of congestion signal in terms of | Otherwise the algorithm operates in graduate update mode. | |||
an equivalent delay: | ||||
x_n = d_tilde_n + p_M*d_M + p_L*d_L, (1) | 4.3. Sender-Side Algorithm | |||
where d_M is a prescribed fictitious delay value associated with ECN | The sender-side algorithm is outlined as follows: | |||
markings (e.g., d_M = 200 ms), and d_L is a prescribed fictitious delay | ||||
value associated with packet losses (e.g., d_L = 1 second). By | ||||
introducing a large fictitious delay penalty for ECN marking and packet | ||||
loss, the proposed scheme leads to low end-to-end actual delay in the | ||||
presence of such events. | ||||
While the value of d_M and d_L are fixed and predetermined in the | on initialization: | |||
current design, a scheme for automatically tuning these values based on | set r_n = RMIN | |||
desired bandwidth sharing behavior in the presence of other competing | set rtt = 0 | |||
loss-based flows (e.g., loss-based TCP) is being studied. | set x_prev = 0 | |||
set t_last and t_curr as current time | ||||
In the absence of ECN marking from the network, the value of x_n falls | on receiving feedback report: | |||
back to the observed queuing delay d_n for packet n when queuing delay | obtain current timestamp: t_curr | |||
is low and no packets are lost over a lightly congested path. In that | obtain values of rmode, x_n, and r_recv from feedback report | |||
case the algorithm operates in purely delay-based mode. | update estimation of rtt | |||
measure feedback interval: delta = t_curr - t_last | ||||
if rmode == 0: | ||||
update r_n following accelerated ramp-up rules | ||||
else: | ||||
update r_n following gradual update rules | ||||
clip rate r_n within the range of [RMIN, RMAX] | ||||
x_prev = x_n | ||||
t_last = t_curr | ||||
4.5 Estimating receiving rate | In accelerated ramp-up mode, the rate r_n is updated as follows: | |||
Estimation of receiving rate of the flow is fairly straightforward. NADA | QBOUND | |||
maintains a recent observation window of 500ms, and simply divides the | gamma = min(GAMMA_MAX, -----------) (3) | |||
total size of packets arriving during that window over the time span. | rtt+DELTA | |||
The receiving rate is denoted as R_r. | ||||
4.6 Sending periodic feedback | r_n = (1+gamma) r_recv (4) | |||
Periodically, the receiver feeds back a tuple of the most recent values | The rate increase multiplier gamma is calculated as a function of | |||
of <d_hat_n, x_n, x'_n, R_r> in RTCP feedback messages to aid the sender | upper bound of self-inflicted queuing delay (QBOUND), round-trip-time | |||
in its calculation of target rate. The queuing delay value d_hat_n is | (rtt), and target feedback interval DELTA. It has a maximum value of | |||
included along with the composite congestion signal x_n so that the | GAMMA_MAX. The rationale behind (3)-(4) is that the longer it takes | |||
sender can decide whether the network is truly underutilized (see Sec. | for the sender to observe self-inflicted queuing delay build-up, the | |||
6.1.1 Accelerated ramp-up). | more conservative the sender should be in increasing its rate, hence | |||
the smaller the rate increase multiplier. | ||||
The value of x'_n corresponds to the derivative (i.e., rate of change) | In gradual update mode, the rate r_n is updated as: | |||
of the composite congestion signal: | ||||
x_n - x_(n-k) | x_offset = x_n - PRIO*X_REF*RMAX/r_n (5) | |||
x'_n = ---------------, (2) | ||||
delta | ||||
where the interval between consecutive RTCP feedback messages is denoted | x_diff = x_n - x_prev (6) | |||
as delta. The packet indices corresponding to the current and previous | ||||
feedback are n and (n-k), respectively. | ||||
The choice of target feedback interval needs to strike the right balance | delta x_offset | |||
between timely feedback and low RTCP feedback message counts. Through | r_n = r_n - KAPPA*-------*------------*r_n | |||
simulation studies and frequency-domain analysis, it was determined that | TAU TAU | |||
a feedback interval below 250ms will not break up the feedback control | ||||
loop of the NADA congestion control algorithm. Thus, it is recommended | ||||
to use a target feed interval of 100ms. This will result in a feedback | ||||
bandwidth of 16Kbps with 200 bytes per feedback message, less than 0.1% | ||||
overhead for a 1Mbps flow. | ||||
4.7 Discussions on delay metrics | x_diff | |||
- KAPPA*ETA*---------*r_n (7) | ||||
TAU | ||||
The current design works with relative one-way-delay (OWD) as the main | The rate changes in proportion to the previous rate decision. It is | |||
indication of congestion. The value of the relative OWD is obtained by | affected by two terms: offset of the aggregate congestion signal from | |||
maintaining the minimum value of observed OWD over a relatively long | its value at equilibrium (x_offset) and its change (x_diff). | |||
time horizon and subtract that out from the observed absolute OWD value. | Calculation of x_offset depends on maximum rate of the flow (RMAX), | |||
Such an approach cancels out the fixed difference between the sender and | its weight of priority (PRIO), as well as a reference congestion | |||
receiver clocks. It has been widely adopted by other delay-based | signal (X_REF). The value of X_REF is chosen that the maximum rate | |||
congestion control approaches such as LEDBAT [RFC6817]. As discussed in | of RMAX can be achieved when the observed congestion signal level is | |||
[RFC6817], the time horizon for tracking the minimum OWD needs to be | below PRIO*X_REF. | |||
chosen with care: it must be long enough for an opportunity to observe | ||||
the minimum OWD with zero queuing delay along the path, and sufficiently | ||||
short so as to timely reflect "true" changes in minimum OWD introduced | ||||
by route changes and other rare events. | ||||
The potential drawback in relying on relative OWD as the congestion | At equilibrium, the aggregated congestion signal stablizes at x_n = | |||
signal is that when multiple flows share the same bottleneck, the flow | PRIO*X_REF*RMAX/r_n. This ensures that when multiple flows share the | |||
arriving late at the network experiencing a non-empty queue may | same bottleneck and observe a common value of x_n, their rates at | |||
mistakenly consider the standing queuing delay as part of the fixed path | equilibrium will be proportional to their respective priority levels | |||
propagation delay. This will lead to slightly unfair bandwidth sharing | (PRIO) and maximum rate (RMAX). | |||
among the flows. | ||||
Alternatively, one could move the per-packet statistical handling to the | As mentioned in the sender-side algorithm, the final rate is clipped | |||
sender instead and use RTT in lieu of OWD, assuming that per-packet ACKs | within the dynamic range specified by the application: | |||
are available. The main drawback of this latter approach is that the | ||||
scheme will be confused by congestion in the reverse direction. | ||||
Note that the choice of either delay metric (relative OWD vs. RTT) | r_n = min(r_n, RMAX) (8) | |||
involves no change in the proposed rate adaptation algorithm at the | ||||
sender. Therefore, comparing the pros and cons regarding which delay | ||||
metric to adopt can be kept as an orthogonal direction of | ||||
investigation. | ||||
5. NADA Sender Behavior | r_n = max(r_n, RMIN) (9) | |||
Figure 1 provides a detailed view of the NADA sender. Upon receipt of an | The above operations ignore many practical issues such as clock | |||
RTCP report from the receiver, the NADA sender updates its calculation | synchronization between sender and receiver, filtering of noise in | |||
of the reference rate R_n. It further adjusts both the target rate for | delay measurements, and base delay expiration. These will be | |||
the live video encoder R_v and the sending rate R_s over the network | addressed in later sections describing practical implementation of | |||
based on the updated value of R_n, as well as the size of the rate | the NADA algorithm. | |||
shaping buffer. | ||||
In the following, we describe these modules in further detail, and | 5. Practical Implementation of NADA | |||
explain how they interact with each other. | ||||
-------------------- | 5.1. Receiver-Side Operation | |||
| | | ||||
| Reference Rate | <---- RTCP report | ||||
| Calculator | | ||||
| | | ||||
-------------------- | ||||
| | ||||
| R_n | ||||
| | ||||
-------------------------- | ||||
| | | ||||
| | | ||||
\ / \ / | ||||
-------------------- ----------------- | ||||
| | | | | ||||
| Video Target | | Sending Rate | | ||||
| Rate Calculator | | Calculator | | ||||
| | | | | ||||
-------------------- ----------------- | ||||
| /|\ /|\ | | ||||
R_v| | | | | ||||
| ----------------------- | | ||||
| | | R_s | ||||
------------ |L_s | | ||||
| | | | | ||||
| | R_o -------------- \|/ | ||||
| Encoder |----------> | | | | | ---------------> | ||||
| | | | | | | video packets | ||||
------------ -------------- | ||||
Rate Shaping Buffer | ||||
Figure 1 NADA Sender Structure | The receiver continuously monitors end-to-end per-packet statistics | |||
in terms of delay, loss, and/or ECN marking ratios. It then | ||||
aggregates all forms of congestion indicators into the form of an | ||||
equivalent delay and periodically reports this back to the sender. | ||||
5.1 Reference rate calculation | In addition, the receiver tracks the receiving rate of the flow and | |||
includes that in the feedback message. | ||||
The sender initializes the reference rate R_n as R-min by default, or to | 5.1.1. Estimation of one-way delay and queuing delay | |||
a value specified by the upper-layer application. [Editor's note: should | ||||
proper choice of starting rate value be within the scope of the CC | ||||
solution? ] | ||||
The reference rate R_n is calculated based on receiver feedback | The delay estimation process in NADA follows a similar approach as in | |||
information regarding queuing delay d_tilde_n, composite congestion | earlier delay-based congestion control schemes, such as LEDBAT | |||
signal x_n, its derivative x'_n, as well as the receiving rate R_r. The | [RFC6817]. NADA estimates the forward delay as having a constant | |||
sender switches between two modes of operation: | base delay component plus a time varying queuing delay component. | |||
The base delay is estimated as the minimum value of one-way delay | ||||
observed over a relatively long period (e.g., tens of minutes), | ||||
whereas the individual queuing delay value is taken to be the | ||||
difference between one-way delay and base delay. | ||||
* Accelerated ramp up: if the reported queuing delay is close to | The individual sample values of queuing delay should be further | |||
zero and both values of x_n and x'_n are close to zero, | filtered against various non-congestion-induced noise, such as spikes | |||
indicating empty queues along the path of the flow and, | due to processing "hiccup" at the network nodes. Current | |||
consequently, underutilized network bandwidth; or | implementation employs a 15-tab minimum filter over per-packet | |||
queuing delay estimates. | ||||
* Gradual rate update: in all other conditions, whereby the | 5.1.2. Estimation of packet loss/marking ratio | |||
receiver reports on a standing or increasing/decreasing queue | ||||
and/or composite congestion signal. | ||||
5.1.1 Accelerated ramp up | The receiver detects packet losses via gaps in the RTP sequence | |||
numbers of received packets. Packets arriving out-of-order are | ||||
discarded, and count towards losses. The instantaneous packet loss | ||||
ratio p_inst is estimated as the ratio between the number of missing | ||||
packets over the number of total transmitted packets within the | ||||
recent observation window LOGWIN. The packet loss ratio p_loss is | ||||
obtained after exponential smoothing: | ||||
In the absence of a non-zero congestion signal to guide the sending rate | p_loss = ALPHA*p_inst + (1-ALPHA)*p_loss. (10) | |||
calculation, the sender needs to ramp up its estimated bandwidth as | ||||
quickly as possible without introducing excessive queuing delay. Ideally | ||||
the flow should inflict no more than T_th milliseconds of queuing delay | ||||
at the bottleneck during the ramp-up process. A typical value of T_th is | ||||
50ms. | ||||
Note that the sender will be aware of any queuing delay introduced by | The filtered result is reported back to the sender as the observed | |||
its rate increase after at least one round-trip time. In addition, the | packet loss ratio p_loss. | |||
bottleneck bandwidth C is greater than or equal to the receive rate R_r | ||||
reported from the most recent "no congestion" feedback message. The rate | ||||
R_n is updated as follows: | ||||
T_th | Estimation of packet marking ratio p_mark follows the same procedure | |||
gamma = min [gamma_0, ---------------] (3) | as above. It is assumed that ECN marking information at the IP | |||
RTT_0+delta_0 | header can be passed to the transport layer by the receiving | |||
endpoint. | ||||
R_n = (1+gamma) R_r (4) | 5.1.3. Estimation of receiving rate | |||
In (3) and (4), the multiplier gamma for rate increase is upper-bounded | It is fairly straighforward to estimate the receiving rate r_recv. | |||
by a fixed ratio gamma_0 (e.g., 20%), as well as a ratio which depends | NADA maintains a recent observation window with time span of LOGWIN, | |||
on T_th, base RTT as measured during the non-congested phase, and target | and simply divides the total size of packets arriving during that | |||
ACK interval delta_0. The rationale behind this is that the rate | window over the time span. The receiving rate (r_recv) is included | |||
increase multiplier should decrease with the delay in the feedback | as part of the feedback report. | |||
control loop, and that RTT_0 + delta_0 provides a worst-case estimate of | ||||
feedback control delay when the network is not congested. | ||||
5.1.2. Gradual rate update | 5.2. Sender-Side Operation | |||
When the receiver reports indicate a standing congestion level, NADA | Figure 4 provides a detailed view of the NADA sender. Upon receipt | |||
operates in gradual update mode, and calculates its reference rate as: | of an RTCP feedback report from the receiver, the NADA sender | |||
calculates the reference rate r_n as specified in Section 4.3. It | ||||
further adjusts both the target rate for the live video encoder r_vin | ||||
and the sending rate r_send over the network based on the updated | ||||
value of r_n and rate shaping buffer occupancy buffer_len. | ||||
kappa * delta_s | The NADA sender behavior stays the same in the presence of all types | |||
R_n <-- R_n + ---------------- * (theta-(R_n-R_min)*x_hat) (5) | of congestion indicators: delay, loss, and ECN marking. This unified | |||
tau_o^2 | approach allows a graceful transition of the scheme as the network | |||
shifts dynamically between light and heavy congestion levels. | ||||
where | +----------------+ | |||
| Calculate | <---- RTCP report | ||||
| Reference Rate | | ||||
-----------------+ | ||||
| r_n | ||||
+------------+-------------+ | ||||
| | | ||||
\|/ \|/ | ||||
+-----------------+ +---------------+ | ||||
| Calculate Video | | Calculate | | ||||
| Target Rate | | Sending Rate | | ||||
+-----------------+ +---------------+ | ||||
| /|\ /|\ | | ||||
r_vin | | | | | ||||
\|/ +-------------------+ | | ||||
+----------+ | buffer_len | r_send | ||||
| Video | r_vout -----------+ \|/ | ||||
| Encoder |--------> |||||||||=================> | ||||
+----------+ -----------+ RTP packets | ||||
Rate Shaping Buffer | ||||
theta = w*(R_max - R_min)*x_ref. (6) | Figure 4: NADA Sender Structure | |||
x_hat = x_n + eta*tau_o* x'_n (7) | 5.2.1. Rate shaping buffer | |||
In (5), delta_s refers to the time interval between current and previous | The operation of the live video encoder is out of the scope of the | |||
rate updates. Note that delta_s is the same as the RTCP report interval | design for the congestion control scheme in NADA. Instead, its | |||
at the receiver (see delta from (2)) when the backward path is un- | behavior is treated as a black box. | |||
congested. | ||||
In (6), R_min and R_max denote the content-dependent rate range the | A rate shaping buffer is employed to absorb any instantaneous | |||
encoder can produce. The weighting factor reflecting a flow's priority | mismatch between encoder rate output r_vout and regulated sending | |||
is w. The reference congestion signal x_ref is chosen so that the | rate r_send. Its current level of occupancy is measured in bytes and | |||
maximum rate of R_max can be achieved when x_hat = w*x_ref. | is denoted as buffer_len. | |||
Proper choice of the scaling parameters eta and kappa in (5) and (7) can | A large rate shaping buffer contributes to higher end-to-end delay, | |||
ensure system stability so long as the RTT falls below the upper bound | which may harm the performance of real-time media communications. | |||
of tau_o. The recommended default value of tau_o is chosen as 500ms. | Therefore, the sender has a strong incentive to prevent the rate | |||
shaping buffer from building up. The mechanisms adopted are: | ||||
For both modes of operations, the final reference rate R_n is clipped | o To deplete the rate shaping buffer faster by increasing the | |||
within the range of [R_min, R_max]. Note also that the sender does not | sending rate r_send; and | |||
need any explicit knowledge of the management scheme inside the network. | ||||
Rather, it reacts to the aggregation of all forms of congestion | ||||
indications (delay, loss, and explicit markings) via the composite | ||||
congestion signals x_n and x'_n from the receiver in a coherent manner. | ||||
5.2 Video encoder rate control | o To limit incoming packets of the rate shaping buffer by reducing | |||
the video encoder target rate r_vin. | ||||
The video encoder rate control procedure has the following | 5.2.2. Adjusting video target rate and sending rate | |||
characteristics: | ||||
* Rate changes can happen only at large intervals, on the order of | The target rate for the live video encoder deviates from the network | |||
seconds. | congestion control rate r_n based on the level of occupancy in the | |||
rate shaping buffer: | ||||
* The encoder output rate may fluctuate around the target rate R_v. | r_vin = r_n - BETA_V*8*buffer_len*FPS. (11) | |||
* The encoder output rate is further constrained by video content | The actual sending rate r_send is regulated in a similar fashion: | |||
complexity. The range of the final rate output is [R_min, R_max]. | ||||
Note that it is content-dependent and may vary over time. | ||||
The operation of the live video encoder is out of the scope of the | r_send = r_n + BETA_S*8*buffer_len*FPS. (12) | |||
design for the congestion control scheme in NADA. Instead, its behavior | ||||
is treated as a black box. | ||||
5.3 Rate shaping buffer | In (11) and (12), the first term indicates the rate calculated from | |||
network congestion feedback alone. The second term indicates the | ||||
influence of the rate shaping buffer. A large rate shaping buffer | ||||
nudges the encoder target rate slightly below -- and the sending rate | ||||
slightly above -- the reference rate r_n. | ||||
A rate shaping buffer is employed to absorb any instantaneous mismatch | Intuitively, the amount of extra rate offset needed to completely | |||
between encoder rate output R_o and regulated sending rate R_s. The size | drain the rate shaping buffer within the duration of a single video | |||
of the buffer evolves from time t-tau to time t as: | frame is given by 8*buffer_len*FPS, where FPS stands for the frame | |||
rate of the video. The scaling parameters BETA_V and BETA_S can be | ||||
tuned to balance between the competing goals of maintaining a small | ||||
rate shaping buffer and deviating the system from the reference rate | ||||
point. | ||||
L_s(t) = max [0, L_s(t-tau)+(R_o-R_s)*tau]. | 6. Discussions and Further Investigations | |||
A large rate shaping buffer contributes to higher end-to-end delay, | 6.1. Choice of delay metrics | |||
which may harm the performance of real-time media communications. | ||||
Therefore, the sender has a strong incentive to constrain the size of | ||||
the shaping buffer. It can either deplete it faster by increasing the | ||||
sending rate R_s, or limit its growth by reducing the target rate for | ||||
the video encoder rate control R_v. | ||||
5.4 Adjusting video target rate and sending rate | The current design works with relative one-way-delay (OWD) as the | |||
main indication of congestion. The value of the relative OWD is | ||||
obtained by maintaining the minimum value of observed OWD over a | ||||
relatively long time horizon and subtract that out from the observed | ||||
absolute OWD value. Such an approach cancels out the fixed | ||||
difference between the sender and receiver clocks. It has been | ||||
widely adopted by other delay-based congestion control approaches | ||||
such as [RFC6817]. As discussed in [RFC6817], the time horizon for | ||||
tracking the minimum OWD needs to be chosen with care: it must be | ||||
long enough for an opportunity to observe the minimum OWD with zero | ||||
queuing delay along the path, and sufficiently short so as to timely | ||||
reflect "true" changes in minimum OWD introduced by route changes and | ||||
other rare events. | ||||
The target rate for the live video encoder is updated based on both the | The potential drawback in relying on relative OWD as the congestion | |||
reference rate R_n and the rate shaping buffer size L_s, as follows: | signal is that when multiple flows share the same bottleneck, the | |||
flow arriving late at the network experiencing a non-empty queue may | ||||
mistakenly consider the standing queuing delay as part of the fixed | ||||
path propagation delay. This will lead to slightly unfair bandwidth | ||||
sharing among the flows. | ||||
L_s | Alternatively, one could move the per-packet statistical handling to | |||
R_v = R_n - beta_v * -------. (8) | the sender instead and use relative round-trip-time (RTT) in lieu of | |||
tau_v | relative OWD, assuming that per-packet acknowledgements are | |||
available. The main drawback of RTT-based approach is the noise in | ||||
the measured delay in the reverse direction. | ||||
Similarly, the outgoing rate is regulated based on both the reference | Note that the choice of either delay metric (relative OWD vs. RTT) | |||
rate R_n and the rate shaping buffer size L_s, such that: | involves no change in the proposed rate adaptation algorithm. | |||
Therefore, comparing the pros and cons regarding which delay metric | ||||
to adopt can be kept as an orthogonal direction of investigation. | ||||
L_s | 6.2. Method for delay, loss, and marking ratio estimation | |||
R_s = R_n + beta_s * -------. (9) | ||||
tau_v | ||||
In (8) and (9), the first term indicates the rate calculated from | Like other delay-based congestion control schemes, performance of | |||
network congestion feedback alone. The second term indicates the | NADA depends on the accuracy of its delay measurement and estimation | |||
influence of the rate shaping buffer. A large rate shaping buffer nudges | module. Appendix A in [RFC6817] provides an extensive discussion on | |||
the encoder target rate slightly below -- and the sending rate slightly | this aspect. | |||
above -- the reference rate R_n. | ||||
Intuitively, the amount of extra rate offset needed to completely drain | The current recommended practice of simply applying a 15-tab minimum | |||
the rate shaping buffer within the same time frame of encoder rate | filter suffices in guarding against processing delay outliers | |||
adaptation tau_v is given by L_s/tau_v. The scaling parameters beta_v | observed in wired connections. For wireless connections with a | |||
and beta_s can be tuned to balance between the competing goals of | higher packet delay variation (PDV), more sophisticated techniques on | |||
maintaining a small rate shaping buffer and deviating the system from | de-noising, outlier rejection, and trend analysis may be needed. | |||
the reference rate point. | ||||
6. Incremental Deployment | More sophisticated methods in packet loss ratio calculation, such as | |||
that adopted by [Floyd-CCR00], will likely be beneficial. These | ||||
alternatives are currently under investigation. | ||||
One nice property of NADA is the consistent video endpoint behavior | 6.3. Impact of parameter values | |||
irrespective of network node variations. This facilitates gradual, | ||||
incremental adoption of the scheme. | ||||
To start off with, the encoder congestion control mechanism can be | In the gradual rate update mode, the parameter TAU indicates the | |||
implemented without any explicit support from the network, and relies | upper bound of round-trip-time (RTT) in feedback control loop. | |||
solely on observed one-way delay measurements and packet loss ratios as | Typically, the observed feedback interval delta is close to the | |||
implicit congestion signals. | target feedback interval DELTA, and the relative ratio of delta/TAU | |||
versus ETA dictates the relative strength of influence from the | ||||
aggregate congestion signal offset term (x_offset) versus its recent | ||||
change (x_diff), respectively. These two terms are analogous to the | ||||
integral and proportional terms in a proportional-integral (PI) | ||||
controller. The recommended choice of TAU=500ms, DELTA=100ms and ETA | ||||
= 2.0 corresponds to a relative ratio of 1:10 between the gains of | ||||
the integral and proportional terms. Consequently, the rate | ||||
adaptation is mostly driven by the change in the congestion signal | ||||
with a long-term shift towards its equilibrium value driven by the | ||||
offset term. Finally, the scaling parameter KAPPA determines the | ||||
overall speed of the adaptation and needs to strike a balance between | ||||
responsiveness and stability. | ||||
When ECN is enabled at the network nodes with RED-based marking, the | The choice of the target feedback interval DELTA needs to strike the | |||
receiver can fold its observations of ECN markings into the calculation | right balance between timely feedback and low RTCP feedback message | |||
of the equivalent delay. The sender can react to these explicit | counts. A target feedback interval of DELTA=100ms is recommended, | |||
congestion signals without any modification. | corresponding to a feedback bandwidth of 16Kbps with 200 bytes per | |||
feedback message --- less than 0.1% overhead for a 1 Mbps flow. | ||||
Furthermore, both simulation studies and frequency-domain analysis | ||||
have established that a feedback interval below 250ms will not break | ||||
up the feedback control loop of NADA congestion control. | ||||
Ultimately, networks equipped with proactive marking based on token | In calculating the non-linear warping of delay in (1), the current | |||
bucket level metering can reap the additional benefits of zero standing | design uses fixed values of QTH and QMAX. It is possible to adapt | |||
queues and lower end-to-end delay and work seamlessly with existing | the value of both based on past observations of queuing delay in the | |||
senders and receivers. | presence of packet losses. | |||
7. Implementation Status | In calculating the aggregate congestion signal x_n, the choice of | |||
DMARK and DLOSS influence the steady-state packet loss/marking ratio | ||||
experienced by the flow at a given available bandwidth. Higher | ||||
values of DMARK and DLOSS result in lower steady-state loss/marking | ||||
ratios, but are more susceptible to the impact of individual packet | ||||
loss/marking events. While the value of DMARK and DLOSS are fixed | ||||
and predetermined in the current design, a scheme for automatically | ||||
tuning these values based on desired bandwidth sharing behavior in | ||||
the presence of other competing loss-based flows (e.g., loss-based | ||||
TCP) is under investigation. | ||||
The NADA scheme has been implemented in the ns-2 simulation platform | [Editor's note: Choice of start value: is this in scope of congestion | |||
[ns2]. Extensive simulation evaluations of an earlier version of the | control, or should this be decided by the application?] | |||
draft are documented in [Zhu-PV13]. Evaluation results of the current | ||||
draft over several test cases in [I-D.draft-sarker-rmcat-eval-test] have | ||||
been presented at recent IETF meetings [IETF-90][IETF-91]. | ||||
The scheme has also been implemented and evaluated in a lab setting as | 6.4. Sender-based vs. receiver-based calculation | |||
described in [IETF-90]. Preliminary evaluation results of NADA in | ||||
single-flow and multi-flow scenarios have been presented in [IETF-91]. | ||||
8. IANA Considerations | In the current design, the aggregated congestion signal x_n is | |||
calculated at the receiver, keeping the sender operation completely | ||||
independent of the form of actual network congestion indications | ||||
(delay, loss, or marking). Alternatively, one can move the logics of | ||||
(1) and (2) to the sender. Such an approach requires slightly higher | ||||
overhead in the feedback messages, which should contain individual | ||||
fields on queuing delay (d_n), packet loss ratio (p_loss), packet | ||||
marking ratio (p_mark), receiving rate (r_recv), and recommended rate | ||||
adaptation mode (rmode). | ||||
There are no actions for IANA. | 6.5. Incremental deployment | |||
9. References | One nice property of NADA is the consistent video endpoint behavior | |||
irrespective of network node variations. This facilitates gradual, | ||||
incremental adoption of the scheme. | ||||
9.1 Normative References | To start off with, the proposed congestion control mechanism can be | |||
implemented without any explicit support from the network, and relies | ||||
solely on observed one-way delay measurements and packet loss ratios | ||||
as implicit congestion signals. | ||||
When ECN is enabled at the network nodes with RED-based marking, the | ||||
receiver can fold its observations of ECN markings into the | ||||
calculation of the equivalent delay. The sender can react to these | ||||
explicit congestion signals without any modification. | ||||
Ultimately, networks equipped with proactive marking based on token | ||||
bucket level metering can reap the additional benefits of zero | ||||
standing queues and lower end-to-end delay and work seamlessly with | ||||
existing senders and receivers. | ||||
7. Implementation Status | ||||
The NADA scheme has been implemented in [ns-2] and [ns-3] simulation | ||||
platforms. Extensive ns-2 simulation evaluations of an earlier | ||||
version of the draft are documented in [Zhu-PV13]. Evaluation | ||||
results of the current draft over several test cases in | ||||
[I-D.ietf-rmcat-eval-test] have been presented at recent IETF | ||||
meetings [IETF-90][IETF-91]. | ||||
The scheme has also been implemented and evaluated in a lab setting | ||||
as described in [IETF-90]. Preliminary evaluation results of NADA in | ||||
single-flow and multi-flow scenarios have been presented in | ||||
[IETF-91]. | ||||
8. IANA Considerations | ||||
This document makes no request of IANA. | ||||
9. Acknowledgements | ||||
The authors would like to thank Randell Jesup, Luca De Cicco, Piers | ||||
O'Hanlon, Ingemar Johansson, Stefan Holmer, Cesar Ilharco Magalhaes, | ||||
Safiqul Islam, Mirja Kuhlewind, and Karen Elisabeth Egede Nielsen for | ||||
their valuable questions and comments on earlier versions of this | ||||
draft. | ||||
10. References | ||||
10.1. Normative References | ||||
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate | [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate | |||
Requirement Levels", BCP 14, RFC 2119, March 1997. | Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/ | |||
RFC2119, March 1997, | ||||
<http://www.rfc-editor.org/info/rfc2119>. | ||||
[RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition | [RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition | |||
of Explicit Congestion Notification (ECN) to IP", | of Explicit Congestion Notification (ECN) to IP", RFC | |||
RFC 3168, September 2001. | 3168, DOI 10.17487/RFC3168, September 2001, | |||
<http://www.rfc-editor.org/info/rfc3168>. | ||||
[RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V. | [RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V. | |||
Jacobson, "RTP: A Transport Protocol for Real-Time | Jacobson, "RTP: A Transport Protocol for Real-Time | |||
Applications", STD 64, RFC 3550, July 2003. | Applications", STD 64, RFC 3550, DOI 10.17487/RFC3550, | |||
July 2003, <http://www.rfc-editor.org/info/rfc3550>. | ||||
9.2 Informative References | [I-D.ietf-rmcat-eval-criteria] | |||
Singh, V. and J. Ott, "Evaluating Congestion Control for | ||||
Interactive Real-time Media", draft-ietf-rmcat-eval- | ||||
criteria-03 (work in progress), March 2015. | ||||
[RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition | [I-D.ietf-rmcat-eval-test] | |||
of Explicit Congestion Notification (ECN) to IP", | Sarker, Z., Singh, V., Zhu, X., and M. Ramalho, "Test | |||
RFC 3168, September 2001. | Cases for Evaluating RMCAT Proposals", draft-ietf-rmcat- | |||
eval-test-02 (work in progress), September 2015. | ||||
[I-D.ietf-rmcat-cc-requirements] | ||||
Jesup, R. and Z. Sarker, "Congestion Control Requirements | ||||
for Interactive Real-Time Media", draft-ietf-rmcat-cc- | ||||
requirements-09 (work in progress), December 2014. | ||||
10.2. Informative References | ||||
[RFC2309] Braden, B., Clark, D., Crowcroft, J., Davie, B., Deering, | [RFC2309] Braden, B., Clark, D., Crowcroft, J., Davie, B., Deering, | |||
S., Estrin, D., Floyd, S., Jacobson, V., Minshall, G., | S., Estrin, D., Floyd, S., Jacobson, V., Minshall, G., | |||
Partridge, C., Peterson, L., Ramakrishnan, K., Shenker, | Partridge, C., Peterson, L., Ramakrishnan, K., Shenker, | |||
S., Wroclawski, J., and L. Zhang, "Recommendations on | S., Wroclawski, J., and L. Zhang, "Recommendations on | |||
Queue Management and Congestion Avoidance in the | Queue Management and Congestion Avoidance in the | |||
Internet", RFC 2309, April 1998. | Internet", RFC 2309, DOI 10.17487/RFC2309, April 1998, | |||
<http://www.rfc-editor.org/info/rfc2309>. | ||||
[RFC6817] Shalunov, S., Hazel, G., Iyengar, J., and Kuehlewind, M., | [RFC5348] Floyd, S., Handley, M., Padhye, J., and J. Widmer, "TCP | |||
Friendly Rate Control (TFRC): Protocol Specification", RFC | ||||
5348, DOI 10.17487/RFC5348, September 2008, | ||||
<http://www.rfc-editor.org/info/rfc5348>. | ||||
[RFC6660] Briscoe, B., Moncaster, T., and M. Menth, "Encoding Three | ||||
Pre-Congestion Notification (PCN) States in the IP Header | ||||
Using a Single Diffserv Codepoint (DSCP)", RFC 6660, DOI | ||||
10.17487/RFC6660, July 2012, | ||||
<http://www.rfc-editor.org/info/rfc6660>. | ||||
[RFC6817] Shalunov, S., Hazel, G., Iyengar, J., and M. Kuehlewind, | ||||
"Low Extra Delay Background Transport (LEDBAT)", RFC 6817, | "Low Extra Delay Background Transport (LEDBAT)", RFC 6817, | |||
December 2012 | DOI 10.17487/RFC6817, December 2012, | |||
<http://www.rfc-editor.org/info/rfc6817>. | ||||
[Floyd-CCR00] Floyd, S., Handley, M., Padhye, J., and Widmer, J., | [Floyd-CCR00] | |||
Floyd, S., Handley, M., Padhye, J., and J. Widmer, | ||||
"Equation-based Congestion Control for Unicast | "Equation-based Congestion Control for Unicast | |||
Applications", ACM SIGCOMM Computer Communications Review, | Applications", ACM SIGCOMM Computer Communications Review | |||
vol. 30. no. 4., pp. 43-56, October 2000. | vol. 30, no. 4, pp. 43-56, October 2000. | |||
[Budzisz-TON11] Budzisz, L. et al., "On the Fair Coexistence of | ||||
Loss- and Delay-Based TCP", IEEE/ACM Transactions on | ||||
Networking, vol. 19, no. 6, pp. 1811-1824, December 2011. | ||||
[ns2] "The Network Simulator - ns-2", http://www.isi.edu/nsnam/ns/ | [Budzisz-TON11] | |||
Budzisz, L., Stanojevic, R., Schlote, A., Baker, F., and | ||||
R. Shorten, "On the Fair Coexistence of Loss- and Delay- | ||||
Based TCP", IEEE/ACM Transactions on Networking vol. 19, | ||||
no. 6, pp. 1811-1824, December 2011. | ||||
[Zhu-PV13] Zhu, X. and Pan, R., "NADA: A Unified Congestion Control | [Zhu-PV13] | |||
Zhu, X. and R. Pan, "NADA: A Unified Congestion Control | ||||
Scheme for Low-Latency Interactive Video", in Proc. IEEE | Scheme for Low-Latency Interactive Video", in Proc. IEEE | |||
International Packet Video Workshop (PV'13). San Jose, CA, | International Packet Video Workshop (PV'13) San Jose, CA, | |||
USA. December 2013. | USA, December 2013. | |||
[I-D.draft-sarker-rmcat-eval-test] Sarker, Z., Singh, V., Zhu, X., | [ns-2] "The Network Simulator - ns-2", | |||
and Ramalho, M., "Test Cases for Evaluating RMCAT | <http://www.isi.edu/nsnam/ns/>. | |||
Proposals", draft-sarker-rmcat-eval-test-01 (work in | ||||
progress), June 2014. | ||||
[IETF-90] Zhu, X. et al., "NADA Update: Algorithm, Implementation, | [ns-3] "The Network Simulator - ns-3", <https://www.nsnam.org/>. | |||
and Test Case Evalua6on Results", presented at IETF 90, | ||||
https://tools.ietf.org/agenda/90/slides/slides-90-rmcat- | ||||
6.pdf | ||||
[IETF-91] Zhu, X. et al., "NADA Algorithm Update and Test Case | [IETF-90] Zhu, X., Ramalho, M., Ganzhorn, C., Jones, P., and R. Pan, | |||
Evaluations", presented at IETF 91 Interium, | "NADA Update: Algorithm, Implementation, and Test Case | |||
https://datatracker.ietf.org/meeting/91/agenda/rmcat/ | Evalua6on Results", July 2014, | |||
<https://tools.ietf.org/agenda/90/slides/slides-90-rmcat- | ||||
6.pdf>. | ||||
Appendix A. Network Node Operations | [IETF-91] Zhu, X., Pan, R., Ramalho, M., Mena, S., Ganzhorn, C., | |||
Jones, P., and S. D'Aronco, "NADA Algorithm Update and | ||||
Test Case Evaluations", November 2014, | ||||
<http://www.ietf.org/proceedings/interim/2014/11/09/rmcat/ | ||||
slides/slides-interim-2014-rmcat-1-2.pdf>. | ||||
NADA can work with different network queue management | Appendix A. Network Node Operations | |||
schemes and does not assume any specific network node | ||||
operation. As an example, this appendix describes three | ||||
variations of queue management behavior at the network | ||||
node, leading to either implicit or explicit congestion | ||||
signals. | ||||
In all three flavors described below, the network queue | NADA can work with different network queue management schemes and | |||
operates with the simple first-in-first-out (FIFO) | does not assume any specific network node operation. As an example, | |||
principle. There is no need to maintain per-flow state. | this appendix describes three variants of queue management behavior | |||
Such a simple design ensures that the system can scale | at the network node, leading to either implicit or explicit | |||
easily with a large number of video flows and high link | congestion signals. | |||
capacity. | ||||
NADA sender behavior stays the same in the presence of all | In all three flavors described below, the network queue operates with | |||
types of congestion indicators: delay, loss, ECN marking | the simple first-in-first-out (FIFO) principle. There is no need to | |||
due to either RED/ECN or PCN algorithms. This unified | maintain per-flow state. The system can scale easily with a large | |||
approach allows a graceful transition of the scheme as the | number of video flows and at high link capacity. | |||
network shifts dynamically between light and heavy | ||||
congestion levels. | ||||
A.1 Default behavior of drop tail | A.1. Default behavior of drop tail queues | |||
In a conventional network with drop tail or RED queues, | In a conventional network with drop tail or RED queues, congestion is | |||
congestion is inferred from the estimation of end-to-end | inferred from the estimation of end-to-end delay and/or packet loss. | |||
delay and/or packet loss. Packet drops at the queue are | Packet drops at the queue are detected at the receiver, and | |||
detected at the receiver, and contributes to the | contributes to the calculation of the aggregated congestion signal | |||
calculation of the equivalent delay x_n. No special action | x_n. No special action is required at network node. | |||
is required at network node. | ||||
A.2 ECN marking | A.2. RED-based ECN marking | |||
In this mode, the network node randomly marks the ECN | In this mode, the network node randomly marks the ECN field in the IP | |||
field in the IP packet header following the Random Early | packet header following the Random Early Detection (RED) algorithm | |||
Detection (RED) algorithm [RFC2309]. Calculation of the | [RFC2309]. Calculation of the marking probability involves the | |||
marking probability involves the following steps: | following steps: | |||
* upon packet arrival, update smoothed queue size q_avg as: | on packet arrival: | |||
update smoothed queue size q_avg as: | ||||
q_avg = w*q + (1-w)*q_avg. | ||||
q_avg = alpha*q + (1-alpha)*q_avg. | calculate marking probability p as: | |||
The smoothing parameter alpha is a value between 0 and 1. A value of | / 0, if q < q_lo; | |||
alpha=1 corresponds to performing no smoothing at all. | | | |||
| q_avg - q_lo | ||||
p= < p_max*--------------, if q_lo <= q < q_hi; | ||||
| q_hi - q_lo | ||||
| | ||||
\ p = 1, if q >= q_hi. | ||||
* calculate marking probability p as: | Here, q_lo and q_hi corresponds to the low and high thresholds of | |||
queue occupancy. The maximum marking probability is p_max. | ||||
p = 0, if q < q_lo; | The ECN markings events will contribute to the calculation of an | |||
equivalent delay x_n at the receiver. No changes are required at the | ||||
sender. | ||||
q_avg - q_lo | A.3. Random Early Marking with Virtual Queues | |||
p = p_max*--------------, if q_lo <= q < q_hi; | ||||
q_hi - q_lo | ||||
p = 1, if q >= q_hi. | Advanced network nodes may support random early marking based on a | |||
token bucket algorithm originally designed for Pre-Congestion | ||||
Notification (PCN) [RFC6660]. The early congestion notification | ||||
(ECN) bit in the IP header of packets are marked randomly. The | ||||
marking probability is calculated based on a token-bucket algorithm | ||||
originally designed for the Pre-Congestion Notification (PCN) | ||||
[RFC6660]. The target link utilization is set as 90%; the marking | ||||
probability is designed to grow linearly with the token bucket size | ||||
when it varies between 1/3 and 2/3 of the full token bucket limit. | ||||
Here, q_lo and q_hi corresponds to the low and high thresholds of queue | * upon packet arrival, meter packet against token bucket (r,b); | |||
occupancy. The maximum marking probability is p_max. | ||||
The ECN markings events will contribute to the calculation of an | * update token level b_tk; | |||
equivalent delay x_n at the receiver. No changes are required at the | ||||
sender. | ||||
A.3 PCN marking | * calculate the marking probability as: | |||
As a more advanced feature, we also envisage network nodes which support | / 0, if b-b_tk < b_lo; | |||
PCN marking based on virtual queues. In such a case, the marking | | | |||
probability of the ECN bit in the IP packet header is calculated as | | b-b_tk-b_lo | |||
follows: | p = < p_max* --------------, if b_lo<= b-b_tk <b_hi; | |||
| b_hi-b_lo | ||||
| | ||||
\ 1, if b-b_tk>=b_hi. | ||||
* upon packet arrival, meter packet against token bucket (r,b); | Here, the token bucket lower and upper limits are denoted by b_lo and | |||
b_hi, respectively. The parameter b indicates the size of the token | ||||
bucket. The parameter r is chosen to be below capacity, resulting in | ||||
slight under-utilization of the link. The maximum marking | ||||
probability is p_max. | ||||
* update token level b_tk; | The ECN markings events will contribute to the calculation of an | |||
equivalent delay x_n at the receiver. No changes are required at the | ||||
sender. The virtual queuing mechanism from the PCN-based marking | ||||
algorithm will lead to additional benefits such as zero standing | ||||
queues. | ||||
* calculate the marking probability as: | Authors' Addresses | |||
p = 0, if b-b_tk < b_lo; | Xiaoqing Zhu | |||
Cisco Systems | ||||
12515 Research Blvd., Building 4 | ||||
Austin, TX 78759 | ||||
USA | ||||
b-b_tk-b_lo | Email: xiaoqzhu@cisco.com | |||
p = p_max* --------------, if b_lo<= b-b_tk <b_hi; | ||||
b_hi-b_lo | ||||
p = 1, if b-b_tk>=b_hi. | Rong Pan | |||
Cisco Systems | ||||
3625 Cisco Way | ||||
San Jose, CA 95134 | ||||
USA | ||||
Here, the token bucket lower and upper limits are denoted by b_lo and | Email: ropan@cisco.com | |||
b_hi, respectively. The parameter b indicates the size of the token | ||||
bucket. The parameter r is chosen as r=gamma*C, where gamma<1 is the | ||||
target utilization ratio and C designates link capacity. The maximum | ||||
marking probability is p_max. | ||||
The ECN markings events will contribute to the calculation of an | Michael A. Ramalho | |||
equivalent delay x_n at the receiver. No changes are required at the | Cisco Systems, Inc. | |||
sender. The virtual queuing mechanism from the PCN marking algorithm | 8000 Hawkins Road | |||
will lead to additional benefits such as zero standing queues. | Sarasota, FL 34241 | |||
USA | ||||
Authors' Addresses | Phone: +1 919 476 2038 | |||
Email: mramalho@cisco.com | ||||
Sergio Mena de la Cruz | ||||
Cisco Systems | ||||
EPFL, Quartier de l'Innovation, Batiment E | ||||
Ecublens, Vaud 1015 | ||||
Switzerland | ||||
Xiaoqing Zhu | Email: semena@cisco.com | |||
Cisco Systems, | ||||
12515 Research Blvd., | ||||
Austin, TX 78759, USA | ||||
Email: xiaoqzhu@cisco.com | ||||
Rong Pan | Paul E. Jones | |||
Cisco Systems | Cisco Systems | |||
510 McCarthy Blvd, | 7025 Kit Creek Rd. | |||
Milpitas, CA 95134, USA | Research Triangle Park, NC 27709 | |||
Email: ropan@cisco.com | USA | |||
Michael A. Ramalho | Email: paulej@packetizer.com | |||
6310 Watercrest Way Unit 203 | ||||
Lakewood Ranch, FL, 34202, USA | ||||
Email: mramalho@cisco.com | ||||
Sergio Mena de la Cruz | ||||
Cisco Systems | ||||
EPFL, Quartier de l'Innovation, Batiment E | ||||
Ecublens, Vaud 1015, Switzerland | ||||
Email: semena@cisco.com | ||||
Charles Ganzhorn | Jiantao Fu | |||
7900 International Drive | Cisco Systems | |||
International Plaza, Suite 400 | 707 Tasman Drive | |||
Bloomington, MN 55425, USA | Milpitas, CA 95035 | |||
Email: charles.ganzhorn@gmail.com | USA | |||
Paul E. Jones | Email: jianfu@cisco.com | |||
7025 Kit Creek Rd. | ||||
Research Triangle Park, NC 27709, USA | ||||
Email: paulej@packetizer.com | ||||
Stefano D'Aronco | Stefano D'Aronco | |||
EPFL STI IEL LTS4 | Ecole Polytechnique Federale de Lausanne | |||
ELD 220 (Batiment ELD), Station 11 | EPFL STI IEL LTS4, ELD 220 (Batiment ELD), Station 11 | |||
CH-1015 Lausanne, Switzerland | Lausanne CH-1015 | |||
Email: stefano.daronco@epfl.ch | Switzerland | |||
Email: stefano.daronco@epfl.ch | ||||
Charles Ganzhorn | ||||
7900 International Drive, International Plaza, Suite 400 | ||||
Bloomington, MN 55425 | ||||
USA | ||||
Email: charles.ganzhorn@gmail.com | ||||
End of changes. 179 change blocks. | ||||
588 lines changed or deleted | 764 lines changed or added | |||
This html diff was produced by rfcdiff 1.42. The latest version is available from http://tools.ietf.org/tools/rfcdiff/ |