draft-ietf-rmcat-eval-test-08.txt   draft-ietf-rmcat-eval-test-09.txt 
Network Working Group Z. Sarker Network Working Group Z. Sarker
Internet-Draft Ericsson AB Internet-Draft Ericsson AB
Intended status: Informational V. Singh Intended status: Informational V. Singh
Expires: May 30, 2019 callstats.io Expires: August 12, 2019 callstats.io
X. Zhu X. Zhu
M. Ramalho M. Ramalho
Cisco Systems Cisco Systems
November 26, 2018 February 08, 2019
Test Cases for Evaluating RMCAT Proposals Test Cases for Evaluating RMCAT Proposals
draft-ietf-rmcat-eval-test-08 draft-ietf-rmcat-eval-test-09
Abstract Abstract
The Real-time Transport Protocol (RTP) is used to transmit media in The Real-time Transport Protocol (RTP) is used to transmit media in
multimedia telephony applications. These applications are typically multimedia telephony applications. These applications are typically
required to implement congestion control. This document describes required to implement congestion control. This document describes
the test cases to be used in the performance evaluation of such the test cases to be used in the performance evaluation of such
congestion control algorithms in a controlled environment. congestion control algorithms in a controlled environment.
Status of This Memo Status of This Memo
skipping to change at page 1, line 38 skipping to change at page 1, line 38
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/. Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on May 30, 2019. This Internet-Draft will expire on August 12, 2019.
Copyright Notice Copyright Notice
Copyright (c) 2018 IETF Trust and the persons identified as the Copyright (c) 2019 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(https://trustee.ietf.org/license-info) in effect on the date of (https://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License. described in the Simplified BSD License.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2
2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3
3. Structure of Test cases . . . . . . . . . . . . . . . . . . . 3 3. Structure of Test cases . . . . . . . . . . . . . . . . . . . 3
4. Recommended Evaluation Settings . . . . . . . . . . . . . . . 7 4. Recommended Evaluation Settings . . . . . . . . . . . . . . . 8
4.1. Evaluation metrics . . . . . . . . . . . . . . . . . . . 8 4.1. Evaluation metrics . . . . . . . . . . . . . . . . . . . 8
4.2. Path characteristics . . . . . . . . . . . . . . . . . . 8 4.2. Path characteristics . . . . . . . . . . . . . . . . . . 8
4.3. Media source . . . . . . . . . . . . . . . . . . . . . . 9 4.3. Media source . . . . . . . . . . . . . . . . . . . . . . 9
5. Basic Test Cases . . . . . . . . . . . . . . . . . . . . . . 10 5. Basic Test Cases . . . . . . . . . . . . . . . . . . . . . . 10
5.1. Variable Available Capacity with a Single Flow . . . . . 10 5.1. Variable Available Capacity with a Single Flow . . . . . 10
5.2. Variable Available Capacity with Multiple Flows . . . . . 13 5.2. Variable Available Capacity with Multiple Flows . . . . . 13
5.3. Congested Feedback Link with Bi-directional Media Flows . 14 5.3. Congested Feedback Link with Bi-directional Media Flows . 14
5.4. Competing Media Flows with same Congestion Control 5.4. Competing Media Flows with same Congestion Control
Algorithm . . . . . . . . . . . . . . . . . . . . . . . . 17 Algorithm . . . . . . . . . . . . . . . . . . . . . . . . 17
5.5. Round Trip Time Fairness . . . . . . . . . . . . . . . . 19 5.5. Round Trip Time Fairness . . . . . . . . . . . . . . . . 19
5.6. Media Flow Competing with a Long TCP Flow . . . . . . . . 21 5.6. Media Flow Competing with a Long TCP Flow . . . . . . . . 21
5.7. Media Flow Competing with Short TCP Flows . . . . . . . . 23 5.7. Media Flow Competing with Short TCP Flows . . . . . . . . 23
5.8. Media Pause and Resume . . . . . . . . . . . . . . . . . 25 5.8. Media Pause and Resume . . . . . . . . . . . . . . . . . 25
6. Other potential test cases . . . . . . . . . . . . . . . . . 27 6. Other potential test cases . . . . . . . . . . . . . . . . . 27
6.1. Media Flows with Priority . . . . . . . . . . . . . . . . 27 6.1. Media Flows with Priority . . . . . . . . . . . . . . . . 27
6.2. Explicit Congestion Notification Usage . . . . . . . . . 27 6.2. Explicit Congestion Notification Usage . . . . . . . . . 27
6.3. Multiple Bottlenecks . . . . . . . . . . . . . . . . . . 27 6.3. Multiple Bottlenecks . . . . . . . . . . . . . . . . . . 28
7. Wireless Access Links . . . . . . . . . . . . . . . . . . . . 30 7. Wireless Access Links . . . . . . . . . . . . . . . . . . . . 30
8. Security Considerations . . . . . . . . . . . . . . . . . . . 30 8. Security Considerations . . . . . . . . . . . . . . . . . . . 30
9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 30 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 30
10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 30 10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 30
11. References . . . . . . . . . . . . . . . . . . . . . . . . . 30 11. References . . . . . . . . . . . . . . . . . . . . . . . . . 30
11.1. Normative References . . . . . . . . . . . . . . . . . . 30 11.1. Normative References . . . . . . . . . . . . . . . . . . 30
11.2. Informative References . . . . . . . . . . . . . . . . . 31 11.2. Informative References . . . . . . . . . . . . . . . . . 32
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 32 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 32
1. Introduction 1. Introduction
This memo describes a set of test cases for evaluating congestion This memo describes a set of test cases for evaluating congestion
control algorithm proposals for real-time interactive media. It is control algorithm proposals in controlled environment for real-time
based on the guidelines enumerated in [I-D.ietf-rmcat-eval-criteria] interactive media. It is based on the guidelines enumerated in
and the requirements discussed in [I-D.ietf-rmcat-cc-requirements]. [I-D.ietf-rmcat-eval-criteria] and the requirements discussed in
The test cases cover basic usage scenarios and are described using a [I-D.ietf-rmcat-cc-requirements]. The test cases cover basic usage
common structure, which allows for additional test cases to be added scenarios and are described using a common structure, which allows
to those described herein to accommodate other topologies and/or the for additional test cases to be added to those described herein to
modelling of different path characteristics. The described test accommodate other topologies and/or the modelling of different path
cases in this memo SHOULD be used to evaluate any proposed congestion characteristics. The described test cases in this memo should be
control algorithm for real-time interactive media. used to evaluate any proposed congestion control algorithm for real-
time interactive media.
2. Terminology 2. Terminology
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The terminology defined in RTP [RFC3550], RTP Profile for Audio and
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this Video Conferences with Minimal Control [RFC3551], RTCP Extended
document are to be interpreted as described in RFC2119 [RFC2119]. Report (XR) [RFC3611], Extended RTP Profile for RTCP-based Feedback
(RTP/AVPF) [RFC4585], and Support for Reduced-Size RTCP [RFC5506]
In addition, the terminology defined in RTP [RFC3550], RTP Profile apply.
for Audio and Video Conferences with Minimal Control [RFC3551], RTCP
Extended Report (XR) [RFC3611], Extended RTP Profile for RTCP-based
Feedback (RTP/AVPF) [RFC4585], and Support for Reduced-Size RTCP
[RFC5506] apply.
3. Structure of Test cases 3. Structure of Test cases
All the test cases in this document follow a basic structure allowing All the test cases in this document follow a basic structure allowing
implementers to describe a new test scenario without repeatedly implementers to describe a new test scenario without repeatedly
explaining common attributes. The structure includes a general explaining common attributes. The structure includes a general
description section that describes the test case and its motivation. description section that describes the test case and its motivation.
Additionally the test case defines a set of attributes that Additionally the test case defines a set of attributes that
characterize the testbed, for example, the network path between characterize the testbed, for example, the network path between
communicating peers and the diverse traffic sources. communicating peers and the diverse traffic sources.
skipping to change at page 5, line 29 skipping to change at page 5, line 26
value. value.
+ One-way propagation delay: describes the end-to-end latency + One-way propagation delay: describes the end-to-end latency
along the path when network queues are empty, i.e., the time along the path when network queues are empty, i.e., the time
it takes for a packet to go from the sender to the receiver it takes for a packet to go from the sender to the receiver
without encountering any queuing delay. without encountering any queuing delay.
+ Maximum end-to-end jitter: defines the maximum jitter that + Maximum end-to-end jitter: defines the maximum jitter that
can be observed along the path. can be observed along the path.
+ Bottleneck queue type: for example, Droptail, FQ-CoDel, or + Bottleneck queue type: for example, "tail drop" [RFC7567],
PIE. FQ-CoDel[RFC8290], or PIE[RFC8033].
+ Bottleneck queue size: defines the size of queue in terms of + Bottleneck queue size: defines the size of queue in terms of
queuing time when the queue is full (in milliseconds). queuing time when the queue is full (in milliseconds).
+ Path loss ratio: characterizes the non-congested, additive, + Path loss ratio: characterizes the non-congested, additive,
losses to be generated on the end-to-end path. MUST losses to be generated on the end-to-end path. This must
describe the loss pattern or loss model used to generate the describe the loss pattern or loss model used to generate the
losses. losses.
* Application-related: defines the traffic source behavior for * Application-related: defines the traffic source behavior for
implementing the test case implementing the test case
+ Media traffic Source: defines the characteristics of the + Media traffic Source: defines the characteristics of the
media sources. When using more than one media source, the media sources. When using more than one media source, the
different attributes are enumerated separately for each different attributes are enumerated separately for each
different media source. different media source.
skipping to change at page 6, line 30 skipping to change at page 6, line 27
defines the range of bit rate adaptation, the sampling defines the range of bit rate adaptation, the sampling
rate variation, and the variation in packetization rate variation, and the variation in packetization
interval. interval.
o Output variation : for a VBR encoder it defines the o Output variation : for a VBR encoder it defines the
encoder output variation from the average target rate encoder output variation from the average target rate
over a particular measurement interval. For example, over a particular measurement interval. For example,
on average the encoder output may vary between 5% to on average the encoder output may vary between 5% to
15% above or below the average target bit rate when 15% above or below the average target bit rate when
measured over a 100 ms time window. The time interval measured over a 100 ms time window. The time interval
over which the variation is specified MUST be over which the variation is specified must be
provided. provided.
o Responsiveness to a new bit rate request: the lag in o Responsiveness to a new bit rate request: the lag in
time between a new bit rate request from the time between a new bit rate request from the
congestion control algorithm and actual rate changes congestion control algorithm and actual rate changes
in encoder output. Depending on the encoder, this in encoder output. Depending on the encoder, this
value may be specified in absolute time (e.g. 10ms to value may be specified in absolute time (e.g. 10ms to
1000ms) or other appropriate metric (e.g. next frame 1000ms) or other appropriate metric (e.g. next frame
interval time). interval time).
More detailed discussions on expected media source More detailed discussions on expected media source
behavior, including those from synthetic video traffic behavior, including those from synthetic video traffic
sources, is at [I-D.ietf-rmcat-video-traffic-model]. sources, is at [I-D.ietf-rmcat-video-traffic-model].
- Media content: describes the chosen media sequences; For - Media content: describes the chosen video scenario. For
example, test sequences are available at: [xiph-seq] and example, video test sequences are available at:
[HEVC-seq]. [xiph-seq] and [HEVC-seq]. Different video scenarios
give different distribution of video frames produced by
the video encoder. Hence, it is important to specify the
media content used in a particular test. If a synthetic
video traffic souce [I-D.ietf-rmcat-video-traffic-model]
is used, then the synthetic video traffic source needs to
configure according to the characteristics of the media
content specified.
- Media timeline: describes the point when the media source - Media timeline: describes the point when the media source
is introduced and removed from the testbed. For example, is introduced and removed from the testbed. For example,
the media source may start transmitting immediately when the media source may start transmitting immediately when
the test case begins, or after a few seconds. the test case begins, or after a few seconds.
- Startup behavior: the media starts at a defined bit rate, - Startup behavior: the media starts at a defined bit rate,
which may be the minimum, maximum bit rate, or a value in which may be the minimum, maximum bit rate, or a value in
between (in Kbps). between (in Kbps).
skipping to change at page 7, line 33 skipping to change at page 7, line 38
sources of each media type per traffic direction. sources of each media type per traffic direction.
- Congestion control: enumerates the congestion control - Congestion control: enumerates the congestion control
used by each type of competing traffic. used by each type of competing traffic.
- Traffic timeline: describes when the competing traffic - Traffic timeline: describes when the competing traffic
starts and ends in the test case. starts and ends in the test case.
* Additional attributes: describes attributes essential for * Additional attributes: describes attributes essential for
implementing a test case which are not included in the above implementing a test case which are not included in the above
structure. These attributes MUST be well defined, so that the structure. These attributes must be well defined, so that the
other implementers of that particular test case are able to other implementers of that particular test case are able to
implement it easily. implement it easily.
Any attribute can have a set of values (enclosed within "[]"). Each Any attribute can have a set of values (enclosed within "[]"). Each
member value of such a set MUST be treated as different value for the member value of such a set must be treated as different value for the
same attribute. It is desired to run separate tests for each such same attribute. It is desired to run separate tests for each such
attribute value. attribute value.
The test cases described in this document follow the above structure. The test cases described in this document follow the above structure.
4. Recommended Evaluation Settings 4. Recommended Evaluation Settings
This section describes recommended test case settings and could be This section describes recommended test case settings and could be
overwritten by the respective test cases. overwritten by the respective test cases.
4.1. Evaluation metrics 4.1. Evaluation metrics
To evaluate the performance of the candidate algorithms the To evaluate the performance of the candidate algorithms the
implementers MUST log enough information to visualize the following implementers must log enough information to visualize the following
metrics at a fine enough time granularity: metrics at a fine enough time granularity:
1. Flow level: 1. Flow level:
A. End-to-end delay for the congestion controlled media flow(s). A. End-to-end delay for the congestion controlled media flow(s).
For example - end-to-end delay overserved on IP packet level,
video frame level.
B. Variation in sending bit rate and goodput. Mainly observing B. Variation in sending bit rate and throughput. Mainly
the frequency and magnitude of oscillations. observing the frequency and magnitude of oscillations.
C. Packet losses observed at the receiving endpoint. C. Packet losses observed at the receiving endpoint.
D. Feedback message overhead. D. Feedback message overhead.
E. Convergence time - time to reach steady state for the E. Convergence time - time to reach steady state for the
congestion controlled media flow(s). congestion controlled media flow(s). Each occurance of
convergence during the test period need to be presented.
2. Transport level: 2. Transport level:
A. Bandwidth utilization. A. Bandwidth utilization.
B. Queue length (milliseconds at specified path capacity): B. Queue length (milliseconds at specified path capacity).
+ average over the length of the session.
+ 5 and 95 percentile.
+ median, maximum, minimum.
4.2. Path characteristics 4.2. Path characteristics
Each path between a sender and receiver as described in Figure 1 have Each path between a sender and receiver as described in Figure 1 have
the following characteristics unless otherwise specified in the test the following characteristics unless otherwise specified in the test
case. case.
o Path direction: forward and backward. o Path direction: forward and backward.
o Reference bottleneck capacity: 1Mbps. o Reference bottleneck capacity: 1Mbps.
o One-Way propagation delay: 50ms. Implementers are encouraged to o One-Way propagation delay: 50ms. Implementers are encouraged to
run the experiment with additional propagation delays mentioned in run the experiment with additional propagation delays mentioned in
[I-D.ietf-rmcat-eval-criteria] [I-D.ietf-rmcat-eval-criteria]
o Maximum end-to-end jitter: 30ms. Jitter models are described in o Maximum end-to-end jitter: 30ms. Jitter models are described in
[I-D.ietf-rmcat-eval-criteria] [I-D.ietf-rmcat-eval-criteria]
o Bottleneck queue type: Drop tail. Implementers are encouraged to o Bottleneck queue type: "tail drop". Implementers are encouraged
run the experiment with other AQM schemes, such as FQ-CoDel and to run the experiment with other AQM schemes, such as FQ-CoDel and
PIE. PIE.
o Bottleneck queue size: 300ms. o Bottleneck queue size: 300ms.
o Path loss ratio: 0%. o Path loss ratio: 0%.
Examples of additional network parameters are discussed in Examples of additional network parameters are discussed in
[I-D.ietf-rmcat-eval-criteria]. [I-D.ietf-rmcat-eval-criteria].
For test cases involving time-varying bottleneck capacity, all For test cases involving time-varying bottleneck capacity, all
skipping to change at page 10, line 19 skipping to change at page 10, line 22
If a different bit rate range is used in the test cases If a different bit rate range is used in the test cases
then the frame resolution range also need to be selected then the frame resolution range also need to be selected
suitably. suitably.
- Frame rate: 10fps - 30fps. This frame rate range is - Frame rate: 10fps - 30fps. This frame rate range is
selected based on the bit rate range. If a different bit selected based on the bit rate range. If a different bit
rate range is used in the test cases then the frame rate rate range is used in the test cases then the frame rate
range also need to be adjusted suitably. range also need to be adjusted suitably.
+ Variation from target bit rate: +/-5%. Unless otherwise + Variation from target bit rate: +/-5%. Unless otherwise
specified in the test case(s), bit rate variation SHOULD be specified in the test case(s), bit rate variation should be
calculated over one (1) second period of time. calculated over one (1) second period of time.
+ Responsiveness to new bit rate request: 100ms + Responsiveness to new bit rate request: 100ms
* Media content: The media content should represent a typical * Media content: The media content should represent a typical
video conversational scenario with head and shoulder movement. video conversational scenario with head and shoulder movement.
We recommend to use Foreman video sequence. We recommend to use Foreman video sequence[xiph-seq].
* Media startup behavior: 150Kbps. It should be noted that * Media startup behavior: 150Kbps. It should be noted that
applications can use smart ways to select an optimal startup applications can use smart ways to select an optimal startup
bit rate value for a certain network condition. In such cases bit rate value for a certain network condition. In such cases
the candidate proposals MAY show the effectiveness of such the candidate proposals MAY show the effectiveness of such
smart approach as an additional information for the evaluation smart approach as an additional information for the evaluation
process. process.
o Media type: Audio o Media type: Audio
skipping to change at page 11, line 24 skipping to change at page 11, line 26
congestion control scheme is expected to stabilize the sending bit congestion control scheme is expected to stabilize the sending bit
rate close to the available bottleneck capacity. rate close to the available bottleneck capacity.
It should be noted that the exact variation in available capacity due It should be noted that the exact variation in available capacity due
to any of the above depends on the underlying technologies. Hence, to any of the above depends on the underlying technologies. Hence,
we describe a set of known factors, which may be extended to devise a we describe a set of known factors, which may be extended to devise a
more specific test case targeting certain behaviors in a certain more specific test case targeting certain behaviors in a certain
network environment. network environment.
Expected behavior: the candidate algorithm is expected to detect the Expected behavior: the candidate algorithm is expected to detect the
path capacity constraint, converges to the bottleneck link's capacity path capacity constraint, converge to the bottleneck link's capacity
and adapt the flow to avoid unwanted media rate oscillation when the and adapt the flow to avoid unwanted media rate oscillation when the
sending bit rate is approaching the bottleneck link's capacity. Such sending bit rate is approaching the bottleneck link's capacity. Such
oscillations might occur when the media flow(s) attempts to reach its oscillations might occur when the media flow(s) attempts to reach its
maximum bit rate but overshoots the usage of the available bottleneck maximum bit rate but overshoots the usage of the available bottleneck
capacity then to rectify, it reduces the bit rate and starts to ramp capacity then to rectify, it reduces the bit rate and starts to ramp
up again. up again.
Evaluation metrics : as described in Section 4.1. Evaluation metrics : as described in Section 4.1.
Testbed topology: One media source S1 is connected to the Testbed topology: One media source S1 is connected to the
skipping to change at page 14, line 36 skipping to change at page 14, line 52
| Four | Forward | 75s | 0.5 | | Four | Forward | 75s | 0.5 |
| Five | Forward | 100s | 1.0 | | Five | Forward | 100s | 1.0 |
+--------------------+--------------+-----------+-------------------+ +--------------------+--------------+-----------+-------------------+
Table 2: Path capacity variation pattern for forward direction Table 2: Path capacity variation pattern for forward direction
5.3. Congested Feedback Link with Bi-directional Media Flows 5.3. Congested Feedback Link with Bi-directional Media Flows
Real-time interactive media uses RTP hence it is assumed that RTCP, Real-time interactive media uses RTP hence it is assumed that RTCP,
RTP header extension or such would be used by the congestion control RTP header extension or such would be used by the congestion control
algorithm in the backchannel. Due to asymmetric nature of the link algorithm in the backchannel. Due to the asymmetric nature of the
between communicating peers it is possible for a participating peer link between communicating peers it is possible for a participating
to not receive such feedback information due to an impaired or peer to not receive such feedback information due to an impaired or
congested backchannel (even when the forward channel might not be congested backchannel (even when the forward channel might not be
impaired). This test case is designed to observe the candidate impaired). This test case is designed to observe the candidate
congestion control behavior in such an event. congestion control behavior in such an event.
Expected behavior: It is expected that the candidate algorithms are Expected behavior: It is expected that the candidate algorithms are
able to cope with the lack of feedback information and adapt to able to cope with the lack of feedback information and adapt to
minimize the performance degradation of media flows in the forward minimize the performance degradation of media flows in the forward
channel. channel.
It should be noted that for this test case: logs are compared with It should be noted that for this test case: logs are compared with
skipping to change at page 17, line 45 skipping to change at page 18, line 20
transported over the backward path. transported over the backward path.
+---+ +---+ +---+ +---+
|S1 |===== \ Forward --> / =======|R1 | |S1 |===== \ Forward --> / =======|R1 |
+---+ \\ // +---+ +---+ \\ // +---+
\\ // \\ //
+---+ +-----+ +-----+ +---+ +---+ +-----+ +-----+ +---+
|S2 |=======| A |------------------------------>| B |=======|R2 | |S2 |=======| A |------------------------------>| B |=======|R2 |
+---+ | |<------------------------------| | +---+ +---+ | |<------------------------------| | +---+
+-----+ +-----+ +-----+ +-----+
// \\ // <-- Backward \\
// <-- Backward \\ +---+ // \\ +---+
+---+ // \\ +---+ |S3 |===== / \ ======|R3 |
|S3 |====== / \ ======|R3 | +---+ +---+
+---+ +---+
Figure 5: Testbed Topology for Multiple congestion controlled media Figure 5: Testbed Topology for Multiple congestion controlled media
Flows Flows
Testbed attributes: Testbed attributes:
o Test duration: 120s o Test duration: 120s
o Path characteristics: o Path characteristics:
skipping to change at page 19, line 30 skipping to change at page 19, line 43
In this test case, multiple media flows share the bottleneck link, In this test case, multiple media flows share the bottleneck link,
but the one-way propagation delay for each flow is different. For but the one-way propagation delay for each flow is different. For
the sake of simplicity it is assumed that there are no other the sake of simplicity it is assumed that there are no other
competing traffic sources in the bottleneck link and that there is competing traffic sources in the bottleneck link and that there is
sufficient capacity to accommodate all the flows. While this appears sufficient capacity to accommodate all the flows. While this appears
to be a variant of test case 5.2, it focuses on the capacity sharing to be a variant of test case 5.2, it focuses on the capacity sharing
aspect of the candidate algorithm under different RTTs. aspect of the candidate algorithm under different RTTs.
Expected behavior: It is expected that the competing flows will Expected behavior: It is expected that the competing flows will
converge to bit rates to accommodate all the flows with minimum converge to bit rates to accommodate all the flows with minimum
possible latency and loss. Specifically, the test introduces five possible latency and loss. The effectiveness of the algorithm
media flows at the same time instance. depends on how fast and fairly the comepting flows converge to their
steady states irrespective of the RTT overserved.
Evaluation metrics : as described in Section 4.1. Evaluation metrics : as described in Section 4.1.
Testbed Topology: Five (5) media sources S1,S2,..,S5 are connected to Testbed Topology: Five (5) media sources S1,S2,..,S5 are connected to
their corresponding media sinks R1,R2,..,R5. The media traffic is their corresponding media sinks R1,R2,..,R5. The media traffic is
transported over the forward path and corresponding feedback/control transported over the forward path and corresponding feedback/control
traffic is transported over the backward path. The topology is the traffic is transported over the backward path. The topology is the
same as in Section 5.4. same as in Section 5.4.
Testbed attributes: Testbed attributes:
skipping to change at page 22, line 5 skipping to change at page 22, line 7
B. Loss for the TCP flow B. Loss for the TCP flow
Testbed topology: One (1) media source S1 is connected to the Testbed topology: One (1) media source S1 is connected to the
corresponding media sink, R1. In addition, there is a long-live TCP corresponding media sink, R1. In addition, there is a long-live TCP
flow sharing the same bottleneck link. The media traffic is flow sharing the same bottleneck link. The media traffic is
transported over the forward path and corresponding feedback/control transported over the forward path and corresponding feedback/control
traffic is transported over the backward path. The TCP traffic goes traffic is transported over the backward path. The TCP traffic goes
over the forward path from, S_tcp with acknowledgment packets go over over the forward path from, S_tcp with acknowledgment packets go over
the backward path from, R_tcp. the backward path from, R_tcp.
+--+ +--+ +--+ +--+
|S1|===== \ Forward --> / =======|R1| |S1|===== \ Forward --> / =====|R1|
+--+ \\ // +--+ +--+ \\ // +--+
\\ // \\ //
+-----+ +-----+ +-----+ +-----+
| A |---------------------------->| B | | A |---------------------------->| B |
| |<----------------------------| | | |<----------------------------| |
+-----+ +-----+ +-----+ +-----+
// \\ // <-- Backward \\
// <-- Backward \\ +-----+ // \\ +-----+
+-----+ // \\ +-----+ |S_tcp|=== / \ ===|R_tcp|
|S_tcp|=== / \ ===|R_tcp| +-----+ +-----+
+-----+ +-----+
Figure 6: Testbed Topology for TCP vs congestion controlled media Figure 6: Testbed Topology for TCP vs congestion controlled media
Flows Flows
Testbed attributes: Testbed attributes:
o Test duration: 120s o Test duration: 120s
o Path characteristics: o Path characteristics:
skipping to change at page 25, line 40 skipping to change at page 25, line 42
test is described in [I-D.ietf-rmcat-eval-criteria]. test is described in [I-D.ietf-rmcat-eval-criteria].
5.8. Media Pause and Resume 5.8. Media Pause and Resume
In this test case, more than one real-time interactive media flows In this test case, more than one real-time interactive media flows
share the link bandwidth and all flows reach to a steady state by share the link bandwidth and all flows reach to a steady state by
utilizing the link capacity in an optimum way. At this stage one of utilizing the link capacity in an optimum way. At this stage one of
the media flows is paused for a moment. This event will result in the media flows is paused for a moment. This event will result in
more available bandwidth for the rest of the flows as they are on a more available bandwidth for the rest of the flows as they are on a
shared link. When the paused media flow resumes it would no longer shared link. When the paused media flow resumes it would no longer
have the same bandwidth share on the link. It has to make it's way have the same bandwidth share on the link. It has to make its way
through the other existing flows in the link to achieve a fair share through the other existing flows in the link to achieve a fair share
of the link capacity. This test case is important specially for of the link capacity. This test case is important specially for
real-time interactive media which consists of more than one media real-time interactive media which consists of more than one media
flows and can pause/resume media flows at any point of time during flows and can pause/resume media flows at any point of time during
the session. This test case directly addresses the requirement the session. This test case directly addresses the requirement
number 5 in [I-D.ietf-rmcat-cc-requirements]. One can think it as a number 5 in [I-D.ietf-rmcat-cc-requirements]. One can think it as a
variation of test case defined in Section 5.4. However, it is variation of test case defined in Section 5.4. However, it is
different as the candidate algorithms can use different strategies to different as the candidate algorithms can use different strategies to
increase its efficiency, for example in terms of fairness, increase its efficiency, for example in terms of fairness,
convergence time, reduce oscillation etc, by capitalizing the fact convergence time, reduce oscillation etc, by capitalizing the fact
skipping to change at page 27, line 29 skipping to change at page 27, line 31
additional test cases can help further evaluation of the candidate additional test cases can help further evaluation of the candidate
algorithm. They are listed as below. algorithm. They are listed as below.
6.1. Media Flows with Priority 6.1. Media Flows with Priority
In this test case media flows will have different priority levels. In this test case media flows will have different priority levels.
This will be an extension of Section 5.4 where the same test will be This will be an extension of Section 5.4 where the same test will be
run with different priority levels imposed on each of the media run with different priority levels imposed on each of the media
flows. For example, the first flow (S1) is assigned a priority of 2 flows. For example, the first flow (S1) is assigned a priority of 2
whereas the remaining two flows (S2 and S3) are assigned a priority whereas the remaining two flows (S2 and S3) are assigned a priority
of 1. The candidate algorithm MUST reflect the relative priorities of 1. The candidate algorithm must reflect the relative priorities
assigned to each media flow. In the previous example, the first flow assigned to each media flow. In this case, the first flow (S1) must
(S1) MUST arrive at a steady-state rate approximately twice of that arrive at a steady-state rate approximately twice of that of the
of the other two flows (S2 and S3). other two flows (S2 and S3).
The candidate algorithm can use a coupled congestion control The candidate algorithm can use a coupled congestion control
mechanism or use a weighted priority scheduler for the bandwidth mechanism [I-D.ietf-rmcat-coupled-cc] or use a weighted priority
distribution according to the respective media flow priority or use. scheduler for the bandwidth distribution according to the respective
media flow priority or use.
6.2. Explicit Congestion Notification Usage 6.2. Explicit Congestion Notification Usage
This test case requires to run all the basic test cases with the This test case requires to run all the basic test cases with the
availability of Explicit Congestion Notification (ECN) [RFC6679] availability of Explicit Congestion Notification (ECN) [RFC6679]
feature enabled. The goal of this test is to exhibit that the feature enabled. The goal of this test is to exhibit that the
candidate algorithms do not fail when ECN signals are available. candidate algorithms do not fail when ECN signals are available.
With ECN signals enabled the algorithms are expected to perform With ECN signals enabled the algorithms are expected to perform
better than their delay based variants. better than their delay based variants.
skipping to change at page 28, line 10 skipping to change at page 28, line 17
In this test case one congestion controlled media flow, S1->R1, In this test case one congestion controlled media flow, S1->R1,
traverses a path with multiple bottlenecks. As illustrated in traverses a path with multiple bottlenecks. As illustrated in
Figure 7, the first flow (S1->R1) competes with the second congestion Figure 7, the first flow (S1->R1) competes with the second congestion
controlled media flow (S2->R2) over the link between A and B which is controlled media flow (S2->R2) over the link between A and B which is
close to the sender side; again, that flow (S1->R1) competes with the close to the sender side; again, that flow (S1->R1) competes with the
third congestion controlled media flow (S3->R3) over the link between third congestion controlled media flow (S3->R3) over the link between
C and D which is close to the receiver side. The goal of this test C and D which is close to the receiver side. The goal of this test
is to ensure that the candidate algorithms work properly in the is to ensure that the candidate algorithms work properly in the
presence of multiple bottleneck links on the end to end path. presence of multiple bottleneck links on the end to end path.
Expected behavior: the candidate algorithm is expected to achieve Expected behavior: The candidate algorithm is expected to achieve
full utilization at both bottleneck links without starving any of the full utilization at both bottleneck links without starving any of the
three congestion controlled media flows. three congestion controlled media flows and esuring fair share of the
available bandwidth at each bottlenecks.
Forward ----> Forward ---->
+---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+
|S2 | |R2 | |S3 | |R3 | |S2 | |R2 | |S3 | |R3 |
+---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+
| | | | | | | |
| | | | | | | |
+---+ +-----+ +-----+ +-----+ +-----+ +---+ +---+ +-----+ +-----+ +-----+ +-----+ +---+
|S1 |=======| A |------>| B |----->| C |---->| D |=======|R1 | |S1 |=======| A |------>| B |----->| C |---->| D |=======|R1 |
skipping to change at page 30, line 15 skipping to change at page 30, line 19
7. Wireless Access Links 7. Wireless Access Links
Additional wireless network (both cellular network and WiFi network) Additional wireless network (both cellular network and WiFi network)
specific test cases are defined in [I-D.ietf-rmcat-wireless-tests]. specific test cases are defined in [I-D.ietf-rmcat-wireless-tests].
8. Security Considerations 8. Security Considerations
The security considerations in [I-D.ietf-rmcat-eval-criteria] and the The security considerations in [I-D.ietf-rmcat-eval-criteria] and the
relevant congestion control algorithms apply. The principles for relevant congestion control algorithms apply. The principles for
congestion control are described in [RFC2914], and in particular any congestion control are described in [RFC2914], and in particular any
new method MUST implement safeguards to avoid congestion collapse of new method must implement safeguards to avoid congestion collapse of
the Internet. the Internet.
The evaluation of the test cases are intended to be run in a The evaluation of the test cases are intended to be run in a
controlled lab environment. Hence, the applications, simulators and controlled lab environment. Hence, the applications, simulators and
network nodes ought to be well-behaved and should not impact the network nodes ought to be well-behaved and should not impact the
desired results. It is important to take appropriate caution to desired results. Moreover, proper measures must be taked to avoid
avoid leaking non-responsive traffic from unproven congestion leaking non-responsive traffic from unproven congestion avoidance
avoidance techniques onto the open Internet. techniques onto the open Internet.
9. IANA Considerations 9. IANA Considerations
There are no IANA impacts in this memo. There are no IANA impacts in this memo.
10. Acknowledgements 10. Acknowledgements
Much of this document is derived from previous work on congestion Much of this document is derived from previous work on congestion
control at the IETF. control at the IETF.
The content and concepts within this document are a product of the The content and concepts within this document are a product of the
discussion carried out in the Design Team. discussion carried out in the Design Team.
11. References 11. References
11.1. Normative References 11.1. Normative References
[I-D.ietf-rmcat-cc-requirements]
Jesup, R. and Z. Sarker, "Congestion Control Requirements
for Interactive Real-Time Media", draft-ietf-rmcat-cc-
requirements-09 (work in progress), December 2014.
[I-D.ietf-rmcat-eval-criteria] [I-D.ietf-rmcat-eval-criteria]
Singh, V., Ott, J., and S. Holmer, "Evaluating Congestion Singh, V., Ott, J., and S. Holmer, "Evaluating Congestion
Control for Interactive Real-time Media", draft-ietf- Control for Interactive Real-time Media", draft-ietf-
rmcat-eval-criteria-08 (work in progress), November 2018. rmcat-eval-criteria-08 (work in progress), November 2018.
[I-D.ietf-rmcat-video-traffic-model] [I-D.ietf-rmcat-video-traffic-model]
Zhu, X., Cruz, S., and Z. Sarker, "Video Traffic Models Zhu, X., Cruz, S., and Z. Sarker, "Video Traffic Models
for RTP Congestion Control Evaluations", draft-ietf-rmcat- for RTP Congestion Control Evaluations", draft-ietf-rmcat-
video-traffic-model-06 (work in progress), November 2018. video-traffic-model-06 (work in progress), November 2018.
[I-D.ietf-rmcat-wireless-tests] [I-D.ietf-rmcat-wireless-tests]
Sarker, Z., Johansson, I., Zhu, X., Fu, J., Tan, W., and Sarker, Z., Johansson, I., Zhu, X., Fu, J., Tan, W., and
M. Ramalho, "Evaluation Test Cases for Interactive Real- M. Ramalho, "Evaluation Test Cases for Interactive Real-
Time Media over Wireless Networks", draft-ietf-rmcat- Time Media over Wireless Networks", draft-ietf-rmcat-
wireless-tests-05 (work in progress), June 2018. wireless-tests-06 (work in progress), December 2018.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
<https://www.rfc-editor.org/info/rfc2119>.
[RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V. [RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V.
Jacobson, "RTP: A Transport Protocol for Real-Time Jacobson, "RTP: A Transport Protocol for Real-Time
Applications", STD 64, RFC 3550, DOI 10.17487/RFC3550, Applications", STD 64, RFC 3550, DOI 10.17487/RFC3550,
July 2003, <https://www.rfc-editor.org/info/rfc3550>. July 2003, <https://www.rfc-editor.org/info/rfc3550>.
[RFC3551] Schulzrinne, H. and S. Casner, "RTP Profile for Audio and [RFC3551] Schulzrinne, H. and S. Casner, "RTP Profile for Audio and
Video Conferences with Minimal Control", STD 65, RFC 3551, Video Conferences with Minimal Control", STD 65, RFC 3551,
DOI 10.17487/RFC3551, July 2003, DOI 10.17487/RFC3551, July 2003,
<https://www.rfc-editor.org/info/rfc3551>. <https://www.rfc-editor.org/info/rfc3551>.
skipping to change at page 32, line 5 skipping to change at page 32, line 11
and K. Carlberg, "Explicit Congestion Notification (ECN) and K. Carlberg, "Explicit Congestion Notification (ECN)
for RTP over UDP", RFC 6679, DOI 10.17487/RFC6679, August for RTP over UDP", RFC 6679, DOI 10.17487/RFC6679, August
2012, <https://www.rfc-editor.org/info/rfc6679>. 2012, <https://www.rfc-editor.org/info/rfc6679>.
11.2. Informative References 11.2. Informative References
[HEVC-seq] [HEVC-seq]
HEVC, "Test Sequences", HEVC, "Test Sequences",
http://www.netlab.tkk.fi/~varun/test_sequences/ . http://www.netlab.tkk.fi/~varun/test_sequences/ .
[I-D.ietf-rmcat-cc-requirements] [I-D.ietf-rmcat-coupled-cc]
Jesup, R. and Z. Sarker, "Congestion Control Requirements Islam, S., Welzl, M., and S. Gjessing, "Coupled congestion
for Interactive Real-Time Media", draft-ietf-rmcat-cc- control for RTP media", draft-ietf-rmcat-coupled-cc-08
requirements-09 (work in progress), December 2014. (work in progress), January 2019.
[RFC2914] Floyd, S., "Congestion Control Principles", BCP 41, [RFC2914] Floyd, S., "Congestion Control Principles", BCP 41,
RFC 2914, DOI 10.17487/RFC2914, September 2000, RFC 2914, DOI 10.17487/RFC2914, September 2000,
<https://www.rfc-editor.org/info/rfc2914>. <https://www.rfc-editor.org/info/rfc2914>.
[RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion [RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion
Control", RFC 5681, DOI 10.17487/RFC5681, September 2009, Control", RFC 5681, DOI 10.17487/RFC5681, September 2009,
<https://www.rfc-editor.org/info/rfc5681>. <https://www.rfc-editor.org/info/rfc5681>.
[RFC7567] Baker, F., Ed. and G. Fairhurst, Ed., "IETF
Recommendations Regarding Active Queue Management",
BCP 197, RFC 7567, DOI 10.17487/RFC7567, July 2015,
<https://www.rfc-editor.org/info/rfc7567>.
[RFC8033] Pan, R., Natarajan, P., Baker, F., and G. White,
"Proportional Integral Controller Enhanced (PIE): A
Lightweight Control Scheme to Address the Bufferbloat
Problem", RFC 8033, DOI 10.17487/RFC8033, February 2017,
<https://www.rfc-editor.org/info/rfc8033>.
[RFC8290] Hoeiland-Joergensen, T., McKenney, P., Taht, D., Gettys,
J., and E. Dumazet, "The Flow Queue CoDel Packet Scheduler
and Active Queue Management Algorithm", RFC 8290,
DOI 10.17487/RFC8290, January 2018,
<https://www.rfc-editor.org/info/rfc8290>.
[xiph-seq] [xiph-seq]
Xiph.org, "Video Test Media", Xiph.org, "Video Test Media",
http://media.xiph.org/video/derf/ . http://media.xiph.org/video/derf/ .
Authors' Addresses Authors' Addresses
Zaheduzzaman Sarker Zaheduzzaman Sarker
Ericsson AB Ericsson AB
Luleae, SE 977 53 Luleae, SE 977 53
Sweden Sweden
Phone: +46 10 717 37 43 Phone: +46 10 717 37 43
Email: zaheduzzaman.sarker@ericsson.com Email: zaheduzzaman.sarker@ericsson.com
Varun Singh Varun Singh
Nemu Dialogue Systems Oy Nemu Dialogue Systems Oy
 End of changes. 41 change blocks. 
98 lines changed or deleted 116 lines changed or added

This html diff was produced by rfcdiff 1.47. The latest version is available from http://tools.ietf.org/tools/rfcdiff/