draft-ietf-rmcat-eval-test-06.txt   draft-ietf-rmcat-eval-test-07.txt 
Network Working Group Z. Sarker Network Working Group Z. Sarker
Internet-Draft Ericsson AB Internet-Draft Ericsson AB
Intended status: Informational V. Singh Intended status: Informational V. Singh
Expires: December 23, 2018 callstats.io Expires: April 25, 2019 callstats.io
X. Zhu X. Zhu
M. Ramalho M. Ramalho
Cisco Systems Cisco Systems
June 21, 2018 October 22, 2018
Test Cases for Evaluating RMCAT Proposals Test Cases for Evaluating RMCAT Proposals
draft-ietf-rmcat-eval-test-06 draft-ietf-rmcat-eval-test-07
Abstract Abstract
The Real-time Transport Protocol (RTP) is used to transmit media in The Real-time Transport Protocol (RTP) is used to transmit media in
multimedia telephony applications, these applications are typically multimedia telephony applications. These applications are typically
required to implement congestion control. This document describes required to implement congestion control. This document describes
the test cases to be used in the performance evaluation of such the test cases to be used in the performance evaluation of such
congestion control algorithms. congestion control algorithms.
Status of This Memo Status of This Memo
This Internet-Draft is submitted in full conformance with the This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79. provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/. Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on December 23, 2018. This Internet-Draft will expire on April 25, 2019.
Copyright Notice Copyright Notice
Copyright (c) 2018 IETF Trust and the persons identified as the Copyright (c) 2018 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(https://trustee.ietf.org/license-info) in effect on the date of (https://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
skipping to change at page 2, line 32 skipping to change at page 2, line 32
5.4. Competing Media Flows with same Congestion Control 5.4. Competing Media Flows with same Congestion Control
Algorithm . . . . . . . . . . . . . . . . . . . . . . . . 17 Algorithm . . . . . . . . . . . . . . . . . . . . . . . . 17
5.5. Round Trip Time Fairness . . . . . . . . . . . . . . . . 19 5.5. Round Trip Time Fairness . . . . . . . . . . . . . . . . 19
5.6. Media Flow Competing with a Long TCP Flow . . . . . . . . 21 5.6. Media Flow Competing with a Long TCP Flow . . . . . . . . 21
5.7. Media Flow Competing with Short TCP Flows . . . . . . . . 23 5.7. Media Flow Competing with Short TCP Flows . . . . . . . . 23
5.8. Media Pause and Resume . . . . . . . . . . . . . . . . . 25 5.8. Media Pause and Resume . . . . . . . . . . . . . . . . . 25
6. Other potential test cases . . . . . . . . . . . . . . . . . 27 6. Other potential test cases . . . . . . . . . . . . . . . . . 27
6.1. Media Flows with Priority . . . . . . . . . . . . . . . . 27 6.1. Media Flows with Priority . . . . . . . . . . . . . . . . 27
6.2. Explicit Congestion Notification Usage . . . . . . . . . 27 6.2. Explicit Congestion Notification Usage . . . . . . . . . 27
6.3. Multiple Bottlenecks . . . . . . . . . . . . . . . . . . 27 6.3. Multiple Bottlenecks . . . . . . . . . . . . . . . . . . 27
7. Wireless Access Links . . . . . . . . . . . . . . . . . . . . 29 7. Wireless Access Links . . . . . . . . . . . . . . . . . . . . 30
8. Security Considerations . . . . . . . . . . . . . . . . . . . 29 8. Security Considerations . . . . . . . . . . . . . . . . . . . 30
9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 29 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 30
10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 30 10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 30
11. References . . . . . . . . . . . . . . . . . . . . . . . . . 30 11. References . . . . . . . . . . . . . . . . . . . . . . . . . 30
11.1. Normative References . . . . . . . . . . . . . . . . . . 30 11.1. Normative References . . . . . . . . . . . . . . . . . . 30
11.2. Informative References . . . . . . . . . . . . . . . . . 31 11.2. Informative References . . . . . . . . . . . . . . . . . 31
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 31 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 32
1. Introduction 1. Introduction
This memo describes a set of test cases for evaluating congestion This memo describes a set of test cases for evaluating congestion
control algorithm proposals for real-time interactive media. It is control algorithm proposals for real-time interactive media. It is
based on the guidelines enumerated in [I-D.ietf-rmcat-eval-criteria] based on the guidelines enumerated in [I-D.ietf-rmcat-eval-criteria]
and the requirements discussed in [I-D.ietf-rmcat-cc-requirements]. and the requirements discussed in [I-D.ietf-rmcat-cc-requirements].
The test cases cover basic usage scenarios and are described using a The test cases cover basic usage scenarios and are described using a
common structure, which allows for additional test cases to be added common structure, which allows for additional test cases to be added
to those described herein to accommodate other topologies and/or the to those described herein to accommodate other topologies and/or the
modeling of different path characteristics. The described test cases modelling of different path characteristics. The described test
in this memo SHOULD be used to evaluate any proposed congestion cases in this memo SHOULD be used to evaluate any proposed congestion
control algorithm for real-time interactive media. control algorithm for real-time interactive media.
2. Terminology 2. Terminology
The terminology defined in RTP [RFC3550], RTP Profile for Audio and The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
Video Conferences with Minimal Control [RFC3551], RTCP Extended "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
Report (XR) [RFC3611], Extended RTP Profile for RTCP-based Feedback document are to be interpreted as described in RFC2119 [RFC2119].
(RTP/AVPF) [RFC4585], and Support for Reduced-Size RTCP [RFC5506]
apply. In addition, the terminology defined in RTP [RFC3550], RTP Profile
for Audio and Video Conferences with Minimal Control [RFC3551], RTCP
Extended Report (XR) [RFC3611], Extended RTP Profile for RTCP-based
Feedback (RTP/AVPF) [RFC4585], and Support for Reduced-Size RTCP
[RFC5506] apply.
3. Structure of Test cases 3. Structure of Test cases
All the test cases in this document follow a basic structure allowing All the test cases in this document follow a basic structure allowing
implementers to describe a new test scenario without repeatedly implementers to describe a new test scenario without repeatedly
explaining common attributes. The structure includes a general explaining common attributes. The structure includes a general
description section that describes the test case and its motivation. description section that describes the test case and its motivation.
Additionally the test case defines a set of attributes that Additionally the test case defines a set of attributes that
characterize the testbed, for example, the network path between characterize the testbed, for example, the network path between
communicating peers and the diverse traffic sources. communicating peers and the diverse traffic sources.
skipping to change at page 3, line 44 skipping to change at page 3, line 48
indicates the minimum set of metrics (e.g., link utilization, indicates the minimum set of metrics (e.g., link utilization,
media sending rate) that a proposed algorithm needs to measure media sending rate) that a proposed algorithm needs to measure
to validate the expected rate adaptation behavior. It should to validate the expected rate adaptation behavior. It should
also indicate the time granularity (e.g., averaged over 10ms, also indicate the time granularity (e.g., averaged over 10ms,
100ms, or 1s) for measuring certain metrics. Typical 100ms, or 1s) for measuring certain metrics. Typical
measurement interval is 200ms. measurement interval is 200ms.
o Define testbed topology: every test case needs to define an o Define testbed topology: every test case needs to define an
evaluation testbed topology. Figure 1 shows such an evaluation evaluation testbed topology. Figure 1 shows such an evaluation
topology. In this evaluation topology, S1..Sn are traffic topology. In this evaluation topology, S1..Sn are traffic
sources. These sources generate media traffic and use either sources. These sources generate media traffic and use the
congestion control algorithm under investigation. R1..Rn are the congestion control algorithm(s) under investigation. R1..Rn are
corresponding receivers. A test case can have one or more such the corresponding receivers. A test case can have one or more
traffic sources (S) and their corresponding receivers (R). The such traffic sources (S) and their corresponding receivers (R).
path from the source to destination is denoted as "forward" and
the path from a destination to a source is denoted as "backward". The path from the source to destination is denoted as "forward"
The following basic structure of the test case has been described and the path from a destination to a source is denoted as
from the perspective of media generating endpoints attached on the "backward". The following basic structure of the test case has
left-hand side of Figure 1. In this setup, the media flows are been described from the perspective of media generating endpoints
transported in forward direction and corresponding feedback/ attached on the left-hand side of Figure 1. In this setup, the
control messages are transported in the backward direction. media flows are transported in forward direction and corresponding
However, it is also possible to set up the test with media in both feedback/control messages are transported in the backward
forward and backward directions. In that case, unless otherwise direction. However, it is also possible to set up the test with
specified by the test case, it is expected that the backward path media in both forward and backward directions. In that case,
does not introduce any congestion related impairments and has unless otherwise specified by the test case, it is expected that
enough capacity to accommodate both media and feedback/control the backward path does not introduce any congestion related
messages. It should be noted that depending on the test cases it impairments and has enough capacity to accommodate both media and
is possible to have different path characteristics in either of feedback/control messages. It should be noted that depending on
the directions. the test cases it is possible to have different path
characteristics in either of the directions.
+---+ +---+ +---+ +---+
|S1 |====== \ Forward --> / =======|R1 | |S1 |====== \ Forward --> / =======|R1 |
+---+ \\ // +---+ +---+ \\ // +---+
\\ // \\ //
+---+ +-----+ +-----+ +---+ +---+ +-----+ +-----+ +---+
|S2 |=======| A |------------------------------>| B |=======|R2 | |S2 |=======| A |------------------------------>| B |=======|R2 |
+---+ | |<------------------------------| | +---+ +---+ | |<------------------------------| | +---+
+-----+ +-----+ +-----+ +-----+
(...) // \\ (...) (...) // \\ (...)
// <-- Backward \\ // <-- Backward \\
+---+ // \\ +---+ +---+ // \\ +---+
|Sn |====== / \ ======|Rn | |Sn |====== / \ ======|Rn |
+---+ +---+ +---+ +---+
Figure 1: Example of A Testbed Topology Figure 1: Example of A Testbed Topology
In a testbed environment where real equipments are used to create In a testbed environment with real equipments, there may exist a
a laboratory, there may exist a significant amount of traffic on significant amount of unwanted traffic on the portions of the
portions of the network path between the endpoints that is not network path between the endpoints. Some of this traffic may be
desired for the purposes of the tests described in the document. generated by other processes on the endpoints themselves (e.g.,
Some of this traffic may be generated by other processes on the discovery protocols) or by other endpoints not presently under
endpoints themselves (e.g., discovery protocols) or by other test. Such unwanted traffic should be removed or avoided to the
endpoints not presently under test. It is recommended not to greatest extent possible.
route traffic generated by endpoints that are not under test
through the test bed and route those traffic generated by the
endpoints under test around the bottleneck links specified herein.
o Define testbed attributes: o Define testbed attributes:
* Duration: defines the duration of the test in seconds. * Duration: defines the duration of the test in seconds.
* Path characteristics: defines the end-to-end transport level * Path characteristics: defines the end-to-end transport level
path characteristics of the testbed for a particular test case. path characteristics of the testbed for a particular test case.
Two sets of attributes describe the path characteristics, one Two sets of attributes describe the path characteristics, one
for the forward path and the other for the backward path. The for the forward path and the other for the backward path. The
path characteristics for a particular path direction is path characteristics for a particular path direction is
skipping to change at page 6, line 30 skipping to change at page 6, line 30
defines the range of bit rate adaptation, the sampling defines the range of bit rate adaptation, the sampling
rate variation, and the variation in packetization rate variation, and the variation in packetization
interval. interval.
o Output variation : for a VBR encoder it defines the o Output variation : for a VBR encoder it defines the
encoder output variation from the average target rate encoder output variation from the average target rate
over a particular measurement interval. For example, over a particular measurement interval. For example,
on average the encoder output may vary between 5% to on average the encoder output may vary between 5% to
15% above or below the average target bit rate when 15% above or below the average target bit rate when
measured over a 100 ms time window. The time interval measured over a 100 ms time window. The time interval
over which the variation is specified must be over which the variation is specified MUST be
provided. provided.
o Responsiveness to a new bit rate request: the lag in o Responsiveness to a new bit rate request: the lag in
time between a new bit rate request from the time between a new bit rate request from the
congestion control algorithm and actual rate changes congestion control algorithm and actual rate changes
in encoder output. Depending on the encoder, this in encoder output. Depending on the encoder, this
value may be specified in absolute time (e.g. 10ms to value may be specified in absolute time (e.g. 10ms to
1000ms) or other appropriate metric (e.g. next frame 1000ms) or other appropriate metric (e.g. next frame
interval time). interval time).
skipping to change at page 8, line 13 skipping to change at page 8, line 13
overwritten by the respective test cases. overwritten by the respective test cases.
4.1. Evaluation metrics 4.1. Evaluation metrics
To evaluate the performance of the candidate algorithms the To evaluate the performance of the candidate algorithms the
implementers MUST log enough information to visualize the following implementers MUST log enough information to visualize the following
metrics at a fine enough time granularity: metrics at a fine enough time granularity:
1. Flow level: 1. Flow level:
A. End-to-end delay for the congestion controlled media flow. A. End-to-end delay for the congestion controlled media flow(s).
B. Variation in sending bit rate and goodput. Mainly observing B. Variation in sending bit rate and goodput. Mainly observing
the frequency and magnitude of oscillations. the frequency and magnitude of oscillations.
C. Packet losses observed at the receiving endpoint. C. Packet losses observed at the receiving endpoint.
D. Feedback message overhead. D. Feedback message overhead.
E. Convergence time - time to reach steady state for the E. Convergence time - time to reach steady state for the
congestion controlled media flow(s). congestion controlled media flow(s).
skipping to change at page 11, line 12 skipping to change at page 11, line 12
requires the algorithm to adapt the flow(s) and provide lower end-to- requires the algorithm to adapt the flow(s) and provide lower end-to-
end latency when there exists: end latency when there exists:
o an intermediate bottleneck o an intermediate bottleneck
o change in available capacity (e.g., due to interface change, o change in available capacity (e.g., due to interface change,
routing change, abrupt arrival/departure of background non- routing change, abrupt arrival/departure of background non-
adaptive traffic). adaptive traffic).
o maximum media bit rate is greater than link capacity. In this o maximum media bit rate is greater than link capacity. In this
case, the application will attempt to ramp up to its maximum bit case, when the application tries to ramp up to its maximum bit
rate, since the link capacity is limited to a value lower, the rate, since the link capacity is limited to a value lower, the
congestion control scheme is expected to stabilize the sending bit congestion control scheme is expected to stabilize the sending bit
rate close to the available bottleneck capacity. rate close to the available bottleneck capacity.
It should be noted that the exact variation in available capacity due It should be noted that the exact variation in available capacity due
to any of the above depends on the underlying technologies. Hence, to any of the above depends on the underlying technologies. Hence,
we describe a set of known factors, which may be extended to devise a we describe a set of known factors, which may be extended to devise a
more specific test case targeting certain behaviors in a certain more specific test case targeting certain behaviors in a certain
network environment. network environment.
Expected behavior: the candidate algorithm is expected to detect the Expected behavior: the candidate algorithm is expected to detect the
path capacity constraint, converges to the bottleneck link's capacity path capacity constraint, converges to the bottleneck link's capacity
and adapt the flow to avoid unwanted oscillation when the sending bit and adapt the flow to avoid unwanted media rate oscillation when the
rate is approaching the bottleneck link's capacity. The oscillations sending bit rate is approaching the bottleneck link's capacity. Such
occur when the media flow(s) attempts to reach its maximum bit rate oscillations might occur when the media flow(s) attempts to reach its
but overshoots the usage of the available bottleneck capacity then to maximum bit rate but overshoots the usage of the available bottleneck
rectify, it reduces the bit rate and starts to ramp up again. capacity then to rectify, it reduces the bit rate and starts to ramp
up again.
Evaluation metrics : as described in Section 4.1. Evaluation metrics : as described in Section 4.1.
Testbed topology: One media source S1 is connected to the Testbed topology: One media source S1 is connected to the
corresponding R1. The media traffic is transported over the forward corresponding R1. The media traffic is transported over the forward
path and corresponding feedback/control traffic is transported over path and corresponding feedback/control traffic is transported over
the backward path. the backward path.
Forward --> Forward -->
+---+ +-----+ +-----+ +---+ +---+ +-----+ +-----+ +---+
skipping to change at page 12, line 46 skipping to change at page 12, line 48
o Test Specific Information: o Test Specific Information:
* One-way propagation delay: [ 50 ms, 100 ms]. on the forward * One-way propagation delay: [ 50 ms, 100 ms]. on the forward
path direction path direction
* This test uses bottleneck path capacity variation as listed in * This test uses bottleneck path capacity variation as listed in
Table 1 Table 1
* When using background non-adaptive UDP traffic to induce time- * When using background non-adaptive UDP traffic to induce time-
varying bottleneck , the physical path capacity remains at varying bottleneck , the physical path capacity remains at
4Mbps and the UDP traffic source rate changes over time as 4Mbps and the UDP traffic source rate changes over time as (4 -
(4-x)Mbps, where x is the bottleneck capacity specified in (Y x r)), where r is the Reference bottleneck capacity in Mbps
Table 1 and Y is the path capacity ratio specified in Table 1
+--------------------+--------------+-----------+-------------------+ +--------------------+--------------+-----------+-------------------+
| Variation pattern | Path | Start | Path capacity | | Variation pattern | Path | Start | Path capacity |
| index | direction | time | ratio | | index | direction | time | ratio |
+--------------------+--------------+-----------+-------------------+ +--------------------+--------------+-----------+-------------------+
| One | Forward | 0s | 1.0 | | One | Forward | 0s | 1.0 |
| Two | Forward | 40s | 2.5 | | Two | Forward | 40s | 2.5 |
| Three | Forward | 60s | 0.6 | | Three | Forward | 60s | 0.6 |
| Four | Forward | 80s | 1.0 | | Four | Forward | 80s | 1.0 |
+--------------------+--------------+-----------+-------------------+ +--------------------+--------------+-----------+-------------------+
skipping to change at page 14, line 15 skipping to change at page 14, line 15
Testbed attributes: Testbed attributes:
Testbed attributes are similar as described in Section 5.1 except the Testbed attributes are similar as described in Section 5.1 except the
test specific capacity variation setup. test specific capacity variation setup.
Test Specific Information: This test uses path capacity variation as Test Specific Information: This test uses path capacity variation as
listed in Table 2 with a corresponding end time of 125 seconds. The listed in Table 2 with a corresponding end time of 125 seconds. The
reference bottleneck capacity is 2Mbps. When using background non- reference bottleneck capacity is 2Mbps. When using background non-
adaptive UDP traffic to induce time-varying bottleneck for congestion adaptive UDP traffic to induce time-varying bottleneck for congestion
controlled media flows, the physical path capacity is 4Mbps and the controlled media flows, the physical path capacity is 4Mbps and the
UDP traffic source rate changes over time as (4-x)Mbps, where x is UDP traffic source rate changes over time as (4 - (Y x r)), where r
the bottleneck capacity specified in Table 2. is the Reference bottleneck capacity in Mbps and Y is the path
capacity ratio specified in Table 2.
+--------------------+--------------+-----------+-------------------+ +--------------------+--------------+-----------+-------------------+
| Variation pattern | Path | Start | Path capacity | | Variation pattern | Path | Start | Path capacity |
| index | direction | time | ratio | | index | direction | time | ratio |
+--------------------+--------------+-----------+-------------------+ +--------------------+--------------+-----------+-------------------+
| One | Forward | 0s | 2.0 | | One | Forward | 0s | 2.0 |
| Two | Forward | 25s | 1.0 | | Two | Forward | 25s | 1.0 |
| Three | Forward | 50s | 1.75 | | Three | Forward | 50s | 1.75 |
| Four | Forward | 75s | 0.5 | | Four | Forward | 75s | 0.5 |
| Five | Forward | 100s | 1.0 | | Five | Forward | 100s | 1.0 |
skipping to change at page 14, line 42 skipping to change at page 14, line 43
Real-time interactive media uses RTP hence it is assumed that RTCP, Real-time interactive media uses RTP hence it is assumed that RTCP,
RTP header extension or such would be used by the congestion control RTP header extension or such would be used by the congestion control
algorithm in the backchannel. Due to asymmetric nature of the link algorithm in the backchannel. Due to asymmetric nature of the link
between communicating peers it is possible for a participating peer between communicating peers it is possible for a participating peer
to not receive such feedback information due to an impaired or to not receive such feedback information due to an impaired or
congested backchannel (even when the forward channel might not be congested backchannel (even when the forward channel might not be
impaired). This test case is designed to observe the candidate impaired). This test case is designed to observe the candidate
congestion control behavior in such an event. congestion control behavior in such an event.
It is expected that the candidate algorithms are able to cope with Expected behavior: It is expected that the candidate algorithms are
the lack of feedback information and adapt to minimize the able to cope with the lack of feedback information and adapt to
performance degradation of media flows in the forward channel. minimize the performance degradation of media flows in the forward
channel.
It should be noted that for this test case: logs are compared with It should be noted that for this test case: logs are compared with
the reference case, i.e, when the backward channel has no the reference case, i.e, when the backward channel has no
impairments. impairments.
Evaluation metrics : as described in Section 4.1. Evaluation metrics : as described in Section 4.1.
Testbed topology: One (1) media source S1 is connected to Testbed topology: One (1) media source S1 is connected to
corresponding R1, but both endpoints are additionally receiving and corresponding R1, but both endpoints are additionally receiving and
sending data, respectively. The media traffic (S1->R1) is sending data, respectively. The media traffic (S1->R1) is
skipping to change at page 17, line 7 skipping to change at page 17, line 7
+--------------------+--------------+-----------+-------------------+ +--------------------+--------------+-----------+-------------------+
| One | Backward | 0s | 2.0 | | One | Backward | 0s | 2.0 |
| Two | Backward | 35s | 0.8 | | Two | Backward | 35s | 0.8 |
| Three | Backward | 70s | 2.0 | | Three | Backward | 70s | 2.0 |
+--------------------+--------------+-----------+-------------------+ +--------------------+--------------+-----------+-------------------+
Table 4: Path capacity variation pattern for backward direction Table 4: Path capacity variation pattern for backward direction
5.4. Competing Media Flows with same Congestion Control Algorithm 5.4. Competing Media Flows with same Congestion Control Algorithm
In this test case, more than one media flows share the bottleneck In this test case, more than one media flow share the bottleneck link
link and each of them uses the same congestion control algorithm. and each of them uses the same congestion control algorithm. This is
This is a typical scenario where a real-time interactive application a typical scenario where a real-time interactive application sends
sends more than one media flow to the same destination and these more than one media flow to the same destination and these flows are
flows are multiplexed over the same port. In such a scenario it is multiplexed over the same port. In such a scenario it is likely that
likely that the flows will be routed via the same path and need to the flows will be routed via the same path and need to share the
share the available bandwidth amongst themselves. For the sake of available bandwidth amongst themselves. For the sake of simplicity
simplicity it is assumed that there are no other competing traffic it is assumed that there are no other competing traffic sources in
sources in the bottleneck link and that there is sufficient capacity the bottleneck link and that there is sufficient capacity to
to accommodate all the flows individually. While this appears to be accommodate all the flows individually. While this appears to be a
a variant of the test case defined in Section 5.2, it focuses on the variant of the test case defined in Section 5.2, it focuses on the
capacity sharing aspect of the candidate algorithm. The previous capacity sharing aspect of the candidate algorithm. The previous
test case, on the other hand, measures adaptability, stability, and test case, on the other hand, measures adaptability, stability, and
responsiveness of the candidate algorithm. responsiveness of the candidate algorithm.
Expected behavior: It is expected that the competing flows will Expected behavior: It is expected that the competing flows will
converge to an optimum bit rate to accommodate all the flows with converge to an optimum bit rate to accommodate all the flows with
minimum possible latency and loss. Specifically, the test introduces minimum possible latency and loss. Specifically, the test introduces
three media flows at different time instances, when the second flow three media flows at different time instances, when the second flow
appears there should still be room to accommodate another flow on the appears there should still be room to accommodate another flow on the
bottleneck link. Lastly, when the third flow appears the bottleneck bottleneck link. Lastly, when the third flow appears the bottleneck
skipping to change at page 19, line 21 skipping to change at page 19, line 21
| 4 | Audio | 0s | 119s | | 4 | Audio | 0s | 119s |
| 5 | Audio | 20s | 119s | | 5 | Audio | 20s | 119s |
| 6 | Audio | 40s | 119s | | 6 | Audio | 40s | 119s |
+---------+------------+------------+----------+ +---------+------------+------------+----------+
Table 5: Media Timeline for Video and Audio media sources Table 5: Media Timeline for Video and Audio media sources
5.5. Round Trip Time Fairness 5.5. Round Trip Time Fairness
In this test case, multiple media flows share the bottleneck link, In this test case, multiple media flows share the bottleneck link,
but the end-to-end path latency for each flow is different. For the but the one-way propagation delay for each flow is different. For
sake of simplicity it is assumed that there are no other competing the sake of simplicity it is assumed that there are no other
traffic sources in the bottleneck link and that there is sufficient competing traffic sources in the bottleneck link and that there is
capacity to accommodate all the flows. While this appears to be a sufficient capacity to accommodate all the flows. While this appears
variant of test case 5.2, it focuses on the capacity sharing aspect to be a variant of test case 5.2, it focuses on the capacity sharing
of the candidate algorithm under different RTTs. aspect of the candidate algorithm under different RTTs.
It is expected that the competing flows will converge to bit rates to Expected behavior: It is expected that the competing flows will
accommodate all the flows with minimum possible latency and loss. converge to bit rates to accommodate all the flows with minimum
Specifically, the test introduces five media flows at the same time possible latency and loss. Specifically, the test introduces five
instance. media flows at the same time instance.
Evaluation metrics : as described in Section 4.1. Evaluation metrics : as described in Section 4.1.
Testbed Topology: Five (5) media sources S1,S2,..,S5 are connected to Testbed Topology: Five (5) media sources S1,S2,..,S5 are connected to
their corresponding media sinks R1,R2,..,R5. The media traffic is their corresponding media sinks R1,R2,..,R5. The media traffic is
transported over the forward path and corresponding feedback/control transported over the forward path and corresponding feedback/control
traffic is transported over the backward path. The topology is the traffic is transported over the backward path. The topology is the
same as in Section 5.4. The end-to-end path delays are: 10ms for same as in Section 5.4.
S1-R1, 25ms for S2-R2, 50ms for S3-R3, 100ms for S4-R4, and 150ms
S5-R5, respectively.
Testbed attributes: Testbed attributes:
o Test duration: 300s o Test duration: 300s
o Path characteristics: o Path characteristics:
* One-Way propagation delay for each flow: 10ms, 25ms, 50ms, * Reference bottleneck capacity: 4Mbps
100ms, 150ms.
* Path capacity ratio: 1.0
* One-Way propagation delay for each flow: 10ms for S1-R1, 25ms
for S2-R2, 50ms for S3-R3, 100ms for S4-R4, and 150ms S5-R5.
o Application-related: o Application-related:
* Media Source: * Media Source:
+ Media type: Video + Media type: Video
- Media direction: forward - Media direction: forward
- Number of media sources: five (5) - Number of media sources: five (5)
skipping to change at page 23, line 36 skipping to change at page 23, line 36
- End time: 119s. - End time: 119s.
o Test Specific Information: none o Test Specific Information: none
5.7. Media Flow Competing with Short TCP Flows 5.7. Media Flow Competing with Short TCP Flows
In this test case, one or more congestion controlled media flow In this test case, one or more congestion controlled media flow
shares the bottleneck link with multiple short-lived TCP flows. shares the bottleneck link with multiple short-lived TCP flows.
Short-lived TCP flows resemble the on/off pattern observed in the web Short-lived TCP flows resemble the on/off pattern observed in the web
traffic, wherein clients (browsers) connect to a server and download traffic, wherein clients ( for example -browsers) connect to a server
a resource (typically a web page, few images, text files, etc.) using and download a resource (typically a web page, few images, text
several TCP connections (up to 4). This scenario shows the files, etc.) using several TCP connections. This scenario shows the
performance of a multimedia application when several browser windows performance of a multimedia application when several browser windows
are active. The test case measures the adaptivity of the candidate are active. The test case measures the adaptivity of the candidate
algorithm to competing web traffic, it addresses the requirements 1.E algorithm to competing web traffic, it addresses the requirements 1.E
in [I-D.ietf-rmcat-cc-requirements]. in [I-D.ietf-rmcat-cc-requirements].
Depending on the number of short TCP flows, the cross-traffic either Depending on the number of short TCP flows, the cross-traffic either
appears as a short burst flow or resembles a long TCP flow. The appears as a short burst flow or resembles a long TCP flow. The
intention of this test is to observe the impact of short-term burst intention of this test is to observe the impact of short-term burst
on the behavior of the candidate algorithm. on the behavior of the candidate algorithm.
Expected behavior: The candidate algorithm is expected to avoid flow
starvation during the presence of short and bursty competing TCP
flows, streaming at least at the minimum media bit rate. After
competing TCP flows terminate, the media streams are expected to be
robust enough to eventually recover to previous steady state
behavior, and at the very least, avoid persistent starvation.
Evaluation metrics : following metrics in addition to as described in Evaluation metrics : following metrics in addition to as described in
Section 4.1. Section 4.1.
1. Flow level: 1. Flow level:
A. Variation in the sending rate of the TCP flow. A. Variation in the sending rate of the TCP flow.
B. TCP throughput. B. TCP throughput.
Testbed topology: The topology described here is same as the one Testbed topology: The topology described here is same as the one
skipping to change at page 25, line 17 skipping to change at page 25, line 24
+ Traffic direction : forward + Traffic direction : forward
+ Congestion algorithm: default TCP Congestion control + Congestion algorithm: default TCP Congestion control
[RFC5681]. [RFC5681].
+ Traffic timeline: each short TCP flow is modeled as a + Traffic timeline: each short TCP flow is modeled as a
sequence of file downloads interleaved with idle periods. sequence of file downloads interleaved with idle periods.
Not all short TCP flows start at the same time, 2 of them Not all short TCP flows start at the same time, 2 of them
start in the ON state while rest of the 8 flows start in an start in the ON state while rest of the 8 flows start in an
OFF stats. For description of short TCP flow model see test OFF state. For description of short TCP flow model see test
specific information below. specific information below.
o Test Specific Information: o Test Specific Information:
* Short-TCP traffic model: The short TCP model to be used in this * Short-TCP traffic model: The short TCP model to be used in this
test is described in [I-D.ietf-rmcat-eval-criteria]. test is described in [I-D.ietf-rmcat-eval-criteria].
5.8. Media Pause and Resume 5.8. Media Pause and Resume
In this test case, more than one real-time interactive media flows In this test case, more than one real-time interactive media flows
skipping to change at page 25, line 46 skipping to change at page 26, line 5
real-time interactive media which consists of more than one media real-time interactive media which consists of more than one media
flows and can pause/resume media flows at any point of time during flows and can pause/resume media flows at any point of time during
the session. This test case directly addresses the requirement the session. This test case directly addresses the requirement
number 5 in [I-D.ietf-rmcat-cc-requirements]. One can think it as a number 5 in [I-D.ietf-rmcat-cc-requirements]. One can think it as a
variation of test case defined in Section 5.4. However, it is variation of test case defined in Section 5.4. However, it is
different as the candidate algorithms can use different strategies to different as the candidate algorithms can use different strategies to
increase its efficiency, for example in terms of fairness, increase its efficiency, for example in terms of fairness,
convergence time, reduce oscillation etc, by capitalizing the fact convergence time, reduce oscillation etc, by capitalizing the fact
that they have previous information of the link. that they have previous information of the link.
Expected behavior: During the period where the third stream is
paused, the two remaining flows are expected to increase their rates
and reach the maximum media bit rate. When the third stream resumes,
all three flows are expected to converge to the same original fair
share of rates prior to the media pause/resume event.
Evaluation metrics : following metrics in addition to as described in Evaluation metrics : following metrics in addition to as described in
Section 4.1. Section 4.1.
1. Flow level: 1. Flow level:
A. Variation in sending bit rate and goodput. Mainly observing A. Variation in sending bit rate and goodput. Mainly observing
the frequency and magnitude of oscillations. the frequency and magnitude of oscillations.
Testbed Topology: Same as test case defined in Section 5.4 Testbed Topology: Same as test case defined in Section 5.4
skipping to change at page 27, line 25 skipping to change at page 27, line 35
This will be an extension of Section 5.4 where the same test will be This will be an extension of Section 5.4 where the same test will be
run with different priority levels imposed on each of the media run with different priority levels imposed on each of the media
flows. For example, the first flow (S1) is assigned a priority of 2 flows. For example, the first flow (S1) is assigned a priority of 2
whereas the remaining two flows (S2 and S3) are assigned a priority whereas the remaining two flows (S2 and S3) are assigned a priority
of 1. The candidate algorithm MUST reflect the relative priorities of 1. The candidate algorithm MUST reflect the relative priorities
assigned to each media flow. In the previous example, the first flow assigned to each media flow. In the previous example, the first flow
(S1) MUST arrive at a steady-state rate approximately twice of that (S1) MUST arrive at a steady-state rate approximately twice of that
of the other two flows (S2 and S3). of the other two flows (S2 and S3).
The candidate algorithm can use a coupled congestion control The candidate algorithm can use a coupled congestion control
mechanism for the bandwidth distribution according to the respective mechanism or use a weighted priority scheduler for the bandwidth
media flow priority. distribution according to the respective media flow priority or use.
6.2. Explicit Congestion Notification Usage 6.2. Explicit Congestion Notification Usage
This test case requires to run all the basic test cases with the This test case requires to run all the basic test cases with the
availability of Explicit Congestion Notification (ECN) [RFC6679] availability of Explicit Congestion Notification (ECN) [RFC6679]
feature enabled. The goal of this test is to exhibit that the feature enabled. The goal of this test is to exhibit that the
candidate algorithms do not fail when ECN signals are available. candidate algorithms do not fail when ECN signals are available.
With ECN signals enabled the algorithms are expected to perform With ECN signals enabled the algorithms are expected to perform
better than their delay based variants. better than their delay based variants.
6.3. Multiple Bottlenecks 6.3. Multiple Bottlenecks
In this test case one congestion controlled media flow, S1->R2, In this test case one congestion controlled media flow, S1->R1,
traverses a path with multiple bottlenecks. As illustrated in traverses a path with multiple bottlenecks. As illustrated in
Figure 7, the first flow (S1->R1) competes with the second congestion Figure 7, the first flow (S1->R1) competes with the second congestion
controlled media flow (S2->R2) over the link between A and B which is controlled media flow (S2->R2) over the link between A and B which is
close to the sender side; again, that flow (S1->R1) competes with the close to the sender side; again, that flow (S1->R1) competes with the
third congestion controlled media flow (S3->R3) over the link between third congestion controlled media flow (S3->R3) over the link between
C and D which is close to the receiver side. The goal of this test C and D which is close to the receiver side. The goal of this test
is to ensure that the candidate algorithms work properly in the is to ensure that the candidate algorithms work properly in the
presence of multiple bottleneck links on the end to end path. presence of multiple bottleneck links on the end to end path.
Expected behavior: the candidate algorithm is expected to achieve Expected behavior: the candidate algorithm is expected to achieve
skipping to change at page 29, line 45 skipping to change at page 30, line 12
+ Number of sources : Zero (0) + Number of sources : Zero (0)
7. Wireless Access Links 7. Wireless Access Links
Additional wireless network (both cellular network and WiFi network) Additional wireless network (both cellular network and WiFi network)
specific test cases are defined in [I-D.ietf-rmcat-wireless-tests]. specific test cases are defined in [I-D.ietf-rmcat-wireless-tests].
8. Security Considerations 8. Security Considerations
Security issues have not been discussed in this memo. The evaluations of the test cases are intended to run in a controlled
lab environment. Hence, the applications, simulators and network
nodes should be well-behaved and should not impact the desired
results. In case the evaluations are not done in a controlled
environment, the security considerations in
[I-D.ietf-rmcat-eval-criteria] and the relevant congestion control
algorithms apply. It is important to take appropriate caution to
avoid leaking non-responsive traffic from unproven congestion
avoidance techniques onto the open Internet.
9. IANA Considerations 9. IANA Considerations
There are no IANA impacts in this memo. There are no IANA impacts in this memo.
10. Acknowledgements 10. Acknowledgements
Much of this document is derived from previous work on congestion Much of this document is derived from previous work on congestion
control at the IETF. control at the IETF.
skipping to change at page 30, line 23 skipping to change at page 30, line 44
11. References 11. References
11.1. Normative References 11.1. Normative References
[I-D.ietf-rmcat-eval-criteria] [I-D.ietf-rmcat-eval-criteria]
Singh, V., Ott, J., and S. Holmer, "Evaluating Congestion Singh, V., Ott, J., and S. Holmer, "Evaluating Congestion
Control for Interactive Real-time Media", draft-ietf- Control for Interactive Real-time Media", draft-ietf-
rmcat-eval-criteria-07 (work in progress), May 2018. rmcat-eval-criteria-07 (work in progress), May 2018.
[I-D.ietf-rmcat-video-traffic-model] [I-D.ietf-rmcat-video-traffic-model]
Zhu, X., Cruz, S., and Z. Sarker, "Modeling Video Traffic Zhu, X., Cruz, S., and Z. Sarker, "Video Traffic Models
Sources for RMCAT Evaluations", draft-ietf-rmcat-video- for RTP Congestion Control Evaluations", draft-ietf-rmcat-
traffic-model-04 (work in progress), January 2018. video-traffic-model-05 (work in progress), July 2018.
[I-D.ietf-rmcat-wireless-tests] [I-D.ietf-rmcat-wireless-tests]
Sarker, Z., Johansson, I., Zhu, X., Fu, J., Tan, W., and Sarker, Z., Johansson, I., Zhu, X., Fu, J., Tan, W., and
M. Ramalho, "Evaluation Test Cases for Interactive Real- M. Ramalho, "Evaluation Test Cases for Interactive Real-
Time Media over Wireless Networks", draft-ietf-rmcat- Time Media over Wireless Networks", draft-ietf-rmcat-
wireless-tests-04 (work in progress), May 2017. wireless-tests-05 (work in progress), June 2018.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
<https://www.rfc-editor.org/info/rfc2119>.
[RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V. [RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V.
Jacobson, "RTP: A Transport Protocol for Real-Time Jacobson, "RTP: A Transport Protocol for Real-Time
Applications", STD 64, RFC 3550, DOI 10.17487/RFC3550, Applications", STD 64, RFC 3550, DOI 10.17487/RFC3550,
July 2003, <https://www.rfc-editor.org/info/rfc3550>. July 2003, <https://www.rfc-editor.org/info/rfc3550>.
[RFC3551] Schulzrinne, H. and S. Casner, "RTP Profile for Audio and [RFC3551] Schulzrinne, H. and S. Casner, "RTP Profile for Audio and
Video Conferences with Minimal Control", STD 65, RFC 3551, Video Conferences with Minimal Control", STD 65, RFC 3551,
DOI 10.17487/RFC3551, July 2003, DOI 10.17487/RFC3551, July 2003,
<https://www.rfc-editor.org/info/rfc3551>. <https://www.rfc-editor.org/info/rfc3551>.
 End of changes. 32 change blocks. 
99 lines changed or deleted 132 lines changed or added

This html diff was produced by rfcdiff 1.47. The latest version is available from http://tools.ietf.org/tools/rfcdiff/