draft-ietf-rmcat-eval-test-02.txt | draft-ietf-rmcat-eval-test-03.txt | |||
---|---|---|---|---|
Network Working Group Z. Sarker | Network Working Group Z. Sarker | |||
Internet-Draft Ericsson AB | Internet-Draft Ericsson AB | |||
Intended status: Informational V. Singh | Intended status: Informational V. Singh | |||
Expires: March 11, 2016 Aalto University | Expires: September 9, 2016 callstats.io | |||
X. Zhu | X. Zhu | |||
M. Ramalho | M. Ramalho | |||
Cisco Systems | Cisco Systems | |||
September 8, 2015 | March 08, 2016 | |||
Test Cases for Evaluating RMCAT Proposals | Test Cases for Evaluating RMCAT Proposals | |||
draft-ietf-rmcat-eval-test-02 | draft-ietf-rmcat-eval-test-03 | |||
Abstract | Abstract | |||
The Real-time Transport Protocol (RTP) is used to transmit media in | The Real-time Transport Protocol (RTP) is used to transmit media in | |||
multimedia telephony applications, these applications are typically | multimedia telephony applications, these applications are typically | |||
required to implement congestion control. The RMCAT working group is | required to implement congestion control. This document describes | |||
currently working on candidate algorithms for such interactive real- | the test cases to be used in the performance evaluation of such | |||
time multimedia applications. This document describes the test cases | congestion control algorithms. | |||
to be used in the performance evaluation of those candidate | ||||
algorithms. | ||||
Status of This Memo | Status of This Memo | |||
This Internet-Draft is submitted in full conformance with the | This Internet-Draft is submitted in full conformance with the | |||
provisions of BCP 78 and BCP 79. | provisions of BCP 78 and BCP 79. | |||
Internet-Drafts are working documents of the Internet Engineering | Internet-Drafts are working documents of the Internet Engineering | |||
Task Force (IETF). Note that other groups may also distribute | Task Force (IETF). Note that other groups may also distribute | |||
working documents as Internet-Drafts. The list of current Internet- | working documents as Internet-Drafts. The list of current Internet- | |||
Drafts is at http://datatracker.ietf.org/drafts/current/. | Drafts is at http://datatracker.ietf.org/drafts/current/. | |||
Internet-Drafts are draft documents valid for a maximum of six months | Internet-Drafts are draft documents valid for a maximum of six months | |||
and may be updated, replaced, or obsoleted by other documents at any | and may be updated, replaced, or obsoleted by other documents at any | |||
time. It is inappropriate to use Internet-Drafts as reference | time. It is inappropriate to use Internet-Drafts as reference | |||
material or to cite them other than as "work in progress." | material or to cite them other than as "work in progress." | |||
This Internet-Draft will expire on March 11, 2016. | This Internet-Draft will expire on September 9, 2016. | |||
Copyright Notice | Copyright Notice | |||
Copyright (c) 2015 IETF Trust and the persons identified as the | Copyright (c) 2016 IETF Trust and the persons identified as the | |||
document authors. All rights reserved. | document authors. All rights reserved. | |||
This document is subject to BCP 78 and the IETF Trust's Legal | This document is subject to BCP 78 and the IETF Trust's Legal | |||
Provisions Relating to IETF Documents | Provisions Relating to IETF Documents | |||
(http://trustee.ietf.org/license-info) in effect on the date of | (http://trustee.ietf.org/license-info) in effect on the date of | |||
publication of this document. Please review these documents | publication of this document. Please review these documents | |||
carefully, as they describe your rights and restrictions with respect | carefully, as they describe your rights and restrictions with respect | |||
to this document. Code Components extracted from this document must | to this document. Code Components extracted from this document must | |||
include Simplified BSD License text as described in Section 4.e of | include Simplified BSD License text as described in Section 4.e of | |||
the Trust Legal Provisions and are provided without warranty as | the Trust Legal Provisions and are provided without warranty as | |||
described in the Simplified BSD License. | described in the Simplified BSD License. | |||
Table of Contents | Table of Contents | |||
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 | 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 | |||
2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3 | 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3 | |||
3. Structure of Test cases . . . . . . . . . . . . . . . . . . . 3 | 3. Structure of Test cases . . . . . . . . . . . . . . . . . . . 3 | |||
4. Recommended Evaluation Settings . . . . . . . . . . . . . . . 7 | 4. Recommended Evaluation Settings . . . . . . . . . . . . . . . 7 | |||
4.1. Evaluation metircs . . . . . . . . . . . . . . . . . . . 7 | 4.1. Evaluation metrics . . . . . . . . . . . . . . . . . . . 8 | |||
4.2. Path characteristics . . . . . . . . . . . . . . . . . . 8 | 4.2. Path characteristics . . . . . . . . . . . . . . . . . . 8 | |||
4.3. Media source . . . . . . . . . . . . . . . . . . . . . . 9 | 4.3. Media source . . . . . . . . . . . . . . . . . . . . . . 9 | |||
5. Basic Test Cases . . . . . . . . . . . . . . . . . . . . . . 10 | 5. Basic Test Cases . . . . . . . . . . . . . . . . . . . . . . 10 | |||
5.1. Variable Available Capacity with Single RMCAT flow . . . 10 | 5.1. Variable Available Capacity with a Single Flow . . . . . 10 | |||
5.2. Variable Available Capacity with Multiple RMCAT flows . . 13 | 5.2. Variable Available Capacity with Multiple Flows . . . . . 13 | |||
5.3. Congested Feedback Link with Bi-directional RMCAT flows . 14 | 5.3. Congested Feedback Link with Bi-directional Media Flows . 14 | |||
5.4. Competing Flows with Same RMCAT Algorithm . . . . . . . . 16 | 5.4. Competing Media Flows with same Congestion Control | |||
Algorithm . . . . . . . . . . . . . . . . . . . . . . . . 17 | ||||
5.5. Round Trip Time Fairness . . . . . . . . . . . . . . . . 19 | 5.5. Round Trip Time Fairness . . . . . . . . . . . . . . . . 19 | |||
5.6. RMCAT Flow competing with a long TCP Flow . . . . . . . . 20 | 5.6. Media Flow Competing with a Long TCP Flow . . . . . . . . 21 | |||
5.7. RMCAT Flow competing with short TCP Flows . . . . . . . . 23 | 5.7. Media Flow Competing with Short TCP Flows . . . . . . . . 23 | |||
5.8. Media Pause and Resume . . . . . . . . . . . . . . . . . 25 | 5.8. Media Pause and Resume . . . . . . . . . . . . . . . . . 25 | |||
6. Other potential test cases . . . . . . . . . . . . . . . . . 26 | 6. Other potential test cases . . . . . . . . . . . . . . . . . 27 | |||
6.1. Explicit Congestion Notification Usage . . . . . . . . . 26 | 6.1. Media Flows with Priority . . . . . . . . . . . . . . . . 27 | |||
6.2. Multiple Bottlenecks . . . . . . . . . . . . . . . . . . 26 | 6.2. Explicit Congestion Notification Usage . . . . . . . . . 27 | |||
6.3. Multiple Bottlenecks . . . . . . . . . . . . . . . . . . 27 | ||||
7. Wireless Access Links . . . . . . . . . . . . . . . . . . . . 29 | 7. Wireless Access Links . . . . . . . . . . . . . . . . . . . . 29 | |||
8. Security Considerations . . . . . . . . . . . . . . . . . . . 29 | 8. Security Considerations . . . . . . . . . . . . . . . . . . . 30 | |||
9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 29 | 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 30 | |||
10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 29 | 10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 30 | |||
11. References . . . . . . . . . . . . . . . . . . . . . . . . . 29 | 11. References . . . . . . . . . . . . . . . . . . . . . . . . . 30 | |||
11.1. Normative References . . . . . . . . . . . . . . . . . . 29 | 11.1. Normative References . . . . . . . . . . . . . . . . . . 30 | |||
11.2. Informative References . . . . . . . . . . . . . . . . . 30 | 11.2. Informative References . . . . . . . . . . . . . . . . . 31 | |||
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 30 | Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 31 | |||
1. Introduction | 1. Introduction | |||
This memo describes a set of test cases for evaluating candidate | This memo describes a set of test cases for evaluating congestion | |||
RMCAT congestion control algorithm proposals, it is based on the | control algorithm proposals for real-time interactive media. It is | |||
guidelines enumerated in [I-D.ietf-rmcat-eval-criteria] and the | based on the guidelines enumerated in [I-D.ietf-rmcat-eval-criteria] | |||
requirements discussed in [I-D.ietf-rmcat-cc-requirements]. The test | and the requirements discussed in [I-D.ietf-rmcat-cc-requirements]. | |||
cases cover basic usage scenarios and are described using a common | The test cases cover basic usage scenarios and are described using a | |||
structure, which allows for additional test cases to be added to | common structure, which allows for additional test cases to be added | |||
those described herein to accommodate other topologies and/or the | to those described herein to accommodate other topologies and/or the | |||
modeling of different path characteristics. It is the intention of | modeling of different path characteristics. The described test cases | |||
this work to capture the consensus of the RMCAT working group | in this memo SHOULD be used to evaluate any proposed congestion | |||
participants regarding the test cases upon which the performance of | control algorithm for real-time interactive media. | |||
the candidate RMCAT proposals should be evaluated. | ||||
2. Terminology | 2. Terminology | |||
The terminology defined in RTP [RFC3550], RTP Profile for Audio and | The terminology defined in RTP [RFC3550], RTP Profile for Audio and | |||
Video Conferences with Minimal Control [RFC3551], RTCP Extended | Video Conferences with Minimal Control [RFC3551], RTCP Extended | |||
Report (XR) [RFC3611], Extended RTP Profile for RTCP-based Feedback | Report (XR) [RFC3611], Extended RTP Profile for RTCP-based Feedback | |||
(RTP/AVPF) [RFC4585], and Support for Reduced-Size RTCP [RFC5506] | (RTP/AVPF) [RFC4585], and Support for Reduced-Size RTCP [RFC5506] | |||
apply. | apply. | |||
3. Structure of Test cases | 3. Structure of Test cases | |||
All test cases in this document follow a basic structure allowing | All the test cases in this document follow a basic structure allowing | |||
implementers to describe a new test scenario without repeatedly | implementers to describe a new test scenario without repeatedly | |||
explaining common attributes. The structure includes a general | explaining common attributes. The structure includes a general | |||
description section that describes the test case and its motivation. | description section that describes the test case and its motivation. | |||
Additionally the test case defines a set of attributes that | Additionally the test case defines a set of attributes that | |||
characterize the testbed, i.e., the network path between | characterize the testbed, for example, the network path between | |||
communicating peers and the diverse traffic sources. | communicating peers and the diverse traffic sources. | |||
o Define the test case: | o Define the test case: | |||
* General description: describes the motivation and the goals of | * General description: describes the motivation and the goals of | |||
the test case. | the test case. | |||
* Expected behavior: describe the desired rate adaptation | * Expected behavior: describes the desired rate adaptation | |||
behaviour. | behavior. | |||
* Define a check-list to evaluate the desired behaviour: this | * Define a list of metrics to evaluate the desired behavior: this | |||
indicates the minimum set of metrics (e.g., link utilization, | indicates the minimum set of metrics (e.g., link utilization, | |||
media sending rate) that a proposed algorithm needs to measure | media sending rate) that a proposed algorithm needs to measure | |||
to validate the expected rate adaptation behaviour. It should | to validate the expected rate adaptation behavior. It should | |||
also indicate the time granularity (e.g., averaged over 10ms, | also indicate the time granularity (e.g., averaged over 10ms, | |||
100ms, or 1s) for measuring certain metrics. Typical | 100ms, or 1s) for measuring certain metrics. Typical | |||
measurement interval is 200ms. | measurement interval is 200ms. | |||
o Define testbed topology: every test case needs to define an | o Define testbed topology: every test case needs to define an | |||
evaluation testbed topology. Figure 1 shows such an evaluation | evaluation testbed topology. Figure 1 shows such an evaluation | |||
topology. In this evaluation topology, S1..Sn are traffic | topology. In this evaluation topology, S1..Sn are traffic | |||
sources. These sources generate media traffic and use either an | sources. These sources generate media traffic and use either | |||
RMCAT candidate congestion control algorithm or other congestion | congestion control algorithm under investigation. R1..Rn are the | |||
control algorithm designed for media, such as TFRC. R1..Rn are | corresponding receivers. A test case can have one or more such | |||
the corresponding receivers. A test case can have one or more | traffic sources (S) and their corresponding receivers (R). The | |||
such traffic sources (S) and corresponding receivers (R). The | path from the source to destination is denoted as "forward" and | |||
path from the source to destination is denoted as forward and the | the path from a destination to a source is denoted as "backward". | |||
path from a destination to a source is denoted as backward. The | The following basic structure of the test case has been described | |||
following basic structure of test case has been described from the | from the perspective of media generating endpoints attached on the | |||
perspective of media generating endpoints attached on the left- | left-hand side of Figure 1. In this setup, the media flows are | |||
hand side of Figure 1. In this setup, media flows in forward | transported in forward direction and corresponding feedback/ | |||
direction and corresponding feedback/control messages flow in the | control messages are transported in the backward direction. | |||
backward direction. However, it is also possible to set up the | However, it is also possible to set up the test with media in both | |||
test with media flowing in both forward and backward directions. | forward and backward directions. In that case, unless otherwise | |||
In that case, unless otherwise specified by the test case, it is | specified by the test case, it is expected that the backward path | |||
expected that the backward path does not introduce any congestion | does not introduce any congestion related impairments and has | |||
related impairments and has enough capacity to accommodate both | enough capacity to accommodate both media and feedback/control | |||
media and feedback/control messages. It should be noted that | messages. It should be noted that depending on the test cases it | |||
depending on the test cases it is possible to have different path | is possible to have different path characteristics in either of | |||
characteristics in of the either directions. | the directions. | |||
o | ||||
+---+ +---+ | +---+ +---+ | |||
|S1 |====== \ Forward --> / =======|R1 | | |S1 |====== \ Forward --> / =======|R1 | | |||
+---+ \\ // +---+ | +---+ \\ // +---+ | |||
\\ // | \\ // | |||
+---+ +-----+ +-----+ +---+ | +---+ +-----+ +-----+ +---+ | |||
|S2 |=======| A |------------------------------>| B |=======|R2 | | |S2 |=======| A |------------------------------>| B |=======|R2 | | |||
+---+ | |<------------------------------| | +---+ | +---+ | |<------------------------------| | +---+ | |||
+-----+ +-----+ | +-----+ +-----+ | |||
(...) // \\ (...) | (...) // \\ (...) | |||
// <-- Backward \\ | // <-- Backward \\ | |||
+---+ // \\ +---+ | +---+ // \\ +---+ | |||
|Sn |====== / \ ======|Rn | | |Sn |====== / \ ======|Rn | | |||
+---+ +---+ | +---+ +---+ | |||
Figure 1: Example of A Testbed Topology | Figure 1: Example of A Testbed Topology | |||
In a laboratory testbed environment there may exist a significant | In a testbed environment where real equipments are used to create | |||
amount of traffic on portions of the network path between the | a laboratory, there may exist a significant amount of traffic on | |||
endpoints that is not desired for the purposes of these RMCAT | portions of the network path between the endpoints that is not | |||
tests. Some of this traffic may be generated by other processes | desired for the purposes of the tests described in the document. | |||
on the endpoints themselves (e.g., discovery protocols) or by | Some of this traffic may be generated by other processes on the | |||
other endpoints not presently under test. It is recommended not | endpoints themselves (e.g., discovery protocols) or by other | |||
to route traffic generated by endpoints that are not under test | endpoints not presently under test. It is recommended not to | |||
through the test bed. Additionally, it is recommended to route | route traffic generated by endpoints that are not under test | |||
non-RMCAT traffic generated by the endpoints under test around the | through the test bed and route those traffic generated by the | |||
bottleneck links specified herein. | endpoints under test around the bottleneck links specified herein. | |||
o Define testbed attributes: | o Define testbed attributes: | |||
* Duration: defines the duration of the test. | * Duration: defines the duration of the test in seconds. | |||
* Path characteristics: defines the end-to-end transport level | * Path characteristics: defines the end-to-end transport level | |||
path characteristics of the testbed in a particular test case. | path characteristics of the testbed for a particular test case. | |||
Two sets of attributes describe the path characteristics, one | Two sets of attributes describe the path characteristics, one | |||
for the forward path and the other for the backward path. The | for the forward path and the other for the backward path. The | |||
path characteristics for a particular path direction is | path characteristics for a particular path direction is | |||
applicable to all the Sources "S" sending traffic on that path. | applicable to all the Sources "S" sending traffic on that path. | |||
If only one attribute is specified, it is used for both path | If only one attribute is specified, it is used for both path | |||
directions, however, unless specified the reverse path has no | directions, however, unless specified the reverse path has no | |||
capacity restrictions and no path loss. | capacity restrictions and no path loss. | |||
+ Path direction: forward or backward. | + Path direction: forward or backward. | |||
skipping to change at page 5, line 33 ¶ | skipping to change at page 5, line 31 ¶ | |||
along the path when network queues are empty, i.e., the time | along the path when network queues are empty, i.e., the time | |||
it takes for a packet to go from the sender to the receiver | it takes for a packet to go from the sender to the receiver | |||
without encountering any queuing delay. | without encountering any queuing delay. | |||
+ Maximum end-to-end jitter: defines the maximum jitter that | + Maximum end-to-end jitter: defines the maximum jitter that | |||
can be observed along the path. | can be observed along the path. | |||
+ Bottleneck queue type: for example, Droptail, FQ-CoDel, or | + Bottleneck queue type: for example, Droptail, FQ-CoDel, or | |||
PIE. | PIE. | |||
+ Bottleneck queue size: defines size of queue in terms of | + Bottleneck queue size: defines the size of queue in terms of | |||
queuing time when the queue is full (in milliseconds). | queuing time when the queue is full (in milliseconds). | |||
+ Path loss ratio: characterizes the non-congested, additive, | + Path loss ratio: characterizes the non-congested, additive, | |||
losses to be generated on the end-to-end path. MUST | losses to be generated on the end-to-end path. MUST | |||
describe the loss pattern or loss model used to generate the | describe the loss pattern or loss model used to generate the | |||
losses. | losses. | |||
+ Values for some characteristics are described in | * Application-related: defines the traffic source behavior for | |||
[I-D.ietf-rmcat-eval-criteria]. | ||||
* Application-related: defines the traffic source behaviour for | ||||
implementing the test case | implementing the test case | |||
+ Media traffic Source: defines the characteristics of the | + Media traffic Source: defines the characteristics of the | |||
media sources. When using more than one media source, the | media sources. When using more than one media source, the | |||
different attributes are enumerated separately for each | different attributes are enumerated separately for each | |||
different media source. | different media source. | |||
- Media type: Video/Voice | - Media type: Video/Voice | |||
- Media flow direction: forward, backward or both. | - Media flow direction: forward, backward or both. | |||
- Number of media sources: defines the total number of | - Number of media sources: defines the total number of | |||
media sources | media sources | |||
- Media codec: Constant Bit Rate (CBR) or Variable Bit Rate | - Media codec: Constant Bit Rate (CBR) or Variable Bit Rate | |||
(VBR) | (VBR) | |||
- Media source behaviour: describes the media encoder | - Media source behavior: describes the media encoder | |||
behavior. It defines the main parameters that affect the | behavior. It defines the main parameters that affect the | |||
adaptation behaviour. This may include but not limited | adaptation behavior. This may include but is not limited | |||
to: | to: | |||
o Adaptability: describes the adaptation options. For | o Adaptability: describes the adaptation options. For | |||
example, in the case of video it defines the following | example, in the case of video it defines the following | |||
ranges of adaptation: bit rate, frame rate, video | ranges of adaptation: bit rate, frame rate, video | |||
resolution. Similarly, in the case of voice, it | resolution. Similarly, in the case of voice, it | |||
defines the range of bit rate adaptation, the sampling | defines the range of bit rate adaptation, the sampling | |||
rate variation, and the variation in packetization | rate variation, and the variation in packetization | |||
interval. | interval. | |||
o Output variation : for a VBR encoder it defines the | o Output variation : for a VBR encoder it defines the | |||
encoder output variation from the average target rate | encoder output variation from the average target rate | |||
over a particular measurement interval. For example, | over a particular measurement interval. For example, | |||
on average the encoder output may vary between 5% to | on average the encoder output may vary between 5% to | |||
15% above or below the average target bit rate when | 15% above or below the average target bit rate when | |||
measured over a 100 ms time window. The time interval | measured over a 100 ms time window. The time interval | |||
over which the variation is specified must be | over which the variation is specified must be | |||
provided. | provided. | |||
o Responsiveness to a new bit rate request: the lag in | o Responsiveness to a new bit rate request: the lag in | |||
time between a new bit rate request and actual rate | time between a new bit rate request from the | |||
changes in encoder output. Depending on the encoder, | congestion control algorithm and actual rate changes | |||
this value may be specified in absolute time (e.g. | in encoder output. Depending on the encoder, this | |||
10ms to 1000ms) or other appropriate metric (next | value may be specified in absolute time (e.g. 10ms to | |||
frame interval time). | 1000ms) or other appropriate metric (e.g. next frame | |||
interval time). | ||||
More detailed discussions on expected media source | ||||
behavior, including those from synthetic video traffic | ||||
sources, is at [I-D.ietf-rmcat-video-traffic-model]. | ||||
- Media content: describes the chosen media sequences; For | - Media content: describes the chosen media sequences; For | |||
example, test sequences are available at: [xiph-seq] and | example, test sequences are available at: [xiph-seq] and | |||
[HEVC-seq]. | [HEVC-seq]. | |||
- Media timeline: describes the point when the media source | - Media timeline: describes the point when the media source | |||
is introduced and removed from the testbed. For example, | is introduced and removed from the testbed. For example, | |||
the media source may start transmitting immediately when | the media source may start transmitting immediately when | |||
the test case begins, or after a few seconds. | the test case begins, or after a few seconds. | |||
- Startup behaviour: the media starts at a defined bit | - Startup behavior: the media starts at a defined bit rate, | |||
rate, which may be the minimum, maximum bit rate, or a | which may be the minimum, maximum bit rate, or a value in | |||
value in between (in Kbps). | between (in Kbps). | |||
+ Competing traffic source: describes the characteristics of | + Competing traffic source: describes the characteristics of | |||
the competing traffic source, the different types of | the competing traffic source, the different types of | |||
competing flows are enumerated in | competing flows are enumerated in | |||
[I-D.ietf-rmcat-eval-criteria]. | [I-D.ietf-rmcat-eval-criteria]. | |||
- Traffic direction: forward, backward or both. | - Traffic direction: forward, backward or both. | |||
- Type of sources: defines the types of competing traffic | - Type of sources: defines the types of competing traffic | |||
sources. Types of competing traffic flows are listed in | sources. Types of competing traffic flows are listed in | |||
[I-D.ietf-rmcat-eval-criteria]. For example, the number | [I-D.ietf-rmcat-eval-criteria]. For example, the number | |||
of TCP flows connected to a web browser, the mean size | of TCP flows connected to a web browser, the mean size | |||
and distribution of the content downloaded. | and distribution of the content downloaded. | |||
- Number of sources: defines the total number of competing | - Number of sources: defines the total number of competing | |||
sources of each media type. | sources of each media type per traffic direction. | |||
- Congestion control: enumerates the congestion control | - Congestion control: enumerates the congestion control | |||
used by each type of competing traffic. | used by each type of competing traffic. | |||
- Traffic timeline: describes when the competing traffic | - Traffic timeline: describes when the competing traffic | |||
starts and ends in the test case. | starts and ends in the test case. | |||
* Additional attributes: describes attributes essential for | * Additional attributes: describes attributes essential for | |||
implementing a test case which are not included in the above | implementing a test case which are not included in the above | |||
structure. These attributes MUST be well defined, so that | structure. These attributes MUST be well defined, so that the | |||
other implementers are able to implement it. | other implementers of that particular test case are able to | |||
implement it easily. | ||||
Any attribute can have a set of values (enclosed within "[]"). Each | Any attribute can have a set of values (enclosed within "[]"). Each | |||
member value of such a set MUST be treated as different value for the | member value of such a set MUST be treated as different value for the | |||
same attribute. It is desired to run separate tests for each such | same attribute. It is desired to run separate tests for each such | |||
attribute value. | attribute value. | |||
The test cases described in this document follow the above structure. | The test cases described in this document follow the above structure. | |||
4. Recommended Evaluation Settings | 4. Recommended Evaluation Settings | |||
This section describes recommended test case settings and could be | This section describes recommended test case settings and could be | |||
overwritten by the respective test cases. | overwritten by the respective test cases. | |||
4.1. Evaluation metircs | 4.1. Evaluation metrics | |||
To evaluate the performance of the candidate algorithms it is | To evaluate the performance of the candidate algorithms the | |||
expected to log enough information to visualize the following metrics | implementers MUST log enough information to visualize the following | |||
at a fine enough time granularity: | metrics at a fine enough time granularity: | |||
1. Flow level: | 1. Flow level: | |||
A. End-to-end delay for the RMCAT flow. | A. End-to-end delay for the congestion controlled media flow. | |||
B. Variation in sending bit rate and goodput. Mainly observing | B. Variation in sending bit rate and goodput. Mainly observing | |||
the frequency and magnitude of oscillations. | the frequency and magnitude of oscillations. | |||
C. Packet losses observed at the receiving endpoint | C. Packet losses observed at the receiving endpoint. | |||
D. Feedback message overhead | D. Feedback message overhead. | |||
E. Convergence time. | E. Convergence time - time to reach steady state for the | |||
congestion controlled media flow(s). | ||||
2. Transport level: | 2. Transport level: | |||
A. Bandwidth utilization | A. Bandwidth utilization. | |||
B. Queue length (milliseconds at specified path capacity): | B. Queue length (milliseconds at specified path capacity): | |||
+ average over the length of the session | + average over the length of the session. | |||
+ 5 and 95 percentile | + 5 and 95 percentile. | |||
+ median, maximum, minimum | + median, maximum, minimum. | |||
4.2. Path characteristics | 4.2. Path characteristics | |||
Each path between a sender and receiver as described in Figure 1 have | Each path between a sender and receiver as described in Figure 1 have | |||
the following characteristics unless otherwise specified in the test | the following characteristics unless otherwise specified in the test | |||
case. | case. | |||
o Path direction: forward and backward. | o Path direction: forward and backward. | |||
o Reference bottleneck capacity: 1Mbps. | o Reference bottleneck capacity: 1Mbps. | |||
skipping to change at page 9, line 17 ¶ | skipping to change at page 9, line 23 ¶ | |||
Examples of additional network parameters are discussed in | Examples of additional network parameters are discussed in | |||
[I-D.ietf-rmcat-eval-criteria]. | [I-D.ietf-rmcat-eval-criteria]. | |||
For test cases involving time-varying bottleneck capacity, all | For test cases involving time-varying bottleneck capacity, all | |||
capacity values are specified as a ratio with respect to a reference | capacity values are specified as a ratio with respect to a reference | |||
capacity value, so as to allow flexible scaling of capacity values | capacity value, so as to allow flexible scaling of capacity values | |||
along with media source rate range. There exist two different | along with media source rate range. There exist two different | |||
mechanisms for inducing path capacity variation: a) by explicitly | mechanisms for inducing path capacity variation: a) by explicitly | |||
modifying the value of physical link capacity; or b) by introducing | modifying the value of physical link capacity; or b) by introducing | |||
background non-adaptive UDP traffic with time-varying traffic rate. | background non-adaptive UDP traffic with time-varying traffic rate. | |||
Implementers are encouraged run the experiments with both mechanisms | Implementers are encouraged to run the experiments with both | |||
for test cases specified in Section 5.1, Section 5.2, and | mechanisms for test cases specified in Section 5.1, Section 5.2, and | |||
Section 5.3. | Section 5.3. | |||
4.3. Media source | 4.3. Media source | |||
Unless otherwise specified, each test case will include one or more | Unless otherwise specified, each test case will include one or more | |||
media sources as described below. | media sources as described below. | |||
o Media type: Video | o Media type: Video | |||
* Media codec: VBR | * Media codec: VBR | |||
* Media source behaviour: | * Media source behavior: | |||
+ Adaptability: | + Adaptability: | |||
- Bit rate range: 150 Kbps - 1.5 Mbps. In real-life | - Bit rate range: 150 Kbps - 1.5 Mbps. In real-life | |||
applications the bitrate range can vary a lot depending | applications the bit rate range can vary a lot depending | |||
on the provided service, for example, the maximum bitrate | on the provided service, for example, the maximum bit | |||
can be up to 4Mbps. However, for running tests to | rate can be up to 4Mbps. However, for running tests to | |||
evaluate the congestion control algorithms it is more | evaluate the congestion control algorithms it is more | |||
important to have a look at how they are reacting to | important to have a look at how they are reacting to | |||
certain amount of bandwidth change. Also it is possible | certain amount of bandwidth change. Also it is possible | |||
that the media traffic generator used in a particular | that the media traffic generator used in a particular | |||
simulator or testbed if not capable of generating higher | simulator or testbed is not capable of generating higher | |||
bitrate. Hence we have selected a suitable bitrate range | bit rate. Hence we have selected a suitable bit rate | |||
typical of consumer-grade video conferencing applications | range typical of consumer-grade video conferencing | |||
in designing the test case. If a different bitrate range | applications in designing the test case. If a different | |||
is used in the test cases, the end-to-end path capacity | bit rate range is used in the test cases, then the end- | |||
values will also need to be scaled accordingly. | to-end path capacity values will also need to be scaled | |||
accordingly. | ||||
- Frame resolution: 144p - 720p (or 1080p) | - Frame resolution: 144p - 720p (or 1080p). This | |||
resolution range is selected based on the bit rate range. | ||||
If a different bit rate range is used in the test cases | ||||
then the frame resolution range also need to be selected | ||||
suitably. | ||||
- Frame rate: 10fps - 30fps | - Frame rate: 10fps - 30fps. This frame rate range is | |||
selected based on the bit rate range. If a different bit | ||||
rate range is used in the test cases then the frame rate | ||||
range also need to be adjusted suitably. | ||||
+ Variation from target bitrate: +/-5%. Unless otherwise | + Variation from target bit rate: +/-5%. Unless otherwise | |||
specified in the test case, bitrate variation SHOULD be | specified in the test case(s), bit rate variation SHOULD be | |||
calculated over one (1) second period of time. | calculated over one (1) second period of time. | |||
+ Responsiveness to new bit rate request: 100ms | + Responsiveness to new bit rate request: 100ms | |||
* Media content: The media content should represent a typical | * Media content: The media content should represent a typical | |||
video conversational scenario with head and shoulder movement. | video conversational scenario with head and shoulder movement. | |||
We recommend to use Foreman video sequence. | We recommend to use Foreman video sequence. | |||
* Media startup behaviour: 150Kbps. It should be noted that | * Media startup behavior: 150Kbps. It should be noted that | |||
applications can use smart ways to select an optimal startup | applications can use smart ways to select an optimal startup | |||
bitrate values for a certain network condition. In such cases | bit rate value for a certain network condition. In such cases | |||
the candidate proposals MAY show the effectiveness of such | the candidate proposals MAY show the effectiveness of such | |||
smart approach as an additional information for the evaluation | smart approach as an additional information for the evaluation | |||
process. | process. | |||
o Media type: Audio | o Media type: Audio | |||
* Media codec: CBR | * Media codec: CBR | |||
* Media bitrate: 20Kbps | * Media bit rate: 20Kbps | |||
5. Basic Test Cases | 5. Basic Test Cases | |||
5.1. Variable Available Capacity with Single RMCAT flow | 5.1. Variable Available Capacity with a Single Flow | |||
In this test case the bottleneck-link capacity between the two | In this test case the bottleneck-link capacity between the two | |||
endpoints varies over time. This test is designed to measure the | endpoints varies over time. This test is designed to measure the | |||
responsiveness of the candidate algorithm. This test tries to | responsiveness of the candidate algorithm. This test tries to | |||
address the requirements in [I-D.ietf-rmcat-cc-requirements], which | address the requirements in [I-D.ietf-rmcat-cc-requirements], which | |||
requires the algorithm to adapt the flow(s) and provide lower end-to- | requires the algorithm to adapt the flow(s) and provide lower end-to- | |||
end latency when there exists: | end latency when there exists: | |||
o an intermediate bottleneck | o an intermediate bottleneck | |||
o change in available capacity (e.g., due to interface change, | o change in available capacity (e.g., due to interface change, | |||
routing change, abrupt arrival/departure of background non- | routing change, abrupt arrival/departure of background non- | |||
adaptive traffic). | adaptive traffic). | |||
o maximum Media Bit Rate is Greater than Link Capacity. In this | o maximum media bit rate is greater than link capacity. In this | |||
case, the application will attempt to ramp up to its maximum bit | case, the application will attempt to ramp up to its maximum bit | |||
rate, since the link capacity is limited to a value lower, the | rate, since the link capacity is limited to a value lower, the | |||
congestion control scheme is expected to stabilize the sending bit | congestion control scheme is expected to stabilize the sending bit | |||
rate close to the available bottleneck capacity. This situation | rate close to the available bottleneck capacity. | |||
can occur when the endpoints are connected via thin long networks | ||||
even though the advertised capacity of the access network may be | ||||
higher. | ||||
It should be noted that the exact variation in available capacity due | It should be noted that the exact variation in available capacity due | |||
to any of the above depends on the under-lying technologies. Hence, | to any of the above depends on the underlying technologies. Hence, | |||
we describe a set of known factors, which may be extended to devise a | we describe a set of known factors, which may be extended to devise a | |||
more specific test case targeting certain behaviour in a certain | more specific test case targeting certain behaviors in a certain | |||
network environment. | network environment. | |||
Expected behavior: the candidate algorithm is expected to detect the | Expected behavior: the candidate algorithm is expected to detect the | |||
path capacity constraint, converges to bottleneck link's capacity and | path capacity constraint, converges to the bottleneck link's capacity | |||
adapt the flow to avoid unwanted oscillation when the sending bit | and adapt the flow to avoid unwanted oscillation when the sending bit | |||
rate is approaching the bottleneck link's capacity. The oscillations | rate is approaching the bottleneck link's capacity. The oscillations | |||
occur when the media flow(s) attempts to reach its maximum bit rate, | occur when the media flow(s) attempts to reach its maximum bit rate | |||
overshoots the usage of the available bottleneck capacity, to rectify | but overshoots the usage of the available bottleneck capacity then to | |||
it reduces the bit rate and starts to ramp up again. | rectify, it reduces the bit rate and starts to ramp up again. | |||
Testbed topology: One media source S1 is connected to corresponding | Evaluation metrics : as described in Section 4.1. | |||
R1. The media traffic is transported over the forward path and | ||||
corresponding feedback/control traffic is transported over the | Testbed topology: One media source S1 is connected to the | |||
backward path. | corresponding R1. The media traffic is transported over the forward | |||
path and corresponding feedback/control traffic is transported over | ||||
the backward path. | ||||
Forward --> | Forward --> | |||
+---+ +-----+ +-----+ +---+ | +---+ +-----+ +-----+ +---+ | |||
|S1 |=======| A |------------------------------>| B |=======|R1 | | |S1 |=======| A |------------------------------>| B |=======|R1 | | |||
+---+ | |<------------------------------| | +---+ | +---+ | |<------------------------------| | +---+ | |||
+-----+ +-----+ | +-----+ +-----+ | |||
<-- Backward | <-- Backward | |||
Figure 2: Testbed Topology for Limited Link Capacity | Figure 2: Testbed Topology for Limited Link Capacity | |||
To evaluate the performance of the candidate algorithms it is | ||||
expected to log enough information to visualize the metrics described | ||||
in Section 4.1 at a fine enough time granularity. | ||||
Testbed attributes: | Testbed attributes: | |||
o Test duration: 100s | o Test duration: 100s | |||
o Path characteristics: as described in Section 4.2 | o Path characteristics: as described in Section 4.2 | |||
o Application-related: | o Application-related: | |||
* Media Traffic: | * Media Traffic: | |||
+ Media type: Video | + Media type: Video | |||
- Media direction: forward. | - Media direction: forward. | |||
- Number of media sources: One (1) | - Number of media sources: one (1) | |||
- Media timeline: | - Media timeline: | |||
o Start time: 0s. | o Start time: 0s. | |||
o End time: 99s. | o End time: 99s. | |||
+ Media type: Audio | + Media type: Audio | |||
- Media direction: forward. | - Media direction: forward. | |||
- Number of media sources: One (1) | - Number of media sources: one (1) | |||
- Media timeline: | - Media timeline: | |||
o Start time: 0s. | o Start time: 0s. | |||
o End time: 99s. | o End time: 99s. | |||
* Competing traffic: | * Competing traffic: | |||
+ Number of sources : Zero (0) | + Number of sources : zero (0) | |||
o Test Specific Information: | o Test Specific Information: | |||
* This test uses the following one way propagation delays of 50 | * One-way propagation delay: [ 50 ms, 100 ms]. on the forward | |||
ms and 100 ms. | path direction | |||
* This test uses bottleneck path capacity variation as listed in | * This test uses bottleneck path capacity variation as listed in | |||
Table 1 | Table 1 | |||
* When using background non-adaptive UDP traffic to induce time- | * When using background non-adaptive UDP traffic to induce time- | |||
varying bottleneck for the RMCAT flow, the physical path | varying bottleneck , the physical path capacity remains at | |||
capacity is 4Mbps and the UDP traffic source rate changes over | 4Mbps and the UDP traffic source rate changes over time as | |||
time as (4-x)Mbps, where x is the bottleneck capacity specified | (4-x)Mbps, where x is the bottleneck capacity specified in | |||
in Table 1 | Table 1 | |||
+--------------------+--------------+-----------+-------------------+ | +--------------------+--------------+-----------+-------------------+ | |||
| Variation pattern | Path | Start | Path capacity | | | Variation pattern | Path | Start | Path capacity | | |||
| index | direction | time | ratio | | | index | direction | time | ratio | | |||
+--------------------+--------------+-----------+-------------------+ | +--------------------+--------------+-----------+-------------------+ | |||
| One | Forward | 0s | 1.0 | | | One | Forward | 0s | 1.0 | | |||
| Two | Forward | 40s | 2.5 | | | Two | Forward | 40s | 2.5 | | |||
| Three | Forward | 60s | 0.6 | | | Three | Forward | 60s | 0.6 | | |||
| Four | Forward | 80s | 1.0 | | | Four | Forward | 80s | 1.0 | | |||
+--------------------+--------------+-----------+-------------------+ | +--------------------+--------------+-----------+-------------------+ | |||
Table 1: Path capacity variation pattern for forward direction | Table 1: Path capacity variation pattern for forward direction | |||
5.2. Variable Available Capacity with Multiple RMCAT flows | 5.2. Variable Available Capacity with Multiple Flows | |||
This test case is similar to Section 5.1. However in addition this | This test case is similar to Section 5.1. However in addition this | |||
test will also consider persistent network load due to competing | test will also consider persistent network load due to competing | |||
traffic. | traffic. | |||
Expected behavior: the candidate algorithms is expected to detect the | Expected behavior: the candidate algorithm is expected to detect the | |||
variation in available capacity and adapt the media stream(s) | variation in available capacity and adapt the media stream(s) | |||
accordingly. The flows stabilize around their maximum bitrate as the | accordingly. The flows stabilize around their maximum bit rate as | |||
as the maximum link capacity is large enough to accommodate the | the maximum link capacity is large enough to accommodate the flows. | |||
flows. When the available capacity drops, the flow(s) adapts by | When the available capacity drops, the flows adapt by decreasing | |||
decreasing its sending bit rate, and when congestion disappears, the | their sending bit rate, and when congestion disappears, the flows are | |||
flow(s) are again expected to ramp up. | again expected to ramp up. | |||
To evaluate the performance of the candidate algorithms it is | Evaluation metrics : as described in Section 4.1. | |||
expected to log enough information to visualize the metrics described | ||||
in Section 4.1 at a fine enough time granularity: | ||||
Testbed Topology: Two (2) media sources S1 and S2 are connected to | Testbed Topology: Two (2) media sources S1 and S2 are connected to | |||
their corresponding destinations R1 and R2. The media traffic is | their corresponding destinations R1 and R2. The media traffic is | |||
transported over the forward path and corresponding feedback/control | transported over the forward path and corresponding feedback/control | |||
traffic is transported over the backward path. | traffic is transported over the backward path. | |||
+---+ +---+ | +---+ +---+ | |||
|S1 |===== \ / =======|R1 | | |S1 |===== \ / =======|R1 | | |||
+---+ \\ Forward --> // +---+ | +---+ \\ Forward --> // +---+ | |||
\\ // | \\ // | |||
skipping to change at page 13, line 52 ¶ | skipping to change at page 14, line 13 ¶ | |||
Figure 3: Testbed Topology for Variable Available Capacity | Figure 3: Testbed Topology for Variable Available Capacity | |||
Testbed attributes: | Testbed attributes: | |||
Testbed attributes are similar as described in Section 5.1 except the | Testbed attributes are similar as described in Section 5.1 except the | |||
test specific capacity variation setup. | test specific capacity variation setup. | |||
Test Specific Information: This test uses path capacity variation as | Test Specific Information: This test uses path capacity variation as | |||
listed in Table 2 with a corresponding end time of 125 seconds. The | listed in Table 2 with a corresponding end time of 125 seconds. The | |||
reference bottleneck capacity is 2Mbps. When using background non- | reference bottleneck capacity is 2Mbps. When using background non- | |||
adaptive UDP traffic to induce time-varying bottleneck for RMCAT | adaptive UDP traffic to induce time-varying bottleneck for congestion | |||
flows, the physical path capacity is 4Mbps and the UDP traffic source | controlled media flows, the physical path capacity is 4Mbps and the | |||
rate changes over time as (4-x)Mbps, where x is the bottleneck | UDP traffic source rate changes over time as (4-x)Mbps, where x is | |||
capacity specified in Table 2. | the bottleneck capacity specified in Table 2. | |||
+--------------------+--------------+-----------+-------------------+ | +--------------------+--------------+-----------+-------------------+ | |||
| Variation pattern | Path | Start | Path capacity | | | Variation pattern | Path | Start | Path capacity | | |||
| index | direction | time | ratio | | | index | direction | time | ratio | | |||
+--------------------+--------------+-----------+-------------------+ | +--------------------+--------------+-----------+-------------------+ | |||
| One | Forward | 0s | 2.0 | | | One | Forward | 0s | 2.0 | | |||
| Two | Forward | 25s | 1.0 | | | Two | Forward | 25s | 1.0 | | |||
| Three | Forward | 50s | 1.75 | | | Three | Forward | 50s | 1.75 | | |||
| Four | Forward | 75s | 0.5 | | | Four | Forward | 75s | 0.5 | | |||
| Five | Forward | 100s | 1.0 | | | Five | Forward | 100s | 1.0 | | |||
+--------------------+--------------+-----------+-------------------+ | +--------------------+--------------+-----------+-------------------+ | |||
Table 2: Path capacity variation pattern for forward direction | Table 2: Path capacity variation pattern for forward direction | |||
5.3. Congested Feedback Link with Bi-directional RMCAT flows | 5.3. Congested Feedback Link with Bi-directional Media Flows | |||
RMCAT WG has been chartered to define algorithms for RTP hence it is | Real-time interactive media uses RTP hence it is assumed that RTCP, | |||
assumed that RTCP, RTP header extension or such would be used by the | RTP header extension or such would be used by the congestion control | |||
congestion control algorithm in the backchannel. Due to asymmetric | algorithm in the backchannel. Due to asymmetric nature of the link | |||
nature of the link between communicating peers it is possible for a | between communicating peers it is possible for a participating peer | |||
participating peer to not receive such feedback information due to an | to not receive such feedback information due to an impaired or | |||
impaired or congested backchannel (even when the forward channel | congested backchannel (even when the forward channel might not be | |||
might not be impaired). This test case is designed to observe the | impaired). This test case is designed to observe the candidate | |||
candidate congestion control behaviour in such an event. | congestion control behavior in such an event. | |||
It is expected that the candidate algorithms is able to cope with the | It is expected that the candidate algorithms are able to cope with | |||
lack of feedback information and adapt to minimize the performance | the lack of feedback information and adapt to minimize the | |||
degradation of media flows in the forward channel. | performance degradation of media flows in the forward channel. | |||
It should be noted that for this test case: logs are compared with | It should be noted that for this test case: logs are compared with | |||
the reference case, i.e, when the backward channel has no impairments | the reference case, i.e, when the backward channel has no | |||
impairments. | ||||
To evaluate the performance of the candidate algorithms it is | Evaluation metrics : as described in Section 4.1. | |||
expected to log enough information to visualize the metrics described | ||||
in Section 4.1 at a fine-grained time intervals: | ||||
Testbed topology: One (1) media source S1 is connected to | Testbed topology: One (1) media source S1 is connected to | |||
corresponding R1, but both endpoints are additionally receiving and | corresponding R1, but both endpoints are additionally receiving and | |||
sending data, respectively. The media traffic (S1->R1) is | sending data, respectively. The media traffic (S1->R1) is | |||
transported over the forward path and corresponding feedback/control | transported over the forward path and corresponding feedback/control | |||
traffic is transported over the backward path. Likewise media | traffic is transported over the backward path. Likewise media | |||
traffic (S2->R2) is transported over the backward path and | traffic (S2->R2) is transported over the backward path and | |||
corresponding feedback/control traffic is transported over the | corresponding feedback/control traffic is transported over the | |||
forward path. | forward path. | |||
skipping to change at page 15, line 37 ¶ | skipping to change at page 15, line 46 ¶ | |||
* Reference bottleneck capacity: 1Mbps. | * Reference bottleneck capacity: 1Mbps. | |||
o Application-related: | o Application-related: | |||
* Media Source: | * Media Source: | |||
+ Media type: Video | + Media type: Video | |||
- Media direction: forward and backward | - Media direction: forward and backward | |||
- Number of media sources: Two (2) | - Number of media sources: two (2) | |||
- Media timeline: | - Media timeline: | |||
o Start time: 0s. | o Start time: 0s. | |||
o End time: 99s. | o End time: 99s. | |||
+ Media type: Audio | + Media type: Audio | |||
- Media direction: forward and backward | - Media direction: forward and backward | |||
- Number of media sources: Two (2) | - Number of media sources: two (2) | |||
- Media timeline: | - Media timeline: | |||
o Start time: 0s. | o Start time: 0s. | |||
o End time: 99s. | o End time: 99s. | |||
* Competing traffic: | * Competing traffic: | |||
+ Number of sources : Zero (0) | + Number of sources : zero (0) | |||
o Test Specific Information: This test uses path capacity variations | o Test Specific Information: this test uses path capacity variations | |||
to create congested feedback link. Table 3 lists the variation | to create congested feedback link. Table 3 lists the variation | |||
patterns applied to the forward path and Table 4 lists the | patterns applied to the forward path and Table 4 lists the | |||
variation patterns applied to the backward path. When using | variation patterns applied to the backward path. When using | |||
background non-adaptive UDP traffic to induce time-varying | background non-adaptive UDP traffic to induce time-varying | |||
bottleneck for RMCAT flows, the physical path capacity is 4Mbps | bottleneck for congestion controlled media flows, the physical | |||
for both directions and the UDP traffic source rate changes over | path capacity is 4Mbps for both directions and the UDP traffic | |||
time as (4-x)Mbps in each direction, where x is the bottleneck | source rate changes over time as (4-x)Mbps in each direction, | |||
capacity specified in Table 4. | where x is the bottleneck capacity specified in Table 4. | |||
+--------------------+--------------+-----------+-------------------+ | +--------------------+--------------+-----------+-------------------+ | |||
| Variation pattern | Path | Start | Path capacity | | | Variation pattern | Path | Start | Path capacity | | |||
| index | direction | time | ratio | | | index | direction | time | ratio | | |||
+--------------------+--------------+-----------+-------------------+ | +--------------------+--------------+-----------+-------------------+ | |||
| One | Forward | 0s | 2.0 | | | One | Forward | 0s | 2.0 | | |||
| Two | Forward | 20s | 1.0 | | | Two | Forward | 20s | 1.0 | | |||
| Three | Forward | 40s | 0.5 | | | Three | Forward | 40s | 0.5 | | |||
| Four | Forward | 60s | 2.0 | | | Four | Forward | 60s | 2.0 | | |||
+--------------------+--------------+-----------+-------------------+ | +--------------------+--------------+-----------+-------------------+ | |||
skipping to change at page 16, line 46 ¶ | skipping to change at page 17, line 5 ¶ | |||
| Variation pattern | Path | Start | Path capacity | | | Variation pattern | Path | Start | Path capacity | | |||
| index | direction | time | ratio | | | index | direction | time | ratio | | |||
+--------------------+--------------+-----------+-------------------+ | +--------------------+--------------+-----------+-------------------+ | |||
| One | Backward | 0s | 2.0 | | | One | Backward | 0s | 2.0 | | |||
| Two | Backward | 35s | 0.8 | | | Two | Backward | 35s | 0.8 | | |||
| Three | Backward | 70s | 2.0 | | | Three | Backward | 70s | 2.0 | | |||
+--------------------+--------------+-----------+-------------------+ | +--------------------+--------------+-----------+-------------------+ | |||
Table 4: Path capacity variation pattern for backward direction | Table 4: Path capacity variation pattern for backward direction | |||
5.4. Competing Flows with Same RMCAT Algorithm | 5.4. Competing Media Flows with same Congestion Control Algorithm | |||
In this test case, more than one RMCAT media flow shares the | In this test case, more than one media flows share the bottleneck | |||
bottleneck link and each of them uses the same congestion control | link and each of them uses the same congestion control algorithm. | |||
algorithm. This is a typical scenario where a real-time interactive | This is a typical scenario where a real-time interactive application | |||
application sends more than one media flows to the same destination | sends more than one media flow to the same destination and these | |||
and these flows are multiplexed over the same port. In such a | flows are multiplexed over the same port. In such a scenario it is | |||
scenario it is likely that the flows will be routed via the same path | likely that the flows will be routed via the same path and need to | |||
and need to share the available bandwidth amongst themselves. For | share the available bandwidth amongst themselves. For the sake of | |||
the sake of simplicity it is assumed that there are no other non- | simplicity it is assumed that there are no other competing traffic | |||
RMCAT competing traffic sources in the bottleneck link and that there | sources in the bottleneck link and that there is sufficient capacity | |||
is sufficient capacity to accommodate all the flows individually. | to accommodate all the flows individually. While this appears to be | |||
While this appears to be a variant of the test case defined in | a variant of the test case defined in Section 5.2, it focuses on the | |||
Section 5.2, it focuses on the capacity sharing aspect of the | capacity sharing aspect of the candidate algorithm. The previous | |||
candidate algorithm. The previous test case, on the other hand, | test case, on the other hand, measures adaptability, stability, and | |||
measures adaptability, stability, and responsiveness of the candidate | responsiveness of the candidate algorithm. | |||
algorithm. | ||||
Expected behavior: It is expected that the competing flows will | Expected behavior: It is expected that the competing flows will | |||
converge to an optimum bit rate to accommodate all the flows with | converge to an optimum bit rate to accommodate all the flows with | |||
minimum possible latency and loss. Specifically, the test introduces | minimum possible latency and loss. Specifically, the test introduces | |||
three media flows at different time instances, when the second flow | three media flows at different time instances, when the second flow | |||
appears there should still be room to accommodate another flow on the | appears there should still be room to accommodate another flow on the | |||
bottleneck link. Lastly, when the third flow appears the bottleneck | bottleneck link. Lastly, when the third flow appears the bottleneck | |||
link should be saturated. | link should be saturated. | |||
To evaluate the performance of the candidate algorithms it is | Evaluation metrics : as described in Section 4.1. | |||
expected to log enough information to visualize the metrics described | ||||
in Section 4.1 at a fine enough time granularity: | ||||
Testbed topology: Three media sources S1, S2, S3 are connected to | Testbed topology: Three media sources S1, S2, S3 are connected to R1, | |||
respective R1, R2, R3. The media traffic is transported over the | R2, R3 respectively. The media traffic is transported over the | |||
forward path and corresponding feedback/control traffic is | forward path and corresponding feedback/control traffic is | |||
transported over the backward path. | transported over the backward path. | |||
+---+ +---+ | +---+ +---+ | |||
|S1 |===== \ Forward --> / =======|R1 | | |S1 |===== \ Forward --> / =======|R1 | | |||
+---+ \\ // +---+ | +---+ \\ // +---+ | |||
\\ // | \\ // | |||
+---+ +-----+ +-----+ +---+ | +---+ +-----+ +-----+ +---+ | |||
|S2 |=======| A |------------------------------>| B |=======|R2 | | |S2 |=======| A |------------------------------>| B |=======|R2 | | |||
+---+ | |<------------------------------| | +---+ | +---+ | |<------------------------------| | +---+ | |||
+-----+ +-----+ | +-----+ +-----+ | |||
// \\ | // \\ | |||
// <-- Backward \\ | // <-- Backward \\ | |||
+---+ // \\ +---+ | +---+ // \\ +---+ | |||
|S3 |====== / \ ======|R3 | | |S3 |====== / \ ======|R3 | | |||
+---+ +---+ | +---+ +---+ | |||
Figure 5: Testbed Topology for Multiple RMCAT Flows | Figure 5: Testbed Topology for Multiple congestion controlled media | |||
Flows | ||||
Testbed attributes: | Testbed attributes: | |||
o Test duration: 120s | o Test duration: 120s | |||
o Path characteristics: | o Path characteristics: | |||
* Reference bottleneck capacity: 3.5Mbps | * Reference bottleneck capacity: 3.5Mbps | |||
* Path capacity ratio: 1.0 | * Path capacity ratio: 1.0 | |||
o Application-related: | o Application-related: | |||
* Media Source: | * Media Source: | |||
skipping to change at page 18, line 18 ¶ | skipping to change at page 18, line 23 ¶ | |||
* Path capacity ratio: 1.0 | * Path capacity ratio: 1.0 | |||
o Application-related: | o Application-related: | |||
* Media Source: | * Media Source: | |||
+ Media type: Video | + Media type: Video | |||
- Media direction: forward. | - Media direction: forward. | |||
- Number of media sources: Three (3) | - Number of media sources: three (3) | |||
- Media timeline: New media flows are added sequentially, | - Media timeline: new media flows are added sequentially, | |||
at short time intervals. See test specific setup below. | at short time intervals. See test specific setup below. | |||
+ Media type: Audio | + Media type: Audio | |||
- Media direction: forward. | - Media direction: forward. | |||
- Number of media sources: Three (3) | - Number of media sources: three (3) | |||
- Media timeline: New media flows are added sequentially, | - Media timeline: new media flows are added sequentially, | |||
at short time intervals. See test specific setup below. | at short time intervals. See test specific setup below. | |||
* Competing traffic: | * Competing traffic: | |||
+ Number of sources : Zero (0) | + Number of sources : zero (0) | |||
o Test Specific Information: Table 5 defines the media timeline for | o Test Specific Information: Table 5 defines the media timeline for | |||
both media type. | both media type. | |||
+---------+------------+------------+----------+ | +---------+------------+------------+----------+ | |||
| Flow IF | Media type | Start time | End time | | | Flow ID | Media type | Start time | End time | | |||
+---------+------------+------------+----------+ | +---------+------------+------------+----------+ | |||
| 1 | Video | 0s | 119s | | | 1 | Video | 0s | 119s | | |||
| 2 | Video | 20s | 119s | | | 2 | Video | 20s | 119s | | |||
| 3 | Video | 40s | 119s | | | 3 | Video | 40s | 119s | | |||
| 4 | Audio | 0s | 119s | | | 4 | Audio | 0s | 119s | | |||
| 5 | Audio | 20s | 119s | | | 5 | Audio | 20s | 119s | | |||
| 6 | Audio | 40s | 119s | | | 6 | Audio | 40s | 119s | | |||
+---------+------------+------------+----------+ | +---------+------------+------------+----------+ | |||
Table 5: Media Timeline for Video and Audio media sources | Table 5: Media Timeline for Video and Audio media sources | |||
5.5. Round Trip Time Fairness | 5.5. Round Trip Time Fairness | |||
In this test case, multiple RMCAT media flows share the bottleneck | In this test case, multiple media flows share the bottleneck link, | |||
link, but the end-to-end path latency for each RMCAT flow is | but the end-to-end path latency for each flow is different. For the | |||
different. For the sake of simplicity it is assumed that there are | sake of simplicity it is assumed that there are no other competing | |||
no other non-RMCAT competing traffic sources in the bottleneck link | traffic sources in the bottleneck link and that there is sufficient | |||
and that there is sufficient capacity to accommodate all the flows. | capacity to accommodate all the flows. While this appears to be a | |||
While this appears to be a variant of test case 5.2, it focuses on | variant of test case 5.2, it focuses on the capacity sharing aspect | |||
the capacity sharing aspect of the candidate algorithm under | of the candidate algorithm under different RTTs. | |||
different RTTs. | ||||
It is expected that the competing flows will converge to bit rates to | It is expected that the competing flows will converge to bit rates to | |||
accommodate all the flows with minimum possible latency and loss. | accommodate all the flows with minimum possible latency and loss. | |||
Specifically, the test introduces five media flows at the same time | Specifically, the test introduces five media flows at the same time | |||
instance. | instance. | |||
To evaluate the performance of the candidate algorithms it is | Evaluation metrics : as described in Section 4.1. | |||
expected to log enough information to visualize the metrics described | ||||
in Section 4.1 at a fine enough time granularity: | ||||
Testbed Topology: Five (5) media sources S1,S2,..,S5 are connected to | Testbed Topology: Five (5) media sources S1,S2,..,S5 are connected to | |||
their corresponding media sinks R1,R2,..,R5. The media traffic is | their corresponding media sinks R1,R2,..,R5. The media traffic is | |||
transported over the forward path and corresponding feedback/control | transported over the forward path and corresponding feedback/control | |||
traffic is transported over the backward path. The topology is the | traffic is transported over the backward path. The topology is the | |||
same as in Section 5.4. The end-to-end path delays are: 10ms for | same as in Section 5.4. The end-to-end path delays are: 10ms for | |||
S1-R1, 25ms for S2-R2, 50ms for S3-R3, 100ms for S4-R4, and 150ms | S1-R1, 25ms for S2-R2, 50ms for S3-R3, 100ms for S4-R4, and 150ms | |||
S5-R5, respectively. | S5-R5, respectively. | |||
Testbed attributes: | Testbed attributes: | |||
skipping to change at page 19, line 50 ¶ | skipping to change at page 20, line 11 ¶ | |||
100ms, 150ms. | 100ms, 150ms. | |||
o Application-related: | o Application-related: | |||
* Media Source: | * Media Source: | |||
+ Media type: Video | + Media type: Video | |||
- Media direction: forward | - Media direction: forward | |||
- Number of media sources: Five (5) | - Number of media sources: five (5) | |||
- Media timeline: New media flows are added sequentially, | ||||
- Media timeline: new media flows are added sequentially, | ||||
at short time intervals. See test specific setup below. | at short time intervals. See test specific setup below. | |||
+ Media type: Audio | + Media type: Audio | |||
- Media direction: forward. | - Media direction: forward. | |||
- Number of media sources: Five (5) | - Number of media sources: five (5) | |||
- Media timeline: New media flows are added sequentially, | - Media timeline: new media flows are added sequentially, | |||
at short time intervals. See test specific setup below. | at short time intervals. See test specific setup below. | |||
* Competing traffic: | * Competing traffic: | |||
+ Number of sources : Zero (0) | + Number of sources : zero (0) | |||
o Test Specific Information: Table 6 defines the media timeline for | o Test Specific Information: Table 6 defines the media timeline for | |||
both media type. | both media type. | |||
+---------+------------+------------+----------+ | +---------+------------+------------+----------+ | |||
| Flow IF | Media type | Start time | End time | | | Flow IF | Media type | Start time | End time | | |||
+---------+------------+------------+----------+ | +---------+------------+------------+----------+ | |||
| 1 | Video | 0s | 299s | | | 1 | Video | 0s | 299s | | |||
| 2 | Video | 10s | 299s | | | 2 | Video | 10s | 299s | | |||
| 3 | Video | 20s | 299s | | | 3 | Video | 20s | 299s | | |||
skipping to change at page 20, line 40 ¶ | skipping to change at page 21, line 5 ¶ | |||
| 5 | Video | 40s | 299s | | | 5 | Video | 40s | 299s | | |||
| 6 | Audio | 0 | 299s | | | 6 | Audio | 0 | 299s | | |||
| 7 | Audio | 10s | 299s | | | 7 | Audio | 10s | 299s | | |||
| 8 | Audio | 20s | 299s | | | 8 | Audio | 20s | 299s | | |||
| 9 | Audio | 30s | 299s | | | 9 | Audio | 30s | 299s | | |||
| 10 | Audio | 40s | 299s | | | 10 | Audio | 40s | 299s | | |||
+---------+------------+------------+----------+ | +---------+------------+------------+----------+ | |||
Table 6: Media Timeline for Video and Audio media sources | Table 6: Media Timeline for Video and Audio media sources | |||
5.6. RMCAT Flow competing with a long TCP Flow | 5.6. Media Flow Competing with a Long TCP Flow | |||
In this test case, one or more RMCAT media flows share the bottleneck | In this test case, one or more media flows share the bottleneck link | |||
link with at least one long lived TCP flows. Long lived TCP flows | with at least one long lived TCP flow. Long lived TCP flows download | |||
download data throughout the session and are expected to have | data throughout the session and are expected to have infinite amount | |||
infinite amount of data to send and receive. This is a scenario | of data to send and receive. This is a scenario where a multimedia | |||
where a multimedia application co-exists with a large file download. | application co-exists with a large file download. The test case | |||
The test case measures the adaptivity of the candidate algorithm to | measures the adaptivity of the candidate algorithm to competing | |||
competing traffic. It addresses the requirement 3 in | traffic. It addresses the requirement 3 in | |||
[I-D.ietf-rmcat-cc-requirements]. | [I-D.ietf-rmcat-cc-requirements]. | |||
Expected behavior: depending on the convergence observed in test case | Expected behavior: depending on the convergence observed in test case | |||
5.1 and 5.2, the candidate algorithm may be able to avoid congestion | 5.1 and 5.2, the candidate algorithm may be able to avoid congestion | |||
collapse. In the worst case, the media stream will fall to the | collapse. In the worst case, the media stream will fall to the | |||
minimum media bit rate. | minimum media bit rate. | |||
To evaluate the performance of the candidate algorithms it is | Evaluation metrics : following metrics in addition to as described in | |||
expected to log enough information to visualize the following metrics | Section 4.1. | |||
in addition to the metrics described in Section 4.1 at a fine enough | ||||
time granularity: | ||||
1. Flow level: | 1. Flow level: | |||
A. TCP throughput. | A. TCP throughput. | |||
Testbed topology: One (1) media source S1 is connected to | B. Loss for the TCP flow | |||
Testbed topology: One (1) media source S1 is connected to the | ||||
corresponding media sink, R1. In addition, there is a long-live TCP | corresponding media sink, R1. In addition, there is a long-live TCP | |||
flow sharing the same bottleneck link. The media traffic is | flow sharing the same bottleneck link. The media traffic is | |||
transported over the forward path and corresponding feedback/control | transported over the forward path and corresponding feedback/control | |||
traffic is transported over the backward path. The TCP traffic goes | traffic is transported over the backward path. The TCP traffic goes | |||
over the forward path from, S_tcp with acknowledgement packets | over the forward path from, S_tcp with acknowledgment packets go over | |||
flowing along the backward path from, R_tcp. | the backward path from, R_tcp. | |||
+--+ +--+ | +--+ +--+ | |||
|S1|===== \ Forward --> / =======|R1| | |S1|===== \ Forward --> / =======|R1| | |||
+--+ \\ // +--+ | +--+ \\ // +--+ | |||
\\ // | \\ // | |||
+-----+ +-----+ | +-----+ +-----+ | |||
| A |---------------------------->| B | | | A |---------------------------->| B | | |||
| |<----------------------------| | | | |<----------------------------| | | |||
+-----+ +-----+ | +-----+ +-----+ | |||
// \\ | // \\ | |||
// <-- Backward \\ | // <-- Backward \\ | |||
+-----+ // \\ +-----+ | +-----+ // \\ +-----+ | |||
|S_tcp|=== / \ ===|R_tcp| | |S_tcp|=== / \ ===|R_tcp| | |||
+-----+ +-----+ | +-----+ +-----+ | |||
Figure 6: Testbed Topology for TCP vs RMCAT Flows | Figure 6: Testbed Topology for TCP vs congestion controlled media | |||
Flows | ||||
Testbed attributes: | Testbed attributes: | |||
o Test duration: 120s | o Test duration: 120s | |||
o Path characteristics: | o Path characteristics: | |||
* Reference bottleneck capacity: 2Mbps | * Reference bottleneck capacity: 2Mbps | |||
* Path capacity ratio: 1.0 | * Path capacity ratio: 1.0 | |||
skipping to change at page 22, line 14 ¶ | skipping to change at page 22, line 42 ¶ | |||
* Bottleneck queue size: [300ms, 1000ms] | * Bottleneck queue size: [300ms, 1000ms] | |||
o Application-related: | o Application-related: | |||
* Media Source: | * Media Source: | |||
+ Media type: Video | + Media type: Video | |||
- Media direction: forward | - Media direction: forward | |||
- Number of media sources: One (1) | - Number of media sources: one (1) | |||
- Media timeline: | - Media timeline: | |||
o Start time: 5s. | o Start time: 5s. | |||
o End time: 119s. | o End time: 119s. | |||
+ Media type: Audio | + Media type: Audio | |||
- Media direction: forward | - Media direction: forward | |||
- Number of media sources: one (1) | ||||
- Number of media sources: One (1) | ||||
- Media timeline: | - Media timeline: | |||
o Start time: 5s. | o Start time: 5s. | |||
o End time: 119s. | o End time: 119s. | |||
* Additionally, implementers are encouraged to run the experiment | * Additionally, implementers are encouraged to run the experiment | |||
with multiple media sources. | with multiple media sources. | |||
* Competing traffic: | * Competing traffic: | |||
+ Number and Types of sources : one (1), long-lived TCP | + Number and Types of sources : one (1) and long-lived TCP | |||
+ Traffic direction : forward | + Traffic direction : forward | |||
+ Congestion control: Default TCP congestion control. | + Congestion control: default TCP congestion control[RFC5681]. | |||
+ Traffic timeline: | + Traffic timeline: | |||
- Start time: 0s. | - Start time: 0s. | |||
- End time: 119s. | - End time: 119s. | |||
o Test Specific Information: None | o Test Specific Information: none | |||
5.7. RMCAT Flow competing with short TCP Flows | 5.7. Media Flow Competing with Short TCP Flows | |||
In this test case, one or more RMCAT media flow shares the bottleneck | In this test case, one or more congestion controlled media flow | |||
link with multiple short-lived TCP flows. Short-lived TCP flows | shares the bottleneck link with multiple short-lived TCP flows. | |||
resemble the on/off pattern observed in the web traffic, wherein | Short-lived TCP flows resemble the on/off pattern observed in the web | |||
clients (browsers) connect to a server and download a resource | traffic, wherein clients (browsers) connect to a server and download | |||
(typically a web page, few images, text files, etc.) using several | a resource (typically a web page, few images, text files, etc.) using | |||
TCP connections (up to 4). This scenario shows the performance of | several TCP connections (up to 4). This scenario shows the | |||
the multimedia application when several browser windows are active. | performance of a multimedia application when several browser windows | |||
The test case measures the adaptivity of the candidate algorithm to | are active. The test case measures the adaptivity of the candidate | |||
competing web traffic, it addresses the requirements 1.E in | algorithm to competing web traffic, it addresses the requirements 1.E | |||
[I-D.ietf-rmcat-cc-requirements]. | in [I-D.ietf-rmcat-cc-requirements]. | |||
Depending on the number of short TCP flows, the cross-traffic either | Depending on the number of short TCP flows, the cross-traffic either | |||
appears as a short burst flow or resembles a long TCP flow. The | appears as a short burst flow or resembles a long TCP flow. The | |||
intention of this test is to observe the impact of short-term burst | intention of this test is to observe the impact of short-term burst | |||
on the behaviour of the candidate algorithm. | on the behavior of the candidate algorithm. | |||
To evaluate the performance of the candidate algorithms it is | Evaluation metrics : following metrics in addition to as described in | |||
expected to log enough information to visualize the following metrics | Section 4.1. | |||
in addition to the metrics described in Section 4.1 at a fine enough | ||||
time granularity: | ||||
1. Flow level: | 1. Flow level: | |||
A. Variation in the sending rate of the TCP flow. | A. Variation in the sending rate of the TCP flow. | |||
B. TCP throughput. | B. TCP throughput. | |||
Testbed topology: The topology described here is same as the one | Testbed topology: The topology described here is same as the one | |||
described in Figure 6. | described in Figure 6. | |||
skipping to change at page 23, line 49 ¶ | skipping to change at page 24, line 26 ¶ | |||
o Test duration: 300s | o Test duration: 300s | |||
o Path characteristics: | o Path characteristics: | |||
* Reference bottleneck capacity: 2.0Mbps | * Reference bottleneck capacity: 2.0Mbps | |||
* Path capacity ratio: 1.0 | * Path capacity ratio: 1.0 | |||
o Application-related: | o Application-related: | |||
* Media Source: | * Media source: | |||
+ Media type: Video | + Media type: Video | |||
- Media direction: forward | - Media direction: forward | |||
- Number of media sources: two (2) | - Number of media sources: two (2) | |||
- Media timeline: | - Media timeline: | |||
o Start time: 5s. | o Start time: 5s. | |||
o End time: 299s. | o End time: 299s. | |||
skipping to change at page 24, line 28 ¶ | skipping to change at page 25, line 5 ¶ | |||
- Number of media sources: two (2) | - Number of media sources: two (2) | |||
- Media timeline: | - Media timeline: | |||
o Start time: 5s. | o Start time: 5s. | |||
o End time: 299s. | o End time: 299s. | |||
* Competing traffic: | * Competing traffic: | |||
+ Number and Types of sources : Ten (10), short-lived TCP | + Number and Types of sources : ten (10), short-lived TCP | |||
flows. | flows. | |||
+ Traffic direction : forward | + Traffic direction : forward | |||
+ Congestion algorithm: Default TCP Congestion control. | + Congestion algorithm: default TCP Congestion control | |||
[RFC5681]. | ||||
+ Traffic timeline: Each short TCP flow is modeled as a | + Traffic timeline: each short TCP flow is modeled as a | |||
sequence of file downloads interleaved with idle periods. | sequence of file downloads interleaved with idle periods. | |||
See test specific setup. Not all short TCPs start at the | See test specific setup. Not all short TCP flows start at | |||
same time, 2 start in the ON state while 8 start in an OFF | the same time, 2 of them start in the ON state while rest on | |||
stats. The model for the idle times for the OFF state is | the 8 flows start in an OFF stats. The model for the idle | |||
discussed in the Short-TCP model. | times for the OFF state is discussed in | |||
[I-D.ietf-rmcat-eval-criteria]. | ||||
o Test Specific Information: | o Test Specific Information: | |||
* Short-TCP traffic model: | * Short-TCP traffic model: | |||
+ File sizes: uniform distribution between 100KB to 1MB | + File sizes: uniform distribution between 100KB to 1MB | |||
+ Idle period: the duration of the OFF state is derived from | + Idle period: the duration of the OFF state is derived from | |||
an exponential distribution with the mean value of 10 | an exponential distribution with the mean value of 10 | |||
seconds. | seconds. | |||
5.8. Media Pause and Resume | 5.8. Media Pause and Resume | |||
In this test case, more than one real-time interactive media flows | In this test case, more than one real-time interactive media flows | |||
share the link bandwidth and all flows reach to a steady state by | share the link bandwidth and all flows reach to a steady state by | |||
utilizing the link capacity in an optimum way. At these stage one of | utilizing the link capacity in an optimum way. At this stage one of | |||
the media flow is paused for a moment. This event will result in | the media flows is paused for a moment. This event will result in | |||
more available bandwidth for the rest of the flows and as they are on | more available bandwidth for the rest of the flows as they are on a | |||
a shared link. When the paused media flow will resume it would no | shared link. When the paused media flow resumes it would no longer | |||
longer have the same bandwidth share on the link. It has to make | have the same bandwidth share on the link. It has to make it's way | |||
it's way through the other existing flows in the link to achieve a | through the other existing flows in the link to achieve a fair share | |||
fair share of the link capacity. This test case is important | of the link capacity. This test case is important specially for | |||
specially for real-time interactive media which consists of more than | real-time interactive media which consists of more than one media | |||
one media flows and can pause/resume media flow at any point of time | flows and can pause/resume media flows at any point of time during | |||
during the session. This test case directly addresses the | the session. This test case directly addresses the requirement | |||
requirement number 5 in [I-D.ietf-rmcat-cc-requirements]. One can | number 5 in [I-D.ietf-rmcat-cc-requirements]. One can think it as a | |||
think it as a variation of test case defined in Section 5.4. | variation of test case defined in Section 5.4. However, it is | |||
However, it is different as the candidate algorithms can use | different as the candidate algorithms can use different strategies to | |||
different strategies to increase its efficiency, for example the | increase its efficiency, for example in terms of fairness, | |||
fairness, convergence time, reduce oscillation etc, by capitalizing | convergence time, reduce oscillation etc, by capitalizing the fact | |||
the fact that they have previous information of the link. | that they have previous information of the link. | |||
To evaluate the performance of the candidate algorithms it is | Evaluation metrics : following metrics in addition to as described in | |||
expected to log enough information to visualize the following metrics | Section 4.1. | |||
in addition to the metrics described in Section 4.1 at a fine enough | ||||
time granularity: | ||||
1. Flow level: | 1. Flow level: | |||
A. Variation in sending bit rate and goodput. Mainly observing | A. Variation in sending bit rate and goodput. Mainly observing | |||
the frequency and magnitude of oscillations. | the frequency and magnitude of oscillations. | |||
Testbed Topology: Same as test case defined in Section 5.4 | Testbed Topology: Same as test case defined in Section 5.4 | |||
Testbed attributes: The general description of the testbed parameters | Testbed attributes: The general description of the testbed parameters | |||
are same as Section 5.4 with changes in the test specific setup as | are same as Section 5.4 with changes in the test specific setup as | |||
below- | below- | |||
o Other test specific setup: | o Other test specific setup: | |||
* Media flow timeline: | * Media flow timeline: | |||
+ Flow ID: One (1) | + Flow ID: one (1) | |||
+ Start time: 0s | + Start time: 0s | |||
+ Flow duration: 119s | + Flow duration: 119s | |||
+ Pause time: not required | + Pause time: not required | |||
+ Resume time: not required | + Resume time: not required | |||
* Media flow timeline: | * Media flow timeline: | |||
+ Flow ID: Two (2) | + Flow ID: two (2) | |||
+ Start time: 0s | + Start time: 0s | |||
+ Flow duration: 119s | + Flow duration: 119s | |||
+ Pause time: at 40s | + Pause time: at 40s | |||
+ Resume time: at 60s | + Resume time: at 60s | |||
* Media flow timeline: | * Media flow timeline: | |||
+ Flow ID: Three (3) | + Flow ID: three (3) | |||
+ Start time: 0s | + Start time: 0s | |||
+ Flow duration:119s | + Flow duration:119s | |||
+ Pause time: not required | + Pause time: not required | |||
+ Resume time: not required | + Resume time: not required | |||
6. Other potential test cases | 6. Other potential test cases | |||
It has been noticed that there are other interesting test cases | It has been noticed that there are other interesting test cases | |||
besides the basis test cases listed above. In many aspects, these | besides the basic test cases listed above. In many aspects, these | |||
additional test cases can help to further evaluate the candidate | additional test cases can help further evaluation of the candidate | |||
algorithm. They are listed as below. | algorithm. They are listed as below. | |||
6.1. Explicit Congestion Notification Usage | 6.1. Media Flows with Priority | |||
In this test case media flows will have different priority levels. | ||||
This will be an extension of Section 5.4 where the same test will be | ||||
run with different priority levels imposed on each of the media | ||||
flows. For example, the first flow (S1) is assigned a priority of 2 | ||||
whereas the remaining two flows (S2 and S3) are assigned a priority | ||||
of 1. The candidate algorithm MUST reflect the relative priorities | ||||
assigned to each media flow. In the previous example, the first flow | ||||
(S1) MUST arrive at a steady-state rate approximately twice of that | ||||
of the other two flows (S2 and S3). | ||||
The candidate algorithm can use a coupled congestion control | ||||
mechanism for the bandwidth distribution according to the respective | ||||
media flow priority. | ||||
6.2. Explicit Congestion Notification Usage | ||||
This test case requires to run all the basic test cases with the | This test case requires to run all the basic test cases with the | |||
availability of Explicit Congestion Notification (ECN) [RFC6679] | availability of Explicit Congestion Notification (ECN) [RFC6679] | |||
feature enabled. The goal of this test is to exhibit that the | feature enabled. The goal of this test is to exhibit that the | |||
candidate algorithms does not fail when ECN signals are available. | candidate algorithms do not fail when ECN signals are available. | |||
With ECN signals enabled the algorithms are expected to perform | With ECN signals enabled the algorithms are expected to perform | |||
better than their delay based variants. | better than their delay based variants. | |||
6.2. Multiple Bottlenecks | 6.3. Multiple Bottlenecks | |||
In this test case one RMCAT flow, S1->R2 traverse a path with | In this test case one congestion controlled media flow, S1->R2, | |||
multiple bottlenecks. As illustrated in Figure 7, the first flow | traverses a path with multiple bottlenecks. As illustrated in | |||
(S1->R1) competes with the second RMCAT flow (S2->R2) over the link | Figure 7, the first flow (S1->R1) competes with the second congestion | |||
between A and B which is close to the sender side; again, that flow | controlled media flow (S2->R2) over the link between A and B which is | |||
(S1->R1) competes with the third RMCAT flow (S3->R3) over the link | close to the sender side; again, that flow (S1->R1) competes with the | |||
between C and D which is close to the receiver side. The goal of | third congestion controlled media flow (S3->R3) over the link between | |||
this test is to ensure that the candidate algorithms work properly in | C and D which is close to the receiver side. The goal of this test | |||
the presence of multiple bottleneck links on the end to end path. | is to ensure that the candidate algorithms work properly in the | |||
presence of multiple bottleneck links on the end to end path. | ||||
Expected behavior: the candidate algorithm is expected to achieve | Expected behavior: the candidate algorithm is expected to achieve | |||
full utilization at both bottleneck links without starving any of the | full utilization at both bottleneck links without starving any of the | |||
three RMCAT flows. | three congestion controlled media flows. | |||
Forward ----> | Forward ----> | |||
+---+ +---+ +---+ +---+ | +---+ +---+ +---+ +---+ | |||
|S2 | |R2 | |S3 | |R3 | | |S2 | |R2 | |S3 | |R3 | | |||
+---+ +---+ +---+ +---+ | +---+ +---+ +---+ +---+ | |||
| | | | | | | | | | |||
| | | | | | | | | | |||
+---+ +-----+ +-----+ +-----+ +-----+ +---+ | +---+ +-----+ +-----+ +-----+ +-----+ +---+ | |||
|S1 |=======| A |------>| B |----->| C |---->| D |=======|R1 | | |S1 |=======| A |------>| B |----->| C |---->| D |=======|R1 | | |||
skipping to change at page 27, line 39 ¶ | skipping to change at page 28, line 35 ¶ | |||
Figure 7: Testbed Topology for Multiple Bottlenecks | Figure 7: Testbed Topology for Multiple Bottlenecks | |||
Testbed topology: Three media sources S1, S2, and S3 are connected to | Testbed topology: Three media sources S1, S2, and S3 are connected to | |||
respective destinations R1, R2, and R3. For all three flows the | respective destinations R1, R2, and R3. For all three flows the | |||
media traffic is transported over the forward path and corresponding | media traffic is transported over the forward path and corresponding | |||
feedback/control traffic is transported over the backward path. | feedback/control traffic is transported over the backward path. | |||
Testbed attributes: | Testbed attributes: | |||
o Test duration: 120s | o Test duration: 300s | |||
o Path characteristics: | o Path characteristics: | |||
* Reference bottleneck capacity between A and B = 2Mbps. | * Reference bottleneck capacity: 2Mbps. | |||
* Path capacity ratio between A and B: 1.0 | * Path capacity ratio between A and B: 1.0 | |||
* Path capacity ratio between B and C: 4.0. | * Path capacity ratio between B and C: 4.0. | |||
* Path capacity ratio between C and D: 0.75. | * Path capacity ratio between C and D: 0.75. | |||
* One-Way propagation delay: | * One-Way propagation delay: | |||
1. Between S1 and R1: 100ms | 1. Between S1 and R1: 100ms | |||
2. Between S2 and R2: 40ms | 2. Between S2 and R2: 40ms | |||
skipping to change at page 28, line 30 ¶ | skipping to change at page 29, line 27 ¶ | |||
+ Media type: Video | + Media type: Video | |||
- Media direction: Forward | - Media direction: Forward | |||
- Number of media sources: Three (3) | - Number of media sources: Three (3) | |||
- Media timeline: | - Media timeline: | |||
o Start time: 0s. | o Start time: 0s. | |||
o End time: 119s. | o End time: 299s. | |||
+ Media type: Audio | + Media type: Audio | |||
- Media direction: Forward | - Media direction: Forward | |||
- Number of media sources: Three (3) | - Number of media sources: Three (3) | |||
- Media timeline: | - Media timeline: | |||
o Start time: 0s. | o Start time: 0s. | |||
o End time: 119s. | o End time: 299s. | |||
* Competing traffic: | * Competing traffic: | |||
+ Number of sources : Zero (0) | + Number of sources : Zero (0) | |||
7. Wireless Access Links | 7. Wireless Access Links | |||
Additional wireless network (both cellular network and WiFi network) | Additional wireless network (both cellular network and WiFi network) | |||
specific test cases are define in [I-D.ietf-rmcat-wireless-tests] | specific test cases are defined in [I-D.ietf-rmcat-wireless-tests]. | |||
8. Security Considerations | 8. Security Considerations | |||
Security issues have not been discussed in this memo. | Security issues have not been discussed in this memo. | |||
9. IANA Considerations | 9. IANA Considerations | |||
There are no IANA impacts in this memo. | There are no IANA impacts in this memo. | |||
10. Acknowledgements | 10. Acknowledgements | |||
skipping to change at page 29, line 46 ¶ | skipping to change at page 30, line 41 ¶ | |||
Jacobson, "RTP: A Transport Protocol for Real-Time | Jacobson, "RTP: A Transport Protocol for Real-Time | |||
Applications", STD 64, RFC 3550, DOI 10.17487/RFC3550, | Applications", STD 64, RFC 3550, DOI 10.17487/RFC3550, | |||
July 2003, <http://www.rfc-editor.org/info/rfc3550>. | July 2003, <http://www.rfc-editor.org/info/rfc3550>. | |||
[RFC3551] Schulzrinne, H. and S. Casner, "RTP Profile for Audio and | [RFC3551] Schulzrinne, H. and S. Casner, "RTP Profile for Audio and | |||
Video Conferences with Minimal Control", STD 65, RFC 3551, | Video Conferences with Minimal Control", STD 65, RFC 3551, | |||
DOI 10.17487/RFC3551, July 2003, | DOI 10.17487/RFC3551, July 2003, | |||
<http://www.rfc-editor.org/info/rfc3551>. | <http://www.rfc-editor.org/info/rfc3551>. | |||
[RFC3611] Friedman, T., Ed., Caceres, R., Ed., and A. Clark, Ed., | [RFC3611] Friedman, T., Ed., Caceres, R., Ed., and A. Clark, Ed., | |||
"RTP Control Protocol Extended Reports (RTCP XR)", RFC | "RTP Control Protocol Extended Reports (RTCP XR)", | |||
3611, DOI 10.17487/RFC3611, November 2003, | RFC 3611, DOI 10.17487/RFC3611, November 2003, | |||
<http://www.rfc-editor.org/info/rfc3611>. | <http://www.rfc-editor.org/info/rfc3611>. | |||
[RFC4585] Ott, J., Wenger, S., Sato, N., Burmeister, C., and J. Rey, | [RFC4585] Ott, J., Wenger, S., Sato, N., Burmeister, C., and J. Rey, | |||
"Extended RTP Profile for Real-time Transport Control | "Extended RTP Profile for Real-time Transport Control | |||
Protocol (RTCP)-Based Feedback (RTP/AVPF)", RFC 4585, DOI | Protocol (RTCP)-Based Feedback (RTP/AVPF)", RFC 4585, | |||
10.17487/RFC4585, July 2006, | DOI 10.17487/RFC4585, July 2006, | |||
<http://www.rfc-editor.org/info/rfc4585>. | <http://www.rfc-editor.org/info/rfc4585>. | |||
[RFC5506] Johansson, I. and M. Westerlund, "Support for Reduced-Size | [RFC5506] Johansson, I. and M. Westerlund, "Support for Reduced-Size | |||
Real-Time Transport Control Protocol (RTCP): Opportunities | Real-Time Transport Control Protocol (RTCP): Opportunities | |||
and Consequences", RFC 5506, DOI 10.17487/RFC5506, April | and Consequences", RFC 5506, DOI 10.17487/RFC5506, April | |||
2009, <http://www.rfc-editor.org/info/rfc5506>. | 2009, <http://www.rfc-editor.org/info/rfc5506>. | |||
[I-D.ietf-rmcat-eval-criteria] | [I-D.ietf-rmcat-eval-criteria] | |||
Singh, V. and J. Ott, "Evaluating Congestion Control for | Singh, V. and J. Ott, "Evaluating Congestion Control for | |||
Interactive Real-time Media", draft-ietf-rmcat-eval- | Interactive Real-time Media", draft-ietf-rmcat-eval- | |||
criteria-03 (work in progress), March 2015. | criteria-04 (work in progress), October 2015. | |||
[I-D.ietf-rmcat-wireless-tests] | [I-D.ietf-rmcat-wireless-tests] | |||
Sarker, Z. and I. Johansson, "Evaluation Test Cases for | Sarker, Z., Johansson, I., Zhu, X., Fu, J., Tan, W., and | |||
Interactive Real-Time Media over Wireless Networks", | M. Ramalho, "Evaluation Test Cases for Interactive Real- | |||
draft-ietf-rmcat-wireless-tests-00 (work in progress), | Time Media over Wireless Networks", draft-ietf-rmcat- | |||
June 2015. | wireless-tests-01 (work in progress), November 2015. | |||
[I-D.ietf-rmcat-video-traffic-model] | ||||
Zhu, X., Cruz, S., and Z. Sarker, "Modeling Video Traffic | ||||
Sources for RMCAT Evaluations", draft-ietf-rmcat-video- | ||||
traffic-model-00 (work in progress), January 2016. | ||||
11.2. Informative References | 11.2. Informative References | |||
[RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion | ||||
Control", RFC 5681, DOI 10.17487/RFC5681, September 2009, | ||||
<http://www.rfc-editor.org/info/rfc5681>. | ||||
[I-D.ietf-rmcat-cc-requirements] | [I-D.ietf-rmcat-cc-requirements] | |||
Jesup, R. and Z. Sarker, "Congestion Control Requirements | Jesup, R. and Z. Sarker, "Congestion Control Requirements | |||
for Interactive Real-Time Media", draft-ietf-rmcat-cc- | for Interactive Real-Time Media", draft-ietf-rmcat-cc- | |||
requirements-09 (work in progress), December 2014. | requirements-09 (work in progress), December 2014. | |||
[xiph-seq] | [xiph-seq] | |||
Xiph.org, , "Video Test Media", | Xiph.org, , "Video Test Media", | |||
http://media.xiph.org/video/derf/ . | http://media.xiph.org/video/derf/ . | |||
[HEVC-seq] | [HEVC-seq] | |||
skipping to change at page 30, line 43 ¶ | skipping to change at page 32, line 4 ¶ | |||
[xiph-seq] | [xiph-seq] | |||
Xiph.org, , "Video Test Media", | Xiph.org, , "Video Test Media", | |||
http://media.xiph.org/video/derf/ . | http://media.xiph.org/video/derf/ . | |||
[HEVC-seq] | [HEVC-seq] | |||
HEVC, , "Test Sequences", | HEVC, , "Test Sequences", | |||
http://www.netlab.tkk.fi/~varun/test_sequences/ . | http://www.netlab.tkk.fi/~varun/test_sequences/ . | |||
Authors' Addresses | Authors' Addresses | |||
Zaheduzzaman Sarker | Zaheduzzaman Sarker | |||
Ericsson AB | Ericsson AB | |||
Luleae, SE 977 53 | Luleae, SE 977 53 | |||
Sweden | Sweden | |||
Phone: +46 10 717 37 43 | Phone: +46 10 717 37 43 | |||
Email: zaheduzzaman.sarker@ericsson.com | Email: zaheduzzaman.sarker@ericsson.com | |||
Varun Singh | Varun Singh | |||
Aalto University | Nemu Dialogue Systems Oy | |||
School of Electrical Engineering | Runeberginkatu 4c A 4 | |||
Otakaari 5 A | Helsinki 00100 | |||
Espoo, FIN 02150 | ||||
Finland | Finland | |||
Email: varun@comnet.tkk.fi | Email: varun.singh@iki.fi | |||
URI: http://www.netlab.tkk.fi/~varun/ | URI: http://www.callstats.io/ | |||
Xiaoqing Zhu | Xiaoqing Zhu | |||
Cisco Systems | Cisco Systems | |||
510 McCarthy Blvd | 12515 Research Blvd | |||
Milpitas, CA 95134 | Austing, TX 78759 | |||
USA | USA | |||
Email: xiaoqzhu@cisco.com | Email: xiaoqzhu@cisco.com | |||
Michael A. Ramalho | Michael A. Ramalho | |||
Cisco Systems, Inc. | Cisco Systems, Inc. | |||
6310 Watercrest Way Unit 203 | 6310 Watercrest Way Unit 203 | |||
Lakewood Ranch, FL 34202-5211 | Lakewood Ranch, FL 34202-5211 | |||
USA | USA | |||
End of changes. 148 change blocks. | ||||
347 lines changed or deleted | 370 lines changed or added | |||
This html diff was produced by rfcdiff 1.44. The latest version is available from http://tools.ietf.org/tools/rfcdiff/ |