draft-ietf-rmcat-eval-test-02.txt   draft-ietf-rmcat-eval-test-03.txt 
Network Working Group Z. Sarker Network Working Group Z. Sarker
Internet-Draft Ericsson AB Internet-Draft Ericsson AB
Intended status: Informational V. Singh Intended status: Informational V. Singh
Expires: March 11, 2016 Aalto University Expires: September 9, 2016 callstats.io
X. Zhu X. Zhu
M. Ramalho M. Ramalho
Cisco Systems Cisco Systems
September 8, 2015 March 08, 2016
Test Cases for Evaluating RMCAT Proposals Test Cases for Evaluating RMCAT Proposals
draft-ietf-rmcat-eval-test-02 draft-ietf-rmcat-eval-test-03
Abstract Abstract
The Real-time Transport Protocol (RTP) is used to transmit media in The Real-time Transport Protocol (RTP) is used to transmit media in
multimedia telephony applications, these applications are typically multimedia telephony applications, these applications are typically
required to implement congestion control. The RMCAT working group is required to implement congestion control. This document describes
currently working on candidate algorithms for such interactive real- the test cases to be used in the performance evaluation of such
time multimedia applications. This document describes the test cases congestion control algorithms.
to be used in the performance evaluation of those candidate
algorithms.
Status of This Memo Status of This Memo
This Internet-Draft is submitted in full conformance with the This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79. provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/. Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on March 11, 2016. This Internet-Draft will expire on September 9, 2016.
Copyright Notice Copyright Notice
Copyright (c) 2015 IETF Trust and the persons identified as the Copyright (c) 2016 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License. described in the Simplified BSD License.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2
2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3
3. Structure of Test cases . . . . . . . . . . . . . . . . . . . 3 3. Structure of Test cases . . . . . . . . . . . . . . . . . . . 3
4. Recommended Evaluation Settings . . . . . . . . . . . . . . . 7 4. Recommended Evaluation Settings . . . . . . . . . . . . . . . 7
4.1. Evaluation metircs . . . . . . . . . . . . . . . . . . . 7 4.1. Evaluation metrics . . . . . . . . . . . . . . . . . . . 8
4.2. Path characteristics . . . . . . . . . . . . . . . . . . 8 4.2. Path characteristics . . . . . . . . . . . . . . . . . . 8
4.3. Media source . . . . . . . . . . . . . . . . . . . . . . 9 4.3. Media source . . . . . . . . . . . . . . . . . . . . . . 9
5. Basic Test Cases . . . . . . . . . . . . . . . . . . . . . . 10 5. Basic Test Cases . . . . . . . . . . . . . . . . . . . . . . 10
5.1. Variable Available Capacity with Single RMCAT flow . . . 10 5.1. Variable Available Capacity with a Single Flow . . . . . 10
5.2. Variable Available Capacity with Multiple RMCAT flows . . 13 5.2. Variable Available Capacity with Multiple Flows . . . . . 13
5.3. Congested Feedback Link with Bi-directional RMCAT flows . 14 5.3. Congested Feedback Link with Bi-directional Media Flows . 14
5.4. Competing Flows with Same RMCAT Algorithm . . . . . . . . 16 5.4. Competing Media Flows with same Congestion Control
Algorithm . . . . . . . . . . . . . . . . . . . . . . . . 17
5.5. Round Trip Time Fairness . . . . . . . . . . . . . . . . 19 5.5. Round Trip Time Fairness . . . . . . . . . . . . . . . . 19
5.6. RMCAT Flow competing with a long TCP Flow . . . . . . . . 20 5.6. Media Flow Competing with a Long TCP Flow . . . . . . . . 21
5.7. RMCAT Flow competing with short TCP Flows . . . . . . . . 23 5.7. Media Flow Competing with Short TCP Flows . . . . . . . . 23
5.8. Media Pause and Resume . . . . . . . . . . . . . . . . . 25 5.8. Media Pause and Resume . . . . . . . . . . . . . . . . . 25
6. Other potential test cases . . . . . . . . . . . . . . . . . 26 6. Other potential test cases . . . . . . . . . . . . . . . . . 27
6.1. Explicit Congestion Notification Usage . . . . . . . . . 26 6.1. Media Flows with Priority . . . . . . . . . . . . . . . . 27
6.2. Multiple Bottlenecks . . . . . . . . . . . . . . . . . . 26 6.2. Explicit Congestion Notification Usage . . . . . . . . . 27
6.3. Multiple Bottlenecks . . . . . . . . . . . . . . . . . . 27
7. Wireless Access Links . . . . . . . . . . . . . . . . . . . . 29 7. Wireless Access Links . . . . . . . . . . . . . . . . . . . . 29
8. Security Considerations . . . . . . . . . . . . . . . . . . . 29 8. Security Considerations . . . . . . . . . . . . . . . . . . . 30
9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 29 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 30
10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 29 10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 30
11. References . . . . . . . . . . . . . . . . . . . . . . . . . 29 11. References . . . . . . . . . . . . . . . . . . . . . . . . . 30
11.1. Normative References . . . . . . . . . . . . . . . . . . 29 11.1. Normative References . . . . . . . . . . . . . . . . . . 30
11.2. Informative References . . . . . . . . . . . . . . . . . 30 11.2. Informative References . . . . . . . . . . . . . . . . . 31
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 30 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 31
1. Introduction 1. Introduction
This memo describes a set of test cases for evaluating candidate This memo describes a set of test cases for evaluating congestion
RMCAT congestion control algorithm proposals, it is based on the control algorithm proposals for real-time interactive media. It is
guidelines enumerated in [I-D.ietf-rmcat-eval-criteria] and the based on the guidelines enumerated in [I-D.ietf-rmcat-eval-criteria]
requirements discussed in [I-D.ietf-rmcat-cc-requirements]. The test and the requirements discussed in [I-D.ietf-rmcat-cc-requirements].
cases cover basic usage scenarios and are described using a common The test cases cover basic usage scenarios and are described using a
structure, which allows for additional test cases to be added to common structure, which allows for additional test cases to be added
those described herein to accommodate other topologies and/or the to those described herein to accommodate other topologies and/or the
modeling of different path characteristics. It is the intention of modeling of different path characteristics. The described test cases
this work to capture the consensus of the RMCAT working group in this memo SHOULD be used to evaluate any proposed congestion
participants regarding the test cases upon which the performance of control algorithm for real-time interactive media.
the candidate RMCAT proposals should be evaluated.
2. Terminology 2. Terminology
The terminology defined in RTP [RFC3550], RTP Profile for Audio and The terminology defined in RTP [RFC3550], RTP Profile for Audio and
Video Conferences with Minimal Control [RFC3551], RTCP Extended Video Conferences with Minimal Control [RFC3551], RTCP Extended
Report (XR) [RFC3611], Extended RTP Profile for RTCP-based Feedback Report (XR) [RFC3611], Extended RTP Profile for RTCP-based Feedback
(RTP/AVPF) [RFC4585], and Support for Reduced-Size RTCP [RFC5506] (RTP/AVPF) [RFC4585], and Support for Reduced-Size RTCP [RFC5506]
apply. apply.
3. Structure of Test cases 3. Structure of Test cases
All test cases in this document follow a basic structure allowing All the test cases in this document follow a basic structure allowing
implementers to describe a new test scenario without repeatedly implementers to describe a new test scenario without repeatedly
explaining common attributes. The structure includes a general explaining common attributes. The structure includes a general
description section that describes the test case and its motivation. description section that describes the test case and its motivation.
Additionally the test case defines a set of attributes that Additionally the test case defines a set of attributes that
characterize the testbed, i.e., the network path between characterize the testbed, for example, the network path between
communicating peers and the diverse traffic sources. communicating peers and the diverse traffic sources.
o Define the test case: o Define the test case:
* General description: describes the motivation and the goals of * General description: describes the motivation and the goals of
the test case. the test case.
* Expected behavior: describe the desired rate adaptation * Expected behavior: describes the desired rate adaptation
behaviour. behavior.
* Define a check-list to evaluate the desired behaviour: this * Define a list of metrics to evaluate the desired behavior: this
indicates the minimum set of metrics (e.g., link utilization, indicates the minimum set of metrics (e.g., link utilization,
media sending rate) that a proposed algorithm needs to measure media sending rate) that a proposed algorithm needs to measure
to validate the expected rate adaptation behaviour. It should to validate the expected rate adaptation behavior. It should
also indicate the time granularity (e.g., averaged over 10ms, also indicate the time granularity (e.g., averaged over 10ms,
100ms, or 1s) for measuring certain metrics. Typical 100ms, or 1s) for measuring certain metrics. Typical
measurement interval is 200ms. measurement interval is 200ms.
o Define testbed topology: every test case needs to define an o Define testbed topology: every test case needs to define an
evaluation testbed topology. Figure 1 shows such an evaluation evaluation testbed topology. Figure 1 shows such an evaluation
topology. In this evaluation topology, S1..Sn are traffic topology. In this evaluation topology, S1..Sn are traffic
sources. These sources generate media traffic and use either an sources. These sources generate media traffic and use either
RMCAT candidate congestion control algorithm or other congestion congestion control algorithm under investigation. R1..Rn are the
control algorithm designed for media, such as TFRC. R1..Rn are corresponding receivers. A test case can have one or more such
the corresponding receivers. A test case can have one or more traffic sources (S) and their corresponding receivers (R). The
such traffic sources (S) and corresponding receivers (R). The path from the source to destination is denoted as "forward" and
path from the source to destination is denoted as forward and the the path from a destination to a source is denoted as "backward".
path from a destination to a source is denoted as backward. The The following basic structure of the test case has been described
following basic structure of test case has been described from the from the perspective of media generating endpoints attached on the
perspective of media generating endpoints attached on the left- left-hand side of Figure 1. In this setup, the media flows are
hand side of Figure 1. In this setup, media flows in forward transported in forward direction and corresponding feedback/
direction and corresponding feedback/control messages flow in the control messages are transported in the backward direction.
backward direction. However, it is also possible to set up the However, it is also possible to set up the test with media in both
test with media flowing in both forward and backward directions. forward and backward directions. In that case, unless otherwise
In that case, unless otherwise specified by the test case, it is specified by the test case, it is expected that the backward path
expected that the backward path does not introduce any congestion does not introduce any congestion related impairments and has
related impairments and has enough capacity to accommodate both enough capacity to accommodate both media and feedback/control
media and feedback/control messages. It should be noted that messages. It should be noted that depending on the test cases it
depending on the test cases it is possible to have different path is possible to have different path characteristics in either of
characteristics in of the either directions. the directions.
o
+---+ +---+ +---+ +---+
|S1 |====== \ Forward --> / =======|R1 | |S1 |====== \ Forward --> / =======|R1 |
+---+ \\ // +---+ +---+ \\ // +---+
\\ // \\ //
+---+ +-----+ +-----+ +---+ +---+ +-----+ +-----+ +---+
|S2 |=======| A |------------------------------>| B |=======|R2 | |S2 |=======| A |------------------------------>| B |=======|R2 |
+---+ | |<------------------------------| | +---+ +---+ | |<------------------------------| | +---+
+-----+ +-----+ +-----+ +-----+
(...) // \\ (...) (...) // \\ (...)
// <-- Backward \\ // <-- Backward \\
+---+ // \\ +---+ +---+ // \\ +---+
|Sn |====== / \ ======|Rn | |Sn |====== / \ ======|Rn |
+---+ +---+ +---+ +---+
Figure 1: Example of A Testbed Topology Figure 1: Example of A Testbed Topology
In a laboratory testbed environment there may exist a significant In a testbed environment where real equipments are used to create
amount of traffic on portions of the network path between the a laboratory, there may exist a significant amount of traffic on
endpoints that is not desired for the purposes of these RMCAT portions of the network path between the endpoints that is not
tests. Some of this traffic may be generated by other processes desired for the purposes of the tests described in the document.
on the endpoints themselves (e.g., discovery protocols) or by Some of this traffic may be generated by other processes on the
other endpoints not presently under test. It is recommended not endpoints themselves (e.g., discovery protocols) or by other
to route traffic generated by endpoints that are not under test endpoints not presently under test. It is recommended not to
through the test bed. Additionally, it is recommended to route route traffic generated by endpoints that are not under test
non-RMCAT traffic generated by the endpoints under test around the through the test bed and route those traffic generated by the
bottleneck links specified herein. endpoints under test around the bottleneck links specified herein.
o Define testbed attributes: o Define testbed attributes:
* Duration: defines the duration of the test. * Duration: defines the duration of the test in seconds.
* Path characteristics: defines the end-to-end transport level * Path characteristics: defines the end-to-end transport level
path characteristics of the testbed in a particular test case. path characteristics of the testbed for a particular test case.
Two sets of attributes describe the path characteristics, one Two sets of attributes describe the path characteristics, one
for the forward path and the other for the backward path. The for the forward path and the other for the backward path. The
path characteristics for a particular path direction is path characteristics for a particular path direction is
applicable to all the Sources "S" sending traffic on that path. applicable to all the Sources "S" sending traffic on that path.
If only one attribute is specified, it is used for both path If only one attribute is specified, it is used for both path
directions, however, unless specified the reverse path has no directions, however, unless specified the reverse path has no
capacity restrictions and no path loss. capacity restrictions and no path loss.
+ Path direction: forward or backward. + Path direction: forward or backward.
skipping to change at page 5, line 33 skipping to change at page 5, line 31
along the path when network queues are empty, i.e., the time along the path when network queues are empty, i.e., the time
it takes for a packet to go from the sender to the receiver it takes for a packet to go from the sender to the receiver
without encountering any queuing delay. without encountering any queuing delay.
+ Maximum end-to-end jitter: defines the maximum jitter that + Maximum end-to-end jitter: defines the maximum jitter that
can be observed along the path. can be observed along the path.
+ Bottleneck queue type: for example, Droptail, FQ-CoDel, or + Bottleneck queue type: for example, Droptail, FQ-CoDel, or
PIE. PIE.
+ Bottleneck queue size: defines size of queue in terms of + Bottleneck queue size: defines the size of queue in terms of
queuing time when the queue is full (in milliseconds). queuing time when the queue is full (in milliseconds).
+ Path loss ratio: characterizes the non-congested, additive, + Path loss ratio: characterizes the non-congested, additive,
losses to be generated on the end-to-end path. MUST losses to be generated on the end-to-end path. MUST
describe the loss pattern or loss model used to generate the describe the loss pattern or loss model used to generate the
losses. losses.
+ Values for some characteristics are described in * Application-related: defines the traffic source behavior for
[I-D.ietf-rmcat-eval-criteria].
* Application-related: defines the traffic source behaviour for
implementing the test case implementing the test case
+ Media traffic Source: defines the characteristics of the + Media traffic Source: defines the characteristics of the
media sources. When using more than one media source, the media sources. When using more than one media source, the
different attributes are enumerated separately for each different attributes are enumerated separately for each
different media source. different media source.
- Media type: Video/Voice - Media type: Video/Voice
- Media flow direction: forward, backward or both. - Media flow direction: forward, backward or both.
- Number of media sources: defines the total number of - Number of media sources: defines the total number of
media sources media sources
- Media codec: Constant Bit Rate (CBR) or Variable Bit Rate - Media codec: Constant Bit Rate (CBR) or Variable Bit Rate
(VBR) (VBR)
- Media source behaviour: describes the media encoder - Media source behavior: describes the media encoder
behavior. It defines the main parameters that affect the behavior. It defines the main parameters that affect the
adaptation behaviour. This may include but not limited adaptation behavior. This may include but is not limited
to: to:
o Adaptability: describes the adaptation options. For o Adaptability: describes the adaptation options. For
example, in the case of video it defines the following example, in the case of video it defines the following
ranges of adaptation: bit rate, frame rate, video ranges of adaptation: bit rate, frame rate, video
resolution. Similarly, in the case of voice, it resolution. Similarly, in the case of voice, it
defines the range of bit rate adaptation, the sampling defines the range of bit rate adaptation, the sampling
rate variation, and the variation in packetization rate variation, and the variation in packetization
interval. interval.
o Output variation : for a VBR encoder it defines the o Output variation : for a VBR encoder it defines the
encoder output variation from the average target rate encoder output variation from the average target rate
over a particular measurement interval. For example, over a particular measurement interval. For example,
on average the encoder output may vary between 5% to on average the encoder output may vary between 5% to
15% above or below the average target bit rate when 15% above or below the average target bit rate when
measured over a 100 ms time window. The time interval measured over a 100 ms time window. The time interval
over which the variation is specified must be over which the variation is specified must be
provided. provided.
o Responsiveness to a new bit rate request: the lag in o Responsiveness to a new bit rate request: the lag in
time between a new bit rate request and actual rate time between a new bit rate request from the
changes in encoder output. Depending on the encoder, congestion control algorithm and actual rate changes
this value may be specified in absolute time (e.g. in encoder output. Depending on the encoder, this
10ms to 1000ms) or other appropriate metric (next value may be specified in absolute time (e.g. 10ms to
frame interval time). 1000ms) or other appropriate metric (e.g. next frame
interval time).
More detailed discussions on expected media source
behavior, including those from synthetic video traffic
sources, is at [I-D.ietf-rmcat-video-traffic-model].
- Media content: describes the chosen media sequences; For - Media content: describes the chosen media sequences; For
example, test sequences are available at: [xiph-seq] and example, test sequences are available at: [xiph-seq] and
[HEVC-seq]. [HEVC-seq].
- Media timeline: describes the point when the media source - Media timeline: describes the point when the media source
is introduced and removed from the testbed. For example, is introduced and removed from the testbed. For example,
the media source may start transmitting immediately when the media source may start transmitting immediately when
the test case begins, or after a few seconds. the test case begins, or after a few seconds.
- Startup behaviour: the media starts at a defined bit - Startup behavior: the media starts at a defined bit rate,
rate, which may be the minimum, maximum bit rate, or a which may be the minimum, maximum bit rate, or a value in
value in between (in Kbps). between (in Kbps).
+ Competing traffic source: describes the characteristics of + Competing traffic source: describes the characteristics of
the competing traffic source, the different types of the competing traffic source, the different types of
competing flows are enumerated in competing flows are enumerated in
[I-D.ietf-rmcat-eval-criteria]. [I-D.ietf-rmcat-eval-criteria].
- Traffic direction: forward, backward or both. - Traffic direction: forward, backward or both.
- Type of sources: defines the types of competing traffic - Type of sources: defines the types of competing traffic
sources. Types of competing traffic flows are listed in sources. Types of competing traffic flows are listed in
[I-D.ietf-rmcat-eval-criteria]. For example, the number [I-D.ietf-rmcat-eval-criteria]. For example, the number
of TCP flows connected to a web browser, the mean size of TCP flows connected to a web browser, the mean size
and distribution of the content downloaded. and distribution of the content downloaded.
- Number of sources: defines the total number of competing - Number of sources: defines the total number of competing
sources of each media type. sources of each media type per traffic direction.
- Congestion control: enumerates the congestion control - Congestion control: enumerates the congestion control
used by each type of competing traffic. used by each type of competing traffic.
- Traffic timeline: describes when the competing traffic - Traffic timeline: describes when the competing traffic
starts and ends in the test case. starts and ends in the test case.
* Additional attributes: describes attributes essential for * Additional attributes: describes attributes essential for
implementing a test case which are not included in the above implementing a test case which are not included in the above
structure. These attributes MUST be well defined, so that structure. These attributes MUST be well defined, so that the
other implementers are able to implement it. other implementers of that particular test case are able to
implement it easily.
Any attribute can have a set of values (enclosed within "[]"). Each Any attribute can have a set of values (enclosed within "[]"). Each
member value of such a set MUST be treated as different value for the member value of such a set MUST be treated as different value for the
same attribute. It is desired to run separate tests for each such same attribute. It is desired to run separate tests for each such
attribute value. attribute value.
The test cases described in this document follow the above structure. The test cases described in this document follow the above structure.
4. Recommended Evaluation Settings 4. Recommended Evaluation Settings
This section describes recommended test case settings and could be This section describes recommended test case settings and could be
overwritten by the respective test cases. overwritten by the respective test cases.
4.1. Evaluation metircs 4.1. Evaluation metrics
To evaluate the performance of the candidate algorithms it is To evaluate the performance of the candidate algorithms the
expected to log enough information to visualize the following metrics implementers MUST log enough information to visualize the following
at a fine enough time granularity: metrics at a fine enough time granularity:
1. Flow level: 1. Flow level:
A. End-to-end delay for the RMCAT flow. A. End-to-end delay for the congestion controlled media flow.
B. Variation in sending bit rate and goodput. Mainly observing B. Variation in sending bit rate and goodput. Mainly observing
the frequency and magnitude of oscillations. the frequency and magnitude of oscillations.
C. Packet losses observed at the receiving endpoint C. Packet losses observed at the receiving endpoint.
D. Feedback message overhead D. Feedback message overhead.
E. Convergence time. E. Convergence time - time to reach steady state for the
congestion controlled media flow(s).
2. Transport level: 2. Transport level:
A. Bandwidth utilization A. Bandwidth utilization.
B. Queue length (milliseconds at specified path capacity): B. Queue length (milliseconds at specified path capacity):
+ average over the length of the session + average over the length of the session.
+ 5 and 95 percentile + 5 and 95 percentile.
+ median, maximum, minimum + median, maximum, minimum.
4.2. Path characteristics 4.2. Path characteristics
Each path between a sender and receiver as described in Figure 1 have Each path between a sender and receiver as described in Figure 1 have
the following characteristics unless otherwise specified in the test the following characteristics unless otherwise specified in the test
case. case.
o Path direction: forward and backward. o Path direction: forward and backward.
o Reference bottleneck capacity: 1Mbps. o Reference bottleneck capacity: 1Mbps.
skipping to change at page 9, line 17 skipping to change at page 9, line 23
Examples of additional network parameters are discussed in Examples of additional network parameters are discussed in
[I-D.ietf-rmcat-eval-criteria]. [I-D.ietf-rmcat-eval-criteria].
For test cases involving time-varying bottleneck capacity, all For test cases involving time-varying bottleneck capacity, all
capacity values are specified as a ratio with respect to a reference capacity values are specified as a ratio with respect to a reference
capacity value, so as to allow flexible scaling of capacity values capacity value, so as to allow flexible scaling of capacity values
along with media source rate range. There exist two different along with media source rate range. There exist two different
mechanisms for inducing path capacity variation: a) by explicitly mechanisms for inducing path capacity variation: a) by explicitly
modifying the value of physical link capacity; or b) by introducing modifying the value of physical link capacity; or b) by introducing
background non-adaptive UDP traffic with time-varying traffic rate. background non-adaptive UDP traffic with time-varying traffic rate.
Implementers are encouraged run the experiments with both mechanisms Implementers are encouraged to run the experiments with both
for test cases specified in Section 5.1, Section 5.2, and mechanisms for test cases specified in Section 5.1, Section 5.2, and
Section 5.3. Section 5.3.
4.3. Media source 4.3. Media source
Unless otherwise specified, each test case will include one or more Unless otherwise specified, each test case will include one or more
media sources as described below. media sources as described below.
o Media type: Video o Media type: Video
* Media codec: VBR * Media codec: VBR
* Media source behaviour: * Media source behavior:
+ Adaptability: + Adaptability:
- Bit rate range: 150 Kbps - 1.5 Mbps. In real-life - Bit rate range: 150 Kbps - 1.5 Mbps. In real-life
applications the bitrate range can vary a lot depending applications the bit rate range can vary a lot depending
on the provided service, for example, the maximum bitrate on the provided service, for example, the maximum bit
can be up to 4Mbps. However, for running tests to rate can be up to 4Mbps. However, for running tests to
evaluate the congestion control algorithms it is more evaluate the congestion control algorithms it is more
important to have a look at how they are reacting to important to have a look at how they are reacting to
certain amount of bandwidth change. Also it is possible certain amount of bandwidth change. Also it is possible
that the media traffic generator used in a particular that the media traffic generator used in a particular
simulator or testbed if not capable of generating higher simulator or testbed is not capable of generating higher
bitrate. Hence we have selected a suitable bitrate range bit rate. Hence we have selected a suitable bit rate
typical of consumer-grade video conferencing applications range typical of consumer-grade video conferencing
in designing the test case. If a different bitrate range applications in designing the test case. If a different
is used in the test cases, the end-to-end path capacity bit rate range is used in the test cases, then the end-
values will also need to be scaled accordingly. to-end path capacity values will also need to be scaled
accordingly.
- Frame resolution: 144p - 720p (or 1080p) - Frame resolution: 144p - 720p (or 1080p). This
resolution range is selected based on the bit rate range.
If a different bit rate range is used in the test cases
then the frame resolution range also need to be selected
suitably.
- Frame rate: 10fps - 30fps - Frame rate: 10fps - 30fps. This frame rate range is
selected based on the bit rate range. If a different bit
rate range is used in the test cases then the frame rate
range also need to be adjusted suitably.
+ Variation from target bitrate: +/-5%. Unless otherwise + Variation from target bit rate: +/-5%. Unless otherwise
specified in the test case, bitrate variation SHOULD be specified in the test case(s), bit rate variation SHOULD be
calculated over one (1) second period of time. calculated over one (1) second period of time.
+ Responsiveness to new bit rate request: 100ms + Responsiveness to new bit rate request: 100ms
* Media content: The media content should represent a typical * Media content: The media content should represent a typical
video conversational scenario with head and shoulder movement. video conversational scenario with head and shoulder movement.
We recommend to use Foreman video sequence. We recommend to use Foreman video sequence.
* Media startup behaviour: 150Kbps. It should be noted that * Media startup behavior: 150Kbps. It should be noted that
applications can use smart ways to select an optimal startup applications can use smart ways to select an optimal startup
bitrate values for a certain network condition. In such cases bit rate value for a certain network condition. In such cases
the candidate proposals MAY show the effectiveness of such the candidate proposals MAY show the effectiveness of such
smart approach as an additional information for the evaluation smart approach as an additional information for the evaluation
process. process.
o Media type: Audio o Media type: Audio
* Media codec: CBR * Media codec: CBR
* Media bitrate: 20Kbps * Media bit rate: 20Kbps
5. Basic Test Cases 5. Basic Test Cases
5.1. Variable Available Capacity with Single RMCAT flow 5.1. Variable Available Capacity with a Single Flow
In this test case the bottleneck-link capacity between the two In this test case the bottleneck-link capacity between the two
endpoints varies over time. This test is designed to measure the endpoints varies over time. This test is designed to measure the
responsiveness of the candidate algorithm. This test tries to responsiveness of the candidate algorithm. This test tries to
address the requirements in [I-D.ietf-rmcat-cc-requirements], which address the requirements in [I-D.ietf-rmcat-cc-requirements], which
requires the algorithm to adapt the flow(s) and provide lower end-to- requires the algorithm to adapt the flow(s) and provide lower end-to-
end latency when there exists: end latency when there exists:
o an intermediate bottleneck o an intermediate bottleneck
o change in available capacity (e.g., due to interface change, o change in available capacity (e.g., due to interface change,
routing change, abrupt arrival/departure of background non- routing change, abrupt arrival/departure of background non-
adaptive traffic). adaptive traffic).
o maximum Media Bit Rate is Greater than Link Capacity. In this o maximum media bit rate is greater than link capacity. In this
case, the application will attempt to ramp up to its maximum bit case, the application will attempt to ramp up to its maximum bit
rate, since the link capacity is limited to a value lower, the rate, since the link capacity is limited to a value lower, the
congestion control scheme is expected to stabilize the sending bit congestion control scheme is expected to stabilize the sending bit
rate close to the available bottleneck capacity. This situation rate close to the available bottleneck capacity.
can occur when the endpoints are connected via thin long networks
even though the advertised capacity of the access network may be
higher.
It should be noted that the exact variation in available capacity due It should be noted that the exact variation in available capacity due
to any of the above depends on the under-lying technologies. Hence, to any of the above depends on the underlying technologies. Hence,
we describe a set of known factors, which may be extended to devise a we describe a set of known factors, which may be extended to devise a
more specific test case targeting certain behaviour in a certain more specific test case targeting certain behaviors in a certain
network environment. network environment.
Expected behavior: the candidate algorithm is expected to detect the Expected behavior: the candidate algorithm is expected to detect the
path capacity constraint, converges to bottleneck link's capacity and path capacity constraint, converges to the bottleneck link's capacity
adapt the flow to avoid unwanted oscillation when the sending bit and adapt the flow to avoid unwanted oscillation when the sending bit
rate is approaching the bottleneck link's capacity. The oscillations rate is approaching the bottleneck link's capacity. The oscillations
occur when the media flow(s) attempts to reach its maximum bit rate, occur when the media flow(s) attempts to reach its maximum bit rate
overshoots the usage of the available bottleneck capacity, to rectify but overshoots the usage of the available bottleneck capacity then to
it reduces the bit rate and starts to ramp up again. rectify, it reduces the bit rate and starts to ramp up again.
Testbed topology: One media source S1 is connected to corresponding Evaluation metrics : as described in Section 4.1.
R1. The media traffic is transported over the forward path and
corresponding feedback/control traffic is transported over the Testbed topology: One media source S1 is connected to the
backward path. corresponding R1. The media traffic is transported over the forward
path and corresponding feedback/control traffic is transported over
the backward path.
Forward --> Forward -->
+---+ +-----+ +-----+ +---+ +---+ +-----+ +-----+ +---+
|S1 |=======| A |------------------------------>| B |=======|R1 | |S1 |=======| A |------------------------------>| B |=======|R1 |
+---+ | |<------------------------------| | +---+ +---+ | |<------------------------------| | +---+
+-----+ +-----+ +-----+ +-----+
<-- Backward <-- Backward
Figure 2: Testbed Topology for Limited Link Capacity Figure 2: Testbed Topology for Limited Link Capacity
To evaluate the performance of the candidate algorithms it is
expected to log enough information to visualize the metrics described
in Section 4.1 at a fine enough time granularity.
Testbed attributes: Testbed attributes:
o Test duration: 100s o Test duration: 100s
o Path characteristics: as described in Section 4.2 o Path characteristics: as described in Section 4.2
o Application-related: o Application-related:
* Media Traffic: * Media Traffic:
+ Media type: Video + Media type: Video
- Media direction: forward. - Media direction: forward.
- Number of media sources: One (1) - Number of media sources: one (1)
- Media timeline: - Media timeline:
o Start time: 0s. o Start time: 0s.
o End time: 99s. o End time: 99s.
+ Media type: Audio + Media type: Audio
- Media direction: forward. - Media direction: forward.
- Number of media sources: One (1) - Number of media sources: one (1)
- Media timeline: - Media timeline:
o Start time: 0s. o Start time: 0s.
o End time: 99s. o End time: 99s.
* Competing traffic: * Competing traffic:
+ Number of sources : Zero (0) + Number of sources : zero (0)
o Test Specific Information: o Test Specific Information:
* This test uses the following one way propagation delays of 50 * One-way propagation delay: [ 50 ms, 100 ms]. on the forward
ms and 100 ms. path direction
* This test uses bottleneck path capacity variation as listed in * This test uses bottleneck path capacity variation as listed in
Table 1 Table 1
* When using background non-adaptive UDP traffic to induce time- * When using background non-adaptive UDP traffic to induce time-
varying bottleneck for the RMCAT flow, the physical path varying bottleneck , the physical path capacity remains at
capacity is 4Mbps and the UDP traffic source rate changes over 4Mbps and the UDP traffic source rate changes over time as
time as (4-x)Mbps, where x is the bottleneck capacity specified (4-x)Mbps, where x is the bottleneck capacity specified in
in Table 1 Table 1
+--------------------+--------------+-----------+-------------------+ +--------------------+--------------+-----------+-------------------+
| Variation pattern | Path | Start | Path capacity | | Variation pattern | Path | Start | Path capacity |
| index | direction | time | ratio | | index | direction | time | ratio |
+--------------------+--------------+-----------+-------------------+ +--------------------+--------------+-----------+-------------------+
| One | Forward | 0s | 1.0 | | One | Forward | 0s | 1.0 |
| Two | Forward | 40s | 2.5 | | Two | Forward | 40s | 2.5 |
| Three | Forward | 60s | 0.6 | | Three | Forward | 60s | 0.6 |
| Four | Forward | 80s | 1.0 | | Four | Forward | 80s | 1.0 |
+--------------------+--------------+-----------+-------------------+ +--------------------+--------------+-----------+-------------------+
Table 1: Path capacity variation pattern for forward direction Table 1: Path capacity variation pattern for forward direction
5.2. Variable Available Capacity with Multiple RMCAT flows 5.2. Variable Available Capacity with Multiple Flows
This test case is similar to Section 5.1. However in addition this This test case is similar to Section 5.1. However in addition this
test will also consider persistent network load due to competing test will also consider persistent network load due to competing
traffic. traffic.
Expected behavior: the candidate algorithms is expected to detect the Expected behavior: the candidate algorithm is expected to detect the
variation in available capacity and adapt the media stream(s) variation in available capacity and adapt the media stream(s)
accordingly. The flows stabilize around their maximum bitrate as the accordingly. The flows stabilize around their maximum bit rate as
as the maximum link capacity is large enough to accommodate the the maximum link capacity is large enough to accommodate the flows.
flows. When the available capacity drops, the flow(s) adapts by When the available capacity drops, the flows adapt by decreasing
decreasing its sending bit rate, and when congestion disappears, the their sending bit rate, and when congestion disappears, the flows are
flow(s) are again expected to ramp up. again expected to ramp up.
To evaluate the performance of the candidate algorithms it is Evaluation metrics : as described in Section 4.1.
expected to log enough information to visualize the metrics described
in Section 4.1 at a fine enough time granularity:
Testbed Topology: Two (2) media sources S1 and S2 are connected to Testbed Topology: Two (2) media sources S1 and S2 are connected to
their corresponding destinations R1 and R2. The media traffic is their corresponding destinations R1 and R2. The media traffic is
transported over the forward path and corresponding feedback/control transported over the forward path and corresponding feedback/control
traffic is transported over the backward path. traffic is transported over the backward path.
+---+ +---+ +---+ +---+
|S1 |===== \ / =======|R1 | |S1 |===== \ / =======|R1 |
+---+ \\ Forward --> // +---+ +---+ \\ Forward --> // +---+
\\ // \\ //
skipping to change at page 13, line 52 skipping to change at page 14, line 13
Figure 3: Testbed Topology for Variable Available Capacity Figure 3: Testbed Topology for Variable Available Capacity
Testbed attributes: Testbed attributes:
Testbed attributes are similar as described in Section 5.1 except the Testbed attributes are similar as described in Section 5.1 except the
test specific capacity variation setup. test specific capacity variation setup.
Test Specific Information: This test uses path capacity variation as Test Specific Information: This test uses path capacity variation as
listed in Table 2 with a corresponding end time of 125 seconds. The listed in Table 2 with a corresponding end time of 125 seconds. The
reference bottleneck capacity is 2Mbps. When using background non- reference bottleneck capacity is 2Mbps. When using background non-
adaptive UDP traffic to induce time-varying bottleneck for RMCAT adaptive UDP traffic to induce time-varying bottleneck for congestion
flows, the physical path capacity is 4Mbps and the UDP traffic source controlled media flows, the physical path capacity is 4Mbps and the
rate changes over time as (4-x)Mbps, where x is the bottleneck UDP traffic source rate changes over time as (4-x)Mbps, where x is
capacity specified in Table 2. the bottleneck capacity specified in Table 2.
+--------------------+--------------+-----------+-------------------+ +--------------------+--------------+-----------+-------------------+
| Variation pattern | Path | Start | Path capacity | | Variation pattern | Path | Start | Path capacity |
| index | direction | time | ratio | | index | direction | time | ratio |
+--------------------+--------------+-----------+-------------------+ +--------------------+--------------+-----------+-------------------+
| One | Forward | 0s | 2.0 | | One | Forward | 0s | 2.0 |
| Two | Forward | 25s | 1.0 | | Two | Forward | 25s | 1.0 |
| Three | Forward | 50s | 1.75 | | Three | Forward | 50s | 1.75 |
| Four | Forward | 75s | 0.5 | | Four | Forward | 75s | 0.5 |
| Five | Forward | 100s | 1.0 | | Five | Forward | 100s | 1.0 |
+--------------------+--------------+-----------+-------------------+ +--------------------+--------------+-----------+-------------------+
Table 2: Path capacity variation pattern for forward direction Table 2: Path capacity variation pattern for forward direction
5.3. Congested Feedback Link with Bi-directional RMCAT flows 5.3. Congested Feedback Link with Bi-directional Media Flows
RMCAT WG has been chartered to define algorithms for RTP hence it is Real-time interactive media uses RTP hence it is assumed that RTCP,
assumed that RTCP, RTP header extension or such would be used by the RTP header extension or such would be used by the congestion control
congestion control algorithm in the backchannel. Due to asymmetric algorithm in the backchannel. Due to asymmetric nature of the link
nature of the link between communicating peers it is possible for a between communicating peers it is possible for a participating peer
participating peer to not receive such feedback information due to an to not receive such feedback information due to an impaired or
impaired or congested backchannel (even when the forward channel congested backchannel (even when the forward channel might not be
might not be impaired). This test case is designed to observe the impaired). This test case is designed to observe the candidate
candidate congestion control behaviour in such an event. congestion control behavior in such an event.
It is expected that the candidate algorithms is able to cope with the It is expected that the candidate algorithms are able to cope with
lack of feedback information and adapt to minimize the performance the lack of feedback information and adapt to minimize the
degradation of media flows in the forward channel. performance degradation of media flows in the forward channel.
It should be noted that for this test case: logs are compared with It should be noted that for this test case: logs are compared with
the reference case, i.e, when the backward channel has no impairments the reference case, i.e, when the backward channel has no
impairments.
To evaluate the performance of the candidate algorithms it is Evaluation metrics : as described in Section 4.1.
expected to log enough information to visualize the metrics described
in Section 4.1 at a fine-grained time intervals:
Testbed topology: One (1) media source S1 is connected to Testbed topology: One (1) media source S1 is connected to
corresponding R1, but both endpoints are additionally receiving and corresponding R1, but both endpoints are additionally receiving and
sending data, respectively. The media traffic (S1->R1) is sending data, respectively. The media traffic (S1->R1) is
transported over the forward path and corresponding feedback/control transported over the forward path and corresponding feedback/control
traffic is transported over the backward path. Likewise media traffic is transported over the backward path. Likewise media
traffic (S2->R2) is transported over the backward path and traffic (S2->R2) is transported over the backward path and
corresponding feedback/control traffic is transported over the corresponding feedback/control traffic is transported over the
forward path. forward path.
skipping to change at page 15, line 37 skipping to change at page 15, line 46
* Reference bottleneck capacity: 1Mbps. * Reference bottleneck capacity: 1Mbps.
o Application-related: o Application-related:
* Media Source: * Media Source:
+ Media type: Video + Media type: Video
- Media direction: forward and backward - Media direction: forward and backward
- Number of media sources: Two (2) - Number of media sources: two (2)
- Media timeline: - Media timeline:
o Start time: 0s. o Start time: 0s.
o End time: 99s. o End time: 99s.
+ Media type: Audio + Media type: Audio
- Media direction: forward and backward - Media direction: forward and backward
- Number of media sources: Two (2) - Number of media sources: two (2)
- Media timeline: - Media timeline:
o Start time: 0s. o Start time: 0s.
o End time: 99s. o End time: 99s.
* Competing traffic: * Competing traffic:
+ Number of sources : Zero (0) + Number of sources : zero (0)
o Test Specific Information: This test uses path capacity variations o Test Specific Information: this test uses path capacity variations
to create congested feedback link. Table 3 lists the variation to create congested feedback link. Table 3 lists the variation
patterns applied to the forward path and Table 4 lists the patterns applied to the forward path and Table 4 lists the
variation patterns applied to the backward path. When using variation patterns applied to the backward path. When using
background non-adaptive UDP traffic to induce time-varying background non-adaptive UDP traffic to induce time-varying
bottleneck for RMCAT flows, the physical path capacity is 4Mbps bottleneck for congestion controlled media flows, the physical
for both directions and the UDP traffic source rate changes over path capacity is 4Mbps for both directions and the UDP traffic
time as (4-x)Mbps in each direction, where x is the bottleneck source rate changes over time as (4-x)Mbps in each direction,
capacity specified in Table 4. where x is the bottleneck capacity specified in Table 4.
+--------------------+--------------+-----------+-------------------+ +--------------------+--------------+-----------+-------------------+
| Variation pattern | Path | Start | Path capacity | | Variation pattern | Path | Start | Path capacity |
| index | direction | time | ratio | | index | direction | time | ratio |
+--------------------+--------------+-----------+-------------------+ +--------------------+--------------+-----------+-------------------+
| One | Forward | 0s | 2.0 | | One | Forward | 0s | 2.0 |
| Two | Forward | 20s | 1.0 | | Two | Forward | 20s | 1.0 |
| Three | Forward | 40s | 0.5 | | Three | Forward | 40s | 0.5 |
| Four | Forward | 60s | 2.0 | | Four | Forward | 60s | 2.0 |
+--------------------+--------------+-----------+-------------------+ +--------------------+--------------+-----------+-------------------+
skipping to change at page 16, line 46 skipping to change at page 17, line 5
| Variation pattern | Path | Start | Path capacity | | Variation pattern | Path | Start | Path capacity |
| index | direction | time | ratio | | index | direction | time | ratio |
+--------------------+--------------+-----------+-------------------+ +--------------------+--------------+-----------+-------------------+
| One | Backward | 0s | 2.0 | | One | Backward | 0s | 2.0 |
| Two | Backward | 35s | 0.8 | | Two | Backward | 35s | 0.8 |
| Three | Backward | 70s | 2.0 | | Three | Backward | 70s | 2.0 |
+--------------------+--------------+-----------+-------------------+ +--------------------+--------------+-----------+-------------------+
Table 4: Path capacity variation pattern for backward direction Table 4: Path capacity variation pattern for backward direction
5.4. Competing Flows with Same RMCAT Algorithm 5.4. Competing Media Flows with same Congestion Control Algorithm
In this test case, more than one RMCAT media flow shares the In this test case, more than one media flows share the bottleneck
bottleneck link and each of them uses the same congestion control link and each of them uses the same congestion control algorithm.
algorithm. This is a typical scenario where a real-time interactive This is a typical scenario where a real-time interactive application
application sends more than one media flows to the same destination sends more than one media flow to the same destination and these
and these flows are multiplexed over the same port. In such a flows are multiplexed over the same port. In such a scenario it is
scenario it is likely that the flows will be routed via the same path likely that the flows will be routed via the same path and need to
and need to share the available bandwidth amongst themselves. For share the available bandwidth amongst themselves. For the sake of
the sake of simplicity it is assumed that there are no other non- simplicity it is assumed that there are no other competing traffic
RMCAT competing traffic sources in the bottleneck link and that there sources in the bottleneck link and that there is sufficient capacity
is sufficient capacity to accommodate all the flows individually. to accommodate all the flows individually. While this appears to be
While this appears to be a variant of the test case defined in a variant of the test case defined in Section 5.2, it focuses on the
Section 5.2, it focuses on the capacity sharing aspect of the capacity sharing aspect of the candidate algorithm. The previous
candidate algorithm. The previous test case, on the other hand, test case, on the other hand, measures adaptability, stability, and
measures adaptability, stability, and responsiveness of the candidate responsiveness of the candidate algorithm.
algorithm.
Expected behavior: It is expected that the competing flows will Expected behavior: It is expected that the competing flows will
converge to an optimum bit rate to accommodate all the flows with converge to an optimum bit rate to accommodate all the flows with
minimum possible latency and loss. Specifically, the test introduces minimum possible latency and loss. Specifically, the test introduces
three media flows at different time instances, when the second flow three media flows at different time instances, when the second flow
appears there should still be room to accommodate another flow on the appears there should still be room to accommodate another flow on the
bottleneck link. Lastly, when the third flow appears the bottleneck bottleneck link. Lastly, when the third flow appears the bottleneck
link should be saturated. link should be saturated.
To evaluate the performance of the candidate algorithms it is Evaluation metrics : as described in Section 4.1.
expected to log enough information to visualize the metrics described
in Section 4.1 at a fine enough time granularity:
Testbed topology: Three media sources S1, S2, S3 are connected to Testbed topology: Three media sources S1, S2, S3 are connected to R1,
respective R1, R2, R3. The media traffic is transported over the R2, R3 respectively. The media traffic is transported over the
forward path and corresponding feedback/control traffic is forward path and corresponding feedback/control traffic is
transported over the backward path. transported over the backward path.
+---+ +---+ +---+ +---+
|S1 |===== \ Forward --> / =======|R1 | |S1 |===== \ Forward --> / =======|R1 |
+---+ \\ // +---+ +---+ \\ // +---+
\\ // \\ //
+---+ +-----+ +-----+ +---+ +---+ +-----+ +-----+ +---+
|S2 |=======| A |------------------------------>| B |=======|R2 | |S2 |=======| A |------------------------------>| B |=======|R2 |
+---+ | |<------------------------------| | +---+ +---+ | |<------------------------------| | +---+
+-----+ +-----+ +-----+ +-----+
// \\ // \\
// <-- Backward \\ // <-- Backward \\
+---+ // \\ +---+ +---+ // \\ +---+
|S3 |====== / \ ======|R3 | |S3 |====== / \ ======|R3 |
+---+ +---+ +---+ +---+
Figure 5: Testbed Topology for Multiple RMCAT Flows Figure 5: Testbed Topology for Multiple congestion controlled media
Flows
Testbed attributes: Testbed attributes:
o Test duration: 120s o Test duration: 120s
o Path characteristics: o Path characteristics:
* Reference bottleneck capacity: 3.5Mbps * Reference bottleneck capacity: 3.5Mbps
* Path capacity ratio: 1.0 * Path capacity ratio: 1.0
o Application-related: o Application-related:
* Media Source: * Media Source:
skipping to change at page 18, line 18 skipping to change at page 18, line 23
* Path capacity ratio: 1.0 * Path capacity ratio: 1.0
o Application-related: o Application-related:
* Media Source: * Media Source:
+ Media type: Video + Media type: Video
- Media direction: forward. - Media direction: forward.
- Number of media sources: Three (3) - Number of media sources: three (3)
- Media timeline: New media flows are added sequentially, - Media timeline: new media flows are added sequentially,
at short time intervals. See test specific setup below. at short time intervals. See test specific setup below.
+ Media type: Audio + Media type: Audio
- Media direction: forward. - Media direction: forward.
- Number of media sources: Three (3) - Number of media sources: three (3)
- Media timeline: New media flows are added sequentially, - Media timeline: new media flows are added sequentially,
at short time intervals. See test specific setup below. at short time intervals. See test specific setup below.
* Competing traffic: * Competing traffic:
+ Number of sources : Zero (0) + Number of sources : zero (0)
o Test Specific Information: Table 5 defines the media timeline for o Test Specific Information: Table 5 defines the media timeline for
both media type. both media type.
+---------+------------+------------+----------+ +---------+------------+------------+----------+
| Flow IF | Media type | Start time | End time | | Flow ID | Media type | Start time | End time |
+---------+------------+------------+----------+ +---------+------------+------------+----------+
| 1 | Video | 0s | 119s | | 1 | Video | 0s | 119s |
| 2 | Video | 20s | 119s | | 2 | Video | 20s | 119s |
| 3 | Video | 40s | 119s | | 3 | Video | 40s | 119s |
| 4 | Audio | 0s | 119s | | 4 | Audio | 0s | 119s |
| 5 | Audio | 20s | 119s | | 5 | Audio | 20s | 119s |
| 6 | Audio | 40s | 119s | | 6 | Audio | 40s | 119s |
+---------+------------+------------+----------+ +---------+------------+------------+----------+
Table 5: Media Timeline for Video and Audio media sources Table 5: Media Timeline for Video and Audio media sources
5.5. Round Trip Time Fairness 5.5. Round Trip Time Fairness
In this test case, multiple RMCAT media flows share the bottleneck In this test case, multiple media flows share the bottleneck link,
link, but the end-to-end path latency for each RMCAT flow is but the end-to-end path latency for each flow is different. For the
different. For the sake of simplicity it is assumed that there are sake of simplicity it is assumed that there are no other competing
no other non-RMCAT competing traffic sources in the bottleneck link traffic sources in the bottleneck link and that there is sufficient
and that there is sufficient capacity to accommodate all the flows. capacity to accommodate all the flows. While this appears to be a
While this appears to be a variant of test case 5.2, it focuses on variant of test case 5.2, it focuses on the capacity sharing aspect
the capacity sharing aspect of the candidate algorithm under of the candidate algorithm under different RTTs.
different RTTs.
It is expected that the competing flows will converge to bit rates to It is expected that the competing flows will converge to bit rates to
accommodate all the flows with minimum possible latency and loss. accommodate all the flows with minimum possible latency and loss.
Specifically, the test introduces five media flows at the same time Specifically, the test introduces five media flows at the same time
instance. instance.
To evaluate the performance of the candidate algorithms it is Evaluation metrics : as described in Section 4.1.
expected to log enough information to visualize the metrics described
in Section 4.1 at a fine enough time granularity:
Testbed Topology: Five (5) media sources S1,S2,..,S5 are connected to Testbed Topology: Five (5) media sources S1,S2,..,S5 are connected to
their corresponding media sinks R1,R2,..,R5. The media traffic is their corresponding media sinks R1,R2,..,R5. The media traffic is
transported over the forward path and corresponding feedback/control transported over the forward path and corresponding feedback/control
traffic is transported over the backward path. The topology is the traffic is transported over the backward path. The topology is the
same as in Section 5.4. The end-to-end path delays are: 10ms for same as in Section 5.4. The end-to-end path delays are: 10ms for
S1-R1, 25ms for S2-R2, 50ms for S3-R3, 100ms for S4-R4, and 150ms S1-R1, 25ms for S2-R2, 50ms for S3-R3, 100ms for S4-R4, and 150ms
S5-R5, respectively. S5-R5, respectively.
Testbed attributes: Testbed attributes:
skipping to change at page 19, line 50 skipping to change at page 20, line 11
100ms, 150ms. 100ms, 150ms.
o Application-related: o Application-related:
* Media Source: * Media Source:
+ Media type: Video + Media type: Video
- Media direction: forward - Media direction: forward
- Number of media sources: Five (5) - Number of media sources: five (5)
- Media timeline: New media flows are added sequentially,
- Media timeline: new media flows are added sequentially,
at short time intervals. See test specific setup below. at short time intervals. See test specific setup below.
+ Media type: Audio + Media type: Audio
- Media direction: forward. - Media direction: forward.
- Number of media sources: Five (5) - Number of media sources: five (5)
- Media timeline: New media flows are added sequentially, - Media timeline: new media flows are added sequentially,
at short time intervals. See test specific setup below. at short time intervals. See test specific setup below.
* Competing traffic: * Competing traffic:
+ Number of sources : Zero (0) + Number of sources : zero (0)
o Test Specific Information: Table 6 defines the media timeline for o Test Specific Information: Table 6 defines the media timeline for
both media type. both media type.
+---------+------------+------------+----------+ +---------+------------+------------+----------+
| Flow IF | Media type | Start time | End time | | Flow IF | Media type | Start time | End time |
+---------+------------+------------+----------+ +---------+------------+------------+----------+
| 1 | Video | 0s | 299s | | 1 | Video | 0s | 299s |
| 2 | Video | 10s | 299s | | 2 | Video | 10s | 299s |
| 3 | Video | 20s | 299s | | 3 | Video | 20s | 299s |
skipping to change at page 20, line 40 skipping to change at page 21, line 5
| 5 | Video | 40s | 299s | | 5 | Video | 40s | 299s |
| 6 | Audio | 0 | 299s | | 6 | Audio | 0 | 299s |
| 7 | Audio | 10s | 299s | | 7 | Audio | 10s | 299s |
| 8 | Audio | 20s | 299s | | 8 | Audio | 20s | 299s |
| 9 | Audio | 30s | 299s | | 9 | Audio | 30s | 299s |
| 10 | Audio | 40s | 299s | | 10 | Audio | 40s | 299s |
+---------+------------+------------+----------+ +---------+------------+------------+----------+
Table 6: Media Timeline for Video and Audio media sources Table 6: Media Timeline for Video and Audio media sources
5.6. RMCAT Flow competing with a long TCP Flow 5.6. Media Flow Competing with a Long TCP Flow
In this test case, one or more RMCAT media flows share the bottleneck In this test case, one or more media flows share the bottleneck link
link with at least one long lived TCP flows. Long lived TCP flows with at least one long lived TCP flow. Long lived TCP flows download
download data throughout the session and are expected to have data throughout the session and are expected to have infinite amount
infinite amount of data to send and receive. This is a scenario of data to send and receive. This is a scenario where a multimedia
where a multimedia application co-exists with a large file download. application co-exists with a large file download. The test case
The test case measures the adaptivity of the candidate algorithm to measures the adaptivity of the candidate algorithm to competing
competing traffic. It addresses the requirement 3 in traffic. It addresses the requirement 3 in
[I-D.ietf-rmcat-cc-requirements]. [I-D.ietf-rmcat-cc-requirements].
Expected behavior: depending on the convergence observed in test case Expected behavior: depending on the convergence observed in test case
5.1 and 5.2, the candidate algorithm may be able to avoid congestion 5.1 and 5.2, the candidate algorithm may be able to avoid congestion
collapse. In the worst case, the media stream will fall to the collapse. In the worst case, the media stream will fall to the
minimum media bit rate. minimum media bit rate.
To evaluate the performance of the candidate algorithms it is Evaluation metrics : following metrics in addition to as described in
expected to log enough information to visualize the following metrics Section 4.1.
in addition to the metrics described in Section 4.1 at a fine enough
time granularity:
1. Flow level: 1. Flow level:
A. TCP throughput. A. TCP throughput.
Testbed topology: One (1) media source S1 is connected to B. Loss for the TCP flow
Testbed topology: One (1) media source S1 is connected to the
corresponding media sink, R1. In addition, there is a long-live TCP corresponding media sink, R1. In addition, there is a long-live TCP
flow sharing the same bottleneck link. The media traffic is flow sharing the same bottleneck link. The media traffic is
transported over the forward path and corresponding feedback/control transported over the forward path and corresponding feedback/control
traffic is transported over the backward path. The TCP traffic goes traffic is transported over the backward path. The TCP traffic goes
over the forward path from, S_tcp with acknowledgement packets over the forward path from, S_tcp with acknowledgment packets go over
flowing along the backward path from, R_tcp. the backward path from, R_tcp.
+--+ +--+ +--+ +--+
|S1|===== \ Forward --> / =======|R1| |S1|===== \ Forward --> / =======|R1|
+--+ \\ // +--+ +--+ \\ // +--+
\\ // \\ //
+-----+ +-----+ +-----+ +-----+
| A |---------------------------->| B | | A |---------------------------->| B |
| |<----------------------------| | | |<----------------------------| |
+-----+ +-----+ +-----+ +-----+
// \\ // \\
// <-- Backward \\ // <-- Backward \\
+-----+ // \\ +-----+ +-----+ // \\ +-----+
|S_tcp|=== / \ ===|R_tcp| |S_tcp|=== / \ ===|R_tcp|
+-----+ +-----+ +-----+ +-----+
Figure 6: Testbed Topology for TCP vs RMCAT Flows Figure 6: Testbed Topology for TCP vs congestion controlled media
Flows
Testbed attributes: Testbed attributes:
o Test duration: 120s o Test duration: 120s
o Path characteristics: o Path characteristics:
* Reference bottleneck capacity: 2Mbps * Reference bottleneck capacity: 2Mbps
* Path capacity ratio: 1.0 * Path capacity ratio: 1.0
skipping to change at page 22, line 14 skipping to change at page 22, line 42
* Bottleneck queue size: [300ms, 1000ms] * Bottleneck queue size: [300ms, 1000ms]
o Application-related: o Application-related:
* Media Source: * Media Source:
+ Media type: Video + Media type: Video
- Media direction: forward - Media direction: forward
- Number of media sources: One (1) - Number of media sources: one (1)
- Media timeline: - Media timeline:
o Start time: 5s. o Start time: 5s.
o End time: 119s. o End time: 119s.
+ Media type: Audio + Media type: Audio
- Media direction: forward - Media direction: forward
- Number of media sources: one (1)
- Number of media sources: One (1)
- Media timeline: - Media timeline:
o Start time: 5s. o Start time: 5s.
o End time: 119s. o End time: 119s.
* Additionally, implementers are encouraged to run the experiment * Additionally, implementers are encouraged to run the experiment
with multiple media sources. with multiple media sources.
* Competing traffic: * Competing traffic:
+ Number and Types of sources : one (1), long-lived TCP + Number and Types of sources : one (1) and long-lived TCP
+ Traffic direction : forward + Traffic direction : forward
+ Congestion control: Default TCP congestion control. + Congestion control: default TCP congestion control[RFC5681].
+ Traffic timeline: + Traffic timeline:
- Start time: 0s. - Start time: 0s.
- End time: 119s. - End time: 119s.
o Test Specific Information: None o Test Specific Information: none
5.7. RMCAT Flow competing with short TCP Flows 5.7. Media Flow Competing with Short TCP Flows
In this test case, one or more RMCAT media flow shares the bottleneck In this test case, one or more congestion controlled media flow
link with multiple short-lived TCP flows. Short-lived TCP flows shares the bottleneck link with multiple short-lived TCP flows.
resemble the on/off pattern observed in the web traffic, wherein Short-lived TCP flows resemble the on/off pattern observed in the web
clients (browsers) connect to a server and download a resource traffic, wherein clients (browsers) connect to a server and download
(typically a web page, few images, text files, etc.) using several a resource (typically a web page, few images, text files, etc.) using
TCP connections (up to 4). This scenario shows the performance of several TCP connections (up to 4). This scenario shows the
the multimedia application when several browser windows are active. performance of a multimedia application when several browser windows
The test case measures the adaptivity of the candidate algorithm to are active. The test case measures the adaptivity of the candidate
competing web traffic, it addresses the requirements 1.E in algorithm to competing web traffic, it addresses the requirements 1.E
[I-D.ietf-rmcat-cc-requirements]. in [I-D.ietf-rmcat-cc-requirements].
Depending on the number of short TCP flows, the cross-traffic either Depending on the number of short TCP flows, the cross-traffic either
appears as a short burst flow or resembles a long TCP flow. The appears as a short burst flow or resembles a long TCP flow. The
intention of this test is to observe the impact of short-term burst intention of this test is to observe the impact of short-term burst
on the behaviour of the candidate algorithm. on the behavior of the candidate algorithm.
To evaluate the performance of the candidate algorithms it is Evaluation metrics : following metrics in addition to as described in
expected to log enough information to visualize the following metrics Section 4.1.
in addition to the metrics described in Section 4.1 at a fine enough
time granularity:
1. Flow level: 1. Flow level:
A. Variation in the sending rate of the TCP flow. A. Variation in the sending rate of the TCP flow.
B. TCP throughput. B. TCP throughput.
Testbed topology: The topology described here is same as the one Testbed topology: The topology described here is same as the one
described in Figure 6. described in Figure 6.
skipping to change at page 23, line 49 skipping to change at page 24, line 26
o Test duration: 300s o Test duration: 300s
o Path characteristics: o Path characteristics:
* Reference bottleneck capacity: 2.0Mbps * Reference bottleneck capacity: 2.0Mbps
* Path capacity ratio: 1.0 * Path capacity ratio: 1.0
o Application-related: o Application-related:
* Media Source: * Media source:
+ Media type: Video + Media type: Video
- Media direction: forward - Media direction: forward
- Number of media sources: two (2) - Number of media sources: two (2)
- Media timeline: - Media timeline:
o Start time: 5s. o Start time: 5s.
o End time: 299s. o End time: 299s.
skipping to change at page 24, line 28 skipping to change at page 25, line 5
- Number of media sources: two (2) - Number of media sources: two (2)
- Media timeline: - Media timeline:
o Start time: 5s. o Start time: 5s.
o End time: 299s. o End time: 299s.
* Competing traffic: * Competing traffic:
+ Number and Types of sources : Ten (10), short-lived TCP + Number and Types of sources : ten (10), short-lived TCP
flows. flows.
+ Traffic direction : forward + Traffic direction : forward
+ Congestion algorithm: Default TCP Congestion control. + Congestion algorithm: default TCP Congestion control
[RFC5681].
+ Traffic timeline: Each short TCP flow is modeled as a + Traffic timeline: each short TCP flow is modeled as a
sequence of file downloads interleaved with idle periods. sequence of file downloads interleaved with idle periods.
See test specific setup. Not all short TCPs start at the See test specific setup. Not all short TCP flows start at
same time, 2 start in the ON state while 8 start in an OFF the same time, 2 of them start in the ON state while rest on
stats. The model for the idle times for the OFF state is the 8 flows start in an OFF stats. The model for the idle
discussed in the Short-TCP model. times for the OFF state is discussed in
[I-D.ietf-rmcat-eval-criteria].
o Test Specific Information: o Test Specific Information:
* Short-TCP traffic model: * Short-TCP traffic model:
+ File sizes: uniform distribution between 100KB to 1MB + File sizes: uniform distribution between 100KB to 1MB
+ Idle period: the duration of the OFF state is derived from + Idle period: the duration of the OFF state is derived from
an exponential distribution with the mean value of 10 an exponential distribution with the mean value of 10
seconds. seconds.
5.8. Media Pause and Resume 5.8. Media Pause and Resume
In this test case, more than one real-time interactive media flows In this test case, more than one real-time interactive media flows
share the link bandwidth and all flows reach to a steady state by share the link bandwidth and all flows reach to a steady state by
utilizing the link capacity in an optimum way. At these stage one of utilizing the link capacity in an optimum way. At this stage one of
the media flow is paused for a moment. This event will result in the media flows is paused for a moment. This event will result in
more available bandwidth for the rest of the flows and as they are on more available bandwidth for the rest of the flows as they are on a
a shared link. When the paused media flow will resume it would no shared link. When the paused media flow resumes it would no longer
longer have the same bandwidth share on the link. It has to make have the same bandwidth share on the link. It has to make it's way
it's way through the other existing flows in the link to achieve a through the other existing flows in the link to achieve a fair share
fair share of the link capacity. This test case is important of the link capacity. This test case is important specially for
specially for real-time interactive media which consists of more than real-time interactive media which consists of more than one media
one media flows and can pause/resume media flow at any point of time flows and can pause/resume media flows at any point of time during
during the session. This test case directly addresses the the session. This test case directly addresses the requirement
requirement number 5 in [I-D.ietf-rmcat-cc-requirements]. One can number 5 in [I-D.ietf-rmcat-cc-requirements]. One can think it as a
think it as a variation of test case defined in Section 5.4. variation of test case defined in Section 5.4. However, it is
However, it is different as the candidate algorithms can use different as the candidate algorithms can use different strategies to
different strategies to increase its efficiency, for example the increase its efficiency, for example in terms of fairness,
fairness, convergence time, reduce oscillation etc, by capitalizing convergence time, reduce oscillation etc, by capitalizing the fact
the fact that they have previous information of the link. that they have previous information of the link.
To evaluate the performance of the candidate algorithms it is Evaluation metrics : following metrics in addition to as described in
expected to log enough information to visualize the following metrics Section 4.1.
in addition to the metrics described in Section 4.1 at a fine enough
time granularity:
1. Flow level: 1. Flow level:
A. Variation in sending bit rate and goodput. Mainly observing A. Variation in sending bit rate and goodput. Mainly observing
the frequency and magnitude of oscillations. the frequency and magnitude of oscillations.
Testbed Topology: Same as test case defined in Section 5.4 Testbed Topology: Same as test case defined in Section 5.4
Testbed attributes: The general description of the testbed parameters Testbed attributes: The general description of the testbed parameters
are same as Section 5.4 with changes in the test specific setup as are same as Section 5.4 with changes in the test specific setup as
below- below-
o Other test specific setup: o Other test specific setup:
* Media flow timeline: * Media flow timeline:
+ Flow ID: One (1) + Flow ID: one (1)
+ Start time: 0s + Start time: 0s
+ Flow duration: 119s + Flow duration: 119s
+ Pause time: not required + Pause time: not required
+ Resume time: not required + Resume time: not required
* Media flow timeline: * Media flow timeline:
+ Flow ID: Two (2) + Flow ID: two (2)
+ Start time: 0s + Start time: 0s
+ Flow duration: 119s + Flow duration: 119s
+ Pause time: at 40s + Pause time: at 40s
+ Resume time: at 60s + Resume time: at 60s
* Media flow timeline: * Media flow timeline:
+ Flow ID: Three (3) + Flow ID: three (3)
+ Start time: 0s + Start time: 0s
+ Flow duration:119s + Flow duration:119s
+ Pause time: not required + Pause time: not required
+ Resume time: not required + Resume time: not required
6. Other potential test cases 6. Other potential test cases
It has been noticed that there are other interesting test cases It has been noticed that there are other interesting test cases
besides the basis test cases listed above. In many aspects, these besides the basic test cases listed above. In many aspects, these
additional test cases can help to further evaluate the candidate additional test cases can help further evaluation of the candidate
algorithm. They are listed as below. algorithm. They are listed as below.
6.1. Explicit Congestion Notification Usage 6.1. Media Flows with Priority
In this test case media flows will have different priority levels.
This will be an extension of Section 5.4 where the same test will be
run with different priority levels imposed on each of the media
flows. For example, the first flow (S1) is assigned a priority of 2
whereas the remaining two flows (S2 and S3) are assigned a priority
of 1. The candidate algorithm MUST reflect the relative priorities
assigned to each media flow. In the previous example, the first flow
(S1) MUST arrive at a steady-state rate approximately twice of that
of the other two flows (S2 and S3).
The candidate algorithm can use a coupled congestion control
mechanism for the bandwidth distribution according to the respective
media flow priority.
6.2. Explicit Congestion Notification Usage
This test case requires to run all the basic test cases with the This test case requires to run all the basic test cases with the
availability of Explicit Congestion Notification (ECN) [RFC6679] availability of Explicit Congestion Notification (ECN) [RFC6679]
feature enabled. The goal of this test is to exhibit that the feature enabled. The goal of this test is to exhibit that the
candidate algorithms does not fail when ECN signals are available. candidate algorithms do not fail when ECN signals are available.
With ECN signals enabled the algorithms are expected to perform With ECN signals enabled the algorithms are expected to perform
better than their delay based variants. better than their delay based variants.
6.2. Multiple Bottlenecks 6.3. Multiple Bottlenecks
In this test case one RMCAT flow, S1->R2 traverse a path with In this test case one congestion controlled media flow, S1->R2,
multiple bottlenecks. As illustrated in Figure 7, the first flow traverses a path with multiple bottlenecks. As illustrated in
(S1->R1) competes with the second RMCAT flow (S2->R2) over the link Figure 7, the first flow (S1->R1) competes with the second congestion
between A and B which is close to the sender side; again, that flow controlled media flow (S2->R2) over the link between A and B which is
(S1->R1) competes with the third RMCAT flow (S3->R3) over the link close to the sender side; again, that flow (S1->R1) competes with the
between C and D which is close to the receiver side. The goal of third congestion controlled media flow (S3->R3) over the link between
this test is to ensure that the candidate algorithms work properly in C and D which is close to the receiver side. The goal of this test
the presence of multiple bottleneck links on the end to end path. is to ensure that the candidate algorithms work properly in the
presence of multiple bottleneck links on the end to end path.
Expected behavior: the candidate algorithm is expected to achieve Expected behavior: the candidate algorithm is expected to achieve
full utilization at both bottleneck links without starving any of the full utilization at both bottleneck links without starving any of the
three RMCAT flows. three congestion controlled media flows.
Forward ----> Forward ---->
+---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+
|S2 | |R2 | |S3 | |R3 | |S2 | |R2 | |S3 | |R3 |
+---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+
| | | | | | | |
| | | | | | | |
+---+ +-----+ +-----+ +-----+ +-----+ +---+ +---+ +-----+ +-----+ +-----+ +-----+ +---+
|S1 |=======| A |------>| B |----->| C |---->| D |=======|R1 | |S1 |=======| A |------>| B |----->| C |---->| D |=======|R1 |
skipping to change at page 27, line 39 skipping to change at page 28, line 35
Figure 7: Testbed Topology for Multiple Bottlenecks Figure 7: Testbed Topology for Multiple Bottlenecks
Testbed topology: Three media sources S1, S2, and S3 are connected to Testbed topology: Three media sources S1, S2, and S3 are connected to
respective destinations R1, R2, and R3. For all three flows the respective destinations R1, R2, and R3. For all three flows the
media traffic is transported over the forward path and corresponding media traffic is transported over the forward path and corresponding
feedback/control traffic is transported over the backward path. feedback/control traffic is transported over the backward path.
Testbed attributes: Testbed attributes:
o Test duration: 120s o Test duration: 300s
o Path characteristics: o Path characteristics:
* Reference bottleneck capacity between A and B = 2Mbps. * Reference bottleneck capacity: 2Mbps.
* Path capacity ratio between A and B: 1.0 * Path capacity ratio between A and B: 1.0
* Path capacity ratio between B and C: 4.0. * Path capacity ratio between B and C: 4.0.
* Path capacity ratio between C and D: 0.75. * Path capacity ratio between C and D: 0.75.
* One-Way propagation delay: * One-Way propagation delay:
1. Between S1 and R1: 100ms 1. Between S1 and R1: 100ms
2. Between S2 and R2: 40ms 2. Between S2 and R2: 40ms
skipping to change at page 28, line 30 skipping to change at page 29, line 27
+ Media type: Video + Media type: Video
- Media direction: Forward - Media direction: Forward
- Number of media sources: Three (3) - Number of media sources: Three (3)
- Media timeline: - Media timeline:
o Start time: 0s. o Start time: 0s.
o End time: 119s. o End time: 299s.
+ Media type: Audio + Media type: Audio
- Media direction: Forward - Media direction: Forward
- Number of media sources: Three (3) - Number of media sources: Three (3)
- Media timeline: - Media timeline:
o Start time: 0s. o Start time: 0s.
o End time: 119s. o End time: 299s.
* Competing traffic: * Competing traffic:
+ Number of sources : Zero (0) + Number of sources : Zero (0)
7. Wireless Access Links 7. Wireless Access Links
Additional wireless network (both cellular network and WiFi network) Additional wireless network (both cellular network and WiFi network)
specific test cases are define in [I-D.ietf-rmcat-wireless-tests] specific test cases are defined in [I-D.ietf-rmcat-wireless-tests].
8. Security Considerations 8. Security Considerations
Security issues have not been discussed in this memo. Security issues have not been discussed in this memo.
9. IANA Considerations 9. IANA Considerations
There are no IANA impacts in this memo. There are no IANA impacts in this memo.
10. Acknowledgements 10. Acknowledgements
skipping to change at page 29, line 46 skipping to change at page 30, line 41
Jacobson, "RTP: A Transport Protocol for Real-Time Jacobson, "RTP: A Transport Protocol for Real-Time
Applications", STD 64, RFC 3550, DOI 10.17487/RFC3550, Applications", STD 64, RFC 3550, DOI 10.17487/RFC3550,
July 2003, <http://www.rfc-editor.org/info/rfc3550>. July 2003, <http://www.rfc-editor.org/info/rfc3550>.
[RFC3551] Schulzrinne, H. and S. Casner, "RTP Profile for Audio and [RFC3551] Schulzrinne, H. and S. Casner, "RTP Profile for Audio and
Video Conferences with Minimal Control", STD 65, RFC 3551, Video Conferences with Minimal Control", STD 65, RFC 3551,
DOI 10.17487/RFC3551, July 2003, DOI 10.17487/RFC3551, July 2003,
<http://www.rfc-editor.org/info/rfc3551>. <http://www.rfc-editor.org/info/rfc3551>.
[RFC3611] Friedman, T., Ed., Caceres, R., Ed., and A. Clark, Ed., [RFC3611] Friedman, T., Ed., Caceres, R., Ed., and A. Clark, Ed.,
"RTP Control Protocol Extended Reports (RTCP XR)", RFC "RTP Control Protocol Extended Reports (RTCP XR)",
3611, DOI 10.17487/RFC3611, November 2003, RFC 3611, DOI 10.17487/RFC3611, November 2003,
<http://www.rfc-editor.org/info/rfc3611>. <http://www.rfc-editor.org/info/rfc3611>.
[RFC4585] Ott, J., Wenger, S., Sato, N., Burmeister, C., and J. Rey, [RFC4585] Ott, J., Wenger, S., Sato, N., Burmeister, C., and J. Rey,
"Extended RTP Profile for Real-time Transport Control "Extended RTP Profile for Real-time Transport Control
Protocol (RTCP)-Based Feedback (RTP/AVPF)", RFC 4585, DOI Protocol (RTCP)-Based Feedback (RTP/AVPF)", RFC 4585,
10.17487/RFC4585, July 2006, DOI 10.17487/RFC4585, July 2006,
<http://www.rfc-editor.org/info/rfc4585>. <http://www.rfc-editor.org/info/rfc4585>.
[RFC5506] Johansson, I. and M. Westerlund, "Support for Reduced-Size [RFC5506] Johansson, I. and M. Westerlund, "Support for Reduced-Size
Real-Time Transport Control Protocol (RTCP): Opportunities Real-Time Transport Control Protocol (RTCP): Opportunities
and Consequences", RFC 5506, DOI 10.17487/RFC5506, April and Consequences", RFC 5506, DOI 10.17487/RFC5506, April
2009, <http://www.rfc-editor.org/info/rfc5506>. 2009, <http://www.rfc-editor.org/info/rfc5506>.
[I-D.ietf-rmcat-eval-criteria] [I-D.ietf-rmcat-eval-criteria]
Singh, V. and J. Ott, "Evaluating Congestion Control for Singh, V. and J. Ott, "Evaluating Congestion Control for
Interactive Real-time Media", draft-ietf-rmcat-eval- Interactive Real-time Media", draft-ietf-rmcat-eval-
criteria-03 (work in progress), March 2015. criteria-04 (work in progress), October 2015.
[I-D.ietf-rmcat-wireless-tests] [I-D.ietf-rmcat-wireless-tests]
Sarker, Z. and I. Johansson, "Evaluation Test Cases for Sarker, Z., Johansson, I., Zhu, X., Fu, J., Tan, W., and
Interactive Real-Time Media over Wireless Networks", M. Ramalho, "Evaluation Test Cases for Interactive Real-
draft-ietf-rmcat-wireless-tests-00 (work in progress), Time Media over Wireless Networks", draft-ietf-rmcat-
June 2015. wireless-tests-01 (work in progress), November 2015.
[I-D.ietf-rmcat-video-traffic-model]
Zhu, X., Cruz, S., and Z. Sarker, "Modeling Video Traffic
Sources for RMCAT Evaluations", draft-ietf-rmcat-video-
traffic-model-00 (work in progress), January 2016.
11.2. Informative References 11.2. Informative References
[RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion
Control", RFC 5681, DOI 10.17487/RFC5681, September 2009,
<http://www.rfc-editor.org/info/rfc5681>.
[I-D.ietf-rmcat-cc-requirements] [I-D.ietf-rmcat-cc-requirements]
Jesup, R. and Z. Sarker, "Congestion Control Requirements Jesup, R. and Z. Sarker, "Congestion Control Requirements
for Interactive Real-Time Media", draft-ietf-rmcat-cc- for Interactive Real-Time Media", draft-ietf-rmcat-cc-
requirements-09 (work in progress), December 2014. requirements-09 (work in progress), December 2014.
[xiph-seq] [xiph-seq]
Xiph.org, , "Video Test Media", Xiph.org, , "Video Test Media",
http://media.xiph.org/video/derf/ . http://media.xiph.org/video/derf/ .
[HEVC-seq] [HEVC-seq]
skipping to change at page 30, line 43 skipping to change at page 32, line 4
[xiph-seq] [xiph-seq]
Xiph.org, , "Video Test Media", Xiph.org, , "Video Test Media",
http://media.xiph.org/video/derf/ . http://media.xiph.org/video/derf/ .
[HEVC-seq] [HEVC-seq]
HEVC, , "Test Sequences", HEVC, , "Test Sequences",
http://www.netlab.tkk.fi/~varun/test_sequences/ . http://www.netlab.tkk.fi/~varun/test_sequences/ .
Authors' Addresses Authors' Addresses
Zaheduzzaman Sarker Zaheduzzaman Sarker
Ericsson AB Ericsson AB
Luleae, SE 977 53 Luleae, SE 977 53
Sweden Sweden
Phone: +46 10 717 37 43 Phone: +46 10 717 37 43
Email: zaheduzzaman.sarker@ericsson.com Email: zaheduzzaman.sarker@ericsson.com
Varun Singh Varun Singh
Aalto University Nemu Dialogue Systems Oy
School of Electrical Engineering Runeberginkatu 4c A 4
Otakaari 5 A Helsinki 00100
Espoo, FIN 02150
Finland Finland
Email: varun@comnet.tkk.fi Email: varun.singh@iki.fi
URI: http://www.netlab.tkk.fi/~varun/ URI: http://www.callstats.io/
Xiaoqing Zhu Xiaoqing Zhu
Cisco Systems Cisco Systems
510 McCarthy Blvd 12515 Research Blvd
Milpitas, CA 95134 Austing, TX 78759
USA USA
Email: xiaoqzhu@cisco.com Email: xiaoqzhu@cisco.com
Michael A. Ramalho Michael A. Ramalho
Cisco Systems, Inc. Cisco Systems, Inc.
6310 Watercrest Way Unit 203 6310 Watercrest Way Unit 203
Lakewood Ranch, FL 34202-5211 Lakewood Ranch, FL 34202-5211
USA USA
 End of changes. 148 change blocks. 
347 lines changed or deleted 370 lines changed or added

This html diff was produced by rfcdiff 1.44. The latest version is available from http://tools.ietf.org/tools/rfcdiff/