--- 1/draft-ietf-rmcat-eval-test-01.txt 2015-09-08 14:14:58.858842563 -0700 +++ 2/draft-ietf-rmcat-eval-test-02.txt 2015-09-08 14:14:58.918844037 -0700 @@ -1,22 +1,22 @@ Network Working Group Z. Sarker Internet-Draft Ericsson AB Intended status: Informational V. Singh -Expires: September 11, 2015 Aalto University +Expires: March 11, 2016 Aalto University X. Zhu M. Ramalho Cisco Systems - March 10, 2015 + September 8, 2015 Test Cases for Evaluating RMCAT Proposals - draft-ietf-rmcat-eval-test-01 + draft-ietf-rmcat-eval-test-02 Abstract The Real-time Transport Protocol (RTP) is used to transmit media in multimedia telephony applications, these applications are typically required to implement congestion control. The RMCAT working group is currently working on candidate algorithms for such interactive real- time multimedia applications. This document describes the test cases to be used in the performance evaluation of those candidate algorithms. @@ -29,21 +29,21 @@ Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at http://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." - This Internet-Draft will expire on September 11, 2015. + This Internet-Draft will expire on March 11, 2016. Copyright Notice Copyright (c) 2015 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents @@ -57,33 +57,31 @@ 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3 3. Structure of Test cases . . . . . . . . . . . . . . . . . . . 3 4. Recommended Evaluation Settings . . . . . . . . . . . . . . . 7 4.1. Evaluation metircs . . . . . . . . . . . . . . . . . . . 7 4.2. Path characteristics . . . . . . . . . . . . . . . . . . 8 4.3. Media source . . . . . . . . . . . . . . . . . . . . . . 9 5. Basic Test Cases . . . . . . . . . . . . . . . . . . . . . . 10 5.1. Variable Available Capacity with Single RMCAT flow . . . 10 - 5.2. Variable Available Capacity with Multiple RMCAT flows . . 12 - 5.3. Congested Feedback Link with Bi-directional RMCAT flows . 13 + 5.2. Variable Available Capacity with Multiple RMCAT flows . . 13 + 5.3. Congested Feedback Link with Bi-directional RMCAT flows . 14 5.4. Competing Flows with Same RMCAT Algorithm . . . . . . . . 16 - 5.5. Round Trip Time Fairness . . . . . . . . . . . . . . . . 18 + 5.5. Round Trip Time Fairness . . . . . . . . . . . . . . . . 19 5.6. RMCAT Flow competing with a long TCP Flow . . . . . . . . 20 - 5.7. RMCAT Flow competing with short TCP Flows . . . . . . . . 22 - 5.8. Media Pause and Resume . . . . . . . . . . . . . . . . . 24 + 5.7. RMCAT Flow competing with short TCP Flows . . . . . . . . 23 + 5.8. Media Pause and Resume . . . . . . . . . . . . . . . . . 25 6. Other potential test cases . . . . . . . . . . . . . . . . . 26 6.1. Explicit Congestion Notification Usage . . . . . . . . . 26 6.2. Multiple Bottlenecks . . . . . . . . . . . . . . . . . . 26 - 7. Wireless Access Links . . . . . . . . . . . . . . . . . . . . 28 - 7.1. Cellular Network Specific Test Cases . . . . . . . . . . 28 - 7.2. Wi-Fi Network Specific Test Cases . . . . . . . . . . . . 28 + 7. Wireless Access Links . . . . . . . . . . . . . . . . . . . . 29 8. Security Considerations . . . . . . . . . . . . . . . . . . . 29 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 29 10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 29 11. References . . . . . . . . . . . . . . . . . . . . . . . . . 29 11.1. Normative References . . . . . . . . . . . . . . . . . . 29 11.2. Informative References . . . . . . . . . . . . . . . . . 30 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 30 1. Introduction @@ -97,22 +95,21 @@ modeling of different path characteristics. It is the intention of this work to capture the consensus of the RMCAT working group participants regarding the test cases upon which the performance of the candidate RMCAT proposals should be evaluated. 2. Terminology The terminology defined in RTP [RFC3550], RTP Profile for Audio and Video Conferences with Minimal Control [RFC3551], RTCP Extended Report (XR) [RFC3611], Extended RTP Profile for RTCP-based Feedback - (RTP/AVPF) [RFC4585], Support for Reduced-Size RTCP [RFC5506], and - RTP Circuit Breaker algorithm [I-D.ietf-avtcore-rtp-circuit-breakers] + (RTP/AVPF) [RFC4585], and Support for Reduced-Size RTCP [RFC5506] apply. 3. Structure of Test cases All test cases in this document follow a basic structure allowing implementers to describe a new test scenario without repeatedly explaining common attributes. The structure includes a general description section that describes the test case and its motivation. Additionally the test case defines a set of attributes that characterize the testbed, i.e., the network path between @@ -198,20 +195,26 @@ applicable to all the Sources "S" sending traffic on that path. If only one attribute is specified, it is used for both path directions, however, unless specified the reverse path has no capacity restrictions and no path loss. + Path direction: forward or backward. + Bottleneck-link capacity: defines minimum capacity of the end-to-end path + + Reference bottleneck capacity: defines a reference value for + the bottleneck capacity for test cases with time-varying + bottleneck capacities. All bottleneck capacities will be + specified as a ratio with respect to the reference capacity + value. + + One-way propagation delay: describes the end-to-end latency along the path when network queues are empty, i.e., the time it takes for a packet to go from the sender to the receiver without encountering any queuing delay. + Maximum end-to-end jitter: defines the maximum jitter that can be observed along the path. + Bottleneck queue type: for example, Droptail, FQ-CoDel, or PIE. @@ -228,21 +231,21 @@ [I-D.ietf-rmcat-eval-criteria]. * Application-related: defines the traffic source behaviour for implementing the test case + Media traffic Source: defines the characteristics of the media sources. When using more than one media source, the different attributes are enumerated separately for each different media source. - - Media type: Video/Voice/Application/Text + - Media type: Video/Voice - Media flow direction: forward, backward or both. - Number of media sources: defines the total number of media sources - Media codec: Constant Bit Rate (CBR) or Variable Bit Rate (VBR) - Media source behaviour: describes the media encoder @@ -358,39 +361,51 @@ + median, maximum, minimum 4.2. Path characteristics Each path between a sender and receiver as described in Figure 1 have the following characteristics unless otherwise specified in the test case. o Path direction: forward and backward. - o Bottleneck-link capacity: 4Mbps. + o Reference bottleneck capacity: 1Mbps. - o One-Way propagation delay: 50ms. It is encouraged to test with - additional propagation delays mentioned in + o One-Way propagation delay: 50ms. Implementers are encouraged to + run the experiment with additional propagation delays mentioned in [I-D.ietf-rmcat-eval-criteria] o Maximum end-to-end jitter: 30ms. Jitter models are described in [I-D.ietf-rmcat-eval-criteria] - o Bottleneck queue type: Drop tail. It is encouraged to test with - other AQM schemes, such as FQ-CoDel and PIE. + o Bottleneck queue type: Drop tail. Implementers are encouraged to + run the experiment with other AQM schemes, such as FQ-CoDel and + PIE. o Bottleneck queue size: 300ms. o Path loss ratio: 0%. Examples of additional network parameters are discussed in [I-D.ietf-rmcat-eval-criteria]. + For test cases involving time-varying bottleneck capacity, all + capacity values are specified as a ratio with respect to a reference + capacity value, so as to allow flexible scaling of capacity values + along with media source rate range. There exist two different + mechanisms for inducing path capacity variation: a) by explicitly + modifying the value of physical link capacity; or b) by introducing + background non-adaptive UDP traffic with time-varying traffic rate. + Implementers are encouraged run the experiments with both mechanisms + for test cases specified in Section 5.1, Section 5.2, and + Section 5.3. + 4.3. Media source Unless otherwise specified, each test case will include one or more media sources as described below. o Media type: Video * Media codec: VBR * Media source behaviour: @@ -446,21 +461,22 @@ In this test case the bottleneck-link capacity between the two endpoints varies over time. This test is designed to measure the responsiveness of the candidate algorithm. This test tries to address the requirements in [I-D.ietf-rmcat-cc-requirements], which requires the algorithm to adapt the flow(s) and provide lower end-to- end latency when there exists: o an intermediate bottleneck o change in available capacity (e.g., due to interface change, - routing change). + routing change, abrupt arrival/departure of background non- + adaptive traffic). o maximum Media Bit Rate is Greater than Link Capacity. In this case, the application will attempt to ramp up to its maximum bit rate, since the link capacity is limited to a value lower, the congestion control scheme is expected to stabilize the sending bit rate close to the available bottleneck capacity. This situation can occur when the endpoints are connected via thin long networks even though the advertised capacity of the access network may be higher. @@ -535,29 +550,35 @@ + Number of sources : Zero (0) o Test Specific Information: * This test uses the following one way propagation delays of 50 ms and 100 ms. * This test uses bottleneck path capacity variation as listed in Table 1 - +---------------------+----------------+------------+---------------+ - | Variation pattern | Path direction | Start time | Path capacity | - | index | | | | - +---------------------+----------------+------------+---------------+ - | One | Forward | 0s | 1Mbps | - | Two | Forward | 40s | 2.5Mbps | - | Three | Forward | 60s | 600Kbps | - | Four | Forward | 80s | 1Mbps | - +---------------------+----------------+------------+---------------+ + * When using background non-adaptive UDP traffic to induce time- + varying bottleneck for the RMCAT flow, the physical path + capacity is 4Mbps and the UDP traffic source rate changes over + time as (4-x)Mbps, where x is the bottleneck capacity specified + in Table 1 + + +--------------------+--------------+-----------+-------------------+ + | Variation pattern | Path | Start | Path capacity | + | index | direction | time | ratio | + +--------------------+--------------+-----------+-------------------+ + | One | Forward | 0s | 1.0 | + | Two | Forward | 40s | 2.5 | + | Three | Forward | 60s | 0.6 | + | Four | Forward | 80s | 1.0 | + +--------------------+--------------+-----------+-------------------+ Table 1: Path capacity variation pattern for forward direction 5.2. Variable Available Capacity with Multiple RMCAT flows This test case is similar to Section 5.1. However in addition this test will also consider persistent network load due to competing traffic. Expected behavior: the candidate algorithms is expected to detect the @@ -592,32 +613,37 @@ +---+ +---+ Figure 3: Testbed Topology for Variable Available Capacity Testbed attributes: Testbed attributes are similar as described in Section 5.1 except the test specific capacity variation setup. Test Specific Information: This test uses path capacity variation as - listed in Table 2 with a corresponding end time of 125 seconds. + listed in Table 2 with a corresponding end time of 125 seconds. The + reference bottleneck capacity is 2Mbps. When using background non- + adaptive UDP traffic to induce time-varying bottleneck for RMCAT + flows, the physical path capacity is 4Mbps and the UDP traffic source + rate changes over time as (4-x)Mbps, where x is the bottleneck + capacity specified in Table 2. - +---------------------+----------------+------------+---------------+ - | Variation pattern | Path direction | Start time | Path capacity | - | index | | | | - +---------------------+----------------+------------+---------------+ - | One | Forward | 0s | 4Mbps | - | Two | Forward | 25s | 2Mbps | - | Three | Forward | 50s | 3.5Mbps | - | Four | Forward | 75s | 1Mbps | - | Five | Forward | 100s | 2Mbps | - +---------------------+----------------+------------+---------------+ + +--------------------+--------------+-----------+-------------------+ + | Variation pattern | Path | Start | Path capacity | + | index | direction | time | ratio | + +--------------------+--------------+-----------+-------------------+ + | One | Forward | 0s | 2.0 | + | Two | Forward | 25s | 1.0 | + | Three | Forward | 50s | 1.75 | + | Four | Forward | 75s | 0.5 | + | Five | Forward | 100s | 1.0 | + +--------------------+--------------+-----------+-------------------+ Table 2: Path capacity variation pattern for forward direction 5.3. Congested Feedback Link with Bi-directional RMCAT flows RMCAT WG has been chartered to define algorithms for RTP hence it is assumed that RTCP, RTP header extension or such would be used by the congestion control algorithm in the backchannel. Due to asymmetric nature of the link between communicating peers it is possible for a participating peer to not receive such feedback information due to an @@ -660,21 +686,21 @@ +---+ +---+ Figure 4: Testbed Topology for Congested Feedback Link Testbed attributes: o Test duration: 100s o Path characteristics: - * Bottleneck-link capacity: 2Mbps. + * Reference bottleneck capacity: 1Mbps. o Application-related: * Media Source: + Media type: Video - Media direction: forward and backward - Number of media sources: Two (2) @@ -697,42 +723,47 @@ o End time: 99s. * Competing traffic: + Number of sources : Zero (0) o Test Specific Information: This test uses path capacity variations to create congested feedback link. Table 3 lists the variation patterns applied to the forward path and Table 4 lists the - variation patterns applied to the backward path. + variation patterns applied to the backward path. When using + background non-adaptive UDP traffic to induce time-varying + bottleneck for RMCAT flows, the physical path capacity is 4Mbps + for both directions and the UDP traffic source rate changes over + time as (4-x)Mbps in each direction, where x is the bottleneck + capacity specified in Table 4. - +---------------------+----------------+------------+---------------+ - | Variation pattern | Path direction | Start time | Path capacity | - | index | | | | - +---------------------+----------------+------------+---------------+ - | One | Forward | 0s | 2Mbps | - | Two | Forward | 20s | 1Mbps | - | Three | Forward | 40s | 500Kbps | - | Four | Forward | 60s | 2Mbps | - +---------------------+----------------+------------+---------------+ + +--------------------+--------------+-----------+-------------------+ + | Variation pattern | Path | Start | Path capacity | + | index | direction | time | ratio | + +--------------------+--------------+-----------+-------------------+ + | One | Forward | 0s | 2.0 | + | Two | Forward | 20s | 1.0 | + | Three | Forward | 40s | 0.5 | + | Four | Forward | 60s | 2.0 | + +--------------------+--------------+-----------+-------------------+ Table 3: Path capacity variation pattern for forward direction - +---------------------+----------------+------------+---------------+ - | Variation pattern | Path direction | Start time | Path capacity | - | index | | | | - +---------------------+----------------+------------+---------------+ - | One | Backward | 0s | 2Mbps | - | Two | Backward | 35s | 800Kbps | - | Three | Backward | 70s | 2Mbps | - +---------------------+----------------+------------+---------------+ + +--------------------+--------------+-----------+-------------------+ + | Variation pattern | Path | Start | Path capacity | + | index | direction | time | ratio | + +--------------------+--------------+-----------+-------------------+ + | One | Backward | 0s | 2.0 | + | Two | Backward | 35s | 0.8 | + | Three | Backward | 70s | 2.0 | + +--------------------+--------------+-----------+-------------------+ Table 4: Path capacity variation pattern for backward direction 5.4. Competing Flows with Same RMCAT Algorithm In this test case, more than one RMCAT media flow shares the bottleneck link and each of them uses the same congestion control algorithm. This is a typical scenario where a real-time interactive application sends more than one media flows to the same destination and these flows are multiplexed over the same port. In such a @@ -776,24 +807,25 @@ // <-- Backward \\ +---+ // \\ +---+ |S3 |====== / \ ======|R3 | +---+ +---+ Figure 5: Testbed Topology for Multiple RMCAT Flows Testbed attributes: o Test duration: 120s - o Path characteristics: - * Bottleneck-link capacity: 3.5Mbps + * Reference bottleneck capacity: 3.5Mbps + + * Path capacity ratio: 1.0 o Application-related: * Media Source: + Media type: Video - Media direction: forward. - Number of media sources: Three (3) @@ -961,23 +992,24 @@ +-----+ +-----+ Figure 6: Testbed Topology for TCP vs RMCAT Flows Testbed attributes: o Test duration: 120s o Path characteristics: - * Bottleneck-link capacity: 2Mbps + * Reference bottleneck capacity: 2Mbps - * Bottleneck queue size: [20ms, 300ms, 1000ms] + * Path capacity ratio: 1.0 + * Bottleneck queue size: [300ms, 1000ms] o Application-related: * Media Source: + Media type: Video - Media direction: forward - Number of media sources: One (1) @@ -1049,28 +1082,29 @@ Testbed topology: The topology described here is same as the one described in Figure 6. Testbed attributes: o Test duration: 300s o Path characteristics: - * Bottleneck-link capacity: 2.0Mbps + * Reference bottleneck capacity: 2.0Mbps + + * Path capacity ratio: 1.0 o Application-related: * Media Source: + Media type: Video - - Media direction: forward - Number of media sources: two (2) - Media timeline: o Start time: 5s. o End time: 299s. @@ -1240,25 +1274,26 @@ respective destinations R1, R2, and R3. For all three flows the media traffic is transported over the forward path and corresponding feedback/control traffic is transported over the backward path. Testbed attributes: o Test duration: 120s o Path characteristics: - * Path capacity between A and B = 2Mbps. + * Reference bottleneck capacity between A and B = 2Mbps. - * Path capacity between B and C = 8Mbps. + * Path capacity ratio between A and B: 1.0 + * Path capacity ratio between B and C: 4.0. - * Path capacity between C and D = 1.5Mbps. + * Path capacity ratio between C and D: 0.75. * One-Way propagation delay: 1. Between S1 and R1 : 100ms 2. Between S2 and R2: 40ms 3. Between S3 and R3: 40ms o Application-related: @@ -1288,31 +1323,22 @@ o Start time: 0s. o End time: 119s. * Competing traffic: + Number of sources : Zero (0) 7. Wireless Access Links -7.1. Cellular Network Specific Test Cases - - Additional cellular network specific test cases are define in - [I-D.draft-sarker-rmcat-cellular-eval-test-cases] - -7.2. Wi-Fi Network Specific Test Cases - - TBD - - [Editor's Note: We should encourage people to come up with possible - WiFi Network specific test cases] + Additional wireless network (both cellular network and WiFi network) + specific test cases are define in [I-D.ietf-rmcat-wireless-tests] 8. Security Considerations Security issues have not been discussed in this memo. 9. IANA Considerations There are no IANA impacts in this memo. 10. Acknowledgements @@ -1322,74 +1348,74 @@ The content and concepts within this document are a product of the discussion carried out in the Design Team. 11. References 11.1. Normative References [RFC6679] Westerlund, M., Johansson, I., Perkins, C., O'Hanlon, P., and K. Carlberg, "Explicit Congestion Notification (ECN) - for RTP over UDP", RFC 6679, August 2012. + for RTP over UDP", RFC 6679, DOI 10.17487/RFC6679, August + 2012, . [RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V. Jacobson, "RTP: A Transport Protocol for Real-Time - Applications", STD 64, RFC 3550, July 2003. + Applications", STD 64, RFC 3550, DOI 10.17487/RFC3550, + July 2003, . [RFC3551] Schulzrinne, H. and S. Casner, "RTP Profile for Audio and Video Conferences with Minimal Control", STD 65, RFC 3551, - July 2003. + DOI 10.17487/RFC3551, July 2003, + . - [RFC3611] Friedman, T., Caceres, R., and A. Clark, "RTP Control - Protocol Extended Reports (RTCP XR)", RFC 3611, November - 2003. + [RFC3611] Friedman, T., Ed., Caceres, R., Ed., and A. Clark, Ed., + "RTP Control Protocol Extended Reports (RTCP XR)", RFC + 3611, DOI 10.17487/RFC3611, November 2003, + . [RFC4585] Ott, J., Wenger, S., Sato, N., Burmeister, C., and J. Rey, "Extended RTP Profile for Real-time Transport Control - Protocol (RTCP)-Based Feedback (RTP/AVPF)", RFC 4585, July - 2006. + Protocol (RTCP)-Based Feedback (RTP/AVPF)", RFC 4585, DOI + 10.17487/RFC4585, July 2006, + . [RFC5506] Johansson, I. and M. Westerlund, "Support for Reduced-Size Real-Time Transport Control Protocol (RTCP): Opportunities - and Consequences", RFC 5506, April 2009. - - [I-D.ietf-avtcore-rtp-circuit-breakers] - Perkins, C. and V. Singh, "Multimedia Congestion Control: - Circuit Breakers for Unicast RTP Sessions", draft-ietf- - avtcore-rtp-circuit-breakers-09 (work in progress), March - 2015. + and Consequences", RFC 5506, DOI 10.17487/RFC5506, April + 2009, . [I-D.ietf-rmcat-eval-criteria] Singh, V. and J. Ott, "Evaluating Congestion Control for Interactive Real-time Media", draft-ietf-rmcat-eval- - criteria-02 (work in progress), July 2014. + criteria-03 (work in progress), March 2015. + + [I-D.ietf-rmcat-wireless-tests] + Sarker, Z. and I. Johansson, "Evaluation Test Cases for + Interactive Real-Time Media over Wireless Networks", + draft-ietf-rmcat-wireless-tests-00 (work in progress), + June 2015. + +11.2. Informative References [I-D.ietf-rmcat-cc-requirements] Jesup, R. and Z. Sarker, "Congestion Control Requirements for Interactive Real-Time Media", draft-ietf-rmcat-cc- requirements-09 (work in progress), December 2014. - [I-D.draft-sarker-rmcat-cellular-eval-test-cases] - Sarker, Z. and I. Johansson, "Evaluation Test Cases for - Interactive Real-Time Media over Cellular Networks", 6 - 2014, . - -11.2. Informative References - [xiph-seq] Xiph.org, , "Video Test Media", - http://media.xiph.org/video/derf/ , . + http://media.xiph.org/video/derf/ . [HEVC-seq] HEVC, , "Test Sequences", - http://www.netlab.tkk.fi/~varun/test_sequences/ , . + http://www.netlab.tkk.fi/~varun/test_sequences/ . Authors' Addresses Zaheduzzaman Sarker Ericsson AB Luleae, SE 977 53 Sweden Phone: +46 10 717 37 43 Email: zaheduzzaman.sarker@ericsson.com