--- 1/draft-ietf-rmcat-eval-test-00.txt 2015-03-09 15:14:33.416151173 -0700 +++ 2/draft-ietf-rmcat-eval-test-01.txt 2015-03-09 15:14:33.476152672 -0700 @@ -1,22 +1,22 @@ Network Working Group Z. Sarker Internet-Draft Ericsson AB Intended status: Informational V. Singh -Expires: February 15, 2015 Aalto University +Expires: September 11, 2015 Aalto University X. Zhu M. Ramalho Cisco Systems - August 14, 2014 + March 10, 2015 Test Cases for Evaluating RMCAT Proposals - draft-ietf-rmcat-eval-test-00 + draft-ietf-rmcat-eval-test-01 Abstract The Real-time Transport Protocol (RTP) is used to transmit media in multimedia telephony applications, these applications are typically required to implement congestion control. The RMCAT working group is currently working on candidate algorithms for such interactive real- time multimedia applications. This document describes the test cases to be used in the performance evaluation of those candidate algorithms. @@ -29,76 +29,68 @@ Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at http://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." - This Internet-Draft will expire on February 15, 2015. + This Internet-Draft will expire on September 11, 2015. Copyright Notice - Copyright (c) 2014 IETF Trust and the persons identified as the + Copyright (c) 2015 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Table of Contents - 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 + 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3 3. Structure of Test cases . . . . . . . . . . . . . . . . . . . 3 4. Recommended Evaluation Settings . . . . . . . . . . . . . . . 7 - 4.1. Evaluation metircs . . . . . . . . . . . . . . . . . . . 8 + 4.1. Evaluation metircs . . . . . . . . . . . . . . . . . . . 7 4.2. Path characteristics . . . . . . . . . . . . . . . . . . 8 4.3. Media source . . . . . . . . . . . . . . . . . . . . . . 9 5. Basic Test Cases . . . . . . . . . . . . . . . . . . . . . . 10 5.1. Variable Available Capacity with Single RMCAT flow . . . 10 5.2. Variable Available Capacity with Multiple RMCAT flows . . 12 - 5.3. Congested Feedback Link with Bi-directional RMCAT flows . 14 + 5.3. Congested Feedback Link with Bi-directional RMCAT flows . 13 5.4. Competing Flows with Same RMCAT Algorithm . . . . . . . . 16 5.5. Round Trip Time Fairness . . . . . . . . . . . . . . . . 18 5.6. RMCAT Flow competing with a long TCP Flow . . . . . . . . 20 5.7. RMCAT Flow competing with short TCP Flows . . . . . . . . 22 5.8. Media Pause and Resume . . . . . . . . . . . . . . . . . 24 6. Other potential test cases . . . . . . . . . . . . . . . . . 26 6.1. Explicit Congestion Notification Usage . . . . . . . . . 26 6.2. Multiple Bottlenecks . . . . . . . . . . . . . . . . . . 26 7. Wireless Access Links . . . . . . . . . . . . . . . . . . . . 28 7.1. Cellular Network Specific Test Cases . . . . . . . . . . 28 7.2. Wi-Fi Network Specific Test Cases . . . . . . . . . . . . 28 8. Security Considerations . . . . . . . . . . . . . . . . . . . 29 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 29 10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 29 11. References . . . . . . . . . . . . . . . . . . . . . . . . . 29 11.1. Normative References . . . . . . . . . . . . . . . . . . 29 11.2. Informative References . . . . . . . . . . . . . . . . . 30 - Appendix A. List of Network Parameters . . . . . . . . . . . . . 31 - A.1. One-way Propagation Delay . . . . . . . . . . . . . . . . 31 - A.2. End-to-end Loss . . . . . . . . . . . . . . . . . . . . . 31 - A.3. DropTail Router Queue Length . . . . . . . . . . . . . . 32 - Appendix B. Models . . . . . . . . . . . . . . . . . . . . . . . 32 - B.1. Jitter models . . . . . . . . . . . . . . . . . . . . . . 32 - B.2. Loss generation model . . . . . . . . . . . . . . . . . . 35 - B.3. TCP taffic model . . . . . . . . . . . . . . . . . . . . 35 - Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 35 + Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 30 1. Introduction This memo describes a set of test cases for evaluating candidate RMCAT congestion control algorithm proposals, it is based on the guidelines enumerated in [I-D.ietf-rmcat-eval-criteria] and the requirements discussed in [I-D.ietf-rmcat-cc-requirements]. The test cases cover basic usage scenarios and are described using a common structure, which allows for additional test cases to be added to those described herein to accommodate other topologies and/or the @@ -225,20 +217,23 @@ PIE. + Bottleneck queue size: defines size of queue in terms of queuing time when the queue is full (in milliseconds). + Path loss ratio: characterizes the non-congested, additive, losses to be generated on the end-to-end path. MUST describe the loss pattern or loss model used to generate the losses. + + Values for some characteristics are described in + [I-D.ietf-rmcat-eval-criteria]. + * Application-related: defines the traffic source behaviour for implementing the test case + Media traffic Source: defines the characteristics of the media sources. When using more than one media source, the different attributes are enumerated separately for each different media source. - Media type: Video/Voice/Application/Text @@ -366,34 +361,35 @@ Each path between a sender and receiver as described in Figure 1 have the following characteristics unless otherwise specified in the test case. o Path direction: forward and backward. o Bottleneck-link capacity: 4Mbps. o One-Way propagation delay: 50ms. It is encouraged to test with - additional propagation delays mentioned in Appendix A.1 + additional propagation delays mentioned in + [I-D.ietf-rmcat-eval-criteria] o Maximum end-to-end jitter: 30ms. Jitter models are described in - Appendix B.1 + [I-D.ietf-rmcat-eval-criteria] o Bottleneck queue type: Drop tail. It is encouraged to test with other AQM schemes, such as FQ-CoDel and PIE. o Bottleneck queue size: 300ms. o Path loss ratio: 0%. Examples of additional network parameters are discussed in - Appendix A. + [I-D.ietf-rmcat-eval-criteria]. 4.3. Media source Unless otherwise specified, each test case will include one or more media sources as described below. o Media type: Video * Media codec: VBR @@ -716,21 +712,21 @@ +---------------------+----------------+------------+---------------+ | One | Forward | 0s | 2Mbps | | Two | Forward | 20s | 1Mbps | | Three | Forward | 40s | 500Kbps | | Four | Forward | 60s | 2Mbps | +---------------------+----------------+------------+---------------+ Table 3: Path capacity variation pattern for forward direction +---------------------+----------------+------------+---------------+ - | Variation pattern | Path direction | Start time | Path Capacity | + | Variation pattern | Path direction | Start time | Path capacity | | index | | | | +---------------------+----------------+------------+---------------+ | One | Backward | 0s | 2Mbps | | Two | Backward | 35s | 800Kbps | | Three | Backward | 70s | 2Mbps | +---------------------+----------------+------------+---------------+ Table 4: Path capacity variation pattern for backward direction 5.4. Competing Flows with Same RMCAT Algorithm @@ -810,45 +807,35 @@ - Number of media sources: Three (3) - Media timeline: New media flows are added sequentially, at short time intervals. See test specific setup below. * Competing traffic: + Number of sources : Zero (0) - o Test Specific Information: - - * Media flow timeline: - - + Flow ID: One (1) - - + Start time: 0s - - + End time: 119s - - * Media flow timeline: - - + Flow ID: Two (2) - - + Start time: 20s - - + End time: 119s - - * Media flow timeline: - - + Flow ID: Three (3) + o Test Specific Information: Table 5 defines the media timeline for + both media type. - + Start time: 40s + +---------+------------+------------+----------+ + | Flow IF | Media type | Start time | End time | + +---------+------------+------------+----------+ + | 1 | Video | 0s | 119s | + | 2 | Video | 20s | 119s | + | 3 | Video | 40s | 119s | + | 4 | Audio | 0s | 119s | + | 5 | Audio | 20s | 119s | + | 6 | Audio | 40s | 119s | + +---------+------------+------------+----------+ - + End time: 119s + Table 5: Media Timeline for Video and Audio media sources 5.5. Round Trip Time Fairness In this test case, multiple RMCAT media flows share the bottleneck link, but the end-to-end path latency for each RMCAT flow is different. For the sake of simplicity it is assumed that there are no other non-RMCAT competing traffic sources in the bottleneck link and that there is sufficient capacity to accommodate all the flows. While this appears to be a variant of test case 5.2, it focuses on the capacity sharing aspect of the candidate algorithm under @@ -866,59 +853,72 @@ Testbed Topology: Five (5) media sources S1,S2,..,S5 are connected to their corresponding media sinks R1,R2,..,R5. The media traffic is transported over the forward path and corresponding feedback/control traffic is transported over the backward path. The topology is the same as in Section 5.4. The end-to-end path delays are: 10ms for S1-R1, 25ms for S2-R2, 50ms for S3-R3, 100ms for S4-R4, and 150ms S5-R5, respectively. Testbed attributes: - o Test duration: 120s + o Test duration: 300s o Path characteristics: * One-Way propagation delay for each flow: 10ms, 25ms, 50ms, 100ms, 150ms. o Application-related: * Media Source: + Media type: Video - Media direction: forward - Number of media sources: Five (5) - - Media timeline: - - o Start time: 0s. - - o End time: 119s. + - Media timeline: New media flows are added sequentially, + at short time intervals. See test specific setup below. + Media type: Audio - Media direction: forward. - Number of media sources: Five (5) - - Media timeline: - o Start time: 0s. - - o End time: 119s. + - Media timeline: New media flows are added sequentially, + at short time intervals. See test specific setup below. * Competing traffic: + Number of sources : Zero (0) - o Test Specific Information: None + o Test Specific Information: Table 6 defines the media timeline for + both media type. + + +---------+------------+------------+----------+ + | Flow IF | Media type | Start time | End time | + +---------+------------+------------+----------+ + | 1 | Video | 0s | 299s | + | 2 | Video | 10s | 299s | + | 3 | Video | 20s | 299s | + | 4 | Video | 30s | 299s | + | 5 | Video | 40s | 299s | + | 6 | Audio | 0 | 299s | + | 7 | Audio | 10s | 299s | + | 8 | Audio | 20s | 299s | + | 9 | Audio | 30s | 299s | + | 10 | Audio | 40s | 299s | + +---------+------------+------------+----------+ + + Table 6: Media Timeline for Video and Audio media sources 5.6. RMCAT Flow competing with a long TCP Flow In this test case, one or more RMCAT media flows share the bottleneck link with at least one long lived TCP flows. Long lived TCP flows download data throughout the session and are expected to have infinite amount of data to send and receive. This is a scenario where a multimedia application co-exists with a large file download. The test case measures the adaptivity of the candidate algorithm to competing traffic. It addresses the requirement 3 in @@ -1348,296 +1348,68 @@ Protocol (RTCP)-Based Feedback (RTP/AVPF)", RFC 4585, July 2006. [RFC5506] Johansson, I. and M. Westerlund, "Support for Reduced-Size Real-Time Transport Control Protocol (RTCP): Opportunities and Consequences", RFC 5506, April 2009. [I-D.ietf-avtcore-rtp-circuit-breakers] Perkins, C. and V. Singh, "Multimedia Congestion Control: Circuit Breakers for Unicast RTP Sessions", draft-ietf- - avtcore-rtp-circuit-breakers-05 (work in progress), - February 2014. + avtcore-rtp-circuit-breakers-09 (work in progress), March + 2015. [I-D.ietf-rmcat-eval-criteria] Singh, V. and J. Ott, "Evaluating Congestion Control for Interactive Real-time Media", draft-ietf-rmcat-eval- - criteria-00 (work in progress), January 2014. + criteria-02 (work in progress), July 2014. [I-D.ietf-rmcat-cc-requirements] - Jesup, R., "Congestion Control Requirements For RMCAT", - draft-ietf-rmcat-cc-requirements-02 (work in progress), - February 2014. + Jesup, R. and Z. Sarker, "Congestion Control Requirements + for Interactive Real-Time Media", draft-ietf-rmcat-cc- + requirements-09 (work in progress), December 2014. [I-D.draft-sarker-rmcat-cellular-eval-test-cases] Sarker, Z. and I. Johansson, "Evaluation Test Cases for Interactive Real-Time Media over Cellular Networks", 6 2014, . 11.2. Informative References - [I-D.ietf-rtcweb-use-cases-and-requirements] - Holmberg, C., Hakansson, S., and G. Eriksson, "Web Real- - Time Communication Use-cases and Requirements", draft- - ietf-rtcweb-use-cases-and-requirements-14 (work in - progress), February 2014. - - [RFC5033] Floyd, S. and M. Allman, "Specifying New Congestion - Control Algorithms", BCP 133, RFC 5033, August 2007. - - [RFC5166] Floyd, S., "Metrics for the Evaluation of Congestion - Control Mechanisms", RFC 5166, March 2008. - - [RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion - Control", RFC 5681, September 2009. - - [SA4-EVAL] - R1-081955, 3GPP., "LTE Link Level Throughput Data for SA4 - Evaluation Framework", 3GPP R1-081955, 5 2008. - - [SA4-LR] S4-050560, 3GPP., "Error Patterns for MBMS Streaming over - UTRAN and GERAN", 3GPP S4-050560, 5 2008. - [xiph-seq] Xiph.org, , "Video Test Media", http://media.xiph.org/video/derf/ , . [HEVC-seq] HEVC, , "Test Sequences", http://www.netlab.tkk.fi/~varun/test_sequences/ , . - [TCP-eval-suite] - Lachlan, A., Marcondes, C., Floyd, S., Dunn, L., Guillier, - R., Gang, W., Eggert, L., Ha, S., and I. Rhee, "Towards a - Common TCP Evaluation Suite", Proc. PFLDnet. 2008, August - 2008. - -Appendix A. List of Network Parameters - - In addition to the recommended evaluation settings in Section 4, the - implemntors can extend their tests by choosing from the following - values: - -A.1. One-way Propagation Delay - - Experiments are expected to verify that the congestion control is - able to work in challenging situations, for example over trans- - continental and/or satellite links. Typical values are: - - 1. Very low latency: 0-1ms - - 2. Low latency: 50ms - - 3. High latency: 150ms - - 4. Extreme latency: 300ms - -A.2. End-to-end Loss - - To model lossy links, the experiments can choose one of the following - loss rates, the fractional loss is the ratio of packets lost and - packets sent. - - 1. no loss: 0% - - 2. 1% - - 3. 5% - - 4. 10% - 5. 20% - -A.3. DropTail Router Queue Length - - The router queue length is measured as the time taken to drain the - FIFO queue. It has been noted in various discussions that the queue - length in the current deployed Internet varies significantly. While - the core backbone network has very short queue length, the home - gateways usually have larger queue length. Those various queue - lengths can be categorized in the following way: - - 1. QoS-aware (or short): 70ms - - 2. Nominal: 300-500ms - - 3. Buffer-bloated: 1000-2000ms - - Here the size of the queue is measured in bytes or packets and to - convert the queue length measured in seconds to queue length in - bytes: - - QueueSize (in bytes) = QueueSize (in sec) x Throughput (in bps)/8 - -Appendix B. Models - -B.1. Jitter models - - This section defines jitter model for the purposes of this document. - When jitter is to be applied to both the RMCAT flow and any competing - flow (such as a TCP competing flow), the competing flow will use the - jitter definition below that does not allow for re-ordering of - packets on the competing flow (see NR-RBPDV definition below). - - Jitter is an overloaded term in communications. Its meaning is - typically associated with the variation of a metric (e.g., delay) - with respect to some reference metric (e.g., average delay or minimum - delay). For example, RFC 3550 jitter is a smoothed estimate of - jitter which is particularly meaningful if the underlying packet - delay variation was caused by a Gaussian random process. - - Because jitter is an overloaded term, we instead use the term Packet - Delay Variation (PDV) to describe the variation of delay of - individual packets in the same sense as the IETF IPPM WG has defined - PDV in their documents (e.g., RFC 3393) and as the ITU-T SG16 has - defined IP Packet Delay Variation (IPDV) in their documents (e.g., - Y.1540). - - Most PDV distributions in packet network systems are one-sided - distributions (the measurement of which with a finite number of - measurement samples result in one-sided histograms). In the usual - packet network transport case there is typically one packet that - transited the network with the minimum delay, then a majority of - packets also transit the system within some variation from this - minimum delay, and then a minority of the packets transits the - network with delays higher than the median or average transit time - (these are outliers). Although infrequent, outliers can cause - significant deleterious operation in adaptive systems and should be - considered in RMCAT adaptation designs. - - In this section we define two different bounded PDV characteristics, - 1) Random Bounded PDV and 2) Approximately Random Subject to No- - Reordering Bounded PDV. - - Random Bounded PDV (RBPDV): - - The RBPDV probability distribution function (pdf) is specified to be - of some mathematically describable function which includes some - practical minimum and maximum discrete values suitable for testing. - For example, the minimum value, x_min, might be specified as the - minimum transit time packet and the maximum value, x_max, might be - idefined to be two standard deviations higher than the mean. - - Since we are typically interested in the distribution relative to the - mean delay packet, we define the zero mean PVD sample, z(n), to be - z(n) = x(n) - x_mean, where x(n) is a sample of the RBPDV random - variable x and x_mean is the mean of x. - - We assume here that s(n) is the original source time of packet n and - the post-jitter induced emmission time, j(n), for packet n is j(n) = - {[z(n) + x_mean] + s(n)}. It follows that the separation in the post- - jitter time of packets n and n+1 is {[s(n+1)-s(n)] - [z(n)-z(n+1)]}. - Since the first term is always a positive quantity, we note that - packet reordering at the receiver is possible whenever the second - term is greater than the first. Said another way, whenever the - difference in possible zero mean PDV sample delays (i.e., [x_max- - x_min]) exceeds the inter-departure time of any two sent packets, we - have the possibility of packet re-ordering. - - There are important use cases in real networks where packets can - become re-ordered such as in load balancing topologies and during - route changes. However, for the vast majority of cases there is no - packet re-ordering because most of the time packets follow the same - path. Due to this, if a packet becomes overly delayed, the packets - after it on that flow are also delayed. This is especially true for - mobile wireless links where there are per-flow queues prior to base - station scheduling. Owing to this important use case, we define - another PDV profile similar to the above, but one that does not allow - for re-ordering within a flow. - - Approximately Random Subject to No-Reordering Bounded PDV (NR-RPVD): - - No Reordering RPDV, NR-RPVD, is defined similarly to the above with - one important exception. Let serial(n) be defined as the - serialization delay of packet n at the lowest bottleneck link rate - (or other appropriate rate) in a given test. Then we produce all the - post-jitter values for j(n) for n = 1, 2, ... N, where N is the - length of the source sequence s to be jittered. The exception can be - stated as follows: We revisit all j(n) beginning from index n=2, and - if j(n) is determined to be less than [j(n-1)+serial(n-1)], we - redefine j(n) to be equal to [j(n-1)+serial(n-1)] and continue for - all remaining n (i.e., n = 3, 4, .. N). This models the case where - the packet n is sent immediately after packet (n-1) at the bottleneck - link rate. Although this is generally the theoretical minimum in - that it assumes that no other packets from other flows are in-between - packet n and n+1 at the bottleneck link, it is a reasonable - assumption for per flow queuing. - - We note that this assumption holds for some important exception - cases, such as packets immediately following outliers. There are a - multitude of software controlled elements common on end-to-end - Internet paths (such as firewalls, ALGs and other middleboxes) which - stop processing packets while servicing other functions (e.g., - garbage collection). Often these devices do not drop packets, but - rather queue them for later processing and cause many of the - outliers. Thus NR-RPVD models this particular use case (assuming - serial(n+1) is defined appropriately for the device causing the - outlier) and thus is believed to be important for adaptation - development for RMCAT. - - [Editor's Note: It may require to define test distributions as well. - Example test distrubution may include- - - 1 - Two-sided: Uniform PDV Distribution. Two quantities to define: - x_min and x_max. - - 2 - Two-sided: Truncated Gaussian PDV Distribution. Four quantities - to define: the appropriate x_min and x_max for test (e.g., +/- two - sigma values), the standard deviation and the mean. - - 3 - One Sided: TBD] - -B.2. Loss generation model - - [Editor's note : Describes the model for generating packet losses, - for example, losses can be generated using traces, or using the - Gilbert-Elliot model, or randomly (uncorrelated loss).] - -B.3. TCP taffic model - - Long-lived TCP flows will download data throughout the session and - are expected to have infinite amount of data to send or receive. - - Each short TCP flow is modeled as a sequence of file downloads - interleaved with idle periods. Not all short TCPs start at the same - time, i.e., some start in the ON state while others start in the OFF - state. - - The short TCP flows can be modelled in two ways, 1) 100s of flows - fetching small (5-20 KB) amounts of data, or 2) 10s of flows fetching - slightly larger (100-1000KB) amounts of data. - - The idle period is typically derived from an exponential distribution - with the mean value of 10 seconds. - - [Open issue: short-lived/bursty TCP cross-traffic parameters are - still to be agreed upon]. - Authors' Addresses Zaheduzzaman Sarker Ericsson AB Luleae, SE 977 53 Sweden Phone: +46 10 717 37 43 Email: zaheduzzaman.sarker@ericsson.com - Varun Singh Aalto University School of Electrical Engineering Otakaari 5 A Espoo, FIN 02150 Finland Email: varun@comnet.tkk.fi URI: http://www.netlab.tkk.fi/~varun/ + Xiaoqing Zhu Cisco Systems 510 McCarthy Blvd Milpitas, CA 95134 USA Email: xiaoqzhu@cisco.com Michael A. Ramalho Cisco Systems, Inc. @@ -1634,16 +1406,16 @@ Xiaoqing Zhu Cisco Systems 510 McCarthy Blvd Milpitas, CA 95134 USA Email: xiaoqzhu@cisco.com Michael A. Ramalho Cisco Systems, Inc. - 8000 Hawkins Road - Sarasota, FL 34241 + 6310 Watercrest Way Unit 203 + Lakewood Ranch, FL 34202-5211 USA Phone: +1 919 476 2038 Email: mramalho@cisco.com