DetNet                                                  J. Korhonen, Ed.
Internet-Draft
Intended status: Standards Track                           B. Varga, Ed.
Expires: April 24, September 11, 2019                                     Ericsson
                                                        October 21, 2018
                                                          March 10, 2019

                  DetNet MPLS Data Plane Encapsulation
                    draft-ietf-detnet-dp-sol-mpls-01
                    draft-ietf-detnet-dp-sol-mpls-02

Abstract

   This document specifies the Deterministic Networking data plane
   encapsulation solutions.  The described data plane solutions is
   applied when
   operating over an MPLS Packet Switched Networks.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at https://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on April 24, September 11, 2019.

Copyright Notice

   Copyright (c) 2018 2019 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (https://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   3
   2.  Terminology . . . . . . . . . . . . . . . . . . . . . . . . .   4
     2.1.  Terms used Used in this document This Document . . . . . . . . . . . . . . .   4
     2.2.  Abbreviations . . . . . . . . . . . . . . . . . . . . . .   4
   3.  Requirements language Language . . . . . . . . . . . . . . . . . . . .   6   5
   4.  MPLS  DetNet data plane overview MPLS Data Plane Overview . . . . . . . . . . . . . . .   6
     4.1.  Layers of DetNet data plane Data Plane . . . . . . . . . . . . . . .   6
     4.2.  MPLS  DetNet data plane scenarios MPLS Data Plane Scenarios  . . . . . . . . . . . .   7
     4.3.  Packet flow example (service protection)  . . . . . . . .  11
   5.
       4.2.1.  IP Over DetNet MPLS Data Considerations . . . . . . . . Plane Scenarios  . . . . . .   9
       4.2.2.  IEEE 802.1 TSN Over DetNet MPLS Data Plane Scenario .  12
     5.1.  End-system specific considerations  . . . .
     4.3.  Packet Flow Example with Service Protection . . . . . . .  13
     5.2.  14
   5.  DetNet domain specific considerations . MPLS Data Plane Considerations . . . . . . . . .  15
       5.2.1.  DetNet Layer Two Service . . .  15
     5.1.  End-System Specific Considerations  . . . . . . . . . . .  16
       5.2.2.  DetNet Routing Service (IP over MPLS) . . . . . . . .  17
     5.3.  DetNet Inter-Working Function (DN-IWF)  . . . . . . . . .  18
       5.3.1.  Networks with multiple technology segments  . .
     5.2.  Sub-Network Considerations  . . .  18
       5.3.2.  DN-IWF related considerations . . . . . . . . . . . .  19  17
   6.  MPLS-based  MPLS-Based DetNet data plane solution Data Plane Solution . . . . . . . . . . . .  20  18
     6.1.  DetNet over Over MPLS Encapsulation Components . . . . . . . .  20  18
     6.2.  MPLS data plane encapsulation Data Plane Encapsulation . . . . . . . . . . . . . .  22
     6.3.  19
       6.2.1.  DetNet control word . . . Control Word and the DetNet Sequence Number  .  20
       6.2.2.  S-Labels  . . . . . . . . . . . . . . .  23
     6.4.  Flow Identification . . . . . . .  21
       6.2.3.  F-Labels  . . . . . . . . . . . .  24
     6.5.  Indication of the DetNet Payload Type . . . . . . . . . .  24
     6.6.
     6.3.  OAM Indication  . . . . . . . . . . . . . . . . . . . . .  25
     6.7.  26
     6.4.  Flow Aggregation  . . . . . . . . . . . . . . . . . . . .  25
       6.7.1.  27
       6.4.1.  Aggregation at the LSP  . . . . . . . . . . . . . . .  26
       6.7.2.  28
       6.4.2.  Aggregating DetNet flows Flows as a new DetNet flow . . . .  26
       6.7.3.  28
       6.4.3.  Simple Aggregation at the DetNet layer Layer  . . . . . . .  27
     6.8.  29
     6.5.  Service Layer Sub-Layer Considerations  . . . . . . . . . . . . . .  28
       6.8.1.  29
       6.5.1.  Edge node processing Node Processing  . . . . . . . . . . . . . . . .  28
       6.8.2.  30
       6.5.2.  Relay node processing Node Processing . . . . . . . . . . . . . . . .  29
     6.9.  Other DetNet data plane considerations  31
     6.6.  Forwarding Sub-Layer Considerations . . . . . . . . .  30
       6.9.1. . .  31
       6.6.1.  Class of Service  . . . . . . . . . . . . . . . . . .  30
       6.9.2.  31
       6.6.2.  Quality of Service  . . . . . . . . . . . . . . . . .  31
       6.9.3.  32
       6.6.3.  Cross-DetNet flow resource aggregation Flow Resource Aggregation  . . . . . . .  32
       6.9.4.
       6.6.4.  Layer 2 addressing Addressing and QoS Considerations . . . . . .  33
       6.9.5.
       6.6.5.  Time Synchronization  . . . . . . . . . . . . . . . .  33  34
   7.  Management  Controller Plane (Management and control considerations . . . . . . . Control)
       Considerations  . . . . .  34
     7.1.  MPLS-based data plane . . . . . . . . . . . . . . . . . .  34
       7.1.1.
     7.1.  S-Label assignment and distribution . . . . . . . . .  34
       7.1.2.  Explicit routes . . . . . . . . . . . . . . F-Label Assignment and Distribution . . . . .  34  35
     7.2.  Packet replication Replication, Elimination, and elimination  . . . . . . . . . Ordering (PREOF) . .  35  36
     7.3.  Congestion protection  Contention Loss and latency control Jitter Reduction  . . . . . . . . . .  36
     7.4.  Bidirectional traffic Traffic . . . . . . . . . . . . . . . . . .  36  37
     7.5.  Flow aggregation control Aggregation Control  . . . . . . . . . . . . . . . .  36
   8.  38
     7.6.  DetNet IP Operation over
           Controller (Control and Management) Plane Requirements  .  38
   8.  DetNet MPLS Service  . . . . . . . .  36
   9. Operation Over IEEE 802.1 TSN Interconnection over DetNet MPLS Service Sub-Networks  . . .  37
   10. DetNet MPLS Transport Layer Operation over IEEE 802.1  39
     8.1.  Mapping of TSN
       Sub-Networks Stream ID and Sequence Number  . . . . . .  41
     8.2.  TSN Usage of FRER . . . . . . . . . . . . . . . . . .  37
     10.1.  Mapping of TSN Stream ID . .  42
     8.3.  Management and Sequence Number Control Implications . . . . . .  38
     10.2.  TSN Usage of FRER . . . . .  42
   9.  DetNet MPLS Operation over DetNet
       IP PSNs . . . . . . . . . . . . . .  40
     10.3.  Management and Control Implications . . . . . . . . . .  40
   11. DetNet MPLS Transport Layer Operation over IP DetNet PSNs . .  40
   12. .  43
   10. Security considerations Considerations . . . . . . . . . . . . . . . . . . .  42
   13.  45
   11. IANA considerations Considerations . . . . . . . . . . . . . . . . . . . . .  42
   14.  45
   12. Contributors  . . . . . . . . . . . . . . . . . . . . . . . .  43
   15.  45
   13. Acknowledgements  . . . . . . . . . . . . . . . . . . . . . .  45
   16.  46
   14. References  . . . . . . . . . . . . . . . . . . . . . . . . .  45
     16.1.  47
     14.1.  Normative references References . . . . . . . . . . . . . . . . . .  45
     16.2.  47
     14.2.  Informative references References . . . . . . . . . . . . . . . . .  48  49
   Appendix A.  Example of DetNet data plane operation Data Plane Operation . . . . . . .  50  52
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  50  52

1.  Introduction

   Deterministic Networking (DetNet) is a service that can be offered by
   a network to DetNet flows.  DetNet provides these flows with a low
   packet loss rates and assured maximum end-to-end delivery latency.
   General background and concepts of DetNet can be found in
   [I-D.ietf-detnet-architecture].

   The DetNet Architecture decomposes the DetNet related data plane
   functions into two sub-layers: a service sub-layer and a forwarding
   sub-layer.  The service sub-layer is used to provide DetNet service
   protection and reordering.  The forwarding sub-layer is used to
   provides congestion protection (low loss, assured latency, and
   limited reordering) leveraging MPLS Traffic Engineering mechanisms.

   This document specifies the DetNet data plane operation and the on-wire on-
   wire encapsulation of DetNet flows over an MPLS-based Packet Switched
   Network (PSN).  The specified encapsulation provides the building
   blocks to enable the DetNet service layer and forwarding sub-layer
   functions and allow supports flow identification as described in the DetNet
   Architecture.

   The DetNet transport layer functionality that provides congestion
   protection for DetNet flows is assumed to be in place in a DetNet
   node.

   Furthermore,  As part of the service sub-layer functions, this
   document also describes how DetNet flows are
   identified, and how a DetNet Relay/Edge/Transit nodes works. node data plane operation.  It also
   describes the function and operation of the Packet Replication (PRF)
   Packet Elimination (PEF) and Packet Ordering (POF) functions in the with an
   MPLS data plane.  It also describes an MPLS-based DetNet forwarding
   sub-layer that eliminates (or reduces) contention loss and provides
   bounded latency for DetNet flows.

   MPLS encapsulated DetNet flows can be carried over network
   technologies that can provide the DetNet required level of service.
   This document defines examples of such, specifically carrying DetNet
   MPLS flows over IEEE 802.1 TSN sub-networks, and over DetNet IP PSN.

   The intent is for this document to support different traffic types
   being mapped over DetNet MPLS, but this is out side the scope of this
   document.  An example of such can be found in
   [I-D.ietf-detnet-dp-sol-ip].  This document also allows for, but does
   not define the define, associated control controller plane functions,
   or and Operations,
   Administration, and Maintenance (OAM).  It also does
   not specify traffic handling capabilities required to deliver
   congestion protection and latency control for DetNet flows at the
   DetNet transport layer. (OAM) functions.

2.  Terminology

2.1.  Terms used Used in this document This Document

   This document uses the terminology established in the DetNet
   architecture [I-D.ietf-detnet-architecture] [I-D.ietf-detnet-architecture], and the DetNet Data Plane
   Solution Alternatives [I-D.ietf-detnet-dp-alt].

   T-Label reader is
   assumed to be familiar with that document and its terminology.

   The following terminology is introduced in this document:

   F-Label       A Detnet "forwarding" label used to identify that identifies the LSP
                 used to transport forward a DetNet flow across an MPLS PSN, e.g.,
                 a hop-by-hop label used between label switching routers
                 (LSR).

   S-Label       A DetNet "service" label that is used between DetNet
                 nodes that implement also the DetNet service layer sub-layer
                 functions.  An S-Label is also used to identify a
                 DetNet flow at DetNet service layer.

   PEF sub-layer.

   d-CW          A Packet Elimination Function (PEF) eliminates
                 duplicate copies of packets received by an edge or a
                 relay node to prevent excess packets flooding the
                 network, or to prevent DetNet Control Word (d-CW) is used for sequencing and
                 identifying duplicate packets being sent out of the DetNet domain.

   PRF           A Packet Replication Function (PRF) replicates a DetNet flow packets and forwards them to one or more next hops
                 in the DetNet domain.  The number of packet copies sent
                 to each next hop is a DetNet flow specific parameter at
                 the node doing the replication.  PRF can be implemented
                 by an edge node, a relay node, or an end system.

   POF           A Packet Ordering Function (POF) re-orders packets
                 within a DetNet flow that are received out of order.
                 This function can be implemented by an edge node, a
                 relay node, or an end system.

   PREOF         Collective name for Packet Replication, Elimination,
                 and Ordering Functions.

   d-CW          A DetNet Control Word (d-CW) is used for sequencing and
                 identifying duplicate packets of a DetNet flow at at the
                 DetNet service layer. sub-layer.

2.2.  Abbreviations

   The following abbreviations are used in this document:

   AC            Attachment Circuit.

   CE            Customer Edge equipment.

   CoS           Class of Service.

   CW            Control Word.

   d-CW          DetNet Control Word.

   DetNet        Deterministic Networking.

   DF            DetNet Flow.

   DN-IWF        DetNet Inter-Working Function.

   L2            Layer 2.

   L2VPN         Layer 2 Virtual Private Network.

   L3            Layer 3.

   LSR           Label Switching Router.

   MPLS          Multiprotocol Label Switching.

   MPLS-TE       Multiprotocol Label Switching - Traffic Engineering.

   MPLS-TP       Multiprotocol Label Switching - Transport Profile.

   MS-PW         Multi-Segment PseudoWire (MS-PW).

   NSP           Native Service Processing.

   OAM           Operations, Administration, and Maintenance.

   PE            Provider Edge.

   PEF           Packet Elimination Function.

   PRF           Packet Replication Function.

   PREOF         Packet Replication, Elimination and Ordering Functions.

   POF           Packet Ordering Function.

   PSN           Packet Switched Network.

   PW            PseudoWire.

   QoS           Quality of Service.

   S-PE          Switching Provider Edge.

   T-PE          Terminating Provider Edge.

   TSN           Time-Sensitive Network.

3.  Requirements language Language

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
   "OPTIONAL" in this document are to be interpreted as described in BCP
   14 [RFC2119] [RFC8174] when, and only when, they appear in all
   capitals, as shown here.

4.  MPLS  DetNet data plane overview MPLS Data Plane Overview

4.1.  Layers of DetNet data plane Data Plane

   This document describes how DetNet flows are carried over MPLS
   networks.  The DetNet Architecture, [I-D.ietf-detnet-architecture],
   decomposes the DetNet data plane into two layers: sub-layers: a service sub-
   layer and a transport layer. forwarding sub-layer.  The basic approach defined in this
   document supports the DetNet service layer sub-layer based on existing
   pseudowire (PW) encapsulations and mechanisms, and supports the
   DetNet transport
   layer forwarding sub-layer based on existing MPLS Traffic
   Engineering encapsulations and mechanisms.  Background on PWs can be
   found in [RFC3985] and [RFC3031].  Background on MPLS Traffic
   Engineering can be found in [RFC3272] and [RFC3209].

                         DetNet        MPLS
                           .
                           .
                        +-----------+
                       +------------+
                       |  Service   | d-CW, S-Label
                        +-----------+
                       +------------+
                       | Transport Forwarding | T-Label(s)
                        +-----------+ F-Label(s)
                       +------------+
                           .
                           .

              Figure 1: DetNet adaptation Adaptation to MPLS data plane Data Plane

   The MPLS DetNet MPLS data plane approach defined in this document is shown
   in Figure 1.  The service layer sub-layer is supported by a DetNet control
   word (d-CW) which conforms to the Generic PW MPLS Control Word
   (PWMCW) defined in [RFC4385].  A d-CW identifying service label
   (S-Label) is also used.  The transport layer is supported by one or labels
   (T-Labels).

   A node operating on a DetNet flow in the Detnet layer, service sub-layer,
   i.e. a node processing a DetNet packet which has the S-label S-Label as top
   of stack uses the local context associated with that S-label S-Label, for
   example a received F-Label, to determine what local DetNet
   operation(s) are applied to that packet.  The S-label has to  An S-Label may be unique on each edge and relay node, which is achieved by using a
   label
   when taken from the platform label space [RFC3031].

4.2.  MPLS [RFC3031], which would
   enable correct DetNet flow identification regardless of which input
   interface or LSP the packet arrives on.

   The DetNet MPLS data plane scenarios

   TSN             Edge builds on MPLS Traffic Engineering
   encapsulations and mechanisms to provide a forwarding sub-layer that
   is responsible for providing resource allocation and explicit routes.
   The forwarding sub-layer is supported by one or more forwarding
   labels (F-Labels).

4.2.  DetNet MPLS Data Plane Scenarios

   DetNet MPLS       Relay       Transit         Edge         TSN         Relay       DetNet MPLS
   End System        Node         Node           Node        End System
      (T-PE)        (S-PE)       (LSR)          (S-PE)         (T-PE)

   +---------+   +.........+                   +.........+   +---------+
   +----------+                                             +----------+
   |   Appl.  |<--:Svc Proxy:--End  |<------------ End to End Svc.--:Svc Proxy:-->| Service ----------->|   Appl.  |
   +----------+   +---------+                 +---------+                   +---------+   +---------+
   |   TSN   +----------+
   |   |TSN| |Svc|<-- Service  |<--| Service |-- DetNet flow -->: --| Service |-->| Service :-->|   TSN  |
   +----------+   +---------+   +---+ +---+    +---------+  +----------+   +---------+   +---------+
   |Transport|   |Trp| |Trp|    |Transport|    |Trp| |Trp|   |Transport|
   +-------.-+   +--.+   +----------+
   |Forwarding|   |Fwd| |Fwd|  |Forwarding|   |Fwd| |Fwd|   |Forwarding|
   +-------.--+   +-.-+    +--.----.-+ +-.-+  +----.---.-+   +-.-+   +---.-----+ +-.-+   +---.------+
           :  Link  :    /  ,-----.  \   : Link :    /  ,-----.  \
           +........+    +-[  Sub  ]-+   +........+   +......+    +-[  Sub  ]-+
                           [Network]                   [Network]
                            `-----'                     `-----'
           |<- TSN ->|   |<----- LSP -->| |<-------- LSP -----------| |<--- LSP -->|

           |<----------------- DetNet MPLS ---->|    |<-- TSN --->| --------------------->|

                      Figure 2: A TSN over DetNet MPLS Enabled Network

   Figure 2 shows several node types defined in
   [I-D.ietf-detnet-architecture].  DetNet Edge Nodes sit at the
   boundary of illustrates a hypothetical DetNet domain.  They are responsible for mapping non- MPLS-only network composed
   of DetNet aware traffic to MPLS enabled end systems, operating over a DetNet services.  They also support the
   imposition
   aware MPLS network.  In this figure, relay nodes sit at MPLS LSP
   boundaries and disposition of the required DetNet encapsulation.
   These are functionally similar to pseudowire (PW) Terminating
   Provider Edge (T-PE) transit nodes which use MPLS-TE LSPs.

   Native TSN flow are LSRs.

   DetNet end system and relay nodes are DetNet MPLS flow differ not only by service sub-layer aware,
   understand the
   additional MPLS specific encapsulation, but particular needs of DetNet MPLS flows have on
   each DetNet node an associated and provide both
   DetNet specific data structure, what
   defines flow related characteristics service and required forwarding sub-layer functions.  Edge Nodes MUST provide a Service Proxy entity that
   "associates" the DetNet flows  They add, remove
   and native flows at process d-CWs, S-Labels and F-labels as needed.  MPLS enabled end
   system and relay nodes can enhance the edge reliability of delivery by
   enabling the replication of packets where multiple copies, possibly
   over multiple paths, are forwarded through the DetNet domain.  It ensures that the DN Flow is properly served at the
   Edge node (and inside the domain).

   Transit nodes are normal MPLS Label Switching Routers (LSRs).  They
   are generally unaware of the special requirements
   can also eliminate surplus previously replicated copies of DetNet flows,
   although they need to provide traffic engineering services and proper
   QoS to the LSPs associated with
   packets.  DetNet flows MPLS nodes provide functionality similar to enhance T-PEs
   when they sit at the prospect edge of an MPLS domain, and functionality
   similar to S-PEs when they are in the LSPs meeting middle of an MPLS domain, see
   [RFC6073].  End system and relay nodes also include DetNet forwarding
   sub-layer functions, support for notably explicit routes, and
   resources allocation to eliminate (or reduce) congestion loss and
   jitter.

   DetNet transit nodes reside wholly within a DetNet domain, and also
   provide DetNet forwarding sub-layer functions in accordance with the
   performance required by a DetNet flow carried over an LSP.  Unlike
   other DetNet node types, transit nodes provide no service requirements.  Some
   implementations of sub-layer
   processing.  In a DetNet MPLS network, transit nodes may be DetNet aware, but
   service aware or may be DetNet unaware MPLS Label Switching Routers
   (LSRs).  In this latter case, such nodes
   just support LSRs would be unaware of the
   special requirements of the DetNet transport layer. service sub-layer, but would still
   provide traffic engineering services and the QoS need to ensure that
   the (TE) LSPs meet the service requirements of the carried DetNet
   flows.

   The MPLS LSP LSPs may be provided by any MPLS method (provisioned, RSVP-
   TE, MPLS- TP, controller method.  For example
   they may be provisioned via a management plane, RSVP-TE, MPLS-TP, or
   MPLS Segment Routing (SR)).

   IP  DetNet       Relay          Transit       Relay       IP (when extended to support resource allocation).

   Figure 3 illustrates how an end to end MPLS-based DetNet
   End System       Node            Node         Node        End System
                    (T-PE)          (LSR)        (T-PE)
   +---------+                                               +---------+
   |  Appl.  |<--------------- End service is
   provided in a more detail.  In this figure, the end systems, CE1 and
   CE2, are able to End Service ---------->|  Appl.  |
   +---------+    .....-----+                  +-----.....   +---------+
   | Service |<---: Service |-- send and receive MPLS encapsulated DetNet flow ---| Service :-->| Service |
   +---------+    +---------+   +---------+    +---------+   +---------+
   |Transport|    |Trp| |Trp|   |Transport|    |Trp| |Trp|   |Transport|
   +-------.-+    +-.-+ +-.-+   +---.---.-+    +-.-+ +-.-+   +---.-----+
           :  Link  :    /  ,-----.  \  :  Link  :    /  ,-----.  \
           +........+    +-[  Sub  ]-+  +........+    +-[  Sub  ]-+
                           [Network]                    [Network]
                            `-----'                      `-----'

           |<-DN IP->|   |<---- DetNet MPLS ---->|      |<-DN IP->|

                Figure 3: DetNet (DN) IP Over MPLS Network

   Figure 3 flows, and Figure 4, show different cases where relay nodes may be
   used.  Relay nodes are similar to edge nodes in that both are aware
   of the needs of particular DetNet flows
   R1, R2 and take care to process them
   in accordance with the required performance needs.  They differ in
   that relay nodes sit within a DetNet domain while edge nodes always
   sit at DetNet domain boundaries.  Both node types can enhance the
   reliability of delivery by enabling the replication of packets so
   that multiple copies, possibly over multiple paths R3 are forwarded
   through the DetNet domain.  They also reduce the impact of
   replication by eliminating surplus copies of DetNet packets.  Relay relay nodes may as they sit in the boundary of an MPLS domain when the non-MPLS domain
   is DetNet aware.  Relay nodes are functionally similar to PW S-PEs
   or, when at the edge middle of an MPLS network, T-PEs [RFC6073].

   Figure 4 illustrates how DetNet can provide services for IEEE
   802.1TSN end systems, CE1 and CE2, over a DetNet enabled
   network.  The edge nodes, E1 and E2, insert and remove required DetNet data
   plane encapsulation.  The 'X' in the edge nodes end systems, and relay node, R1,
   represent a nodes represents
   potential DetNet compound flow packet replication and elimination
   point.  This conceptually parallels L2VPN services,
   points.  In this example, service protection is supported over four
   DetNet member flows and could
   leverage existing related solutions as discussed below.

        TSN    |<------- End to End DetNet Service ------>|  TSN
       Service |         Transit          Transit         | Service
   TSN  (AC)   |        |<-Tnl->|        |<-Tnl->|        |  (AC)  TSN
   End    |    V        V    1  V        V   2   V        V   |    End
   System |    +--------+       +--------+       +--------+   |  System
   +---+  |    |   E1   |=======|   R1   |=======|   E2   |   |   +---+
   |   |--|----|._X_....|..DF1..|.._ _...|..DF3..|...._X_.|---|---|   |
   |CE1|  |    |    \   |       |   X    |       |   /    |   |   |CE2|
   |   |       |     \_.|..DF2..|._/ \_..|..DF4..|._/     |       |   |
   +---+       |        |=======|        |=======|        |       +---+
       ^       +--------+       +--------+       +--------+       ^
       |        Edge Node       Relay Node        Edge Node       |
       |          (T-PE)           (S-PE)          (T-PE)         |
       |                                                          |
       |<-TSN->  <------- TSN Over DetNet MPLS --------->  <-TSN->|
       |                                                          |
       |<--- Emulated Time Sensitive Networking (TSN) Service --->|

       DFx = DetNet Flow x

                    Figure 4: IEEE 802.1TSN over DetNet

   [Editor's note: TSN Over DetNet MPLS arrows extended beyond the 'X'
   (the PREF points).]

   Figure 5 illustrates how an end to end MPLS-based DetNet service is
   provided in TE LSPs.  For a more detail.  In this case, the end systems, CE1 and
   CE2, are able to send and receive DetNet flows, and unidirectional flow, R1 and
   supports PRF, R2 are
   relay nodes as they sit in the middle of a DetNet network.  For
   example, an end system sends data encapsulated in MPLS.  The 'X' in
   the end systems, supports PREOF and relay nodes represents potential DetNet flow
   packet replication R3 supports PEF and elimination points.  In this figure, POF.  Note
   that the relay nodes may change the underlying transport, forwarding sub-layer,
   for example tunneling MPLS over IP IEEE 802.1 TSN Section 11, 8, or simply
   over interconnect network segments. links.

         DetNet                                           DetNet
   MPLS  Service          Transit          Transit       Service MPLS
   DetNet  |             |<-Tnl->|        |<-Tnl->|            | DetNet
   End     |             V   1   V        V   2   V            | End
   System  |    +--------+       +--------+       +--------+   | System
   +---+   |    |   R1   |=======|   R2   |=======|   R3   |   |  +---+
   |  X...DFa...|._X_....|..DF1..|.__ ___.|..DF3..|...._X_.|.DFa..|.X |
   |CE1|========|    \   |       |   X    |       |   /    |======|CE2|
   |   |   |    |     \_.|..DF2..|._/ \__.|..DF4..|._/     |   |  |   |
   +---+        |        |=======|        |=======|        |      +---+
       ^        +--------+       +--------+       +--------+      ^
       |        Relay Node       Relay Node       Relay Node      |
       |          (S-PE)           (S-PE)           (S-PE)        |
       |                                                          |
       |<---------------------- DetNet MPLS --------------------->|
       |                                                          |
       |<--------------- End to End DetNet Service -------------->|

                    Figure 5: MPLS-Based Native

      -------------------------- Data Flow ------------------------->

       X   = Optional service protection (none, PRF, PREOF, PEF/POF)
       DFx = DetNet

   Figure 6 illustrates member flow x over a TE LSP

                    Figure 3: MPLS-Based Native DetNet

   As previously mentioned, this document specifies how an end MPLS is used to end MPLS-based
   support DetNet service is
   provided where the end systems are not able flows using an MPLS data plane as well as how such can
   be mapped to send IEEE 802.1 TSN and receive IP DetNet flows.  In PSNs.  An equally import
   scenario is when IP is supported over DetNet MPLS and this example, the nodes labeled CE1 is covered
   in [I-D.ietf-detnet-dp-sol-ip].  Another important scenario is where
   an Ethernet Layer 2 service is supported over DetNet MPLS and CE2 could
   be non-DetNet aware this is
   covered in [TBD-TSN-OVER-DETNET].

4.2.1.  IP routers or hosts.  Note that E1 and E2 are
   edge nodes as they sit boundaries of the Over DetNet enabled domain. MPLS Data Plane Scenarios

   [Author's note: this section to be moved to IP sol draft]
   IP
   Non   Service          Transit          Transit       Service Non  DetNet                |<-Tnl->|        |<-Tnl->|        Relay       Transit         Relay       IP DetNet
   End     |             V   1   V        V   2   V            | End System  |    +--------+       +--------+       +--------+   | System
   +---+   |    |   E1   |=======|   R2   |=======|   E3   |   |  +---+
   |   |--------|._X_....|..DF1..|.__ ___.|..DF3..|...._X_.|------|   |
   |CE1|   |    |    \   |       |   X    |       |   /    |   |  |CE2|
   |   |   |    |     \_.|..DF2..|._/ \__.|..DF4..|._/     |   |  |   |
   +---+        |        |=======|        |=======|        |      +---+
                +--------+       +--------+       +--------+
                ^ Edge        Node      Relay         Node       Edge Node^
                |           Node        End System
                     (T-PE)           (S-PE)       (LSR)          (T-PE)
   +----------+                                             +----------+
   |
                |                                          |
        <--IP-->| <-------- IP Over DetNet MPLS ---------> |<--IP-->
                |                                          |
                |<------   Appl.  |<------------ End to End DetNet Service ------->|

             Figure 6: MPLS-Based DetNet (non-MPLS End System)

   Figure 7 illustrates how end to end ----------->|   Appl.  |
   +----------+   .....-----+                 +-----.....   +----------+
   | Service  |<--: Service |-- DetNet service is provided where
   the end systems are able flow --| Service :-->| Service  |
   +----------+   +---------+  +----------+   +---------+   +----------+
   |Forwarding|   |Fwd| |Fwd|  |Forwarding|   |Fwd| |Fwd|   |Forwarding|
   +-------.--+   +-.-+ +-.-+  +----.---.-+   +-.-+ +-.-+   +---.------+
           :  Link  :    /  ,-----.  \   : Link :    /  ,-----.  \
           +........+    +-[  Sub  ]-+   +......+    +-[  Sub  ]-+
                           [Network]                   [Network]
                            `-----'                     `-----'

           |<- DN IP->| |<---- DetNet MPLS ---->| |< -DN IP ->|

                   Figure 4: DetNet IP Over MPLS Network

   Figure 4 illustrates DetNet enabled End Systems (hosts), connected to send and receive
   DetNet (DN) enabled IP networks, operating over a DetNet flows, e.g.,
   per [I-D.ietf-detnet-dp-sol-ip], and aware MPLS
   network.  In this figure, Relay nodes sit at the boundary of the MPLS
   domain since the non-MPLS domain is DetNet aware.  This figure is
   very similar to Figure 2.  The primary difference is that the Relay
   nodes optionally are at the edges of the MPLS domain and therefore function as
   T-PEs, and that service sub-layer functions are not provided over the
   DetNet IP network.  There is no difference in transit node function.

   Figure 5 illustrates how relay nodes can provide service protection. protection
   over the MPLS domain.  In this case R1 case, CE1 and R3 CE2 are T-PEs IP DetNet end
   systems which are interconnected via a MPLS domain such as previously
   shown in Figure 3.  Note that R1 and R2
   is R3 sit at the edges of an S-PE MPLS
   domain and therefore are similar to T-PEs, while R2 sits in the DetNet service
   middle of the domain and is end-to-end. therefore similar to an S-PE.

         DetNet                                         DetNet
   IP    Service         Transit          Transit       Service  IP
   DetNet               |<-Tnl->|        |<-Tnl->|               DetNet
   End     |            V   1   V        V   2   V            |  End
   System  |   +--------+       +--------+       +--------+   |  System
   +---+   |   |   R1   |=======|   R2   |=======|   R3   |   |   +---+
   |   |-------|._X_....|..DF1..|.__ ___.|..DF3..|...._X_.|-------|   |
   |CE1|   |   |    \   |       |   X    |       |   /    |   |   |CE2|
   |   |   |   |     \_.|..DF2..|._/ \__.|..DF4..|._/     |   |   |   |
   +---+       |        |=======|        |=======|        |       +---+
       ^       +--------+       +--------+       +--------+       ^
       |        Relay Node       Relay Node       Relay Node      |
       |          (T-PE)           (S-PE)          (T-PE)         |
       |                                                          |
       |<-DN IP-> <-------- DetNet MPLS ---------------> <-DN IP->|
       |                                                          |
       |<-------------- End to End DetNet Service --------------->|

      -------------------------- Data Flow ------------------------->

       X   = Service protection (PRF, PREOF, PEF/POF)
       DFx = DetNet member flow x over a TE LSP

               Figure 7: 5: DetNet IP over Over DetNet (DN) MPLS

4.3.  Packet flow example (service protection)

   An example MPLS DetNet network fragment and packet flow is
   illustrated in Figure 8.

      1      1.1       1.1      1.2.1    1.2.1      1.2.2
   CE1----EN1--------R1-------R2-------R3--------EN2-----CE2
            \           1.2.1 / Network

    IP               Edge                        Edge        IP
    End System       Node                        Node        End System
                    (T-PE)       (LSR)          (T-PE)
   +----------+   +....-----+                 +-----....+   +----------+
   |   Appl.  |<--:Svc Proxy|-- E2E Service --|Svc Proxy:-->|   Appl.  |
   +----------+   +.....+---+                 +---+.....+   +----------+
   |    IP    |<--:IP : |Svc|-- IP/DN Flow ---|Svc| :IP :-->|    IP    |
   +----------+   +---+ +---+  +----------+   +---+ +---+   +----------+
   |Forwarding|   |Fwd| |Fwd|  |Forwarding|   |Fwd| |Fwd|   |Forwarding|
   +-------.--+   +-.-+ +-.-+  +----.---.-+   +-.-+ +-.-+   +---.------+
           :  Link  :    /
             \1.2     /-----+  ,-----.  \   : Link :    /
              +------R4------------------------+
                        1.2.2  ,-----.  \
           +........+    +-[  Sub  ]-+   +......+    +-[  Sub  ]-+
                           [Network]                   [Network]
                            `-----'                     `-----'

         |<--- IP --->| |<----- DetNet MPLS ----->| |<--- IP --->|

          Figure 8: Example Packet flow in 6: Non-DetNet Aware IP Over DetNet Enabled MPLS Network

   In

   Figure 8 the numbers are used 6 illustrates non-DetNet enabled End Systems (hosts),
   connected to identify the instance of a
   packet.  Packet 1 is the original packet, and packets 1.1, and 1.2
   are two first generation copies of packet 1.  Packet 1.2.1 is a
   second generation copy of packet 1.2 etc.  Note that these numbers
   never appear DetNet (DN) enabled MPLS network.  It differs from
   Figure 4 in that the packet, hosts and edge IP networks are not to be confused with sequence
   numbers, labels or any other identifier that appears in the packet.
   They simply indicate DetNet aware.
   In this case, edge nodes sit at the generation number boundary of the original packet so
   that its passage through the network fragment can be identified to
   the reader.

   Customer Equipment CE1 sends a packet into the DetNet enabled MPLS
   network.  This domain since
   it is packet (1).  Edge Node EN1 encapsulates the packet
   as also a DetNet Packet domain boundary.  The edge nodes provide DetNet
   service proxies for the end applications by initiating and sends it to Relay node R1 (packet 1.1).  EN1
   makes a copy of
   terminating DetNet service for the packet (1.2), encapsulates application's IP flows.  See
   [I-D.ietf-detnet-dp-sol-ip] for more information.

   Figure 7 illustrates how it and sends this copy
   to Relay node R4.

   Note that along the MPLS path from EN1 is still possible to R1 there may be zero or
   more LSRs which, provided DetNet
   service protection for clarity, are not shown.  The same non-DetNet aware end systems.  This figures is true for
   any other path between two DetNet entities shown in
   basically the same as Figure 8.

   Relay node R4 has been configured to send one copy of 5, with the packet to
   Relay Node R2 (packet 1.2.1) exception that CE1 and one copy to Edge Node EN2 (packet
   1.2.2).

   R2 receives packet copy 1.2.1 before packet copy 1.1 arrives, and,
   having been configured to perform packet elimination on this DetNet
   flow, forwards packet 1.2.1 to Relay Node R3.  Packet copy 1.1 is of
   no further use CE2
   are non-DetNet aware end systems and so is discarded by R2.

   Edge Node EN2 receives packet copy 1.2.2 from R4 before it receives
   packet copy 1.2.1 from R2 via relay Node R3.  EN2 therefore strips
   any DetNet encapsulation from packet copy 1.2.2 E1 and forwards the
   packet to CE2.  When EN2 receives the later packet copy 1.2.1 this is
   discarded.

   The above is of course illustrative of many network scenarios that
   can be configured.  Between a pair of relay nodes there may be one or
   more transport E3 are edge nodes that simply forward
   replace the relay nodes R1 and R3.

         IP                                              IP
   Non   Service          Transit          Transit       Service Non
   DetNet traffic, but
   these are omitted for clarity.

5.                |<-Tnl->|        |<-Tnl->|              DetNet MPLS Data Considerations

   This section
   End     |             V   1   V        V   2   V            | End
   System  |    +--------+       +--------+       +--------+   | System
   +---+   |    |   E1   |=======|   R2   |=======|   E3   |   |  +---+
   |   |--------|._X_....|..DF1..|.__ ___.|..DF3..|...._X_.|------|   |
   |CE1|   |    |    \   |       |   X    |       |   /    |   |  |CE2|
   |   |   |    |     \_.|..DF2..|._/ \__.|..DF4..|._/     |   |  |   |
   +---+        |        |=======|        |=======|        |      +---+
                +--------+       +--------+       +--------+
                ^ Edge Node      Relay Node       Edge Node^
                | (T-PE)           (S-PE)          (T-PE)  |
                |                                          |
        <--IP-->| <-------- IP Over DetNet MPLS ---------> |<--IP-->
                |                                          |
                |<------ End to End DetNet Service ------->|

       X   = Optional service protection (none, PRF, PREOF, PEF/POF)
       DFx = DetNet member flow x over a TE LSP

             Figure 7: MPLS-Based DetNet (non-MPLS End System)

4.2.2.  IEEE 802.1 TSN Over DetNet MPLS Data Plane Scenario

   [Author's note: this section to be moved to TSN over mpls sol draft -
   TBD-TSN-OVER-DETNET]
    TSN             Edge          Transit        Edge        TSN
    End System      Node           Node          Node        End System
                   (T-PE)         (LSR)          (T-PE)

   +----------+  +.........+                   +.........+  +----------+
   |   Appl.  |<-:Svc Proxy:--End to End Svc.--:Svc Proxy:->|   Appl.  |
   +----------+  +---------+                   +---------+  +----------+
   |    TSN   |  |TSN| |Svc|<-- DetNet flow -->|Svc| |TSN|  |    TSN   |
   +----------+  +---+ +---+    +----------+   +---+ +---+  +----------+
   |Forwarding|  |Fwd| |Fwd|    |Forwarding|   |Fwd| |Fwd|  |Forwarding|
   +------.---+  +--.+ +-.-+    +---.----.-+   +--.+ +-.-+  +----.-----+
          :   Link  :    /  ,-----.  \   :  Link  :   /  ,-----.  \
          +.........+    +-[  Sub  ]-+   +........+   +-[  Sub  ]-+
                           [Network]                    [Network]
                            `-----'                      `-----'

           |<- TSN ->| |<------ DetNet MPLS ------>| |<-- TSN --->|

             Figure 8: A TSN over DetNet MPLS Enabled Network

   Figure 8 shows IEEE 802.1 TSN end stations operating over a TSN aware
   DetNet service running over an MPLS network.  DetNet Edge Nodes sit
   at the boundary of a DetNet domain.  They are responsible for mapping
   non-DetNet aware L2 traffic to DetNet services.  They also support
   the imposition and disposition of the required DetNet encapsulation.
   These are functionally similar to pseudowire (PW) Terminating
   Provider Edge (T-PE) nodes which use MPLS-TE LSPs.  In this example
   they understand and support IEEE 802.1 TSN and are able to map TSN
   flows into DetNet flows.  The specific of this operation are
   discussed in [TBD-TSN-OVER-DETNET].

   Native TSN flow and DetNet MPLS flow differ not only by the
   additional MPLS specific encapsulation, but DetNet MPLS flows have on
   each DetNet node an associated DetNet specific data structure, what
   defines flow related characteristics and required forwarding
   functions.  In this example, edge Nodes provide a service proxy
   function that "associates" the DetNet flows and native flows at the
   edge of the DetNet domain.  This ensures that the DN Flow is properly
   served at the Edge node (and inside the domain).

   Figure 9 illustrates how DetNet can provide services for IEEE
   802.1TSN end systems, CE1 and CE2, over a DetNet enabled MPLS
   network.  Similar to Figure 6, the edge nodes, E1 and E2, insert and
   remove required DetNet data plane encapsulation.  The 'X' in the edge
   nodes and relay node, R1, represent a potential DetNet compound flow
   packet replication and elimination point.  This conceptually
   parallels L2VPN services, and could leverage existing related
   solutions as discussed below.

        TSN    |<------- End to End DetNet Service ------>|  TSN
       Service |         Transit          Transit         | Service
   TSN  (AC)   |        |<-Tnl->|        |<-Tnl->|        |  (AC)  TSN
   End    |    V        V    1  V        V   2   V        V   |    End
   System |    +--------+       +--------+       +--------+   |  System
   +---+  |    |   E1   |=======|   R1   |=======|   E2   |   |   +---+
   |   |--|----|._X_....|..DF1..|.._ _...|..DF3..|...._X_.|---|---|   |
   |CE1|  |    |    \   |       |   X    |       |   /    |   |   |CE2|
   |   |       |     \_.|..DF2..|._/ \_..|..DF4..|._/     |       |   |
   +---+       |        |=======|        |=======|        |       +---+
       ^       +--------+       +--------+       +--------+       ^
       |        Edge Node       Relay Node        Edge Node       |
       |          (T-PE)           (S-PE)          (T-PE)         |
       |                                                          |
       |<- TSN -> <------- TSN Over DetNet MPLS -------> <- TSN ->|
       |                                                          |
       |<--- Emulated Time Sensitive Networking (TSN) Service --->|

       X   = Service protection
       DFx = DetNet member flow x over a TE LSP

                    Figure 9: IEEE 802.1TSN Over DetNet

4.3.  Packet Flow Example with Service Protection

   An example DetNet MPLS network fragment and packet flow is
   illustrated in Figure 10.

      1      1.1       1.1      1.2.1    1.2.1      1.2.2
   CE1----EN1--------R1-------R2-------R3--------EN2-----CE2
            \           1.2.1 /                   /
             \1.2     /-----+                   /
              +------R4------------------------+
                        1.2.2

       Figure 10: Example Packet Flow in DetNet Enabled MPLS Network

   In Figure 10 the numbers are used to identify the instance of a
   packet.  Packet 1 is the original packet, and packets 1.1, and 1.2
   are two first generation copies of packet 1.  Packet 1.2.1 is a
   second generation copy of packet 1.2 etc.  Note that these numbers
   never appear in the packet, and are not to be confused with sequence
   numbers, labels or any other identifier that appears in the packet.
   They simply indicate the generation number of the original packet so
   that its passage through the network fragment can be identified to
   the reader.

   Customer Equipment CE1 sends a packet into the DetNet enabled MPLS
   network.  This is packet (1).  Edge Node EN1 encapsulates the packet
   as a DetNet Packet and sends it to Relay node R1 (packet 1.1).  EN1
   makes a copy of the packet (1.2), encapsulates it and sends this copy
   to Relay node R4.

   Note that along the MPLS path from EN1 to R1 there may be zero or
   more LSRs which, for clarity, are not shown.  The same is true for
   any other path between two DetNet entities shown in Figure 10.

   Relay node R4 has been configured to send one copy of the packet to
   Relay Node R2 (packet 1.2.1) and one copy to Edge Node EN2 (packet
   1.2.2).

   R2 receives packet copy 1.2.1 before packet copy 1.1 arrives, and,
   having been configured to perform packet elimination on this DetNet
   flow, forwards packet 1.2.1 to Relay Node R3.  Packet copy 1.1 is of
   no further use and so is discarded by R2.

   Edge Node EN2 receives packet copy 1.2.2 from R4 before it receives
   packet copy 1.2.1 from R2 via relay Node R3.  EN2 therefore strips
   any DetNet encapsulation from packet copy 1.2.2 and forwards the
   packet to CE2.  When EN2 receives the later packet copy 1.2.1 this is
   discarded.

   The above is of course illustrative of many network scenarios that
   can be configured.  Between a pair of relay nodes there may be one or
   more transit nodes that simply forward the DetNet traffic, but these
   are omitted for clarity.

5.  DetNet MPLS Data Plane Considerations

   This section provides informative considerations related to providing providing
   DetNet service to flows which are identified based on their header
   information.  At a high level, the following are provided on a per
   flow basis:

   Eliminating contention loss and jitter reduction:

      Use of allocated resources (queuing, policing, shaping) to ensure
      that the congestion-related loss and latency/jitter requirements
      of a DetNet flow are met.

   Explicit routes:

      Use of a specific path for a flow.  This limits misordering and
      bounds latency.

   Service protection:

      Which in the case of this document primarily relates to
      replication and elimination.  Changing the explicit path after a
      failure is detected in order to restore delivery of the required
      DetNet service characteristics is also possible.  Path changes,
      even in the case of failure recovery, can lead to the out of order
      delivery of data.

   Load sharing:

      Generally, distributing packets of the same DetNet flow over
      multiple paths is not recommended.  Such load sharing, e.g., via
      ECMP or UCMP, impacts ordering and possibly jitter.

   Troubleshooting:

      For example, to support identification of misbehaving flows.

   Recognize flow(s) for analytics:

      For example, increase counters.

   Correlate events with flows:

      For example, unexpected loss.

   The DetNet data plane also allows for the aggregation of DetNet
   flows, e.g., via MPLS hierarchical LSPs, to improved scaling.  When
   DetNet flows are aggregated, transit nodes provide service to the
   aggregate and not on a per-DetNet flow basis.  In this case, nodes
   performing aggregation will ensure that per-flow service requirements
   are achieved.

5.1.  End-System Specific Considerations

   Data-flows requiring DetNet service are generated and terminated on
   end-systems.  Encapsulation depends on application and its
   preferences.  In a DetNet MPLS domain the DN functions use the d-CWs,
   S-Labels and F-Labels to provide DetNet services.  However, an
   application may exchange further flow related parameters (e.g., time-
   stamp), which are not provided by DN functions.

   Specifics related to non-MPLS DetNet end station behavior are out
   side the scope of this document.  For example, details on support for
   DetNet IP data flows can be found in [I-D.ietf-detnet-dp-sol-ip].
   This document is also useful for end stations that map IP flows to
   DetNet flows.

   As a general rule, DetNet MPLS domains are capable of forwarding any
   DetNet MPLS flows and the DetNet domain does not mandate the end-
   system or edge system encapsulation format.  Unless there is a proxy
   of some form present, end-systems peer with similar end-systems using
   the same application encapsulation format.  For example, as shown in
   Figure 11, IP applications peer with IP applications and Ethernet
   L2VPN applications peer with Ethernet L2VPN applications.

             +-----+
             |  X  |                               +-----+
             +-----+                               |  X  |
             | Eth |               ________        +-----+
             +-----+     _____    /        \       | Eth |
                    \   /     \__/          \___   +-----+
                     \ /                        \ /
                      0======== tunnel-1 ========0_
                      |                             \
                       \                             |
                       0========= tunnel-2 =========0
                      / \                        __/ \
               +-----+   \__ DetNet MPLS domain /     \
               |  X  |      \         __       /       +-----+
               +-----+       \_______/  \_____/        |  X  |
               |  IP |                                 +-----+
               +-----+                                 |  IP |
                                                       +-----+

             Figure 11: End-Systems and The DetNet MPLS Domain

5.2.  Sub-Network Considerations

   As shown in Figure 2, MPLS nodes are interconnected by different sub-
   network technologies, which may include point-to-point links.  Each
   of these need to provide appropriate service to DetNet flows.  In
   some cases, e.g., on dedicated point-to-point links or TDM
   technologies, all that is required is for a DetNet node to
   appropriately queue its output traffic.  In other cases, DetNet nodes
   will need to map DetNet flows to the flow semantics (i.e.,
   identifiers) and mechanisms used by an underlying sub-network
   technology.  Figure 12 shows several examples of header formats that
   can be used to carry DetNet MPLS flows over different sub-network
   technologies.  L2 represent a generic layer-2 encapsulation that
   might be used on a point-to-point link.  TSN represents the
   encapsulation used on an IEEE 802.1 TSN network, as described in
   Section 8.  UDP/IP represents the encapsulation used on a DetNet IP
   PSN, as described in Section 9 .

                              +------+  +------+  +------+
           App-Flow           |  X   |  |  X   |  |  X   |
                        +-----+======+--+======+--+======+-----+
           DetNet-MPLS        | d-CW |  | d-CW |  | d-CW |
                              +------+  +------+  +------+
                              |Labels|  |Labels|  |Labels|
                        +-----+======+--+======+--+======+-----+
           Sub-Network        |  L2  |  | TSN  |  | UDP  |
                              +------+  +------+  +------+
                                                  |  IP  |
                                                  +------+
                                                  |  L2  |
                                                  +------+

            Figure 12: Example DetNet MPLS Sub-Network Formats

6.  MPLS-Based DetNet Data Plane Solution

6.1.  DetNet Over MPLS Encapsulation Components

   To carry DetNet over MPLS the following is required:

   1.  A method of identifying the MPLS payload type.

   2.  A method of identifying the DetNet flow group to the processing
       element.

   3.  A method of distinguishing DetNet OAM packets from DetNet data
       packets.

   4.  A method of carrying the DetNet sequence number.

   5.  A suitable LSP to deliver the packet to the egress PE.

   6.  A method of carrying queuing and forwarding indication.

   In this design an MPLS service label (the S-Label), similar to flows a
   pseudowire (PW) label [RFC3985], is used to identify both the DetNet
   flow identity and the payload MPLS payload type satisfying (1) and
   (2) in the list above.  OAM traffic discrimination happens through
   the use of the Associated Channel method described in [RFC4385].  The
   DetNet sequence number is carried in the DetNet Control word which
   carries the Data/OAM discriminator.  To simplify implementation and
   to maximize interoperability two sequence number sizes are identified based on their header
   information.  At supported:
   a 16 bit sequence number and a 28 bit sequence number.  The 16 bit
   sequence number is needed to support some types of legacy clients.
   The 28 bit sequence number is used in situations where it is
   necessary ensure that in high level, speed networks the following sequence number
   space does not wrap whilst packets are provided on a per
   flow basis:

   Congestion protection and latency control:

      Usage in flight.

   The LSP used to forward the DetNet packet may be of allocated resources (queuing, policing, shaping) any type (MPLS-
   LDP, MPLS-TE, MPLS-TP [RFC5921], or MPLS-SR
   [I-D.ietf-spring-segment-routing-mpls]).  The LSP (F-Label) label
   and/or the S-Label may be used to
      ensure indicate the queue processing as
   well as the forwarding parameters.  Note that the congestion-related loss and latency/jitter
      requirements possible use of
   Penultimate Hop Popping (PHP) means that the only label in a received
   label stack may be the S-Label.

6.2.  MPLS Data Plane Encapsulation

   Figure 13 illustrates a DetNet flow are met.

   Explicit routes:

      Use data plane MPLS encapsulation.  The
   MPLS-based encapsulation of the DetNet flows is a specific path good fit for a flow.  This limits miss-ordering and
      latency.

   Service protection:

      Which in the case of this document primarily relates
   scenarios described in Section 4.2.1 and Section 4.2.2.  Furthermore,
   end to end DetNet service i.e., native DetNet deployment (see
   Section 4.2) is also possible if DetNet end systems are capable of
   initiating and termination MPLS encapsulated packets.

   The MPLS-based DetNet data plane encapsulation consists of:

   o  DetNet control word (d-CW) containing sequencing information for
      packet replication and elimination.  Changing duplicate elimination purposes, and the explicit path after OAM
      indicator.

   o  DetNet service Label (S-Label) that identifies a
      failure is detected in order to restore delivery of DetNet flow at
      the required receiving DetNet service characteristics sub-layer processing node.

   o  Zero or more Detnet MPLS Forwarding label(s) (F-Label) used to
      direct the packet along the label switched path (LSP) to the next
      service sub-layer processing node along the path.  When
      Penultimate Hop Popping is also possible.  Path changes,
      even in use there may be no label F-Label in
      the case of failure recovery, can lead to the out of order
      delivery of data.

   Load sharing:

      Generally, distributing packets of protocol stack on the same DetNet flow over
      multiple paths final hop.

   o  The necessary data-link encapsulation is not recommended.  Such load sharing, e.g., via
      ECMP or UCMP, impacts ordering and end-to-end jitter.

   Troubleshooting:

      For example, then applied prior to support identification of misbehaving flows.

   Recognize flow(s) for analytics:

      For example, increase counters.

   Correlate events with flows:

      For example, unexpected loss.

   The
      transmission over the physical media.

        DetNet MPLS-based encapsulation

      +---------------------------------+
      |                                 |
      |         DetNet App-Flow         |
      |         Payload  Packet         |
      |                                 |
      +---------------------------------+ <--\
      |       DetNet Control Word       |    |
      +---------------------------------+    +--> DetNet data plane also allows for the aggregation
      |           S-Label               |    |    MPLS encapsulation
      +---------------------------------+    |
      |           F-Label(s)            |    |
      +---------------------------------+ <--/
      |           Data-Link             |
      +---------------------------------+
      |           Physical              |
      +---------------------------------+

     Figure 13: Encapsulation of a DetNet
   flows, e.g., via MPLS hierarchical LSPs, App-Flow in an MPLS(-TP) PSN

6.2.1.  DetNet Control Word and the DetNet Sequence Number

   A DetNet control word (d-CW) conforms to improved scaling.  When the Generic PW MPLS Control
   Word (PWMCW) defined in [RFC4385].  The d-CW formatted as shown in
   Figure 14 MUST be present in all DetNet flows are aggregated, transit nodes may have limited ability packets containing app-flow
   data.

      0                   1                   2                   3
      0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |0 0 0 0|                Sequence Number                        |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                      Figure 14: DetNet Control Word

   (bits 0 to provide service on per-flow 3)

      Per [RFC4385], MUST be set to zero (0).

   Sequence Number (bits 4 to 31)

      An unsigned value implementing the DetNet identifiers.  Therefore,
   identifying sequence number.

   A separate sequence number space MUST be maintained by the the node
   that adds the d-CW for each individual DetNet flow on a transit node may not app-flow.  The following sequence
   number field lengths MUST be
   achieved in some network scenarios, but DetNet service can still supported:

      0 bits

      16 bits

      28 bits

   The sequence number length MUST be
   assured in these scenarios through resource allocation and control.

5.1.  End-system specific considerations

   Data-flows requiring DetNet service are generated and terminated on
   end-systems.  Encapsulation depends provisioned (configured) on application and its
   preferences.  In a DetNet (or even a TSN) domain per
   app-flow basis via configuration, e.g., the DN (TSN)
   functions use at most two flow parameters, namely Flow-ID and
   Sequence Number.  However, an application may exchange further flow
   related parameters (e.g., time-stamp), which are not considered by DN
   functions.

   Two types of end-systems are distinguished:

   o  L2 (Ethernet) end-system: application directly over L2.

   o  L3 (IP) end-system: application over L3.

   Note: An MPLS DetNet end system (as shown controller plane
   described in Figure 5) can be treated
   as a combination of an L3 (IP) end-system and an MPLS DetNet edge
   node.

   In case of Ethernet end-systems the application data Section 7.

   A 0 bit sequence number field length indicates that there is encapsulated
   directly in L2.  From the DN domain perspective no upper layer
   protocols are visible.  The Data-flow uses only Ethernet tag(s) and
   further flow specific parameters (if needed) are hidden inside
   DetNet sequence number used for the
   protocol data unit (PDU).

   The IP end-system scenario is different.  In this case, data-flows
   are encapsulated directly in IP and, typically, other higher layer
   protocols such as UDP and Real-time Transport Protocol (RTP).  Many
   valid combinations exist and it is up to applications to select
   specific headers to flow.  When the length is zero,
   the sequence number field MUST be used.  Details set to zero (0) on support all packets sent
   for the flow.

   When the sequence number field length is 16 or 28 bits for DetNet IP data
   flows can be found in [I-D.ietf-detnet-dp-sol-ip].

   As a general rule, DetNet domains flow,
   the sequence number MUST be capable of forwarding any
   Data-flows and incremented by one for each new app-flow
   packet sent.  When the DetNet domain field length is 16 bits, d-CW bits 4 to 15
   MUST NOT mandate the end-system
   encapsulation format.

   Furthermore, no application-level-proxy function be set to zero (0).  This values carried in this field can wrap
   and it is envisioned inside
   the DetNet domain, so end-systems peer with end-systems using important to note that zero (0) is a valid field value.
   For example, were the
   same application encapsulation format (see figure below):

   o  L2 end-systems peer with L2 end-systems and

   o  L3 end-systems peer with L3 end-systems.

             +-----+
             |  X  |                               +-----+
             +-----+                               |  X  |
             | Eth |               ________        +-----+
             +-----+     _____    /        \       | Eth |
                    \   /     \__/          \___   +-----+
                     \ /                        \ /
                      0======== tunnel-1 ========0_
                      |                             \
                       \                             |
                       0========= tunnel-2 =========0
                      / \                        __/ \
               +-----+   \__     DetNet domain  /     \
               |  X  |      \         __       /       +-----+
               +-----+       \_______/  \_____/        |  X  |
               |  IP |                                 +-----+
               +-----+                                 |  IP |
                                                       +-----+

                Figure 9: End-systems and sequence number size is 16 bits, the DetNet domain

5.2.  DetNet domain specific considerations

   From a connection type perspective, three scenarios are
   distinguished: sequence
   will contain: 65535, 0, 1.  Directly attached: end-system  In this case, zero (0) is an ordinary
   sequence number.  This differs from [RFC4448] where a sequence number
   of zero (0) does not indicate that no sequence number field value is directly connected to an edge
       node.

   2.  Indirectly attached: end-system
   in use.

   The sequence number is behind optionally used during receive processing as
   described below in Section 6.2.2.1 and Section 6.2.2.2.

6.2.2.  S-Labels

   App-flow identification at a (L2-TSN / L3-DetNet)
       sub-network.

   3.  DN integrated: end-system DetNet service sub-layer is part of realized by
   an S-Label.  Each app-flow MUST be sent by the node that adds a d-CW
   with a single specific S-Label value.  MPLS-aware DetNet domain.

   L3 end-systems may use any of these connection types, however L2 end- end systems may use only the first two (directly or indirectly attached).
   DetNet domain
   and edge nodes, which are by definition MPLS ingress and egress
   nodes, MUST allow communication between any end-systems of the
   same type (L2-L2, L3-L3), independent of their connection type add and
   DetNet capability.  However directly attached remove the d-CW and indirectly attached
   end-systems have no knowledge about S-Label.  Relay nodes MAY
   swap S-Label values when processing an app-flow.

   The S-Label value MUST be provisioned per app-flow via configuration,
   e.g., via the controller plane described in Section 7.  Note that
   S-Labels provide app-flow identification at the downstream DetNet domain
   service sub-layer receiver, not the sender.  As such, S-Labels MUST
   be allocated by the entity that controls the service sub-layer
   receiving node's label space, and its
   encapsulation format MAY be allocated from the platform
   label space [RFC3031].

   The S-Label will normally be at all.  See Figure 10 for L3 end-system
   scenarios.

                                               ____+----+
                       +----+        _____    /    | ES3|
                       | ES1|____   /     \__/     +----+___
                       +----+    \ /                        \
                                  +                          |
                          ____     \                        _/
            +----+     __/    \     +__    DetNet domain   /
            | ES2|____/  L2/L3 |___/   \         __     __/
            +----+    \_______/         \_______/  \___/

               Figure 10: Connection types the bottom of L3 end-systems

5.2.1.  DetNet Layer Two Service

   The simplest DetNet service is to provide tunneling for layer two,
   where the connected hosts are label stack,
   immediately preceding the d-CW.  To support service sub-layer level
   OAM, an OAM Associated Channel Header (ACH) [RFC4385] together with a
   Generic Associated Channel Label (GAL) [RFC5586] MAY be used in place
   of a d-CW.

   Similarly, an Entropy Label Indicator/Entropy Label (ELI/EL)
   [RFC6790] MAY be carried below the same broadcast (BC) domain.
   Forwarding over S-Label in the label stack in
   networks where DetNet domain is based on L2 (MAC) addresses
   (i.e. dst-MAC), or on flows would otherwise received interface [RFC3985].  In both cases ECMP treatment.
   When ELs are used, the L2 headers MUST either be kept, or provision must same EL value SHOULD be made used for
   their reconstruction at egress from all of the DetNet domain.

                                   +------+
                                   |  X   |
                          +------+ +------+
                          |  X   | |  IP  |
                          +------+ +------+
              End-system  |  L2  | |  L2  |
                    +-----+======+-+======+--+------+
              DetNet tunnel                  | Shim |
                                             +------+
                                             | MPLS |
                                             +------+
                                             |  L2  |
                                             +------+

              Examples:

                                    +------+  +------+
                                    |  X   |  |  X   |
                          +------+  +------+  +------+
                          |  X   |  |  IP  |  |  IP  |
                          +------+  +------+  +------+
                          |  L2  |  |  L2  |  |  L2  |
                    +-----+======+--+======+--+======+-----+
                          | d-CW |  | d-CW |  | d-CW |
                          +------+  +------+  +------+
                          | MPLS |  | MPLS |  | MPLS |
                          +------+  +------+  +------+
                          |  L2  |  |  L2  |  | UDP  |
                          +------+  +------+  +------+
                                              |  IP  |
                                              +------+
                                              |  L2  |
                                              +------+

       Figure 11: Encapsulation format for DetNet Layer Two Service

   As shown in Figure 11 both L2 and L3 end-systems can be served by
   such
   packets sent using a specific S-Label to force the flow to follow the
   same path.  However, as previously stated in Section 5, the use of
   ECMP for DetNet L2 encapsulation service.  This encapsulation service
   may flows is NOT RECOMMENDED.  ECMP MAY be carried over MPLS natively Section 6.2, or over MPLS over IP
   Section 11.

5.2.2. used for non-
   DetNet Routing Service (IP over MPLS)

   IP traffic and IP flows within a DetNet flows, see [I-D.ietf-detnet-dp-sol-ip], can
   be carried over domain.

   When receiving a DetNet MPLS domain. flow, an implementation MUST identify
   the app-flow associated with the incoming packet based on the
   S-Label.  When a node is using platform labels for S-Labels, no
   additional information is needed as the S-label uniquely identifies
   the app-flow.  In such cases, the IP headers case where platform labels are modified per standard router behavior, e.g., TTL handling.

   Figure 12 shows not used, one or
   more F-Labels proceeding the encapsulation of S-Label MUST be used together with the
   S-Label to uniquely identify the incoming app-flows.  When PHP is
   used, the incoming interface MAY also be used to together with any
   present F-Label(s) and the S-Label to uniquely identify an IP flow over MPLS as well as
   when MPLS incoming
   app-flows.  Note that choice to use platform label space for S-Label
   or S-Label plus one or more F-Labels to identify app flows is carried over a local
   implementation choice, with one caveat.  When one or more F-labels,
   or incoming interface, is needed together with an IP PSN, see Section 11.

                               +------+
                               |  X   |
                               +------+
                   End-system  |  IP  |
                         +-----+======+--+------+ S-Label to uniquely
   identify, the controller plane MUST ensure that incoming DetNet tunnel         | Shim |
                                         +------+
                                         | MPLS |
                                         +------+
                                         |  L2  |
                                         +------+

                   Examples:

                               +------+  +------+
                               |  X   |  |  X   |
                               +------+  +------+
                               |  IP  |  |  IP  |
                         +-----+======+--+======+-----+
                               | d-CW |  | d-CW |
                               +------+  +------+
                               | MPLS |  | MPLS |
                               +------+  +------+
                               |  L2  |  | UDP  |
                               +------+  +------+
                                         |  IP  |
                                         +------+
                                         |  L2  |
                                         +------+

   Figure 12: Encapsulation format
   packets arrive with the needed information (F-label(s) and/or
   incoming interface); the details of such are outside the scope of
   this document.

   While NOT REQUIRED, the use of platform labels for DetNet Routing S-Labels matches
   other pseudowire encapsulations.  This implementation choice also
   impacts PEF and POF processing as described in MPLS PSN for L3
                                end-systems

5.3.  DetNet Inter-Working the next section.

6.2.2.1.  Packet Elimination Function (DN-IWF)

5.3.1.  Networks with multiple technology segments

   There are networking scenarios, where Processing

   Implementations MAY support the Packet Elimination Function (PEF) for
   received DetNet domain contains
   multiple technology segments (IP, MPLS, ..) and all those segments
   are under MPLS flows.  When supported, use of the same administrative control (see Figure 13).
   Furthermore, DetNet nodes may PEF for a
   particular app-flow MUST be interconnected provisioned via TSN segments.

   An important aspect of configuration, e.g., via
   the controller plane described in Section 7.

   After an app-flow is identified for a received DetNet network design MPLS packet, as
   described above, an implementation MUST check if PEF is configured
   for that app-flow.  When configured the placement of
   DetNet functions across implementation MUST track the domain.  Designs based on segment-by-
   segment optimization can provide only sub-optimal solutions.  In
   order to achieve global optimized Inter-Working Functions (DN-IWF)
   can be placed at segment edge nodes, which stitch together DetNet
   flows across connected segments.

   DN-IWF may
   sequence number contained in received d-CWs and MUST ensure that flow attributes are correlated across segment
   edges.  For example, there are two DetNet functions which require
   Sequence Numbers: (1) PEF: removes duplications from flows and (2)
   POF: ensures in-order-delivery
   duplicate (replicated) instances of packet in a flow.  Stitching flows
   together and correlating attributes means particular sequence number are
   discarded.  The specific mechanisms used for example that
   replication of an implementation to
   identify which received packets can happen in one segment and elimination of are duplicates in a different one.

                                      ______
                            ____     /      \__
               ____        /     \__/          \___   ______
   +----+   __/    +======+                       +==+      \     +----+
   |src |__/  Seg1  )     |                       |  \  Seg3 \____| dst|
   +----+  \_______+      \       Segment-2       |   \+_____/    +----+
                    \======+_                    _+===/
                             \         __     __/
                              \_______/  \___/

                                          +------------+
                +---------------E----+    |            |
   +----+       |               |    +----R---+        |          +----+
   |src |-------R           +---+             |        E----------+ dst|
   +----+       |           |                 E--------+          +----+
                +-----------R                 |
                            +-----------------+

      Figure 13: Optimal replication and elimination placement across
                        technology segments example

5.3.2.  DN-IWF related considerations

   The goal of DN-IWF which are new is to (1) match
   an implementation choice.  Note that per Section 6.2.1 the sequence
   number field length may be 16 or 28 bits, and (2) translate segment specific
   flow attributes.  The DN-IWF ensures that segment specific attributes
   comprise per domain unique attributes for the whole DetNet domain.
   This characteristic field value can ensure
   wrap.

   Note that DetNet functions can be based an implementation MAY wish to constrain the maximum number
   sequence numbers that are tracked, on platform-wide or per domain attributes and not per segment attributes.

   The two DetNet specific attributes have flow
   basis.  Some implementations MAY support the following
   characteristics:

   o  Flow-ID: it is same in all packets provisioning of the
   maximum number sequence numbers that are tracked number on either a
   platform-wide or per flow

   o  Sequence Number: it basis.

6.2.2.2.  Packet Ordering Function Processing

   A function that is related to PEF is different packet-by-packet

   For the Flow-ID Packet Ordering Function
   (POF).  Implementations MAY support POF.  When supported, use of the DN-IWF can implement
   POF for a static mapping.  The
   situation particular app-flow MUST be provisioned via configuration,
   e.g., via the controller plane described by Section 7.
   Implementations MAY required that PEF and POF be used in combination.
   There is more complicated no requirement related to the order of execution of the
   Packet Elimination and Ordering Functions in an implementation.

   After an app-flow is identified for Sequence Number a received DetNet MPLS packet, as it is different
   packet-by-packet, so it may need more sophisticated translation
   unless its format
   described above, an implementation MUST check if POF is exactly the same in the two technology segments.
   In this later case configured
   for that app-flow.  When configured the DN-IWF can simple copy implementation MUST track the Sequence Number
   field between
   sequence number contained in received d-CWs and MUST ensure that
   packets are processed in the tunneling encapsulation of order indicated in the two technology
   segments.

   In case of three technology segments (IP, MPLS and TSN) three DN-IWF
   functions can received d-CW
   sequence number field, which may not be specified.  In in the rest of this section order the packets are
   received.  As defined in Section 6.2.1 the focus sequence number field
   length may be 16 or 28 bits, is
   on the incremened by one (1) IP - MPLS network scenario.  Note: for each new
   app-flow packet sent, and the use-cases are out-
   of-scope field value can wrap.  The specific
   mechanisms used for (2) TSN - IP, (3) TSN - MPLS.

   Simplest an implementation of DN-IWF is provided if the flow attributes
   have to identify the same format.  Such a common denominator order of the tunnel
   encapsulation format is the pseudowire encapsulation over both IP and
   MPLS.

                                 +--------+
                                 |        |
                                 | X    X |
                                 |  ____  |
                                 | /    \ |
                                 |        |
                                 |/\/\/\/\|

                                    Oops!
                               404 Not Found

                  Figure 14: FIGURE Placeholder PW over X

   [Editor's note: Where
   received packets is the text describing how 802.1 TSN Streams
   are mapping an implementation choice.

   Note that an implementation MAY wish to DetNet services/flows. i.e., EVPN+]

6.  MPLS-based DetNet data plane solution

6.1.  DetNet over MPLS Encapsulation Components

   To carry DetNet over MPLS constrain the following is required:

   1.  A method maximum number
   of identifying the MPLS payload type.

   2.  A method out of identifying the DetNet order packets that can be processed, on platform-wide or
   per flow group to basis.  Some implementations MAY support the processing
       element.

   3.  A method provisioning of distinguishing DetNet OAM
   this number on either a platform-wide or per flow basis.  The number
   of out of order packets from DetNet data
       packets.

   4.  A method that can be processed also impacts the
   latency of carrying a flow.

6.2.3.  F-Labels

   F-Labels are support the DetNet sequence number.

   5.  A suitable LSP to deliver the packet to the egress PE.

   6.  A method of carrying queuing and forwarding indication.

   In this design an MPLS service label (the S-Label), similar to a
   pseudowire (PW) label [RFC3985], is sub-layer.  F-Labels are
   used to identify both the provide LSP-based connectivity between DetNet
   flow identity service sub-
   layer processing nodes.

6.2.3.1.  Service Sub-Layer and the payload Packet Replication Function Processing

   DetNet MPLS payload type satisfying (1) end systems, edge nodes and
   (2) in the list above.  OAM traffic discrimination happens through relay nodes may operate at
   the use DetNet service sub-layer with understand of app-flows and their
   requirements.  As mentioned above, when operating at this layer such
   nodes can push, pop or swap (pop then push) S-Labels.  In all cases,
   the Associated Channel method described in [RFC4385].  The
   sequence number is carried in F-Labels used for the app-flow are always replaced and this
   section applies.

   When sending a DetNet Control word which carries flow, Zero or more F-Labels MAY be added on top
   of an S-Label by the Data/OAM discriminator. node pushing an S-Label.  The LSP used F-Labels to transport the DetNet
   packet may be
   pushed when sending a particular app-flow MUST be provisioned per
   app-flow via configuration, e.g., via the controller plane discussed
   in Section 7.  To allow for the omission of any type (MPLS-LDP, MPLS-TE, MPLS-TP [RFC5921], or
   MPLS-SR [I-D.ietf-spring-segment-routing-mpls]). F-Labels, an
   implementation SHOULD also allow an outgoing interface to be
   provisioned.

   The LSP (T-Label)
   label and/or the S-Label may Packet Replication Function (PRF) function MAY be used to indicate the queue processing
   as well as supported by an
   implementation for outgoing DetNet flows.  When supported, the same
   app-flow data will be sent over multiple outgoing forwarding parameters. sub-
   layer LSPs.  To simplify support PRF an implementation and MUST support the
   setting of different sets of F-Labels.  Hereto, to maximize interoperability two
   sequence number sizes are supported: allow for the
   omission of F-Labels, an implementation SHOULD also allow multiple
   outgoing interfaces to be provisioned.  PRF MUST NOT be used with
   app-flows configured with a 16 bit d-CW sequence number and field length of 0
   bits.

   When a
   28 bit sequence number.  The 16 bit sequence number single set of F-Labels is needed to
   support some types provisioned for a particular
   outgoing app-flow, that set of legacy clients. F-labels MUST be pushed after the
   S-Label is pushed.  The 28 bit sequence number outgoing packet is
   used then forwarded as
   described below in situations where it Section 6.2.3.2.  When a single outgoing interface
   is necessary ensure that in high speed
   networks provisioned, the sequence number space does not wrap whilst packets are outgoing packet is then forwarded as described
   below in flight.  In addition it must be possible to send Section 6.2.3.2.

   When multiple sets of F-Labels or interfaces are provisioned for a packet with
   particular outgoing app-flow, a
   zero length sequence number, to support copy of the case where sequence
   numbers outgoing packet,
   including the pushed S-Label, MUST be made per F-label set and
   outgoing interface.  Each set of provisioned F-Labels are not required by then pushed
   onto a copy of the packet.  Each copy is then forwarded as described
   below in Section 6.2.3.2.

   As described in the previous section, when receiving a particular DetNet flow.

   Note that MPLS
   flow, an implementation identifies the concept of app-flow associated with the
   incoming packet based on the S-Label.  When a zero length sequence number node is not to using platform
   labels for S-Labels, any F-Labels can be
   confused with a sequence number of zero.  For example, were popped and the
   sequence number size is 16 bits, S-label
   uniquely identifies the sequence will contain: 65535, 0,
   1. app-flow.  In this the case zero is an ordinary sequence number.  Unlike
   [RFC4448] a sequence number of zero does not indicate that no
   sequence number is in use.  Where sequence numbers where platform labels
   are not in use,
   and thus a zero length sequence number used, F-Label(s) MUST also be provisioned for incoming app-
   flows.  When PHP is used, incoming interface MUST be provisioned.
   The provisioned information MUST then be used to identify incoming
   app-flows based on the combination of S-Label and F-Label(s) or
   incoming interface.

6.2.3.2.  Common F-Label Processing

   All DetNet aware MPLS nodes process F-Labels as needed to meet the
   service requirements of the DetNet flow or flows carried in used, the sequence
   number field in LSPs
   represented by the packet F-Labels.  This includes normal push, pop and swap
   operations.  Such processing is sent as zero.  The DetNet packet
   forwarder knows which essentially the same type of these cases applies through configuration
   parameters associated with each
   processing enabled for TE LSPs, although the specific DetNet flow.

   Note that when service
   parameters, or traffic specification, can differ.  When the network consists only of DetNet enabled nodes with
   no aggregation, Penultimate Hop Popping (PHP) means that
   service parameters of the only
   label app-flow or flows carried in the label stack may an LSP
   represented by an F-Label can be met by an exiting TE mechanism, the S-label.

6.2.  MPLS data plane encapsulation

   Figure 15 illustrates
   forwarding sub-layer processing node MAY be a DetNet data plane unaware, i.e.,
   standard, MPLS encapsulation.  The
   MPLS-based encapsulation of LSR.  Such TE LSPs may provide LSP forwarding service
   as defined in, but not limited to, [RFC3209], [RFC3270], [RFC3272],
   [RFC3473], [RFC4875], [RFC5440], and [RFC6006].

   More specifically, as mentioned above, the DetNet flows forwarding sub-
   layer provides explicit routes and allocated resources, and F-Labels
   are used to map to each.  Explicit routes are supported based on the
   topmost (outermost) F-Label that is pushed or swapped and the LSP
   that corresponds to this label.  This topmost (outgoing) label MUST
   be associated with a good fit provisioned outgoing interface and, for non-
   point-to-point outgoing interfaces, a next hop LSR.  Note that this
   information MUST be provisioned via configuration or the
   Layer-2 interconnect deployment cases (see Figure 4).  Furthermore,
   end to end DetNet service i.e., native DetNet deployment (see
   Figure 5) controller
   plane.  In the previously mentioned special case where there is also possible if DetNet end systems are capable of
   initiating and termination MPLS encapsulated packets.

   The MPLS-based DetNet data plane encapsulation consists of:

   o  DetNet control word (d-CW) containing sequencing information for
      packet replication and duplicate elimination purposes, no
   added F-labels and the OAM
      indicator.  There outgoing interface is not a point-to-point
   interface, the outgoing interface MUST also be associated with a separate sequence number space for next
   hop LSR.

   Resources may be allocated in a hierarchical fashion per LSP that is
   represented by each DetNet flow.

   o  DetNet F-Label.  Each LSP MAY be provisioned with a
   service Label (S-label) parameters that identifies a DetNet flow to dictates the peer node that is specific traffic treatment to process it.  The S-Label is allocated
      from be
   received by the platform label space [RFC3031].

   o  Zero or more MPLS transport traffic carried over that LSP.  Implementations of
   this document MUST ensure that traffic carried over each LSP label(s) (T-label)
   represented by an F-Label receives the traffic treatment provisioned
   for that LSP.  Typical mechanisms used to direct
      the packet along the label switched path (LSP) provide different treatment
   to different flows includes the next peer
      node along the path.  When Penultimate Hop Popping is in use there
      may allocation of system resources (such
   as queues and buffers) and provisioning or related parameters (such
   as shaping, and policing).  Support can also be no label T-label provided via an
   underlying network technology such IEEE802.1 TSN Section 8.  Other
   than in the protocol stack on the final hop.

   o  The necessary data-link encapsulation is then applied prior to
      transmission over TSN case, the physical media.

        DetNet MPLS-based encapsulation

      +---------------------------------+
      |                                 |
      | specific mechanisms used by a DetNet Flow           |
      |         Payload  Packet         |
      |                                 |
      +---------------------------------+ <--\
      | node
   to ensure DetNet Control Word       |    |
      +---------------------------------+    +--> service delivery requirements are met for supported
   DetNet data plane
      |           S-Label               |    |    MPLS encapsulation
      +---------------------------------+ <--/
      |           T-Label(s)            |
      +---------------------------------+
      |           Data-Link             |
      +---------------------------------+
      |           Physical              |
      +---------------------------------+

       Figure 15: Encapsulation flows is outside the scope of this document.

   Packets that are marked in a way that does not correspond to
   allocated resources, e.g., lack a provisioned F-Label, can disrupt
   the QoS offered to properly reserved DetNet flows by using resources
   allocated to the reserved flows.  Therefore, the network nodes of a
   DetNet flow in an MPLS(-TP) PSN

6.3. network:

   o  MUST defend the DetNet control word

   A QoS by discarding or remarking (to an
      allocated DetNet control word (d-CW) conforms to flow or non-competing non-DetNet flow) packets
      received that are not the Generic PW MPLS Control
   Word (PWMCW) defined in [RFC4385] and is illustrated in Figure 16.
   The upper nibble subject of the d-CW a completed resource
      allocation.

   o  MUST be set to zero (0).  Two sequence
   number sizes are supported: 16 bits and 28 bits.  The sequence number
   size in NOT use for the d-CW associated with a DetNet flow (S-Label) is
   configured either by allocated resource, e.g. a control plane queue or manually shaper
      reserved for each DetNet
   flow.  The sequence number is aligned to flows, for any packet that does match the right (least significant
   bits) and unused bits MUST be set to zero (0).  Each
      corresponding DetNet flow flow.

   o  MUST
   have ensure a QoS flow does not exceed its own sequence number counter.  The sequence number allocated resources or
      provisioned service level,

   o  MUST ensure a CoS flow or service class does not impact the
      service delivered to other flows.  This requirement is
   incremented by one similar to
      requirement for each new packet.

   As discussed in Section 6, zero is an ordinary sequence number with
   no special meaning.  Also as discussed therein, where no sequence
   number is used by a particular DetNet flow, MPLS LSRs to that CoS LSPs do not impact the
      resources allocated to TE LSPs, e.g., via [RFC3473].

   Subsequent sections provide additional considerations related to CoS
   (Section 6.6.1), QoS (Section 6.6.2) and aggregation (Section 6.6.3).

6.3.  OAM Indication

   OAM follows the sequence number field procedures set out in [RFC5085] with the d-CW restriction
   that only Virtual Circuit Connectivity Verification (VCCV) type 1 is set to zero.

   The d-CW MUST always be present
   supported.

   As shown in a packet.  In a case where Figure 3 of [RFC5085] when the
   sequence number is not used (e.g., for DetNet-t-flows) a zero length
   sequence number first nibble of the d-CW
   is used and 0x0 the sequence number MUST be set to zero
   (0).

      0                   1                   2                   3
      0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |0 0 0 0|                Sequence Number                        |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                      Figure 16: DetNet Control Word

6.4.  Flow Identification

   DetNet flow identification at a DetNet service layer payload following the d-CW is realized by
   an S-label.  The S-label normal user data.  However,
   when the first nibble of the d-CW is allocated from 0X1, the platform label space
   [RFC3031] which means payload that follows
   the DetNet flow d-DW is correctly identified
   and matched to an OAM payload with the flow parameters, including OAM type indicated by the flow history,
   regardless of which input interface value
   in the packet arrives on. d-CW Channel Type field.

   The
   S-label MUST be at the bottom label of the label stack reader is referred to [RFC5085] for a DetNet-
   s- or DetNet-st-flow more detailed description
   of the Associated Channel mechanism, and MUST precede to the d-CW.

   The S-label DetNet work on OAM
   for a specific more information DetNet flow is unique OAM.

6.4.  Flow Aggregation

   [Author's note: need to revisit this section and ensure that we cover
   (and fully specify) desired types of aggregation.]

   1.  Aggregate at the LSP (Forwarding)

   2.  Aggregating DetNet flow
   on flows as a specific node, but is not required to be identical with the
   S-label for that new DetNet flow in any other node within

   3.  Simple Aggregation at the DetNet
   domain.  Thus layer

   The resource control and management aspects of aggregation (including
   the S-label can only queuing/shaping/ policing implications) will be used covered in other
   documents.

   The ability to identify aggregate individual flows, and their associated
   resource control, into a larger aggregate is an important technique
   for improving scaling of control in the data, management and control
   planes.  The DetNet
   flow at data plane allows for the intended receiving node.

6.5.  Indication aggregation of the DetNet Payload Type

   The only nodes that needs
   flows, to know the payload type improved scaling.  There are three methods of a introducing
   flow are the
   DetNet ingress node and the DetNet egress nodes. aggregation:

   [Editor's note:]

   The ingress node
   has following review comments were received when this section was
   committed to know how github.

   General comment: We should points to process the packet it receives from major issue of aggregation,
   namely the ingress AC
   or IP flow, Seq.Num related problem.  The aggregated flows have their
   own Seq.Num and the egress edge node has to know how to prepare the
   packet for transmission those are independent.  We should consider to group
   the next hop.

   On ingress aggregation techniques as per their impact on what DetNet
   functions they allow on a DetNet edge node has to classify the packets into those
   that are for transmission as Detnet packets flow.  (E.g., aggregation without
   new Aggregate.Seq.Num would prohibit usage of FR, EF and those that are for
   transmission in-order-
   delivery function on the aggregate flow).

   SR based aggregation can be treated as "normal" packets at one a form of more lower priorities.
   The packet type is indicated to H-LSP aggregation.
   Should we differentiate them?  What are the egress edge node through differences?

   What are the
   value issues when aggregating of different payload types?
   Should we add an editor note on this?

   Simple-aggregation-at-the-detnet-layer: is this not the S-label.  Thus, when same as
   H-LSP?  The A-label can be treated just as an additional F-Label.

   [Editor's note: End of review comment.]

6.4.1.  Aggregation at the egress edge node looks up LSP

   DetNet flows forwarded via MPLS can leverage MPLS-TE's existing
   support for hierarchical LSPs (H-LSPs), see [RFC4206].  H-LSPs are
   typically used to aggregate control and resources, they may also be
   used to provide OAM or protection for the
   S-label one aggregated LSPs.  Arbitrary
   levels of aggregation naturally falls out of the parameters returned is definition for
   hierarchy and the packet type MPLS label stack [RFC3032].  DetNet nodes which in
   turn tells the egress edge node how to prepare
   support aggregation (LSP hierarchy) map one or more LSPs (labels)
   into and from an H-LSP.  Both carried LSPs and H-LSPs may or may not
   use the packet for
   transmission TC field, i.e., L-LSPs or E-LSPs.  Such nodes will need to a next hop.

   The consequence of this approach is
   ensure that if multiple packet
   encapsulations traffic from aggregated LSPs are processed on a node pair, each encapsulation will
   need its own S-Label.  That is not generally placed (shaped/policed/
   enqueued) onto the H-LSPs in a problems, since it is
   anticipated fashion that only one encapsulation type will be present for each ensures the required
   DetNet flow.  Of course, if for some reason service is preserved.

   Additional details of the multiple
   encapsulations are traffic control capabilities needed to support at a single DetNet service,
   multiple S-labels will
   DetNet-aware node may be required for that service.  Note that covered in the unlikely case that Ipv4 new service descriptions
   mentioned above or in separate future documents.  Management and IPv6
   control plane mechanisms will map also need to ensure that the same DetNet
   flow, different S-labels will be needed to differentiate between service
   required on the
   versions of IP.

6.6.  OAM Indication

   OAM follows aggregate flow (H-LSP or DSCP) are provided, which
   may include the procedures set out discarding or remarking mentioned in [RFC5085] with the restriction
   that only Virtual Circuit Connectivity Verification (VCCV) type 1 is
   supported.

   As previous
   sections.

6.4.2.  Aggregating DetNet Flows as a new DetNet flow

   An aggregate can be built by layering DetNet flows as shown in Figure 3 of [RFC5085] when below:

   +---------------------------------+
   |                                 |
   |           DetNet Flow           |
   |         Payload  Packet         |
   |                                 |
   +---------------------------------+ <--\
   |       DetNet Control Word       |    |
   +=================================+    |
   |            S-Label              |    |
   +---------------------------------+    +----DetNet data plane
   |       DetNet Control Word       |    |    MPLS encapsulation
   +=================================+    |
   |            A-Label              |    |
   +---------------------------------+    |
   |           F-Label(s)            | <--/
   +---------------------------------+
   |           Data-Link             |
   +---------------------------------+
   |           Physical              |
   +---------------------------------+
   Both the first nibble Aggregation (A) label and the S-Label have their MPLS S bit
   set indicating bottom of stack, and the d-CW
   is 0x0 the payload following allows the d-CW PREOF to
   work.

   It is normal user data.  However,
   when the first nibble a property of the d-CW is 0X1, the payload A-label that what follows
   the d-DW is an OAM payload with the OAM type indicated d-CW followed by
   an S-Label.  A relay node processing the value
   in A-label would not know the d-CW Channel Type field.

   The reader is referred
   underlying payload type.  This would only be known to [RFC5085] for a more detailed description node that was
   a peer of the Associated Channel mechanism, and to node imposing the DetNet work on OAM S-Label.  However there is no real
   need for more information DetNet OAM.

6.7.  Flow Aggregation

   1.  Aggregate at it to know the LSP (Transport)

   2.  Aggregating DetNet flows as a new DetNet flow

   3. payload type during aggregation processing.

6.4.3.  Simple Aggregation at the DetNet layer

   A further method of using SR Layer

   Another approach would be not to perform aggregation is include a d-CW for further
   study.

   The resource control and management aspects the aggregated
   flow.  This would be functionally similar to aggregation at the
   forwarding sub-layer using H-LSPs, but would confine knowledge of the
   aggregation (including to the queuing/shaping/ policing implications) will DetNet layer.  Such an approach shares the
   disadvantage that PREOF operations would not be covered possible.  OAM
   operation in other
   documents.

   The ability to aggregate individual flows, and their associated
   resource control, into a larger aggregate this mode is an important technique for improving scaling of control in the data, management and control
   planes.  The further study.

    +---------------------------------+
    |                                 |
    |           DetNet Flow           |
    |         Payload  Packet         |
    |                                 |
    +---------------------------------+ <--\
    |       DetNet Control Word       |    |
    +=================================+    |
    |            S-Label              |    +----DetNet data plane allows for the aggregation of DetNet
   flows, to improved scaling.  There are three methods of introducing
   flow aggregation:

   The following review comments were received when this section was
   committed to github.

   General comment: We should points to the major issue of aggregation,
   namely the Seq.Num related problem.
    +---------------------------------+    |    MPLS encapsulation
    |            A-Label              |    |
    +---------------------------------+    |
    |           F-Label(s)            | <--/
    +---------------------------------+
    |           Data-Link             |
    +---------------------------------+
    |           Physical              |
    +---------------------------------+

6.5.  Service Sub-Layer Considerations

   The aggregated flows have their
   own Seq.Num edge and those are independent.  We should consider relay node internal procedures related to group
   the aggregation techniques as per their impact on what DetNet
   functions they allow on PREOF are
   implementation specific.  The order of a DetNet flow.  (E.g., aggregation without
   new Aggregate.Seq.Num would prohibit usage packet elimination or
   replication is out of FR, EF and in-order-
   delivery function on the aggregate flow).

   SR based aggregation can scope in this specification.  However, care
   should be treated taken that the replication function does not actually
   loopback packets as a form of H-LSP aggregation.
   Should we differentiate them?  What are "replicas".  Looped back packets include
   artificial delay when the differences?

   What are node that originally initiated the issues when aggregating of different payload types?
   Should we add an editor note on this?

   Simple-aggregation-at-the-detnet-layer: packet
   receives it again.  Also, looped back packets may make the network
   condition to look healthier than it actually is this (in some cases link
   failures are not reflected properly because looped back packets make
   the same as
   H-LSP?  The A-label can be treated just as an additional T-label.

   End of review comment.

6.7.1.  Aggregation at situation appear better than it actually is).

   It is important that the LSP DetNet flows transported via MPLS can leverage MPLS-TE?s existing
   support for hierarchical LSPs (H-LSPs), see [RFC4206].  H-LSPs are
   typically used to aggregate control and resources, they may also be
   used to provide OAM or protection for the aggregated LSPs.  Arbitrary
   levels of aggregation naturally falls out of layer is configured such that a
   DetNet node never receives its own replicated packets.  If it were to
   receive such packets the definition for
   hierarchy and replication function would make the MPLS label stack [RFC3032].  DetNet nodes which
   support aggregation (LSP hierarchy) map one or loop
   more LSPs (labels)
   into and from an H-LSP.  Both carried LSPs and H-LSPs may or may not
   use destructive of bandwidth than a conventional unicast loop.
   Ultimately the TC field, i.e., L-LSPs or E-LSPs.  Such nodes TTL in the S-Label will need to
   ensure that traffic from aggregated LSPs are placed (shaped/policed/
   enqueued) onto cause the H-LSPs in packet to die during
   a fashion that ensures transient, but given the sensitivity of applications to packet
   latency the impact on the required DetNet service application would be severe.

6.5.1.  Edge Node Processing

   An edge node is preserved.

   Additional details of resposable for matching ingress packets to the traffic control capabilities needed at a
   DetNet-aware
   service they require and encapsulating them accordingly.An edge node
   may be covered participate in the new service descriptions
   mentioned above or in separate future documents.  Management packet replication and
   control plane mechanisms will also need to ensure that duplication
   elimination.

   The DetNet-aware forwarder selects the service
   required egress DetNet member flow
   segment based on the aggregate flow (H-LSP or DSCP) are provided, which
   may include the discarding or remarking mentioned in the previous
   sections.

6.7.2.  Aggregating DetNet flows as a new identification.  The mapping of ingress
   DetNet member flow

   An aggregate can be built by layering DetNet flows as shown below:

   +---------------------------------+
   |                                 |
   |           DetNet Flow           |
   |         Payload  Packet         |
   |                                 |
   +---------------------------------+ <--\
   |       DetNet Control Word       |    |
   +=================================+    |
   |            S-Label              |    |
   +---------------------------------+    +----DetNet data plane
   | segment to egress DetNet Control Word       |    |    MPLS encapsulation
   +=================================+    |
   |            A-Label              |    |
   +---------------------------------+ <--/
   |           T-Label(s)            |
   +---------------------------------+
   |           Data-Link             |
   +---------------------------------+
   |           Physical              |
   +---------------------------------+

   Both member flow segment may
   be statically or dynamically configured.  Additionally the Aggregation (A) label DetNet-
   aware forwarder does duplicate frame elimination based on the flow
   identification and the S-label have their MPLS S bit
   set indicating bottom of stack, sequence number combination.  The packet
   replication is also done within the DetNet-aware forwarder.  During
   elimination and the d-CW allows replication process the PREOF sequence number of the
   DetNet member flow MUST be preserved and copied to
   work.

   It is the egress DetNet
   member flow.

   The internal design of a property relay node is out of scope of this document.
   However the A-label that what follows reader's attention is d-CW followed by
   an S-label.  A relay node processing drawn to the A-label would not know need to make any PREOF
   state available to the
   underlying payload type.  This would only packet processor(s) dealing with packets to
   which the PREOF functions must be known applied, and to a node maintain that was
   a peer of the node imposing the S-label.  However there state
   is no real
   need for such as way that it is available to know the payload type during aggregation processing.

6.7.3.  Simple Aggregation at packet processor operation
   on the next packet in the DetNet layer

   Another approach would flow (which may be not to include a d-CW for duplicate, a
   late packet, or the aggregated
   flow.  This would be functionally similar to aggregation at next packet in sequence.

   [Editor's note: I think the
   transport layer using H-LSPs, but would confine knowledge rest of the
   aggregation to this section belongs in a new
   "802.1 TSN (island Interconnect) over DetNet MPLS" section.]

   This may be done in the DetNet layer.  Such an approach shares layer, or where the
   disadvantage that PREOF operations would not native service
   processing (NSP) [RFC3985] is IEEE 802.1CB [IEEE8021CB] capable, the
   packet replication and duplicate elimination MAY entirely be possible.  OAM
   operation done in this mode is for further study.

    +---------------------------------+
    |                                 |
    |           DetNet Flow           |
    |         Payload  Packet         |
    |                                 |
    +---------------------------------+ <--\
    |
   the NSP, bypassing the DetNet Control Word       |    |
    +=================================+    |
    |            S-Label              |    +----DetNet data plane
    +---------------------------------+    |    MPLS flow encapsulation
    |            A-Label              |    |
    +---------------------------------+ <--/
    |           T-Label(s)            |
    +---------------------------------+
    |           Data-Link             |
    +---------------------------------+
    |           Physical              |
    +---------------------------------+

6.8.  Service Layer Considerations and logic entirely.
   This enables operating over unmodified implementations and
   deployments.  The NSP approach works only between edge nodes and
   cannot make use of relay node internal procedures related nodes.

   The NSP approach is useful end to end tunnel and for for "island
   interconnect" scenarios.  However, when there is a need to do PREOF are
   implementation specific.  The order of
   in a middle of the network, such plain edge to edge operation is not
   sufficient.

   The extended forwarder MAY copy the sequencing information from the
   native DetNet packet elimination or
   replication into the DetNet sequence number field and vice
   versa.  If there is out of scope no existing sequencing information available in this specification.  However, care
   should be taken that
   the replication function does native packet or the forwarder chose not actually
   loopback packets as "replicas".  Looped back packets include
   artificial delay when to copy it from the
   native packet, then the extended forwarder MUST maintain a sequence
   number counter for each DetNet flow (indexed by the DetNet flow
   identification).

6.5.2.  Relay Node Processing

   A DetNet Relay node that originally initiated operates in the packet DetNet forwarding sub-layer .
   This processing is done within an extended forwarder function.
   Whether an ingress DetNet member flow receives it again.  Also, looped back packets may make DetNet specific
   processing depends on how the network
   condition forwarding is programmed.  Some relay
   nodes may be DetNet service aware, while others may be unmodified
   LSRs that only understand how to look healthier than it actually swicth MPLS-TE LSPs.

   It is (in some cases link
   failures are not reflected properly because looped back packets make also possible to treat the situation appear better than it actually is).

   It relay node as a transit node, see
   Section 6.6.3.  Again, this is important entirely up to how the forwarding has
   been programmed.

6.6.  Forwarding Sub-Layer Considerations

6.6.1.  Class of Service

   Class and quality of service, i.e., CoS and QoS, are terms that are
   often used interchangeably and confused with each other.  In the DetNet layer
   context of DetNet, CoS is configured such used to refer to mechanisms that provide
   traffic forwarding treatment based on aggregate group basis and QoS
   is used to refer to mechanisms that provide traffic forwarding
   treatment based on a specific DetNet node never receives its own replicated packets.  If it were to
   receive such packets flow basis.  Examples of
   existing network level CoS mechanisms include DiffServ which is
   enabled by IP header differentiated services code point (DSCP) field
   [RFC2474] and MPLS label traffic class field [RFC5462], and at Layer-
   2, by IEEE 802.1p priority code point (PCP).

   CoS for DetNet flows carried in PWs and MPLS is provided using the replication function would make
   existing MPLS Differentiated Services (DiffServ) architecture
   [RFC3270].  Both E-LSP and L-LSP MPLS DiffServ modes MAY be used to
   support DetNet flows.  The Traffic Class field (formerly the loop
   more destructive EXP
   field) of bandwidth than a conventional unicast loop.
   Ultimately an MPLS label follows the definition of [RFC5462] and
   [RFC3270].  The Uniform, Pipe, and Short Pipe DiffServ tunneling and
   TTL processing models are described in the S-Label will cause the packet to die during
   a transient, but given the sensitivity [RFC3270] and [RFC3443] and
   MAY be used for MPLS LSPs supporting DetNet flows.  MPLS ECN MAY also
   be used as defined in ECN [RFC5129] and updated by [RFC5462].

6.6.2.  Quality of applications Service

   In addition to explicit routes, and packet
   latency the impact on the replication and
   elimination, described in Section 6 above, DetNet application would be severe.

6.8.1.  Edge node processing

   An edge node is resposable for matching ingress packets provides zero
   congestion loss and bounded latency and jitter.  As described in
   [I-D.ietf-detnet-architecture], there are different mechanisms that
   maybe used separately or in combination to deliver a zero congestion
   loss service.  This includes Quality of Service (QoS) mechanisms at
   the
   service they require and encapsulating them accordingly.An edge node MPLS layer, that may participate in be combined with the packet replication and duplication
   elimination.

   The DetNet-aware forwarder selects mechanisms defined by
   the egress DetNet member underlying network layer such as 802.1TSN.

   Quality of Service (QoS) mechanisms for flow
   segment based on specific traffic
   treatment typically includes a guarantee/agreement for the flow identification.  The mapping service,
   and allocation of ingress
   DetNet member flow segment resources to egress DetNet member flow segment may
   be statically or dynamically configured.  Additionally the DetNet-
   aware forwarder does duplicate frame elimination based on support the service.  Example QoS
   mechanisms include discrete resource allocation, admission control,
   flow identification and the sequence number combination. isolation, and sometimes path control,
   traffic protection, shaping, policing and remarking.  Example
   protocols that support QoS control include Resource ReSerVation
   Protocol (RSVP) [RFC2205] (RSVP) and RSVP-TE [RFC3209] and [RFC3473].
   The packet
   replication is existing MPLS mechanisms defined to support CoS [RFC3270] can
   also done within the DetNet-aware forwarder.  During
   elimination and the replication process the sequence number be used to reserve resources for specific traffic classes.

   A baseline set of the QoS capabilities for DetNet member flow MUST be preserved flows carried in PWs
   and copied to MPLS can provided by MPLS with Traffic Engineering (MPLS-TE)
   [RFC3209] and [RFC3473].  TE LSPs can also support explicit routes
   (path pinning).  Current service definitions for packet TE LSPs can
   be found in "Specification of the egress DetNet
   member flow.

   The internal design Controlled Load Quality of a relay node is out
   Service", [RFC2211], "Specification of scope Guaranteed Quality of this document.
   However the reader's attention is drawn to the need to make any PREOF
   state available
   Service", [RFC2212], and "Ethernet Traffic Parameters", [RFC6003].
   Additional service definitions are expected in future documents to
   support the packet processor(s) dealing with packets to
   which full range of DetNet services.  In all cases, the PREOF functions must be applied,
   existing label-based marking mechanisms defined for TE-LSPs and even
   E-LSPs are use to maintain that state
   is such as way that it is available to the packet processor operation
   on the next packet in support the identification of flows requiring
   DetNet flow (which may be a duplicate, a
   late packet, or the next packet in sequence. QoS.

6.6.3.  Cross-DetNet Flow Resource Aggregation

   [Editor's note: I think the rest of NOTE: Isn't this section belongs in a new
   "802.1 TSN (island Interconnect) over MPLS DetNet" section.]

   This may be done in the DetNet layer, or where the native service
   processing (NSP) [RFC3985] is IEEE 802.1CB [IEEE8021CB] capable, same as "Aggregation at the
   packet replication
   LSP". -- Address as part of aggregation section cleanup.]

   The ability to aggregate individual flows, and duplicate elimination MAY entirely be done their associated
   resource control, into a larger aggregate is an important technique
   for improving scaling of control in the NSP, bypassing the DetNet flow encapsulation data, management and logic entirely. control
   planes.  This enables operating over unmodified implementations and
   deployments. document identifies the traffic identification related
   aspects of aggregation of DetNet flows.  The NSP approach works only between edge nodes resource control and
   cannot make use
   management aspects of relay nodes. aggregation (including the queuing/shaping/
   policing implications) will be covered in other documents.  The NSP approach is useful end to end tunnel and data
   plane implications of aggregation are independent for PW/MPLS and IP
   encapsulated DetNet flows.

   DetNet flows forwarded via MPLS can leverage MPLS-TE's existing
   support for "island
   interconnect" scenarios.  However, when there is a need hierarchical LSPs (H-LSPs), see [RFC4206].  H-LSPs are
   typically used to do PREOF
   in a middle of the network, such plain edge aggregate control and resources, they may also be
   used to edge operation is not
   sufficient.

   The extended forwarder MAY copy provide OAM or protection for the sequencing information from aggregated LSPs.  Arbitrary
   levels of aggregation naturally falls out of the
   native DetNet packet into definition for
   hierarchy and the MPLS label stack [RFC3032].  DetNet sequence number field nodes which
   support aggregation (LSP hierarchy) map one or more LSPs (labels)
   into and vice
   versa.  If there is no existing sequencing information available in
   the native packet from an H-LSP.  Both carried LSPs and H-LSPs may or the forwarder chose may not
   use the TC field, i.e., L-LSPs or E-LSPs.  Such nodes will need to copy it
   ensure that traffic from aggregated LSPs are placed (shaped/policed/
   enqueued) onto the
   native packet, then the extended forwarder MUST maintain a sequence
   number counter for each DetNet flow (indexed by the DetNet flow
   identification).

6.8.2.  Relay node processing

   A DetNet Relay node operates H-LSPs in a fashion that ensures the required
   DetNet transport layer . This
   processing service is done within an extended forwarder function.  Whether an
   ingress DetNet member flow receives DetNet specific processing
   depends on how preserved.

   [NOTE: This needs to be revised:] Additional details of the forwarding is programmed.  Some relay nodes traffic
   control capabilities needed at a DetNet-aware node may be
   DetNet covered in
   the new service aware, while others may be unmodified LSRs that only
   understand how to swicth MPLS-TE LSPs.

   It is descriptions mentioned above or in separate future
   documents.  Management and control plane mechanisms will also possible need to treat
   ensure that the relay node as a transit node, see
   Section 6.9.3.  Again, this is entirely up to how service required on the forwarding has
   been programmed.

6.9.  Other DetNet data plane considerations

6.9.1.  Class of Service aggregate flow (H-LSP or
   DSCP) are provided, which may include the discarding or remarking
   mentioned in the previous sections.

6.6.4.  Layer 2 Addressing and QoS Considerations

   [Editor's note: this section needs to updated to discuss how DetNet
   service is mapped to E- NOTE: review and L-LSPs.  Perhaps simplify this gets merged with section.  Doesn't this
   belong in the aggregation section or dropped?]

   Class and quality TSN section?  Alternatively, describe in generic/non
   sub-network technology specific terms.]

   The Time-Sensitive Networking (TSN) Task Group of service, i.e., CoS and QoS, are terms that are
   often used interchangeably and confused with each other.  In the
   context IEEE 802.1
   Working Group have defined (and are defining) a number of DetNet, CoS is used to refer amendments
   to mechanisms IEEE 802.1Q [IEEE8021Q] that provide
   traffic forwarding treatment based on aggregate group basis zero congestion loss and QoS
   is used to refer to mechanisms
   bounded latency in bridged networks.  IEEE 802.1CB [IEEE8021CB]
   defines packet replication and elimination functions that provide traffic forwarding
   treatment based on should
   prove both compatible with and useful to, DetNet networks.

   As is the case for DetNet, a Layer 2 network node such as a bridge
   may need to identify the specific DetNet flow basis.  Examples of
   existing network level CoS mechanisms include DiffServ to which is
   enabled by IP header differentiated services code point (DSCP) field
   [RFC2474] and MPLS label traffic class field [RFC5462], and at Layer-
   2, by IEEE 802.1p priority code point (PCP).

   CoS for DetNet flows carried a packet
   belongs in PWs and MPLS is provided using the
   existing MPLS Differentiated Services (DiffServ) architecture
   [RFC3270].  Both E-LSP and L-LSP MPLS DiffServ modes MAY be used order to
   support DetNet flows.  The Traffic Class field (formerly provide the TSN/DetNet QoS for that packet.  It
   also will likely need a CoS marking, such as the EXP
   field) priority field of an MPLS label follows
   IEEE Std 802.1Q VLAN tag, to give the definition of [RFC5462] and
   [RFC3270].  The Uniform, Pipe, and Short Pipe DiffServ tunneling and
   TTL processing models are packet proper service.

   Although the flow identification methods described in [RFC3270] and [RFC3443] IEEE 802.1CB
   [IEEE8021CB] are flexible, and
   MAY be used for MPLS LSPs supporting DetNet flows.  MPLS ECN MAY also
   be used as defined in ECN [RFC5129] and updated by [RFC5462].

   CoS for fact, include IP 5-tuple
   identification methods, the baseline TSN standards assume that every
   Ethernet frame belonging to a TSN stream (i.e.  DetNet flows carried in IPv6 flow) carries
   a multicast destination MAC address that is provided using unique to that flow
   within the standard
   differentiated services code point (DSCP) field [RFC2474] and related
   mechanisms.  The 2-bit explicit congestion notification (ECN)
   [RFC3168] field MAY also be used.

   One additional consideration for DetNet nodes bridged network over which support CoS
   services it is carried.  Furthermore,
   IEEE 802.1CB [IEEE8021CB] describes three methods by which a packet
   sequence number can be encoded in an Ethernet frame.

   Ensuring that they MUST ensure that the CoS service classes do not
   impact the congestion protection proper Ethernet VLAN tag priority and latency control mechanisms destination
   MAC address are used on a DetNet/TSN packet may require further
   clarification of the customary L2/L3 transformations carried out by
   routers and edge label switches.  Edge nodes may also have to provide DetNet QoS.  This requirement is similar to requirement
   for MPLS LSRs to that CoS LSPs do not impact move
   sequence number fields among Layer 2, PW, and IP encapsulations.

6.6.5.  Time Synchronization

   [Editor's Note: A detailed discussion of time synchronization is
   outside the resources allocated
   to TE LSPs via [RFC3473].

6.9.2.  Quality scope of Service

   Quality this document, and the production of Service (QoS) mechanisms for flow specific traffic
   treatment typically includes a guarantee/agreement for
   specialist text discussing this topic is encouraged.  This section
   will be updated/removed if such a document is available before
   publication of this text.]

   Time synchronization is important both from the service,
   and allocation perspective of resources to support
   operating the service.  Example QoS
   mechanisms include discrete resource allocation, admission control,
   flow identification and isolation, and sometimes path control,
   traffic protection, shaping, policing and remarking.  Example
   protocols that support QoS control include Resource ReSerVation
   Protocol (RSVP) [RFC2205] (RSVP) and RSVP-TE [RFC3209] DetNet network itself and [RFC3473].
   The existing MPLS mechanisms defined to support CoS [RFC3270] can
   also from the perspective of
   transferring time across the network between client applications.
   Some clients may be used to reserve resources for specific traffic classes.

   In addition able to explicit routes, and packet replication and
   elimination, described in Section 6 above, use the DetNet provides zero
   congestion loss and bounded latency as their provider of time
   and jitter.  As described in
   [I-D.ietf-detnet-architecture], there are different mechanisms that
   maybe used separately or in combination frequency, others may require the DetNet to deliver transfer time between
   a zero congestion
   loss service.  These mechanisms are provided by the either client clock source and a client clock user.

   For example, [RFC8169] describes a method of recording the packet
   queuing time in an MPLS
   or IP layers, LSR on a packet by per packet basis and may
   forwarding this information to the egress edge system.  This allows
   compensation for any variable packet queuing delay to be combined with applied at
   the packet receiver.  Other mechanisms for IP/MPLS networks are
   defined by the
   underlying network layer based on IEEE Standard 1588 [IEEE1588], such as 802.1TSN. ITU-T
   [G.8275.1] and [G.8275.2].

   A baseline set more detailed discussion of QoS capabilities for DetNet flows carried in PWs time synchronization is outside the
   scope of this document.

7.  Controller Plane (Management and MPLS can Control) Considerations

   While management plane and control planes are traditionally
   considered separately, from the Data Plane perspective there is no
   practical difference based on the origin of flow provisioning
   information, and the DetNet architecture
   [I-D.ietf-detnet-architecture] refers to these collectively as the
   'Controller Plane'.  This document therefore does not distinguish
   between information provided by MPLS with Traffic Engineering (MPLS-TE) distributed control plane protocols,
   e.g., RSVP-TE [RFC3209] and [RFC3473].  TE LSPs can also support explicit routes
   (path pinning).  Current service definitions for packet TE LSPs can
   be found in "Specification of [RFC3473], or by centralized network
   management mechanisms, e.g., RestConf [RFC8040], YANG [RFC7950], and
   the Controlled Load Quality of
   Service", [RFC2211], "Specification of Guaranteed Quality of
   Service", [RFC2212], Path Computation Element Communication Protocol (PCEP)
   [I-D.ietf-pce-pcep-extension-for-pce-controller] or any combination
   thereof.  Specific considerations and "Ethernet Traffic Parameters", [RFC6003].
   Additional service definitions requirements for the DetNet
   Controller Plane are expected discussed in future documents Section 7.6.

7.1.  S-Label and F-Label Assignment and Distribution

   [Editor's note - we may need additional text on resource allocation
   in this section.]

   DetNet S-Labels (see Section 6.2.2 for their definition) are similar
   to
   support other MPLS service labels that denote the full range contents of DetNet services.  In all cases, the
   existing label-based marking mechanisms defined for TE-LSPs and even
   E-LSPs MPLS
   packet payload such as a layer 2 pseudowire, an IP packet that is
   routed in a VPN context with a private address, or an Ethernet
   virtual private network (EVPN) service.

   S-Labels are use expected to support be allocated in the identification of flows requiring
   DetNet QoS.

   Packets that are marked with same manner as any other
   service labels.  S-Labels uniquely identify a particular DetNet Class of Service value, but
   that have not been flow,
   and are local to the subject of a completed reservation, can
   disrupt node on which the label is allocated.  In the QoS offered to properly reserved
   DetNet flows by using
   resources allocated to service sub-layer the reserved flows.  Therefore, explicit route consists of the network
   nodes set of a
   Relay Nodes that the DetNet network:

   o  MUST defend flow must traverse.  They can be used to
   identify the DetNet QoS by discarding or remarking (to flow that a packet belongs to as it traverses a
   particular node in a non- DetNet CoS) packets received that domain.  Because labels are not the subject of local to each
   node rather than being a
      completed reservation.

   o  MUST NOT use global identifier within a domain, they must
   be advertised to their upstream DetNet reserved resource, e.g. service-aware peer nodes
   (e.g., a queue or shaper
      reserved for DetNet flows, for any packet that does not carry MPLS End System as shown in Figure 3, or a DetNet Class of Service marker.

6.9.3.  Cross-DetNet flow resource aggregation

   [Editor's NOTE: keep and extend this section.]

   The ability to aggregate individual flows, and their associated
   resource control, into a larger aggregate is an important technique
   for improving scaling of control
   Relay or Edge Node as shown in the data, management Figure 7) and control
   planes.  This document identifies interpreted in the traffic identification related
   aspects of aggregation
   context of their received F-Label.

   As discussed in Section 4, the forwarding sub-layer uses one or more
   F-Labels to forward DetNet flows.  The resource control and
   management aspects of aggregation (including packets between DetNet service-aware nodes
   along explicitly defined routes at the queuing/shaping/
   policing implications) will be covered DetNet forwarding sub-layer,
   which in other documents.  The data
   plane implications the context of aggregation are independent for PW/MPLS and IP
   encapsulated DetNet flows.

   DetNet flows transported via this document is the MPLS layer.  F-Labels
   can leverage MPLS-TE's existing
   support for hierarchical LSPs (H-LSPs), see [RFC4206].  H-LSPs are
   typically used to aggregate control and resources, they may also be
   used to provide OAM or protection context for an S-Label.  In the aggregated LSPs.  Arbitrary
   levels of aggregation naturally falls out of DetNet Forwarding
   (MPLS) sub-layer the definition for
   hierarchy and explicit route consists of the MPLS label stack [RFC3032]. set of DetNet
   nodes which
   support aggregation (LSP hierarchy) map one or more LSPs (labels)
   into are LSRs, links, and from an H-LSP.  Both carried LSPs possibly link bundle members and H-LSPs may or may not
   use the TC field, i.e., L-LSPs or E-LSPs.  Such nodes will need to
   ensure that traffic from aggregated LSPs are placed (shaped/policed/
   enqueued) onto the H-LSPs in a fashion
   queues that ensures the required DetNet service is preserved.

   DetNet flows transported via IP have more limited aggregation
   options, due to the available traffic flow identification fields packets of
   the IP solution.  One available approach is to manage the resources
   associated with a DSCP identified traffic class and to map (remark)
   individually controlled DetNet flows onto that traffic class.  This
   approach also requires that flow must traverse between nodes support aggregation ensure that
   traffic from aggregated LSPs are placed (shaped/policed/enqueued)
   in
   a fashion that ensures the required DetNet service is preserved.

   In both the MPLS sub-layer (i.e. between a specific Edge Node
   and IP cases, additional details of the traffic
   control capabilities needed at next hop Relay Node, between specific Relay Nodes, and
   between a DetNet-aware specific Relay node may be covered in
   the new service descriptions mentioned above or in separate future
   documents.  Management and control plane mechanisms will also need the egress Edge Node.  Resource
   allocation corresponding to
   ensure that the service required on set of Services supported over the aggregate flow (H-LSP or
   DSCP) are provided,
   forwarding sub-layer, which may include the discarding or remarking
   mentioned in the previous sections.

6.9.4.  Layer 2 addressing and QoS Considerations

   [Editor's NOTE: review and simplify may not include aggregation, is
   required at this section.]

   The Time-Sensitive Networking (TSN) Task Group of the IEEE 802.1
   Working Group have defined (and sub-layer.  Explicit routes are defining) a number of amendments used to IEEE 802.1Q [IEEE8021Q] that provide zero congestion loss and
   bounded latency in bridged networks.  IEEE 802.1CB [IEEE8021CB]
   defines packet replication and elimination functions ensure that should
   prove both compatible with and useful to, DetNet networks.

   As is
   packets are routed through the case resources that have been reserved for DetNet, a Layer 2 network node such as a bridge
   may need to identify the specific DetNet flow to which a packet
   belongs in order to
   them, and hence provide the TSN/DetNet QoS for that packet.  It
   also will likely need a CoS marking, such as DetNet application with the priority field of required
   service.  Multiple F-Labels may be pushed after an
   IEEE Std 802.1Q VLAN tag, S-Label and there
   is no requirement for all F-Labels to give the packet proper service.

   Although be controlled via the flow identification methods described same
   controller mechanisms.  For example in IEEE 802.1CB
   [IEEE8021CB] EVPN, some labels are flexible,
   distributed using BGP while others are distributed using LDP or RSVP.

   Whether configuring, calculating and instantiating these routes is a
   single-stage or multi-stage process, or in fact, include IP 5-tuple
   identification methods, the baseline TSN standards assume that every
   Ethernet frame belonging to a TSN stream (i.e.  DetNet flow) carries centralized or
   distributed manner, is out of scope of this document.

   There are a multicast destination MAC address number of approaches that is unique could be used to that flow
   within provide
   explicit routes and resource allocation in the bridged network over which it is carried.  Furthermore,
   IEEE 802.1CB [IEEE8021CB] describes three methods MPLS layer:

   o  The path could be explicitly set up by a controller which
      calculates the path and explicitly configures each node along that
      path with the appropriate forwarding and resource allocation
      information.

   o  The path could be set up using RSVP-TE signaling.

   o  The path could be implemented using MPLS-based segment routing
      when extended to support resource allocation.

   See Section 7.6 for further discussion of these alternatives.

   Much like other MPLS labels, there are a packet
   sequence number can be encoded in of alternatives
   available for DetNet S-Label and F-Label advertisement to an Ethernet frame.

   Ensuring upstream
   peer node.  These include distributed signaling protocols such as
   RSVP-TE, centralized label distribution via a controller that manages
   both the proper Ethernet VLAN tag priority sender and destination
   MAC address are used on a DetNet/TSN packet may require further
   clarification the receiver using NETCONF/YANG, BGP, PCEP, etc.,
   and hybrid combinations of the customary L2/L3 transformations carried out by
   routers two.  The details of the controller
   plane solution required for the label distribution and edge the management
   of the label switches.  Edge nodes may also have to move
   sequence number fields among Layer 2, PW, and IPv6 encapsulations.

6.9.5.  Time Synchronization

   [Editor's Note: A detailed discussion space are out of time synchronization scope of this document, but as
   mentioned above, there are particular DetNet considerations and
   requirements that are discussed in Section 7.6.

7.2.  Packet Replication, Elimination, and Ordering (PREOF)

   The controller plane protocol solution required for managing the
   PREOF processing is outside the scope of this document, and the production of a
   specialist text discussing this topic is encouraged.  This section
   will document.  That said,
   it should be updated/removed if such a document is available before
   publication of this text.]

   Time synchronization is important both from noted that the perspective of
   operating ability to determine, for a particular
   flow, optimal packet replication and elimination points in the DetNet network itself
   domain requires explicit support.  There are be capabilities that can
   be used, or extended, for example GMPLS end-to-end recovery [RFC4872]
   and from the perspective of
   transferring time across GMPLS segment recovery [RFC4873].

7.3.  Contention Loss and Jitter Reduction

   As discussed in Section 1, this document does not specify the network between client applications.
   Some clients may be able
   mechanisms needed to use the eliminate contention loss or reduce jitter for
   DetNet as their provider of time
   and frequency, others may require flows at the DetNet forwarding sub-layer.  The ability to transfer time between
   a client clock source and a client clock user.

   For example, [RFC8169] describes a method of recording the packet
   queuing time in an MPLS LSR on a packet by per packet basis
   manage node and
   forwarding this information link resources to the egress edge system.  This allows
   compensation for any variable packet queuing delay be able to provide these functions
   will be applied at
   the packet receiver.  Other mechanisms for IP/MPLS networks are
   defined based on IEEE Standard 1588 [IEEE1588], such as ITU-T
   [G.8275.1] and [G.8275.2].

   A more detailed discussion a necessary part of time synchronization is outside the
   scope of this document.

7.  Management and control considerations

   [Editor's note: This section needs DetNet controller plane.  It will
   also be necessary to be different able to control the required queuing
   mechanisms used to provide these functions along a flow's path
   through the network.  See Section 7.6 for further discussion of these
   requirements.

7.4.  Bidirectional Traffic

   Some DetNet applications generate bidirectional traffic.  Using MPLS and IP
   solutions.  Most solutions
   definitions [RFC5654] there are technology dependant.  Currently most
   text in this section is just a draft associated bidirectional flows, and may have bits that are
   already moved
   co-routed bidirectional flows.  MPLS defines a point-to-point
   associated bidirectional LSP as consisting of two unidirectional
   point-to-point LSPs, one from A to other places/documents.]

   While management plane B and control planes are traditionally
   considered separately, from the Data Plane perspective there is no
   practical difference based on the origin of flow provisioning
   information.  This document therefore does not distinguish between
   information provided by other from B to A, which
   are regarded as providing a control plane protocol, e.g., RSVP-TE
   [RFC3209] and [RFC3473], single logical bidirectional forwarding
   path.  This would be analogous of standard IP routing, or by PWs running
   over two reciprocal unidirection LSPs.  MPLS defines a network management mechanisms, e.g.,
   RestConf [RFC8040] point-to-point
   co-routed bidirectional LSP as an associated bidirectional LSP which
   satisfies the additional constraint that its two unidirectional
   component LSPs follow the same path (in terms of both nodes and YANG [RFC7950].

   [Editor's note: This section is a work
   links) in progress.  discuss here
   what kind both directions.  An important property of enhancements are needed for DetNet and specifically for
   PREOF and DetNet zero congest loss and latency control.  Need to
   cover co-routed
   bidirectional LSPs is that their unidirectional component LSPs share
   fate.  In both traffic control (queuing) types of bidirectional LSPs, resource reservations may
   differ in each direction.  The concepts of associated bidirectional
   flows and connection control (control
   plane).]

7.1.  MPLS-based co-routed bidirectional flows can be applied to DetNet
   flows.

   While the MPLS data plane

7.1.1.  S-Label assignment and distribution

   [Editor's note: Outdated and needs more work.]

   The must support bidirectional DetNet S-Label distribution follows the same mechanisms specified
   for XYZ . The details of flows,
   there are no special bidirectional features with respect to the control data
   plane protocol solution required
   for the label distribution and other than the management of need for the label number
   space are out of scope two directions of this document.

7.1.2.  Explicit routes

   It is necessary a co-routed
   bidirectional flow to consider explicit routes both at take the DetNet layer same path.  Fate sharing and in the MPLS layer.  In the DetNet layer the explicit route
   consists of
   associated vs co-routed bidirectional flows can be managed at the set of Relay Nodes
   control level.  Note that the there is no stated requirement for
   bidirectional DetNet flow must
   traverse.  In flows to be supported using the same MPLS layer the explicit route consists Labels
   in each direction.

   DetNet's use of PREOF may increase the set complexity of LSRs, links, and possibly link bundle members and queues that using co-routing
   bidirectional flows, since if PREOF is used, then the
   DetNet packets of a flow must traverse between nodes replication
   points in one direction would have to match the DetNet
   layer (i.e. between a specific Edge Node and elimination points in
   the next hop Relay Node,
   between specific Relay Nodes, other direction, and between a specific Relay node vice versa, and the egress Edge Node.  This detailed steering is needed to ensure
   that packets are routed through the resources that have been reserved optimal points for them, and hence provide these
   functions in one direction may not match the DetNet application with optimal points in the required
   performance.

   Whether configuring, calculating
   other.

   Control and instantiating this is a multi-
   stage process, or a single stage process is management mechanisms will need to support bidirectional
   flows, but the specification of such mechanisms are out of scope of
   this document.

   The one method of explicitly setting up the explicit path at the
   DetNet layer  Related control plan mechanisms have been defined in
   [RFC3473], [RFC6387] and [RFC7551].

   This is through further discussed in Section 7.6.

7.5.  Flow Aggregation Control

   Section 6.4 discusses the use of flow aggregation in DetNet.  It
   includes flow aggregation accomplished through the management controller.

   [Editor's note: a method use of setting up
   hierarchical LSPs, aggregating multiple DetNet flows into a graph through single
   new DetNet flow, and simple aggregation at the DetNet
   Nodes using layer.  It will
   be the IGP has been proposed.  A reference is needed to
   e.g., RFC 7813 IS-IS Path Control and Reservation.]

   There are a number responsibility of approaches that can the DetNet controller plane to be taken able to provide
   explicit routes/paths
   properly provision the use of these mechanisms.  These requirements
   are included in the MPLS layer:

   o  The path can be explicitly set up by next section.

7.6.  DetNet Controller (Control and Management) Plane Requirements

   While the management definition of controller
      calculating plane for DetNet is out of the path
   scope of this document, there are particular considerations and explicitly configuring each node along
   requirements for such that path.

   o  The LSP can be set up using RSVP-TE.  Such an approach confines result from the packet to unique characteristics of
   the explicit path.

   o DetNet architecture [I-D.ietf-detnet-architecture] and data plane
   as defined herein.

   The path can be implemented using segment routing.

   Where primary requirements of the DetNet traffic is carried over IP Section 11 controller plane are that it
   must be able to:

   o  Instantiate DetNet flows in a DetNet domain (which may include
      some or all of explicit paths
   may need to be provided in the IP layer.  This is for further study.

7.2.  Packet path and PREOF replication and elimination

   [Editor's note: Outdated
      node determination, link bandwidth reservations, node buffer and at the functional level technology
   independent.. but needs more work.]

   The control plane protocol solution
      other resource reservations, specification of required for managing the PREOF
   processing is outside queuing
      disciplines along the scope of this document.

7.3.  Congestion protection and latency control

   [Editor's note: TBD]

7.4.  Bidirectional traffic

   [Editor's NOTE: this section needs to be updated to have its scope
   limited path, ability to management and control.]

   Some DetNet applications generate bidirectional traffic.  Using MPLS
   definitions [RFC5654] there are associated manage bidirectional flows, and
   co-routed bidirectional flows.  MPLS defines a point-to-point
   associated bidirectional LSP
      etc.) as consisting of two unidirectional
   point-to-point LSPs, one from A to B needed for a flow.

   o  Manage DetNet S-Label and F-Label allocation and distribution,
      when the other from B DetNet MPLS encapsulation is in use

   o  The ability to A, which
   are regarded support DetNet flow aggregation

   o  Advertise static and dynamic node and link resources such as providing a single logical bidirectional transport
   path.  This would be analogous of standard IP routing,
      capabilities and adjacencies to other network nodes (for dynamic
      signaling approaches) or PWs running
   over two reciprocal unidirection LSPs.  MPLS defines a point-to-point
   co-routed bidirectional LSP as an associated bidirectional LSP which
   satisfies the additional constraint that its two unidirectional
   component LSPs follow to network controllers (for centralized
      approaches)

   o  Scale to handle the same path (in terms number of both nodes and
   links) DetNet flows expected in both directions.  An important property of co-routed
   bidirectional LSPs is that their unidirectional component LSPs share
   fate.  In both types a domain
      (which may require per-flow signaling or provisioning)

   o  Provision flow identification information at each of bidirectional LSPs, resource allocations the nodes
      along the path, and it may differ depending on the location in each direction.  The concepts of associated bidirectional
   flows the
      network and co-routed bidirectional flows can be applied to the DetNet
   flows functionality (e.g. transit node vs. relay
      node).

   These requirements, as well whether IPv6 stated earlier, could be satisfied using
   distributed control protocol signaling (such as RSVP-TE), centralized
   network management provisioning mechanisms (such as BGP, PCEP, YANG
   [I-D.ietf-detnet-flow-information-model], etc.) or MPLS is used.

   While hybrid
   combinations of the IPv6 two, and MPLS data planes must support bidirectional DetNet
   flows, there are no special bidirectional features with respect to could also make use of MPLS-based
   segment routing.

   In the abstract, the results of either distributed signaling or
   centralized provisioning are equivalent from a DetNet data plane other than need for the two directions take the same
   paths.  Fate sharing and associated vs co-routed bidirectional
   perspective - flows
   can be managed at are instantiated, explicit routes are determined,
   resources are reserved, and packets are forwarded through the control level.  Note, that there is no stated
   requirement for bidirectional DetNet flows to be supported domain
   using the
   same IPv6 Flow Labels or MPLS Labels in each direction.  Control
   mechanisms will need to support such bidirectional flows for both
   IPv6 data plane.

   However, from a practical and MPLS, but such mechanisms implementation standpoint, they are out of scope of this document.
   An example control plane solution for MPLS can be found not
   equivalent at all.  Some approaches are more scalable than others in [RFC7551].

7.5.  Flow aggregation control

   [TBD]

8.  DetNet IP Operation over DetNet MPLS Service

   [Editor's note: this is a place holder section.  A standalone section
   terms of signaling load on operation the network.  Some can take advantage of IP flows over DetNet MPLS data plane.  Includes
   RFC2119 Language.]

9.  IEEE 802.1 TSN Interconnection over
   global tracking of resources in the DetNet MPLS Service

   [Editor's note: this domain for better overall
   network resource optimization.  Some are more resilient than others
   if link, node, or management equipment failures occur.  While a
   detailed analysis of the control plane alternatives is out of the
   scope of this document, the requirements from this document can be
   used as the basis of a place holder section.  A standalone section
   on TSN "island" interconnect over DetNet".  Includes RFC2119
   Language.]

10. later analysis of the alternatives.

8.  DetNet MPLS Transport Layer Operation over Over IEEE 802.1 TSN Sub-
     Networks Sub-Networks

   [Editor's note: this is a place holder section.  A standalone section
   on MPLS over IEEE 802.1 TSN.  Includes RFC2119 Language.]

   This section covers how MPLS DetNet MPLS flows operate over an IEEE 802.1
   TSN sub-network.  Figure 17 15 illustrates such a scenario, where two
   MPLS (DetNet) nodes are interconnected by a TSN sub-network.  Node-1
   is single homed and Node-2 is dual-homed.  MPLS nodes can be (1) MPLS
   DetNet MPLS End System, (2) MPLS DetNet MPLS Edge or Relay node or (3)
   MPLS Transit node.

   Note: in case of MPLS Transit node there is no DetNet Service sub-
   layer.
   layer processing.

      MPLS (DetNet)                 MPLS (DetNet)
         Node-1                        Node-2

      +---------+                  +---------+

      +----------+                  +----------+
   <--| Service*|-- Service* |-- DetNet flow ---| Service*|-->
      +---------+                  +---------+
      |Transport|                  |Transport|
      +-------.-+ Service* |-->
      +----------+                  +----------+
      |Forwarding|                  |Forwarding|
      +--------.-+    <-TSN Str->   +-.-----.-+   +-.-----.--+
                \      ,-------.     /     /
                 +----[ TSN-Sub ]---+     /
                      [ Network ]--------+
                       `-------'
   <---------------- DetNet MPLS --------------->

   Note: * no service sub-layer required for transit nodes

       Figure 17: 15: DetNet Enabled MPLS Network over Over a TSN sub-network Sub-Network

   The Time-Sensitive Networking (TSN) Task Group of the IEEE 802.1
   Working Group have defined (and are defining) a number of amendments
   to IEEE 802.1Q [IEEE8021Q] that provide zero congestion loss and
   bounded latency in bridged networks.  Furthermore IEEE 802.1CB
   [IEEE8021CB] defines frame replication and elimination functions for
   reliability that should prove both compatible with and useful to,
   DetNet networks.  All these functions have to identify flows those
   require TSN treatment.

   As is the case for DetNet, a Layer 2 network node such as a bridge
   may need to identify the specific DetNet flow to which a packet
   belongs in order to provide the TSN/DetNet QoS for that packet.  It
   also may need a CoS marking, such as the priority field of an IEEE
   Std 802.1Q VLAN tag, to give the packet proper service.

   The challange for MPLS DeNet flows is that the protocol interworking
   function defined in IEEE 802.1CB [IEEE8021CB] works only for IP
   flows.  The aim of the protocol interworking function is to convert
   an ingress flow to use a specific multicast destination MAC address
   and VLAN, for example to direct the packets through a specific path
   inside the bridged network.  A similar interworking pair at the other
   end of the TSN sub-network would restore the packet to its original
   destination MAC address and VLAN.

   As protocol interworking function defined in [IEEE8021CB] does not
   work for MPLS labeled flows, the MPLS DetNet MPLS nodes MUST ensure proper
   TSN sub-network specific Ethernet encapsulation of the MPLS DetNet MPLS
   packets.  For a given TSN Stream (i.e., DetNet flow) an MPLS (DetNet)
   node MUST behave as a TSN-aware Talker or a Listener inside the TSN
   sub-network.

10.1.

8.1.  Mapping of TSN Stream ID and Sequence Number

   TSN capable MPLS (DetNet) nodes are TSN-aware Talker/Listener as
   shown in Figure 18. 16.  MPLS (DetNet) node MUST provide the TSN sub-
   network specific Ethernet encapsulation over the link(s) towards the
   sub-network.  An TSN-aware MPLS (DetNet) node MUST support the
   following TSN components:

   1.  For recognizing flows:

       *  Stream Identification (MPLS-flow-aware)

   2.  For FRER used inside the TSN domain, additonaly:

       *  Sequencing function (MPLS-flow-aware)

       *  Sequence encode/decode function

   3.  For FRER when the node is a TSN replication or elimination point,
       additionally:

       *  Stream splitting function

       *  Individual recovery function

   [Editor's note: Should we added here requirements regarding IEEE
   802.1Q C-VLAN component?]

   The Stream Identification and The Sequencing functions are slightly
   modified for frames passed down the protocol stack from the upper
   layers.

   Stream Identification MUST pair MPLS flows and TSN Streams and encode
   that in data plane formats as well.  The packet's stream_handle
   subparameter (see IEEE 802.1CB [IEEE8021CB]) inside the Talker/
   Listener is defined based on the Flow-ID used in the upper MPLS DetNet
   MPLS layer.  Stream Identification function MUST encode Ethernet
   header fields namely (1) the destination MAC-address, (2) the VLAN-ID
   and (3) priority parameters with TSN sub-network specific values.
   Encoding is provided for the frame passed down the stack from the
   upper layers.

   The sequence generation function resides in the Sequencing function.
   It generates a sequence_number subparameter for each packet of a
   Stream passed down to the lower layers.  Sequencing function MUST
   copy sequence information from the MPLS d-CW of the packet to the
   sequence_number subparameter for the frame passed down the stack from
   the upper layers.

      MPLS (DetNet)
         Node-1
      <--------->

      +---------+
      <---------->

      +----------+
   <--| Service  |-- DetNet flow ------------------
      +---------+
      |Transport|
      +---------+
      +----------+
      |Forwarding|
      +----------+    +---------------+
      | L2 with  |<---| L2 Relay with |---- TSN ----
      |   TSN    |    | TSN function  |    Stream
      +----.----+
      +-----.----+    +--.---------.--+
             \__________/           \______

       TSN-aware
        Talker /          TSN-Bridge
        Listener             Relay

            <--------- TSN sub-network ------------

             Figure 18: 16: MPLS (DetNet) node Node with TSN functions Functions

   The Sequence encode/decode function MUST support the Redundancy tag
   (R-TAG) format as per Clause 7.8 of IEEE 802.1CB [IEEE8021CB].

10.2.

8.2.  TSN Usage of FRER

   TSN Streams supporting DetNet flows may use Frame Replication and
   Elimination for Redundancy (FRER) [802.1CB] based on the loss service
   requirements of the TSN Stream, which is derived from the DetNet
   service requirements of the DetNet mapped flow.  The specific
   operation of FRER is not modified by the use of DetNet and follows
   IEEE 802.1CB [IEEE8021CB].

   FRER function and the provided service recovery is available only
   within the TSN sub-network however as the Stream-ID and the TSN
   sequence number are paired with the MPLS flow parameters they can be
   combined with PREOF functions.

10.3.

8.3.  Management and Control Implications

   [Editor's note: This section is TBD Covers Creation, mapping, removal
   of TSN Stream IDs, related parameters and,when needed, configuration
   of FRER.  Supported by management/control plane.]

11.

9.  DetNet MPLS Transport Layer Operation over IP DetNet IP PSNs

   This section specifies the DetNet encapsulation over an IP transport network.
   The approach is modeled on the operation of MPLS and PseudoWires (PW)
   over an IP Packet Switched Network (PSN) [RFC3985][RFC4385][RFC7510].
   It is also based on maps the MPLS data plane encapsulation described in Section 6.2. 6.2 to
   the DetNet IP data plane define in [I-D.ietf-detnet-dp-sol-ip].

   To carry DetNet with full functionality at the DetNet layer over an
   IP transport network, the following components are required (these are a subset
   of the requirements for MPLS encapsulation listed in Section 6.1):

   1.  A method of identifying the DetNet flow group to the processing
       element.

   2.  A method of carrying the DetNet sequence number.

   3.  A method of distinguishing DetNet OAM packets from DetNet data
       packets.

   4.  A method of carrying queuing and forwarding indication.

   These requirements are satisfied by the DetNet over MPLS
   Encapsulation described in Section 6.2.

   To simplify operations and implementations, rather than inventing a
   new encapsulation, the IP encapsulation takes advantage of

   This document builds on the MPLS
   encapsulation.  By using the specification of MPLS over UDP and IP in
   [RFC7510], the T-Label(s) shown in Figure 15 in Section 6.2 can be
   replaced by UDP and IP, resulting in the following encapsulation:

      +---------------------------------+
      |                                 |
      |           DetNet Flow           |
      |         Payload  Packet         |
      |                                 |
      +---------------------------------+ <--\
      |       DetNet Control Word       |    |
      +---------------------------------+    +--> DetNet data plane
      |             S-Label             |    |    MPLS encapsulation
      +---------------------------------+ <--/
      |           UDP Header            |
      +---------------------------------+
      |           IP Header             |
      +---------------------------------+
      |           Data-Link             |
      +---------------------------------+
      |           Physical              |
      +---------------------------------+

                   Figure 19: IP Encapsulation of DetNet

   Where the UDP header is used as defined in Section 3 of [RFC7510].

   As in Section 6.2, the S-Label is used to identify a DetNet flow to
   the peer node that processes it, IP
   defined in this case [RFC7510].  It replaces the node addressed by the IP Header F-Label(s) used in Figure 19.
   Section 6.2 with UDP and IP headers.  The S-Label UDP and IP header
   information is allocated from the
   receiving node?s platform label space [RFC3031].

   In ingress Edge Nodes, the encapsulation in Figure 19 will be imposed
   on Detnet Flow Payload Packets as received from used to identify DetNet End Systems,
   and the flows, including member flows,
   per [I-D.ietf-detnet-dp-sol-ip].  The resulting encapsulation will be removed is
   shown in egress Edge Nodes as they
   transmit the Payload Packets to the End Systems. Figure 17.

   Note that this encapsulation works equally well with IPv4 and IPv6.

   This encapsulation can also be used in conjunction with segment
   routing as specified in [I-D.ietf-spring-segment-routing-mpls].  In
   this case, the T-Label(s) in Figure 19 should be retained, and at
   each hop, the top T-label is popped and mapped to a corresponding
   UDP/IP tunnel, resulting in the following encapsulation:

      +---------------------------------+
      |                                 |
      |         DetNet Flow App-Flow         |
      |         Payload  Packet         |
      |                                 |
      +---------------------------------+ <--\
      |       DetNet Control Word       |    |
      +---------------------------------+    +--> DetNet data plane
      |             S-Label             |    |    MPLS encapsulation
      +---------------------------------+ <--/
       |           T-Label(s)            |
       +---------------------------------+ <--X.
      |           UDP Header            |    |
      +---------------------------------+    +--> DetNet data plane
      |           IP Header             |    |    IP encapsulation
      +---------------------------------+ <--/
      |           Data-Link             |
      +---------------------------------+
      |           Physical              |
      +---------------------------------+

                Figure 20: 17: IP Encapsulation of DetNet with MPLS-SR

   Again, the UDP header is MPLS

   d-CW and and S-Labels are used as defined in Section 3 6.2 and are not
   modified by this section.

   To support outgoing DetNet MPLS over IP, an implementation MUST
   support the provisioning of [RFC7510]. IP/UDP header information in place of
   sets of F-Labels.  Note that if required in both the case multiple sets of IP Encapsulation F-Labels can be
   provisioned to support PRF on transmitted DetNet flows and therefore,
   when PRF is supported, multiple IP/UDP headers MAY be provisioned.
   When multiple IP/UDP headers are provisioned for a particular
   outgoing app-flow, a copy of the outgoing packet, including the
   pushed S-Label, MUST be made for each.  The headers for each outgoing
   packet MUST be based on the configuration information and as defined
   in [RFC7510], with one exception.  The one exceptions is that the UDP
   Source Port value MUST be set to uniquely identify the DetNet
   (forwarding sub-layer) flow.  The packet MUST then be handed as a
   DetNet
   Figure 19, and of IP Encapsulation packet, per [I-D.ietf-detnet-dp-sol-ip].

   To support receive processing an implementation MUST also support the
   provisioning of DetNet with MPLS-SR Figure 20,
   it received IP/UDP header information.  When S-Labels
   are taken from platform label space, all that is required is possible to omit
   provision that receiving IP/UDP encapsulated DetNet MPLS packets is
   permitted.  Once the UDP IP/UDP header if required.  Operation of MPLS
   directly over IP is described in [RFC4023].  In this case DetNet
   Service can stripped, the S-label uniquely
   identifies the app-flow.  When S-Labels are not taken from platform
   label space, IP/UDP header information MUST be provided provisioned.  The
   provisioned information MUST then be used to identify incoming app-
   flows based on a per IP flow basis as described in
   [I-D.ietf-detnet-dp-sol-ip].

12. the combination of S-Label and incoming IP/UDP header.
   Normal receive processing, including PEOF can then take place.

10.  Security considerations Considerations

   The security considerations of DetNet in general are discussed in
   [I-D.ietf-detnet-architecture] and [I-D.sdt-detnet-security].  Other
   security considerations will be added in a future version of this
   draft.

13.

11.  IANA considerations Considerations

   This document makes no IANA requests.

14.

12.  Contributors

   RFC7322 limits the number of authors listed on the front page of a
   draft to a maximum of 5, far fewer than the 20 many individuals below
   who made important contributions to this draft.  The editor wishes to
   thank and acknowledge each of the following authors for contributing
   text to this draft.  See also Section 15. 13.

      Loa Andersson
      Huawei
      Email: loa@pi.nu

      Yuanlong Jiang
      Huawei
      Email: jiangyuanlong@huawei.com

      Norman Finn
      Huawei
      3101 Rio Way
      Spring Valley, CA  91977
      USA
      Email: norman.finn@mail01.huawei.com

      Janos Farkas
      Ericsson
      Magyar Tudosok krt. 11.
      Budapest  1117
      Hungary
      Email: janos.farkas@ericsson.com

      Carlos J. Bernardos
      Universidad Carlos III de Madrid
      Av. Universidad, 30
      Leganes, Madrid  28911
      Spain
      Email: cjbc@it.uc3m.es

      Tal Mizrahi
      Marvell
      6 Hamada st.
      Yokneam
      Israel
      Email: talmi@marvell.com

      Lou Berger
      LabN Consulting, L.L.C.
      Email: lberger@labn.net

      Stewart Bryant
      Huawei Technologies
      Email: stewart.bryant@gmail.com

      Mach Chen
      Huawei Technologies
      Email: mach.chen@huawei.com

15.

      Andrew G. Malis
      Huawei Technologies
      Email: agmalis@gmail.com

13.  Acknowledgements

   The author(s) ACK and NACK.

   The following people were part of the DetNet Data Plane Solution
   Design Team:

      Jouni Korhonen

      Janos Farkas

      Norman Finn

      Balazs Varga

      Loa Andersson

      Tal Mizrahi

      David Mozes

      Yuanlong Jiang
      Andrew Malis

      Carlos J.  Bernardos

   The DetNet chairs serving during the DetNet Data Plane Solution
   Design Team:

      Lou Berger

      Pat Thaler

   Thanks for Stewart Bryant for his extensive review of the previous
   versions of the document.

16.

14.  References

16.1.

14.1.  Normative references

   [I-D.ietf-spring-segment-routing-mpls]
              Bashandy, A., Filsfils, C., Previdi, S., Decraene, B.,
              Litkowski, S., and R. Shakir, "Segment Routing with MPLS
              data plane", draft-ietf-spring-segment-routing-mpls-14
              (work in progress), June 2018. References

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119,
              DOI 10.17487/RFC2119, March 1997,
              <https://www.rfc-editor.org/info/rfc2119>.

   [RFC2211]  Wroclawski, J., "Specification of the Controlled-Load
              Network Element Service", RFC 2211, DOI 10.17487/RFC2211,
              September 1997, <https://www.rfc-editor.org/info/rfc2211>.

   [RFC2212]  Shenker, S., Partridge, C., and R. Guerin, "Specification
              of Guaranteed Quality of Service", RFC 2212,
              DOI 10.17487/RFC2212, September 1997,
              <https://www.rfc-editor.org/info/rfc2212>.

   [RFC2474]  Nichols, K., Blake, S., Baker, F., and D. Black,
              "Definition of the Differentiated Services Field (DS
              Field) in the IPv4 and IPv6 Headers", RFC 2474,
              DOI 10.17487/RFC2474, December 1998,
              <https://www.rfc-editor.org/info/rfc2474>.

   [RFC3031]  Rosen, E., Viswanathan, A., and R. Callon, "Multiprotocol
              Label Switching Architecture", RFC 3031,
              DOI 10.17487/RFC3031, January 2001,
              <https://www.rfc-editor.org/info/rfc3031>.

   [RFC3032]  Rosen, E., Tappan, D., Fedorkow, G., Rekhter, Y.,
              Farinacci, D., Li, T., and A. Conta, "MPLS Label Stack
              Encoding", RFC 3032, DOI 10.17487/RFC3032, January 2001,
              <https://www.rfc-editor.org/info/rfc3032>.

   [RFC3168]  Ramakrishnan, K., Floyd, S., and D. Black, "The Addition
              of Explicit Congestion Notification (ECN) to IP",
              RFC 3168, DOI 10.17487/RFC3168, September 2001,
              <https://www.rfc-editor.org/info/rfc3168>.

   [RFC3209]  Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V.,
              and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP
              Tunnels", RFC 3209, DOI 10.17487/RFC3209, December 2001,
              <https://www.rfc-editor.org/info/rfc3209>.

   [RFC3270]  Le Faucheur, F., Wu, L., Davie, B., Davari, S., Vaananen,
              P., Krishnan, R., Cheval, P., and J. Heinanen, "Multi-
              Protocol Label Switching (MPLS) Support of Differentiated
              Services", RFC 3270, DOI 10.17487/RFC3270, May 2002,
              <https://www.rfc-editor.org/info/rfc3270>.

   [RFC3443]  Agarwal, P. and B. Akyol, "Time To Live (TTL) Processing
              in Multi-Protocol Label Switching (MPLS) Networks",
              RFC 3443, DOI 10.17487/RFC3443, January 2003,
              <https://www.rfc-editor.org/info/rfc3443>.

   [RFC3473]  Berger, L., Ed., "Generalized Multi-Protocol Label
              Switching (GMPLS) Signaling Resource ReserVation Protocol-
              Traffic Engineering (RSVP-TE) Extensions", RFC 3473,
              DOI 10.17487/RFC3473, January 2003,
              <https://www.rfc-editor.org/info/rfc3473>.

   [RFC4023]  Worster, T., Rekhter, Y., and E. Rosen, Ed.,
              "Encapsulating MPLS in IP or Generic Routing Encapsulation
              (GRE)", RFC 4023, DOI 10.17487/RFC4023, March 2005,
              <https://www.rfc-editor.org/info/rfc4023>.

   [RFC4206]  Kompella, K. and Y. Rekhter, "Label Switched Paths (LSP)
              Hierarchy with Generalized Multi-Protocol Label Switching
              (GMPLS) Traffic Engineering (TE)", RFC 4206,
              DOI 10.17487/RFC4206, October 2005,
              <https://www.rfc-editor.org/info/rfc4206>.

   [RFC4385]  Bryant, S., Swallow, G., Martini, L., and D. McPherson,
              "Pseudowire Emulation Edge-to-Edge (PWE3) Control Word for
              Use over an MPLS PSN", RFC 4385, DOI 10.17487/RFC4385,
              February 2006, <https://www.rfc-editor.org/info/rfc4385>.

   [RFC5085]  Nadeau, T., Ed. and C. Pignataro, Ed., "Pseudowire Virtual
              Circuit Connectivity Verification (VCCV): A Control
              Channel for Pseudowires", RFC 5085, DOI 10.17487/RFC5085,
              December 2007, <https://www.rfc-editor.org/info/rfc5085>.

   [RFC5129]  Davie, B., Briscoe, B., and J. Tay, "Explicit Congestion
              Marking in MPLS", RFC 5129, DOI 10.17487/RFC5129, January
              2008, <https://www.rfc-editor.org/info/rfc5129>.

   [RFC5462]  Andersson, L. and R. Asati, "Multiprotocol Label Switching
              (MPLS) Label Stack Entry: "EXP" Field Renamed to "Traffic
              Class" Field", RFC 5462, DOI 10.17487/RFC5462, February
              2009, <https://www.rfc-editor.org/info/rfc5462>.

   [RFC6003]  Papadimitriou, D., "Ethernet Traffic Parameters",
              RFC 6003, DOI 10.17487/RFC6003, October 2010,
              <https://www.rfc-editor.org/info/rfc6003>.

   [RFC7510]  Xu, X., Sheth, N., Yong, L., Callon, R., and D. Black,
              "Encapsulating MPLS in UDP", RFC 7510,
              DOI 10.17487/RFC7510, April 2015,
              <https://www.rfc-editor.org/info/rfc7510>.

   [RFC8174]  Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
              2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
              May 2017, <https://www.rfc-editor.org/info/rfc8174>.

16.2.

14.2.  Informative references References

   [G.8275.1]
              International Telecommunication Union, "Precision time
              protocol telecom profile for phase/time synchronization
              with full timing support from the network", ITU-T
              G.8275.1/Y.1369.1 G.8275.1, June 2016,
              <https://www.itu.int/rec/T-REC-G.8275.1/en>.

   [G.8275.2]
              International Telecommunication Union, "Precision time
              protocol telecom profile for phase/time synchronization
              with partial timing support from the network", ITU-T
              G.8275.2/Y.1369.2 G.8275.2, June 2016,
              <https://www.itu.int/rec/T-REC-G.8275.2/en>.

   [I-D.ietf-detnet-architecture]
              Finn, N., Thubert, P., Varga, B., and J. Farkas,
              "Deterministic Networking Architecture", draft-ietf-
              detnet-architecture-08
              detnet-architecture-11 (work in progress), September 2018.

   [I-D.ietf-detnet-dp-alt] February 2019.

   [I-D.ietf-detnet-dp-sol-ip]
              Korhonen, J., Varga, B., "DetNet IP Data Plane
              Encapsulation", 2018.

   [I-D.ietf-detnet-flow-information-model]
              Farkas, J., Mirsky, G., Thubert, P.,
              Zhuangyan, Varga, B., Cummings, R., and Y. Jiang, "DetNet
              Flow Information Model", draft-ietf-detnet-flow-
              information-model-03 (work in progress), March 2019.

   [I-D.ietf-pce-pcep-extension-for-pce-controller]
              Zhao, Q., Li, Z., Negi, M., and L. Berger, "DetNet Data Plane Protocol C. Zhou, "PCEP Procedures
              and Solution Alternatives", draft-ietf-detnet-dp-alt-00 Protocol Extensions for Using PCE as a Central
              Controller (PCECC) of LSPs", draft-ietf-pce-pcep-
              extension-for-pce-controller-01 (work in progress), October 2016.

   [I-D.ietf-detnet-dp-sol-ip]
              Korhonen, J., Varga,
              February 2019.

   [I-D.ietf-spring-segment-routing-mpls]
              Bashandy, A., Filsfils, C., Previdi, S., Decraene, B., "DetNet IP Data Plane
              Encapsulation",
              Litkowski, S., and R. Shakir, "Segment Routing with MPLS
              data plane", draft-ietf-spring-segment-routing-mpls-18
              (work in progress), December 2018.

   [I-D.sdt-detnet-security]
              Mizrahi, T., Grossman, E., Hacker, A., Das, S.,
              "Deterministic Networking (DetNet) Security
              Considerations, draft-sdt-detnet-security, work in
              progress", 2017.

   [IEEE1588]
              IEEE, "IEEE 1588 Standard for a Precision Clock
              Synchronization Protocol for Networked Measurement and
              Control Systems Version 2", 2008.

   [IEEE8021CB]
              Finn, N., "Draft Standard for Local and metropolitan area
              networks - Seamless Redundancy", IEEE P802.1CB
              /D2.1 P802.1CB, December 2015,
              <http://www.ieee802.org/1/files/private/cb-drafts/
              d2/802-1CB-d2-1.pdf>.

   [IEEE8021Q]
              IEEE 802.1, "Standard for Local and metropolitan area
              networks--Bridges and Bridged Networks (IEEE Std 802.1Q-
              2014)", 2014, <http://standards.ieee.org/about/get/>.

   [RFC2205]  Braden, R., Ed., Zhang, L., Berson, S., Herzog, S., and S.
              Jamin, "Resource ReSerVation Protocol (RSVP) -- Version 1
              Functional Specification", RFC 2205, DOI 10.17487/RFC2205,
              September 1997, <https://www.rfc-editor.org/info/rfc2205>.

   [RFC2474]  Nichols, K., Blake, S., Baker, F., and D. Black,
              "Definition of the Differentiated Services Field (DS
              Field) in the IPv4 and IPv6 Headers", RFC 2474,
              DOI 10.17487/RFC2474, December 1998,
              <https://www.rfc-editor.org/info/rfc2474>.

   [RFC3272]  Awduche, D., Chiu, A., Elwalid, A., Widjaja, I., and X.
              Xiao, "Overview and Principles of Internet Traffic
              Engineering", RFC 3272, DOI 10.17487/RFC3272, May 2002,
              <https://www.rfc-editor.org/info/rfc3272>.

   [RFC3985]  Bryant, S., Ed. and P. Pate, Ed., "Pseudo Wire Emulation
              Edge-to-Edge (PWE3) Architecture", RFC 3985,
              DOI 10.17487/RFC3985, March 2005,
              <https://www.rfc-editor.org/info/rfc3985>.

   [RFC4448]  Martini, L., Ed., Rosen, E., El-Aawar, N., and G. Heron,
              "Encapsulation Methods for Transport of Ethernet over MPLS
              Networks", RFC 4448, DOI 10.17487/RFC4448, April 2006,
              <https://www.rfc-editor.org/info/rfc4448>.

   [RFC4872]  Lang, J., Ed., Rekhter, Y., Ed., and D. Papadimitriou,
              Ed., "RSVP-TE Extensions in Support of End-to-End
              Generalized Multi-Protocol Label Switching (GMPLS)
              Recovery", RFC 4872, DOI 10.17487/RFC4872, May 2007,
              <https://www.rfc-editor.org/info/rfc4872>.

   [RFC4873]  Berger, L., Bryskin, I., Papadimitriou, D., and A. Farrel,
              "GMPLS Segment Recovery", RFC 4873, DOI 10.17487/RFC4873,
              May 2007, <https://www.rfc-editor.org/info/rfc4873>.

   [RFC4875]  Aggarwal, R., Ed., Papadimitriou, D., Ed., and S.
              Yasukawa, Ed., "Extensions to Resource Reservation
              Protocol - Traffic Engineering (RSVP-TE) for Point-to-
              Multipoint TE Label Switched Paths (LSPs)", RFC 4875,
              DOI 10.17487/RFC4875, May 2007,
              <https://www.rfc-editor.org/info/rfc4875>.

   [RFC5440]  Vasseur, JP., Ed. and JL. Le Roux, Ed., "Path Computation
              Element (PCE) Communication Protocol (PCEP)", RFC 5440,
              DOI 10.17487/RFC5440, March 2009,
              <https://www.rfc-editor.org/info/rfc5440>.

   [RFC5586]  Bocci, M., Ed., Vigoureux, M., Ed., and S. Bryant, Ed.,
              "MPLS Generic Associated Channel", RFC 5586,
              DOI 10.17487/RFC5586, June 2009,
              <https://www.rfc-editor.org/info/rfc5586>.

   [RFC5654]  Niven-Jenkins, B., Ed., Brungard, D., Ed., Betts, M., Ed.,
              Sprecher, N., and S. Ueno, "Requirements of an MPLS
              Transport Profile", RFC 5654, DOI 10.17487/RFC5654,
              September 2009, <https://www.rfc-editor.org/info/rfc5654>.

   [RFC5921]  Bocci, M., Ed., Bryant, S., Ed., Frost, D., Ed., Levrau,
              L., and L. Berger, "A Framework for MPLS in Transport
              Networks", RFC 5921, DOI 10.17487/RFC5921, July 2010,
              <https://www.rfc-editor.org/info/rfc5921>.

   [RFC6003]  Papadimitriou, D., "Ethernet Traffic Parameters",
              RFC 6003, DOI 10.17487/RFC6003, October 2010,
              <https://www.rfc-editor.org/info/rfc6003>.

   [RFC6006]  Zhao, Q., Ed., King, D., Ed., Verhaeghe, F., Takeda, T.,
              Ali, Z., and J. Meuric, "Extensions to the Path
              Computation Element Communication Protocol (PCEP) for
              Point-to-Multipoint Traffic Engineering Label Switched
              Paths", RFC 6006, DOI 10.17487/RFC6006, September 2010,
              <https://www.rfc-editor.org/info/rfc6006>.

   [RFC6073]  Martini, L., Metz, C., Nadeau, T., Bocci, M., and M.
              Aissaoui, "Segmented Pseudowire", RFC 6073,
              DOI 10.17487/RFC6073, January 2011,
              <https://www.rfc-editor.org/info/rfc6073>.

   [RFC6387]  Takacs, A., Berger, L., Caviglia, D., Fedyk, D., and J.
              Meuric, "GMPLS Asymmetric Bandwidth Bidirectional Label
              Switched Paths (LSPs)", RFC 6387, DOI 10.17487/RFC6387,
              September 2011, <https://www.rfc-editor.org/info/rfc6387>.

   [RFC6790]  Kompella, K., Drake, J., Amante, S., Henderickx, W., and
              L. Yong, "The Use of Entropy Labels in MPLS Forwarding",
              RFC 6790, DOI 10.17487/RFC6790, November 2012,
              <https://www.rfc-editor.org/info/rfc6790>.

   [RFC7551]  Zhang, F., Ed., Jing, R., and R. Gandhi, Ed., "RSVP-TE
              Extensions for Associated Bidirectional Label Switched
              Paths (LSPs)", RFC 7551, DOI 10.17487/RFC7551, May 2015,
              <https://www.rfc-editor.org/info/rfc7551>.

   [RFC7950]  Bjorklund, M., Ed., "The YANG 1.1 Data Modeling Language",
              RFC 7950, DOI 10.17487/RFC7950, August 2016,
              <https://www.rfc-editor.org/info/rfc7950>.

   [RFC8040]  Bierman, A., Bjorklund, M., and K. Watsen, "RESTCONF
              Protocol", RFC 8040, DOI 10.17487/RFC8040, January 2017,
              <https://www.rfc-editor.org/info/rfc8040>.

   [RFC8169]  Mirsky, G., Ruffini, S., Gray, E., Drake, J., Bryant, S.,
              and A. Vainshtein, "Residence Time Measurement in MPLS
              Networks", RFC 8169, DOI 10.17487/RFC8169, May 2017,
              <https://www.rfc-editor.org/info/rfc8169>.

Appendix A.  Example of DetNet data plane operation Data Plane Operation

   [Editor's note: Add a simplified example of DetNet data plane and how
   labels etc work in the case of MPLS-based PSN and utilizing PREOF.
   The figure is subject to change depending on the further DT decisions
   on the label handling..]

Authors' Addresses

   Jouni Korhonen (editor)

   Email: jouni.nospam@gmail.com
   Balazs Varga (editor)
   Ericsson
   Magyar Tudosok krt. 11.
   Budapest  1117
   Hungary

   Email: balazs.a.varga@ericsson.com