L2VPN Workgroup                                          A. Sajassi, Ed.
INTERNET-DRAFT                                                  S. Salam
Intended Status: Standards Track                               S. Thoria
                                                                   Cisco
                                                                J. Drake
                                                                 Juniper
                                                              J. Rabadan
                                                                   Nokia
                                                                 L. Yong
                                                                  Huawei

Expires: August 8, 2017                                 February 8, 2017 January 2, 2019                                    July 2, 2018

                Integrated Routing and Bridging in EVPN
            draft-ietf-bess-evpn-inter-subnet-forwarding-03
            draft-ietf-bess-evpn-inter-subnet-forwarding-04

Abstract

   EVPN provides an extensible and flexible multi-homing VPN solution
   over an MPLS/IP network for intra-subnet connectivity among hosts/VMs over an MPLS/IP
   network. Tenant
   Systems and End Devices that can be physical or virtual. However,
   there are scenarios in for which there is a need for a dynamic and
   efficient inter-subnet
   forwarding connectivity among hosts/VMs across different IP subnets is required, these Tenant Systems and
   End Devices while maintaining the multi-homing capabilities of EVPN.
   This document describes an Integrated Routing and Bridging (IRB)
   solution based on EVPN to address such requirements.

Status of this Memo

   This Internet-Draft is submitted to IETF in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF), its areas, and its working groups.  Note that
   other groups may also distribute working documents as
   Internet-Drafts.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   The list of current Internet-Drafts can be accessed at
   http://www.ietf.org/1id-abstracts.html

   The list of Internet-Draft Shadow Directories can be accessed at
   http://www.ietf.org/shadow.html

Copyright and License Notice

   Copyright (c) 2014 IETF Trust and the persons identified as the
   document authors. All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (http://trustee.ietf.org/license-info) in effect on the date of
   publication of this document. Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document. Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

Table of Contents

   1  Introduction  . . . . . . . . . . . . . . . . . . . . . . . . .  5  6
   2  Inter-Subnet Forwarding Scenarios EVPN PE Model for IRB Operation  . . . . . . . . . . . . . . .  6
     2.1 Switching among IP subnets within a DC .  7
   3  Symmetric and Asymmetric IRB  . . . . . . . . . .  7
     2.2 Switching among IP subnets in different DCs without GW . . .  8
     2.3 Switching among IP subnets in different DCs with GW . . . .  8
     2.4 Switching among IP subnets spread across IP-VPN
     3.1 IRB Interface and EVPN
         networks with GW its MAC & IP addresses . . . . . . . . . . 11
     3.2 Symmetric IRB Procedures . . . . . . . . . . . .  8
   3 Default L3 Gateway for Tenant System . . . . . . 12
       3.2.1 Control Plane - Ingress PE . . . . . . . .  9
     3.1 Homogeneous Environment . . . . . . . 12
       3.2.2 Control Plane - Egress PE  . . . . . . . . . . . .  9
     3.2 Heterogeneous Environment . . . 13
       3.2.3 Data Plane - Ingress PE  . . . . . . . . . . . . . . 10
   4  Operational Models for Asymmetric Inter-Subnet Forwarding . . 14
       3.2.4 Data Plane - Egress PE . 10
     4.1 Among EVPN NVEs within a DC . . . . . . . . . . . . . . . . 10
     4.2 Among EVPN NVEs in Different DCs Without GW 14
     3.3 Asymmetric IRB Procedures  . . . . . . . . 11
     4.3 Among EVPN NVEs in Different DCs with GW . . . . . . . . . 15
       3.3.1 Control Plane - Ingress PE . . 13
     4.4 Among IP-VPN Sites and EVPN NVEs with GW . . . . . . . . . . 14
     4.5 Use of Centralized Gateway . . . 15
       3.3.2 Control Plane - Egress PE  . . . . . . . . . . . . . . . 15
   5 Operational Models for Symmetric Inter-Subnet Forwarding
       3.3.3 Data Plane - Ingress PE  . . . . . . . 16
     5.1 IRB forwarding on NVEs for Tenant Systems . . . . . . . . . 16
       5.1.1 Control
       3.3.4 Data Plane Operation - Egress PE . . . . . . . . . . . . . . . . . 17
       5.1.2 Data Plane Operation - Inter Subnet
   4 BGP Encoding . . . . . . . . . . 18
       5.1.3 TS Move Operation . . . . . . . . . . . . . . . . 17
     4.1 Router's MAC Extended Community  . . . 19
     5.2 IRB forwarding on NVEs for Subnets behind . . . . . . . . . . . 17
   5 Operational Models for Symmetric Inter-Subnet Forwarding . . . . 18
     5.1 IRB forwarding on NVEs for Tenant Systems  . . 20
       5.2.1 . . . . . . . 18
       5.1.1 Control Plane Operation  . . . . . . . . . . . . . . . . 22
       5.2.2 19
       5.1.2 Data Plane Operation - Inter Subnet  . . . . . . . . . . 21
       5.1.3 TS Move Operation  . . . . . . . . . . 23
   6 BGP Encoding . . . . . . . . . 22
     5.2 IRB forwarding on NVEs for Subnets behind Tenant Systems . . 23
       5.2.1 Control Plane Operation  . . . . . . . . . . . . . . . . 24
     6.1 Router's MAC Extended Community
       5.2.2 Data Plane Operation . . . . . . . . . . . . . . 24 . . . . 25
   6  Inter-Subnet DCI Scenarios  . . . . . . . . . . . . . . . . . . 26
     6.1 Switching among IP subnets in different DCs without GW . . . 27
     6.2 Switching among IP subnets in different DCs with GW  . . . . 29
   7 TS Mobility  . . . . . . . . . . . . . . . . . . . . . . . . . . 24 31
     7.1 TS Mobility & Optimum Forwarding for TS Outbound Traffic . . 24 31
     7.2 TS Mobility & Optimum Forwarding for TS Inbound Traffic  . . 24 31
       7.2.1 Mobility without Route Aggregation . . . . . . . . . . . 25 31
   8  Acknowledgements  . . . . . . . . . . . . . . . . . . . . . . . 25 32
   9  Security Considerations . . . . . . . . . . . . . . . . . . . . 25 32
   10  IANA Considerations  . . . . . . . . . . . . . . . . . . . . . 25 32
   11  References . . . . . . . . . . . . . . . . . . . . . . . . . . 25 32
     11.1  Normative References . . . . . . . . . . . . . . . . . . . 25 32
     11.2  Informative References . . . . . . . . . . . . . . . . . . 26 32
   12  Contributors . . . . . . . . . . . . . . . . . . . . . . . . . 26 33
   Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 27 33

Terminology

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
   "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119]. BCP
   14 [RFC2119] [RFC8174] when, and only when, they appear in all
   capitals, as shown here.

   AC: Attachment Circuit.

   ARP: Address Resolution Protocol.

   BD: Broadcast Domain: In a bridged network, the broadcast domain
   corresponds to a Virtual LAN (VLAN), where a VLAN is typically
   represented by a single VLAN ID (VID) but can be represented by
   several VIDs where Shared VLAN Learning (SVL) is used Domain. As per [802.1Q].

   EVI : An EVPN instance spanning the Provider Edge (PE) devices
   participating in that EVPN

   IRB: Integrated Routing and Bridging

   MAC-VRF: A Virtual Routing and Forwarding table for Media Access
   Control (MAC) addresses on a PE for [RFC7432], an EVI

   Bridge Table: An instantiation consists of a broadcast domain on a MAC-VRF

   IP-VRF: A Virtual Routing single
   or multiple BDs. In case of VLAN-bundle and Forwarding table for IP addresses on VLAN-based service models
   (see [RFC7432]), a
   PE that BD is associated with one or more EVIs

   IRB Interface: A virtual interface that connects a bridge table in a
   MAC-VRF equivalent to an IP-VRF in EVI. In case of VLAN-aware
   bundle service model, an NVE.

   NVE: Network Virtualization Endpoint

   TS: Tenant System EVI contains multiple BDs. Also, in this
   document, BD and subnet are equivalent terms.

   BD Route Target: refers to the Broadcast Domain assigned Route Target
   [RFC4364]. In case of VLAN-aware bundle service model, all the BD
   instances in the MAC-VRF share the same Route Target.

   BT: Bridge Table. The instantiation of a BD in a MAC-VRF, as per
   [RFC7432].

   DGW: Data Center Gateway.

   Ethernet A-D route: Ethernet Auto-Discovery (A-D) route, as per
   [RFC7432].

   Ethernet NVO tunnel: It refers to Network Virtualization Overlay tunnels
   with Ethernet payload. Example Examples of this type of tunnels are
   VxLAN and NvGRE. VXLAN or
   GENEVE.

   EVI: EVPN Instance spanning the NVE/PE devices that are participating
   on that EVPN, as per [RFC7432].

   EVPN: Ethernet Virtual Private Networks, as per [RFC7432].

   GRE: Generic Routing Encapsulation.

   GW IP: Gateway IP Address.

   IPL: IP Prefix Length.

   IP NVO tunnel: It it refers to Network Virtualization Overlay tunnels
   with IP payload (no MAC header in the payload). Examples of IP NVO
   tunnels are VxLAN GPE or MPLSoGRE (both with IP payload).

1  Introduction

   EVPN provides an extensible and flexible multi-homing

   IP-VRF: A VPN solution Routing and Forwarding table for intra-subnet connectivity among Tenant Systems (TS's) over an
   MPLS/IP network; where, an IP subnet is represented by an EVI for a
   VLAN-based service or by routes on an <EVI, VLAN> for a VLAN-aware bundle
   service. However, there are scenarios where, in addition to intra-
   subnet forwarding, inter-subnet forwarding is required among TS's
   across different
   NVE/PE. The IP subnets at routes could be populated by EVPN PE nodes, and IP-VPN address
   families. An IP-VRF is also known as EVPN NVE
   nodes throughout this document, while maintaining the multi-homing
   capabilities an instantiation of EVPN. This document describes a layer 3 VPN in an
   NVE/PE.

   IRB: Integrated Routing and Bridging (IRB) solution based on EVPN interface. It connects an IP-VRF
   to address such
   requirements.

   The inter-subnet communication is traditionally achieved at
   centralized a BD (or subnet).

   MAC-VRF: A Virtual Routing and Forwarding table for Media Access
   Control (MAC) addresses on an NVE/PE, as per [RFC7432]. A MAC-VRF is
   also an instantiation of an EVI in an NVE/PE.

   ML: MAC address length.

   ND: Neighbor Discovery Protocol.

   NVE: Network Virtualization Edge.

   GENEVE: Generic Network Virtualization Encapsulation, [GENEVE].

   NVO: Network Virtualization Overlays.

   RT-2: EVPN route type 2, i.e., MAC/IP advertisement route, as defined
   in [RFC7432].

   RT-5: EVPN route type 5, i.e., IP Prefix route. As defined in Section
   3 of [EVPN-PREFIX].

   SBD: Supplementary Broadcast Domain. A BD that does not have any ACs,
   only IRB interfaces, and it is used to provide connectivity among all
   the IP-VRFs of the tenant. The SBD is only required in IP-VRF- to-IP-
   VRF use-cases (see Section 4.4.).

   SN: Subnet.

   TS: Tenant System.

   VA: Virtual Appliance.

   VNI: Virtual Network Identifier. As in [RFC8365], the term is used as
   a representation of a 24-bit NVO instance identifier, with the
   understanding that VNI will refer to a VXLAN Network Identifier in
   VXLAN, or Virtual Network Identifier in GENEVE, etc. unless it is
   stated otherwise.

   VTEP: VXLAN Termination End Point, as in [RFC7348].

   VXLAN: Virtual Extensible LAN, as in [RFC7348].

   This document also assumes familiarity with the terminology of
   [RFC7432], [RFC8365] and [RFC7365].

1  Introduction

   EVPN provides an extensible and flexible multi-homing VPN solution
   over an MPLS/IP network for intra-subnet connectivity among Tenant
   Systems (TS's) and End Devices that can be physical or virtual; where
   an IP subnet is represented by an EVI for a VLAN-based service or by
   an <EVI, VLAN> for a VLAN-aware bundle service. However, there are
   scenarios for which there is a need for a dynamic and efficient
   inter-subnet connectivity among these Tenant Systems and End Devices
   while maintaining the multi-homing capabilities of EVPN. This
   document describes an Integrated Routing and Bridging (IRB) solution
   based on EVPN to address such requirements.

   The inter-subnet communication is traditionally achieved at
   centralized L3 Gateway (L3GW) nodes where all the inter-subnet
   communication policies are enforced. When two Tenant Systems (TS's)
   belonging to two different subnets connected to the same PE node,
   wanted to talk to communicate with each other, their traffic needed to be
   back hauled from the PE node all the way to the centralized gateway
   nodes where inter-subnet switching is performed and then back to the
   PE node. For today's large multi-tenant data center, this scheme is
   very inefficient and sometimes impractical.

   In order to overcome the drawback of centralized L3GW approach, IRB
   functionality is needed on the PE nodes (i.e., NVE devices) as close (also referred to TS as possible EVPN
   NVEs) attached to TS's in order to avoid hair pinning inefficient forwarding of user traffic
   unnecessarily.  Under this design, all
   tenant traffic between hosts attached
   to one NVE can be routed (i.e., avoid back-hauling and hair-pinning). A PE with
   IRB capability, can not only locally bridged locally, thus avoiding traffic
   hair-pinning issue of the centralized L3GW.

   There can be scenarios where both centralized and distributed
   approaches may be preferred simultaneously. For example, to allow
   NVEs to switch inter-subnet tenant intra-subnet
   traffic belonging to one but also can locally route the tenant or one
   security zone locally; whereas, to back haul inter-subnet traffic
   belonging to two different tenants or security zones to on
   a packet by packet basis thus meeting the
   centralized gateway nodes requirements for both intra
   and perform switching there after the inter-subnet forwarding and avoiding non-optimum traffic is subjected to Firewall (FW) or Deep Packet Inspection
   (DPI).
   forwarding associate with centralized L3GW approach.

   Some TS's run non-IP protocols in conjunction with their IP traffic.
   Therefore, it is important to handle both kinds of traffic optimally
   - e.g., to bridge non-IP and intra-subnet traffic and to route inter-
   subnet IP traffic. Therefore, the solution needs to meet the
   following requirements:

   R1: The solution MUST allow for inter-subnet traffic to be locally
   switched at NVEs.

   R2: The solution MUST allow for both inter-subnet and intra-subnet
   traffic belonging to the same tenant to be locally routed and bridged
   respectively. The solution MUST provide IP routing for inter-subnet
   traffic and Ethernet Bridging for intra-subnet traffic.

   R3:

   R2: The solution MUST support bridging of non-IP traffic.

   R4:

   R3: The solution MUST allow inter-subnet switching to be disabled on
   a per VLAN basis on NVEs PEs where the traffic needs to be back hauled to
   another node (i.e., for performing FW or DPI functionality).

2  Inter-Subnet Forwarding Scenarios

   The inter-subnet forwarding scenarios performed by an EVPN NVE can be
   divided into the following five categories. The last scenario, along
   with its corresponding solution, are described in [EVPN-IPVPN-
   INTEROP]. The first four scenarios are covered in PE Model for IRB Operation

   Since this document.

   1. Switching among IP subnets within a DC using EVPN

   2. Switching among IP subnets in different DCs using EVPN without GW

   3. Switching among IP subnets document discusses IRB operation in different DCs using EVPN with GW

   4. Switching among IP subnets spread across IP-VPN and relationship to EVPN networks
   with GW

   5. Switching among IP subnets spread across IP-VPN
   MAC-VRF, IP-VRF, EVI, Bridge Domain (BD), Bridge Table (BT), and EVPN networks
   without GW

   In IRB
   interfaces, it is important to understand the above scenario, relationship among
   these components. Therefore, the term "GW" refers following PE model is demonstrated
   below to a) describe these components and b) illustrate the case where a node
   situated at the WAN edge of the data center network behaves as a
   default gateway (GW) for all the destinations that are outside the
   data center. The absence of GW refers to the scenario where NVEs
   within a data center maintain individual (host) routes that are
   outside of the data center.

   In the case (4), the WAN edge node also performs route aggregation
   for all the destinations within its own data center, and acts as an
   interworking unit between EVPN and IP VPN (it implements both EVPN
   and IP-VPN functionality).

                             +---+    Enterprise Site 1
                             |PE1|----- H1
                             +---+
                               /
                         ,---------.             Enterprise Site 2
                       ,'           `.    +---+
        ,---------.  /(    MPLS/IP    )---|PE2|-----  H2
       '   DCN 3   `./ `.   Core    ,'    +---+
        `-+------+'     `-+------+'
        __/__           /
   relationship among them.

      +-------------------------------------------------------------+
      |                                                             |
      |              +------------------+                    IRB PE |
      | Attachment   | +------------------+                         |
      | Circuit(AC1) | |  +----------+    |                MPLS/NVO tnl
    ----------------------*Bridge    |    |                    +-----
      |              | |  |Table(BT1)|    |    +-----------+  / \    \
       :NVE4 :        +---+
      |              | |  |          *---------*           |<--> |Eth|
      |              | |  |Eth-Tag x |    |IRB1|           |  \ /    /
      |              | |  +----------+    |    |           |   +-----
      |              | |     ...          |    |  IP-VRF1  |        |
      |              | |  +----------+    |    |  RD2/RT2  |MPLS/NVO tnl
      |              | |  |Bridge    |    |    |           |   +-----
      |              | |  |Table(BT2)|    |IRB2|           |  / \
       '-----'   ,----|GW |.    \
      |              | |  |          *---------*           |<--> |IP |
    ----------------------*Eth-Tag y |    |    +-----------+  \ /    /
      |    ,'     +---+ `.      ,---------.
         TS6  (      DCN 1    )   ,'           `.
               `.           ,'   (      DCN 2    )
                 `-+------+'      `.           ,'
                   __/__            `-+------+'
                  :NVE1 :           __/__   __\__
                  '-----'          :NVE2 :  :NVE3 :  AC2         | |            '-----'  '-----'
                  TS1 TS2  +----------+    |                    +-----
      |              |
                                    TS3 TS4   TS5 |    MAC-VRF1      |                         |
      |              +-+    RD1/RT1       |                         |
      |                +------------------+                         |
      |                                                             |
      |                                                             |
      +-------------------------------------------------------------+

                      Figure 2: Interoperability Use-Cases

   In what follows, we will describe scenarios 1 through 4 in more
   detail.

2.1 Switching among IP subnets within 1: EVPN IRB PE Model

   A tenant needing IRB services on a DC

   In this scenario, connectivity PE, requires an IP Virtual Routing
   and Forwarding table (IP-V RF) along with one or more MAC Virtual
   Routing and Forwarding tables (MAC-VRFs). An IP-VRF, as defined in
   [RFC4364], is required between TS's the instantiation of an IPVPN in a PE. A MAC-VRF, as
   defined in [RFC7432], is the same
   data center, instantiation of an EVI (EVPN Instancce)
   in a PE. A MAC-VRF can consists of one or more Bridge Tables (BTs)
   where those hosts belong to different IP subnets. All
   these subnets belong each BT corresponds to a VLAN (broadcast domain - BD). If the same tenant or are part of
   service interface for the same IP
   VPN. Each subnet EVPN PE is associated with configured in VLAN-Based mode
   (i.e., section 6.1 of [RFC7432]), then there is only a single EVI (or <EVI,VLAN>)
   realized by a collection of MAC-VRFs (one BT per NVE) residing on the
   NVEs configured for that EVI.

   As an example, consider TS3 and TS5 of Figure 2 above. Assume that
   connectivity
   MAC-VRF (per EVI) - i.e., there is required between these two TS's where TS3 belongs to
   the IP-subnet 3 (SN3) whereas TS5 belongs to the IP-subnet 5 (SN5).
   Both SN3 and SN5 subnets belong to only one tenant VLAN per EVI.
   However, if the same tenant. NVE2 has an EVI3
   associated with service interface for the SN3 and this EVI EVPN PE is represented by a configured in
   VLAN-Aware Bundle mode (i.e., section 6.3 of [RFC7432]), then there
   are several BTs per MAC-VRF
   which (per EVI) - i.e., there are several
   tenant VLANs per EVI.

   Each BT is associated with an connected to a IP-VRF (for that tenant) via an a L3 interface called IRB
   interface. NVE3 respectively has an EVI5 associated with the SN5 and
   this EVI Since a single tenant subnet is typically (and in this
   document) represented by an MAC-VRF which is associated with the
   same IP-VRF via a different IRB interface.

2.2 Switching among IP subnets in different DCs without GW

   This case is similar to that of section 2.1 above albeit VLAN (and thus supported by a single BT),
   for the fact
   that the TS's belong to different data centers that are
   interconnected over a WAN (e.g. MPLS/IP PSN). The data centers in
   question here given tenant there are seamlessly interconnected to as many BTs as there are subnets and
   thus there are also as many IRB interfaces between the WAN, i.e., tenant IP-VRF
   and the WAN
   edge devices do not maintain any TS-specific addresses associated BTs as shown in the
   forwarding path - e.g., there PE model above.

   IP-VRF is no WAN edge GW(s) between these DCs.

   As an example, consider TS3 identified by its corresponding route target and TS6 of Figure 2 above. Assume that
   connectivity route
   distinguisher and MAC-VRF is required between these two TS's where TS3 belongs to
   the SN3 whereas TS6 belongs to the SN6. NVE2 has an EVI3 associated
   with SN3 also identified by its corresponding
   route target and NVE4 has route distinguisher. If operating in EVPN VLAN-Based
   mode, then a receiving PE that receives an EVI6 associated EVPN route with MAC-VRF
   route target can identify the SN6. Both SN3 and
   SN6 are part of the same IP-VRF.

2.3 Switching among IP subnets in different DCs with GW

   In this scenario, connectivity is required between TS's corresponding BT; however, if operating
   in different
   data centers, EVPN VLAN-Aware Bundle mode, then the receiving PE needs both the
   MAC-VRF route target and those hosts belong VLAN ID in order to different IP subnets. What
   makes this case different from that of Section 2.2 is that at least
   one of the data centers has a gateway as the WAN edge switch. Because
   of that, identify the NVE's IP-VRF  within that data center need not maintain
   (host) routes to individual TS's outside of that data center.

   As an example, consider a tenant with TS1
   corresponding BT.

3  Symmetric and TS5 of Figure 2 above.
   Assume that connectivity is required between these two TS's where TS1
   belongs to the SN1 whereas TS5 belongs to the SN5. NVE3 has an EVI5
   associated with the SN5 Asymmetric IRB

   This document defines and this EVI is represented by the MAC-VRF
   which is connected to the IP-VRF via an describes two types of IRB interface. NVE1 has an
   EVI1 associated with the SN1 solutions -
   namely symmetric and this EVI is represented by asymmetric IRB. In symmetric IRB as its name
   implies, the MAC-
   VRF which lookup operation is connected to the IP-VRF representing the same tenant.
   Due to the gateway symmetric at the edge of DCN 1, NVE1's IP-VRF does not need
   to have the both ingress and egress
   PEs - i.e., both ingress and egress PEs perform lookups on both TS's
   MAC and IP addresses - i.e., ingress PE performs lookup on
   destination TS's MAC address of TS5 but instead it has a default route in followed by its
   IP-VRF with the next-hop being the GW.

2.4 Switching among IP subnets spread across IP-VPN address and EVPN networks
   with GW

   In this scenario, connectivity is required between egress PE
   performs lookup on destination TS's IP address followed by its MAC
   address as depicted in a data
   center and hosts in an enterprise site that belongs to a given IP-
   VPN. The NVE within figure 2.

               Ingress PE                   Egress PE
         +-------------------+        +------------------+
         |                   |        |                  |
         |    +-> IP-VFF ----|---->---|-----> IP-VRF -+  |
         |    |              |        |               |  |
         |   BT1        BT2  |        |  BT3         BT2 |
         |    |              |        |               |  |
         |    ^              |        |               v  |
         |    |              |        |               |  |
         +-------------------+        +------------------+
              ^                                       |
              |                                       |
        TS1->-+                                       +->-TS2
                        Figure 2: Symmetric IRB

   In symmetric IRB as shown in figure-2, the data center inter-subnet forwarding
   between two PEs is an EVPN NVE, whereas the
   enterprise site has an IP-VPN PE. Furthermore, the data center in
   question has a gateway as done between their associated IP-VRFs. Therefore,
   the WAN edge switch. Because tunnel connecting these IP-VRFs can be either IP-only tunnel (in
   case of that, the
   NVE in MPLS or GENEVE encapsulation) or Ethernet NVO tunnel (in case
   of VxLAN encapsulation). If it is Ethernet NOV tunnel, the data center does not need to maintain individual TS's IP
   prefixes advertised by enterprise sites (by IP-VPN PEs).

   As
   packet is encapsulated in an example, consider end-station H1 and TS2 Ethernet header consisting of Figure 2. Assume
   that connectivity is required between the end-station ingress
   and the TS,
   where TS2 belongs egress PEs MAC addresses - i.e., there is no need for ingress PE
   to use the SN2 that destination TS's MAC address. Therefore, in symmetric IRB,
   there is realized using EVPN, whereas H1
   belongs no need for the ingress PE to an hold destination TS's IP VPN site connected to PE1 (PE1 and
   MAC association in its ARP table. Each PE participating in symmetric
   IRB only maintains an IP-VRF
   associated with that IP VPN). NVE1 has an EVI2 associated with ARP entries for locally connected hosts and
   maintain MAC-VRFs/BTs for only locally configured subnets.

   In asymmetric IRB, the
   SN2. Moreover, EVI2 on NVE1 lookup operation is connected to an IP-VRF associated with
   that IP VPN.  PE1 originates a VPN-IP route that covers H1. The
   gateway at asymmetric and the edge of DCN1 ingress
   PE performs three lookups; whereas the egress PE performs interworking function between
   IP-VPN and EVPN.  As a result of this, a default route in single
   lookup - i.e., the IP-VRF ingress PE performs lookups on destination TS's
   MAC address, followed by its IP address, followed by its MAC address
   again; whereas, the NVE1, pointing to the gateway as the next hop, and egress PE performs just a route to
   the TS2  (or maybe SN2) single lookup on the PE1's
   destination TS's MAC address as depicted in figure 3 below.

            Ingress PE                       Egress PE
         +-------------------+        +------------------+
         |                   |        |                  |
         |    +-> IP-VFF ->  |        |      IP-VRF are sufficient for the
   connectivity between H1 and TS2.      |
         |    |           |  |        |                  |
         |   BT1        BT2  |        |  BT3         BT2 |
         |    |           |  |        |              | | |
         |    |           +--|--->----|--------------+ | |
         |    |              |        |                v |
         +-------------------+        +----------------|-+
              ^                                        |
              |                                        |
        TS1->-+                                        +->-TS2
                        Figure 3: Asymmetric IRB

   In this scenario, the NVE1's IP-VRF
   does not need to maintain a route to H1 because it has the default
   route to asymmetric IRB as shown in figure-2, the gateway.

3 Default L3 Gateway for Tenant System

3.1 Homogeneous Environment

   This is an environment where all NVEs to which an EVPN instance could
   potentially be attached (or moved), perform inter-subnet switching. forwarding
   between two PEs is done between their associated MAC-VRFs/BTs.
   Therefore, the MPLS or NVO tunnel used for inter-subnet traffic can forwarding
   MUST be locally switched by of type Ethernet. Since at the EVPN
   NVE connecting egress PE only MAC lookup is
   performed (e.g., no IP lookup), the TS's belonging IP packet needs to different subnets.

   To support such inter-subnet forwarding, be
   encapsulated with the NVE behaves as an IP
   Default Gateway from the perspective of the attached TS's. Two models
   are possible:

   1. All the NVEs of a given EVPN instance use the same anycast default
   gateway IP address and the same anycast default gateway destination TS's MAC address.
   On each NVE, this default gateway IP/MAC address correspond In order for
   ingress PE to the
   IRB interface connecting the MAC-VRF of that EVI perform such encapsulation, it needs to the corresponding
   IP-VRF.

   2. Each NVE of a given EVPN instance uses its own default gateway maintain TS's
   IP and MAC addresses, and these addresses are aliased to the same
   conceptual gateway through the use of the Default Gateway extended
   community as specified in [EVPN], which is carried address association in the EVPN its ARP table. Furthermore, it
   needs to maintain destination TS's MAC
   Advertisement routes. On each NVE, this default gateway IP/MAC address correspond to in the IRB interface connecting corresponding
   BT even though it does not have the MAC-VRF of
   that EVI corresponding subnet locally
   configured. In other words, each PE participating in asymmetric IRB
   MUST maintain ARP entries for remote hosts (hosts connected to other
   PEs) as well as maintaining MAC-VRFs/BTs for subnets that are not
   locally present on that PE.

   The following subsection defines the corresponding IP-VRF.

   Both of these models enable a packet forwarding paradigm control and data planes
   procedures for both symmetric and asymmetric IRB forwarding. In case of asymmetric IRB, a
   packet is forwarded through the MAC-VRF followed by the IP-VRF on the ingress NVE, and then forwarded through the the MAC-VRF on the egress
   (disposition) NVE.
   PEs. The egress NVE merely needs to perform a lookup in
   the associated MAC-VRF and forward the Ethernet frames unmodified,
   i.e. without rewriting the source MAC address.  This following figure is different
   from symmetric IRB forwarding used in description of these procedures
   where it shows a packet is forwarded through the
   MAC-VRF followed by the single IP-VRF on the ingress NVE, and then forwarded
   through the IP-VRF followed by the MAC-VRF on the egress NVE.

   It is worth noting that if the applications that are running on the
   TS's are employing or relying on any form of MAC security, then the
   first model (i.e. using anycast addresses) would be required to
   ensure that the applications receive traffic from the same source MAC
   address that they are sending to.

3.2 Heterogeneous Environment

   For large data centers with thousands a number of servers and ToR (or Access)
   switches, some BTs on each PE for a
   given tenant. The IP-VRF of them may not have the capability of maintaining or
   enforcing policies for inter-subnet switching. Even though policies
   among multiple subnets belonging to same tenant can be simpler, hosts
   belonging to one tenant can also send traffic to peers belonging (i.e., IP-VRF1) is connected
   to
   different tenants or security zones. In such scenarios, each BT via its associated IRB interface. Each BT on a WAN edge PE is
   associated with a unique VLAN (e.g., L3GW) may not only need to enforce policies for communication
   among subnets belonging to with a BD) where in turn is
   associated with a single tenant, but also it may need to
   know how to handle traffic destined towards peers MAC-VRF in different
   tenants. Therefore, there case of VLAN-Based mode or a
   number of BTs can be associated with a mixed environment where an NVE
   performs inter-subnet switching for some EVPN instances and single MAC-VRF in case of
   VLAN-Aware Bundle mode. Whether the L3GW
   for others.

4  Operational Models for Asymmetric Inter-Subnet Forwarding

4.1 Among EVPN NVEs within service interface on a DC

   When an EVPN MAC/IP advertisement route PE is received by a NVE,
   VLAN-Based or VLAN-Aware Bundle mode does not impact the IP
   address associated with the route is used to populate the IP-VRF
   table, whereas IRB
   operation and procedures. It only impacts the setting of Ethernet tag
   field in EVPN routes as described in [RFC7432].

                    PE 1         +---------+
              +-------------+    |         |
      TS1-----|         MACx|    |         |        PE2
    (IP1/M1)  |(BT1)        |    |         |   +-------------+
      TS5-----|      \      |    |  MPLS/  |   |MACy  (BT3)  |-----TS3
    (IP5/M5)  |Mx/IPx \     |    |  VxLAN/ |   |     /       |  (IP3/M3)
              |    (IP-VRF1)|----|  NVGRE  |---|(IP-VRF1)    |
              |       /     |    |         |   |     \       |
      TS2-----|(BT2) /      |    |         |   |      (BT1)  |-----TS4
    (IP2/M2)  |             |    |         |   |             |   (IP4/M4)
              +-------------+    |         |   +-------------+
                                 |         |
                                 +---------+

                       Figure 4: IRB forwarding

3.1 IRB Interface and its MAC address associated with the route is used to
   populate both & IP addresses

   To support inter-subnet forwarding on a PE, the MAC-VRF table, as well PE acts as an IP
   Default Gateway from the adjacency perspective of the attached Tenant Systems
   where default gateway MAC and IP addresses are configured on each IRB
   interface associated with its subnet and falls into one of the IP route in
   following two options:

   1. All the IP-VRF table (i.e., ARP table).

   When an Ethernet frame is received by an ingress NVE, it performs PEs for a
   lookup on given tenant subnet use the destination same anycast default
   gateway IP and MAC address in addresses . On each PE, this default gateway IP
   and MAC addresses correspond to the IRB interface connecting the BT
   associated MAC-VRF with the tenant's <EVI, VLAN> to the corresponding
   tenant's IP-VRF.

   2. Each PE for
   that EVI. If a given tenant subnet uses the MAC same anycast default
   gateway IP address corresponds to but its IRB Interface own MAC
   address, the ingress NVE deduces that address. These MAC addresses are
   aliased to the packet MUST be inter-subnet
   routed. Hence, the ingress NVE performs an same anycast default gateway IP lookup in address through the
   associated IP-VRF table. The lookup identifies an adjacency that
   contains a MAC rewrite and in turn
   use of the next-hop (i.e., egress) NVE to Default Gateway extended community as specified in [EVPN],
   which is carried in the packet must be forwarded and the EVPN MAC/IP Advertisement routes. On each PE,
   this default gateway IP address along with its associated MPLS label
   stack. The MAC rewrite holds
   addresses correspond to the MAC address IRB interface connecting the BT
   associated with the
   destination host (as populated by tenant's <EVI, VLAN> to the EVPN MAC route), instead corresponding
   tenant's IP-VRF.

   It is worth noting that if the applications that are running on the
   TS's are employing or relying on any form of MAC security, then
   either the first model (i.e. using anycast MAC address of address) should be
   used to ensure that the next-hop NVE. The ingress NVE then rewrites applications receive traffic from the
   destination same
   IRB interface MAC address in that they are sending to, or if the packet with second
   model is used, then the IRB interface MAC address specified MUST be the one
   used in the adjacency. It also rewrites initial ARP reply for that TS.

   Although both of these options are equally applicable to both
   symmetric and asymmetric IRB, the source option-1 is recommended because of
   the ease of anycast MAC address with its provisioning on not only the IRB
   Interface MAC address. The ingress NVE, then, forwards
   interface associated with a given subnet across all the frame PEs
   corresponding to
   the next-hop (i.e. egress) NVE after encapsulating it that EVI but also on all IRB interfaces associated
   with all the MPLS
   label stack. Note that this label stack includes tenant's subnets across all the LSP label as
   well as PEs corresponding to all
   the EVPN label EVIs for that was advertised by tenant. Furthermore, it simplifies the egress NVE. operation as
   there is no need for Default Gateway extended community advertisement
   and its associated MAC aliasing procedure.

   When a TS sends an ARP request to the MPLS encapsulated packet PE that is received by the egress NVE, it uses attached to, the EVPN label to identify ARP
   request is sent for the MAC-VRF table. It then performs a MAC
   lookup in that table, which yields IP address of the outbound IRB interface to which
   the Ethernet frame must be forwarded. Figure 2 below depicts associated
   with the
   packet flow, where NVE1 and NVE2 are TS's subnet. For example, in figure 4, TS1 is configured
   with the ingress anycast IPx address as its default gateway IP address and egress NVEs,
   respectively.

                    NVE1                NVE2
              +------------+     +------------+
              |            |     |            |
              |(MAC -
   thus when it sends an ARP request for IPx (IP  |     |(IP  - (MAC |
              | VRF)   VRF)|     | VRF)   VRF)|
              |  |     |   |     |       |  | |
              +------------+     +------------+
                 ^     v                 ^  V
                 |     |                 |  |
           TS1->-+     +-->--------------+  +->-TS2

     Figure 2: Inter-Subnet Forwarding Among EVPN NVEs within a DC

   Note that the forwarding behavior on address of the egress NVE is similar to
   EVPN intra-subnet forwarding. In other words, all IRB
   interface for BT1), the packet
   processing associated PE1 sends an ARP reply with the inter-subnet forwarding semantics MACx which is
   confined to
   the ingress NVE and MAC address of that is why it is called Asymmetric
   IRB.

   It should also IRB interface.

   In addition to anycast addresses, IRB interfaces can be noted that [EVPN] provides different level of
   granularity configured
   with non-anycast IP addresses for the EVPN label.  Besides identifying bridge domain
   table, it can be used purpose of OAM (such as
   traceroute/ping to identify the egress interface or a
   destination MAC address on that interface. If EVPN label is used these interfaces) for
   egress interface or destination MAC address identification, then no
   MAC lookup is needed in the egress EVI both symmetric and the packet can
   asymmetric IRB. These IP addresses need to be directly
   forwarded distributed as VPN
   routes when PEs operating in symmetric IRB mode. However, they don't
   need to be distributed if the egress interface just based on EVPN label lookup.

4.2 Among EVPN NVEs PEs are operating in Different DCs Without GW asymmetric IRB
   mode and the IRB interfaces are configured with individual MACs.

3.2 Symmetric IRB Procedures

3.2.1 Control Plane - Ingress PE

   When an EVPN MAC advertisement route is received by a NVE, the PE (e.g., PE1 in figure 4 above) learns MAC and IP address associated with the route is used to populate the IP-VRF
   table, whereas of
   a TS (via an ARP request), it adds the MAC address associated with the route is used to
   populate both the MAC-VRF table, as well as the adjacency associated
   with
   corresponding MAC-VRF/BT of that tenant's subnet and adds the IP route in
   address to the IP-VRF table (i.e., ARP table).

   When an Ethernet frame is received by an ingress NVE, for that tenant. Furthermore, it performs a
   lookup on the destination MAC address in the associated EVI. If the adds this TS's
   MAC and IP address corresponds association to its IRB Interface MAC address, the ingress
   NVE deduces ARP table. It then builds an
   EVPN MAC/IP Advertisement route (type 2) as follow and advertises it
   to other PEs participating in that tenant's VPN.

   - The Length field of the packet BGP EVPN NLRI for an EVPN MAC/IP
   Advertisement route MUST be inter-subnet routed. Hence, the
   ingress NVE performs an either 40 (if IPv4 address is carried) or
   52 (if IPv6 address is carried).

   - Route Distinguisher (RD), Ethernet Segment Identifier, Ethernet Tag
   ID, MAC Address Length, MAC Address, IP lookup Address Length, IP Address,
   and MPLS Label1 fields MUST be used as defined in the associated IP-VRF table. [RFC7432] and
   [RFC8365].

   - The
   lookup identifies MPLS Label2 field is set to either an adjacency that contains MPLS label or a MAC rewrite and in
   turn the next-hop (i.e. egress) Gateway VNI
   corresponding to which the packet must be
   forwarded along with the associated tenant's IP-VRF. In case of an MPLS label stack. The MAC rewrite
   holds label, this
   field is encoded as 3 octets, where the MAC address associated with high-order 20 bits contain
   the destination host (as
   populated by label value.

   Just as in [RFC7432], the EVPN RD, Ethernet Tag ID, MAC route), instead of the Address Length,
   MAC address Address, IP Address Length, and IP Address fields are part of
   the
   next-hop Gateway. route key used by BGP to compare routes. The ingress NVE then rewrites rest of the destination MAC
   address in fields
   are not part of the packet route key.

   This route is advertised along with the address specified in the adjacency. It
   also rewrites the source following two extended
   communities:
        1) Tunnel Type Extended Community
        2) Router's MAC address with its Extended Community

   For symmetric IRB Interface mode, Router's MAC
   address. The ingress NVE, then, forwards the frame EC is needed to carry the next-hop
   (i.e. egress) Gateway after encapsulating it PE's
   overlay MAC address which is used for IP-VRF to IP-VRF communications
   with the Ethernet NVO tunnel. If MPLS label
   stack.

   Note that this label stack includes the LSP label as well as an EVPN
   label. The EVPN label could be either advertised by the ingress
   Gateway, if inter-AS option B or IP-only NVO tunnel is used, or advertised by the egress
   NVE, if inter-AS option C then
   there is used. When no need to send Router's MAC Extended Community along with
   this route.

   This route MUST be advertised with two route targets - one
   corresponding to the MPLS encapsulated packet
   is received by MAC-VRF of the ingress Gateway, tenant's subnet and another
   corresponding to the processing again differs
   depending on whether inter-AS option B or option C is employed: tenant's IP-VRF.

3.2.2 Control Plane - Egress PE

   When a PE (e.g., PE2 in
   the former case, the ingress Gateway swaps the figure 4 above) receives this EVPN label in the
   packets with MAC/IP
   Advertisement route advertisement, it performs the EVPN label value received from following:

   - Using MAC-VRF route target, it identifies the egress Gateway.
   In corresponding MAC-
   VRF. If the latter case, MAC-VRF exists (e.g., it is locally configured) then it
   imports the ingress Gateway MAC address into it. Otherwise, it does not modify import the EVPN
   label
   MAC address.

   - Using IP-VRF route target, it identifies the corresponding IP-VRF
   and performs normal label switching on imports the LSP label.
   Similarly on IP address into it.

   The inclusion of MPLS label2 field in this route signals to the egress Gateway,
   receiving PE that this route is for option B, symmetric IRB mode and MPLS
   label2 needs to be installed in forwarding path to identify the egress Gateway
   swaps
   corresponding IP-VRF.

   If the EVPN label receiving PE receives this route with both the value advertised by the egress NVE.
   Whereas, for option C, MAC-VRF and IP-
   VRF route targets but the egress Gateway MAC/IP Advertisement route does not modify the EVPN
   label, include
   MPLS label2 field and performs normal label switching on the LSP label. When if the
   MPLS encapsulated packet is received by receiving PE does not support asymmetric
   IRB mode, then if it has the egress NVE, corresponding MAC-VRF, it uses only imports
   the
   EVPN label to identify the bridge-domain table. It then performs a MAC lookup in that table, which yields the outbound interface to
   which the Ethernet frame must be forwarded. Figure 3 below depicts
   the packet flow.

            NVE1            GW1             GW2            NVE2
      +------------+  +------------+  +------------+  +------------+
      |            |  |            |  |            |  |            |
      |(MAC - (IP  |  |    [LS]    |  |    [LS]    |  |(IP  - (MAC |
      | VRF)   VRF)|  |            |  |            |  | VRF)   VRF)|
      |  |     |   |  |    |  |    |  |    |  |    |  |       |  | |
      +------------+  +------------+  +------------+  +------------+
         ^     v           ^  V            ^  V               ^  V
         |     |           |  |            |  |               |  |
   TS1->-+     +-->--------+  +------------+  +---------------+  +->-TS2

  Figure 3: Inter-Subnet Forwarding Among EVPN NVEs in Different DCs
   without GW

4.3 Among EVPN NVEs in Different DCs with GW

   In this scenario, the NVEs within a given data center do not have
   entries for the MAC/IP addresses of hosts in remote data centers.
   Rather, the NVEs address; otherwise, if it doesn't have a default IP route pointing to the WAN gateway
   for each VRF. This is accomplished by corresponding MAC-
   VRF, it MUST treat the WAN gateway advertising for
   a given EVPN that spans multiple DC a default VPN-IP route that is
   imported by the NVEs of that VPN that are in the gateway's own DC. as withdraw [RFC7606].

3.2.3 Data Plane - Ingress PE

   When an Ethernet frame is received by an ingress NVE, PE (e.g., PE1 in
   figure 4 above), the PE uses the AC ID (e.g., VLAN ID) to identify
   the associated MAC-VRF/BT and it performs a lookup on the destination
   MAC address in the associated MAC-VRF
   table. address. If the MAC address corresponds to the its IRB Interface MAC
   address, the ingress NVE PE deduces that the packet MUST must be inter-subnet
   routed. Hence, the ingress NVE PE performs an IP lookup in the associated
   IP-VRF table. The lookup, in this case, matches lookup identifies BGP next hop of egress PE along
   with the
   default host route which points to tunnel/encapsulation type and the local WAN gateway. The ingress
   NVE associated MPLS/VNI
   values.

   If the tunnel type is that of MPLS or IP-only NVO tunnel, then rewrites TS's
   IP packet is sent over the destination MAC address in tunnel without any Ethernet header.
   However, if the packet with tunnel type is that of Eternet NVO tunnel, then an
   Ethernet header needs to be added to the
   router's TS's IP packet. The source
   MAC address of this Ethernet header is set to the local WAN gateway. It also rewrites ingress PE's router
   MAC address and the
   source destination MAC address with its own IRB Interface of this Ethernet header
   is set to the egress PE's router MAC address. The
   ingress NVE, then, forwards MPLS VPN label or
   VNI fields are set accordingly and the frame packet is forwarded to the WAN gateway after
   encapsulating it with the MPLS label stack. Note that this label
   stack includes
   egress PE.

   If case of NVO tunnel encapsulation, the LSP label as well as outer source IP address is
   set to the label for default host
   route that was advertised by ingress PE's BGP next-hop address and outer destination IP
   address is set to the local WAN gateway. egress PE's BGP next-hop address.

3.2.4 Data Plane - Egress PE

   When the tenant's MPLS or NVO encapsulated packet is received over an
   MPLS or NVO tunnel by the local WAN gateway, it egress PE, the egress PE removes NVO tunnel
   encapsulation and uses the
   default host route VPN MPLS label (for MPLS encapsulation) or
   VNI (for VxLAN encapsulation) to identify the IP-VRF table. It then
   performs an in which IP
   lookup in that table. needs to be performed.

   The lookup identifies an
   adjacency that contains a MAC rewrite and in turn the remote WAN
   gateway (of the remote data center) local adjacency to which the packet must be
   forwarded along with the associated MPLS label stack. The MAC rewrite
   holds the MAC address IRB interface
   associated with the ultimate destination host
   (as populated by the EVPN MAC route). egress subnet's MAC-VRF/BT.

   The local WAN gateway then
   rewrites egress PE gets the destination TS's MAC address in for that TS's IP
   address from its ARP table, it encapsulates the packet with the that
   destination MAC address
   specified in the adjacency. It also rewrites the and a source MAC address
   with its router's MAC address. The local WAN gateway, then, forwards
   the frame corresponding to the remote WAN gateway after encapsulating it with the
   MPLS label stack. Note that this label stack includes the LSP label
   as well as a EVPN label
   that was advertised by the remote WAN
   gateway. When the MPLS encapsulated packet is received by the remote
   WAN gateway, it simply swaps the EVPN label IRB interface and forwards sends the packet to the egress NVE. This implies that the GW1 needs to keep the remote
   host its destination subnet
   MAC-VRF/BT.

   The destination MAC addresses along with the corresponding EVPN labels address lookup in the MAC-VRF/BT results in local
   adjacency entries of the IP-VRF table (i.e., its ARP table). The
   remote WAN gateway then forward the packet to (e.g., local interface) over which the egress NVE. The
   egress NVE then performs Ethernet frame is
   sent on.

3.3 Asymmetric IRB Procedures

3.3.1 Control Plane - Ingress PE

   When a PE (e.g., PE1 in figure 4 above) learns MAC lookup and IP address of
   a TS (via an ARP request), it populates its MAC-VRF/BT, IP-VRF, and
   ARP table just as in the MAC-VRF (identified by
   the received case for symmetric IRB. It then builds an
   EVPN label) MAC/IP Advertisement route (type 2) as follow and advertises it
   to determine other PEs participating in that tenant's VPN.

   - The Length field of the outbound port to send the
   traffic on.

   Figure 4 below depicts the forwarding model.

            NVE1            GW1             GW2            NVE2
      +------------+  +------------+  +------------+  +------------+
      |            |  |            |  |            |  |            |
      |(MAC - (IP  |  |(IP  - (MAC |  |    [LS]    |  |(IP  - (MAC |
      | VRF)   VRF)|  | VRF)   VRF)|  |    |  |    |  | VRF)   VRF)|
      |  |     |   |  | |  |       |  |    |  |    |  |       |  | |
      +------------+  +------------+  +------------+  +------------+
         ^     v        ^  V               ^  V               ^  V
         |     |        |  |               |  |               |  |
   TS1->-+     +-->-----+  +---------------+  +---------------+  +->-TS2

  Figure 4: Inter-Subnet Forwarding Among BGP EVPN NVEs NLRI for an EVPN MAC/IP
   Advertisement route MUST be either 37 (if IPv4 address is carried) or
   49 (if IPv6 address is carried).

   - Route Distinguisher (RD), Ethernet Segment Identifier, Ethernet Tag
   ID, MAC Address Length, MAC Address, IP Address Length, IP Address,
   and MPLS Label1 fields MUST be used as defined in Different DCs
   with GW

4.4 Among IP-VPN Sites [RFC7432] and EVPN NVEs with GW

   In
   [RFC8365].

   - The MPLS Label2 field MUST NOT be included in this scenario, the NVEs within a given data center do not have
   entries for route.

   Just as in [RFC7432], the RD, Ethernet Tag ID, MAC Address Length,
   MAC Address, IP addresses Address Length, and IP Address fields are part of hosts in remote enterprise sites.
   Rather,
   the NVEs have a default IP route pointing key used by BGP to compare routes. The rest of the WAN gateway for
   each IP-VRF.

   When an Ethernet frame fields
   are not part of the route key.

   This route is received by an ingress NVE, it performs a
   lookup on advertised along with the following extended
   communitiy:
        1) Tunnel Type Extended Community

   For asymmetric IRB mode, Router's MAC EC is not needed because
   forwarding is performed using destination TS's MAC address in the associated which is
   carried in this route advertisement.

   This route MUST always be advertised with MAC-VRF
   table. If the MAC address corresponds route target. It
   MAY also be advertised with a second route target corresponding to
   the IRB Interface MAC
   address, IP-VRF. If only MAC-VRF route target is used, then the ingress NVE deduces that receiving
   PE uses the packet MUST be inter-subnet
   routed. Hence, MAC-VRF route target to identify the ingress NVE performs an IP lookup in corresponding IP-VRF
   - i.e., many MAC-VRF route targets map to the
   associated same IP-VRF table. The lookup, for a given
   tenant.

3.3.2 Control Plane - Egress PE
   When a PE (e.g., PE2 in figure 4 above) receives this case, matches EVPN MAC/IP
   Advertisement route advertisement, it performs the
   default following:

   - Using MAC-VRF route which points to target, it identifies the local WAN gateway. The ingress NVE
   then rewrites corresponding MAC-VRF
   and imports the destination MAC address into it. For asymmetric IRB mode, it is
   assumed that all PEs participating in the packet a tenant's VPN are configured
   with the
   router's all subnets and corresponding MAC-VRFs/BTs even if there are no
   locally attached TS's for some of these subnets. The reason for this
   is because ingress PE needs to do forwarding based on destination
   TS's MAC address and does proper NVO tunnel encapsulation which are
   property of a lookup in MAC-VRF/BT. An implementation may choose to
   consolidate the local WAN gateway. It also rewrites lookup at the
   source MAC address with its own IRB Interface MAC address. The ingress NVE, then, forwards PE's IP-VRF with the frame to lookup at
   the local WAN gateway after
   encapsulating it with ingress PE's destination subnet MAC-VRF. Consideration for such
   consolidation of lookups is outside the MPLS label stack. Note that scope of this label
   stack includes document.

   - Using MAC-VRF route target, it identifies the LSP label as well as corresponding ARP
   table for the default host route label tenant and it adds an entry to the ARP table for the
   TS's MAC and IP address association. It should be noted that was advertised by the local WAN gateway. When
   tenant's ARP table at the MPLS
   encapsulated packet receiving PE is received identified by all the local WAN gateway, it uses the
   default host MAC-
   VRF route label to identify the targets for that tenant. If IP-VRF table. It route target is included
   with this route advertisement, then
   performs an IP lookup in that table. The lookup identifies it MAY be used for the next
   hop ASBR to which
   identification of tenant's ARP table.

   For asymmetric IRB mode, the packet must MPLS label2 field SHOULD not be forwarded. The local gateway included
   in the route; however, if the receiving PE receives this case strips route with
   the MPLS label2 field, then it SHOULD ignore it.

3.3.3 Data Plane - Ingress PE

   When an Ethernet encapsulation and perform frame is received by an IP lookup ingress PE (e.g., PE1 in its IP-VRF and forwards
   figure 4 above), the IP packet PE uses the AC ID (e.g., VLAN ID) to identify
   the ASBR using a label
   stack comprising of an LSP label associated MAC-VRF/BT and an IP-VPN label that was
   advertised by the ASBR. When the MPLS encapsulated packet is received
   by the ASBR, it simply swaps performs a lookup on the IP-VPN label with destination
   MAC address. If the one advertised
   by MAC address corresponds to its IRB Interface MAC
   address, the egress PE. The ASBR then forwards ingress PE deduces that the packet to must be inter-subnet
   routed. Hence, the egress PE.
   The egress ingress PE then performs an IP lookup in the associated
   IP-VRF (identified by
   the received IP-VPN label) to determine where table. The lookup identifies a local adjacency to forward the traffic.

   Figure 5 below depicts IRB
   interface associated with the forwarding model.

            NVE1            GW1             ASBR egress subnet's MAC-VRF/BT.

   The ingress PE gets the destination TS's MAC address for that TS's IP
   address from its ARP table, it encapsulates the packet with that
   destination MAC address and a source MAC address corresponding to
   that IRB interface and sends the packet to its destination subnet
   MAC-VRF/BT.

   The destination MAC address lookup in the MAC-VRF/BT results in BGP
   next hop address of egress PE. The ingress PE encapsulates the packet
   using Ethernet NVO tunnel of the choice (e.g., VxLAN or GENEVE) and
   sends the packet to the egress PE. Since the packet forwarding is
   between ingress PE's MAC-VRF/BT and egress PE's MAC-VRF/BT, the
   packet encapsulation procedures follows that of [RFC7432] for MPLS
   and [RFC8365] for VxLAN encapsulations.

3.3.4 Data Plane - Egress PE

   When a tenant's Ethernet frame is received over an NVO tunnel by the
   egress PE, the egress PE removes NVO tunnel encapsulation and uses
   the VPN MPLS label (for MPLS encapsulation) or VNI (for VxLAN
   encapsulation) to identify the MAC-VRF/BT in which MAC lookup needs
   to be performed.

   The MAC lookup results in local adjacency (e.g., local interface)
   over which the packet needs to get sent.

   Note that the forwarding behavior on the egress PE is the same as
   EVPN intra-subnet forwarding described in [RFC7432] for MPLS and
   [RFC8365] for VxLAN networks. In other words, all the packet
   processing associated with the inter-subnet forwarding semantics is
   confined to the ingress PE.

   It should also be noted that [RFC7432] provides different level of
   granularity for the EVPN label.  Besides identifying bridge domain
   table, it can be used to identify the egress interface or a
   destination MAC address on that interface. If EVPN label is used for
   egress interface or individual MAC address identification, then no
   MAC lookup is needed in the egress PE for MPLS encapsulation and the
   packet can be directly forwarded to the egress interface just based
   on EVPN label lookup.

4 BGP Encoding

   This document defines one new BGP Extended Community for EVPN.

4.1 Router's MAC Extended Community

   A new EVPN BGP Extended Community called Router's MAC is introduced
   here. This new extended community is a transitive extended community
   with the Type field of 0x06 (EVPN) and the Sub-Type of 0x03. It may
   be advertised along with BGP Encapsulation Extended Community define
   in section 4.5 of [TUNNEL-ENCAP].

   The Router's MAC Extended Community is encoded as an 8-octet value as
   follows:

        0                   1                   2                   3
        0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       | Type=0x06     | Sub-Type=0x03 |        Router's MAC           |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |                      Router's MAC Cont'd                      |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

               Figure 5: Router's MAC Extended Community

   This extended community is used to carry the PE's MAC address for
   symmetric IRB scenarios and it is sent with RT-2.

5 Operational Models for Symmetric Inter-Subnet Forwarding

   The following sections describe two main symmetric IRB forwarding
   scenarios (within a DC - i.e., intra-DC) along with their
   corresponding procedures. In the following scenarios, without loss of
   generality, it is assumed that a given tenant is represented by a
   single IP-VPN instance. Therefore, on a given PE, a tenant is
   represented by a single IP-VRF table and one or more MAC-VRF tables.

5.1 IRB forwarding on NVEs for Tenant Systems

   This section covers the symmetric IRB procedures for the scenario
   where each Tenant System (TS) is attached to one or more NVEs and its
   host IP and MAC addresses are learned by the attached NVEs and are
   distributed to all other NVEs that are interested in participating in
   both intra-subnet and inter-subnet communications with that TS.

   In this scenario, for a given tenant, an NVE has typically one MAC-
   VRF for each tenant's subnet (VLAN) that is configured for, assuming
   VLAN-based service which is typically the case for VxLAN and NVGRE
   encapsulation and each MAC-VRF consists of a single bridge domain. In
   case of MPLS encapsulation with VLAN-aware bundling, then each MAC-
   VRF consists of multiple bridge domains (one bridge domain per VLAN).
   The MAC-VRFs on an NVE for a given tenant are associated with an IP-
   VRF corresponding to that tenant (or IP-VPN instance) via their IRB
   interfaces.

   Each NVE MUST support QoS, Security, and OAM policies per IP-VRF
   to/from the core network. This is not to be confused with the QoS,
   Security, and OAM policies per Attachment Circuits (AC) to/from the
   Tenant Systems. How this requirement is met is an implementation
   choice and it is outside the scope of this document.

   Since VxLAN and NVGRE encapsulations require inner Ethernet header
   (inner MAC SA/DA), and since for inter-subnet traffic, TS MAC address
   cannot be used, the ingress NVE's MAC address is used as inner MAC
   SA. The NVE's MAC address is the device MAC address and it is common
   across all MAC-VRFs and IP-VRFs. This MAC address is advertised using
   the new EVPN Router's MAC Extended Community (section 6.1).

   Figure 6 below illustrates this scenario where a given tenant (e.g.,
   an IP-VPN instance) has three subnets represented by MAC-VRF1, MAC-
   VRF2, and MAC-VRF3 across two NVEs. There are five TS's that are
   associated with these three MAC-VRFs - i.e., TS1, TS4, and TS5 are
   sitting on the same subnet (e.g., same MAC-VRF/VLAN);where, TS1 and
   TS5 are associated with MAC-VRF1 on NVE1, TS4 is associated with MAC-
   VRF1 on NVE2.  TS2 is associated with MAC-VRF2 on NVE1, and TS3 is
   associated with MAC-VRF3 on NVE2. MAC-VRF1 and MAC-VRF2 on NVE1 are
   in turn associated with IP-VRF1 on NVE1 and MAC-VRF1 and MAC-VRF3 on
   NVE2 are associated with IP-VRF1 on NVE2. When TS1, TS5, and TS4
   exchange traffic with each other, only L2 forwarding (bridging) part
   of the IRB solution is exercised because all these TS's sit on the
   same subnet. However, when TS1 wants to exchange traffic with TS2 or
   TS3 which belong to different subnets, then both bridging and routing
   parts of the IRB solution are exercised. The following subsections
   describe the control and data planes operations for this IRB scenario
   in details.

                     NVE1         +---------+
               +-------------+    |         |
       TS1-----|         MACx|    |         |        NVE2
     (IP1/M1)  |(MAC-        |    |         |   +-------------+
       TS5-----| VRF1)\      |    |  MPLS/  |   |MACy  (MAC-  |-----TS3
     (IP5/M5)  |       \     |    |  VxLAN/ |   |     / VRF3) |  (IP3/M3)
               |    (IP-VRF1)|----|  NVGRE  |---|(IP-VRF1)    |
               |       /     |    |         |   |     \       |
       TS2-----|(MAC- /      |    |         |   |      (MAC-  |-----TS4
     (IP2/M2)  | VRF2)       |    |         |   |       VRF1) |   (IP4/M4)
               +-------------+    |         |   +-------------+
                                  |         |
                                  +---------+

          Figure 6: IRB forwarding on NVEs for Tenant Systems

5.1.1 Control Plane Operation

   Each NVE advertises a MAC/IP Advertisement route (i.e., Route Type 2)
   for each of its TS's with the following field set:

   - RD and ESI per [EVPN]
   - Ethernet Tag = 0; assuming VLAN-based service
   - MAC Address Length = 48
   - MAC Address = Mi ; where i = 1,2,3,4, or 5 in the above example
   - IP Address Length = 32 or 128
   - IP Address = IPi ; where i = 1,2,3,4, or 5 in the above example
   - Label-1 = MPLS Label or VNID corresponding to MAC-VRF
   - Label-2 = MPLS Label or VNID corresponding to IP-VRF

   Each NVE advertises an RT-2 route with two Route Targets (one
   corresponding to its MAC-VRF and the other corresponding to its IP-
   VRF. Furthermore, the RT-2 is advertised with two BGP Extended
   Communities. The first BGP Extended Community identifies the tunnel
   type per section 4.5 of [TUNNEL-ENCAP] and the second BGP Extended
   Community includes the MAC address of the NVE (e.g., MACx for NVE1 or
   MACy for NVE2) as defined in section 6.1. This second Extended
   Community (for the MAC address of NVE) is only required when Ethernet
   NVO tunnel type is used. If IP NVO tunnel type is used, then there is
   no need to send this second Extended Community. It should be noted
   that IP NVO tunnel type is only applicable to symmetric IRB
   procedures.

   Upon receiving this advertisement, the receiving NVE performs the
   following:

   - It uses Route Targets corresponding to its MAC-VRF and IP-VRF for
   identifying these tables and subsequently importing this route into
   them.

   - It imports the MAC address from MAC/IP Advertisement route into the
   MAC-VRF with BGP Next Hop address as underlay tunnel destination
   address (e.g., VTEP DA for VxLAN encapsulation) and Label-1 as VNID
   for VxLAN encapsulation or EVPN label for MPLS encapsulation.

   - If the route carries the new Router's MAC Extended Community, and
   if the receiving NVE is using Ethernet NVO tunnel, then the receiving
   NVE imports the IP address into IP-VRF with NVE's MAC address (from
   the new Router's MAC Extended Community) as inner MAC DA and BGP Next
   Hop address as underlay tunnel destination address, VTEP DA for VxLAN
   encapsulation and Label-2 as IP-VPN VNID for VxLAN encapsulation.

   - If the receiving NVE is going to use MPLS encapsulation, then the
   receiving NVE imports the IP address into IP-VRF with BGP Next Hop
   address as underlay tunnel destination address, and Label-2 as IP-VPN
   label for MPLS encapsulation.

   If the receiving NVE receives a RT-2 with only a single Route Target
   corresponding to IP-VRF and Label-1, or if it receives a RT-2 with
   only a single Route Target corresponding to MAC-VRF but with both
   Label-1 and Label-2, or if it receives a RT-2 with MAC Address Length
   of zero, then it must not import it to either IP-VRF or MAC-VRF and
   it must log an error.

5.1.2 Data Plane Operation - Inter Subnet

   The following description of the data-plane operation describes just
   the logical functions and the actual implementation may differ. Lets
   consider data-plane operation when TS1 in subnet-1 (MAC-VRF1) on NVE1
   wants to send traffic to TS3 in subnet-3 (MAC-VRF3) on NVE2.

   - NVE1 receives a packet with MAC DA corresponding to the MAC-VRF1
   IRB interface on NVE1 (the interface between MAC-VRF1 and IP-VRF1),
   and VLAN-tag corresponding to MAC-VRF1.

   - Upon receiving the packet, the NVE1 uses VLAN-tag to identify the
   MAC-VRF1. It then looks up the MAC DA and forwards the frame to its
   IRB interface.

   -  The Ethernet header of the packet is stripped and the packet is
   fed to the IP-VRF where IP lookup is performed on the destination
   address. This lookup yields an outgoing interface and the required
   encapsulation. If the encapsulation is for Ethernet NVO tunnel, then
   it includes a MAC address to be used as inner MAC DA, an IP address
   to be used as VTEP DA, and a VPN-ID to be used as VNID. The inner MAC
   SA and VTEP SA is set to NVE's MAC and IP addresses respectively. If
   it is a MPLS encapsulation, then corresponding EVPN and LSP labels
   are added to the packet. The packet is then forwarded to the egress
   NVE.

   - On the egress NVE, if the packet arrives on Ethernet NVO tunnel
   (e.g., it is VxLAN encapsulated), then the VxLAN header is removed.
   Since the inner MAC DA is the egress NVE's MAC address, the egress
   NVE knows that it needs to perform an IP lookup. It uses VNID to
   identify the IP-VRF table. If the packet is MPLS encapsulated, then
   the EVPN label lookup identifies the IP-VRF table. Next, an IP lookup
   is performed for the destination TS (TS3) which results in access-
   facing IRB interface over which the packet is sent. Before sending
   the packet over this interface, the ARP table is consulted to get the
   destination TS's MAC address.

   - The IP packet is encapsulated with an Ethernet header with MAC SA
   set to that of IRB interface MAC address and MAC DA set to that of
   destination TS (TS3) MAC address. The packet is sent to the
   corresponding MAC-VRF3 and after a lookup of MAC DA, is forwarded to
   the destination TS (TS3) over the corresponding interface.

   In this symmetric IRB scenario, inter-subnet traffic between NVEs
   will always use the IP-VRF VNID/MPLS label. For instance, traffic
   from TS2 to TS4 will be encapsulated by NVE1 using NVE2's IP-VRF
   VNID/MPLS label, as long as TS4's host IP is present in NVE1's IP-
   VRF.

5.1.3 TS Move Operation

   When a TS move from one NVE to other, it is important that the MAC
   mobility procedures are properly executed and the corresponding MAC-
   VRF and IP-VRF tables on all participating NVEs are updated. [EVPN]
   describes the MAC mobility procedures for L2-only services for both
   single-homed TS and multi-homed TS. This section describes the
   incremental procedures and BGP Extended Communities needed to handle
   the MAC mobility for a mixed of L2 and L3 connectivity (aka IRB). In
   order to place the emphasis on the differences between L2-only versus
   L2-and-L3 use cases, the incremental procedure is described for
   single-homed TS with the expectation that the reader can easily
   extrapolate multi-homed TS based on the procedures described in
   section 15 of [EVPN].

   Lets consider TS1 in figure-6 above where it moves from NVE1 to NVE2.
   In such move, NVE2
      +------------+  +------------+  +------------+  +------------+
      |            |  |            |  |            |  |            |
      |(MAC - (IP  |  |(IP  - (MAC |  |    [LS]    |  |       (IP  |
      | VRF)   VRF)|  | VRF)   VRF)|  |    |  |    |  |        VRF)|
      |  |     |   |  | |  |       |  |    |  |    |  |       |  | |
      +------------+  +------------+  +------------+  +------------+
         ^     v        ^  V              ^   V               ^  V
         |     |        |  |              |   |               |  |
   TS1->-+     +-->-----+  +--------------+   +---------------+  +->-H1

  Figure 5: Inter-Subnet Forwarding Among IP-VPN Sites discovers IP1/MAC1 of TS1 and realizes that it is
   a MAC move and it advertises a MAC/IP route per section 5.1.1 above
   with MAC Mobility Extended Community.

   Since NVE2 learns TS1's MAC/IP addresses locally, it updates its MAC-
   VRF1 and IP-VRF1 for TS1 with its local interface.

   If the local learning at NVE1 is performed using control or
   management planes, then these interactions serve as the trigger for
   NVE1 to withdraw the MAC and EVPN NVEs IP addresses associated with GW

4.5 Use TS1.
   However, if the local learning at NVE1 is performed using data-plane
   learning, then the reception of Centralized Gateway

   In this scenario, the NVEs within a given data center need to forward
   traffic in L2 MAC/IP Advertisement route (for
   TS1) from NVE2 with MAC Mobility extended community serve as the
   trigger for NVE1 to a centralized L3GW withdraw the MAC and IP addresses associated with
   TS1.

   All other remote NVE devices upon receiving the MAC/IP advertisement
   route for a TS1 from NVE2 with MAC Mobility extended community compare
   the sequence number of reasons: a) they
   don't have IRB capabilities or b) in this advertisement with the one previously
   received. If the new sequence number is greater than the old one,
   then they don't have required policy for
   switching traffic between different tenants or security zones. The
   centralized L3GW performs both update the IRB function MAC/IP addresses of TS1 in their corresponding
   MAC-VRFs and IP-VRFs to point to NVE2. Furthermore, upon receiving
   the MAC/IP withdraw for switching traffic
   among different EVPN instances as well as it performs interworking
   function when TS1 from NVE1, these remote PEs perform the traffic needs to be switched between IP-VPN sites
   and EVPN instances.

5 Operational Models
   cleanups for Symmetric Inter-Subnet Forwarding

   The following sections describe several main symmetric IRB forwarding
   scenarios.

5.1 their BGP tables.

5.2 IRB forwarding on NVEs for Subnets behind Tenant Systems

   This section covers the symmetric IRB procedures for the scenario
   where each some Tenant System (TS) is attached to Systems (TS's) support one or more NVEs and its
   host IP and MAC addresses are learned by the attached NVEs subnets and
   these TS's are
   distributed to all other NVEs that are interested in participating in
   both intra-subnet and inter-subnet communications associated with that TS.

   In this scenario, for a given tenant (e.g., an IP-VPN instance), an
   NVE has typically one MAC-VRF or more NVEs. Therefore, besides
   the advertisement of MAC/IP addresses for each tenant's subnet (VLAN) that is
   configured for, Assuming VLAN-based service TS which is typically can be multi-
   homed with All-Active redundancy mode, the
   case for VxLAN associated NVE needs to
   also advertise the subnets statically configured on each TS.

   The main difference between this solution and NVGRE encapsulation, the previous one is the
   additional advertisement corresponding to each MAC-VRF consists of a
   single bridge domain. In case of MPLS encapsulation subnet. These subnet
   advertisements are accomplished using EVPN IP Prefix route defined in
   [EVPN-PREFIX]. These subnet prefixes are advertised with VLAN-aware
   bundling, then each MAC-VRF consists the IP
   address of multiple bridge domains (one
   bridge domain per VLAN). The MAC-VRFs on an NVE for a given tenant
   are their associated with an IP-VRF corresponding TS (which is in overlay address space) as
   their next hop. The receiving NVEs perform recursive route resolution
   to resolve the subnet prefix with its associated ingress NVE so that tenant (or IP-VPN
   instance) via their IRB interfaces.

   Each
   they know which NVE MUST support QoS, Security, and OAM policies per IP-VRF
   to/from the core network. This is not to be confused with the QoS,
   Security, and OAM policies per Attachment Circuits (AC) to/from forward the
   Tenant Systems. How packets to when they are destined
   for that subnet prefix.

   The advantage of this requirement recursive route resolution is met that when a TS
   moves from one NVE to another, there is an implementation
   choice and no need to re-advertise any
   of the subnet prefixes for that TS. All it is outside needed is to advertise
   the scope of this document.

   Since VxLAN IP/MAC addresses associated with the TS itself and NVGRE encapsulations require inner Ethernet header
   (inner exercise MAC SA/DA), and since
   mobility procedures for inter-subnet traffic, TS MAC address
   cannot be used, the ingress NVE's MAC address is used as inner MAC
   SA. that TS. The NVE's MAC address is recursive route resolution
   automatically takes care of the device MAC address and it is common
   across all MAC-VRFs and IP-VRFs. This MAC address is advertised using updates for the new EVPN Router's MAC Extended Community (section 6.1). subnet prefixes of
   that TS.

   Figure below illustrates this scenario where a given tenant (e.g., an
   IP-VPN instance) service) has three subnets represented by MAC-VRF1, MAC-VRF2,
   and MAC-VRF3 across two NVEs. There are five four TS's that are associated with
   these three MAC-VRFs - i.e., TS1, TS4, and TS5 are sitting on
   the same subnet (e.g., same MAC-VRF/VLAN);where, TS1 and TS5 are
   associated with connected to MAC-VRF1 on
   NVE1, TS4 is associated with MAC-VRF1 on
   NVE2. TS2 is associated with connected to MAC-VRF2 on NVE1, and  TS3 is associated
   with MAC-VRF3 connected to MAC-
   VRF3 on NVE2. MAC-VRF1 NVE2, and MAC-VRF2 on NVE1 are in turn
   associated with IP-VRF1 TS4 is connected to MAC-VRF1 on NVE1 NVE2. TS1 has two
   subnet prefixes (SN1 and MAC-VRF1 SN2) and MAC-VRF3 TS3 has a single subnet prefix,
   SN3. The MAC-VRFs on NVE2 each NVE are associated with IP-VRF1 on NVE2. their corresponding
   IP-VRF using their IRB interfaces. When TS1, TS5, and TS4 and TS1 exchange
   traffic with each other, intra-
   subnet traffic, only L2 forwarding (bridging) part of the IRB
   solution is exercised because all these TS's sit on used (i.e., the same
   subnet. However, traffic only goes through their MAC-
   VRFs); however, when TS1 TS3 wants to exchange forward traffic with TS2 or TS3
   which belong to different subnets, SN1 or SN2
   sitting behind TS1 (inter-subnet traffic), then both bridging and
   routing parts of the IRB solution are exercised. exercised (i.e., the traffic
   goes through the corresponding MAC-VRFs and IP-VRFs). The following
   subsections describe the control and data planes operations for this
   IRB scenario in details.

                             NVE1         +---------+      +----------+
     SN1--+          +-------------+   |          |
       TS1-----|         MACx|
          |--TS1-----|(MAC- \      |   |        NVE2
     (IP1/M1)  |(MAC-          |
     SN2--+ IP1/M1   | VRF1) \     |   +-------------+
       TS5-----| VRF1)\   |          |  MPLS/
                     |   |MACy  (MAC-  |-----TS3
     (IP5/M5)     (IP-VRF)|---|          |       \
                     |       /     |  VxLAN/   |          |
             TS2-----|(MAC- / VRF3)      |  (IP3/M3)   |    (IP-VRF1)|----|  MPLS/   |
            IP2/M2   | VRF2)       |   |  VxLAN/  |
                     +-------------+   |  NVGRE  |---|(IP-VRF1)   |
                     +-------------+   |          |       /
     SN3--+--TS3-----|(MAC-\       |   |          |
            IP3/M3   |     \ VRF3)\      |
       TS2-----|(MAC- /   |          |
                     |     (IP-VRF)|---|          |      (MAC-  |-----TS4
     (IP2/M2)
                     | VRF2)       /     |   |          |
             TS4-----|(MAC- /      |       VRF1)   |   (IP4/M4)
               +-------------+          |
            IP4/M4   |   +-------------+ VRF1)       |   |
                                  +---------+          |
                     +-------------+   +----------+
                            NVE2

Figure 6: 7: IRB forwarding on NVEs for Tenant Systems

5.1.1 with configured subnets

5.2.1 Control Plane Operation

   Each NVE advertises a Route Type-2 (RT-2, MAC/IP Advertisement Route) Type-5 (RT-5, IP Prefix Route defined in
   [EVPN-PREFIX]) for each of its TS's subnet prefixes with the following field set: IP address of
   its TS as the next hop (gateway address field) as follow:

   - RD and associated with the IP-VRF
   - ESI per [EVPN] = 0
   - Ethernet Tag = 0; assuming VLAN-based service
   - MAC Address Length = 48
   - MAC Address = Mi ; where i = 1,2,3,4, or 5 in the above example
   - IP Address Prefix Length = 32 or 128
   - IP Address = IPi ; where i = 1,2,3,4, or 5 in the above example
   - Label-1 Prefix = MPLS Label or VNID corresponding to MAC-VRF SNi
   - Label-2 Gateway Address = MPLS Label or VNID corresponding to IP-VRF

   Each NVE advertises an RT-2 route with two Route Targets (one
   corresponding to its MAC-VRF and the other corresponding to its IP-
   VRF. Furthermore, the RT-2 is advertised with two BGP Extended
   Communities. The first BGP Extended Community identifies the tunnel
   type per section 4.5 of [TUNNEL-ENCAP] and the second BGP Extended
   Community includes the MAC IPi; IP address of TS
   - Label = 0

   This RT-5 is advertised with one or more Route Targets that have been
   configured as "export route targets" of the IP-VRF from which the
   route is originated.

   Each NVE (e.g., MACx for NVE1 or
   MACy also advertises an RT-2 (MAC/IP Advertisement Route) along
   with their associated Route Targets and Extended Communities for NVE2) each
   of its TS's exactly as defined described in section 6.1. This second Extended
   Community (for the MAC address of NVE) is only required when Ethernet
   NVO tunnel type is used. If IP NVO tunnel type is used, then there is
   no need to send this second Extended Community. 5.1.1.

   Upon receiving this the RT-5 advertisement, the receiving NVE performs the
   following:

   - It uses the Route Targets corresponding Target to its MAC-VRF and identify the corresponding IP-VRF for
   identifying these tables and subsequently importing this route into
   them.
   - It imports the MAC address IP prefix into the MAC-VRF its corresponding IP-VRF that is
   configured with BGP Next Hop
   address as underlay tunnel destination address (e.g., VTEP DA for
   VxLAN encapsulation) and Label-1 as VNID for VxLAN encapsulation or
   EVPN label for MPLS encapsulation.

   - If the route carries the new Router's MAC Extended Community, and
   if the receiving NVE an import RT that is using Ethernet NVO tunnel, then one of the receiving
   NVE imports RTs being carried by
   the IP address into IP-VRF RT-5 route along with NVE's MAC address (from the new Router's MAC Extended Community) as inner MAC DA and BGP Next
   Hop IP address as underlay tunnel destination address, VTEP DA for VxLAN
   encapsulation and Label-2 as IP-VPN VNID for VxLAN encapsulation.

   - If of the associated TS as its
   next hop.

   When receiving NVE is going to use MPLS encapsulation, then the RT-2 advertisement, the receiving NVE imports
   MAC/IP addresses of the IP address TS into the corresponding MAC-VRF and IP-VRF with BGP Next Hop
   per section 5.1.1. When both routes exist, recursive route resolution
   is performed to resolve the IP prefix (received in RT-5) to its
   corresponding NVE's IP address (e.g., its BGP next hop). BGP next hop
   will be used as underlay tunnel destination address, address (e.g., VTEP DA
   for VxLAN encapsulation) and Label-2 Router's MAC will be used as IP-VPN
   label inner MAC
   for MPLS VxLAN encapsulation.

   If the receiving NVE receives a RT-2 with only a single Route Target
   corresponding to IP-VRF and Label-1, then it must discard this route
   and log an error. If the receiving NVE receives a RT-2 with only a
   single Route Target corresponding to MAC-VRF but with both Label-1
   and Label-2, then it must discard this route and log an error. If the
   receiving NVE receives a RT-2 with MAC Address Length of zero, then
   it must discard this route and log an error.

5.1.2

5.2.2 Data Plane Operation - Inter Subnet

   The following description of the data-plane operation describes just
   the logical functions and the actual implementation may differ. Lets
   consider data-plane operation when TS1 in subnet-1 (MAC-VRF1) a host on NVE1 SN1 sitting behind TS1
   wants to send traffic to TS3 in subnet-3 (MAC-VRF3) on NVE2. a host sitting behind SN3 behind TS3.

   - TS1 send a packet with MAC DA corresponding to the MAC-VRF1 IRB
   interface on NVE1 (the interface between MAC-VRF1 and IP-VRF1), of NVE1, and VLAN-tag corresponding to MAC-VRF1.

   - Upon receiving the packet, the ingress NVE1 uses VLAN-tag to
   identify the MAC-VRF1. It then looks up the MAC DA and forwards the
   frame to its IRB interface. interface just like section 5.1.1.

   - The Ethernet header of the packet is stripped and the packet is fed
   to the IP-VRF where IP-VRF; where, IP lookup is performed on the destination
   address. This lookup yields an outgoing interface and the required
   encapsulation. If the encapsulation is fields needed for Ethernet NVO tunnel, then
   it includes a VxLAN encapsulation
   with NVE2's MAC address to be used as the inner MAC DA, an NVE'2 IP address
   to be used as the
   VTEP DA, and a VPN-ID to be used as the VNID. MAC SA is set to NVE1's MAC address and VTEP
   SA is set to NVE1's IP address.

   -  The packet is then encapsulated with the proper header based on
   the above info. The inner MAC SA and VTEP SA is set to NVE's MAC info and
   IP addresses respectively. The packet is then forwarded to the egress
   NVE. NVE (NVE2).

   - On the egress NVE, if NVE (NVE2), assuming the packet arrives on Ethernet NOV tunnel
   (e.g., it is VxLAN encapsulated), then
   encapsulated, the VxLAN header is removed.
   Since and the inner MAC DA is Ethernet headers are removed
   and the egress NVE's MAC address, resultant IP packet is fed to the egress
   NVE knows IP-VRF associated with that it needs to perform an
   the VNID.

   - Next, a lookup is performed based on IP lookup. It uses VNID to
   identify DA (which is in SN3) in the
   associated IP-VRF table and then performs an of NVE2. The IP lookup for yields the
   destination TS (TS3) which results in access-facing IRB
   interface over which the packet is needs to be sent. Before sending the
   packet over this interface, the ARP table is consulted to get the
   destination TS's TS (TS3) MAC address.

   - The IP packet is encapsulated with an Ethernet header with the MAC
   SA set to that of the access-facing IRB interface MAC address of the egress NVE
   (NVE2) and the MAC DA is set to that of destination TS (TS3) MAC
   address. The packet is sent to the corresponding MAC-VRF3 and after a
   lookup of MAC DA, is forwarded to the destination TS (TS3) over the
   corresponding interface.

   In this symmetric IRB scenario,

6  Inter-Subnet DCI Scenarios

   The inter-subnet traffic between NVEs
   will always use the IP-VRF VNID/MPLS label. For instance, traffic
   from TS2 to TS4 will DCI scenarios can be encapsulated by NVE1 categorized into the following
   four categories. The last two scenarios, along with its corresponding
   solution, are described in [EVPN-IPVPN-INTEROP]. The first two
   scenarios are covered in this document.

   1. Switching among IP subnets in different DCs using NVE2's IP-VRF
   VNID/MPLS label, as long as TS4's host EVPN without GW

   2. Switching among IP is present subnets in NVE1's IP-
   VRF.

5.1.3 TS Move Operation

   When a TS move from one NVE to other, it is important that the MAC
   mobility procedures are properly executed different DCs using EVPN with GW

   3. Switching among IP subnets spread across IP-VPN and the corresponding MAC-
   VRF EVPN networks
   with GW

   4. Switching among IP subnets spread across IP-VPN and IP-VRF tables on all participating NVEs are updated. [EVPN]
   describes EVPN networks
   without GW

   In the MAC mobility procedures for L2-only services for both
   single-homed TS and multi-homed TS. This section describes above scenario, the
   incremental procedures and BGP Extended Communities needed term "GW" refers to handle the MAC mobility for case where a mixed of L2 and L3 connectivity (aka IRB). In
   order to place the emphasis on node
   situated at the differences between L2-only versus
   L2-and-L3 use cases, WAN edge of the incremental procedure is described data center network behaves as a
   default gateway (GW) for
   single-homed TS with all the expectation destinations that are outside the reader can easily
   extrapolate multi-homed TS based on
   data center. The absence of GW refers to the procedures described in
   section 15 scenario where NVEs
   within a data center maintain individual (host) routes that are
   outside of [EVPN].

   Lets consider the data center.

   In the case (3), the WAN edge node also performs route aggregation
   for all the destinations within its own data center, and acts as an
   interworking unit between EVPN and IP VPN (it implements both EVPN
   and IP-VPN functionality).

                             +---+    Enterprise Site 1
                             |PE1|----- H1
                             +---+
                               /
                         ,---------.             Enterprise Site 2
                       ,'           `.    +---+
        ,---------.  /(    MPLS/IP    )---|PE2|-----  H2
       '   DCN 3   `./ `.   Core    ,'    +---+
        `-+------+'     `-+------+'
        __/__           / /      \ \
       :NVE4 :        +---+       \ \
       '-----'   ,----|GW |.       \ \
          |    ,'     +---+ `.      ,---------.
         TS6  (      DCN 1    )   ,'           `.
               `.           ,'   (      DCN 2    )
                 `-+------+'      `.           ,'
                   __/__            `-+------+'
                  :NVE1 :           __/__   __\__
                  '-----'          :NVE2 :  :NVE3 :
                   |  |            '-----'  '-----'
                  TS1 in figure-6 above where it moves from NVE1 to NVE2. TS2            |  |      |
                                    TS3 TS4   TS5

                  Figure 8: Interoperability Use-Cases

   In such move, NVE2 discovers IP1/MAC1 of TS1 what follows, we will describe scenarios 1 and realizes that it 2 in more details.

6.1 Switching among IP subnets in different DCs without GW

   This case is
   a MAC move and it advertises a MAC/IP route per similar to that of section 5.1.1 2.1 above
   with MAC Mobility Extended Community. In this IRB use case, both MAC
   and IP addresses of albeit for the TS along with their corresponding VNI/MPLS
   labels fact
   that the TS's belong to different data centers that are included
   interconnected over a WAN (e.g. MPLS/IP PSN). The data centers in the EVPN MAC/IP Advertisement route.
   Furthermore, besides MAC mobility Extended Community and Route Target
   corresponding
   question here are seamlessly interconnected to the MAC-VRF, WAN, i.e., the following additional BGP Extended
   Communities are advertised along with WAN
   edge devices do not maintain any TS-specific addresses in the MAC/IP Advertisement route:

   	- Route Target associated with IP-VRF
   	- Router's MAC Extended Community
   forwarding path - Tunnel Type Extended Community

   Since NVE2 learns TS1's MAC/IP addresses locally, it updates its MAC-
   VRF1 e.g., there is no WAN edge GW(s) between these DCs.

   As an example, consider TS3 and IP-VRF1 for TS1 with its local interface.

   If the local learning at NVE1 TS6 of Figure 2 above. Assume that
   connectivity is performed using control or
   management planes, then required between these interactions serve as two TS's where TS3 belongs to
   the trigger for
   NVE1 SN3 whereas TS6 belongs to withdraw the MAC/IP addresses SN6. NVE2 has an EVI3 associated
   with SN3 and NVE4 has an EVI6 associated with TS1. However,
   if the local learning at NVE1 is performed using data-plane learning,
   then the reception SN6. Both SN3 and
   SN6 are part of the MAC/IP Advertisement same IP-VRF.

   When an EVPN MAC advertisement route (for TS1) from
   NVE2 is received by a NVE, the IP
   address associated with MAC Mobility extended community serve as the trigger for
   NVE1 route is used to withdraw populate the MAC/IP addresses IP-VRF
   table, whereas the MAC address associated with TS1.

   All other remote NVE devices upon receiving the MAC/IP advertisement route for TS1 from NVE2 with MAC Mobility extended community compare is used to
   populate both the sequence number in this advertisement MAC-VRF table, as well as the adjacency associated
   with the one previously
   received. If IP route in the new sequence number IP-VRF table (i.e., ARP table).

   When an Ethernet frame is greater than the old one,
   then they update received by an ingress NVE, it performs a
   lookup on the MAC/IP addresses of TS1 destination MAC address in their corresponding
   MAC-VRFs and IP-VRFs to point to NVE2. Furthermore, upon receiving
   the MAC/IP withdraw for TS1 from NVE1, these remote PEs perform the
   cleanups for their BGP tables.

5.2 IRB forwarding on NVEs for Subnets behind Tenant Systems

   This section covers associated EVI. If the symmetric
   MAC address corresponds to its IRB procedures for Interface MAC address, the scenario
   where some Tenant Systems (TS's) support one or more subnets and
   these TS's are associated with one ore more NVEs. Therefore, besides ingress
   NVE deduces that the advertisement of MAC/IP addresses for each TS which can packet MUST be in inter-subnet routed. Hence, the
   presence of All-Active multi-homing,
   ingress NVE performs an IP lookup in the associated NVE needs IP-VRF table. The
   lookup identifies an adjacency that contains a MAC rewrite and in
   turn the next-hop (i.e. egress) Gateway to also
   advertise which the subnets behind each TS. packet must be
   forwarded along with the associated MPLS label stack. The main difference between this scenario and MAC rewrite
   holds the previous one is MAC address associated with the destination host (as
   populated by the
   additional advertisement corresponding to each subnet. These subnet
   advertisements are accomplished using EVPN IP Prefix route defined in
   [EVPN-PREFIX]. These subnet prefixes are advertised with MAC route), instead of the IP MAC address of their associated TS (which is the
   next-hop Gateway. The ingress NVE then rewrites the destination MAC
   address in overlay the packet with the address space) as
   their next hop. The receiving NVEs perform recursive route resolution
   to resolve specified in the subnet prefix adjacency. It
   also rewrites the source MAC address with its associated IRB Interface MAC
   address for the destination subnet. The ingress NVE so that
   they know which NVE to forward NVE, then, forwards
   the packets frame to when they are destined
   for the next-hop (i.e. egress) Gateway after encapsulating
   it with the MPLS label stack.

   Note that subnet prefix.

   The advantage of this recursive route resolution label stack includes the LSP label as well as an EVPN
   label. The EVPN label could be either advertised by the ingress
   Gateway, if inter-AS option B is that when a TS
   moves from one NVE to another, there used, or advertised by the egress
   NVE, if inter-AS option C is no need to re-advertise any
   of used. When the subnet prefixes for that TS. All it MPLS encapsulated packet
   is needed received by the ingress Gateway, the processing again differs
   depending on whether inter-AS option B or option C is to advertise employed: in
   the IP/MAC addresses associated with former case, the TS itself and exercise MAC
   mobility procedures for that TS. The recursive route resolution
   automatically takes care of ingress Gateway swaps the updates for EVPN label in the subnet prefixes of
   that TS.

   Figure below illustrates this scenario where a given tenant (e.g., an
   IP-VPN service) has three subnets represented by MAC-VRF1, MAC-VRF2,
   and MAC-VRF3 across two NVEs. There are four TS's associated
   packets with
   these three MAC-VRFs - i.e., TS1, TS5 are connected to MAC-VRF1 on
   NVE1, TS2 is connected to MAC-VRF2 on NVE1,  TS3 is connected to MAC-
   VRF3 on NVE2, the EVPN label value received from the egress Gateway.
   In the latter case, the ingress Gateway does not modify the EVPN
   label and TS4 is connected to MAC-VRF1 performs normal label switching on NVE2. TS1 has two
   subnet prefixes (SN1 and SN2) and TS3 has a single subnet prefix,
   SN3. The MAC-VRFs the LSP label.
   Similarly on each NVE are associated the egress Gateway, for option B, the egress Gateway
   swaps the EVPN label with their corresponding
   IP-VRF using their IRB interfaces. When TS4 and TS1 exchange intra-
   subnet traffic, only L2 forwarding (bridging) part of the IRB
   solution value advertised by the egress NVE.
   Whereas, for option C, the egress Gateway does not modify the EVPN
   label, and performs normal label switching on the LSP label. When the
   MPLS encapsulated packet is used (i.e., received by the traffic only goes through their MAC-
   VRFs); however, when TS3 wants to forward traffic egress NVE, it uses the
   EVPN label to SN1 or SN2
   sitting behind TS1 (inter-subnet traffic), then both bridging and
   routing parts of identify the IRB solution are exercised (i.e., bridge-domain table. It then performs a
   MAC lookup in that table, which yields the traffic
   goes through outbound interface to
   which the corresponding MAC-VRFs and IP-VRFs). The following
   subsections describe Ethernet frame must be forwarded. Figure 3 below depicts
   the control and data planes operations for this
   IRB scenario in details. packet flow.

            NVE1      +----------+
     SN1--+          +-------------+   |          |
          |--TS1-----|(MAC- \            ASBR1          ASBR2           NVE2
      +------------+  +------------+  +------------+  +------------+
      |            |  |
     SN2--+ IP1/M1            | VRF1) \  |            |  |            |     (IP-VRF)|---|
      |(MAC - (IP  |  |       /    [LS]    |  |    [LS]    |
             TS2-----|(MAC- /  |(IP  - (MAC |
      |  MPLS/ VRF)   VRF)|  |
            IP2/M2            | VRF2)  |            |  VxLAN/  |
                     +-------------+ VRF)   VRF)|
      |  NVGRE  |
                     +-------------+     |   |
     SN3--+--TS3-----|(MAC-\  |    |  |
            IP3/M3    | VRF3)\  |    |  |    |     (IP-VRF)|---|  |       |       /  | |
      +------------+  +------------+  +------------+  +------------+
         ^     v           ^  V            ^  V               ^  V
         |
             TS4-----|(MAC- /     |           |  |
            IP4/M4            | VRF1)  |               |  |
                     +-------------+   +----------+
                            NVE2
   TS1->-+     +-->--------+  +------------+  +---------------+  +->-TS2

  Figure 7: IRB forwarding on 9: Inter-Subnet Forwarding Among EVPN NVEs for Tenant Systems with configured subnets

5.2.1 Control Plane Operation

   Each NVE advertises a Route Type-5 (RT-5, IP Prefix Route defined in
   [EVPN-PREFIX]) for each of its subnet prefixes with the IP address of
   its TS as the next hop (gateway address field) as follow:

   - RD per VPN
   - ESI = 0
   - Ethernet Tag = 0;
   - IP Prefix Length = 32 or 128
   - in Different DCs
   without GW

6.2 Switching among IP Prefix = SNi
   - Gateway Address = IPi; subnets in different DCs with GW

   In this scenario, connectivity is required between TS's in different
   data centers, and those hosts belong to different IP address subnets. What
   makes this case different from that of TS
   - Label = 0

   This RT-5 Section 2.2 is advertised with that at least
   one of the data centers has a Route Target corresponding to gateway as the IP-
   VPN service.

   Each NVE also advertises WAN edge switch. Because
   of that, the NVE's IP-VRF  within that data center need not maintain
   (host) routes to individual TS's outside of that data center.

   As an RT-2 (MAC/IP Advertisement Route) along example, consider a tenant with their associated Route Targets TS1 and Extended Communities for each TS5 of its Figure 2 above.
   Assume that connectivity is required between these two TS's exactly as described in section 5.1.1.

   Upon receiving where TS1
   belongs to the RT-5 advertisement, SN1 whereas TS5 belongs to the receiving NVE performs SN5. NVE3 has an EVI5
   associated with the
   following:

   - It uses SN5 and this EVI is represented by the Route Target MAC-VRF
   which is connected to identify the corresponding IP-VRF
   - It imports via an IRB interface. NVE1 has an
   EVI1 associated with the SN1 and this EVI is represented by the MAC-
   VRF which is connected to the IP prefix into its corresponding IP-VRF with representing the IP
   address same tenant.
   Due to the gateway at the edge of DCN 1, NVE1's IP-VRF does not need
   to have the associated TS as address of TS5 but instead it has a default route in its next hop.

   Upon receiving
   IP-VRF with the RT-2 advertisement, next-hop being the GW.

   In this scenario, the NVEs within a given data center do not have
   entries for the receiving NVE imports MAC/IP addresses of hosts in remote data centers.
   Rather, the TS into the corresponding MAC-VRF and IP-VRF
   per section 5.1.1. Furthermore, it performs recursive NVEs have a default IP route
   resolution pointing to resolve the IP prefix (received in RT-5) to its
   corresponding NVE's IP address (e.g., its BGP next hop). BGP next hop
   will be used as underlay tunnel destination address (e.g., VTEP DA
   for VxLAN encapsulation) and Router's MAC will be used as inner MAC WAN gateway
   for VxLAN encapsulation.

5.2.2 Data Plane Operation

   The following description of each VRF. This is accomplished by the data-plane operation describes just WAN gateway advertising for
   a given EVPN that spans multiple DC a default VPN-IP route that is
   imported by the logical functions and NVEs of that VPN that are in the actual implementation may differ. Lets
   consider data-plane operation when gateway's own DC.

   When an Ethernet frame is received by an ingress NVE, it performs a host
   lookup on SN1 sitting behind TS1
   wants to send traffic to a host sitting behind SN3 behind TS3.

   - TS1 send a packet with the destination MAC DA corresponding address in the associated MAC-VRF
   table. If the MAC address corresponds to the MAC-VRF1 IRB
   interface of NVE1, and VLAN-tag corresponding to MAC-VRF1.

   - Upon receiving Interface MAC
   address, the packet, ingress NVE deduces that the packet MUST be inter-subnet
   routed. Hence, the ingress NVE performs an IP lookup in the
   associated IP-VRF table. The lookup, in this case, matches the ingress NVE1 uses VLAN-tag
   default host route which points to
   identify the MAC-VRF1. It local WAN gateway (GW1). The
   ingress NVE (NVE1) then looks up rewrites the destination MAC DA and forwards address in the
   frame to its
   packet with the MAC address of core-facing IRB interface just like section 5.1.1.

   - The Ethernet header of GW1 (not
   shown in the packet is stripped and the packet is fed
   to the IP-VRF; where, IP lookup is performed on the destination
   address. This lookup yields the fields needed for VxLAN encapsulation figure) or it can rewrite it with NVE2's the router's MAC
   address as of GW1. It also rewrites the inner source MAC DA, NVE'2 IP address as with its own
   core-facing IRB Interface's MAC address for the
   VTEP DA, and destination subnet
   (i.e., the VNID. MAC SA is set to NVE1's subnet between NVE1 and GW1) or it can rewrite it with its
   own router's MAC address and VTEP
   SA is set to NVE1's IP address.

   - of NVE1. The packet is then encapsulated ingress NVE, then, forwards the
   frame to GW1  after encapsulating it with the proper header based on MPLS label stack. Note
   that this label stack includes the above info and is forwarded to LSP label as well as the egress NVE (NVE2).

   - On label for
   default host route that was advertised by the egress NVE (NVE2), assuming local WAN gateway. When
   the MPLS encapsulated packet is VxLAN
   encapsulated, received by GW1, it uses the VxLAN and default
   host route MPLS label to identify the inner Ethernet headers are removed core-facing MAC-VRF. It does a
   MAC-DA lookup and forwards the resultant IP packet is fed to the IP-VRF associated with that after stripping
   the VNID.

   - Next, a lookup is performed based on Ethernet header. It then performs an IP DA (which is in SN3) lookup in the
   associated IP-VRF of NVE2. that table. The IP
   lookup yields identifies an adjacency that contains a MAC rewrite and in
   turn the access-facing IRB
   interface over remote WAN gateway (GW2) to which the packet needs to must be sent. Before sending
   forwarded along with the
   packet over this interface, associated MPLS label stack. The MAC rewrite
   holds the ARP table is consulted to get MAC address associated with the ultimate destination TS (TS3) host
   (as populated by the EVPN MAC address.

   - The IP route). GW1 then rewrites the
   destination MAC address in the packet is encapsulated with an Ethernet header the address specified in
   the adjacency. It also rewrites the source MAC address with the MAC
   SA set to that
   address of the access-facing its core-facing IRB interface of the egress NVE
   (NVE2) and (not shown in the MAC DA is set to that of destination TS (TS3) figure) or
   its router's MAC address. The packet is sent GW1, then, forwards the frame to the corresponding MAC-VRF3 and GW2
   after encapsulating it with the MPLS label stack. Note that this
   label stack includes the LSP label as well as a
   lookup of MAC DA, EVPN label that was
   advertised by GW2. When the MPLS encapsulated packet is forwarded received by
   GW2, it uses the EVPN label to identify the destination TS (TS3) over MAC-VRF. It
   then performs a MAC-DA lookup and grabs the
   corresponding interface.

6 BGP Encoding

   This document defines one new BGP Extended Community for EVPN.

6.1 Router's MAC Extended Community

   A new EVPN BGP Extended Community called Router's MAC is introduced
   here. This new extended community is a transitive extended community label advertised by
   NVE2 along with adjacencies info. It then encapsulates the packet
   with the Type field of 0x06 (EVPN) corresponding label stack and forwards the Sub-Type of 0x03. packet to NVE2.
   It may should be advertised noted that no MAC header re-write is performed on GW2.
   This implies that both GW1 and GW2 need to keep the remote host MAC
   addresses along with BGP Encapsulation Extended Community define the corresponding EVPN labels in section 4.5 of [TUNNEL-ENCAP]. their tables.
   The Router's egress NVE (NVE2) then upon receiving the packet, performs a MAC Extended Community is encoded as an 8-octet value as
   follows:

        0                   1                   2                   3
        0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3
   lookup in the MAC-VRF (identified by the received EVPN label) to
   determine the outbound port to send the traffic on.

   Figure 4 5 6 7 8 9 0 1
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ below depicts the forwarding model.

            NVE1            GW1             GW2            NVE2
      +------------+  +------------+  +------------+  +------------+
      | Type=0x06            | Sub-Type=0x03  |        Router's MAC            |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  |                      Router's MAC Cont'd            |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   This extended community is used to carry the NVE's MAC address for
   symmetric IRB scenarios and it is sent with RT-2 as described  |            |
      |(MAC - (IP  |  |(IP  - (MAC |  |    (MAC    |  |(IP  - (MAC |
      | VRF)   VRF)|  | VRF)   VRF)|  |     VRF)   |  | VRF)   VRF)|
      |  |     |   |  | |  |       |  |    |  |    |  |       |  | |
      +------------+  +------------+  +------------+  +------------+
         ^     v        ^  V               ^  V               ^  V
         |     |        |  |               |  |               |  |
   TS1->-+     +-->-----+  +---------------+  +---------------+  +->-TS2

  Figure 10: Inter-Subnet Forwarding Among EVPN NVEs in
   section 5.1.1 and 5.2.1. Different DCs
   with GW

7 TS Mobility

7.1 TS Mobility & Optimum Forwarding for TS Outbound Traffic

   Optimum forwarding for the TS outbound traffic, upon TS mobility, can
   be achieved using either the anycast default Gateway MAC and IP
   addresses, or using the address aliasing as discussed in [DC-
   MOBILITY].

7.2 TS Mobility & Optimum Forwarding for TS Inbound Traffic

   For optimum forwarding of the TS inbound traffic, upon TS mobility,
   all the NVEs and/or IP-VPN PEs need to know the up to date location
   of the TS. Two scenarios must be considered, as discussed next.

   In what follows, we use the following terminology:

   - source NVE refers to the NVE behind which the TS used to reside
   prior to the TS mobility event.

   - target NVE refers to the new NVE behind which the TS has moved
   after the mobility event.

7.2.1 Mobility without Route Aggregation

   In this scenario, when a target NVE detects that a MAC mobility event
   has occurred, it initiates the MAC mobility handshake in BGP as
   specified in section 5.1.3. The WAN Gateways, acting as ASBRs in this
   case, re-advertise the MAC route of the target NVE with the MAC
   Mobility extended community attribute unmodified. Because the WAN
   Gateway for a given data center re-advertises BGP routes received
   from the WAN into the data center, the source NVE will receive the
   MAC Advertisement route of the target NVE (with the next hop
   attribute adjusted depending on which inter-AS option is employed).
   The source NVE will then withdraw its original MAC Advertisement
   route as a result of evaluating the Sequence Number field of the MAC
   Mobility extended community in the received MAC Advertisement route.
   This is per the procedures already defined in [EVPN].

8  Acknowledgements

   The authors would like to thank Sami Boutros and Jeffrey Zhang for his
   their valuable comments.

9  Security Considerations

   The security considerations discussed in [EVPN] apply to this
   document.

10  IANA Considerations

   IANA has allocated a new transitive extended community Type of 0x06
   and Sub-Type of 0x03 for EVPN Router's MAC Extended Community.

11  References

11.1  Normative References

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119, March 1997.

   [EVPN] Sajassi et al., "BGP MPLS Based Ethernet VPN", RFC 7432,
              February, 2015.

   [TUNNEL-ENCAP] Rosen et al., "The BGP Tunnel Encapsulation
              Attribute", draft-ietf-idr-tunnel-encaps-03, November
              2016.

   [EVPN-PREFIX] Rabadan et al., "IP Prefix Advertisement in EVPN",
              draft-ietf-bess-evpn-prefix-advertisement-03, September,
              2016.

11.2  Informative References

   [RFC7606]  Chen, E., Scudder, J., Mohapatra, P., and K. Patel,
   "Revised Error Handling for BGP UPDATE Messages", RFC 7606, August
   2015, <http://www.rfc-editor.org/info/rfc7606>.

   [802.1Q] "IEEE Standard for Local and metropolitan area networks -
   Media Access Control (MAC) Bridges and Virtual Bridged Local Area
   Networks", IEEE Std 802.1Q(tm), 2014 Edition, November 2014.

   [EVPN-IPVPN-INTEROP] Sajassi et al., "EVPN Seamless Interoperability
   with IP-VPN", draft-sajassi-l2vpn-evpn-ipvpn-interop-01, work in
   progress, October, 2012.

   [DC-MOBILITY] Aggarwal et al., "Data Center Mobility based on
   BGP/MPLS, IP Routing and NHRP", draft-raggarwa-data-center-mobility-
   05.txt, work in progress, June, 2013.

12  Contributors

   In addition to the authors listed on the front page, the following
   co-authors have also contributed to this document:

   Samer Salam

   Florin Balus
   Cisco

   Yakov Rekhter
   Juniper

   Wim Henderickx
   Nokia

   Linda Dunbar
   Huawei

   Dennis Cai
   Alibaba

Authors' Addresses

   Ali Sajassi (Editor)
   Cisco
   Email: sajassi@cisco.com

   Samer Salam
   Cisco
   Email: sslam@cisco.com

   Samir Thoria
   Cisco
   Email: sthoria@cisco.com
   John E. Drake
   Juniper Networks
   Email: jdrake@juniper.net

   Lucy Yong
   Huawei Technologies
   Email: lucy.yong@huawei.com

   Jorge Rabadan
   Nokia
   Email: jorge.rabadan@nokia.com