L2VPN Workgroup                                          A. Sajassi, Ed.
INTERNET-DRAFT                                                  S. Salam
Intended Status: Standards Track                               S. Thoria
                                                                   Cisco
                                                                J. Drake
                                                                 Juniper
                                                              J. Rabadan
                                                                   Nokia

Expires: January 2, 18, 2019                                  July 2, 18, 2018

                Integrated Routing and Bridging in EVPN
            draft-ietf-bess-evpn-inter-subnet-forwarding-04
            draft-ietf-bess-evpn-inter-subnet-forwarding-05

Abstract

   EVPN provides an extensible and flexible multi-homing VPN solution
   over an MPLS/IP network for intra-subnet connectivity among Tenant
   Systems and End Devices that can be physical or virtual. However,
   there are scenarios for which there is a need for a dynamic and
   efficient inter-subnet connectivity among these Tenant Systems and
   End Devices while maintaining the multi-homing capabilities of EVPN.
   This document describes an Integrated Routing and Bridging (IRB)
   solution based on EVPN to address such requirements.

Status of this Memo

   This Internet-Draft is submitted to IETF in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF), its areas, and its working groups.  Note that
   other groups may also distribute working documents as
   Internet-Drafts.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   The list of current Internet-Drafts can be accessed at
   http://www.ietf.org/1id-abstracts.html

   The list of Internet-Draft Shadow Directories can be accessed at
   http://www.ietf.org/shadow.html

Copyright and License Notice

   Copyright (c) 2014 IETF Trust and the persons identified as the
   document authors. All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (http://trustee.ietf.org/license-info) in effect on the date of
   publication of this document. Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document. Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

Table of Contents

   1  Introduction  . . . . . . . . . . . . . . . . . . . . . . . . .  6
   2 EVPN PE Model for IRB Operation  . . . . . . . . . . . . . . . .  7
   3  Symmetric and Asymmetric IRB  . . . . . . . . . . . . . . . . .  8
     3.1 IRB Interface and its MAC & IP addresses . . . . . . . . . . 11
     3.2 Symmetric IRB Procedures . . . . . . . . . . . . . . . . . . 12 13
       3.2.1 Control Plane - Ingress PE . . . . . . . . . . . . . . . 12 13
       3.2.2 Control Plane - Egress PE  . . . . . . . . . . . . . . . 13
       3.2.3 Data Plane - Ingress PE  . . . . . . . . . . . . . . . . 14
       3.2.4 Data Plane - Egress PE . . . . . . . . . . . . . . . . . 14 15
     3.3 Asymmetric IRB Procedures  . . . . . . . . . . . . . . . . . 15
       3.3.1 Control Plane - Ingress PE . . . . . . . . . . . . . . . 15
       3.3.2 Control Plane - Egress PE  . . . . . . . . . . . . . . . 15 16
       3.3.3 Data Plane - Ingress PE  . . . . . . . . . . . . . . . . 16 17
       3.3.4 Data Plane - Egress PE . . . . . . . . . . . . . . . . . 17 18
   4 BGP Encoding . . . Mobility Procedure . . . . . . . . . . . . . . . . . . . . . . . 17 18
     4.1 Router's MAC Extended Community  . . Mobility Procedure for Symmetric IRB . . . . . . . . . . . . 17
   5 Operational Models for Symmetric Inter-Subnet Forwarding 19
       4.1.1 Initiating an ARP Request upon a Move  . . . . 18
     5.1 IRB forwarding on NVEs for Tenant Systems . . . . . 19
       4.1.2 Sending Data Traffic without an ARP Request  . . . . 18
       5.1.1 Control Plane Operation . . 20
       4.1.3 Silent Host  . . . . . . . . . . . . . . 19
       5.1.2 Data Plane Operation - Inter Subnet . . . . . . . . 21
   5 BGP Encoding . . 21
       5.1.3 TS Move Operation . . . . . . . . . . . . . . . . . . . 22
     5.2 IRB forwarding on NVEs for Subnets behind Tenant Systems . . 23
       5.2.1 Control Plane Operation . . . 22
     5.1 Router's MAC Extended Community  . . . . . . . . . . . . . 24
       5.2.2 Data Plane Operation . 22
   6 Operational Models for Symmetric Inter-Subnet Forwarding . . . . 23
     6.1 IRB forwarding on NVEs for Tenant Systems  . . . . . . . . . 23
       6.1.1 Control Plane Operation  . . . . 25
   6  Inter-Subnet DCI Scenarios . . . . . . . . . . . . 24
       6.1.2 Data Plane Operation . . . . . . 26
     6.1 Switching among IP subnets in different DCs without GW . . . 27
     6.2 Switching among IP subnets in different DCs with GW . . . . 29
   7 TS Mobility . . . . . 26
     6.2 IRB forwarding on NVEs for Subnets behind Tenant Systems . . 27
       6.2.1 Control Plane Operation  . . . . . . . . . . . . . . . . 28
       6.2.2 Data Plane Operation . . . 31
     7.1 TS Mobility & Optimum Forwarding for TS Outbound Traffic . . 31
     7.2 TS Mobility & Optimum Forwarding for TS Inbound Traffic . . 31
       7.2.1 Mobility without Route Aggregation . . . . . . . . . . . 31
   8 29
   7  Acknowledgements  . . . . . . . . . . . . . . . . . . . . . . . 32
   9 30
   8  Security Considerations . . . . . . . . . . . . . . . . . . . . 32
   10 30
   9  IANA Considerations . . . . . . . . . . . . . . . . . . . . . 32
   11 . 31
   10  References . . . . . . . . . . . . . . . . . . . . . . . . . . 32
     11.1 31
     10.1  Normative References . . . . . . . . . . . . . . . . . . . 32
     11.2 31
     10.2  Informative References . . . . . . . . . . . . . . . . . . 32
   12 31
   11  Contributors . . . . . . . . . . . . . . . . . . . . . . . . . 33 32
   Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 33 32

Terminology

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
   "OPTIONAL" in this document are to be interpreted as described in BCP
   14 [RFC2119] [RFC8174] when, and only when, they appear in all
   capitals, as shown here.

   AC: Attachment Circuit.

   ARP: Address Resolution Protocol.

   BD: Broadcast Domain. As per [RFC7432], an EVI consists of a single
   or multiple BDs. In case of VLAN-bundle and VLAN-based service models
   (see [RFC7432]), a BD is equivalent to an EVI. In case of VLAN-aware
   bundle service model, an EVI contains multiple BDs. Also, in this
   document, BD and subnet are equivalent terms.

   BD Route Target: refers to the Broadcast Domain assigned Route Target
   [RFC4364]. In case of VLAN-aware bundle service model, all the BD
   instances in the MAC-VRF share the same Route Target.

   BT: Bridge Table. The instantiation of a BD in a MAC-VRF, as per
   [RFC7432].

   DGW: Data Center Gateway.

   Ethernet A-D route: Ethernet Auto-Discovery (A-D) route, as per
   [RFC7432].

   Ethernet NVO tunnel: refers to Network Virtualization Overlay tunnels
   with Ethernet payload. Examples of this type of tunnels are VXLAN or
   GENEVE.

   EVI: EVPN Instance spanning the NVE/PE devices that are participating
   on that EVPN, as per [RFC7432].

   EVPN: Ethernet Virtual Private Networks, as per [RFC7432].

   GRE: Generic Routing Encapsulation.

   GW IP: Gateway IP Address.

   IPL: IP Prefix Length.

   IP NVO tunnel: it refers to Network Virtualization Overlay tunnels
   with IP payload (no MAC header in the payload).

   IP-VRF: A VPN Routing and Forwarding table for IP routes on an
   NVE/PE. The IP routes could be populated by EVPN and IP-VPN address
   families. An IP-VRF is also an instantiation of a layer 3 VPN in an
   NVE/PE.

   IRB: Integrated Routing and Bridging interface. It connects an IP-VRF
   to a BD (or subnet).

   MAC-VRF: A Virtual Routing and Forwarding table for Media Access
   Control (MAC) addresses on an NVE/PE, as per [RFC7432]. A MAC-VRF is
   also an instantiation of an EVI in an NVE/PE.

   ML: MAC address length.

   ND: Neighbor Discovery Protocol.

   NVE: Network Virtualization Edge.

   GENEVE: Generic Network Virtualization Encapsulation, [GENEVE].

   NVO: Network Virtualization Overlays.

   RT-2: EVPN route type 2, i.e., MAC/IP advertisement route, as defined
   in [RFC7432].

   RT-5: EVPN route type 5, i.e., IP Prefix route. As defined in Section
   3 of [EVPN-PREFIX].

   SBD: Supplementary Broadcast Domain. A BD that does not have any ACs,
   only IRB interfaces, and it is used to provide connectivity among all
   the IP-VRFs of the tenant. The SBD is only required in IP-VRF- to-IP-
   VRF use-cases (see Section 4.4.).

   SN: Subnet.

   TS: Tenant System.

   VA: Virtual Appliance.

   VNI: Virtual Network Identifier. As in [RFC8365], the term is used as
   a representation of a 24-bit NVO instance identifier, with the
   understanding that VNI will refer to a VXLAN Network Identifier in
   VXLAN, or Virtual Network Identifier in GENEVE, etc. unless it is
   stated otherwise.

   VTEP: VXLAN Termination End Point, as in [RFC7348].

   VXLAN: Virtual Extensible LAN, as in [RFC7348].

   This document also assumes familiarity with the terminology of
   [RFC7432], [RFC8365] and [RFC7365].

1  Introduction

   EVPN provides an extensible and flexible multi-homing VPN solution
   over an MPLS/IP network for intra-subnet connectivity among Tenant
   Systems (TS's) and End Devices that can be physical or virtual; where
   an IP subnet is represented by an EVI for a VLAN-based service or by
   an <EVI, VLAN> for a VLAN-aware bundle service. However, there are
   scenarios for which there is a need for a dynamic and efficient
   inter-subnet connectivity among these Tenant Systems and End Devices
   while maintaining the multi-homing capabilities of EVPN. This
   document describes an Integrated Routing and Bridging (IRB) solution
   based on EVPN to address such requirements.

   The inter-subnet communication is traditionally achieved at
   centralized L3 Gateway (L3GW) nodes devices where all the inter-subnet
   forwarding are performed and all the inter-subnet communication
   policies are enforced. When two Tenant Systems (TS's) belonging to
   two different subnets connected to the same PE node, wanted to
   communicate with each other, their traffic needed to be back hauled
   from the PE node all the way to the centralized gateway
   nodes node where
   inter-subnet switching is performed and then back to the PE node. For
   today's large multi-tenant data center, this scheme is very
   inefficient and sometimes impractical.

   In order to overcome the drawback of centralized L3GW approach, IRB
   functionality is needed on the PE nodes (also referred to as EVPN
   NVEs) attached to TS's in order to avoid inefficient forwarding of
   tenant traffic (i.e., avoid back-hauling and hair-pinning). A When a PE
   with IRB capability, capability receives tenant traffic over a single Attachment
   Circuit (AC), it can not only locally bridged the tenant intra-subnet
   traffic but also can locally route the tenant inter-subnet traffic on
   a packet by packet basis thus meeting the requirements for both intra
   and inter-subnet forwarding and avoiding non-optimum traffic
   forwarding associate with centralized L3GW approach.

   Some TS's run non-IP protocols in conjunction with their IP traffic.
   Therefore, it is important to handle both kinds of traffic optimally
   - e.g., to bridge non-IP and intra-subnet traffic and to route inter-
   subnet IP traffic. Therefore, the solution needs to meet the
   following requirements:

   R1: The solution MUST allow for both inter-subnet and intra-subnet
   traffic belonging to the same tenant to be locally routed and bridged
   respectively. The solution MUST provide IP routing for inter-subnet
   traffic and Ethernet Bridging for intra-subnet traffic.

   R2: The solution MUST support bridging of for non-IP traffic.

   R3: The solution MUST allow inter-subnet switching to be disabled on
   a per VLAN basis on PEs where the traffic needs to be back hauled to
   another node (i.e., for performing FW or DPI functionality).

2 EVPN PE Model for IRB Operation

   Since this document discusses IRB operation in relationship to EVPN
   MAC-VRF, IP-VRF, EVI, Bridge Domain (BD), Bridge Table (BT), and IRB
   interfaces, it is important to understand the relationship among
   these components. Therefore, the following PE model is demonstrated
   below to a) describe these components and b) illustrate the
   relationship among them.

      +-------------------------------------------------------------+
      |                                                             |
      |              +------------------+                    IRB PE |
      | Attachment   | +------------------+                         |
      | Circuit(AC1) | |  +----------+    |                MPLS/NVO tnl
    ----------------------*Bridge    |    |                    +-----
      |              | |  |Table(BT1)|    |    +-----------+  / \    \
      |              | |  |          *---------*           |<--> |Eth|
      |              | |  |Eth-Tag x |    |IRB1|           |  \ /    /
      |              | |  +----------+    |    |           |   +-----
      |              | |     ...          |    |  IP-VRF1  |        |
      |              | |  +----------+    |    |  RD2/RT2  |MPLS/NVO tnl
      |              | |  |Bridge    |    |    |           |   +-----
      |              | |  |Table(BT2)|    |IRB2|           |  / \    \
      |              | |  |          *---------*           |<--> |IP |
    ----------------------*Eth-Tag y |    |    +-----------+  \ /    /
      |  AC2         | |  +----------+    |                    +-----
      |              | |    MAC-VRF1      |                         |
      |              +-+    RD1/RT1       |                         |
      |                +------------------+                         |
      |                                                             |
      |                                                             |
      +-------------------------------------------------------------+

                      Figure 1: EVPN IRB PE Model

   A tenant needing IRB services on a PE, requires an IP Virtual Routing
   and Forwarding table (IP-V RF) (IP-VRF) along with one or more MAC Virtual
   Routing and Forwarding tables (MAC-VRFs). An IP-VRF, as defined in
   [RFC4364], is the instantiation of an IPVPN in a PE. A MAC-VRF, as
   defined in [RFC7432], is the instantiation of an EVI (EVPN Instancce)
   in a PE. A MAC-VRF can consists of one or more Bridge Tables (BTs)
   where each BT corresponds to a VLAN (broadcast domain - BD). If the
   service interface interfaces for the an EVPN PE is are configured in VLAN-Based mode
   (i.e., section 6.1 of [RFC7432]), then there is only a single BT per
   MAC-VRF (per EVI) - i.e., there is only one tenant VLAN per EVI.
   However, if the service interface interfaces for the an EVPN PE is are configured in
   VLAN-Aware VLAN-
   Aware Bundle mode (i.e., section 6.3 of [RFC7432]), then there are
   several BTs per MAC-VRF (per EVI) - i.e., there are several tenant
   VLANs per EVI.

   Each BT is connected to a IP-VRF via a L3 interface called IRB
   interface. Since a single tenant subnet is typically (and in this
   document) represented by a VLAN (and thus supported by a single BT),
   for a given tenant there are as many BTs as there are subnets and
   thus there are also as many IRB interfaces between the tenant IP-VRF
   and the associated BTs as shown in the PE model above.

   IP-VRF is identified by its corresponding route target and route
   distinguisher and MAC-VRF is also identified by its corresponding
   route target and route distinguisher. If operating in EVPN VLAN-Based
   mode, then a receiving PE that receives an EVPN route with MAC-VRF
   route target can identify the corresponding BT; however, if operating
   in EVPN VLAN-Aware Bundle mode, then the receiving PE needs both the
   MAC-VRF route target and VLAN ID in order to identify the
   corresponding BT.

3  Symmetric and Asymmetric IRB

   This document defines and describes two types of IRB solutions -
   namely symmetric and asymmetric IRB. In symmetric IRB as its name
   implies, the lookup operation is symmetric at both ingress and egress
   PEs - i.e., both ingress and egress PEs perform lookups on both TS's MAC
   and IP addresses - i.e., addresses. The ingress PE performs lookup on
   destination TS's a MAC address lookup followed by its an
   IP address lookup and the egress PE performs lookup on destination TS's a IP address lookup followed by its a MAC
   address
   lookup as depicted in figure 2.

               Ingress PE                   Egress PE
         +-------------------+        +------------------+
         |                   |        |                  |
         |    +-> IP-VFF IP-VRF ----|---->---|-----> IP-VRF -+  |
         |    |              |        |               |  |
         |   BT1        BT2  |        |  BT3         BT2 |
         |    |              |        |               |  |
         |    ^              |        |               v  |
         |    |              |        |               |  |
         +-------------------+        +------------------+
              ^                                       |
              |                                       |
        TS1->-+                                       +->-TS2
                        Figure 2: Symmetric IRB

   In symmetric IRB as shown in figure-2, the inter-subnet forwarding
   between two PEs is done between their associated IP-VRFs. Therefore,
   the tunnel connecting these IP-VRFs can be either IP-only tunnel (in
   case of MPLS or GENEVE encapsulation) or Ethernet NVO tunnel (in case
   of VxLAN encapsulation). If it is an Ethernet NOV NVO tunnel, the TS's IP
   packet is encapsulated in an Ethernet header consisting of ingress
   and egress PEs MAC addresses - i.e., there is no need for ingress PE
   to use the destination TS's MAC address. Therefore, in symmetric IRB,
   there is no need for the ingress PE to hold maintain ARP entries for
   destination TS's TS IP and MAC addresses association in its ARP table.
   Each PE participating in symmetric IRB only maintains ARP entries for
   locally connected hosts and
   maintain maintains MAC-VRFs/BTs for only locally
   configured subnets.

   In asymmetric IRB, the lookup operation is asymmetric and the ingress
   PE performs three lookups; whereas the egress PE performs a single
   lookup - i.e., the ingress PE performs lookups on destination TS's a MAC address, lookup, followed by its an
   IP address, lookup, followed by its a MAC address lookup again; whereas, the egress PE
   performs just a single lookup on
   destination TS's MAC address lookup as depicted in figure 3 below.

            Ingress PE                       Egress PE
         +-------------------+        +------------------+
         |                   |        |                  |
         |    +-> IP-VFF IP-VRF ->  |        |      IP-VRF      |
         |    |           |  |        |                  |
         |   BT1        BT2  |        |  BT3         BT2 |
         |    |           |  |        |              | | |
         |    |           +--|--->----|--------------+ | |
         |    |              |        |                v |
         +-------------------+        +----------------|-+
              ^                                        |
              |                                        |
        TS1->-+                                        +->-TS2
                        Figure 3: Asymmetric IRB

   In asymmetric IRB as shown in figure-2, figure-3, the inter-subnet forwarding
   between two PEs is done between their associated MAC-VRFs/BTs.
   Therefore, the MPLS or NVO tunnel used for inter-subnet forwarding
   MUST be of type Ethernet. Since at the egress PE only MAC lookup is
   performed (e.g., no IP lookup), the TS's IP packet needs packets need to be
   encapsulated with the destination TS's MAC address. In order for
   ingress PE to perform such encapsulation, it needs to maintain TS's
   IP and MAC address association in its ARP table. Furthermore, it
   needs to maintain destination TS's MAC address in the corresponding
   BT even though it does may not have any TS of the corresponding subnet
   locally
   configured. attached. In other words, each PE participating in asymmetric
   IRB MUST maintain ARP entries for remote hosts (hosts connected to
   other PEs) as well as maintaining MAC-VRFs/BTs for subnets that are may
   not be locally present on that PE.

   The following subsection defines the control and data planes
   procedures for symmetric and asymmetric IRB on ingress and egress
   PEs. The following figure is used in description of these procedures
   where it shows a single IP-VRF and a number of BTs on each PE for a
   given tenant. The IP-VRF of the tenant (i.e., IP-VRF1) is connected
   to each BT via its associated IRB interface. Each BT on a PE is
   associated with a unique VLAN (e.g., with a BD) where in turn is
   associated with a single MAC-VRF in case of VLAN-Based mode or a
   number of BTs can be associated with a single MAC-VRF in case of
   VLAN-Aware Bundle mode. Whether the service interface on a PE is
   VLAN-Based or VLAN-Aware Bundle mode does not impact the IRB
   operation and procedures. It only impacts the setting of Ethernet tag
   field in EVPN BGP routes as described in [RFC7432].

                    PE 1         +---------+
              +-------------+    |         |
      TS1-----|         MACx|    |         |        PE2
    (IP1/M1)  |(BT1)        |    |         |   +-------------+
      TS5-----|      \      |    |  MPLS/  |   |MACy  (BT3)  |-----TS3
    (IP5/M5)  |Mx/IPx \     |    |  VxLAN/ |   |     /       |  (IP3/M3)
              |    (IP-VRF1)|----|  NVGRE  |---|(IP-VRF1)    |
              |       /     |    |         |   |     \       |
      TS2-----|(BT2) /      |    |         |   |      (BT1)  |-----TS4
    (IP2/M2)  |             |    |         |   |             |   (IP4/M4)
              +-------------+    |         |   +-------------+
                                 |         |
                                 +---------+

                       Figure 4: IRB forwarding

3.1 IRB Interface and its MAC & IP addresses

   To support inter-subnet forwarding on a PE, the PE acts as an IP
   Default Gateway from the perspective of the attached Tenant Systems
   where default gateway MAC and IP addresses are configured on each IRB
   interface associated with its subnet and falls into one of the
   following two options:

   1. All the PEs for a given tenant subnet use the same anycast default
   gateway IP and MAC addresses . On each PE, this default gateway IP
   and MAC addresses correspond to the IRB interface connecting the BT
   associated with the tenant's <EVI, VLAN> to the corresponding
   tenant's IP-VRF.

   2. Each PE for a given tenant subnet uses the same anycast default
   gateway IP address but its own MAC address. These MAC addresses are
   aliased to the same anycast default gateway IP address through the
   use of the Default Gateway extended community as specified in [EVPN],
   [RFC7432], which is carried in the EVPN MAC/IP Advertisement routes.
   On each PE, this default gateway IP address along with its associated
   MAC addresses correspond to the IRB interface connecting the BT
   associated with the tenant's <EVI, VLAN> to the corresponding
   tenant's IP-VRF.

   It is worth noting that if the applications that are running on the
   TS's are employing or relying on any form of MAC security, then
   either the first model (i.e. using anycast MAC address) should be
   used to ensure that the applications receive traffic from the same
   IRB interface MAC address that they are sending to, or if the second
   model is used, then the IRB interface MAC address MUST be the one
   used in the initial ARP reply for that TS.

   Although both of these options are equally applicable to both
   symmetric and asymmetric IRB, the option-1 is recommended because of
   the ease of anycast MAC address provisioning on not only the IRB
   interface associated with a given subnet across all the PEs
   corresponding to that EVI but also on all IRB interfaces associated
   with all the tenant's subnets across all the PEs corresponding to all
   the EVIs for that tenant. Furthermore, it simplifies the operation as
   there is no need for Default Gateway extended community advertisement
   and its associated MAC aliasing procedure.

   When a TS sends an ARP request Yet another advantage is
   that following host mobility, the host does not need to refresh the PE that
   default GW ARP entry.

   If option-1 is attached to, used, an implementation MAY choose to auto-derive the ARP
   request
   anycast MAC address. If auto-derivation is sent for used, the IP address anycast MAC MUST
   be auto-derived out of the following ranges (which are defined in
   [RFC5798]):

   - Anycast IPv4 IRB interface associated
   with the TS's subnet. For example, case: 00-00-5E-00-01-{VRID} (in hex, in figure 4, TS1 Internet
   standard bit-order)

   - Anycast IPv6 IRB case: 00-00-5E-00-02-{VRID} (in hex, in Internet
   standard bit-order)

   Where the last octet is configured
   with generated based on a configurable Virtual
   Router ID (VRID, range 1-255)). If not explicitly configured, the anycast IPx address as its
   default gateway IP address and
   thus when it sends an ARP request for IPx (IP address of the IRB
   interface value for BT1), the PE1 sends an ARP reply with the MACx which VRID octet is '01'. Auto-derivation of the
   anycast MAC address of can only be used if there is certainty that IRB interface. the auto-
   derived MAC does not collide with any customer MAC address.

   In addition to IP anycast addresses, IRB interfaces can be configured
   with non-anycast IP addresses for the purpose of OAM (such as
   traceroute/ping to these interfaces) for both symmetric and
   asymmetric IRB. These IP addresses need to be distributed as VPN
   routes when PEs operating in symmetric IRB mode. However, they don't
   need to be distributed if the PEs are operating in asymmetric IRB
   mode and the IRB interfaces non-anycast IP addresses are configured with individual
   MACs.

3.2 Symmetric IRB Procedures

3.2.1 Control Plane - Ingress PE

   When a PE (e.g., PE1 in figure 4 above) learns MAC and IP address

   Irrespective of
   a TS (via an ARP request), it adds using only the MAC anycast address to or both anycast and
   non-anycast addresses on the same IRB, when a TS sends an ARP request
   to the PE that is attached to, the ARP request is sent for the
   anycast IP address of the IRB interface associated with the TS's
   subnet. For example, in figure 4, TS1 is configured with the anycast
   IPx address as its default gateway IP address and thus when it sends
   an ARP request for IPx (anycast IP address of the IRB interface for
   BT1), the PE1 sends an ARP reply with the MACx which is the anycast
   MAC address of that IRB interface. Traffic routed from IP-VRF1 to TS1
   SHOULD use the anycast MAC address as source MAC address.

3.2 Symmetric IRB Procedures

3.2.1 Control Plane - Ingress PE

   When a PE (e.g., PE1 in figure 4 above) learns MAC and IP address of
   a TS (via an ARP request), it adds the MAC address to the
   corresponding MAC-VRF/BT of that tenant's subnet and adds the IP
   address to the IP-VRF for that tenant. Furthermore, it adds this TS's
   MAC and IP address association to its ARP table. It then builds an
   EVPN MAC/IP Advertisement route (type 2) as follow follows and advertises it
   to other PEs participating in that tenant's VPN.

   - The Length field of the BGP EVPN NLRI for an EVPN MAC/IP
   Advertisement route MUST be either 40 (if IPv4 address is carried) or
   52 (if IPv6 address is carried).

   - Route Distinguisher (RD), Ethernet Segment Identifier, Ethernet Tag
   ID, MAC Address Length, MAC Address, IP Address Length, IP Address,
   and MPLS Label1 fields MUST be used as defined in set per [RFC7432] and [RFC8365].

   - The MPLS Label2 field is set to either an MPLS label or a VNI
   corresponding to the tenant's IP-VRF. In case of an MPLS label, this
   field is encoded as 3 octets, where the high-order 20 bits contain
   the label value.

   Just as in [RFC7432], the RD, Ethernet Tag ID, MAC Address Length,
   MAC Address, IP Address Length, and IP Address fields are part of
   the route key used by BGP to compare routes. The rest of the fields
   are not part of the route key.

   This route is advertised along with the following two extended
   communities:

        1) Tunnel Type Extended Community
        2) Router's MAC Extended Community

   For symmetric IRB mode, Router's MAC EC is needed to carry the PE's
   overlay MAC address (e.g., inner MAC address in NVO encapsulation)
   which is used for IP-VRF to IP-VRF communications with Ethernet NVO
   tunnel. If MPLS or IP-only NVO tunnel is used, then there is no need
   to send Router's MAC Extended Community along with this route.

   This route MUST be advertised with two route targets - one
   corresponding to the MAC-VRF of the tenant's subnet and another
   corresponding to the tenant's IP-VRF.

3.2.2 Control Plane - Egress PE
   When a PE (e.g., PE2 in figure 4 above) receives this EVPN MAC/IP
   Advertisement route advertisement, it performs the following:

   - Using MAC-VRF route target, Route Target (and Ethernet Tag if different from
   zero), it identifies the corresponding MAC-
   VRF. MAC-VRF (and BT). If the MAC-VRF MAC-
   VRF (and BT) exists (e.g., it is locally configured) then it imports
   the MAC address into it. Otherwise, it does not import the MAC
   address.

   - Using IP-VRF route target, it identifies the corresponding IP-VRF
   and imports the IP address into it.

   The inclusion of MPLS label2 field in this route signals to the
   receiving PE that this route is for symmetric IRB mode and MPLS
   label2 needs to be installed in forwarding path to identify the
   corresponding IP-VRF.

   If the receiving PE receives this route with both the MAC-VRF and IP-
   VRF route targets but the MAC/IP Advertisement route does not include
   MPLS label2 field and if the receiving PE supports asymmetric IRB
   mode, then the receiving PE installs the MAC address in the
   corresponding MAC-VRF and <IP, MAC> association in the ARP table for
   that tenant (identified by the corresponding IP-VRF route target).

   If the receiving PE receives this route with both the MAC-VRF and IP-
   VRF route targets but the MAC/IP Advertisement route does not include
   MPLS label2 field and if the receiving PE does not support asymmetric
   IRB mode, then if it has the corresponding MAC-VRF, it only imports
   the MAC address; otherwise, if it doesn't have the corresponding MAC-
   VRF, it MUST treat the route as withdraw [RFC7606].

3.2.3 Data Plane - Ingress PE

   When [RFC7606] and log an error
   message.

   If the receiving PE receives this route with both the MAC-VRF and IP-
   VRF route targets and the MAC/IP Advertisement route includes MPLS
   label2 field but the receiving PE only supports asymmetric IRB mode,
   then the receiving PE MUST ignore MPLS label2 field and install the
   MAC address in the corresponding MAC-VRF and <IP, MAC> association in
   the ARP table for that tenant (identified by the corresponding IP-VRF
   route target).

3.2.3 Data Plane - Ingress PE

   When an Ethernet frame is received by an ingress PE (e.g., PE1 in
   figure 4 above), the PE uses the AC ID (e.g., VLAN ID) to identify
   the associated MAC-VRF/BT and it performs a lookup on the destination
   MAC address. If the MAC address corresponds to its IRB Interface MAC
   address, the ingress PE deduces that the packet must be inter-subnet
   routed. Hence, the ingress PE performs an IP lookup in the associated
   IP-VRF table. The lookup identifies BGP next hop of egress PE along
   with the tunnel/encapsulation type and the associated MPLS/VNI
   values.

   If the tunnel type is that of MPLS or IP-only NVO tunnel, then TS's
   IP packet is sent over the tunnel without any Ethernet header.
   However, if the tunnel type is that of Eternet Ethernet NVO tunnel, then an
   Ethernet header needs to be added to the TS's IP packet. The source
   MAC address of this inner Ethernet header is set to the ingress PE's
   router MAC address and the destination MAC address of this inner
   Ethernet header is set to the egress PE's router MAC address. The
   MPLS VPN label or VNI fields are set accordingly and the packet is
   forwarded to the egress PE.

   If case of NVO tunnel encapsulation, the outer source IP address is
   set to the ingress PE's BGP next-hop address and outer destination
   IP
   address is addresses are set to the ingress and egress PE's PE BGP next-hop address. IP
   addresses respectively.

3.2.4 Data Plane - Egress PE

   When the tenant's MPLS or NVO encapsulated packet is received over an
   MPLS or NVO tunnel by the egress PE, the egress PE removes NVO tunnel
   encapsulation and uses the VPN MPLS label (for MPLS encapsulation) or
   VNI (for VxLAN NVO encapsulation) to identify the IP-VRF in which IP lookup
   needs to be performed. If the VPN MPLS label or VNI identifies a MAC-
   VRF instead of an IP-VRF, then the procedures in section 3.3.4 for
   asymmetric IRB are executed.

   The lookup in the IP-VRF identifies a local adjacency to the IRB
   interface associated with the egress subnet's MAC-VRF/BT.

   The egress PE gets the destination TS's MAC address for that TS's IP
   address from its ARP table, it encapsulates the packet with that
   destination MAC address and a source MAC address corresponding to
   that IRB interface and sends the packet to its destination subnet
   MAC-VRF/BT.

   The destination MAC address lookup in the MAC-VRF/BT results in local
   adjacency (e.g., local interface) over which the Ethernet frame is
   sent on.

3.3 Asymmetric IRB Procedures

3.3.1 Control Plane - Ingress PE
   When a PE (e.g., PE1 in figure 4 above) learns MAC and IP address of
   a TS (via (e.g., via an ARP request), it populates its MAC-VRF/BT, IP-VRF,
   and ARP table just as in the case for symmetric IRB. It then builds
   an EVPN MAC/IP Advertisement route (type 2) as follow and advertises
   it to other PEs participating in that tenant's VPN.

   - The Length field of the BGP EVPN NLRI for an EVPN MAC/IP
   Advertisement route MUST be either 37 (if IPv4 address is carried) or
   49 (if IPv6 address is carried).

   - Route Distinguisher (RD), Ethernet Segment Identifier, Ethernet Tag
   ID, MAC Address Length, MAC Address, IP Address Length, IP Address,
   and MPLS Label1 fields MUST be used as defined in set per [RFC7432] and [RFC8365].

   - The MPLS Label2 field MUST NOT be included in this route.

   Just as in [RFC7432], the RD, Ethernet Tag ID, MAC Address Length,
   MAC Address, IP Address Length, and IP Address fields are part of
   the route key used by BGP to compare routes. The rest of the fields
   are not part of the route key.

   This route is advertised along with the following extended
   communitiy:

        1) Tunnel Type Extended Community

   For asymmetric IRB mode, Router's MAC EC is not needed because
   forwarding is performed using destination TS's MAC address which is
   carried in this EVPN route type-2 advertisement.

   This route MUST always be advertised with the MAC-VRF route target.
   It MAY also be advertised with a second route target corresponding to
   the IP-VRF. If only MAC-VRF route target is used, then the receiving
   PE uses the MAC-VRF route target to identify the corresponding IP-VRF
   - i.e., many MAC-VRF route targets map to the same IP-VRF for a given
   tenant. Since in this asymmetric IRB mode, each PE is configured with
   every BD for a tenant, the MAC-VRF route target has the same
   reachability as the IP-VRF route target and that is why the use of
   IP-VRF route target is optional for this IRB mode.

3.3.2 Control Plane - Egress PE

   When a PE (e.g., PE2 in figure 4 above) receives this EVPN MAC/IP
   Advertisement route advertisement, it performs the following:

   - Using MAC-VRF route target, it identifies the corresponding MAC-VRF
   and imports the MAC address into it. For asymmetric IRB mode, it is
   assumed that all PEs participating in a tenant's VPN are configured
   with all subnets and corresponding MAC-VRFs/BTs even if there are no
   locally attached TS's for some of these subnets. The reason for this
   is because ingress PE needs to do forwarding based on destination
   TS's MAC address and does proper NVO tunnel encapsulation which are
   property of a lookup in MAC-VRF/BT. An implementation may choose to
   consolidate the lookup at the ingress PE's IP-VRF with the lookup at
   the ingress PE's destination subnet MAC-VRF. Consideration for such
   consolidation of lookups is an implementation exercise and thus its
   specification is outside the scope of this document.

   - Using MAC-VRF route target, it identifies the corresponding ARP
   table for the tenant and it adds an entry to the ARP table for the
   TS's MAC and IP address association. It should be noted that the
   tenant's ARP table at the receiving PE is identified by all the MAC-
   VRF route targets for that tenant. If IP-VRF route target is included
   with this route advertisement, then it MAY be used for the
   identification of tenant's ARP table.

   For

   If the receiving PE receives the MAC/IP Advertisement route with MPLS
   label2 field but the receiving PE only supports asymmetric IRB mode,
   then the receiving PE MUST ignore MPLS label2 field SHOULD not be included and install the
   MAC address in the route; however, if corresponding MAC-VRF and <IP, MAC> association in
   the ARP table for that tenant (identified by either MAC-VRF or IP-VRF
   route targets).

   If the receiving PE receives this the MAC/IP Advertisement route with
   the MPLS
   label2 field, field and it can support symmetric IRB mode, then it SHOULD ignore it. should
   use the MAC-VRF route target to identify its corresponding MAC-VRF
   table and import the MAC address. It should use the IP-VRF route
   target to identify the corresponding IP-VRF table and import the IP
   address. It MUST not import <IP, MAC> association into its ARP
   table.

3.3.3 Data Plane - Ingress PE

   When an Ethernet frame is received by an ingress PE (e.g., PE1 in
   figure 4 above), the PE uses the AC ID (e.g., VLAN ID) to identify
   the associated MAC-VRF/BT and it performs a lookup on the destination
   MAC address. If the MAC address corresponds to its IRB Interface MAC
   address, the ingress PE deduces that the packet must be inter-subnet
   routed. Hence, the ingress PE performs an IP lookup in the associated
   IP-VRF table. The lookup identifies a local adjacency to the IRB
   interface associated with the egress subnet's MAC-VRF/BT.

   The ingress PE gets the destination TS's MAC address for that TS's IP
   address from its ARP table, it encapsulates the packet with that
   destination MAC address and a source MAC address corresponding to
   that IRB interface and sends the packet to its destination subnet
   MAC-VRF/BT.

   The destination MAC address lookup in the MAC-VRF/BT results in BGP
   next hop address of egress PE. PE along with VPN MPLS label or VNI. The
   ingress PE encapsulates the packet using Ethernet NVO tunnel of the
   choice (e.g., VxLAN or GENEVE) and sends the packet to the egress PE.
   Since the packet forwarding is between ingress PE's MAC-VRF/BT and
   egress PE's MAC-VRF/BT, the packet encapsulation procedures follows
   that of [RFC7432] for MPLS and [RFC8365] for VxLAN encapsulations.

3.3.4 Data Plane - Egress PE

   When a tenant's Ethernet frame is received over an NVO tunnel by the
   egress PE, the egress PE removes NVO tunnel encapsulation and uses
   the VPN MPLS label (for MPLS encapsulation) or VNI (for VxLAN NVO
   encapsulation) to identify the MAC-VRF/BT in which MAC lookup needs
   to be performed.

   The MAC lookup results in local adjacency (e.g., local interface)
   over which the packet needs to get sent.

   Note that the forwarding behavior on the egress PE is the same as
   EVPN intra-subnet forwarding described in [RFC7432] for MPLS and
   [RFC8365] for VxLAN NVO networks. In other words, all the packet processing
   associated with the inter-subnet forwarding semantics is confined to
   the ingress PE. PE for asymmetric IRB mode.

   It should also be noted that [RFC7432] provides different level of
   granularity for the EVPN label.  Besides identifying bridge domain
   table, it can be used to identify the egress interface or a
   destination MAC address on that interface. If EVPN label is used for
   egress interface or individual MAC address identification, then no
   MAC lookup is needed in the egress PE for MPLS encapsulation and the
   packet can be directly forwarded to the egress interface just based
   on EVPN label lookup.

4 BGP Encoding

   This document defines Mobility Procedure

   When a TS move from one new BGP Extended Community for EVPN.

4.1 Router's NVE (aka source NVE) to another NVE (aka
   target NVE), it is important that the MAC Extended Community

   A new EVPN mobility procedures are
   properly executed and the corresponding MAC-VRF and IP-VRF tables on
   all participating NVEs are updated. [RFC7432] describes the MAC
   mobility procedures for L2-only services for both single-homed TS and
   multi-homed TS. This section describes the incremental procedures and
   BGP Extended Community called Router's Communities needed to handle the MAC is introduced
   here. This new extended community is a transitive extended community
   with mobility for IRB.

   In order to place the Type field of 0x06 (EVPN) emphasis on the differences between L2-only and
   IRB use cases, the Sub-Type of 0x03. It may
   be advertised along incremental procedure is described for single-
   homed TS with BGP Encapsulation Extended Community define the expectation that the reader can easily extrapolate
   multi-homed TS based on the procedures described in section 4.5 15 of [TUNNEL-ENCAP].

   The Router's MAC Extended Community is encoded as an 8-octet value as
   follows:

        0                   1                   2                   3
        0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       | Type=0x06     | Sub-Type=0x03 |        Router's MAC           |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |                      Router's MAC Cont'd                      |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

               Figure 5: Router's MAC Extended Community
   [RFC7432]. This extended community is used to carry the PE's MAC address section describes mobility procedures for both
   symmetric IRB scenarios and it is sent with RT-2.

5 Operational Models asymmetric IRB.

4.1 Mobility Procedure for Symmetric Inter-Subnet Forwarding

   The following sections describe two main symmetric IRB forwarding
   scenarios (within a DC - i.e., intra-DC) along with their
   corresponding procedures. In the following scenarios, without loss of
   generality, it is assumed that

   When a given tenant is represented by TS moves from a
   single IP-VPN instance. Therefore, on source NVE to a given PE, target NVE, it can behave in
   one of the following three ways:

   1) TS initiates an ARP request upon a tenant move to the target NVE

   2) TS sends data packet without first initiating an ARP request to
   the target NVE

   3) TS is
   represented by a single IP-VRF table silent host and one or more MAC-VRF tables.

5.1 IRB forwarding on NVEs for Tenant Systems

   This section covers neither initiates an ARP request nor sends
   any packets

   The following subsections describe the symmetric IRB procedures for the scenario
   where each Tenant System (TS) of the
   above options. In the following subsections, it is assumed that the
   MAC & IP addresses of a TS have one-to-one relationship (i.e., there
   is attached to one or more NVEs and its
   host IP and address per MAC addresses are learned by the attached NVEs address and are
   distributed to all other NVEs vise versa). If such there is
   many-to-one relationship such that there are interested in participating in
   both intra-subnet and inter-subnet communications with that TS.

   In this scenario, for many host IP addresses
   correspond to a given tenant, an NVE has typically one MAC-
   VRF for each tenant's subnet (VLAN) that is configured for, assuming
   VLAN-based service which is typically the case for VxLAN and NVGRE
   encapsulation and each MAC-VRF consists of single host MAC address or there are many host MAC
   addresses correspond to a single bridge domain. In
   case of MPLS encapsulation with VLAN-aware bundling, IP address, then each MAC-
   VRF consists of multiple bridge domains (one bridge domain per VLAN).
   The MAC-VRFs on to detect host
   mobility, the procedures in [IRB-EXT-MOBILITY] must be exercised
   followed by the procedures described below.

4.1.1 Initiating an ARP Request upon a Move

   In this scenario when a TS moves from a source NVE for to a given tenant are associated with target NVE,
   the TS initiates an IP-
   VRF corresponding ARP request upon the move (e.g., gratuitous ARP)
   to that tenant (or IP-VPN instance) via their IRB
   interfaces.

   Each the target NVE.

   The target NVE MUST support QoS, Security, upon receiving this ARP request, updates its MAC-VRF,
   IP-VRF, and OAM policies per IP-VRF
   to/from the core network. This is not to be confused ARP table with the QoS,
   Security, host MAC, IP, and OAM policies per Attachment Circuits (AC) to/from the
   Tenant Systems. How local adjacency
   information (e.g., local interface).

   Since this requirement is met is an implementation
   choice and it is outside NVE has previously learned the scope of this document.

   Since VxLAN same MAC and NVGRE encapsulations require inner Ethernet header
   (inner IP addresses
   from the source NVE, it recognizes that there has been a MAC SA/DA), move and since for inter-subnet traffic, TS
   it initiates MAC address
   cannot be used, the ingress NVE's MAC address is used as inner MAC
   SA. The NVE's MAC address is mobility procedures per [RFC7432] by advertising an
   EVPN MAC/IP route with both the device MAC address and it is common
   across all MAC-VRFs and IP-VRFs. This MAC address is advertised using
   the new EVPN Router's IP addresses filled in along
   with MAC Mobility Extended Community (section 6.1).

   Figure 6 below illustrates with the sequence number
   incremented by one.

   The source NVE upon receiving this scenario where a given tenant (e.g.,
   an IP-VPN instance) MAC/IP advertisement, realizes
   that the MAC has three subnets represented by MAC-VRF1, MAC-
   VRF2, moved to the target NVE. It updates its MAC-VRF and MAC-VRF3 across two NVEs. There are five TS's that are
   associated
   IP-VRF table accordingly with these three MAC-VRFs - i.e., TS1, TS4, and TS5 are
   sitting on the same subnet (e.g., same MAC-VRF/VLAN);where, TS1 adjacency information of the target
   NVE and
   TS5 are associated with MAC-VRF1 on NVE1, TS4 is associated with MAC-
   VRF1 on NVE2.  TS2 withdraws its EVPN MAC/IP route. Furthermore, it sends an ARP
   probe locally to ensure that the MAC is associated with MAC-VRF2 on NVE1, gone and TS3 it deletes its ARP
   entry corresponding to that <IP, MAC> when there is
   associated no ARP response.

   All other remote NVE devices upon receiving the MAC/IP advertisement
   route with MAC-VRF3 on NVE2. MAC-VRF1 and MAC-VRF2 on NVE1 are MAC Mobility extended community compare the sequence
   number in turn associated with IP-VRF1 on NVE1 and MAC-VRF1 and MAC-VRF3 on
   NVE2 are associated with IP-VRF1 on NVE2. When TS1, TS5, and TS4
   exchange traffic this advertisement with each other, only L2 forwarding (bridging) part
   of the IRB solution is exercised because all these TS's sit on one previously received. If the
   same subnet. However, when TS1 wants to exchange traffic with TS2 or
   TS3 which belong to different subnets,
   new sequence number is greater than the old one, then both bridging and routing
   parts of they update the IRB solution are exercised. The following subsections
   describe
   MAC/IP addresses of the control TS in their corresponding MAC-VRF and data planes operations IP-VRF
   tables to point to the target NVE. Furthermore, upon receiving the
   MAC/IP withdraw for the TS from the source NVE, these remote PEs
   perform the cleanups for their BGP tables.

4.1.2 Sending Data Traffic without an ARP Request

   In this IRB scenario
   in details.

                     NVE1         +---------+
               +-------------+    |         |
       TS1-----|         MACx|    |         |        NVE2
     (IP1/M1)  |(MAC-        |    |         |   +-------------+
       TS5-----| VRF1)\      |    |  MPLS/  |   |MACy  (MAC-  |-----TS3
     (IP5/M5)  |       \     |    |  VxLAN/ |   |     / VRF3) |  (IP3/M3)
               |    (IP-VRF1)|----|  NVGRE  |---|(IP-VRF1)    |
               |       /     |    |         |   |     \       |
       TS2-----|(MAC- /      |    |         |   |      (MAC-  |-----TS4
     (IP2/M2)  | VRF2)       |    |         |   |       VRF1) |   (IP4/M4)
               +-------------+    |         |   +-------------+
                                  |         |
                                  +---------+

          Figure 6: IRB forwarding on NVEs for Tenant Systems

5.1.1 Control Plane Operation

   Each when a TS moves from a source NVE advertises to a MAC/IP Advertisement route (i.e., Route Type 2)
   for each target NVE,
   the TS starts sending data traffic without first initiating an ARP
   request.

   The target NVE upon receiving the first data packet, it learns the
   MAC address of the TS in data plane and updates its TS's MAC-VRF table
   with the following field set:

   - RD MAC address and ESI per [EVPN]
   - Ethernet Tag = 0; assuming VLAN-based service
   - the local adjacency information (e.g., local
   interface) accordingly. The target NVE realizes that there has been a
   MAC Address Length = 48
   - move because the same MAC Address = Mi ; where i = 1,2,3,4, or 5 address has been learned remotely from
   the source NVE.

   If EVPN-IRB NVEs are configured to advertise MAC-only routes in
   addition to MAC-and-IP EVPN routes, then the above example
   - IP Address Length = 32 or 128 following steps are
   taken:

   - IP Address = IPi ; where i = 1,2,3,4, or 5 The target NVE upon learning this MAC address in data-plane,
   updates this MAC address entry in the above example
   - Label-1 = MPLS Label or VNID corresponding to MAC-VRF
   - Label-2 = MPLS Label or VNID corresponding to IP-VRF

   Each NVE advertises with the
   local adjacency information (e.g., local interface). It also
   recognizes that this MAC has moved and initiates MAC mobility
   procedures per [RFC7432] by advertising an RT-2 EVPN MAC/IP route with two Route Targets (one
   corresponding to its MAC-VRF and the other corresponding to its IP-
   VRF. Furthermore,
   only the RT-2 is advertised MAC address filled in along with two BGP Extended
   Communities. The first BGP MAC Mobility Extended
   Community identifies with the tunnel
   type per section 4.5 of [TUNNEL-ENCAP] and sequence number incremented by one.

   - The source NVE upon receiving this MAC/IP advertisement, realizes
   that the second BGP Extended
   Community includes MAC has moved to the new NVE. It updates its MAC-VRF table
   accordingly by updating the adjacency information for that MAC
   address of to point to the target NVE (e.g., MACx for NVE1 or
   MACy for NVE2) as defined in section 6.1. This second Extended
   Community (for and withdraws its EVPN MAC/IP
   route that has only the MAC address of NVE) is only required when Ethernet
   NVO tunnel type is used. If IP NVO tunnel type is used, then there (if it has advertised such route
   previously). Furthermore, it searches its ARP table for this MAC and
   sends an ARP probe for this <MAC,IP> pair. The ARP request message is
   no need
   sent both locally to send this second Extended Community. It should be noted all attached TS's in that IP NVO tunnel type subnet as well as it
   is only applicable sent to symmetric IRB
   procedures.

   Upon receiving this advertisement, other NVEs participating in that subnet including the receiving
   target NVE.

   - The target NVE performs passes the
   following:

   - It uses Route Targets corresponding ARP request to its MAC-VRF and IP-VRF for
   identifying these tables locally attached TS's
   and subsequently importing this route into
   them.

   - It imports when it receives the MAC address from ARP response, it sends an EVPN MAC/IP Advertisement
   advertisement route into the
   MAC-VRF with BGP Next Hop address as underlay tunnel destination
   address (e.g., VTEP DA for VxLAN encapsulation) and Label-1 as VNID
   for VxLAN encapsulation or EVPN label for MPLS encapsulation.

   - If the route carries both the new Router's MAC Extended Community, and
   if the receiving NVE is using Ethernet NVO tunnel, then the receiving
   NVE imports the IP address into IP-VRF addresses filled in
   along with NVE's MAC address (from
   the new Router's MAC Mobility Extended Community) as inner MAC DA and BGP Next
   Hop address as underlay tunnel destination address, VTEP DA for VxLAN
   encapsulation and Label-2 Community with the sequence number
   set to the same value as IP-VPN VNID the one for VxLAN encapsulation. MAC-only advertisement route it
   sent previously.

   - If When the receiving source NVE is going to use MPLS encapsulation, then receives the
   receiving NVE imports the IP address into EVPN MAC/IP advertisement, it
   updates its IP-VRF table with BGP Next Hop
   address as underlay tunnel destination address, and Label-2 as IP-VPN
   label for MPLS encapsulation.

   If the receiving NVE receives a RT-2 with only a single Route Target
   corresponding new adjacency information
   (pointing to IP-VRF the target NVE) and Label-1, or if deletes the associated ARP entry
   from its ARP table. Furthermore, it receives a RT-2 with
   only a single Route Target corresponding to MAC-VRF but withdraws its previously
   advertised EVPN MAC/IP route with both
   Label-1 and Label-2, or if it receives a RT-2 with the MAC Address Length
   of zero, then it must not import it to either IP-VRF or MAC-VRF and
   it must log an error.

5.1.2 Data Plane Operation IP addresses.

   - Inter Subnet

   The following description of the data-plane operation describes just
   the logical functions and All other remote NVE devices upon receiving the actual implementation may differ. Lets
   consider data-plane operation when TS1 in subnet-1 (MAC-VRF1) on NVE1
   wants to send traffic to TS3 in subnet-3 (MAC-VRF3) on NVE2.

   - NVE1 receives a packet MAC/IP
   advertisement route with MAC DA corresponding to Mobility extended community compare the MAC-VRF1
   IRB interface on NVE1 (the interface between MAC-VRF1 and IP-VRF1),
   and VLAN-tag corresponding to MAC-VRF1.

   - Upon receiving
   sequence number in this advertisement with the packet, one previously
   received. If the NVE1 uses VLAN-tag to identify new sequence number is greater than the
   MAC-VRF1. It old one,
   then looks up the MAC DA and forwards they update the frame to its
   IRB interface.

   -  The Ethernet header MAC/IP addresses of the packet is stripped TS in their
   corresponding MAC-VRF and the packet is
   fed IP-VRF tables to point to the IP-VRF where IP lookup is performed on new NVE.
   Furthermore, upon receiving the destination
   address. This lookup yields an outgoing interface and MAC/IP withdraw for the required
   encapsulation. If TS from the encapsulation is
   old NVE, these remote PEs perform the cleanups for Ethernet NVO tunnel, their BGP tables.

   If EVPN-IRB NVEs are configured not to advertise MAC-only routes,
   then upon receiving the first data packet, it includes a MAC address to be used as inner learns the MAC DA, an IP address
   to be used as VTEP DA,
   of the TS and updates the MAC entry in the corresponding MAC-VRF
   table with the local adjacency information (e.g., local interface).
   It also realizes that there has been a VPN-ID to be used as VNID. The inner MAC
   SA and VTEP SA is set to NVE's move because the same MAC and IP addresses respectively. If
   it is a MPLS encapsulation, then corresponding EVPN and LSP labels
   are added to
   address has been learned remotely from the packet. The packet is source NVE. It then forwarded sends
   an unicast ARP request to the egress
   NVE.

   - On host and when receiving an ARP
   response, it follows the egress procedure outlined in section 4.1.1.

4.1.3 Silent Host

   In this scenario when a TS moves from a source NVE to a target NVE, if
   the packet arrives on Ethernet NVO tunnel
   (e.g., it TS is VxLAN encapsulated), then silent and it neither initiates an ARP request nor it sends
   any data traffic. Therefore, neither the VxLAN header is removed.
   Since target nor the source NVEs
   are aware of the inner MAC DA is move.

   On the source NVE, the egress NVE's MAC address, age-out timer expires and as the egress
   NVE knows that result it needs to perform
   triggers an IP lookup. It uses VNID to
   identify the IP-VRF table. If ARP probe on the packet is MPLS encapsulated, then source NVE. The ARP request gets sent
   both locally to all the EVPN label lookup identifies attached TS's on that subnet as well as it
   gets sent to all the IP-VRF table. Next, an IP lookup
   is performed for remote NVEs (including the destination TS (TS3) which results target NVE)
   participating in access-
   facing IRB interface over which that subnet. It also withdraw the packet is sent. Before sending EVPN MAC/IP route
   with only the packet over this interface, MAC address (if it has previously advertised such a
   route).

   The target NVE passes the ARP table is consulted request to get the
   destination its locally attached TS's
   and when it receives the ARP response, it sends an EVPN MAC/IP
   advertisement route with both the MAC address.

   - The and IP packet is encapsulated with an Ethernet header addresses filled in
   along with MAC SA
   set Mobility Extended Community with the sequence number
   incremented by one.

   When the source NVE receives the EVPN MAC/IP advertisement, it
   updates its IP-VRF table with the new adjacency information
   (pointing to that of IRB interface MAC address the target NVE) and deletes the associated ARP entry
   from its ARP table. Furthermore, it withdraws its previously
   advertised EVPN MAC/IP route with both the MAC DA set to that of
   destination TS (TS3) and IP addresses.

   All other remote NVE devices upon receiving the MAC/IP advertisement
   route with MAC address. The packet Mobility extended community compare the sequence
   number in this advertisement with the one previously received. If the
   new sequence number is sent to greater than the
   corresponding MAC-VRF3 and after a lookup old one, then they update the
   MAC/IP addresses of MAC DA, is forwarded to the destination TS (TS3) over the in their corresponding interface.

   In this symmetric IRB scenario, inter-subnet traffic between NVEs
   will always use the MAC-VRF and IP-VRF VNID/MPLS label. For instance, traffic
   from TS2
   tables to TS4 will be encapsulated by NVE1 using NVE2's IP-VRF
   VNID/MPLS label, as long as TS4's host IP is present in NVE1's IP-
   VRF.

5.1.3 TS Move Operation

   When a TS move from one NVE point to other, it is important that the MAC
   mobility procedures are properly executed and the corresponding MAC-
   VRF and IP-VRF tables on all participating NVEs are updated. [EVPN]
   describes new NVE. Furthermore, upon receiving the MAC mobility procedures for L2-only services
   MAC/IP withdraw for both
   single-homed the TS and multi-homed TS. This section describes from the
   incremental procedures and old NVE, these remote PEs perform
   the cleanups for their BGP tables.

5 BGP Encoding

   This document defines one new BGP Extended Communities needed to handle
   the MAC mobility Community for EVPN.

5.1 Router's MAC Extended Community

   A new EVPN BGP Extended Community called Router's MAC is introduced
   here. This new extended community is a mixed transitive extended community
   with the Type field of L2 0x06 (EVPN) and L3 connectivity (aka IRB). In
   order to place the emphasis on the differences between L2-only versus
   L2-and-L3 use cases, the incremental procedure is described for
   single-homed TS Sub-Type of 0x03. It may
   be advertised along with the expectation that the reader can easily
   extrapolate multi-homed TS based on the procedures described BGP Encapsulation Extended Community define
   in section 15 of [EVPN].

   Lets consider TS1 in figure-6 above where it moves from NVE1 to NVE2.
   In such move, NVE2 discovers IP1/MAC1 4.5 of TS1 and realizes that it is
   a [TUNNEL-ENCAP].

   The Router's MAC move and it advertises a MAC/IP route per section 5.1.1 above
   with MAC Mobility Extended Community.

   Since NVE2 learns TS1's MAC/IP addresses locally, it updates its MAC-
   VRF1 and IP-VRF1 for TS1 with its local interface.

   If the local learning at NVE1 Community is performed using control or
   management planes, then these interactions serve encoded as the trigger for
   NVE1 to withdraw the an 8-octet value as
   follows:

        0                   1                   2                   3
        0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       | Type=0x06     | Sub-Type=0x03 |        Router's MAC and IP addresses associated with TS1.
   However, if the local learning at NVE1 is performed using data-plane
   learning, then the reception of the MAC/IP Advertisement route (for
   TS1) from NVE2 with           |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |                      Router's MAC Mobility Cont'd                      |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

               Figure 5: Router's MAC Extended Community

   This extended community serve as the
   trigger for NVE1 is used to withdraw carry the PE's MAC address for
   symmetric IRB scenarios and IP addresses associated it is sent with
   TS1.

   All other remote NVE devices upon receiving the MAC/IP advertisement
   route RT-2.

6 Operational Models for TS1 from NVE2 with MAC Mobility extended community compare
   the sequence number in this advertisement Symmetric Inter-Subnet Forwarding

   The following sections describe two main symmetric IRB forwarding
   scenarios (within a DC - i.e., intra-DC) along with the one previously
   received. If the new sequence number is greater than the old one,
   then they update the MAC/IP addresses of TS1 in their
   corresponding
   MAC-VRFs and IP-VRFs to point to NVE2. Furthermore, upon receiving
   the MAC/IP withdraw for TS1 from NVE1, these remote PEs perform procedures. In the
   cleanups for their BGP tables.

5.2 following scenarios, without loss of
   generality, it is assumed that a given tenant is represented by a
   single IP-VPN instance. Therefore, on a given PE, a tenant is
   represented by a single IP-VRF table and one or more MAC-VRF tables.

6.1 IRB forwarding on NVEs for Subnets behind Tenant Systems

   This section covers the symmetric IRB procedures for the scenario
   where some each Tenant Systems (TS's) support System (TS) is attached to one or more subnets NVEs and
   these TS's are associated with one or more NVEs. Therefore, besides
   the advertisement of MAC/IP its
   host IP and MAC addresses for each TS which can be multi-
   homed with All-Active redundancy mode, the associated NVE needs to
   also advertise are learned by the subnets statically configured on each TS.

   The main difference between this solution attached NVEs and the previous one is the
   additional advertisement corresponding are
   distributed to each subnet. These subnet
   advertisements all other NVEs that are accomplished using EVPN IP Prefix route defined interested in
   [EVPN-PREFIX]. These subnet prefixes are advertised participating in
   both intra-subnet and inter-subnet communications with the IP
   address that TS.

   In this scenario, without loss of their associated TS (which generality, it is in overlay address space) as
   their next hop. The receiving assumed that NVEs perform recursive route resolution
   to resolve the subnet prefix
   operate in VLAN-based service interface mode with its associated ingress one Bridge Table
   (BT) per MAC-VRF. Thus for a given tenant, an NVE so has one MAC-VRF for
   each tenant's subnet (e.g., each VLAN) that
   they know is configured for which NVE to forward
   is typically the packets to when they are destined case for that subnet prefix.

   The advantage VxLAN and GENEVE encapsulation. In case of this recursive route resolution is that when a TS
   moves from one
   VLAN-aware bundling, then each MAC-VRF consists of multiple Bridge
   Tables (e.g., one BT per VLAN). The MAC-VRFs on an NVE for a given
   tenant are associated with an IP-VRF corresponding to another, there that tenant (or
   IP-VPN instance) via their IRB interfaces.

   Each NVE MUST support QoS, Security, and OAM policies per IP-VRF
   to/from the core network. This is no need not to re-advertise any
   of be confused with the subnet prefixes for that TS. All it QoS,
   Security, and OAM policies per Attachment Circuits (AC) to/from the
   Tenant Systems. How this requirement is needed met is to advertise
   the IP/MAC addresses associated with an implementation
   choice and it is outside the TS itself scope of this document.

   Since VxLAN and exercise GENEVE encapsulations require inner Ethernet header
   (inner MAC
   mobility procedures SA/DA), and since for that TS. inter-subnet traffic, TS MAC address
   cannot be used, the ingress NVE's MAC address is used as inner MAC
   SA. The recursive route resolution
   automatically takes care of NVE's MAC address is the updates for device MAC address and it is common
   across all MAC-VRFs and IP-VRFs. This MAC address is advertised using
   the subnet prefixes of
   that TS. new EVPN Router's MAC Extended Community (section 5.1).

   Figure 6 below illustrates this scenario where a given tenant (e.g.,
   an IP-VPN service) instance) has three subnets represented by MAC-VRF1, MAC-VRF2, MAC-
   VRF2, and MAC-VRF3 across two NVEs. There are four five TS's that are
   associated with these three MAC-VRFs - i.e., TS1, TS4, and TS5 are connected to
   sitting on the same subnet (e.g., same MAC-VRF/VLAN);where, TS1 and
   TS5 are associated with MAC-VRF1 on NVE1, TS4 is associated with MAC-
   VRF1 on NVE2.  TS2 is connected to associated with MAC-VRF2 on NVE1,  TS3 is connected to MAC-
   VRF3 on NVE2, and TS4 TS3 is connected to MAC-VRF1
   associated with MAC-VRF3 on NVE2. TS1 has two
   subnet prefixes (SN1 and SN2) MAC-VRF1 and TS3 has a single subnet prefix,
   SN3. The MAC-VRFs MAC-VRF2 on each NVE NVE1 are
   in turn associated with their corresponding
   IP-VRF using their IRB interfaces. IP-VRF1 on NVE1 and MAC-VRF1 and MAC-VRF3 on
   NVE2 are associated with IP-VRF1 on NVE2. When TS4 TS1, TS5, and TS1 TS4
   exchange intra-
   subnet traffic, traffic with each other, only L2 forwarding (bridging) part
   of the IRB solution is used (i.e., exercised because all these TS's sit on the traffic only goes through their MAC-
   VRFs); however,
   same subnet. However, when TS3 TS1 wants to forward exchange traffic to SN1 with TS2 or SN2
   sitting behind TS1 (inter-subnet traffic),
   TS3 which belong to different subnets, then both bridging and routing
   parts of the IRB solution are exercised (i.e., the traffic
   goes through the corresponding MAC-VRFs and IP-VRFs). exercised. The following subsections
   describe the control and data planes operations for this IRB scenario
   in details.

                     NVE1      +----------+
     SN1--+         +---------+
               +-------------+    |         |
          |--TS1-----|(MAC- \      |   |          |
     SN2--+ IP1/M1   | VRF1) \     |   |          |
                     |     (IP-VRF)|---|
       TS1-----|         MACx|    |         |       /        NVE2
     (IP1/M1)  |(MAC-        |    |         |
             TS2-----|(MAC- /   +-------------+
       TS5-----| VRF1)\      |    |  MPLS/  |
            IP2/M2   |MACy  (MAC-  |-----TS3
     (IP5/M5)  | VRF2)       \     |    |  VxLAN/ |
                     +-------------+   |  NVGRE   |
                     +-------------+   |     / VRF3) |
     SN3--+--TS3-----|(MAC-\  (IP3/M3)
               |    (IP-VRF1)|----|  NVGRE  |---|(IP-VRF1)    |
               |
            IP3/M3       /     | VRF3)\    |         |   |     \       |     (IP-VRF)|---|
       TS2-----|(MAC- /      |    |       /         |   |      (MAC-  |-----TS4
     (IP2/M2)  |
             TS4-----|(MAC- / VRF2)       |    |         |
            IP4/M4   |       VRF1) |   (IP4/M4)
               +-------------+    |         |   +-------------+   +----------+
                            NVE2
                                  |         |
                                  +---------+

          Figure 7: 6: IRB forwarding on NVEs for Tenant Systems with configured subnets

5.2.1

6.1.1 Control Plane Operation

   Each NVE advertises a MAC/IP Advertisement route (i.e., Route Type-5 (RT-5, IP Prefix Route defined in
   [EVPN-PREFIX]) Type 2)
   for each of its subnet prefixes TS's with the IP address of
   its TS as the next hop (gateway address field) as follow: following field set:

   - RD associated with the IP-VRF
   - and ESI = 0 per [RFC7432]
   - Ethernet Tag = 0; assuming VLAN-based service
   - MAC Address Length = 48
   - MAC Address = Mi ; where i = 1,2,3,4, or 5 in the above example
   - IP Prefix Address Length = 32 or 128
   - IP Prefix = SNi
   - Gateway Address = IPi; IP address of TS
   - Label IPi ; where i = 0

   This RT-5 is advertised with one 1,2,3,4, or more Route Targets that have been
   configured as "export route targets" of 5 in the above example
   - Label-1 = MPLS Label or VNI corresponding to MAC-VRF
   - Label-2 = MPLS Label or VNI corresponding to IP-VRF from which the
   route is originated.

   Each NVE also advertises an RT-2 (MAC/IP Advertisement Route) along route with their associated two Route Targets (one
   corresponding to its MAC-VRF and the other corresponding to its IP-
   VRF. Furthermore, the RT-2 is advertised with two BGP Extended Communities for each
   Communities. The first BGP Extended Community identifies the tunnel
   type per section 4.5 of its TS's exactly [TUNNEL-ENCAP] and the second BGP Extended
   Community includes the MAC address of the NVE (e.g., MACx for NVE1 or
   MACy for NVE2) as described defined in section 5.1.1. 5.1. This second Extended
   Community (for the MAC address of NVE) is only required when Ethernet
   NVO tunnel type is used. If IP NVO tunnel type is used, then there is
   no need to send this second Extended Community. It should be noted
   that IP NVO tunnel type is only applicable to symmetric IRB
   procedures.

   Upon receiving the RT-5 this advertisement, the receiving NVE performs the
   following:

   - It uses the Route Target to identify the Targets corresponding to its MAC-VRF and IP-VRF for
   identifying these tables and subsequently importing the MAC and IP
   addresses into them respectively.

   - It imports the IP prefix MAC address from MAC/IP Advertisement route into its corresponding IP-VRF that is
   configured with an import RT that is one of the RTs being carried by the RT-5 route along
   MAC-VRF with the IP BGP Next Hop address of the associated TS as its
   next hop.

   When receiving the RT-2 advertisement, underlay tunnel destination
   address (e.g., VTEP DA for VxLAN encapsulation) and Label-1 as VNI
   for VxLAN encapsulation or EVPN label for MPLS encapsulation.

   - If the route carries the new Router's MAC Extended Community, and
   if the receiving NVE is using Ethernet NVO tunnel, then the receiving
   NVE imports
   MAC/IP addresses of the TS IP address into the corresponding MAC-VRF and IP-VRF
   per section 5.1.1. When both routes exist, recursive route resolution
   is performed to resolve the IP prefix (received in RT-5) to its
   corresponding with NVE's IP MAC address (e.g., its BGP next hop). (from
   the new Router's MAC Extended Community) as inner MAC DA and BGP next hop
   will be used Next
   Hop address as underlay tunnel destination address (e.g., address, VTEP DA for VxLAN encapsulation)
   encapsulation and Router's MAC will be used Label-2 as inner MAC IP-VPN VNI for VxLAN encapsulation.

5.2.2

   - If the receiving NVE is going to use MPLS encapsulation, then the
   receiving NVE imports the IP address into IP-VRF with BGP Next Hop
   address as underlay tunnel destination address, and Label-2 as IP-VPN
   label for MPLS encapsulation.

   If the receiving NVE receives a RT-2 with only Label-1 and only a
   single Route Target corresponding to IP-VRF, or if it receives a RT-2
   with only a single Route Target corresponding to MAC-VRF but with
   both Label-1 and Label-2, or if it receives a RT-2 with MAC Address
   Length of zero, then it MUST treat the route as withdraw [RFC7606]
   and log an error message.

6.1.2 Data Plane Operation

   The following description of the data-plane operation describes just
   the logical functions and the actual implementation may differ. Lets
   consider data-plane operation when a host on SN1 sitting behind TS1 in subnet-1 (MAC-VRF1) on NVE1
   wants to send traffic to a host sitting behind SN3 behind TS3. TS3 in subnet-3 (MAC-VRF3) on NVE2.

   - TS1 send NVE1 receives a packet with MAC DA corresponding to the MAC-VRF1
   IRB interface of NVE1, and VLAN-tag corresponding on NVE1 (the interface between MAC-VRF1 and IP-VRF1),
   and VLAN-tag corresponding to MAC-VRF1.

   - Upon receiving the packet, the ingress NVE1 uses VLAN-tag to identify the
   MAC-VRF1. It then looks up the MAC DA and forwards the frame to its
   IRB interface just like section 5.1.1. interface.

   -  The Ethernet header of the packet is stripped and the packet is
   fed to the IP-VRF; where, IP-VRF where IP lookup is performed on the destination IP
   address. This lookup yields the fields needed for VxLAN outgoing NVO tunnel and the required
   encapsulation. If the encapsulation
   with NVE2's is for Ethernet NVO tunnel, then
   it includes the egress NVE's MAC address as the inner MAC DA, NVE'2 the egress
   NVE's IP address (e.g., BGP Next Hop address) as the VTEP DA, and the VNID.
   VPN-ID as the VNI. The inner MAC SA is set to NVE1's MAC address and VTEP SA is are set to NVE1's NVE's MAC
   and IP address.

   -  The packet addresses respectively. If it is a MPLS encapsulation, then encapsulated with the proper header based on
   the above info
   corresponding EVPN and LSP labels are added to the packet. The packet
   is then forwarded to the egress NVE (NVE2). NVE.

   - On the egress NVE (NVE2), assuming NVE, if the packet arrives on Ethernet NVO tunnel
   (e.g., it is VxLAN
   encapsulated, encapsulated), then the VxLAN and NVO tunnel header is
   removed. Since the inner Ethernet headers are removed
   and MAC DA is the resultant egress NVE's MAC address, the
   egress NVE knows that it needs to perform an IP packet is fed lookup. It uses the
   VNI to identify the IP-VRF associated with that table. If the VNID.

   - Next, a lookup is performed based on IP DA (which packet is in SN3) in MPLS encapsulated,
   then the EVPN label lookup identifies the
   associated IP-VRF of NVE2. The table. Next, an IP
   lookup yields is performed for the destination TS (TS3) which results in
   access-facing IRB interface over which the packet needs to be is sent. Before
   sending the packet over this interface, the ARP table is consulted to
   get the destination TS (TS3) TS's MAC address.

   - The IP packet is encapsulated with an Ethernet header with the MAC SA
   set to that of the access-facing IRB interface of the egress NVE
   (NVE2) MAC address (i.e, IRB interface between
   MAC-VRF3 and IP-VRF1 on NVE2) and the MAC DA is set to that of destination
   TS (TS3) MAC address. The packet is sent to the corresponding MAC-VRF3 MAC-VRF
   (i.e., MAC-VRF3) and after a lookup of MAC DA, is forwarded to the
   destination TS (TS3) over the corresponding interface.

6  Inter-Subnet DCI Scenarios

   The

   In this symmetric IRB scenario, inter-subnet DCI scenarios can be categorized into the following
   four categories. The last two scenarios, along with its corresponding
   solution, are described in [EVPN-IPVPN-INTEROP]. The first two
   scenarios are covered in this document.

   1. Switching among IP subnets in different DCs traffic between NVEs
   will always use the IP-VRF VNI/MPLS label. For instance, traffic from
   TS2 to TS4 will be encapsulated by NVE1 using EVPN without GW

   2. Switching among NVE2's IP-VRF VNI/MPLS
   label, as long as TS4's host IP subnets is present in different DCs using EVPN with GW

   3. Switching among IP NVE1's IP-VRF.

6.2 IRB forwarding on NVEs for Subnets behind Tenant Systems

   This section covers the symmetric IRB procedures for the scenario
   where some Tenant Systems (TS's) support one or more subnets spread across IP-VPN and EVPN networks
   these TS's are associated with GW

   4. Switching among IP one or more NVEs. Therefore, besides
   the advertisement of MAC/IP addresses for each TS which can be multi-
   homed with All-Active redundancy mode, the associated NVE needs to
   also advertise the subnets spread across IP-VPN statically configured on each TS.

   The main difference between this solution and EVPN networks
   without GW

   In the above scenario, previous one is the term "GW" refers
   additional advertisement corresponding to each subnet. These subnet
   advertisements are accomplished using EVPN IP Prefix route defined in
   [EVPN-PREFIX]. These subnet prefixes are advertised with the case where a node
   situated at the WAN edge IP
   address of the data center network behaves their associated TS (which is in overlay address space) as a
   default gateway (GW) for all
   their next hop. The receiving NVEs perform recursive route resolution
   to resolve the destinations subnet prefix with its associated ingress NVE so that are outside
   they know which NVE to forward the
   data center. packets to when they are destined
   for that subnet prefix.

   The absence advantage of GW refers to the scenario where NVEs
   within a data center maintain individual (host) routes that are
   outside of the data center.

   In the case (3), the WAN edge node also performs this recursive route aggregation
   for all the destinations within its own data center, and acts as an
   interworking unit between EVPN and IP VPN (it implements both EVPN
   and IP-VPN functionality).

                             +---+    Enterprise Site 1
                             |PE1|----- H1
                             +---+
                               /
                         ,---------.             Enterprise Site 2
                       ,'           `.    +---+
        ,---------.  /(    MPLS/IP    )---|PE2|-----  H2
       '   DCN 3   `./ `.   Core    ,'    +---+
        `-+------+'     `-+------+'
        __/__           / /      \ \
       :NVE4 :        +---+       \ \
       '-----'   ,----|GW |.       \ \
          |    ,'     +---+ `.      ,---------.
         TS6  (      DCN 1    )   ,'           `.
               `.           ,'   (      DCN 2    )
                 `-+------+'      `.           ,'
                   __/__            `-+------+'
                  :NVE1 :           __/__   __\__
                  '-----'          :NVE2 :  :NVE3 :
                   |  |            '-----'  '-----'
                  TS1 TS2            |  |      |
                                    TS3 TS4   TS5

                  Figure 8: Interoperability Use-Cases

   In what follows, we will describe scenarios 1 and 2 in more details.

6.1 Switching among IP subnets in different DCs without GW

   This case resolution is similar to that of section 2.1 above albeit for the fact
   that the TS's belong to different data centers that are
   interconnected over when a WAN (e.g. MPLS/IP PSN). The data centers in
   question here are seamlessly interconnected TS
   moves from one NVE to the WAN, i.e., the WAN
   edge devices do not maintain any TS-specific addresses in the
   forwarding path - e.g., another, there is no WAN edge GW(s) between these DCs.

   As an example, consider TS3 and TS6 need to re-advertise any
   of Figure 2 above. Assume the subnet prefixes for that
   connectivity TS. All it is needed is required between these two TS's where TS3 belongs to
   the SN3 whereas TS6 belongs to advertise
   the SN6. NVE2 has an EVI3 associated
   with SN3 and NVE4 has an EVI6 IP/MAC addresses associated with the SN6. Both SN3 TS itself and
   SN6 are part exercise MAC
   mobility procedures for that TS. The recursive route resolution
   automatically takes care of the same IP-VRF.

   When updates for the subnet prefixes of
   that TS.

   Figure below illustrates this scenario where a given tenant (e.g., an EVPN MAC advertisement route is received
   IP-VPN service) has three subnets represented by a NVE, the IP
   address MAC-VRF1, MAC-VRF2,
   and MAC-VRF3 across two NVEs. There are four TS's associated with the route
   these three MAC-VRFs - i.e., TS1, TS5 are connected to MAC-VRF1 on
   NVE1, TS2 is used connected to populate the IP-VRF
   table, whereas the MAC address associated with the route MAC-VRF2 on NVE1,  TS3 is used connected to
   populate both the MAC-VRF table, as well as the adjacency associated
   with the IP route in the IP-VRF table (i.e., ARP table).

   When an Ethernet frame MAC-
   VRF3 on NVE2, and TS4 is received by an ingress NVE, it performs connected to MAC-VRF1 on NVE2. TS1 has two
   subnet prefixes (SN1 and SN2) and TS3 has a
   lookup single subnet prefix,
   SN3. The MAC-VRFs on the destination MAC address in the associated EVI. If the
   MAC address corresponds to its IRB Interface MAC address, the ingress
   NVE deduces that the packet MUST be inter-subnet routed. Hence, the
   ingress each NVE performs an IP lookup in the are associated with their corresponding
   IP-VRF table. The
   lookup identifies an adjacency that contains a MAC rewrite using their IRB interfaces. When TS4 and in
   turn the next-hop (i.e. egress) Gateway to which the packet must be
   forwarded along with the associated MPLS label stack. The MAC rewrite
   holds the MAC address associated with the destination host (as
   populated by the EVPN MAC route), instead TS1 exchange intra-
   subnet traffic, only L2 forwarding (bridging) part of the MAC address of IRB
   solution is used (i.e., the
   next-hop Gateway. The ingress NVE traffic only goes through their MAC-
   VRFs); however, when TS3 wants to forward traffic to SN1 or SN2
   sitting behind TS1 (inter-subnet traffic), then rewrites both bridging and
   routing parts of the destination MAC
   address in IRB solution are exercised (i.e., the packet with traffic
   goes through the address specified in the adjacency. It
   also rewrites the source MAC address with its IRB Interface MAC
   address for the destination subnet. The ingress NVE, then, forwards
   the frame to the next-hop (i.e. egress) Gateway after encapsulating
   it with the MPLS label stack.

   Note that this label stack includes the LSP label as well as an EVPN
   label. corresponding MAC-VRFs and IP-VRFs). The EVPN label could be either advertised by the ingress
   Gateway, if inter-AS option B is used, or advertised by the egress
   NVE, if inter-AS option C is used. When the MPLS encapsulated packet
   is received by the ingress Gateway, the processing again differs
   depending on whether inter-AS option B or option C is employed: in
   the former case, the ingress Gateway swaps the EVPN label in the
   packets with the EVPN label value received from the egress Gateway.
   In the latter case, the ingress Gateway does not modify following
   subsections describe the EVPN
   label control and performs normal label switching on the LSP label.
   Similarly on the egress Gateway, for option B, the egress Gateway
   swaps the EVPN label with the value advertised by the egress NVE.
   Whereas, data planes operations for option C, the egress Gateway does not modify the EVPN
   label, and performs normal label switching on the LSP label. When the
   MPLS encapsulated packet is received by the egress NVE, it uses the
   EVPN label to identify the bridge-domain table. It then performs a
   MAC lookup this
   IRB scenario in that table, which yields the outbound interface to
   which the Ethernet frame must be forwarded. Figure 3 below depicts
   the packet flow. details.

                             NVE1            ASBR1          ASBR2           NVE2
      +------------+  +------------+  +------------+  +------------+      +----------+
     SN1--+          +-------------+   |          |
          |--TS1-----|(MAC- \      |   |          |
     SN2--+ IP1/M1   | VRF1) \     |   |
      |(MAC - (IP          |
                     |    [LS]     (IP-VRF)|---|          |
                     |    [LS]       /     |  |(IP  - (MAC   |          | VRF)   VRF)|
             TS2-----|(MAC- /      |   |  MPLS/   |
            IP2/M2   | VRF2)       | VRF)   VRF)|   |  VxLAN/  |
                     +-------------+   |  NVGRE   |
                     +-------------+   |          |
     SN3--+--TS3-----|(MAC-\       |   |          |
            IP3/M3   | VRF3)\      |   |          |
                     |     (IP-VRF)|---|          |
                     |
      +------------+  +------------+  +------------+  +------------+
         ^     v           ^  V            ^  V               ^  V       /     |   |          |
             TS4-----|(MAC- /      |   |          |
            IP4/M4   | VRF1)       |
   TS1->-+     +-->--------+  +------------+  +---------------+  +->-TS2

  Figure 9: Inter-Subnet Forwarding Among EVPN NVEs in Different DCs
   without GW

6.2 Switching among IP subnets in different DCs with GW

   In this scenario, connectivity is required between TS's in different
   data centers, and those hosts belong to different IP subnets. What
   makes this case different from that of Section 2.2 is that at least
   one of the data centers has a gateway as the WAN edge switch. Because
   of that, the NVE's IP-VRF  within that data center need not maintain
   (host) routes to individual TS's outside of that data center.

   As an example, consider a tenant with TS1 and TS5 of Figure 2 above.
   Assume that connectivity is required between these two TS's where TS1
   belongs to the SN1 whereas TS5 belongs to the SN5. NVE3 has an EVI5
   associated with the SN5 and this EVI is represented by the MAC-VRF
   which is connected to the IP-VRF via an IRB interface. NVE1 has an
   EVI1 associated with the SN1 and this EVI is represented by the MAC-
   VRF which is connected to the IP-VRF representing the same tenant.
   Due to the gateway at the edge of DCN 1, NVE1's IP-VRF does not need
   to have the address of TS5 but instead it has a default route in its
   IP-VRF with the next-hop being the GW.

   In this scenario, the NVEs within a given data center do not have
   entries for the MAC/IP addresses of hosts in remote data centers.
   Rather, the NVEs have a default IP route pointing to the WAN gateway
   for each VRF. This is accomplished by the WAN gateway advertising for
   a given EVPN that spans multiple DC a default VPN-IP route that is
   imported by the NVEs of that VPN that are in the gateway's own DC.

   When an Ethernet frame is received by an ingress NVE, it performs a
   lookup on the destination MAC address in the associated MAC-VRF
   table. If the MAC address corresponds to the IRB Interface MAC
   address, the ingress NVE deduces that the packet MUST be inter-subnet
   routed. Hence, the ingress NVE performs an IP lookup in the
   associated IP-VRF table. The lookup, in this case, matches the
   default host route which points to the local WAN gateway (GW1). The
   ingress   |          |
                     +-------------+   +----------+
                            NVE2

        Figure 7: IRB forwarding on NVEs for subnets behind TS's

6.2.1 Control Plane Operation

   Each NVE (NVE1) then rewrites the destination MAC address advertises a Route Type-5 (RT-5, IP Prefix Route defined in the
   packet with the MAC address of core-facing IRB interface
   [EVPN-PREFIX]) for each of GW1 (not
   shown in the figure) or it can rewrite it its subnet prefixes with the router's MAC IP address of GW1. It also rewrites
   its TS as the source MAC next hop (gateway address field) as follow:

   - RD associated with its own
   core-facing IRB Interface's MAC address for the destination subnet
   (i.e., the subnet between NVE1 and GW1) IP-VRF
   - ESI = 0
   - Ethernet Tag = 0;
   - IP Prefix Length = 32 or it can rewrite it with its
   own router's MAC 128
   - IP Prefix = SNi
   - Gateway Address = IPi; IP address of NVE1. The ingress NVE, then, forwards the
   frame to GW1  after encapsulating it TS
   - Label = 0

   This RT-5 is advertised with the MPLS label stack. Note one or more Route Targets that this label stack includes the LSP label as well have been
   configured as the label for
   default host "export route that was advertised by targets" of the local WAN gateway. When IP-VRF from which the MPLS encapsulated packet
   route is received by GW1, it uses originated.

   Each NVE also advertises an RT-2 (MAC/IP Advertisement Route) along
   with their associated Route Targets and Extended Communities for each
   of its TS's exactly as described in section 6.1.1.

   Upon receiving the default
   host route MPLS label to identify RT-5 advertisement, the core-facing MAC-VRF. receiving NVE performs the
   following:

   - It does a
   MAC-DA lookup and forwards uses the packet Route Target to identify the corresponding IP-VRF after stripping
   the Ethernet header.

   - It then performs an imports the IP lookup in prefix into its corresponding IP-VRF that table. The
   lookup identifies is
   configured with an adjacency import RT that contains a MAC rewrite and in
   turn is one of the remote WAN gateway (GW2) to which RTs being carried by
   the packet must be
   forwarded RT-5 route along with the associated MPLS label stack. The MAC rewrite
   holds the MAC IP address of the associated with TS as its
   next hop.

   When receiving the ultimate destination host
   (as populated by RT-2 advertisement, the EVPN MAC route). GW1 then rewrites receiving NVE imports
   MAC/IP addresses of the
   destination MAC address in TS into the packet with corresponding MAC-VRF and IP-VRF
   per section 6.1.1. When both routes exist, recursive route resolution
   is performed to resolve the address specified IP prefix (received in
   the adjacency. It also rewrites the source MAC RT-5) to its
   corresponding NVE's IP address with the MAC (e.g., its BGP next hop). BGP next hop
   will be used as underlay tunnel destination address (e.g., VTEP DA
   for VxLAN encapsulation) and Router's MAC will be used as inner MAC
   for VxLAN encapsulation.

6.2.2 Data Plane Operation

   The following description of its core-facing IRB interface (not shown in the figure) or
   its router's MAC address. GW1, then, forwards data-plane operation describes just
   the frame to logical functions and the GW2
   after encapsulating it actual implementation may differ. Lets
   consider data-plane operation when a host on SN1 sitting behind TS1
   wants to send traffic to a host sitting behind SN3 behind TS3.

   - TS1 send a packet with MAC DA corresponding to the MPLS label stack. Note that this
   label stack includes the LSP label as well as a EVPN label that was
   advertised by GW2. When MAC-VRF1 IRB
   interface of NVE1, and VLAN-tag corresponding to MAC-VRF1.

   - Upon receiving the MPLS encapsulated packet is received by
   GW2, it uses packet, the EVPN label ingress NVE1 uses VLAN-tag to
   identify the destination MAC-VRF. MAC-VRF1. It then performs a MAC-DA lookup looks up the MAC DA and grabs forwards the EVPN label advertised by
   NVE2 along with adjacencies info. It then encapsulates
   frame to its IRB interface just like section 6.1.1.

   - The Ethernet header of the packet
   with the corresponding label stack is stripped and forwards the packet is fed
   to NVE2.
   It should be noted that no MAC header re-write the IP-VRF; where, IP lookup is performed on GW2.
   This implies that both GW1 and GW2 need to keep the remote host MAC
   addresses along with the corresponding EVPN labels in their tables.
   The egress NVE (NVE2) then upon receiving the packet, performs a MAC destination
   address. This lookup in the MAC-VRF (identified by the received EVPN label) to
   determine the outbound port to send the traffic on.

   Figure 4 below depicts yields the forwarding model.

            NVE1            GW1             GW2            NVE2
      +------------+  +------------+  +------------+  +------------+
      |            |  |            |  |            |  |            |
      |(MAC - (IP  |  |(IP  - (MAC |  |    (MAC    |  |(IP  - (MAC |
      | VRF)   VRF)|  | VRF)   VRF)|  |     VRF)   |  | VRF)   VRF)|
      |  |     |   |  | |  |       |  |    |  |    |  |       |  | |
      +------------+  +------------+  +------------+  +------------+
         ^     v        ^  V               ^  V               ^  V
         |     |        |  |               |  |               |  |
   TS1->-+     +-->-----+  +---------------+  +---------------+  +->-TS2

  Figure 10: Inter-Subnet Forwarding Among EVPN NVEs in Different DCs
   with GW

7 TS Mobility

7.1 TS Mobility & Optimum Forwarding for TS Outbound Traffic

   Optimum forwarding fields needed for VxLAN encapsulation
   with NVE2's MAC address as the TS outbound traffic, upon TS mobility, can
   be achieved using either the anycast default Gateway inner MAC and DA, NVE'2 IP
   addresses, or using the address aliasing as discussed in [DC-
   MOBILITY].

7.2 TS Mobility & Optimum Forwarding for TS Inbound Traffic

   For optimum forwarding of the TS inbound traffic, upon TS mobility,
   all
   VTEP DA, and the NVEs and/or IP-VPN PEs need VNI. MAC SA is set to NVE1's MAC address and VTEP SA
   is set to know NVE1's IP address.

   -  The packet is then encapsulated with the up to date location
   of proper header based on
   the TS. Two scenarios must be considered, as discussed next.

   In what follows, we use above info and is forwarded to the following terminology:

   - source egress NVE refers to (NVE2).

   - On the egress NVE behind which (NVE2), assuming the TS used to reside
   prior to packet is VxLAN
   encapsulated, the TS mobility event.

   - target NVE refers to VxLAN and the new NVE behind which inner Ethernet headers are removed
   and the TS has moved
   after resultant IP packet is fed to the mobility event.

7.2.1 Mobility without Route Aggregation

   In this scenario, when a target NVE detects IP-VRF associated with that a MAC mobility event
   has occurred, it initiates
   the MAC mobility handshake in BGP as
   specified VNI.

   - Next, a lookup is performed based on IP DA (which is in section 5.1.3. The WAN Gateways, acting as ASBRs SN3) in this
   case, re-advertise the MAC route
   associated IP-VRF of NVE2. The IP lookup yields the target NVE with the MAC
   Mobility extended community attribute unmodified. Because access-facing IRB
   interface over which the WAN
   Gateway for a given data center re-advertises BGP routes received
   from packet needs to be sent. Before sending the WAN into
   packet over this interface, the data center, ARP table is consulted to get the source NVE will receive
   destination TS (TS3) MAC address.

   - The IP packet is encapsulated with an Ethernet header with the MAC Advertisement route
   SA set to that of the target NVE (with access-facing IRB interface of the next hop
   attribute adjusted depending on which inter-AS option is employed).
   The source egress NVE will then withdraw its original MAC Advertisement
   route as a result of evaluating
   (NVE2) and the Sequence Number field MAC DA is set to that of the destination TS (TS3) MAC
   Mobility extended community in
   address. The packet is sent to the received corresponding MAC-VRF3 and after a
   lookup of MAC Advertisement route.
   This DA, is per forwarded to the procedures already defined in [EVPN].

8 destination TS (TS3) over the
   corresponding interface.

7  Acknowledgements

   The authors would like to thank Sami Boutros and Boutros, Jeffrey Zhang Zhang,
   Krzysztof Szarkowicz, and Neeraj Malhotra for their valuable
   comments.

9

8  Security Considerations

   This document describes a set of procedures for Inter-Subnet
   Forwarding of tenant traffic across PEs (or NVEs). These procedures
   include both layer-2 forwarding and layer-3 routing on a packet by
   packet basis. The security considerations discussed consideration for layer-2 forwarding in [EVPN] apply
   this document follow that of [RFC7432] for MPLS encapsulation and it
   follows that of [RFC8365] for VxLAN or GENEVE encapsulations.

   Furthermore, the security consideration for layer-3 routing is this
   document follows that of [RFC4365] with the exception for application
   of routing protocols between CEs and PEs. Contrary to [RFC4364], this
   document.

10
   document does not describe route distribution techniques between CEs
   and PEs, but rather considers the CEs as TSes or VAs that do not run
   dynamic routing protocols. This can be considered a security
   advantage, since dynamic routing protocols can be blocked on the
   NVE/PE ACs, not allowing the tenant to interact with the
   infrastructure's dynamic routing protocols.

   In this document, the RT-5 is used for certain scenarios. This route
   uses an Overlay Index that requires a recursive resolution to a
   different EVPN route (an RT-2). Because of this, it is worth noting
   that any action that ends up filtering or modifying the RT-2 route
   used to convey the Overlay Indexes, will modify the resolution of the
   RT-5 and therefore the forwarding of packets to the remote subnet.

9  IANA Considerations

   IANA has allocated a new transitive extended community Type of 0x06
   and Sub-Type of 0x03 for EVPN Router's MAC Extended Community.

11

10  References

11.1

10.1  Normative References

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119, March 1997.

   [EVPN]

   [RFC8174]  Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC2119
              Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May
              2017.

   [RFC7432] Sajassi et al., "BGP MPLS Based Ethernet VPN", RFC 7432,
              February, 2015.

   [RFC8365] Sajassi et al., "A Network Virtualization Overlay Solution
              Using Ethernet VPN (EVPN)", RFC 8365, March, 2018.

   [TUNNEL-ENCAP] Rosen et al., "The BGP Tunnel Encapsulation
              Attribute", draft-ietf-idr-tunnel-encaps-03, November
              2016.

   [EVPN-PREFIX] Rabadan et al., "IP Prefix Advertisement in EVPN",
              draft-ietf-bess-evpn-prefix-advertisement-03, September,
              2016.

11.2

10.2  Informative References

   [RFC7606]  Chen, E., Scudder, J., Mohapatra, P., and K. Patel,
   "Revised Error Handling for BGP UPDATE Messages", RFC 7606, August
   2015, <http://www.rfc-editor.org/info/rfc7606>.

   [802.1Q] "IEEE Standard for Local and metropolitan area networks -
   Media Access Control (MAC) Bridges and Virtual Bridged Local Area
   Networks", IEEE Std 802.1Q(tm), 2014 Edition, November 2014.

   [EVPN-IPVPN-INTEROP] Sajassi

   [RFC7348]  Mahalingam, M., et al., "EVPN Seamless Interoperability
   with IP-VPN", draft-sajassi-l2vpn-evpn-ipvpn-interop-01, work in
   progress, October, 2012.

   [DC-MOBILITY] Aggarwal "Virtual eXtensible Local Area
   Network (VXLAN): A Framework for Overlaying Virtualized Layer 2
   Networks over Layer 3 Networks", RFC 7348, DOI 10.17487/RFC7348,
   August 2014.

   [GENEVE]  Gross, J., et al., "Data Center "Geneve: Generic Network Virtualization
   Encapsulation", Work in Progress, draft-ietf-nvo3-geneve-06, March
   2018.

   [IRB-EXT-MOBILITY] Malhotra, N., al., "Extended Mobility based on
   BGP/MPLS, IP Routing and NHRP", draft-raggarwa-data-center-mobility-
   05.txt, work in progress, June, 2013.

12 Procedures
   for EVPN-IRB", Work in Progress, draft-malhotra-bess-evpn-irb-
   extended-mobility-02, February 2018.

11  Contributors

   In addition to the authors listed on the front page, the following
   co-authors have also contributed to this document:

   Florin Balus
   Cisco

   Yakov Rekhter
   Juniper

   Wim Henderickx
   Nokia

   Lucy Yong
   Linda Dunbar
   Huawei

   Dennis Cai
   Alibaba

Authors' Addresses

   Ali Sajassi (Editor)
   Cisco
   Email: sajassi@cisco.com

   Samer Salam
   Cisco
   Email: sslam@cisco.com

   Samir Thoria
   Cisco
   Email: sthoria@cisco.com

   John E. Drake
   Juniper Networks
   Email: jdrake@juniper.net

   Lucy Yong
   Huawei Technologies
   Email: lucy.yong@huawei.com

   Jorge Rabadan
   Nokia
   Email: jorge.rabadan@nokia.com