draft-ietf-bess-service-chaining-04.txt   draft-ietf-bess-service-chaining-05.txt 
BGP Enabled Services (bess) R. Fernando Network Working Group R. Fernando
INTERNET-DRAFT Cisco Internet-Draft Cisco Systems
Intended status: Standards Track S. Mackie Intended status: Standards Track S. Mackie
Expires: August 18, 2018 Juniper Expires: February 7, 2019 Juniper Networks
D. Rao D. Rao
Cisco Cisco Systems
B. Rijsman B. Rijsman
Juniper
M. Napierala M. Napierala
AT&T ATT Labs
T. Morin T. Morin
Orange Orange
August 6, 2018
February 12, 2018 Service Function Chaining using Virtual Networks with BGP VPNs
draft-ietf-bess-service-chaining-05
Service Chaining using Virtual Networks with BGP VPNs
draft-ietf-bess-service-chaining-04
Abstract Abstract
This document describes how service function chains (SFC) can be This document describes how service function chains (SFC) can be
applied to traffic flows using routing in a virtual (overlay) network applied to traffic flows using routing in a virtual (overlay) network
to steer traffic between service nodes. Chains can include services to steer traffic between service nodes. Chains can include services
running in routers, on physical appliances or in virtual machines. running in routers, on physical appliances or in virtual machines.
Service chains have applicability at the subscriber edge, business Service chains have applicability at the subscriber edge, business
edge and in multi-tenant datacenters. The routing function into SFCs edge and in multi-tenant datacenters. The routing function into SFCs
and between service functions within an SFC can be performed by and between service functions within an SFC can be performed by
physical devices (routers), be virtualized inside hypervisors, or run physical devices (routers), be virtualized inside hypervisors, or run
as part of a host OS. as part of a host OS.
A BGP control plane for route distribution is used to create virtual A BGP control plane for route distribution is used to create virtual
networks implemented using IP MPLS, VXLAN or other suitable networks implemented using IP MPLS, VXLAN or other suitable
encapsulation, where the routes within the virtual networks cause encapsulation, where the routes within the virtual networks cause
traffic to flow through a sequence of service nodes that apply packet traffic to flow through a sequence of service nodes that apply packet
processing functions to the flows. processing functions to the flows.
skipping to change at page 2, line 12 skipping to change at page 2, line 10
In both techniques, service chains can be created by manual In both techniques, service chains can be created by manual
configuration of routes and route targets in routing systems, or configuration of routes and route targets in routing systems, or
through the use of a controller which contains a topological model of through the use of a controller which contains a topological model of
the desired service chains. the desired service chains.
This document also contains discussion of load balancing between This document also contains discussion of load balancing between
network functions, symmetric forward and reverse paths when stateful network functions, symmetric forward and reverse paths when stateful
services are involved, and use of classifiers to direct traffic into services are involved, and use of classifiers to direct traffic into
a service chain. a service chain.
Status of this Memo Status of This Memo
This Internet-Draft is submitted in full conformance with the This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79. provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that Task Force (IETF). Note that other groups may also distribute
other groups may also distribute working documents as Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts. Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at This Internet-Draft will expire on February 7, 2019.
http://www.ietf.org/ietf/1id-abstracts.txt
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html
This Internet-Draft will expire on October 13, 2016.
Copyright Notice and License Notice Copyright Notice
Copyright (c) 2016 IETF Trust and the persons identified as the Copyright (c) 2018 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (https://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License. described in the Simplified BSD License.
Table of Contents Table of Contents
1 Introduction ...................................................4 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1 Terminology................................................5 1.1. Terminology . . . . . . . . . . . . . . . . . . . . . . . 5
2 Service Function Chain Architecture Using Virtual Networking ...8 2. Service Function Chain Architecture Using Virtual Networking 7
2.1 High Level Architecture....................................9 2.1. High Level Architecture . . . . . . . . . . . . . . . . . 8
2.2 Service Function Chain Logical Model......................10 2.2. Service Function Chain Logical Model . . . . . . . . . . 10
2.3 Service Function Implemented in a Set of SF Instances.....11 2.3. Service Function Implemented in a Set of SF Instances . . 11
2.4 SF Instance Connections to VRFs...........................13 2.4. SF Instance Connections to VRFs . . . . . . . . . . . . . 12
2.4.1 SF Instance in Physical Appliance....................13 2.4.1. SF Instance in Physical Appliance . . . . . . . . . . 12
2.4.2 SF Instance in a Virtualized Environment.............14 2.4.2. SF Instance in a Virtualized Environment . . . . . . 13
2.5 Encapsulation Tunneling for Transport.....................15
2.6 SFC Creation Procedure....................................15
2.6.1 SFC Provisioning Using Sequential VPNs...............16
2.6.2 Modified-Route SFC Creation..........................18
2.7 Controller Function.......................................20
2.8 Variations on Setting Prefixes in an SFC..................21
2.8.1 Using a Default Route................................21
2.8.2 Using a Default Route and a Large Prefix.............21
2.8.3 Disaggregated Gateway Routers........................22
2.8.4 Optimizing VRF usage.................................23
2.8.5 Dynamic Entry and Exit Signaling.....................23
2.8.6 Dynamic Re-Advertisements in Intermediate systems....24
2.9 Layer-2 Virtual Networks and Services.....................24
2.10 Header Transforming Service Functions....................25
3 Load Balancing Along a Service Function Chain .................25
3.1 SF Instances Connected to Separate VRFs...................25
3.2 SF Instances Connected to the Same VRF....................26
3.3 Combination of Egress and Ingress VRF Load Balancing......27
3.4 Forward and Reverse Flow Load Balancing...................29
3.4.1 Issues with Equal Cost Multi-Path Routing............29
3.4.2 Modified ECMP with Consistent Hash...................29
3.4.3 ECMP with Flow Table.................................30
3.4.4 Dealing with different hash algorithms in an SFC.....32
4 Steering into SFCs Using a Classifier .........................32
5 External Domain Co-ordination .................................34
6 Fine-grained steering using BGP Flow-Spec .....................35
7 Controller Federation .........................................35
8 Coordination Between SF Instances and Controller using BGP ....35
9 BGP Extended Communities ......................................36
10 Summary and Conclusion........................................38
11 Security Considerations.......................................38
12 IANA Considerations...........................................38
13 Informative References........................................38
14 Acknowledgments...............................................40 2.5. Encapsulation Tunneling for Transport . . . . . . . . . . 15
2.6. SFC Creation Procedure . . . . . . . . . . . . . . . . . 15
2.6.1. SFC Provisioning Using Sequential VPNs . . . . . . . 16
2.6.2. Modified-Route SFC Creation . . . . . . . . . . . . . 17
2.6.3. Common SFC provisioning considerations . . . . . . . 19
2.7. Controller Function . . . . . . . . . . . . . . . . . . . 19
2.8. Variations on Setting Prefixes in an SFC . . . . . . . . 20
2.8.1. Using a Default Route . . . . . . . . . . . . . . . . 20
2.8.2. Using a Default Route and a Large Prefix . . . . . . 20
2.8.3. Disaggregated Gateway Routers . . . . . . . . . . . . 21
2.8.4. Optimizing VRF usage . . . . . . . . . . . . . . . . 22
2.8.5. Dynamic Entry and Exit Signaling . . . . . . . . . . 22
2.8.6. Dynamic Re-Advertisements in Intermediate Systems . . 22
2.9. Layer-2 Virtual Networks and Service Functions . . . . . 23
2.10. Header Transforming Service Functions . . . . . . . . . . 23
3. Load Balancing Along a Service Function Chain . . . . . . . . 24
3.1. SF Instances Connected to Separate VRFs . . . . . . . . . 24
3.2. SF Instances Connected to the Same VRF . . . . . . . . . 25
3.3. Combination of Egress and Ingress VRF Load Balancing . . 26
3.4. Forward and Reverse Flow Load Balancing . . . . . . . . . 27
3.4.1. Issues with Equal Cost Multi-Path Routing . . . . . . 27
3.4.2. Modified ECMP with Consistent Hash . . . . . . . . . 28
3.4.3. ECMP with Flow Table . . . . . . . . . . . . . . . . 28
3.4.4. Dealing with Different Hash Algorithms in an SFC . . 30
4. Steering into SFCs Using a Classifier . . . . . . . . . . . . 30
5. External Domain Co-ordination . . . . . . . . . . . . . . . . 32
6. Fine-grained steering using BGP Flow-Spec . . . . . . . . . . 33
7. Controller Federation . . . . . . . . . . . . . . . . . . . . 33
8. Coordination Between SF Instances and Controller using BGP . 33
9. BGP Extended Communities . . . . . . . . . . . . . . . . . . 34
9.1. Route-Target Record . . . . . . . . . . . . . . . . . . . 34
9.2. Consistent Hash Sort Order . . . . . . . . . . . . . . . 35
9.3. Load Balance Settings . . . . . . . . . . . . . . . . . . 35
10. Summary and Conclusion . . . . . . . . . . . . . . . . . . . 36
11. Security Considerations . . . . . . . . . . . . . . . . . . . 36
12. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 36
13. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 37
14. References . . . . . . . . . . . . . . . . . . . . . . . . . 37
14.1. Normative References . . . . . . . . . . . . . . . . . . 37
14.2. Informational References . . . . . . . . . . . . . . . . 38
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 39
1 Introduction 1. Introduction
The purpose of networks is to allow computing systems to communicate The purpose of networks is to allow computing systems to communicate
with each other. Requests are usually made from the client or with each other. Requests are usually made from the client or
customer side of a network, and responses are generated by customer side of a network, and responses are generated by
applications residing in a datacenter. Over time, the network between applications residing in a datacenter. Over time, the network
the client and the application has become more complex, and traffic between the client and the application has become more complex, and
between the client and the application is acted on by intermediate traffic between the client and the application is acted on by
systems that apply network services. Some of these activities, like intermediate systems that apply network services. Some of these
firewall filtering, subscriber attachment and network address activities, like firewall filtering, subscriber attachment and
translation are generally carried out in network devices along the network address translation are generally carried out in network
traffic path, while others are carried out by dedicated appliances, devices along the traffic path, while others are carried out by
such as media proxy and deep packet inspection (DPI). Deployment of dedicated appliances, such as media proxy and deep packet inspection
these in-network services is complex, time- consuming and costly, (DPI). Deployment of these in-network services is complex, time-
since they require configuration of devices with vendor-specific consuming and costly, since they require configuration of devices
operating systems, sometimes with co-processing cards, or deployment with vendor-specific operating systems, sometimes with co-processing
of physical devices in the network, which requires cabling and cards, or deployment of physical devices in the network, which
configuration of the devices that they connect to. Additionally, requires cabling and configuration of the devices that they connect
other devices in the network need to be configured to ensure that to. Additionally, other devices in the network need to be configured
traffic is correctly steered through the systems that services are to ensure that traffic is correctly steered through the systems that
running on. services are running on.
The current mode of operations does not easily allow common The current mode of operations does not easily allow common
operational processes to be applied to the lifecycle of services in operational processes to be applied to the lifecycle of services in
the network, or for steering of traffic through them. the network, or for steering of traffic through them.
The recent emergence of Network Functions Virtualization (NFV) The recent emergence of Network Functions Virtualization (NFV)
[NFVE2E] to provide a standard deployment model for network services [NFVE2E] to provide a standard deployment model for network services
as software appliances, combined with Software Defined Networking as software appliances, combined with Software Defined Networking
(SDN) for more dynamic traffic steering can provide foundational (SDN) for more dynamic traffic steering can provide foundational
elements that will allow network services to be deployed and managed elements that will allow network services to be deployed and managed
far more efficiently and with more agility than is possible today. far more efficiently and with more agility than is possible today.
This document describes how the combination of several existing This document describes how the combination of several existing
technologies can be used to create chains of functions, while technologies can be used to create chains of functions, while
preserving the requirements of scale, performance and reliability for preserving the requirements of scale, performance and reliability for
service provider networks. The technologies employed are: service provider networks. The technologies employed are:
o Traffic flow between service functions described by routing and o Traffic flow between service functions described by routing and
network policies rather than by static physical or logical network policies rather than by static physical or logical
connectivity connectivity
o Packet header encapsulation in order to create virtual private o Packet header encapsulation in order to create virtual private
networks using network overlays networks using network overlays
o VRFs on both physical devices and in hypervisors to implement o VRFs on both physical devices and in hypervisors to implement
forwarding policies that are specific to each virtual network forwarding policies that are specific to each virtual network
o Optional use of a controller to calculate routes to be installed o Optional use of a controller to calculate routes to be installed
in routing systems to form a service chain. The controller uses a in routing systems to form a service chain. The controller uses a
topological model that stores service function instance topological model that stores service function instance
connectivity to network devices and intended connectivity between connectivity to network devices and intended connectivity between
service functions. service functions.
o MPLS or other labeling to facilitate identification of the next o MPLS or other labeling to facilitate identification of the next
interface to send packets to in a service function chain interface to send packets to in a service function chain
o BGP or BGP-style signaling to distribute routes in order to create o BGP or BGP-style signaling to distribute routes in order to create
service function chains service function chains
o Distributed load balancing between service functions performed in o Distributed load balancing between service functions performed in
the VRFs that service function instance connect to. the VRFs that service function instance connect to.
Virtualized environments can be supported without necessarily running Virtualized environments can be supported without necessarily running
BGP or MPLS natively. Messaging protocols such as NC/YANG, XMPP or BGP or MPLS natively. Messaging protocols such as NC/YANG, XMPP or
OpenFlow may be used to signal forwarding information. Encapsulation OpenFlow may be used to signal forwarding information. Encapsulation
mechanisms such as VXLAN or GRE may be used for overlay transport. mechanisms such as VXLAN or GRE may be used for overlay transport.
The term 'BGP-style', above, refers to this type of signaling. The term 'BGP-style', above, refers to this type of signaling.
Traffic can be directed into service function chains using IP routing Traffic can be directed into service function chains using IP routing
at each end of the service function chain, or be directed into the at each end of the service function chain, or be directed into the
chain by a classifier function that can determine which service chain chain by a classifier function that can determine which service chain
a traffic flow should pass through based on deep packet inspection a traffic flow should pass through based on deep packet inspection
(DPI) and/or subscriber identity. (DPI) and/or subscriber identity.
The techniques can support an evolution from services implemented in The techniques can support an evolution from services implemented in
physical devices attached to physical forwarding systems (routers) to physical devices attached to physical forwarding systems (routers) to
fully virtualized implementations as well as intermediate hybrid fully virtualized implementations as well as intermediate hybrid
implementations. implementations.
1.1 Terminology 1.1. Terminology
This document uses the following acronyms and terms. This document uses the following acronyms and terms.
Terms Meaning Terms Meaning
----- ----------------------------------------------- ----- -----------------------------------------------
AS Autonomous System AS Autonomous System
ASBR Autonomous System Border Router ASBR Autonomous System Border Router
RR Route Reflector
RT Route Target
SDN Software Defined Network
VM Virtual Machine
VPN Virtual Private Network
VRF VPN Routing and Forwarding table [RFC4364]
FW Firewall Table 1
I2RS Interface to the Routing System
L3VPN Layer 3 VPN
LB Load Balancer
NLRI Network Layer Reachability Information [RFC4271]
P Provider backbone router
proxy-arp proxy-Address Resolution Protocol
RR Route Reflector
RT Route Target
SDN Software Defined Network
vCE virtual Customer Edge router
[I-D.fang-l3vpn-virtual-ce]
vFW virtual Firewall
vLB virtual Load Balancer
VM Virtual Machine
vPC virtual Private Cloud
vPE virtual Provider Edge router
[I-D.fang-l3vpn-virtual-pe]
VPN Virtual Private Network
VRF VPN Routing and Forwarding table [RFC4364]
vRR virtual Route Reflector
This document follows some of the terminology used in [sfc-arch] and This document follows some of the terminology used in [RFC7665] and
adds some new terminology: adds some new terminology:
Network Service: An externally visible service offered by a network Network Service: An externally visible service offered by a network
operator; a service may consist of a single service function or a operator; a service may consist of a single service function or a
composite built from several service functions executed in one or composite built from several service functions executed in one or
more pre-determined sequences and delivered by software executing more pre-determined sequences and delivered by software executing
in physical or virtual devices. in physical or virtual devices.
Classification: Customer/network/service policy used to identify and Classification: Customer/network/service policy used to identify and
select traffic flow(s) requiring certain outbound forwarding select traffic flow(s) requiring certain outbound forwarding
actions, in particular, to direct specific traffic flows into the actions, in particular, to direct specific traffic flows into the
ingress of a particular service function chain, or causing ingress of a particular service function chain, or causing
branching within a service function chain. branching within a service function chain.
Virtual Network: A logical overlay network built using virtual links Virtual Network: A logical overlay network built using virtual links
or packet encapsulation, over an existing network (the underlay). or packet encapsulation, over an existing network (the underlay).
Service Function Chain (SFC): A service function chain defines an Service Function Chain (SFC): A service function chain defines an
ordered set of service functions that must be applied to packets ordered set of service functions that must be applied to packets
and/or frames selected as a result of classification. An SFC may be and/or frames selected as a result of classification. An SFC may
either a linear chain or a complex service graph with multiple be either a linear chain or a complex service graph with multiple
branches. The term 'Service Chain' is often used in place of branches. The term 'Service Chain' is often used in place of
'Service Function Chain'. 'Service Function Chain'.
SFC Set: The pair of SFCs through which the forward and reverse SFC Set: The pair of SFCs through which the forward and reverse
directions of a given classified flow will pass. directions of a given classified flow will pass.
Service Function (SF): A logical function that is applied to Service Function (SF): A logical function that is applied to
packets. A service function can act at the network layer or other packets. A service function can act at the network layer or other
OSI layers. A service function can be embedded in one or more OSI layers. A service function can be embedded in one or more
physical network elements, or can be implemented in one or more physical network elements, or can be implemented in one or more
software instances running on physical or virtual hosts. One or software instances running on physical or virtual hosts. One or
multiple service functions can be embedded in the same network multiple service functions can be embedded in the same network
element or run on the same host. Multiple instances of a service element or run on the same host. Multiple instances of a service
function can be enabled in the same administrative domain. We will function can be enabled in the same administrative domain. We
also refer to 'Service Function' as, simply, 'Service' for will also refer to 'Service Function' as, simply, 'Service' for
simplicity. simplicity.
A non-exhaustive list of services includes: firewalls, DDOS
protection, anti-malware/ant-virus systems, WAN and application
acceleration, Deep Packet Inspection (DPI), server load balancers,
network address translation, HTTP Header Enrichment functions,
video optimization, TCP optimization, etc.
SF Instance: An instance of software that implements the packet A non-exhaustive list of services includes: firewalls, DDOS
processing of a service function protection, anti-malware/ant-virus systems, WAN and application
acceleration, Deep Packet Inspection (DPI), server load balancers,
network address translation, HTTP Header Enrichment functions,
video optimization, TCP optimization, etc.
SF Instance Set: A group of SF instances that, in parallel, implement SF Instance: An instance of software that implements the packet
a service function in an SFC. processing of a service function
Routing System: A hardware or software system that performs layer 3 SF Instance Set: A group of SF instances that, in parallel,
routing and/or forwarding functions. The term includes physical implement a service function in an SFC.
routers as well as hypervisor or Host OS implementations of the
forwarding plane of a conventional router.
Gateway: A routing system attached to the source or destination Routing System: A hardware or software system that performs layer 3
network that peers with the controller, or with the routing system routing and/or forwarding functions. The term includes physical
at one end of an SFC. A source network gateway directs traffic from routers as well as hypervisor or Host OS implementations of the
the source network into an SFC, while a destination network gateway forwarding plane of a conventional router.
distributes traffic towards destinations. The routing systems at
each end of an SFC can themselves act as gateways and in a
bidirectional SF instance set, gateways can act in both directions
VRF: A subsystem within a routing system as defined in [RFC4364] that Gateway: A routing system attached to the source or destination
contains private routing and forwarding tables and has physical network that peers with the controller, or with the routing system
and/or logical interfaces associated with it. In the case of at one end of an SFC. A source network gateway directs traffic
hypervisor/Host OS implementations, the term refers only to the from the source network into an SFC, while a destination network
forwarding function of a VRF, and this will be referred to as a gateway distributes traffic towards destinations. The routing
'VPN forwarder.' systems at each end of an SFC can themselves act as gateways and
in a bidirectional SF instance set, gateways can act in both
directions VRF: A subsystem within a routing system as defined in
[RFC4364] that contains private routing and forwarding tables and
has physical and/or logical interfaces associated with it. In the
case of hypervisor/Host OS implementations, the term refers only
to the forwarding function of a VRF, and this will be referred to
as a 'VPN forwarder.'
Ingress VRF: A VRF containing an ingress interface of a SF instance Ingress VRF: A VRF containing an ingress interface of a SF instance
Egress VRF: A VRF containing an egress interface of a SF instance Egress VRF: A VRF containing an egress interface of a SF instance
Note that in this document the terms 'ingress' and 'egress' are used Note that in this document the terms 'ingress' and 'egress' are
with respect to SF instances rather than the tunnels that connect SF used with respect to SF instances rather than the tunnels that
instances. This is different usage than in VPN literature in general. connect SF instances. This is different usage than in VPN
literature in general.
Entry VRF: A VRF through which traffic enters the SFC from the source Entry VRF: A VRF through which traffic enters the SFC from the
network. This VRF may be used to advertise the destination source network. This VRF may be used to advertise the destination
network's routes to the source network. It could be placed on a network's routes to the source network. It could be placed on a
gateway router or be collocated with the first ingress VRF. gateway router or be collocated with the first ingress VRF.
Exit VRF: A VRF through which traffic exits the SFC into the Exit VRF: A VRF through which traffic exits the SFC into the
destination network. This VRF contains the routes from the destination network. This VRF contains the routes from the
destination network and could be located on a gateway router. destination network and could be located on a gateway router.
Alternatively, the egress VRF attached to the last SF instance may Alternatively, the egress VRF attached to the last SF instance may
also function as the exit VRF. also function as the exit VRF.
2 Service Function Chain Architecture Using Virtual Networking 2. Service Function Chain Architecture Using Virtual Networking
The techniques described in this document use virtual networks to The techniques described in this document use virtual networks to
implement service function chains. Service function chains can be implement service function chains. Service function chains can be
implemented on devices that support existing MPLS VPN and BGP implemented on devices that support existing MPLS VPN and BGP
standards [RFC4364, RFC4271, RFC4760], as well as other standards [RFC4364], [RFC4271], [RFC4760], as well as other
encapsulations, such as VXLAN [RFC7348]. Similarly, equivalent encapsulations, such as VXLAN [RFC7348]. Similarly, equivalent
control plane protocols such as BGP-EVPN with type-2 and type-5 route control plane protocols such as BGP-EVPN with type-2 and type-5 route
types can also be used where supported. The set of techniques types can also be used where supported [RFC8365]. The set of
described in this document represent one implementation approach to techniques described in this document represent one implementation
realize the SFC architecture described in [sfc-arch]. approach to realize the SFC architecture described in [RFC7665].
The following sections detail the building blocks of the SFC The following sections detail the building blocks of the SFC
architecture, and outline the processes of route installation and architecture, and outline the processes of route installation and
subsequent route exchange to create an SFC. subsequent route exchange to create an SFC.
2.1 High Level Architecture 2.1. High Level Architecture
Service function chains can be deployed with or without a classifier. Service function chains can be deployed with or without a classifier.
Use cases where SFCs may be deployed without a classifier include Use cases where SFCs may be deployed without a classifier include
multi-tenant data centers, private and public cloud and virtual CPE multi-tenant data centers, private and public cloud and virtual CPE
for business services. Classifiers will primarily be used in mobile for business services. Classifiers will primarily be used in mobile
and wireline subscriber edge use cases. Use of a classifier is and wireline subscriber edge use cases. Use of a classifier is
discussed in Section 4. discussed in Section 4.
A high-level architecture diagram of an SFC without a classifier, A high-level architecture diagram of an SFC without a classifier,
where traffic is routed into and out of the SFC, is shown in Figure where traffic is routed into and out of the SFC, is shown in
1, below. An optional controller is shown that contains a topological Figure 1, below. An optional controller is shown that contains a
model of the SFC and which configures the network resources to topological model of the SFC and which configures the network
implement the SFC. resources to implement the SFC.
+-------------------------+ +-------------------------+
|--- Data plane connection| |--- Data plane connection|
|=== Encapsulation tunnel | |=== Encapsulation tunnel |
| O VRF | | O VRF |
+-------------------------+ +-------------------------+
Control +------------------------------------------------+ Control +------------------------------------------------+
Plane | Controller | Plane | Controller |
....... +-+------------+----------+----------+---------+-+ ....... +-+------------+----------+----------+---------+-+
| | | | | | | | | |
Service | +---+ | +---+ | +---+ | | Service | +---+ | +---+ | +---+ | |
Plane | |SF1| | |SF2| | |SF3| | | Plane | |SF1| | |SF2| | |SF3| | |
| +---+ | +---+ | +---+ | | | +---+ | +---+ | +---+ | |
....... / | | / | | / | | / / ....... / | | / | | / | | / /
+-----+ +--|-|--+ +--|-|--+ +--|-|--+ +-----+ +-----+ +--|-|--+ +--|-|--+ +--|-|--+ +-----+
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
Net-A-->---O==========O O========O O========O O=========O---->Net-B Net-A-->---O==========O O========O O========O O=========O---->Net-B
| | | | | | | | | | | | | | | | | | | |
Data | R-A | | R-1 | | R-2 | | R-3 | | R-B | Data | R-A | | R-1 | | R-2 | | R-3 | | R-B |
Plane +-----+ +-------+ +-------+ +-------+ +-----+ Plane +-----+ +-------+ +-------+ +-------+ +-----+
^ ^ ^ ^ ^ ^ ^ ^
| | | | | | | |
| Ingress Egress | | Ingress Egress |
| VRF VRF | | VRF VRF |
SFC Entry SFC Exit SFC Entry SFC Exit
VRF VRF VRF VRF
Figure 1 - High level SFC Architecture High Level SFC Architecture
Figure 1
Traffic from Network-A destined for Network-B will pass through the Traffic from Network-A destined for Network-B will pass through the
SFC composed of SF instances, SF1, SF2 and SF3. Routing system R-A SFC composed of SF instances, SF1, SF2 and SF3. Routing system R-A
contains a VRF (shown as 'O' symbol) that is the SFC entry point. contains a VRF (shown as 'O' symbol) that is the SFC entry point.
This VRF will advertise a route to reach Network-B into Network-A This VRF will advertise a route to reach Network-B into Network-A
causing any traffic from a source in Network-A with a destination in causing any traffic from a source in Network-A with a destination in
Network-B to arrive in this VRF. The forwarding table in the VRF in Network-B to arrive in this VRF. The forwarding table in the VRF in
R-A will direct traffic destined for Network-B into an encapsulation R-A will direct traffic destined for Network-B into an encapsulation
tunnel with destination R-1 and a label that identifies the ingress tunnel with destination R-1 and a label that identifies the ingress
(left) interface of SF1 that R-1 should send the packets out on. The (left) interface of SF1 that R-1 should send the packets out on. The
packets are processed by service instance SF-1 and arrive in the packets are processed by service instance SF-1 and arrive in the
egress (right) VRF in R-1. The forwarding entries in the egress VRF egress (right) VRF in R-1. The forwarding entries in the egress VRF
direct traffic to the next ingress VRF using encapsulation tunneling. direct traffic to the next ingress VRF using encapsulation tunneling.
The process is repeated for each service instance in the SFC until The process is repeated for each service instance in the SFC until
packets arrive at the SFC exit VRF (in R-B). This VRF is peered with packets arrive at the SFC exit VRF (in R-B). This VRF is peered with
Network-B and routes packets towards their destinations in the user Network-B and routes packets towards their destinations in the user
data plane. In this example, routing systems R-A and R-B are gateway data plane. In this example, routing systems R-A and R-B are gateway
routing systems. routing systems.
In the example, each pair of ingress and egress VRFs are configured In the example, each pair of ingress and egress VRFs are configured
in separate routing systems, but such pairs could be collocated in in separate routing systems, but such pairs could be collocated in
the same routing system, and it is possible for the ingress and the same routing system, and it is possible for the ingress and
egress VRFs for a given SF instance to be in different routing egress VRFs for a given SF instance to be in different routing
systems. The SFC entry and exit VRFs can be collocated in the same systems. The SFC entry and exit VRFs can be collocated in the same
routing system, and the service instances can be local or remote from routing system, and the service instances can be local or remote from
either or both of the routing systems containing the entry and exit either or both of the routing systems containing the entry and exit
VRFs, and from each other. It is also possible that the ingress and VRFs, and from each other. It is also possible that the ingress and
egress VRFs are implemented using alternative mechanisms. egress VRFs are implemented using alternative mechanisms.
The controller is responsible for configuring the VRFs in each The controller is responsible for configuring the VRFs in each
routing system, installing the routes in each of the VRFs to routing system, installing the routes in each of the VRFs to
implement the SFC, and, in the case of virtualized services, may implement the SFC, and, in the case of virtualized services, may
instantiate the service instances. instantiate the service instances.
2.2 Service Function Chain Logical Model 2.2. Service Function Chain Logical Model
A service function chain is a set of logically connected service A service function chain is a set of logically connected service
functions through which traffic can flow. Each egress interface of functions through which traffic can flow. Each egress interface of
one service function is logically connected to an ingress interface one service function is logically connected to an ingress interface
of the next service function. of the next service function.
+------+ +------+ +------+ +------+ +------+ +------+
Network-A-->| SF-1 |-->| SF-2 |-->| SF-3 |-->Network-B Network-A-->| SF-1 |-->| SF-2 |-->| SF-3 |-->Network-B
+------+ +------+ +------+ +------+ +------+ +------+
Figure 2 - A Chain of Service Functions A Chain of Service Functions
Figure 2
In Figure 2, above, a service function chain has been created that In Figure 2, above, a service function chain has been created that
connects Network-A to Network-B, such that traffic from a host in connects Network-A to Network-B, such that traffic from a host in
Network-A to a host in Network-B will traverse the service function Network-A to a host in Network-B will traverse the service function
chain. chain.
As defined in [sfc-arch], a service function chain can be uni- As defined in [RFC7665], a service function chain can be uni-
directional or bi-directional. In this document, in order to allow directional or bi-directional. In this document, in order to allow
for the possibility that the forward and reverse paths may not be for the possibility that the forward and reverse paths may not be
symmetrical, SFCs are defined as uni-directional, and the term 'SFC symmetrical, SFCs are defined as uni-directional, and the term 'SFC
set' is used to refer to a pair of forward and reverse direction SFCs set' is used to refer to a pair of forward and reverse direction SFCs
for some set of routed or classified traffic. for some set of routed or classified traffic.
2.3 Service Function Implemented in a Set of SF Instances 2.3. Service Function Implemented in a Set of SF Instances
A service function instance is a software system that acts on packets A service function instance is a software system that acts on packets
that arrive on an ingress interface of that software system. Service that arrive on an ingress interface of that software system. Service
function instances may run on a physical appliance or in a virtual function instances may run on a physical appliance or in a virtual
machine. A service function instance may be transparent at layer 2 machine. A service function instance may be transparent at layer 2
and/or layer 3, and may support branching across multiple egress and/or layer 3, and may support branching across multiple egress
interfaces and may support aggregation across ingress interfaces. For interfaces and may support aggregation across ingress interfaces.
simplicity, the examples in this document have a single ingress and a For simplicity, the examples in this document have a single ingress
single egress interface. and a single egress interface.
Each service function in a chain can be implemented by a single Each service function in a chain can be implemented by a single
service function instance, or by a set of instances in order to service function instance, or by a set of instances in order to
provide scale and resilience. provide scale and resilience.
+------------------------------------------------------------------+ +------------------------------------------------------------------+
| Logical Service Functions Connected in a Chain | | Logical Service Functions Connected in a Chain |
| | | |
| +--------+ +--------+ | | +--------+ +--------+ |
| Net-A--->| SF-1 |----------->| SF-2 |--->Net-B | | Net-A--->| SF-1 |----------->| SF-2 |--->Net-B |
skipping to change at page 12, line 29 skipping to change at page 11, line 44
| : : +------+ : : +------+ : : | | : : +------+ : : +------+ : : |
| A->: VN-1 :-->|SFI-12|-->: VN-2 : : VN-3 :-->B | | A->: VN-1 :-->|SFI-12|-->: VN-2 : : VN-3 :-->B |
| : : +------+ : : +------+ : : | | : : +------+ : : +------+ : : |
| : : : :-->|SFI-22|-->: : | | : : : :-->|SFI-22|-->: : |
| : : +------+ : : +------+ : : | | : : +------+ : : +------+ : : |
| : :-->|SFI-13|-->: : '''''' | | : :-->|SFI-13|-->: : '''''' |
| : : +------+ : : | | : : +------+ : : |
| '''''' '''''' | | '''''' '''''' |
+------------------------------------------------------------------+ +------------------------------------------------------------------+
Figure 3 - Service Functions Are Composed of SF Instances Connected Service Functions Are Composed of SF Instances
Via Virtual Networks Connected Via Virtual Networks
Figure 3
In Figure 3, service function SF-1 is implemented in three service In Figure 3, service function SF-1 is implemented in three service
function instances, SFI-11, SFI-12, and SFI-13. Service function SF- function instances, SFI-11, SFI-12, and SFI-13. Service function SF-
2 is implemented in two SF instances. The service function instances 2 is implemented in two SF instances. The service function instances
are connected to the next service function in the chain using a are connected to the next service function in the chain using a
virtual network, VN-2. Additionally, a virtual network (VN-1) is used virtual network, VN-2. Additionally, a virtual network (VN-1) is
to enter the SFC and another (VN-3) is used at the exit. used to enter the SFC and another (VN-3) is used at the exit.
The logical connection between two service functions is implemented The logical connection between two service functions is implemented
using a virtual network that contains egress interfaces for instances using a virtual network that contains egress interfaces for instances
of one service function, and ingress interfaces of instances of the of one service function, and ingress interfaces of instances of the
next service function. Traffic is directed across the virtual network next service function. Traffic is directed across the virtual
between the two sets of service function instances using layer 3 network between the two sets of service function instances using
forwarding (e.g. an MPLS VPN) or layer 2 forwarding (e.g. a VXLAN). layer 3 forwarding (e.g. an MPLS VPN) or layer 2 forwarding (e.g. a
VXLAN).
The virtual networks could be described as "directed half-mesh", in The virtual networks could be described as "directed half-mesh", in
that the egress interface of each SF instance of one service function that the egress interface of each SF instance of one service function
can reach any ingress interface of the SF instances of the connected can reach any ingress interface of the SF instances of the connected
service function. service function.
Details on how routing across virtual networks is achieved, and Details on how routing across virtual networks is achieved, and
requirements on load balancing across ingress interfaces are requirements on load balancing across ingress interfaces are
discussed in later sections of this document. discussed in later sections of this document.
2.4 SF Instance Connections to VRFs 2.4. SF Instance Connections to VRFs
SF instances can be deployed as software running on physical SF instances can be deployed as software running on physical
appliances, or in virtual machines running on a hypervisor. These two appliances, or in virtual machines running on a hypervisor. These
types are described in more detail in the following sections. two types are described in more detail in the following sections.
2.4.1 SF Instance in Physical Appliance 2.4.1. SF Instance in Physical Appliance
The case of a SF instance running on a physical appliance is shown in The case of a SF instance running on a physical appliance is shown in
Figure 4, below. Figure 4, below.
+---------------------------------+ +---------------------------------+
| | | |
| +-----------------------------+ | | +-----------------------------+ |
| | Service Function Instance | | | | Service Function Instance | |
| +-------^-------------|-------+ | | +-------^-------------|-------+ |
| | Host | | | | Host | |
+---------|-------------|---------+ +---------|-------------|---------+
| | | |
+------ |-------------|-------+ +------ |-------------|-------+
| | | | | | | |
| +----|----+ +-----v----+ | | +----|----+ +-----v----+ |
---------+ Ingress | | Egress +--------- ---------+ Ingress | | Egress +---------
---------> VRF | | VRF ----------> ---------> VRF | | VRF ---------->
---------+ | | +--------- ---------+ | | +---------
| +---------+ +----------+ | | +---------+ +----------+ |
| Routing System | | Routing System |
+-----------------------------+ +-----------------------------+
Figure 4 - Ingress and Egress VRFs for a Physical Routing System and Ingress and Egress VRFs for a Physical Routing System
Physical SF Instance and Physical SF Instance
Figure 4
The routing system is a physical device and the service function The routing system is a physical device and the service function
instance is implemented as software running in a physical appliance instance is implemented as software running in a physical appliance
(host) connected to it. The connection between the physical device (host) connected to it. The connection between the physical device
and the routing system may use physical or logical interfaces. and the routing system may use physical or logical interfaces.
Transport between VRFs on different routing systems that are Transport between VRFs on different routing systems that are
connected to other SF instances in an SFC is via encapsulation connected to other SF instances in an SFC is via encapsulation
tunnels, such as MPLS over GRE, or VXLAN. tunnels, such as MPLS over GRE, or VXLAN.
2.4.2 SF Instance in a Virtualized Environment 2.4.2. SF Instance in a Virtualized Environment
In virtualized environments, a routing system with VRFs that act as In virtualized environments, a routing system with VRFs that act as
VPN forwarders is resident in the hypervisor/Host OS, and is co- VPN forwarders is resident in the hypervisor/Host OS, and is co-
resident in the host with one or more SF instances that run in resident in the host with one or more SF instances that run in
virtual machines. The egress VPN forwarder performs tunnel virtual machines. The egress VPN forwarder performs tunnel
encapsulation to send packets to other physical or virtual routing encapsulation to send packets to other physical or virtual routing
systems with attached SF instances to form an SFC. The tunneled systems with attached SF instances to form an SFC. The tunneled
packets are sent through the physical interfaces of the host to the packets are sent through the physical interfaces of the host to the
other hosts or physical routers. This is illustrated in Figure 5, other hosts or physical routers. This is illustrated in Figure 5,
below. below.
+-------------------------------------+ +-------------------------------------+
| +-----------------------------+ | | +-----------------------------+ |
| | Service Function Instance | | | | Service Function Instance | |
| +-------^-------------|-------+ | | +-------^-------------|-------+ |
| | | | | | | |
| +---------|-------------|---------+ | | +---------|-------------|---------+ |
| | +-------|-------------|-------+ | | | | +-------|-------------|-------+ | |
| | | | | | | | | | | | | | | |
| | | +----|----+ +-----v----+ | | | | | | +----|----+ +-----v----+ | | |
------------+ Ingress | | Egress +----------- ------------+ Ingress | | Egress +-----------
------------> VRF | | VRF ------------> ------------> VRF | | VRF ------------>
------------+ | | +----------- ------------+ | | +-----------
| | | +---------+ +----------+ | | | | | | +---------+ +----------+ | | |
| | | Routing System | | | | | | Routing System | | |
| | +-----------------------------+ | | | | +-----------------------------+ | |
| | Hypervisor or Host OS | | | | Hypervisor or Host OS | |
| +---------------------------------+ | | +---------------------------------+ |
| Host | | Host |
+-------------------------------------+ +-------------------------------------+
Figure 5 - Ingress and Egress VRFs for a Virtual Routing System and Ingress and Egress VRFs for a Virtual Routing System
Virtualized SF Instance and Virtualized SF Instance
Figure 5
When more than one instance of an SF is running on a hypervisor, they When more than one instance of an SF is running on a hypervisor, they
can be connected to the same VRF for scale out of an SF within an can be connected to the same VRF for scale out of an SF within an
SFC. SFC.
The routing mechanisms in the VRFs into and between service function The routing mechanisms in the VRFs into and between service function
instances, and the encapsulation tunneling between routing systems instances, and the encapsulation tunneling between routing systems
are identical in the physical and virtual implementations of SFCs and are identical in the physical and virtual implementations of SFCs and
routing systems described in this document. Physical and virtual routing systems described in this document. Physical and virtual
service functions can be mixed as needed with different combinations service functions can be mixed as needed with different combinations
of physical and virtual routing systems, within a single service of physical and virtual routing systems, within a single service
chain. chain.
The SF instances are attached to the routing systems via physical, The SF instances are attached to the routing systems via physical,
virtual or logical (e.g, 802.1q) interfaces, and are assumed to virtual or logical (e.g, 802.1q) interfaces, and are assumed to
perform basic L3 or L2 forwarding. perform basic L3 or L2 forwarding.
A single SF instance can be part of multiple service chains. In this A single SF instance can be part of multiple service chains. In this
case, the SF instance will have dedicated interfaces (typically case, the SF instance will have dedicated interfaces (typically
logical) and forwarding contexts associated with each service chain. logical) and forwarding contexts associated with each service chain.
2.5 Encapsulation Tunneling for Transport 2.5. Encapsulation Tunneling for Transport
Encapsulation tunneling is used to transport packets between SF Encapsulation tunneling is used to transport packets between SF
instances in the chain and, when a classifier is not used, from the instances in the chain and, when a classifier is not used, from the
originating network into the SFC and from the SFC into the originating network into the SFC and from the SFC into the
destination network. destination network.
The tunnels can be MPLS over GRE [RFC4023], MPLS over UDP [draft- The tunnels can be MPLS over GRE [RFC4023], MPLS over UDP [RFC7510],
ietf-mpls-in-udp], MPLS over MPLS [RFC3031], VXLAN [RFC7348], or MPLS over MPLS [RFC3031], VXLAN [RFC7348][RFC7348], or another
another suitable encapsulation methods. suitable encapsulation method.
Tunneling capabilities may be enabled in each routing system as part Tunneling capabilities may be enabled in each routing system as part
of a base configuration or may be configured by the controller. of a base configuration or may be configured by the controller.
Tunnel encapsulations may be programmed by the controller or signaled Tunnel encapsulations may be programmed by the controller or signaled
using BGP. The encapsulation to be used for a given route is signaled using BGP. The encapsulation to be used for a given route is
in BGP using the procedures described in [draft-rosen- signaled in BGP using the procedures described in
idr-tunnel-encaps], i.e. typically relying on the BGP Tunnel [idr-tunnel-encaps], i.e. typically relying on the BGP Tunnel
Encapsulation Extended Community. Encapsulation Extended Community.
2.6 SFC Creation Procedure 2.6. SFC Creation Procedure
This section describes how service chains are created using two This section describes how service chains are created using two
methods: methods:
o Sequential VPNs - where a conventional VPN is created between each o Sequential VPNs - where a conventional VPN is created between each
set of SF instances to create the links in the SFC set of SF instances to create the links in the SFC
o Route Modification - where each routing system modifies advertised o Route Modification - where each routing system modifies advertised
routes that it receives, to realize the links in an SFC on the routes that it receives, to realize the links in an SFC on the
basis of a special service topology RT and a route- policy that basis of a special service topology RT and a route- policy that
describes the service chain logical topology describes the service chain logical topology
In both cases the controller, when present, is responsible for In both cases the controller, when present, is responsible for
creating ingress and egress VRFs, configuring the interfaces creating ingress and egress VRFs, configuring the interfaces
connected to SF instances in each VRF, and allocating and configuring connected to SF instances in each VRF, and allocating and configuring
import and export RTs for each VRF. Additionally, in the second import and export RTs for each VRF. Additionally, in the second
method, the controller also sends the route-policy containing the method, the controller also sends the route-policy containing the
service chain logical topology to each routing system. If a service chain logical topology to each routing system. If a
controller is not used, these procedures will require to be performed controller is not used, these procedures will require to be performed
manually or through scripting, for instance. manually or through scripting, for instance.
The source and destination networks' prefixes can be configured in The source and destination networks' prefixes can be configured in
the controller, or may be automatically learned through peering the controller, or may be automatically learned through peering
between the controller and each network's gateway. This is further between the controller and each network's gateway. This is further
described in Section 2.8.5 and Section 5. described in Section 2.8.5 and Section 5.
The following sub-sections describe how RT configuration, local route The following sub-sections describe how RT configuration, local route
installation and route distribution occur in each of the methods. installation and route distribution occur in each of the methods.
It should be noted that depending on the capabilities of the routing It should be noted that depending on the capabilities of the routing
systems, a controller can use one or more techniques to realize systems, a controller can use one or more techniques to realize
forwarding along the service chain, ranging from fully centralized to forwarding along the service chain, ranging from fully centralized to
fully distributed. The goal of describing the following two methods fully distributed. The goal of describing the following two methods
is to illustrate the broad approaches and as a base for various is to illustrate the broad approaches and as a base for various
optimization options. optimization options.
Interoperability between a controller implementing one method and a Interoperability between a controller implementing one method and a
controller implementing a different method is achieved by relying on controller implementing a different method is achieved by relying on
the techniques described in section 5 and section 8, that describe the techniques described in section 5 and section 8, that describe
the use of BGP-style service chaining within domains that are the use of BGP-style service chaining within domains that are
interconnected using standard BGP VPN route exchanges. interconnected using standard BGP VPN route exchanges.
2.6.1 SFC Provisioning Using Sequential VPNs 2.6.1. SFC Provisioning Using Sequential VPNs
The task of the controller in this method of SFC provisioning is to The task of the controller in this method of SFC provisioning is to
create a set of VPNs that carry traffic to the destination network create a set of VPNs that carry traffic to the destination network
through instances of each service function in turn. This is achieved through instances of each service function in turn. This is achieved
by allocating and configuring RTs such that the egress VRFs of one by allocating and configuring RTs such that the egress VRFs of one
set of SF instances import an RT that is an export RT for the ingress set of SF instances import an RT that is an export RT for the ingress
VRFs of the next, logically connected, set of SF instances. VRFs of the next, logically connected, set of SF instances.
The process of SFC creation is as follows: The process of SFC creation is as follows:
1. Controller creates a VRF in each routing system that is 1. Controller creates a VRF in each routing system that is connected
connected to a service instance that will be used in the SFC to a service instance that will be used in the SFC
2. Controller configures each VRF to contain the logical 2. Controller configures each VRF to contain the logical interface
interface that connects to a SF instance. that connects to a SF instance.
3. Controller implements route target import and export 3. Controller implements route target import and export policies in
policies in the VRFs using the same route targets for the the VRFs using the same route targets for the egress VRFs of a
egress VRFs of a service function and the ingress VRFs of service function and the ingress VRFs of the next logically
the next logically connected service function in the SFC. connected service function in the SFC.
4. Controller installs a static route in each ingress VRF whose 4. Controller installs a static route in each ingress VRF whose next
next hop is the interface that a SF instance is connected hop is the interface that a SF instance is connected to. The
to. The prefix for the route is the destination network to prefix for the route is the destination network to be reached by
be reached by passing through the SFC. The following passing through the SFC. The following sections describe
sections describe variations that can be used. variations that can be used.
5. Routing systems advertise the static routes via BGP as VPN 5. Routing systems advertise the static routes via BGP as VPN routes
routes with next hop being the IP address of the router, with next hop being the IP address of the router, with an
with an encapsulation specified and a label that identifies encapsulation specified and a label that identifies the service
the service instance interface. instance interface.
6. Routing systems containing VRFs with matching route targets 6. Routing systems containing VRFs with matching route targets
receive the updates. receive the updates.
7. Routes are installed in egress VRFs with matching import 7. Routes are installed in egress VRFs with matching import targets.
targets. The egress VRFs of each SF instance will now The egress VRFs of each SF instance will now contain VPN routes
contain VPN routes to one or more routers containing ingress to one or more routers containing ingress VRFs for SF instances
VRFs for SF instances of the next service function in the of the next service function in the SFC.
SFC.
Routes to the destination network via the first set of SF instances Routes to the destination network via the first set of SF instances
are advertised into the source network, and the egress VRFs of the are advertised into the source network, and the egress VRFs of the
last SF instance set have routes into the destination network. last SF instance set have routes into the destination network.
As discussed further in Section 3, egress VRFs can load balance As discussed further in Section 3, egress VRFs can load balance
across the multiple next hops advertised from the next set of ingress across the multiple next hops advertised from the next set of ingress
VRFs. VRFs.
2.6.2 Modified-Route SFC Creation 2.6.2. Modified-Route SFC Creation
In this method of SFC configuration, all the VRFs connected to SF In this method of SFC configuration, all the VRFs connected to SF
instances for a given SFC are configured with same import and export instances for a given SFC are configured with same import and export
RT, so they form a VPN-connected mesh between the SF instance RT, so they form a VPN-connected mesh between the SF instance
interfaces. This is termed the 'Service VPN'. A route is configured interfaces. This is termed the 'Service VPN'. A route is configured
or learnt in each VRF with destination being the IP address of a or learnt in each VRF with destination being the IP address of a
connected SF instance via an interface configured in the VRF. The connected SF instance via an interface configured in the VRF. The
interface may be a physical or logical interface. The routing system interface may be a physical or logical interface. The routing system
that hosts such a VRF advertises a VPN route for each locally that hosts such a VRF advertises a VPN route for each locally
connected SF instance, with a forwarding label that enables it to connected SF instance, with a forwarding label that enables it to
forward incoming traffic from other routing systems to the connected forward incoming traffic from other routing systems to the connected
SF instance. The VPN routes may be advertised via an RR or the SF instance. The VPN routes may be advertised via an RR or the
controller, which sends these updates to all the other routing controller, which sends these updates to all the other routing
systems that have VRFs with the service VPN RT. At this point all the systems that have VRFs with the service VPN RT. At this point all
VRFs have a route to reach every SF instance. The same virtual IP the VRFs have a route to reach every SF instance. The same virtual
address may be used for each SF instance in a set, enabling load- IP address may be used for each SF instance in a set, enabling load-
balancing among multiple SF instances in the set. balancing among multiple SF instances in the set.
The controller builds a route-policy for the routing systems in the The controller builds a route-policy for the routing systems in the
VPN, that describes the logical topology of each service chain that VPN, that describes the logical topology of each service chain that
it belongs to. The route-policy contains entries in the form of a it belongs to. The route-policy contains entries in the form of a
tuple for each service chain: tuple for each service chain:
{Service-topology-name, Service-topology-RT, Service-node- {Service-topology-name, Service-topology-RT, Service-node- sequence}
sequence}
where Service-node-sequence is simply an ordered list of the service where Service-node-sequence is simply an ordered list of the service
function interface IP addresses that are in the chain. function interface IP addresses that are in the chain.
Every service function chain has a single unique service-topology-RT Every service function chain has a single unique service-topology-RT
that is allocated and provisioned on all participating routing that is allocated and provisioned on all participating routing
systems in the relevant VRFs. systems in the relevant VRFs.
The VRF in the routing system that connects to the destination The VRF in the routing system that connects to the destination
network (i.e. the exit VRF) is configured to attach the Service- network (i.e. the exit VRF) is configured to attach the Service-
topology-RT to exported routes, and the VRF connected to the source topology-RT to exported routes, and the VRF connected to the source
network (i.e. the entry VRF) will import routes using the Service- network (i.e. the entry VRF) will import routes using the Service-
topology-RT. The controller may also be used to originate the topology-RT. The controller may also be used to originate the
Service-topology-RT attached routes. Service-topology-RT attached routes.
The route-policy may be described in a variety of formats and The route-policy may be described in a variety of formats and
installed on the routing system using a suitable mechanism. For installed on the routing system using a suitable mechanism. For
instance, the policy may be defined in YANG and provisioned using instance, the policy may be defined in YANG and provisioned using
Netconf. Netconf [RFC6241].
Using Figure 1 for reference, when the gateway R-B advertises a VPN Using Figure 1 for reference, when the gateway R-B advertises a VPN
route to Network-B, it attaches the Service-topology-RT. BGP route route to Network-B, it attaches the Service-topology-RT. BGP route
updates are sent to all the routing systems in the service VPN. The updates are sent to all the routing systems in the service VPN. The
routing systems perform a modified set of actions for next-hop routing systems perform a modified set of actions for next-hop
resolution and route installation in the ingress VRFs compared to resolution and route installation in the ingress VRFs compared to
normal BGP VPN behavior in routing systems, but no changes are normal BGP VPN behavior in routing systems, but no changes are
required in the operation of the BGP protocol itself. The required in the operation of the BGP protocol itself. The
modification of behavior in the routing systems allows the automatic modification of behavior in the routing systems allows the automatic
and constrained flow of traffic through the service chain. and constrained flow of traffic through the service chain.
Each routing system in the service VPN will process the VPN route to Each routing system in the service VPN will process the VPN route to
Network-B via R-B as follows: Network-B via R-B as follows:
1. If the routing system contains VRFs that import the 1. If the routing system contains VRFs that import the Service-
Service-topology-RT, continue, otherwise ignore the route. topology-RT, continue, otherwise ignore the route.
2. The routing system identifies the position and role 2. The routing system identifies the position and role (ingress/
(ingress/egress) of each of its VRFs in the SFC by comparing egress) of each of its VRFs in the SFC by comparing the IP
the IP address of the route in the VRF to the connected SF address of the route in the VRF to the connected SF instance with
instance with those in the Service-node- sequence in the those in the Service-node- sequence in the route-policy.
route-policy. Alternatively, the controller may provision Alternatively, the controller may provision the specific service
the specific service node IP to be used as the next-hop in node IP to be used as the next-hop in each VRF, in the route-
each VRF, in the route-policy for the VRF. policy for the VRF.
3. The routing system modifies the next-hop of the imported 3. The routing system modifies the next-hop of the imported route
route with the Service-topology-RT, to select the with the Service-topology-RT, to select the appropriate next-hop
appropriate next-hop as per the route-policy. It ignores the as per the route-policy. It ignores the next-hop and label in
next-hop and label in the received route. It resolves the the received route. It resolves the selected next-hop in the
selected next-hop in the local VRF routing table. local VRF routing table.
a. The imported route to Network-B in the ingress VRF is 4.
modified to have a next-hop of the IP address of the
logically connected SF instance.
b. The imported route to Network-B in the egress VRF is a. The imported route to Network-B in the ingress VRF is
modified to have a next hop of the IP address of the modified to have a next-hop of the IP address of the
next SF instance in the SFC. logically connected SF instance.
4. The egress VRFs for the last service function install the b. The imported route to Network-B in the egress VRF is modified
VPN route via the gateway R-B unmodified. to have a next hop of the IP address of the next SF instance
in the SFC.
5. The egress VRFs for the last service function install the VPN
route via the gateway R-B unmodified.
Note that the modified routes are not re-advertised into the VPN by Note that the modified routes are not re-advertised into the VPN by
the various intermediate routing systems in the SFC. the various intermediate routing systems in the SFC.
2.6.3 Common SFC provisioning considerations 2.6.3. Common SFC provisioning considerations
In both the methods, for physical routers, the creation and In both the methods, for physical routers, the creation and
configuration of VRFs, interfaces and local static routes can be configuration of VRFs, interfaces and local static routes can be
performed programmatically using Netconf; and BGP route distribution performed programmatically using Netconf; and BGP route distribution
can use a route reflector (which may be part of the controller). In can use a route reflector (which may be part of the controller). In
the virtualized case, where a VPN forwarder is present, creation and the virtualized case, where a VPN forwarder is present, creation and
configuration of VRFs, interfaces and installation of routes may configuration of VRFs, interfaces and installation of routes may
instead be performed using a single protocol like XMPP, NC/YANG or an instead be performed using a single protocol like XMPP, NC/YANG or an
equivalent programmatic interface. equivalent programmatic interface.
Also in the virtualized case, the actual forwarding table entries to Also in the virtualized case, the actual forwarding table entries to
be installed in the ingress and egress VRFs may be calculated by the be installed in the ingress and egress VRFs may be calculated by the
controller based on its internal knowledge of the required SFC controller based on its internal knowledge of the required SFC
topology and the connectivity of SF instances to routing systems. In topology and the connectivity of SF instances to routing systems. In
this case, the routes may be directly installed in the forwarders this case, the routes may be directly installed in the forwarders
using the programmatic interface and no BGP route advertisement is using the programmatic interface and no BGP route advertisement is
necessary, except when coordination with external domains (Section 5) necessary, except when coordination with external domains (Section 5)
or federation between controller domains is employed (Section 7). or federation between controller domains is employed (Section 7).
Note however that this is just one typical model for a virtual Note however that this is just one typical model for a virtual
forwarding based system. In general, physical and virtual routing forwarding based system. In general, physical and virtual routing
systems can be treated exactly the same if they have the same systems can be treated exactly the same if they have the same
capabilities. capabilities.
In both the methods, the SF instance may also need to be set up In both the methods, the SF instance may also need to be set up
appropriately to forward traffic between it's input and output appropriately to forward traffic between it's input and output
interfaces, either via static, dynamic or policy-based routing. If interfaces, either via static, dynamic or policy-based routing. If
the service function is a transparent L2 service, then the static the service function is a transparent L2 service, then the static
route installed in the ingress VRF will have a next-hop of the IP route installed in the ingress VRF will have a next-hop of the IP
address of the routing system interface that the service instance is address of the routing system interface that the service instance is
attached to on its other interface. attached to on its other interface.
2.7 Controller Function 2.7. Controller Function
The purpose of the controller is to manage instantiation of SFCs in The purpose of the controller is to manage instantiation of SFCs in
networks and datacenters. When an SFC is to be instantiated, a model networks and datacenters. When an SFC is to be instantiated, a model
of the desired topology (service functions, number of instances, of the desired topology (service functions, number of instances,
connectivity) is built in the controller either via an API or GUI. connectivity) is built in the controller either via an API or GUI.
The controller then selects resources in the infrastructure that will The controller then selects resources in the infrastructure that will
support the SFC and configures them. This can involve instantiation support the SFC and configures them. This can involve instantiation
of SF instances to implement each service function, the instantiation of SF instances to implement each service function, the instantiation
of VRFs that will form virtual networks between SF instances, and of VRFs that will form virtual networks between SF instances, and
installation of routes to cause traffic to flow into and between SF installation of routes to cause traffic to flow into and between SF
instances. It can also include provisioning the necessary static, instances. It can also include provisioning the necessary static,
dynamic or policy based forwarding on the service function instance dynamic or policy based forwarding on the service function instance
to enable it to forward traffic. to enable it to forward traffic.
For simplicity, in this document, the controller is assumed to For simplicity, in this document, the controller is assumed to
contain all the required features for management of SFCs. In actual contain all the required features for management of SFCs. In actual
implementations, these features may be distributed among multiple implementations, these features may be distributed among multiple
inter-connected systems. E.g. An overarching orchestrator might inter-connected systems. E.g. An overarching orchestrator might
manage the overall SFC model, sending instructions to a separate manage the overall SFC model, sending instructions to a separate
virtual machine manager to instantiate service function instances, virtual machine manager to instantiate service function instances,
and to a virtual network manager to set up the service chain and to a virtual network manager to set up the service chain
connections between them. connections between them.
The controller can also perform necessary BGP signaling and route The controller can also perform necessary BGP signaling and route
distribution actions as described throughout this document. distribution actions as described throughout this document.
2.8 Variations on Setting Prefixes in an SFC 2.8. Variations on Setting Prefixes in an SFC
The SFC Creation section above described the basic procedures for a The SFC Creation section above described the basic procedures for a
couple of SFC creation methods. This section describes some couple of SFC creation methods. This section describes some
techniques that can extend and provide optimizations on top of the techniques that can extend and provide optimizations on top of the
basic procedures. basic procedures.
2.8.1 Using a Default Route 2.8.1. Using a Default Route
In the methods described above, it can be noted that only the gateway In the methods described above, it can be noted that only the gateway
routing systems need the specific network prefixes to steer traffic routing systems need the specific network prefixes to steer traffic
in and out of the SFC. The intermediate systems can direct traffic in in and out of the SFC. The intermediate systems can direct traffic
the ingress and egress VRFs by using only a default route. Hence, it in the ingress and egress VRFs by using only a default route. Hence,
is possible to avoid installing the network prefixes in the it is possible to avoid installing the network prefixes in the
intermediate systems. This can be done by splitting the SFC into two intermediate systems. This can be done by splitting the SFC into two
sections - one linking the entry and exit VRFs and the other sections - one linking the entry and exit VRFs and the other
including the intermediate systems. For instance, this may be including the intermediate systems. For instance, this may be
achieved by using two different Service-topology-RTs in the second achieved by using two different Service-topology-RTs in the second
method. method.
2.8.2 Using a Default Route and a Large Prefix 2.8.2. Using a Default Route and a Large Prefix
In the configuration methods described above, the network prefixes In the configuration methods described above, the network prefixes
for each network (Network-A and Network-B in the example above) for each network (Network-A and Network-B in the example above)
connected to the SFC are used in the routes that direct traffic connected to the SFC are used in the routes that direct traffic
through the SFC. This creates an operational linkage between the through the SFC. This creates an operational linkage between the
implementation of the SFC and the insertion of the SFC into a implementation of the SFC and the insertion of the SFC into a
network. network.
For instance, subscriber network prefixes will normally be segmented For instance, subscriber network prefixes will normally be segmented
across subscriber attachment points such as broadband or mobile across subscriber attachment points such as broadband or mobile
gateways. This means that each SFC would have to be configured with gateways. This means that each SFC would have to be configured with
the subscriber network prefixes whose traffic it is handling. the subscriber network prefixes whose traffic it is handling.
In a variation of the SFC configuration method described above, the In a variation of the SFC configuration method described above, the
prefixes used in each direction can be such that they include all prefixes used in each direction can be such that they include all
possible addresses at each side of the SFC. For example, in Figure 1, possible addresses at each side of the SFC. For example, in
the prefix for Network-A could include all subscriber IP addresses Figure 1, the prefix for Network-A could include all subscriber IP
and the prefix for Network-B could be the default route, 0/0. addresses and the prefix for Network-B could be the default route,
0/0.
Using this technique, the same routes can be installed in all Using this technique, the same routes can be installed in all
instances of an SFC that serve different groups of subscribers in instances of an SFC that serve different groups of subscribers in
different geographic locations. different geographic locations.
The routes forwarding traffic into a SF instance and to the next SF The routes forwarding traffic into a SF instance and to the next SF
instance are installed when an SFC is initially built, and each time instance are installed when an SFC is initially built, and each time
a SF instance is connected into the SFC, but there is no requirement a SF instance is connected into the SFC, but there is no requirement
for VRFs to be reconfigured when traffic from different networks pass for VRFs to be reconfigured when traffic from different networks pass
through the service chain, so long as their prefix is included in the through the service chain, so long as their prefix is included in the
prefixes in the VRFs along the SFC. prefixes in the VRFs along the SFC.
In this variation, it is assumed that no subscriber-originated In this variation, it is assumed that no subscriber-originated
traffic will enter the SFC destined for an IP address also in the traffic will enter the SFC destined for an IP address also in the
subscriber network address range. This will not be a restriction in subscriber network address range. This will not be a restriction in
many cases. many cases.
2.8.3 Disaggregated Gateway Routers 2.8.3. Disaggregated Gateway Routers
As a slight variation of the above, a network prefix may be As a slight variation of the above, a network prefix may be
disaggregated and spread out among various gateway routers, for disaggregated and spread out among various gateway routers, for
instance, in the case of virtual machines in a data-center. In order instance, in the case of virtual machines in a data-center. In order
to reduce the scaling requirements on the routing systems along the to reduce the scaling requirements on the routing systems along the
SFC, the SFC can again be split into two sections as described above. SFC, the SFC can again be split into two sections as described above.
In addition, the last egress VRF may act as the exit VRF and install In addition, the last egress VRF may act as the exit VRF and install
the destination network's disaggregated routes. If the destination the destination network's disaggregated routes. If the destination
network's prefixes can be aggregated, for instance into a subnet network's prefixes can be aggregated, for instance into a subnet
prefix, then the aggregate prefix may be advertised and installed in prefix, then the aggregate prefix may be advertised and installed in
the entry VRF. the entry VRF.
2.8.4 Optimizing VRF usage 2.8.4. Optimizing VRF usage
It may be desirable to avoid using distinct ingress and egress VRFs It may be desirable to avoid using distinct ingress and egress VRFs
for the service instances in order to make more efficient use of VRF for the service instances in order to make more efficient use of VRF
resources, especially on physical routing systems. The ingress VRF resources, especially on physical routing systems. The ingress VRF
and egress VRF may be treated as conceptual entities and the and egress VRF may be treated as conceptual entities and the
forwarding realized using one or more options described in this forwarding realized using one or more options described in this
section, combined with the methods described earlier. section, combined with the methods described earlier.
For instance, the next-hop forwarding label described earlier serves For instance, the next-hop forwarding label described earlier serves
the purpose of directing traffic received from other routing systems the purpose of directing traffic received from other routing systems
directly towards an attached service instance. On the other hand, if directly towards an attached service instance. On the other hand, if
the encapsulation mechanism or the device in use requires an IP the encapsulation mechanism or the device in use requires an IP
lookup for incoming packets from other routing systems, then the lookup for incoming packets from other routing systems, then the
specific network prefixes may be installed in the intermediate specific network prefixes may be installed in the intermediate
service VRFs to direct traffic towards the attached service service VRFs to direct traffic towards the attached service
instances. instances.
Similarly, a per-interface policy-based-routing rule applied to an Similarly, a per-interface policy-based-routing rule applied to an
access interface can serve to direct traffic coming in from attached access interface can serve to direct traffic coming in from attached
service instances towards the next SF set. service instances towards the next SF set.
2.8.5 Dynamic Entry and Exit Signaling 2.8.5. Dynamic Entry and Exit Signaling
When either of the methods of the previous sections are employed, the When either of the methods of the previous sections are employed, the
prefixes of the attached networks at each end of an SFC can be prefixes of the attached networks at each end of an SFC can be
signaled into the corresponding VRFs dynamically. This requires that signaled into the corresponding VRFs dynamically. This requires that
a BGP session is configured either from the network device at each a BGP session is configured either from the network device at each
end of the SFC into each network or from the controller. end of the SFC into each network or from the controller.
If dynamic signaling is performed, and a bidirectional SFC set is If dynamic signaling is performed, and a bidirectional SFC set is
configured, and the gateways to the networks connected via the SFC configured, and the gateways to the networks connected via the SFC
exchange routes, steps must be taken to ensure that routes to both exchange routes, steps must be taken to ensure that routes to both
networks do not get advertised from both ends of the SFC set by re- networks do not get advertised from both ends of the SFC set by re-
origination. This can be achieved if a new BGP Extended Community is origination. This can be achieved if a new BGP Extended Community
implemented to control re-origination. When a route is re-originated, [RFC4360] is implemented to control re-origination. When a route is
the RTs of the re-originated routes are appended to the new Route- re- originated, the RTs of the re-originated routes are appended to
Target Record Extended Community, and if the RT for the route already the new RT-Record Extended Community, and if the RT for the route
exists in the Extended Community, the route is not re-originated (see already exists in the Extended Community, the route is not re-
Section 9.1). originated (see Section 9.1).
2.8.6 Dynamic Re-Advertisements in Intermediate systems 2.8.6. Dynamic Re-Advertisements in Intermediate Systems
The intermediate routing systems attached to the service instances The intermediate routing systems attached to the service instances
may also use the dynamic signaling technique from the previous may also use the dynamic signaling technique from the previous
section to re-advertise received routes up the chain. In this case, section to re-advertise received routes up the chain. In this case,
the ingress and egress VRFs are combined into one; and a local the ingress and egress VRFs are combined into one; and a local route-
route-policy ensures the re-advertised routes are associated with policy ensures the re-advertised routes are associated with labels
labels that direct incoming traffic directly to the attached service that direct incoming traffic directly to the attached service
instances on that routing system. instances on that routing system.
2.9 Layer-2 Virtual Networks and Service Functions 2.9. Layer-2 Virtual Networks and Service Functions
There are SFs that operate at layer-2, in a transparent mode, and There are SFs that operate at layer-2, in a transparent mode, and
forward traffic based on the MAC DA. When such a SF is present in the forward traffic based on the MAC DA. When such a SF is present in
SFC, the procedures at the routing system are modified slightly. In the SFC, the procedures at the routing system are modified slightly.
this case, the IP address associated with the SF instance (and used In this case, the IP address associated with the SF instance (and
as the next-hop of routes in the above procedures) is actually the used as the next-hop of routes in the above procedures) is actually
one assigned to the routing system interface attached to the other the one assigned to the routing system interface attached to the
end of the SF instance, or it could be a virtual IP address logically other end of the SF instance, or it could be a virtual IP address
associated with the service function with a next-hop of the other logically associated with the service function with a next-hop of the
routing system interface. The routing system interface uses distinct other routing system interface. The routing system interface uses
interface MAC addresses. This allows the current scheme to be distinct interface MAC addresses. This allows the current scheme to
supported, while allowing the transparent service function to work be supported, while allowing the transparent service function to work
using its existing behavior. using its existing behavior.
A SFC may be also be set up between end systems or network segments A SFC may be also be set up between end systems or network segments
within the same Layer-2 bridged network. In this case, applying the within the same Layer-2 bridged network. In this case, applying the
procedures described earlier, the segments or groups of end systems procedures described earlier, the segments or groups of end systems
are placed in distinct Layer-2 virtual networks, which are then then are placed in distinct Layer-2 virtual networks, which are then then
inter-connected via a sequence of intermediate Layer-2 virtual inter-connected via a sequence of intermediate Layer-2 virtual
networks that form the links in the SFC. Each virtual network maps to networks that form the links in the SFC. Each virtual network maps
a pair of ingress and egress MAC VRFs on the routing systems to which to a pair of ingress and egress MAC VRFs on the routing systems to
the SF instances are attached. The routing systems at the ends of the which the SF instances are attached. The routing systems at the ends
SFC will advertise the locally learnt or installed MAC entries using of the SFC will advertise the locally learnt or installed MAC entries
BGP-EVPN type-2 routes, which will get installed in the MAC VRFs at using BGP-EVPN type-2 routes, which will get installed in the MAC
the other end. The intermediate systems may use default MAC routes VRFs at the other end. The intermediate systems may use default MAC
installed in the ingress and egress MAC VRFs, or the other variations routes installed in the ingress and egress MAC VRFs, or the other
described earlier in this document. variations described earlier in this document.
2.10 Header Transforming Service Functions 2.10. Header Transforming Service Functions
If a service function performs an action that changes the source If a service function performs an action that changes the source
address in the packet header (e.g., NAT), the routes that were address in the packet header (e.g., NAT), the routes that were
installed as described above may not support reverse flow traffic. installed as described above may not support reverse flow traffic.
The solution to this is for the controller modify the routes in the The solution to this is for the controller modify the routes in the
reverse direction to direct traffic into instances of the reverse direction to direct traffic into instances of the
transforming service function. The original routes with a source transforming service function. The original routes with a source
prefix (Network-A in Figure 2) are replaced with a route that has a prefix (Network-A in Figure 2) are replaced with a route that has a
prefix that includes all the possible addresses that the source prefix that includes all the possible addresses that the source
address could be mapped to. In the case of network address address could be mapped to. In the case of network address
translation, this would correspond to the NAT pool. translation, this would correspond to the NAT pool.
3 Load Balancing Along a Service Function Chain 3. Load Balancing Along a Service Function Chain
One of the key concepts driving NFV [NFVE2E]is the idea that each One of the key concepts driving NFV [NFVE2E]is the idea that each
service function along an SFC can be separately scaled by changing service function along an SFC can be separately scaled by changing
the number of service function instances that implement it. This the number of service function instances that implement it. This
requires that load balancing be performed before entry into each requires that load balancing be performed before entry into each
service function. In this architecture, load balancing is performed service function. In this architecture, load balancing is performed
in either or both of egress and ingress VRFs depending on the type of in either or both of egress and ingress VRFs depending on the type of
load balancing being performed, and if more than one service instance load balancing being performed, and if more than one service instance
is connected to the same ingress VRF. is connected to the same ingress VRF.
3.1 SF Instances Connected to Separate VRFs 3.1. SF Instances Connected to Separate VRFs
If SF instances implementing a service in an SFC are each connected If SF instances implementing a service in an SFC are each connected
to separate VRFs(e.g. instances are connected to different routers or to separate VRFs(e.g. instances are connected to different routers or
are running on different hosts), load balancing is performed in the are running on different hosts), load balancing is performed in the
egress VRFs of the previous service, or in the VRF that is the entry egress VRFs of the previous service, or in the VRF that is the entry
to the SFC. The controller distributes BGP multi-path routes to the to the SFC. The controller distributes BGP multi-path routes to the
egress VRFs. The destination prefix of each route is the ultimate egress VRFs. The destination prefix of each route is the ultimate
destination network, or its representative aggregate or default. The destination network, or its representative aggregate or default. The
next-hops in the ECMP set are BGP next-hops of the service instances next-hops in the ECMP set are BGP next-hops of the service instances
attached to ingress VRFs of the next service in the SFC. The load attached to ingress VRFs of the next service in the SFC. The load
balancing corresponds to BGP Multipath, which requires that the route balancing corresponds to BGP Multipath, which requires that the route
distinguishers for each route are distinct in order to recognize that distinguishers for each route are distinct in order to recognize that
distinct paths should be used. Hence, each VRF in a distributed, SFC distinct paths should be used. Hence, each VRF in a distributed, SFC
environment should have a unique route distinguisher. environment should have a unique route distinguisher.
+------+ +-------------------------+ +------+ +-------------------------+
O----|SFI-11|---O |--- Data plane connection| O----|SFI-11|---O |--- Data plane connection|
// +------+ \\ |=== Encapsulation tunnel | // +------+ \\ |=== Encapsulation tunnel |
// \\ | O VRF | // \\ | O VRF |
// \\ | * Load balancer | // \\ | * Load balancer |
// \\ +-------------------------+ // \\ +-------------------------+
// +------+ \\ // +------+ \\
Net-A-->O*====O----|SFI-12|---O====O-->Net-B Net-A-->O*====O---|SFI-12|---O====O-->Net-B
\\ +------+ // \\ +------+ //
\\ // \\ //
\\ // \\ //
\\ // \\ //
\\ +------+ // \\ +------+ //
O----|SFI-13|---O O----|SFI-13|---O
+------+ +------+
Figure 6 - Egress VRF Load Balancing across SF Instances Connected Egress VRF Load Balancing across SF Instances
to Different VRFs Connected to Different VRFs
Figure 6
In the diagram, above, a service function is implemented in three In the diagram, above, a service function is implemented in three
service instances each connected to separate VRFs. Traffic from service instances each connected to separate VRFs. Traffic from
Network-A arrives at VRF at the start of the SFC, and is load Network-A arrives at VRF at the start of the SFC, and is load
balanced across the service instances using a set of ECMP routes with balanced across the service instances using a set of ECMP routes with
next hops being the addresses of the routing systems containing the next hops being the addresses of the routing systems containing the
ingress VRFs and with labels that identify the ingress interfaces of ingress VRFs and with labels that identify the ingress interfaces of
the service instances. the service instances.
3.2 SF Instances Connected to the Same VRF In the case that the bandwidth of the links between the load balancer
and the ingress VRFs are unequal, or that the bandwidth capacity of
the service function instances are unequal, this can be signalled in
the routes for each ingress VRF using the extended community
described in [draft-ietf-idr-link-bandwidth] and procedures from
[draft-malhotra-bess-evpn-unequal-lb] could be followed.
3.2. SF Instances Connected to the Same VRF
When SF instances implementing a service in an SFC are connected to When SF instances implementing a service in an SFC are connected to
the same ingress VRF, load balancing is performed in the ingress VRF the same ingress VRF, load balancing is performed in the ingress VRF
across the service instances connected to it. The controller will across the service instances connected to it. The controller will
install routes in the ingress VRF to the destination network with the install routes in the ingress VRF to the destination network with the
interfaces connected to each service instance as next hops. The interfaces connected to each service instance as next hops. The
ingress VRF will then use ECMP to load balance across the service ingress VRF will then use ECMP to load balance across the service
instances. instances.
+------+ +-------------------------+ +------+ +-------------------------+
|SFI-11| |--- Data plane connection| |SFI-11| |--- Data plane connection|
+------+ |=== Encapsulation tunnel | +------+ |=== Encapsulation tunnel |
/ \ | O VRF | / \ | O VRF |
/ \ | * Load balancer | / \ | * Load balancer |
/ \ +-------------------------+ / \ +-------------------------+
/ +------+ \ / +------+ \
Net-A-->O====O*----|SFI-12|----O====O-->Net-B Net-A-->O====O*---|SFI-12|----O====O-->Net-B
\ +------+ / \ +------+ /
\ / \ /
\ / \ /
\ / \ /
+------+ +------+
|SFI-13| |SFI-13|
+------+ +------+
Figure 7 - Ingress VRF Load Balancing across SF Instances Ingress VRF Load Balancing across SF Instances
Connected to the Same VRF Connected to the Same VRF
Figure 7
In the diagram, above, a service is implemented by three service In the diagram, above, a service is implemented by three service
instances that are connected to the same ingress and egress VRFs. The instances that are connected to the same ingress and egress VRFs.
ingress VRF load balances across the ingress interfaces using ECMP,
and the egress traffic is aggregated in the egress VRF. The ingress VRF load balances across the ingress interfaces using
ECMP, and the egress traffic is aggregated in the egress VRF.
If forwarding labels that identify each SFI ingress interface are If forwarding labels that identify each SFI ingress interface are
used, and if the routes to each SF instance are advertised with used, and if the routes to each SF instance are advertised with
different route distinguishers, then it is possible to perform ECMP different route distinguishers, then it is possible to perform ECMP
load balancing at the routing instance at the beginning of the load balancing at the routing instance at the beginning of the
encapsulation tunnel (which could be the egress VRF of the previous encapsulation tunnel (which could be the egress VRF of the previous
SF in the SFC). SF in the SFC).
3.3 Combination of Egress and Ingress VRF Load Balancing 3.3. Combination of Egress and Ingress VRF Load Balancing
In Figure 8, below, an example SFC is shown where load balancing is In Figure 8, below, an example SFC is shown where load balancing is
performed in both ingress and egress VRFs. performed in both ingress and egress VRFs.
+-------------------------+ +-------------------------+
|--- Data plane connection| |--- Data plane connection|
+------+ |=== Encapsulation tunnel | +------+ |=== Encapsulation tunnel |
|SFI-11| | O VRF | |SFI-11| | O VRF |
+------+ | * Load balancer | +------+ | * Load balancer |
/ \ +-------------------------+ / \ +-------------------------+
/ \ / \
/ +------+ \ +------+ / +------+ \ +------+
O*---|SFI-12|---O*====O---|SFI-21|---O O*---|SFI-12|---O*====O---|SFI-21|---O
// +------+ \\ // +------+ \\ // +------+ \\ // +------+ \\
// \\// \\ // \\// \\
// \\ \\ // \\ \\
// //\\ \\ // //\\ \\
// +------+ // \\ +------+ \\ // +------+ // \\ +------+ \\
Net-A-->O*====O----|SFI-13|---O*====O---|SFI-22|---O====O-->Net-B Net-A-->O*====O----|SFI-13|---O*====O---|SFI-22|---O====O-->Net-B
+------+ +------+ +------+ +------+
^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^
| | | | | | | | | | | |
| Ingress Egress | | | | Ingress Egress | | |
| Ingress Egress | | Ingress Egress |
SFC Entry SFC Exit SFC Entry SFC Exit
Figure 8 - Load Balancing across SF Instances Load Balancing across SF Instances
Figure 8
In Figure 8, above, an SFC is composed of two services implemented by In Figure 8, above, an SFC is composed of two services implemented by
three service instances and two service instances, respectively. The three service instances and two service instances, respectively. The
service instances SFI-11 and SFI-12 are connected to the same ingress service instances SFI-11 and SFI-12 are connected to the same ingress
and egress VRFs, and all the other service instances are connected to and egress VRFs, and all the other service instances are connected to
separate VRFs. separate VRFs.
Traffic entering the SFC from Network-A is load balanced across the Traffic entering the SFC from Network-A is load balanced across the
ingress VRFs of the first service function by the chain entry VRF, ingress VRFs of the first service function by the chain entry VRF,
and then load balanced again across the ingress interfaces of SFI-11 and then load balanced again across the ingress interfaces of SFI-11
and SFI-12 by the shared ingress VRF. Note that use of standard ECMP and SFI-12 by the shared ingress VRF. Note that use of standard ECMP
will lead to an uneven distribution of traffic between the three will lead to an uneven distribution of traffic between the three
service instances (25% to SFI-11, 25% to SFI-12, and 50% to SFI-13). service instances (25% to SFI-11, 25% to SFI-12, and 50% to SFI-13).
This issue can be mitigated through the use of BGP link bandwidth This issue can be mitigated through the use of BGP link bandwidth
extended community [draft-ietf-idr-link-bandwidth]. As described in extended community [draft-ietf-idr-link-bandwidth] and use of
the previous section, if a next-hop forwarding label is used, another procedures described in [draft-malhotra-bess-evpn-unequal-lb]. As
way to mitigate this effect would be to advertise routes to each SF described in the previous section, if a next-hop forwarding label is
instance connected to a VRF with a different route distinguisher. used, another way to mitigate this effect would be to advertise
routes to each SF instance connected to a VRF with a different route
distinguisher.
After traffic passes through the first set of service instances, it After traffic passes through the first set of service instances, it
is load balanced in each of the egress VRFs of the first set of is load balanced in each of the egress VRFs of the first set of
service instances across the ingress VRFs of the next set of service service instances across the ingress VRFs of the next set of service
instances. instances.
3.4 Forward and Reverse Flow Load Balancing 3.4. Forward and Reverse Flow Load Balancing
This section discusses requirements in load balancing for forward and This section discusses requirements in load balancing for forward and
reverse paths when stateful service functions are deployed. reverse paths when stateful service functions are deployed.
3.4.1 Issues with Equal Cost Multi-Path Routing 3.4.1. Issues with Equal Cost Multi-Path Routing
As discussed in the previous sections, load balancing in the forward As discussed in the previous sections, load balancing in the forward
SFC in the above example can automatically occur with standard BGP, SFC in the above example can automatically occur with standard BGP,
if multiple equal cost routes to Network-B are installed into all the if multiple equal cost routes to Network-B are installed into all the
ingress VRFs, and each route directs traffic through a different ingress VRFs, and each route directs traffic through a different
service function instance in the next set. The multiple BGP routes in service function instance in the next set. The multiple BGP routes
the routing table will translate to Equal Cost Multi-Path in the in the routing table will translate to Equal Cost Multi-Path in the
forwarding table. The hash used in the load balancing algorithm (per forwarding table. The hash used in the load balancing algorithm (per
packet, per flow or per prefix) is implementation specific. packet, per flow or per prefix) is implementation specific.
If a service function is stateful, it is required that forward flows If a service function is stateful, it is required that forward flows
and reverse flows always pass through the same service function and reverse flows always pass through the same service function
instance. Standard ECMP does not provide this capability, since the instance. Standard ECMP does not provide this capability, since the
hash calculation will see different input data for the same flow in hash calculation will see different input data for the same flow in
the forward and reverse directions (since the source and destination the forward and reverse directions (since the source and destination
fields are reversed). fields are reversed).
Additionally, if the number of SF instances changes, either Additionally, if the number of SF instances changes, either
increasing to expand capacity, or decreases (planned, or due to a SF increasing to expand capacity, or decreases (planned, or due to a SF
instance failure), the hash table in ECMP is recalculated, and most instance failure), the hash table in ECMP is recalculated, and most
flows will be directed to a different SF instance and user sessions flows will be directed to a different SF instance and user sessions
will be disrupted. will be disrupted.
There are a number of ways to satisfy the requirements of symmetric There are a number of ways to satisfy the requirements of symmetric
forward/reverse paths for flows and minimal disruption when SF forward/reverse paths for flows and minimal disruption when SF
instances are added to or removed from a set. Two techniques that can instances are added to or removed from a set. Two techniques that
be employed are described in the following sections. can be employed are described in the following sections.
3.4.2 Modified ECMP with Consistent Hash 3.4.2. Modified ECMP with Consistent Hash
Symmetric forwarding into each side of an SF instance set can be Symmetric forwarding into each side of an SF instance set can be
achieved with a small modification to ECMP if the packet headers are achieved with a small modification to ECMP if the packet headers are
preserved after passing through the SF instance set and assuming that preserved after passing through the SF instance set and assuming that
the same hash function, same hash salt and same ordering association the same hash function, same hash salt and same ordering association
of hash buckets to ECMP routes is used in both directions. Each packet's of hash buckets to ECMP routes is used in both directions. Each
5-tuple data is used to calculate which hash bucket, and therefore packet's 5-tuple data is used to calculate which hash bucket, and
which service instance, that the packet will be sent to, but the source therefore which service instance, that the packet will be sent to,
and destination IP address and port information are swapped in the but the source and destination IP address and port information are
calculation in the reverse direction. This method only requires that the swapped in the calculation in the reverse direction. This method
list of available service function instances is consistently maintained only requires that the list of available service function instances
in load balance tables in all the routing systems rather than maintaining is consistently maintained in load balance tables in all the routing
flow tables. This requirement can be met by the use of a distinct VPN systems rather than maintaining flow tables. This requirement can be
route for each instance. met by the use of a distinct VPN route for each instance.
In the SFC architecture described in this document, when SF instances In the SFC architecture described in this document, when SF instances
are added or removed, the controller is required to install (or are added or removed, the controller is required to install (or
remove) routes to the SF instances. The controller could configure remove) routes to the SF instances. The controller could configure
the load balancing function in VRFs that connect to each added (or the load balancing function in VRFs that connect to each added (or
removed) SF instance as part of the same network transaction as route removed) SF instance as part of the same network transaction as route
updates to ensure that the load balancer configuration is updates to ensure that the load balancer configuration is
synchronized with the set of SF instances. synchronized with the set of SF instances.
The consistent ordering among ECMP routes in the routing systems The consistent ordering among ECMP routes in the routing systems
could be achieved through configuration of the routing systems by the could be achieved through configuration of the routing systems by the
controller using, for instance, Netconf; or when the routes are controller using, for instance, Netconf; or when the routes are
signaled using BGP by the controller or a routing system, the order signaled using BGP by the controller or a routing system, the order
for a given instance can be sent in a new 'Consistent Hash Sort for a given instance can be sent in a new 'Consistent Hash Sort
Order' BGP Extended Community (defined in Section 9.2). Order' BGP Extended Community (defined in Section 9.2).
The effect of rehashing when SF instances are added or removed can be The effect of rehashing when SF instances are added or removed can be
minimized, or even eliminated using variations of the technique of minimized, or even eliminated using variations of the technique of
consistent hashing [consistent-hash]. Details are outside the scope consistent hashing [consistent-hash]. Details are outside the scope
of this document. of this document.
3.4.3 ECMP with Flow Table 3.4.3. ECMP with Flow Table
A second refinement that can ensure forward/reverse flow consistency, A second refinement that can ensure forward/reverse flow consistency,
and also provides stability when the number of SF instances changes and also provides stability when the number of SF instances changes
('flow-stickiness'), is the use of dynamically configured IP flow ('flow-stickiness'), is the use of dynamically configured IP flow
tables in the VRFs. In this technique, flow tables are used to ensure tables in the VRFs. In this technique, flow tables are used to
that existing flows are unaffected if the number of ECMP routes ensure that existing flows are unaffected if the number of ECMP
changes, and that forward and reverse traffic passes through the same routes changes, and that forward and reverse traffic passes through
SF instance in each set of SF instances implementing a service the same SF instance in each set of SF instances implementing a
function. service function.
The flow tables are set up as follows: The flow tables are set up as follows:
1. User traffic with a new 5-tuple enters an egress VRF from a 1. User traffic with a new 5-tuple enters an egress VRF from a
connected SF instance. connected SF instance.
2. The VRF calculates the ECMP hash across available routes 2. The VRF calculates the ECMP hash across available routes (i.e.,
(i.e., ECMP group) to the ingress interfaces of the SF ECMP group) to the ingress interfaces of the SF instances in the
instances in the next SF instance set. The consistent hash next SF instance set. The consistent hash technique described in
technique described in section 3.4.2 must be used here and section 3.4.2 must be used here and in subsequent steps.
in subsequent steps.
3. The VRF creates a new flow entry for the 5-tuple of the new 3. The VRF creates a new flow entry for the 5-tuple of the new
traffic with the next-hop being the chosen downstream ECMP traffic with the next-hop being the chosen downstream ECMP group
group member (determined in the step 2. above) . All member (determined in the step 2. above). All subsequent packets
subsequent packets for the same flow will be forwarded using for the same flow will be forwarded using flow lookup and, hence,
flow lookup and, hence, will use the same next-hop. will use the same next-hop.
4. The encapsulated packet arrives in the routing system that 4. The encapsulated packet arrives in the routing system that hosts
hosts the ingress VRF for the selected SF instance. the ingress VRF for the selected SF instance.
5. The ingress VRF of the next service instance determines if 5. The ingress VRF of the next service instance determines if the
the packet came from a routing system that is in an ECMP packet came from a routing system that is in an ECMP group in the
group in the reverse direction(i.e., from this ingress VRF reverse direction(i.e., from this ingress VRF back to the
back to the previous set of SF instances). previous set of SF instances).
6. If an ECMP group is found, the ingress VRF creates a flow 6. If an ECMP group is found, the ingress VRF creates a flow entry
entry for the reversed 5-tuple with next-hop of the tunnel for the reversed 5-tuple with next-hop of the tunnel on which
on which traffic arrived. This is for the traffic in the traffic arrived. This is for the traffic in the reverse
reverse direction. direction.
7. If multiple SF instances are connected to the ingress VRF, 7. If multiple SF instances are connected to the ingress VRF, the
the ECMP consistent hash is used to choose which one to send ECMP consistent hash is used to choose which one to send the
the traffic into. traffic into.
8. A forward flow table entry is created for the traffic's 5- 8. A forward flow table entry is created for the traffic's 5-tuple
tuple with next hop of the interface of the SF instance with next hop of the interface of the SF instance chosen in the
chosen in the previous step. previous step.
9. The packet is sent into the selected SF instance. 9. The packet is sent into the selected SF instance.
The above method ensures that forward and reverse flows pass through The above method ensures that forward and reverse flows pass through
the same SF instances, and that if the number of ECMP routes changes the same SF instances, and that if the number of ECMP routes changes
when SF instances are added or removed, all existing flows will when SF instances are added or removed, all existing flows will
continue to flow through the same SF instances, but new flows will continue to flow through the same SF instances, but new flows will
use the new ECMP hash. The only flows affected will be those that use the new ECMP hash. The only flows affected will be those that
were passing through an SF instance that was removed, and those will were passing through an SF instance that was removed, and those will
be spread among the remaining SF instances using the updated ECMP be spread among the remaining SF instances using the updated ECMP
hash. hash.
If the consistent hash algorithm is used in both directions, then If the consistent hash algorithm is used in both directions, then
only the forwarding flow entries would be required, and would be only the forwarding flow entries would be required, and would be
built independently in each direction. If distinct VPN routes with built independently in each direction. If distinct VPN routes with
next-hop forwarding labels are used, then only the flow table in step next-hop forwarding labels are used, then only the flow table in step
3 is sufficient to provide flow stickiness. 3 is sufficient to provide flow stickiness.
3.4.4 Dealing with different hash algorithms in an SFC 3.4.4. Dealing with Different Hash Algorithms in an SFC
In some cases, there will be two or more hash algorithms in In some cases, there will be two or more hash algorithms in
forwarders along an SFC. E.g. when a physical router is at the entry forwarders along an SFC. E.g. when a physical router is at the entry
and exit of the chain, and virtual forwarders are used within the and exit of the chain, and virtual forwarders are used within the
chain. Forward and reverse flows will mostly not pass through the chain. Forward and reverse flows will mostly not pass through the
same SF instances of the first SF, and the SFC will not operate as same SF instances of the first SF, and the SFC will not operate as
intended if the first SF is stateful. It may be impractical, or intended if the first SF is stateful. It may be impractical, or
prohibitively expensive to implement the flow table-based methods prohibitively expensive to implement the flow table-based methods
described above to achieve flow stability and symmetry. This issue described above to achieve flow stability and symmetry. This issue
can be mitigated by ensuring that the first SF is not stateful, or by can be mitigated by ensuring that the first SF is not stateful, or by
placing a null SF between the physical router and the first actual SF placing a null SF between the physical router and the first actual SF
in the SFC. This ensures that the hash method on both sides of in the SFC. This ensures that the hash method on both sides of
stateful service instances is the same, and the SFC will operate with stateful service instances is the same, and the SFC will operate with
flow stability and symmetry if the methods described above are flow stability and symmetry if the methods described above are
employed. employed.
4 Steering into SFCs Using a Classifier 4. Steering into SFCs Using a Classifier
In many applications of SFCs, a classifier will be used to direct In many applications of SFCs, a classifier will be used to direct
traffic into SFCs. The classifier inspects the first or first few traffic into SFCs. The classifier inspects the first or first few
packets in a flow to determine which SFC the flow should be sent packets in a flow to determine which SFC the flow should be sent
into. The decision criteria can be based on just the IP 5-tuple of into. The decision criteria can be based on just the IP 5-tuple of
the header (i.e filter-based forwarding), or could involve analysis the header (i.e filter-based forwarding), or could involve analysis
of the payload of packets using deep packet inspection. Integration of the payload of packets using deep packet inspection. Integration
with a subscriber management system such as PCRF or AAA may be with a subscriber management system such as PCRF or AAA may be
required in order to identify which SFC to send traffic to based on required in order to identify which SFC to send traffic to based on
subscriber policy. subscriber policy.
An example logical architecture is shown in Figure 9, below where a An example logical architecture is shown in Figure 9, below where a
classifier is external to a physical router that is hosting the VRFs classifier is external to a physical router that is hosting the VRFs
that form the ends of two SFC sets. In the case of filter-based that form the ends of two SFC sets. In the case of filter-based
forwarding, classification could occur in a VRF on the router. forwarding, classification could occur in a VRF on the router.
+----------+ +----------+
| PCRF/AAA | | PCRF/AAA |
+-----+----+ +-----+----+
: :
: :
Subscriber +-----+------+ Subscriber +-----+------+
Traffic----->| Classifier | Traffic----->| Classifier |
+------------+ +------------+
| | | |
+-------|---|------------------------+ +-------|---|------------------------+
| | | Router | | | | Router |
| | | | | | | |
| O O X--------->Internet | O O X--------->Internet
| | | / \ | | | | / \ |
| | | O O | | | | O O |
+-------|---|----------------|---|---+ +-------|---|----------------|---|---+
| | +---+ +---+ | | | | +---+ +---+ | |
| +--+ U +---+ V +-+ | | +--+ U +---+ V +-+ |
| +---+ +---+ | | +---+ +---+ |
| | | |
| +---+ +---+ +---+ | | +---+ +---+ +---+ |
+--+ X +---+ Y +---+ Z +-+ +--+ X +---+ Y +---+ Z +-+
+---+ +---+ +---+ +---+ +---+ +---+
Figure 9 - Subscriber/Application-Aware Steering with a Classifier Subscriber/Application-Aware Steering with a Classifier
Figure 9
In the diagram, the classifier receives subscriber traffic and sends In the diagram, the classifier receives subscriber traffic and sends
the traffic out of one of two logical interfaces, depending on the traffic out of one of two logical interfaces, depending on
classification criteria. The logical interfaces of the classifier are classification criteria. The logical interfaces of the classifier
connected to VRFs in a router that are entries to two SFCs (shown as are connected to VRFs in a router that are entries to two SFCs (shown
O in the diagram). as O in the diagram).
In this scenario, the entry VRF for each chain does not advertise the In this scenario, the entry VRF for each chain does not advertise the
destination network prefixes and the modified method of setting destination network prefixes and the modified method of setting
prefixes, described in Section 2.8.2 can be employed. Also, the exit prefixes, described in Section 2.8.2 can be employed. Also, the exit
VRF for each SFC does not peer with a gateway or proxy node in the VRF for each SFC does not peer with a gateway or proxy node in the
destination network and packets are forwarded using IP lookup in the destination network and packets are forwarded using IP lookup in the
main routing table or in a VRF that the exit traffic from the SFCs is main routing table or in a VRF that the exit traffic from the SFCs is
directed into (shown as X in the diagram). A flow table may be directed into (shown as X in the diagram). A flow table may be
required to ensure that reverse traffic is sent into the correct SFC. required to ensure that reverse traffic is sent into the correct SFC.
An alternative would be where the classifier is itself a distributed, An alternative would be where the classifier is itself a distributed,
virtualized service function, but with multiple egress interfaces. In virtualized service function, but with multiple egress interfaces.
that case, each virtual classifier instance could be attached to a set In that case, each virtual classifier instance could be attached to a
of VRFs that connect to different SFCs. Each chain entry VRF would set of VRFs that connect to different SFCs. Each chain
load balance across the first SF instance set in its SFC. The reverse entry VRF would load balance across the first SF instance set in its
flow table mechanism described in Section 3.4.3 could be employed to SFC. The reverse flow table mechanism described in Section 3.4.3
ensure that flows return to the originating classifier instance which could be employed to ensure that flows return to the originating
may maintain subscriber context and perform charging and accounting. classifier instance which may maintain subscriber context and perform
charging and accounting.
5 External Domain Co-ordination 5. External Domain Co-ordination
It is likely that SFCs will be managed as a separate administrative It is likely that SFCs will be managed as a separate administrative
domain from the networks that they receive traffic from, and send domain from the networks that they receive traffic from, and send
traffic to. If the connected networks use BGP for route distribution, traffic to. If the connected networks use BGP for route
the controller in the SFC domain can join the network domains by distribution, the controller in the SFC domain can join the network
creating BGP peering sessions with routing systems or route domains by creating BGP peering sessions with routing systems or
reflectors in those network domains to exchange VPN routes, or with route reflectors in those network domains to exchange VPN routes, or
local border routers that peer with the external domains. While a with local border routers that peer with the external domains. While
controller can modify route targets for the VRFs within its SFC a controller can modify route targets for the VRFs within its SFC
domain, it is likely to not have any control over the external domain, it is likely to not have any control over the external
networks with which it is peering. Hence, the design does not assume networks with which it is peering. Hence, the design does not assume
that the RTs of external network domains can be modified by the that the RTs of external network domains can be modified by the
controller. It may however learn those RTs and use them in it's controller. It may however learn those RTs and use them in it's
modified route advertisements. modified route advertisements.
In order to steer traffic from external network domains into an SFC, In order to steer traffic from external network domains into an SFC,
the controller will advertise a destination network's prefixes into the controller will advertise a destination network's prefixes into
the peering source network domain with a BGP next-hop and label the peering source network domain with a BGP next-hop and label
associated with the SFC entry point that may be on a routing system associated with the SFC entry point that may be on a routing system
attached to the first SF instance. This advertisement may be over attached to the first SF instance. This advertisement may be over
regular MP-BGP/VPN peering which assumes existing standard VPN regular MP-BGP/VPN peering which assumes existing standard VPN
routing/forwarding behavior on the network domain's routers routing/forwarding behavior on the network domain's routers (PEs/
(PEs/ASBRs). The controller can learn routes to networks in external ASBRs). The controller can learn routes to networks in external
domains at the egress of an SFC and advertise routes to those network domains at the egress of an SFC and advertise routes to those network
into other external domains using the first ingress routing instance into other external domains using the first ingress routing instance
as the next hop thus allowing dynamic steering through re- as the next hop thus allowing dynamic steering through re-
origination of routes. origination of routes.
An operational benefit of this approach is that the SFC topology An operational benefit of this approach is that the SFC topology
within a domain need not be exposed to other domains. Additionally, within a domain need not be exposed to other domains. Additionally,
using non-specific routes inside an SFC, as described in Section using non-specific routes inside an SFC, as described in
2.8.1, means that new networks can be attached to a SFC without Section 2.8.1, means that new networks can be attached to a SFC
needing to configure prefixes inside the chain. without needing to configure prefixes inside the chain.
The controller will typically remove the destination network's RTs The controller will typically remove the destination network's RTs
and replace them with the RTs of the source network while advertising and replace them with the RTs of the source network while advertising
the modified routes. Alternatively, an external domain may be the modified routes. Alternatively, an external domain may be
provisioned with an additional export-only RT and an import- only RT provisioned with an additional export-only RT and an import- only RT
that the controller can use. that the controller can use.
6 Fine-grained steering using BGP Flow-Spec 6. Fine-grained steering using BGP Flow-Spec
When steering traffic from an external network domain into an SFC When steering traffic from an external network domain into an SFC
based on attributes of the packet flow, BGP Flow-spec can be used as based on attributes of the packet flow, BGP Flow-spec can be used as
a signaling option. a signaling option.
In this case, the controller can advertise one or more flow-spec In this case, the controller can advertise one or more flow-spec
routes into the entry VRF with the appropriate Service-topology-RT routes into the entry VRF with the appropriate Service-topology-RT
for the SFC. Alternatively, it can use the procedures described in for the SFC. Alternatively, it can use the procedures described in
RFC5575 or [flowspec-redirect-ip] on the gateway router to redirect [RFC5575] or [flowspec-redirect-ip] on the gateway router to redirect
traffic towards the first SF. traffic towards the first SF.
If it is desired to steer specific flows from a network domain's If it is desired to steer specific flows from a network domain's
existing routers, the controller can advertise the above flow-spec existing routers, the controller can advertise the above flow-spec
routes to the network domain's border routers or route reflectors. routes to the network domain's border routers or route reflectors.
7 Controller Federation 7. Controller Federation
When SFCs are distributed geographically, or in very large-scale When SFCs are distributed geographically, or in very large-scale
environments, there may be multiple SFC controllers present and they environments, there may be multiple SFC controllers present and they
may variously employ both of the SFC creation methods described in may variously employ both of the SFC creation methods described in
Section 2.6. If there is a requirement for SFCs to span controller Section 2.6. If there is a requirement for SFCs to span controller
domains there may be a requirement to exchange information between domains there may be a requirement to exchange information between
controllers. Again, a BGP session between controllers can be used to controllers. Again, a BGP session between controllers can be used to
exchange route information as described in the previous sections and exchange route information as described in the previous sections and
allow such domain spanning SFCs to be created. allow such domain spanning SFCs to be created.
8 Coordination Between SF Instances and Controller using BGP 8. Coordination Between SF Instances and Controller using BGP
In many cases, the configuration of SF instance determines its In many cases, the configuration of SF instance determines its
network behavior. E.g. when NAT pools are set up, or when an SSL network behavior. E.g. when NAT pools are set up, or when an SSL
gateway is configured with a set of enterprise IP addresses to use. gateway is configured with a set of enterprise IP addresses to use.
In these cases, the addresses that will be used by the SFs need to be In these cases, the addresses that will be used by the SFs need to be
known in the networks connecting to them in order that traffic can be known in the networks connecting to them in order that traffic can be
properly routed. When SFCs are involved, this means that the properly routed. When SFCs are involved, this means that the
controller has to be notified when such configuration changes are controller has to be notified when such configuration changes are
made in SF instances. Sometimes, the changes will be made by end- made in SF instances. Sometimes, the changes will be made by end-
customers and it is desirable the controller adjust the SFC routing customers and it is desirable the controller adjust the SFC routing
configuration automatically when the change is made, and without configuration automatically when the change is made, and without
customers needing to notify the service provider via a portal, for customers needing to notify the service provider via a portal, for
instance, or requiring development of integration modules linking the instance, or requiring development of integration modules linking the
SF instances and the controller. SF instances and the controller.
One option for automatic notification for SFs that support BGP is for One option for automatic notification for SFs that support BGP is for
the connected forwarding system (physical or virtual SFF) to also the connected forwarding system (physical or virtual SFF) to also
support BGP, and for SF instances to be configured to peer with the support BGP, and for SF instances to be configured to peer with the
SFF. When changes are made to the configuration of a SF instance, SFF. When changes are made to the configuration of a SF instance,
that for example, the SF will accept packets from a particular that for example, the SF will accept packets from a particular
network prefix on one of its interfaces, the SF instance will send a network prefix on one of its interfaces, the SF instance will send a
BGP route update to the SFF it is connected to and which it has a BGP BGP route update to the SFF it is connected to and which it has a BGP
session with. The controller can then adjust the routes along SFCs to session with. The controller can then adjust the routes along SFCs
ensure that packets with destinations in the new prefix reach the to ensure that packets with destinations in the new prefix reach the
reconfigured SF instance. reconfigured SF instance.
BGP could also be used to signal from the controller to a SF instance BGP could also be used to signal from the controller to a SF instance
that certain traffic should be sent out from a particular interface. that certain traffic should be sent out from a particular interface.
This could be used to direct suspect traffic to a security scrubbing This could be used to direct suspect traffic to a security scrubbing
center,for example. center,for example.
Note that the SFF need not support a BGP stack itself; it can proxy Note that the SFF need not support a BGP stack itself; it can proxy
BGP messages to the controller which will support such a stack. BGP messages to the controller which will support such a stack.
9 BGP Extended Communities 9. BGP Extended Communities
9.1 Route-Target Record 9.1. Route-Target Record
Route-Target Record (RT Record) is defined as a transitive BGP Route-Target Record (RT-Record)is defined as a transitive BGP
Extended Community, that contains a Route-Target value representing Extended Community, that contains an Route-Target value representing
one of the RTs that the route has been attached with previously, and one of the RTs that the route has been attached with previously, and
which may no longer be attached to the route on subsequent re- which may no longer be attached to the route on subsequent re-
advertisements (see Section 2.8.5). advertisements (see Section 2.8.5).
A Sub-Type code 0x13 is assigned in the three BGP Extended Community A Sub-Type code 0x13 is assigned in the three BGP Extended Community
types - Two-Octet AS-Specific 0x00, IPv4-Address-Specific 0x01 and types - Two-Octet AS-Specific 0x00, IPv4-Address-Specific 0x01 and
Four-Octet AS-Specific 0x02. A Sub-Type code 0x0013 is also assigned Four-Octet AS-Specific 0x02. A Sub-Type code 0x0013 is also assigned
in the BGP Transitive IPv6 Address-Specific Extended Community. in the BGP Transitive IPv6 Address-Specific Extended Community.
The Extended Community is encoded as follows: The Extended Community is encoded as follows:
0 1 2 3 0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 0x00,0x01,0x02| Sub-Type=0x13 | Route-Target Value | | 0x00,0x01,0x02| Sub-Type=0x13 | Route-Target Value |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Route-Target Value contd. | | Route-Target Value contd. |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
The Type field of the BGP Route-Target Extended Community is The Type field of the BGP Route-Target Extended Community is copied
copied into the Type field of the RT Record Extended Community. into the Type field of the RT Record Extended Community.
The Value field (Global Administrator and Local Administrator) of The Value field (Global Administrator and Local Administrator) of the
the Route-Target Extended Community is copied into the Route- Route-Target Extended Community is copied into the Route-Target Value
Target Value field of the RT Record Extended Community. field of the RT Record Extended Community.
When comparing a RT-Record to a Route-Target, only the Type and When comparing a RT-Record to a Route-Target, only the Type and the
the Route-Target value fields are used in the comparison. The sub- Route-Target value fields are used in the comparison. The sub-type
type field is masked out. field is masked out.
When a speaker re-originates a route that contains one or more When a speaker re-originates a route that contains one or more RTs,
RTs, it must add each of these RTs as RT Record extended communities it must add each of these RTs as RT Record extended communities in
in the re-originated route. the re-originated route.
A speaker must not re-originate a route with an RT, if this RT is A speaker must not re-originate a route with an RT, if this RT is
already present as an RT Record extended community. already present as an RT Record extended community.
9.2 CONSISTENT_HASH_SORT_ORDER 9.2. Consistent Hash Sort Order
Consistent Hash Sort Order is an optional transitive Opaque BGP Consistent Hash Sort Order is an optional transitive Opaque BGP
Extended Community of sub-type 0x14, defined as follows: Extended Community of type 0x14, defined as follows:
Type Field : The value of the high-order octet is determined by Type Field : The value of the high-order octet is 0x03 (transitive
provisioning as per [RFC4360]. The value of the low- opaque). The value of the low-order octet is assigned
order octet is assigned as 0x14 by IANA from the as 0x14 by IANA from the Transitive Opaque Extended
Transitive Opaque Extended Community Sub-Types registry. Community Sub-Types registry.
Value Field : The value field contains a Sort Order sub-field that Value Field : The value field contains a Sort Order sub-field that
indicates the relative order of this route among the indicates the relative order of this route among the
ECMP set for the prefix, to be sorted in increasing ECMP set for the prefix, to be sorted in increasing
order. It is a 32-bit unsigned integer. The field is order. It is a 32-bit unsigned integer. The field is
encoded as shown below: encoded as shown below:
+------------------------------+ +------------------------------+
| Sort Order (4 octets) | | Sort Order (4 octets) |
+------------------------------+ +------------------------------+
| Reserved (2 octets) | | Reserved (2 octets) |
+------------------------------+ +------------------------------+
10 Summary and Conclusion 9.3. Load Balance Settings
Consistent Hash Sort Order is an optional transitive Opaque BGP
Extended Community of type 0x14, defined as follows:
Type Field : The value of the high-order octet is 0x03 (transitive
opaque). The value of the low-order octet is assigned
as 0xaa by IANA from the Transitive Opaque Extended
Community Sub-Types registry.
Value Field : The value field contains flags that indicate which
values in an IP packet's 5-tuple should be used as
inputs to the ECMP hash algorithm. The field is
encoded as shown below:
* 0 1 2 3
* 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
* +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
* | Type 0x03 | Sub-Type 0xaa |s d c p P R R R|R R R R R R R R|
* +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
* | Reserved |B R R R R R R R| Reserved | Reserved |
* +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
*
* Type: 0x03 Opaque
* SubType: 0xAA LoadBalance attribute information (TBA)
* s: Use l3_source_address ECMP Load-balancing
* d: Use l3_destination_address ECMP Load-balancing
* c: Use l4_protocol ECMP Load-balancing
* p: Use l4_source_port ECMP Load-balancing
* P: Use l4_destination_port ECMP Load-balancing
* B: Use source_bias (instead of ECMP load-balancing)
* R: Reserved
10. Summary and Conclusion
The architecture for service function chains described in this The architecture for service function chains described in this
document uses virtual networks implemented as overlays in order to document uses virtual networks implemented as overlays in order to
create service function chains. The virtual networks use standards- create service function chains. The virtual networks use standards-
based encapsulation tunneling, such as MPLS over GRE/UDP or VXLAN, to based encapsulation tunneling, such as MPLS over GRE/UDP or VXLAN, to
transport packets into an SFC and between service function instances transport packets into an SFC and between service function instances
without routing in the user address space. Two methods of installing without routing in the user address space. Two methods of installing
routes to form service chains are described. routes to form service chains are described.
In environments with physical routers, a controller may operate in In environments with physical routers, a controller may operate in
tandem with existing BGP route reflectors, and would contain the SFC tandem with existing BGP route reflectors, and would contain the SFC
topology model, and the ability to install the local static interface topology model, and the ability to install the local static interface
routes to SF instances. In a virtualized environment, the controller routes to SF instances. In a virtualized environment, the controller
can emulate route refection internally and simply install required can emulate route refection internally and simply install required
routes directly without advertisements occurring. routes directly without advertisements occurring.
11 Security Considerations 11. Security Considerations
The security considerations for SFCs are broadly similar to those The security considerations for SFCs are broadly similar to those
concerning the data, control and management planes of any device concerning the data, control and management planes of any device
placed in a network. Details are out of scope for this document. placed in a network. Details are out of scope for this document.
12 IANA Considerations
The new BGP Extended Communities in Section 9 are assigned types as
defined above in the IANA registry for extended communities.
13 Informative References
[NFVE2E] "Network Functions Virtualisation: End to End Architecture, 12. IANA Considerations
http://docbox.etsi.org/ISG/NFV/70-
DRAFT/0010/NFV-0010v016.zip".
[RFC2328] J. Moy, "OSPF Version 2", RFC 2328, April, 1998. The new BGP Extended Communities in are assigned types as defined
above in the IANA registry for extended communities.
[sfc-arch] Halpern, J. and Pignataro, C., "Service Function Chaining 13. Acknowledgments
(SFC) Architecture", RFC 7665, October 2015.
[RFC4364] Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private The authors would like to thank D. Daino, D.R. Lopez, D. Bernier,
Networks (VPNs)", RFC 4364, February 2006. W. Haeffner, A. Farrel, L. Fang, and N. So, for their
contributions to the earlier drafts. The authors would also like to
thank the following individuals for their review and feedback on the
original proposals: E. Rosen, J. Guchard, P. Quinn, P. Bosch, D.
Ward, A. Ganesan, N. Seth, G. Pildush and N. Bitar. The authors
also thank Wim Henderickx for his useful suggestions on several
aspects of the draft.
[RFC4271] Rekhter, Y., Li, T., and S. Hares, "A Border Gateway 14. References
Protocol 4 (BGP-4)", RFC 4271, January 2006.
[RFC4760] Bates, T., Chandra, R., Katz, D., and Y. Rekhter, 14.1. Normative References
"Multiprotocol Extensions for BGP-4", RFC 4760, January
2007.
[RFC7348] Mahalingam, M., et al. "VXLAN: A Framework for Overlaying [RFC3031] Rosen, E., Viswanathan, A., and R. Callon, "Multiprotocol
Virtualized Layer 2 Networks over Layer 3 Networks", RFC Label Switching Architecture", RFC 3031,
7348, August 2014. DOI 10.17487/RFC3031, January 2001,
<https://www.rfc-editor.org/info/rfc3031>.
[draft-ietf-idr-tunnel-encaps] [RFC4023] Worster, T., Rekhter, Y., and E. Rosen, Ed.,
Rosen, E, Ed et al. "The BGP Tunnel Encapsulation "Encapsulating MPLS in IP or Generic Routing Encapsulation
Attribute", draft-ietf-idr-tunnel-encaps-08, January 11, (GRE)", RFC 4023, DOI 10.17487/RFC4023, March 2005,
2018 <https://www.rfc-editor.org/info/rfc4023>.
[draft-ietf-l3vpn-end-system] Marques, P., et al., "BGP-signaled [RFC4271] Rekhter, Y., Ed., Li, T., Ed., and S. Hares, Ed., "A
end-system IP/VPNs", draft-ietf-l3vpn-end-system-06, Border Gateway Protocol 4 (BGP-4)", RFC 4271,
December 15, 2016. DOI 10.17487/RFC4271, January 2006,
<https://www.rfc-editor.org/info/rfc4271>.
[RFC5575] Marques, P., Sheth, N., Raszuk, R., et al., "Dissemination [RFC4360] Sangli, S., Tappan, D., and Y. Rekhter, "BGP Extended
of Flow Specification Rules", RFC 5575, Ausust 2009. Communities Attribute", RFC 4360, DOI 10.17487/RFC4360,
February 2006, <https://www.rfc-editor.org/info/rfc4360>.
[draft-ietf-bess-evpn-overlay-12] [RFC4364] Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private
Networks (VPNs)", RFC 4364, DOI 10.17487/RFC4364, February
2006, <https://www.rfc-editor.org/info/rfc4364>.
A. Sajassi, et al, "A Network Virtualization Overlay [RFC4760] Bates, T., Chandra, R., Katz, D., and Y. Rekhter,
Solution using EVPN", draft-ietf-bess-evpn-overlay, "Multiprotocol Extensions for BGP-4", RFC 4760,
February 2018. DOI 10.17487/RFC4760, January 2007,
<https://www.rfc-editor.org/info/rfc4760>.
[RFC 8300] [RFC5575] Marques, P., Sheth, N., Raszuk, R., Greene, B., Mauch, J.,
Quinn, P., et al, "Network Service Header", RFC 8300, and D. McPherson, "Dissemination of Flow Specification
November 2017. Rules", RFC 5575, DOI 10.17487/RFC5575, August 2009,
<https://www.rfc-editor.org/info/rfc5575>.
[draft-niu-sfc-mechanism] [RFC6241] Enns, R., Ed., Bjorklund, M., Ed., Schoenwaelder, J., Ed.,
Niu, L., Li, H., and Jiang, Y., "A Service Function and A. Bierman, Ed., "Network Configuration Protocol
Chaining Header and its Mechanism", draft-niu-sfc- (NETCONF)", RFC 6241, DOI 10.17487/RFC6241, June 2011,
mechanism-00, January 2014. <https://www.rfc-editor.org/info/rfc6241>.
[draft-rijsman-sfc-metadata-considerations] [RFC7348] Mahalingam, M., Dutt, D., Duda, K., Agarwal, P., Kreeger,
B. Rijsman, et al. "Metadata Considerations", draft- L., Sridhar, T., Bursell, M., and C. Wright, "Virtual
rijsman-sfc-metadata-considerations-00, February 12, 2014 eXtensible Local Area Network (VXLAN): A Framework for
Overlaying Virtualized Layer 2 Networks over Layer 3
Networks", RFC 7348, DOI 10.17487/RFC7348, August 2014,
<https://www.rfc-editor.org/info/rfc7348>.
[RFC6241] Enns, R., Bjorklund, M., Schoenwaelder, J., and A. Bierman, [RFC7510] Xu, X., Sheth, N., Yong, L., Callon, R., and D. Black,
"Network Configuration Protocol (NETCONF)", RFC 6241, June "Encapsulating MPLS in UDP", RFC 7510,
2011. DOI 10.17487/RFC7510, April 2015,
<https://www.rfc-editor.org/info/rfc7510>.
[RFC4023] Worster, T., Rekhter, Y., and E. Rosen, "Encapsulating MPLS [RFC7665] Halpern, J., Ed. and C. Pignataro, Ed., "Service Function
in IP or Generic Routing Encapsulation (GRE)", RFC 4023, Chaining (SFC) Architecture", RFC 7665,
March 2005. DOI 10.17487/RFC7665, October 2015,
<https://www.rfc-editor.org/info/rfc7665>.
[RFC7510] Xu, X., Sheth, N. et al, "Encapsulating MPLS in UDP", RFC [RFC8365] Sajassi, A., Ed., Drake, J., Ed., Bitar, N., Shekhar, R.,
7510, April 2015. Uttaro, J., and W. Henderickx, "A Network Virtualization
Overlay Solution Using Ethernet VPN (EVPN)", RFC 8365,
DOI 10.17487/RFC8365, March 2018,
<https://www.rfc-editor.org/info/rfc8365>.
[draft-ietf-i2rs-architecture] 14.2. Informational References
Atlas, A., Halpern, J., Hares, S., Ward, D., and T Nadeau,
"An Architecture for the Interface to the Routing System",
draft-ietf-i2rs-architecture, work in progress, March
2015.
[consistent-hash] [consistent-hash]
Karger, D.; Lehman, E.; Leighton, T.; Panigrahy, R.; Karger, D., Lehman, E., Leighton, T., Panigrahy, R.,
Levine, M.; Lewin, D. (1997). "Consistent Hashing and Levine, M., and D. Lewin, ""Consistent Hashing and Random
Random Trees: Distributed Caching Protocols for Relieving Trees: Distributed Caching Protocols for Relieving Hot
Hot Spots on the World Wide Web". Proceedings of the Spots on the World Wide Web"", 1997, <Proceedings of the
Twenty-ninth Annual ACM Symposium on Theory of Computing. Twenty-ninth Annual ACM Symposium on Theory of Computing.
ACM Press New York, NY, USA. pp. 654-663. ACM Press New York, NY, USA. pp. 654--663>.
[draft-ietf-idr-link-bandwidth] [draft-ietf-idr-link-bandwidth]
P. Mohapatra, R. Fernando, "BGP Link Bandwidth Extended Mohapatra, P. and R. Fernando, ""BGP Link Bandwidth
Community", draft-ietf-idr-link-bandwidth, work in Extended Community"", March 2018.
progress.
[flowspec-redirect-ip]
Uttaro, J. et al. "BGP Flow-Spec Redirect to IP Action",
draft-ietf-idr-flowspec-redirect-ip-02, February 2015.
14 Acknowledgments [draft-malhotra-bess-evpn-unequal-lb]
Malhotra, N., Sajassi, A., Rabadan, J., Drake, J.,
Lingala, A., and S. Thoria, ""Weighted Multi-Path
Procedures for EVPN All-Active Multi-Homing"", June 2018.
This document was prepared using 2-Word-v2.0.template.dot. [flowspec-redirect-ip]
Uttaro, J., Haas, J., Texier, M., Karch, A., Sreekanth,
A., Ray, S., Simpson, A., and W. Henderickx, ""BGP Flow-
Spec Redirect to IP Action"", February 2015.
This document is based on earlier drafts [draft-rfernando-bess- [idr-tunnel-encaps]
service-chaining] and [draft-mackie-sfc-using-virtual-networking]. Rosen, E., Patel, K., and G. van de Velde, ""The BGP
Tunnel Encapsulation Attribute"", February 2018.
The authors would like to thank D. Daino, D.R. Lopez, D. Bernier, W. [NFVE2E] ETSI, ""Network Functions Virtualisation (NFV):
Haeffner, A. Farrel, L. Fang, and N. So, for their contributions to Architectural Framework"", 2013.
the earlier drafts. The authors would also like to thank the
following individuals for their review and feedback on the original
proposals: E. Rosen, J. Guchard, P. Quinn, P. Bosch, D. Ward, A.
Ganesan, N. Seth, G. Pildush and N. Bitar. The authors also thank Wim
Henderickx for his useful suggestions on several aspects of the
Authors' Addresses Authors' Addresses
Rex Fernando Rex Fernando
Cisco Cisco Systems
170 W Tasman Drive 170 W. Tasman Drive
San Jose, CA 95134 San Jose, CA 95134
USA
Email: rex@cisco.com Email: rex@cisco.com
Stuart Mackie Stuart Mackie
Juniper Networks Juniper Networks
1133 Innovation Way 1133 Innovation Way
Sunnyvale, CA 94089 Sunnyvale, CA 94089
USA
Email: wsmackie@juniper.net Email: wsmackie@juniper.net
Dhananjaya Rao Dhananjaya Rao
Cisco Cisco Systems
170 W. Tasman Drive 170 W. Tasman Drive
San Jose, CA 95134 San Jose, CA 95134
USA
Email: dhrao@cisco.com Email: dhrao@cisco.com
Bruno Rijsman Bruno Rijsman
Juniper Networks
1133 Innovation Way
Sunnyvale, CA 94089
USA
Email: brijsman@juniper.net
Email: brunorijsman@gmail.com
Maria Napierala Maria Napierala
AT&T Labs ATT Labs
200 Laurel Avenue 200 Laurel Avenue
Middletown, NJ 07748 Middletown, NJ 07748
USA
Email: mnapierala@att.com Email: mnapierala@att.com
Thomas Morin Thomas Morin
Orange Orange
2, avenue Pierre Marzin 2, Avenue Pierre Marzin
Lannion 22307 Lannion, France 22307
France
Email: thomas.morin@orange.com Email: thomas.morin@orange.com
 End of changes. 299 change blocks. 
812 lines changed or deleted 860 lines changed or added

This html diff was produced by rfcdiff 1.47. The latest version is available from http://tools.ietf.org/tools/rfcdiff/