We are all proud parents of our products as developers, much like our own children, we see them born, care and feed for them, watch them carefully as they are unstable during early years, we do not go out much, they become more stable over time, and then something happens -they grow up and there is a need to interact with others.  This could describe some of early customer experiences with first generation SDN LAN Emulation technologies.

Cisco Systems introduction and support of Multi-Protocol BGP eVPN control plane for VXLAN https://tools.ietf.org/html/draft-ietf-l2vpn-evpn-11 is an indication that the SDN industry is growing up, leveraging standards-track protocols, and enabling SDN to scale and interact with others.  This is far more significant to the SDN industry than one can read in a single press release and we will expand on its relevance in this blog.

Let's start with some basic understanding and a bit of SDN history.

SDN encapsulation into overlays or tunnels are not new technologies and have been supported for many years https://tools.ietf.org/html/rfc1701 describes GRE encapsulation and was written in 1994.  Anyone who uses a VPN also uses encapsulation such as IPSEC, so nothing new.  What is new are the SDN controller applications, how they enable logical network functions, and support centralized automation of the infrastructure for data center networks.  I will not go into all of the use cases for SDN overlays as you can find those readily by speaking to your vendor or searching for them on the web.

There are multiple controller architectures available for SDN. I will simply characterize them in three buckets and two additional qualifiers; OpenFlow, Integrated, and Decoupled are the three buckets;  SDN LAN Emulation and Policy-based are two qualifiers.  Today much of the confusion for customers is that vendors are still debating about, and attempting to monetize, "their" method of SDN.

There are key distinctions between the two qualifiers SDN LAN Emulation and Policy-based.  SDN LAN Emulation controllers reproduce properties of the layer 2 and layer 3 networks in the overlay including address learning and distribution, leveraging x86 servers to emulate LAN functions, where the overlay termination end points map logical network destinations to physical next hop in the overlay.

Policy-based controllers use fewer x86 servers by just mapping policy at the physical or virtual switch, benefitting from the integration of the overlay into vSwitches, merchant, and custom ASIC switches in an an open and cooperative manner which eliminates the need for LAN Emulation x86 components, providing more scale and far fewer components than SDN LAN Emulation models.

Five to six years ago the SDN industry started with controller applications providing a software function described with an ability to reproduce network functions from the physical network into a logical network, and overlay that logical network on top of the physical infrastructure.  I refer to this reproduction as SDN LAN Emulation as it has similarities to ATM LAN Emulation https://www.broadband-forum.org/ftp/pub/approved-specs/af-lane-0021.000.pdf .  

Similar to how virtualization evolved in compute, starting with software only, followed by Intel introducing VT and AMD introducing -V, for the purpose that virtualization worked better when it was cooperative with hardware.  In the early days of SDN LAN Emulation controllers, none of the overlay or gateway functions existed in hardware, but today 70 to 90 percent of the SDN LAN Emulation controller use cases exist in every merchant ASIC from Broadcom.

SDN controllers do three basic operations, they run the SDN application, described often as a distributed computing application, they expose a northbound API for orchestration, and they expose a southbound API for programming physical and virtual overlay termination end points.  The overlay termination end points are referred to here as VTEP's (VXLAN Tunnel End Point) for VXLAN, as it is the most common encapsulation and end point discussed for SDN today.

This is a basic, but fair characterization of SDN controllers, irrespective of whether they are coupled, decoupled, LAN Emulation, or Policy-based.  Integrated controllers provide the SDN controller running the application, the northbound API, the southbound API, and the VTEP.  Decoupled controllers do all of the items mentioned below, but they are meant to support the integration of separate components from third party vendors in each of the afore mentioned categories.

Examples of integrated controllers are VMware NSX and Cisco ACI.  In each of these implementations, the SDN controller application, the API's north and south, and either a physical or virtual VTEP is provided by the same vendor.

VMware NSX is an SDN LAN Emulation controller that integrates with the NSX vSwitch VTEP provided by VMware for vSphere. Today VMware has a Multi-hypervisor product that enables the NSX Multi-hypervisor controller with a VMware supplied version of Open vSwtich to speak with Xen and KVM hypervisors (you must get VMware's version of OVS).  VMware tightly controls the vSwitch API's for VTEP's in the kernel in vSphere, unlike that of RedHat, Xen Server, and Microsoft.  VMware leverages the informational RFC, OVSDB https://tools.ietf.org/html/rfc7047 to integrate with some vSwitches and third party hardware VTEP's.

Cisco Systems Application Centric Infrastructure (ACI) is a policy-based controller architecture with the Application Policy Infrastructure Controller or APIC, API's north and south, physical and virtual VTEP's.  Cisco works with open hypervisor vswitches such as OVS from Xen and KVM, Hyper-V, VMware VDS, VMware VSS, Cisco Systems Application Virtual Switch (AVS), and the Cisco Nexus 1000v, third party hardware VTEP vendors, virtual and physical layer 4 -7 appliance vendors, each integrating the OPFLEX control protocol (outside of VMware provided vSwitches) http://tools.ietf.org/id/draft-smith-opflex-01.txt for a southbound API and distributed control system leveraging a declarative policy model.  The northbound and southbound API's are fully published from Cisco with ACI.  Cisco provided VTEP's, both physical and virtual, also support or integrate directly with Multi-protocol BGP eVPN as a control plane for VXLAN.

Multi-protocol BGP eVPN as a Control Plane for VXLAN is a standards-track, distributed control plane offering a significant shift in customers ability to build and interconnect SDN overlay networks, while removing the need to run or configure multicast routing in the physical network.

A little more background is required to understand why Multi-protocol BGP eVPN as a control plane for VXLAN is so significant, so please bear with me a few more paragraphs, as the point is coming.

Various SDN controllers including VMware NSX, leverage the informational RFC OVSDB https://tools.ietf.org/html/rfc7047. OVSDB is a management protocol supporting programmability between an SDN controller a vSwitch or hardware VTEP, providing configuration such as termination of tunnels in an overlay network.  The OVSDB VTEP.5 schema is shown below:

VTEP5 Schema - http://openvswitch.org/docs/vtep.5.pdf Table - Purpose Global - Top-level configuration
Manager - OVSDB management connection
Physical_Switch - A physical switch
Physical_Port - A port within a physical switch
Logical_Binding_Stats - Statistics for a VLAN on a physical port bound to a logical network
Logical_Switch - A layer?2 domain
Ucast_Macs_Local - Unicast MACs (local)
Ucast_Macs_Remote - Unicast MACs (remote)
Mcast_Macs_Local - Multicast MACs (local)
Mcast_Macs_Remote - Multicast MACs (remote)
Logical_Router - A logical L3 router.
Physical_Locator_Set - Physical_Locator_Set configuration
Physical_Locator - Physical_Locator configuration

Looking at the table above you quickly realize this represents a limited set of options that will require more interaction between the SDN controller and the VTEP being configured leveraging OVSDB than what is defined in the spec.  There are multiple elements in this table, but the primary element is to carry layer 2 reachability information in the overlay and communicate that between the controllers and VTEP's.  An SDN LAN Emulation controller leveraging OVSDB is involved in address learning and distribution of addresses to the VTEP's.  This means that the data path is dependent upon the capacity of the x86 platforms running the controller software, it's ability to learn and distribute addresses to the VTEP's, and the VTEP's need to be tightly coupled to the OVSDB spec leveraging an imperative model.  Any feature must be conceptualized in the SDN LAN Emulation environment and then mapped to the data path at the VTEP's doing the forwarding or gateway functions.

This is a major friction point in large SDN installations because the controller dictates the feature velocity and scale, the VTEP features must be tightly aligned with this model, and any feature changes are limited by the development of this specification which is an informational draft. VTEP's are primarily ToR's and vSwitches.  Any other configuration or innovation must be controlled through vendor integration outside of the specification and coordinated across the platforms -features such as VTEP or gateway HA, link management, or others as an example. This is where marketing open moves to the reality of vendor dependence and integration.

Vendors exclusively supporting OVSDB as a management protocol and schema for third party hardware VTEP's and to integrate with vSwitches are limited by the scale and open or integration implications of this model.  Think back however, the basic function for OVSDB is to enable layer 2 reachability information in the overlay.

What happens if you want to extend your layer 2 and layer 3 information across a data center interconnect, to WAN routers, or across overlay networks that may have other SDN controllers, leveraging a standards-based protocol?

Enter eVPN MP-BGP EVPN control plane for VXLAN.

MP-BGP EVPN control plane for VXLAN offers the following key benefits: Leveraging an industry standards-track control protocol it enables multi-vendor interoperability and the following benefits:

• Control plane learning for end host Layer-2 and Layer-3 reachability information to build more robust and scalable VXLAN overlay networks.
• Leverages the decade-long MP-BGP VPN technology to support scalable multi-tenant VXLAN overlay networks.
• EVPN address family carries both Layer 2 and Layer 3 reachability information. This provides integrated bridging and routing in VXLAN overlay networks.
• Minimizes network flooding through protocol-driven host MAC/IP route distribution and ARP suppression on the local VTEPs.
• Provides optimal forwarding for east-west and north-south bound traffic with the distributed anycast function
• Provides VTEP peer discovery and authentication which mitigates the risk of rouge VTEPs in the VXLAN overlay network.

Now you no longer have to be limited to one controller, one vSwitch, and one SDN domain.  Leveraging MP-BGP EVPN control plane for VXLAN can create independent exchanges of layer 2 and layer 3 reachability information across overlays, VXLAN gateways, DC or WAN devices, and dramatically improves scale as MP-BGP EVPN control plane for VXLAN is a distributed to control plane not limited to the scale implications or the lock-in control and development of one schema.  Cisco Nexus 9000 with NXOS, Cisco ACI, and vSwtiches all integrate or directly support MP-BGP EVPN control plane for VXLAN and is an expansion to the open choices customers have for SDN from Cisco.

So what should you be asking from your vendors?  Every VTEP in your network should have the ability to integrate or support MP-BGP EVPN control plane for VXLAN, and it should be in every RFP. You should ensure each API is fully published without 3rd party vendor's being restricted from accessing or integrating with these API's -this includes vSwitches inside of the hypervisor, top of rack switches, and layer 4 -7 appliances.

In the transformation of traditional IT models to supporting DevOps and Cloud operations, vendor's willingness to cooperate varies over time. Leveraging standards-track protocols such as MP-BGP EVPN control plane for VXLAN and keeping the API's fully published, ensures the customer is no longer trapped by one vendor's implementation and the customer can drive their own integration or automation by calling the URI objects delivered through open and published RESTful API's.