BGP EVPN/VXLAN in the Data Center: A Practical Guide

Walk into almost any modern data center and you will find the same architectural pattern humming away beneath the racks: a leaf-spine fabric stitched together with VXLAN tunnels and steered by an EVPN control plane. It has quietly become the default way to build networks that carry enormous volumes of east-west traffic, isolate dozens or hundreds of tenants, and grow without disruptive forklift upgrades. Yet for many engineers the terminology still feels like alphabet soup — overlay, underlay, VTEP, route type 2, symmetric IRB.

This guide unpacks how BGP EVPN and VXLAN actually work together, why the industry moved on from older Layer 2 designs, and what the combination means in practice when it sits underneath a private cloud. The aim is not to drown you in protocol minutiae but to give you a mental model you can reason about and defend in a design review.

The problem EVPN/VXLAN was built to solve

Classic data center networks stretched VLANs across many switches and leaned on Spanning Tree Protocol to prevent loops. That worked when traffic was mostly north-south — clients talking to servers — but it aged badly. Spanning Tree disables links to break loops, so you pay for ports you cannot use. The VLAN namespace tops out at 4096 segments, which is nowhere near enough for a multi-tenant platform. And every switch in a Layer 2 domain has to learn every MAC address it might ever see, so the failure domain and the hardware tables both balloon as you scale.

The shift to virtualization and distributed applications made this worse. Suddenly the bulk of traffic was east-west: microservices, storage replication, live migration, and database clusters chatting constantly between racks. A topology optimized for north-south flows and capped at a few thousand segments simply could not keep up. EVPN/VXLAN is the answer the industry converged on, and it rests on one elegant idea — separate the network that moves packets from the network that decides where they go.

Underlay and overlay: two networks in one

The foundation is the underlay: a plain, routed IP network connecting every leaf and spine. There is no Spanning Tree here. Instead, links are routed and every leaf reaches every spine, so equal-cost multipath (ECMP) spreads traffic across all available uplinks at once. If a link or a spine fails, routing reconverges in well under a second and the remaining paths absorb the load. The underlay has exactly one job: deliver packets between switch loopback addresses quickly and reliably.

On top of that sits the overlay — a mesh of virtual tunnels that carry the actual tenant traffic. The devices that originate and terminate these tunnels are called VTEPs (VXLAN Tunnel Endpoints), and in a leaf-spine fabric the leaf switches play that role. To a virtual machine or container, the overlay looks like a flat Layer 2 segment, even though its packets are really being routed across an IP fabric underneath. This separation is what lets you scale the physical network and the logical networks independently.

What VXLAN does: the data plane

VXLAN is the encapsulation that builds the overlay. When a server sends a frame, the ingress VTEP wraps it in a UDP packet and adds a VXLAN header before shipping it across the underlay to the egress VTEP, which strips the wrapper and delivers the original frame. Because the outer packet is ordinary IP, the underlay does not need to know or care what is inside — it just forwards UDP.

The key field in that header is the 24-bit VXLAN Network Identifier, or VNI. Where VLANs gave you about 4,000 segments, the VNI gives you roughly 16 million. That headroom is precisely what a multi-tenant platform needs: each tenant network, each isolated environment, each project can get its own identifier without ever colliding. The trade-off is overhead — the encapsulation adds 50 bytes per packet — which is why MTU planning matters, a point we will return to.

What EVPN does: the control plane

VXLAN on its own has a weakness. Early deployments used flood-and-learn: to find an unknown MAC, a VTEP would flood the frame to every other VTEP and learn locations from the replies. That is noisy, slow, and scales poorly. EVPN fixes it by giving the overlay a real control plane built on Multiprotocol BGP (MP-BGP), using a dedicated address family for Layer 2 VPN EVPN routes.

Instead of flooding to discover endpoints, every VTEP advertises what it knows through BGP: the MAC addresses, the IP-to-MAC bindings, and the segments it serves. Other VTEPs install that information directly into their forwarding tables. The result is a network that behaves predictably, converges quickly, and suppresses most of the broadcast chatter that plagued older designs.

EVPN route types in plain terms

You do not need to memorize the full catalogue, but three route types carry most of the weight. Type 2 routes advertise individual MAC and IP addresses, so a VTEP learns exactly where a given host lives and can answer ARP requests locally instead of flooding them. Type 3 routes handle multicast and broadcast membership, telling each VTEP which peers belong to a segment so genuinely unknown traffic reaches only the right places. Type 5 routes advertise IP prefixes, which is how the fabric routes between subnets and stitches in external networks. Together they let the fabric do both bridging and routing intelligently, without ever resorting to brute-force flooding.

How a packet actually flows

Picture a VM on a server attached to leaf 1 sending traffic to a VM on leaf 9. The destination MAC and IP were already advertised to leaf 1 via an EVPN Type 2 route, so leaf 1 knows the target sits behind the VTEP on leaf 9. It encapsulates the frame in VXLAN, stamps it with the correct VNI, and sends it into the underlay addressed to leaf 9. ECMP hashing picks one of the spine uplinks; the spine, which only sees the outer IP header, routes it onward. Leaf 9 receives the packet, removes the VXLAN wrapper, and hands the original frame to the destination VM. No flooding, no Spanning Tree, no guesswork — just a routed lookup and a clean decapsulation.

Multi-tenancy and the private cloud angle

This is where the architecture earns its keep in a cloud platform. By mapping tenant networks to distinct VNIs and binding routing domains to VRFs, the fabric can host many customers or many projects on shared hardware while keeping their traffic genuinely isolated. Distributed routing — where every leaf acts as the default gateway for its locally attached segments — means traffic between subnets is routed at the first hop rather than tromboning back to a central router. East-west latency drops, and there is no single chokepoint to overwhelm.

For a private cloud, this maps almost one-to-one onto how the virtualization layer thinks about networking. Tenant overlays created by the cloud platform ride on the same encapsulation the physical fabric already speaks, so the logical networks tenants see and the physical paths their packets take stay cleanly decoupled. That is the foundation a leaf-spine fabric provides, and it is why EVPN/VXLAN has become the connective tissue of on-premise private clouds.

Operational realities worth planning for

The architecture is robust, but a few details separate a smooth deployment from a frustrating one. MTU is the classic trap: because VXLAN adds 50 bytes of overhead, the underlay must carry jumbo frames (typically 9216 bytes) so encapsulated traffic is never fragmented. Get this wrong and you will chase mysterious performance problems for days. ECMP hashing deserves attention too — you want flows spread evenly across uplinks, and entropy in the outer UDP source port is what makes that work. ARP suppression, enabled by EVPN Type 2 routes, dramatically cuts broadcast traffic but means your monitoring should watch the control plane rather than expecting to see floods. Finally, treat the underlay routing protocol as sacred: keep it simple, keep it fast, and let it do nothing but move packets between VTEPs.

Bringing it together

BGP EVPN and VXLAN succeed because they divide a hard problem into two manageable ones. VXLAN provides a scalable data plane with millions of segments and clean encapsulation; EVPN provides an intelligent control plane that distributes reachability without flooding. The result is a fabric that scales horizontally, isolates tenants cleanly, and recovers from failures in well under a second — exactly the properties a serious private cloud demands.

Designing, deploying, and operating such a fabric is still real engineering work, from leaf-spine cabling and switch selection to BGP policy and MTU hygiene. At clouditiv we build sovereign, OpenStack-based private clouds on Arista leaf-spine fabrics, where this overlay model underpins tenant isolation and the kind of predictable east-west performance modern workloads expect. When the network beneath the cloud is designed this way, the platform on top has room to grow — and that is ultimately what EVPN/VXLAN buys you.