Modern Data Center Networking using EVPN

November 20, 2025 / Balaji Bandi / 0 Comments

Ethernet VPN (EVPN) represents a fundamental shift in how modern data centers approach network architecture. By moving from reactive, flooding-based MAC learning to proactive, control-plane-driven endpoint discovery, EVPN solves critical scalability, availability, and operational challenges inherent in traditional VXLAN deployments. This guide provides a deep technical exploration of EVPN architecture, mechanisms, and operational benefits.

The Traditional VXLAN Approach

Traditional VXLAN networks rely on “flood-and-learn” behavior inherited from Ethernet switching principles. When a VTEP (VXLAN Tunnel Endpoint) needs to deliver a frame to a destination MAC address it doesn’t know, it floods the frame across all other VTEPs in the VXLAN segment, waiting for responses to learn the correct location.

The Process:

1. Host A sends an ARP request for Host B (unknown destination)
2. The ingress VTEP receives this ARP request with an unknown destination MAC
3. VTEP treats the packet as broadcast and floods it to all other VTEPs in the same VXLAN segment
4. Every other VTEP processes this broadcast, even if Host B is not connected to them
5. Only when a response arrives does the original VTEP learn that Host B exists on a specific VTEP
6. MAC learning occurs reactively, through data-plane traffic.

Why Flood-and-Learn Becomes Problematic

As data centers scale, flood-and-learn creates multiple critical issues:

Broadcast Storms and Bandwidth Waste

• Every ARP request, DHCP discovery, and unknown unicast frame floods the entire VXLAN segment
• In large data centers with thousands of virtual machines, this creates constant broadcast traffic
• Network bandwidth gets consumed by unnecessary flooding traffic rather than productive data transfers

CPU Overload on VTEPs

• Each VTEP must process every broadcast packet, even if it’s not relevant
• VTEP CPUs spend cycles examining and processing frames that don’t belong to connected hosts
• As hosts move or new services are deployed, repeated flooding events spike CPU utilization

Unpredictable Traffic Paths

• Traffic paths depend on which host responds first to flooding
• No deterministic routing behavior makes troubleshooting extremely difficult
• Network engineers cannot predict or control traffic flow patterns

Scalability Limitations

• Large data centers quickly exhaust VTEP CPU and memory capacity
• Networks become increasingly unstable as scale increases
• ARP storms become more frequent and severe

Host Mobility Issues

• When a host moves between VTEPs, it triggers new flooding events
• Modern cloud environments with high mobility suffer constant relearning cycles
• Convergence times are unpredictable and reactive

Introducing EVPN – Control Plane Intelligence

What is EVPN?

EVPN (Ethernet VPN) is an extension to BGP (Border Gateway Protocol) that fundamentally changes how VXLAN networks discover and communicate endpoint information. Instead of learning MAC and IP addresses reactively through data-plane flooding, EVPN uses the control plane to proactively advertise and distribute endpoint reachability information.

Core Innovation:

• Learning moves from the data plane (reactive) to the control plane (proactive)
• BGP becomes the signaling mechanism for VXLAN endpoint discovery
• All VTEPs know exactly where destinations are before traffic arrives

Key Architectural Concepts

BGP as the Control Plane
EVPN leverages BGP’s proven, scalable routing capabilities to distribute endpoint information. Each VTEP advertises the MAC addresses, IP addresses, and virtual network associations of locally connected hosts. BGP ensures all VTEPs receive this information consistently and efficiently.

Route Reflector Architecture
In large deployments, EVPN uses BGP route reflectors to simplify the control plane topology. Rather than requiring full-mesh BGP peering between all VTEPs, route reflectors collect advertisements from VTEPs and efficiently distribute them. This dramatically reduces control plane complexity and CPU overhead.

Standardized Route Types
EVPN defines specific BGP route types for different purposes, ensuring interoperability across vendors and consistent behavior across the fabric.

EVPN Route Types – The Foundation of Control Plane Communication

Understanding Route Types

EVPN uses different BGP route types to advertise different categories of information. Each route type serves a specific purpose in the network.

Type 2: MAC/IP Advertisement Routes

Purpose: Advertise MAC addresses and associated IP addresses of locally connected hosts.

What Gets Advertised:

• MAC address of the host
• IP address of the host (optional, for IP-based operations)
• Virtual Network Identifier (VXLAN VNI or VLAN)
• Router Distinguisher (RD) for uniqueness across the fabric

Practical Example:
When a server connects to VTEP 1, VTEP 1 learns the server’s MAC address and IP through local detection (ARP snooping or DHCP snooping). VTEP 1 creates a Type 2 route and advertises it via BGP:

• Route: MAC=00:11:22:33:44:55, IP=192.168.1.100, VNI=100
• All other VTEPs receive this advertisement and store it in their local MAC/IP tables
• Now, any VTEP sending traffic to this MAC or IP knows exactly where to send it

Why This Matters:
Before any traffic arrives, the entire fabric knows where this host is located. No flooding is needed.

Type 3: IMET (Inclusive Multicast Ethernet Tag) Routes

Purpose: Enable VTEP discovery and coordinate Broadcast, Unknown Unicast, and Multicast (BUM) traffic handling.

What Gets Advertised:

• VTEP IP address
• Indication that this VTEP participates in a specific VXLAN segment
• BUM traffic handling preferences

Practical Example:
When VTEP 1 is configured for VXLAN segment 100, it advertises a Type 3 route:

• Route: VTEP IP=10.0.0.1, VNI=100
• This tells all other VTEPs: “I am active in segment 100, send me BUM traffic for this segment”
• VTEPs use this information to build multicast groups or point-to-multipoint tunnels

Why This Matters:
Type 3 routes establish the foundation for controlled BUM traffic distribution without unlimited flooding.

Type 5: IP Prefix Routes

Purpose: Enable IP-based routing across different VXLAN segments (VNIs) or to external networks.

What Gets Advertised:

• IP prefix (e.g., 192.168.1.0/24)
• Gateway information
• Route target communities indicating which VNIs can reach this prefix

Practical Example:
A VTEP connected to both VNI 100 (database servers) and VNI 200 (application servers) advertises Type 5 routes to enable cross-subnet communication:

• Route: Prefix=192.168.2.0/24 (reachable from VNI 200) via VTEP 2
• Route: Prefix=192.168.1.0/24 (reachable from VNI 100) via VTEP 1
• Hosts in different subnets can now communicate, with routing handled at the VTEP level

Why This Matters:
Type 5 routes enable true Layer 3 connectivity within the VXLAN fabric without requiring separate routers.

Traffic Flow and ARP Suppression

How EVPN Changes Traffic Patterns

Before EVPN (Traditional VXLAN):

1. Host sends ARP request for unknown destination
2. VTEP floods ARP to all VTEPs
3. One VTEP responds with ARP reply
4. Original VTEP learns MAC location
5. Original traffic finally flows

With EVPN:

1. Both VTEPs have already learned MAC/IP information via BGP Type 2 routes
2. Ingress VTEP knows exactly which VTEP has the destination
3. Traffic goes directly to the correct VTEP
4. No flooding occurs

ARP Suppression: Stopping Flooding at the Source

ARP suppression is one of EVPN’s most powerful features for reducing broadcast traffic. Instead of allowing ARP requests to flood across the fabric, VTEPs intercept local ARP requests and respond locally using information learned from the control plane.

How ARP Suppression Works:

Step 1: Host Sends ARP Request

Host A: "Who has IP 192.168.1.50?"
(ARP request with destination MAC = ff:ff:ff:ff:ff:ff)

Step 2: VTEP Intercepts the Request
VTEP 1 receives the ARP request locally. Before flooding it, the VTEP checks its EVPN-learned MAC/IP table:

• Does this IP (192.168.1.50) exist in our database?
• Yes! MAC=00:aa:bb:cc:dd:ee is on VTEP 3 (learned from Type 2 route)

Step 3: VTEP Responds Locally
Instead of flooding, VTEP 1 creates and sends an ARP reply:

ARP Reply: IP 192.168.1.50 is at MAC 00:aa:bb:cc:dd:ee
(Response sent directly from VTEP 1 to Host A)

Step 4: Traffic Flows Directly
Host A receives the ARP reply and sends traffic to the MAC address. VTEP 1 has already learned from the BGP Type 2 route that this MAC exists on VTEP 3, so it tunnels the traffic directly.

Result: The ARP request never left the local VXLAN segment. No flooding across the fabric. Zero broadcast traffic.

Benefits of ARP Suppression

EVPN Multihoming – Redundancy Without Spanning Tree

The Problem with Traditional Redundancy

Traditional networks use Spanning Tree Protocol (STP) to prevent loops when hosts connect to multiple switches. However, STP has fundamental limitations:

• Only one active uplink per host (blocks other links)
• Convergence times are measured in tens of seconds
• Recalculation is CPU-intensive
• Unpredictable failover behavior

In virtual machine deployments with thousands of hosts, STP becomes a severe bottleneck.

EVPN Multihoming Solution

EVPN multihoming allows a single host (or group of hosts) to connect to multiple VTEPs simultaneously, with all links active and load-balanced. The solution uses three key components:

1. Ethernet Segment Identifier (ESI)

ESI is a unique identifier that groups multiple physical links connecting a host to the VXLAN fabric. It tells all VTEPs that multiple links belong to the same host.

Host 1 → VTEP 1 (Link 1)  ─┐
                           ├─ Assigned ESI = 00:11:22:33:44:55:00:00:00:01
Host 1 → VTEP 2 (Link 2)  ─┘

All links with the same ESI are treated as a single logical connection by the fabric.

2. BGP Signaling of ESI Information

Each VTEP advertises the ESIs it’s connected to via EVPN. This ensures all VTEPs in the fabric know which VTEPs have access to which hosts.

3. Split-Horizon Mechanism

When a VTEP receives traffic from a host over one ESI link, it prevents that traffic from being sent back out another ESI link to the same host (preventing loops).

All-Active Multihoming Mode

In All-Active mode, all links to a multihomed host carry traffic simultaneously.

How It Works:

1. Outbound Traffic (Host to Fabric):
   • Host can send traffic over Link 1 or Link 2 (or both via LAG)
   • Both links are equally active
   • Bandwidth utilization is maximized
2. Inbound Traffic (Fabric to Host):
   • Remote VTEPs learn the host is reachable via both VTEP 1 and VTEP 2
   • Traffic can arrive over either VTEP
   • Load balancing occurs naturally
3. VTEP Coordination:
   • When a packet arrives at VTEP 1, VTEP 1 forwards it directly to the host
   • VTEP 1 doesn’t send the packet to VTEP 2 (split horizon)
   • When traffic arrives at VTEP 2, it forwards directly to the host

Example Traffic Flow:

Remote Host (on VTEP 3) sends to Host 1 (multihomed):
- Packet 1 → VTEP 1 → Host 1 (Link 1)
- Packet 2 → VTEP 2 → Host 1 (Link 2)
- Packet 3 → VTEP 1 → Host 1 (Link 1)
- Load distributed across both VTEPs and links

Benefits of EVPN Multihoming

Failure Detection and Convergence

How EVPN Detects Failures

EVPN detects failures through multiple mechanisms, enabling rapid recovery:

VTEP Failure Detection

• BGP keepalives detect when a VTEP goes offline
• Failed VTEP stops advertising its routes
• Default convergence: 3 seconds (90-second BFD can reduce to milliseconds)

Link Failure Detection

• Physical link failures are detected immediately by the access layer
• VTEP updates its endpoint information
• New route advertisements are sent immediately

Host Mobility Detection

• Host sends gratuitous ARP or DHCP renewal from new VTEP
• New VTEP learns the host locally and advertises via Type 2 route
• Old VTEP’s route is withdrawn or superseded
• Convergence is sub-second

EVPN Convergence Process

Step 1: Failure Detection

Time = 0ms: VTEP 1 fails (link goes down)
BGP keepalive timer expires on connected VTEPs
All VTEPs detect loss of VTEP 1

Step 2: Route Withdrawal

Time = 50-100ms: Route reflector detects VTEP 1 failure
All Type 2 and Type 3 routes from VTEP 1 are withdrawn
Route updates are sent to remaining VTEPs

Step 3: Traffic Reroute

Time = 100-200ms: All VTEPs update their forwarding tables
Traffic destined for hosts on VTEP 1 is rerouted
If multihoming is configured, traffic uses alternate VTEP
Hosts experience brief pause (~100ms) but no packet loss

Step 4: Stabilization

Time = 200-500ms: Network reaches stable state
All hosts are reachable via surviving VTEPs
No additional flooding or relearning occurs

Comparison: Traditional vs EVPN Convergence

Scalability Benefits

Control Plane Scaling

Traditional VXLAN uses data-plane learning, which causes:

• Every host movement triggers network-wide learning events
• No way to prioritize or control learning
• MAC tables fill and age inconsistently
• Large networks experience instability

EVPN provides superior scalability:

Why EVPN Scales Better

1. Explicit Control: Every MAC/IP mapping is explicitly advertised once
2. No Implicit Learning: VTEPs don’t learn from traffic patterns
3. Deterministic: Same information is always advertised the same way
4. Efficient BGP: BGP is designed to handle millions of routes
5. Route Aggregation: Multiple hosts can be represented with fewer routes

Real-World Scalability Numbers

Data Center with 10,000 Virtual Machines:

• Traditional VXLAN:
  • ARP requests per hour: 50,000-100,000
  • Average flooding events: 500+ per minute during normal operation
  • VTEP CPU during convergence: 85-95%
  • Convergence time after changes: 30-120 seconds
• EVPN:
  • ARP suppression eliminates 95% of flooding
  • Learning events: Deterministic, tied to VM lifecycle (startup/shutdown)
  • VTEP CPU during changes: 20-30%
  • Convergence time: 100-500ms

Large Fabric Architecture

EVPN enables deployment of single-fabric architectures with thousands of VTEPs:

┌─────────────────────────────────────┐
│   BGP Route Reflectors (3-5)        │
│   Centralized Control Plane         │
└─────────────────────────────────────┘
              ↑    ↑    ↑
     ┌────────┴────┴────┴──────────┐
     ↓                              ↓
  VTEP 1                         VTEP N
  (Leaf)                         (Leaf)
  │                              │
  ├─ Servers/VMs                 ├─ Servers/VMs
  └─ vPC to VTEP 2               └─ vPC to VTEP 1

This architecture eliminates the need for large spanning trees and allows unlimited fabric growth.

Anycast Gateway and Distributed Routing

Traditional Gateway Architecture Problems

In traditional networks with multiple subnets:

• A single gateway IP is designated as default route
• All hosts send default-route traffic to that gateway
• Single point of failure
• Gateway becomes bottleneck for inter-subnet traffic

EVPN Anycast Gateway Solution

EVPN enables distributed routing by allowing multiple VTEPs to present the same gateway IP and MAC address.

How Anycast Gateway Works:

1. Same Gateway IP/MAC on Multiple VTEPs

VTEP 1: Gateway IP = 192.168.1.1, Gateway MAC = aa:aa:aa:aa:aa:aa
VTEP 2: Gateway IP = 192.168.1.1, Gateway MAC = aa:aa:aa:aa:aa:aa
VTEP 3: Gateway IP = 192.168.1.1, Gateway MAC = aa:aa:aa:aa:aa:aa

2. Hosts Use Nearest Gateway

Host 1 connected to VTEP 1 → Uses VTEP 1 gateway (local)
Host 2 connected to VTEP 2 → Uses VTEP 2 gateway (local)
Host 3 connected to VTEP 3 → Uses VTEP 3 gateway (local)

3. EVPN Coordinates Gateway Information

Each VTEP advertises: "I am 192.168.1.1 for hosts in VNI 100"
All VTEPs learn all gateway locations
Inter-subnet routing is deterministic and optimal

Benefits:

• Redundancy: No single gateway failure point
• Local Exit: Each host uses the nearest gateway
• Optimal Paths: Traffic doesn’t traverse unnecessary hops
• Stateless: Gateways are stateless; any can handle any traffic
• Multitenancy: Different VNIs can have different gateways with same IP

Operational Advantages

Deterministic Behavior

EVPN provides deterministic network behavior through control-plane visibility:

Before (Traditional VXLAN):

• Traffic paths are unpredictable (depend on learning order)
• Troubleshooting requires packet captures and guesswork
• Network behavior is reactive and event-driven

With EVPN:

• Every path is known before traffic arrives
• BGP can be queried to understand routing decisions
• Network behavior is predictable and verifiable

Practical Troubleshooting:

# Verify host reachability
show bgp evpn route-type 2  # See all known hosts
show bgp evpn route type 2 mac-address aa:bb:cc:dd:ee:ff

# Verify VTEP status
show bgp evpn summary  # See EVPN route reflector status
show bgp evpn route-type 3  # See which VTEPs are active

Visibility and Monitoring

EVPN provides unprecedented network visibility:

1. MAC/IP Table Accuracy
   • MAC tables are always accurate (learned from BGP, not guessed)
   • No stale entries or inconsistencies
   • One source of truth across fabric
2. Endpoint Tracking
   • Know exactly which VTEP hosts are connected to
   • Track host movements in real-time
   • Audit host connectivity for compliance
3. Convergence Monitoring
   • Monitor BGP route churn
   • Alert on abnormal movement patterns
   • Detect routing instabilities early

Deployment Considerations

Prerequisites for EVPN

Hardware Requirements:

• VTEPs must support EVPN (most modern switches do)
• BGP capability required
• Sufficient TCAM for control-plane routes
• Route reflectors for larger deployments

Network Design:

• Underlay network (physical network) must support BGP or static routing
• Underlay should be stable; EVPN adds minimal overhead
• Route reflector placement for redundancy
• Proper bandwidth provisioning for BGP convergence

Configuration Elements:

• VXLAN VNI planning
• Route Distinguisher (RD) assignment
• Route Target (RT) communities for segmentation
• BGP ASN and neighbor configuration
• EVPN address family under BGP

Implementation Strategy

Phase 1: Control Plane Setup

1. Deploy route reflectors
2. Configure BGP peering to VTEPs
3. Enable EVPN address family
4. Verify BGP convergence

Phase 2: VXLAN Configuration

1. Configure VXLAN VNIs on VTEPs
2. Map VLANs to VNIs
3. Configure EVPN route-targets
4. Verify Type 2 route advertisements

Phase 3: Advanced Features

1. Configure ARP suppression
2. Deploy Anycast gateways
3. Configure multihoming (ESI)
4. Enable BUM optimization

Conclusion

EVPN represents a fundamental architectural shift in data center networking. By moving MAC learning from reactive data-plane flooding to proactive control-plane signaling via BGP, EVPN addresses the critical scalability, reliability, and operational challenges of traditional VXLAN.

Key Takeaways:

• EVPN eliminates unnecessary flooding through ARP suppression
• Control-plane learning enables faster, more predictable convergence
• Multihoming without STP provides superior redundancy
• Distributed gateways enable optimal routing at scale
• Deterministic behavior simplifies operations and troubleshooting

Modern data centers scaling to thousands of servers depend on EVPN to provide stable, predictable, and efficient network service delivery.

Happy Labinggggggggggggggggggggggggggggg!