VXLAN Explained: Architecture, VTEPs, EVPN and Data Center Scaling

VXLAN Explained: Architecture, VTEPs, EVPN and Data Center Scaling

Modern data centers must support thousands of workloads, multiple tenants, distributed applications, and increasingly dynamic infrastructure. Traditional VLAN-based Layer 2 networks can become difficult to scale as the number of virtual networks and endpoints grows.

VXLAN (Virtual Extensible LAN) addresses these limitations by creating an overlay network that carries Layer 2 Ethernet traffic across a Layer 3 IP underlay. This separation allows the physical network to remain relatively simple while virtual networks scale independently on top of it.

VXLAN is especially powerful when combined with EVPN (Ethernet VPN) and MP-BGP, providing a control plane for distributing endpoint information and enabling highly scalable, resilient data center fabrics.

🌐 What Is VXLAN?

#

VXLAN is a network virtualization technology that encapsulates Ethernet frames inside UDP/IP packets. Instead of requiring the physical network to behave as one large Layer 2 domain, VXLAN allows Layer 2 segments to operate across a routed Layer 3 infrastructure.

The architecture consists of two primary components:

• VXLAN overlay: The logical network carrying tenant or application traffic.
• IP underlay: The physical Layer 3 network that transports VXLAN packets between tunnel endpoints.

This separation is fundamental to modern data center networking. The underlay focuses on IP reachability, while the overlay provides logical connectivity between workloads.

The VXLAN Network Identifier

#

VXLAN uses a 24-bit VXLAN Network Identifier (VNI). This provides up to approximately 16.7 million possible VNIs, compared with the roughly 4,000 usable VLAN IDs available in traditional IEEE 802.1Q networks.

The larger identifier space allows operators to create isolated logical networks for tenants, applications, environments, or other organizational boundaries without requiring a corresponding Layer 2 broadcast domain in the physical network.

VXLAN Tunnel Endpoints

#

VXLAN traffic enters and exits the overlay through VXLAN Tunnel Endpoints (VTEPs).

A VTEP performs two fundamental functions:

1. Encapsulates Ethernet frames into VXLAN/UDP packets when traffic enters the overlay.
2. Decapsulates VXLAN packets when traffic exits the overlay.

VTEPs are commonly implemented in data center switches, although they can also exist in software depending on the deployment architecture.

⚙️ How VXLAN Works

#

VXLAN separates the logical network from the physical topology by tunneling Ethernet frames across an IP network.

Encapsulation

#

Consider a virtual machine communicating with another workload located behind a different VTEP.

The process is approximately:

1. The source VM generates an Ethernet frame.
2. The local VTEP identifies the destination endpoint.
3. The VTEP adds a VXLAN header containing the appropriate VNI.
4. The VXLAN packet is encapsulated inside UDP/IP.
5. The IP underlay forwards the packet toward the remote VTEP.
6. The remote VTEP removes the outer headers.
7. The original Ethernet frame is forwarded toward its destination.

The physical network therefore does not need to understand the tenant’s Layer 2 topology. It only needs IP reachability between VTEPs.

VXLAN Packet Structure

#

A simplified VXLAN packet can be viewed as:

+-----------------------------+
| Outer Ethernet Header       |
+-----------------------------+
| Outer IP Header             |
+-----------------------------+
| UDP Header                  |
+-----------------------------+
| VXLAN Header                |
| VNI: 24 bits                |
+-----------------------------+
| Original Ethernet Frame     |
+-----------------------------+

VXLAN normally uses UDP as its transport, with UDP destination port 4789 assigned to VXLAN.

The outer IP addresses identify the source and destination VTEPs, while the VNI identifies the logical VXLAN segment.

Decapsulation

#

When the packet reaches the destination VTEP:

1. The VTEP receives the VXLAN/UDP packet.
2. It validates the tunnel and extracts the VNI.
3. The VXLAN and outer IP/UDP headers are removed.
4. The original Ethernet frame is forwarded to the destination endpoint.

The workload sees an ordinary Ethernet frame even though the traffic may have crossed multiple routed Layer 3 hops.

🧭 VXLAN Underlay and Overlay Architecture

#

A production VXLAN deployment is typically divided into an underlay and an overlay.

The IP Underlay

#

The underlay is the physical network that provides IP connectivity between VTEPs.

Common designs use routed leaf-spine fabrics with protocols such as:

• OSPF
• IS-IS
• eBGP
• ECMP

The underlay does not need to maintain tenant-specific MAC-address information. Its primary responsibility is reliable IP reachability.

The VXLAN Overlay

#

The overlay provides logical connectivity between workloads.

It contains concepts such as:

• VNIs
• VTEPs
• Tenant networks
• Virtual routing instances
• Endpoint information
• Overlay forwarding policies

This architectural separation makes it possible to change the physical topology without necessarily changing the logical network design.

🧩 VXLAN and EVPN

#

VXLAN defines the encapsulation mechanism, but large-scale deployments also require an efficient way to distribute endpoint information.

This is where EVPN becomes important.

EVPN (Ethernet VPN) commonly uses MP-BGP as its control plane to distribute information about MAC addresses, IP addresses, VNIs, and reachable endpoints.

Instead of relying exclusively on data-plane learning and flooding, VTEPs can learn remote endpoint information through the EVPN control plane.

Why MP-BGP EVPN Matters

#

A VXLAN EVPN fabric can provide:

• Distributed endpoint discovery.
• Reduced dependence on flooding.
• MAC and IP route distribution.
• Layer 2 and Layer 3 VPN services.
• Multi-tenant segmentation.
• ECMP-based forwarding.
• Improved convergence and operational visibility.

This combination is commonly referred to as VXLAN EVPN and has become a standard architecture for modern data center fabrics.

🚀 Advantages of VXLAN

#

VXLAN provides several important benefits compared with large traditional Layer 2 designs.

Massive Network Scalability

#

The 24-bit VNI space supports millions of logical network identifiers. This makes VXLAN suitable for cloud platforms, service providers, and large enterprise environments where VLAN space can become restrictive.

Layer 2 Connectivity Over Layer 3

#

VXLAN allows logical Layer 2 networks to cross routed infrastructure.

This means workloads can remain connected to the same logical segment even when their physical locations are separated by multiple Layer 3 hops.

Physical and Logical Topology Separation

#

The overlay is decoupled from the physical network.

Operators can build a standardized routed fabric underneath while allowing different applications and tenants to maintain independent logical networks above it.

ECMP and High Availability

#

VXLAN fabrics can use Equal-Cost Multi-Path (ECMP) routing in the underlay. Multiple paths between leaf and spine switches can therefore carry traffic simultaneously.

This improves:

• Link utilization.
• Bandwidth availability.
• Resilience.
• Failure recovery.
• Network scalability.

Automation and Programmability

#

Because VXLAN fabrics are commonly built around standardized Layer 3 underlays and BGP-based control planes, they integrate well with automation systems and infrastructure-as-code workflows.

This is particularly useful in environments where networks are frequently created, modified, and removed.

🏢 VXLAN Data Center Use Cases

#

VXLAN is used across a range of modern infrastructure architectures.

Virtual Machine Mobility

#

Virtual machines can move between hosts or racks while maintaining logical network connectivity.

VXLAN therefore helps separate workload mobility from the physical boundaries of the data center network.

Multi-Tenant Cloud Infrastructure

#

Cloud platforms can assign different VNIs to different tenants or application environments.

Traffic remains logically isolated while all tenants share the same physical switching fabric.

Application Segmentation

#

Organizations can create separate logical networks for:

• Production applications.
• Development environments.
• Database tiers.
• Management systems.
• Security zones.
• Individual application services.

This provides segmentation without requiring a dedicated physical network for every environment.

Data Center Interconnect

#

VXLAN-based architectures can also participate in data center interconnection designs. However, extending VXLAN between geographically separated facilities requires careful consideration of latency, failure domains, routing, MTU, control-plane design, and operational complexity.

Network Functions Virtualization

#

VXLAN can provide connectivity for virtualized network functions such as firewalls, load balancers, routers, and other software-based appliances.

The overlay allows these functions to be deployed flexibly without tying their logical connectivity to a particular physical switch or rack.

🔐 VXLAN and Network Security

#

VXLAN itself is primarily an encapsulation and virtualization technology; it should not be treated as an encryption mechanism.

VXLAN provides logical segmentation through VNIs, but confidentiality and integrity require additional technologies when traffic must be protected from inspection or modification across an untrusted transport.

Depending on the architecture, technologies such as IPsec can be used to secure traffic.

A secure VXLAN design should therefore consider multiple layers:

• VNI-based segmentation.
• Routing and forwarding policy.
• Access-control policies.
• Microsegmentation.
• Encryption where required.
• Identity and authentication.
• Monitoring and logging.

Segmentation and encryption solve different problems and should not be conflated.

📏 VXLAN MTU Considerations

#

VXLAN adds encapsulation overhead to the original Ethernet frame. The underlay therefore needs sufficient MTU capacity to carry the larger packet without unwanted fragmentation.

A simplified model is:

Original Ethernet frame
        +
VXLAN header
        +
UDP header
        +
Outer IP header
        +
Outer Ethernet overhead

For this reason, data center operators commonly configure jumbo-capable MTUs on the underlay and verify end-to-end MTU consistency.

An inconsistent MTU can produce difficult-to-diagnose problems such as dropped packets, failed application connections, or degraded performance.

🔬 VXLAN vs. Traditional VLANs

#

VXLAN does not necessarily replace VLANs everywhere. Instead, the technologies operate at different architectural layers.

The major architectural shift is that VXLAN allows operators to build scalable logical networks on top of a routed IP fabric instead of extending large Layer 2 domains throughout the physical topology.

🛠️ Operational Considerations

#

Deploying VXLAN successfully requires more than enabling VXLAN on a switch.

Underlay Reachability

#

Every VTEP must be able to reach the other VTEPs through the IP underlay.

Troubleshooting should therefore begin with basic Layer 3 connectivity before investigating overlay behavior.

Control-Plane Consistency

#

In EVPN deployments, BGP sessions, route advertisements, VNIs, and route-target policies must be consistent across the fabric.

A control-plane problem can prevent endpoints from becoming reachable even when the physical network itself is operational.

MTU Validation

#

The entire path between VTEPs must support the required packet size. Checking MTU configuration at every relevant hop is essential.

Endpoint Learning

#

Operators should verify whether endpoint information is being learned through the expected mechanism and whether MAC and IP bindings are correctly distributed.

Failure Testing

#

A resilient VXLAN fabric should be tested under realistic failure scenarios, including:

• Link failures.
• Spine failures.
• Leaf failures.
• VTEP failures.
• BGP session failures.
• Endpoint mobility.
• Network congestion.

The objective is not only to confirm connectivity, but also to verify convergence behavior and application impact.

🔮 Why VXLAN Remains Important

#

Modern data centers are moving toward architectures built around routed fabrics, distributed applications, virtualization, containers, and large-scale automation.

VXLAN fits this evolution because it separates logical network design from physical network topology.

The physical network can remain a predictable IP fabric, while the overlay provides the logical segmentation and connectivity required by applications and tenants. When combined with EVPN and MP-BGP, the architecture becomes suitable for highly scalable data center environments with large numbers of endpoints and virtual networks.

🚀 Conclusion

#

VXLAN is more than a way to extend Layer 2 across Layer 3. It is a foundational building block for scalable data center network virtualization.

By combining VXLAN encapsulation with routed IP underlays, VTEPs, ECMP, and increasingly EVPN/MP-BGP control planes, operators can build networks that scale beyond traditional VLAN boundaries while improving flexibility and resilience.

For modern data centers, the key architectural idea is simple:

Keep the underlay simple and routed; build scalable logical networks in the overlay.

That separation enables data centers to support cloud-scale multi-tenancy, workload mobility, distributed applications, and increasingly automated infrastructure without turning the physical network into a massive Layer 2 domain.