# $Id$ [@background] {tr small clock, a} {bl, {small, Luigi Rizzo, UNIPI - CoNEXT 2012}} {br, @@fl h5,img/unipi.jpg@@ @@fr h5,img/OpenLab_WebLogo.png@@ @@fr h5,img/change-logo-small.jpg@@ } [VALE] {title, VALE: a switched ethernet for virtual machines} {b author, Luigi Rizzo, Giuseppe Lettieri \\ Universita` di Pisa} \\ {author, [[http://info.iet.unipi.it/\~luigi/vale/]] } [Motivation] Make sw packet processing cope with ever increasing hw speeds - netmap -- fast packet I/O - QFQ -- high speed fair queueing scheduler - DXR -- fast IPv4 lookup Virtualization is the next step - consolidation and better use of hw resources - VM performance not too far from bare metal - still room for improvement in the network path [The VALE switch] - functionally equivalent to linux bridge (or simplified OVS) - implements the netmap API - over 18~Mpps for 64-byte frames, and over 70~Gbit/s with 1500-byte frames - available for Linux and FreeBSD as add-on kernel module - dynamic creation of switch instances and ports - qemu/kvm backend [The rest of the paper is structured as follows: ...] - Background - VALE architecture and performance - VM interaction - Conclusion and future work [Background] @@fr sz60,img/conext-fig-arch.png@@ {i,Guest OS}es run in an imaginary world - {b,hypervisor (VMM)} emulates a virtual machine - {b,frontend} emulates guest devices - {b,backend} talks to the actual physical hardware - {b,virtual switch} interconnects the virtual machines [Switch technologies and performance] Many possible options: - native bridge (linux, BSD, OVS) - UDP multicast encapsulation - Virtual Machine Device queue (VMDq), physical switch support - Proprietary solutions (VMware, Xen) 0.5..1~Mpps for sw-based switching, up to 10~Gbit/s for HW supported solutions, slightly more for proprietary switches [Netmap key ideas] {b,Efficient I/O API} - simplified packet format - reduced system call overhead (batching) {b,Standard system calls} - open(), ioctl() for configuration - ioctl(), poll() for send and receive - libpcap emulation - FreeBSD and Linux versions available {b,Performance matches HW speed} - 14.88~Mpps on 10~Gbit/s links at 0.9 GHz [Creating a Virtual Local Ethernet] VALE extends the netmap API to virtual interfaces @@ctr sz60,img/conext-fig-sw.png@@ - dynamically create switch instances and ports - ports named {tt,valeX:Y} where X is the switch instance, Y is the local port name - each switch runs the Learning Bridge algorithm [Operation] VALE operation is sender-driven: - each incoming packet is dispatched to the correct destination(s) - writes are non-blocking, packets dropped when input queue full - reads are blocking - the cost of forwarding/copying is charged to the sender. [Basic performance] A straightforward implementation (one packet at a time) reaches 2..5~Mpps - much better than native bridging - but can still be improved by a large factor - locking overhead and cache misses are the performance bottlenecks [Multi-stage forwarding] @@fr sz40,img/fwd2.png@@ Fetch batches of packets of {b,bounded} maximum size - prefetch payload, reducing cache miss penalties - control maximum latency Compute destinations for each packet in the batch - allow further batching during the next stage Forward traffic iterating on interfaces - one lock/unlock per interface {red b,18-20~Mpps with minimum-size frames} [VALE performance] Performance analysis of the VALE switch in various conditions: - packet size - number of ports - batch size (on sender, receiver and kernel side) Comparison is between - VALE, our switch - NIC, hardware-assisted switching (using PCI passthrough and NIC virtual queues, and/or external switch support) - TAP and linux bridging (OpenvSwitch shows similar performance) [Throughput versus number of replicas] Throughput decreases as 1/N - VALE and native bridging are CPU/memory bound; - NIC is mostly limited by bus speed @@ctr sz50,img/plot-rate-ports.png@@ [Processing time versus packet size] Packet processing time (lower is better) grows with packet size. - VALE operates at memory speed (80-100~Gbit/s) - TAP operates almost at memory speed, but has higher per-packet overhead - NIC operates at link/bus speed, hence limited to 10-15~Gbit/s @@ctr sz50,img/plot-rate-size.png@@ [Effect of batching] Batching occurs in three places - {b,transmit side}: controlled by the sender, can amortize syscall and locking costs during forwarding - {b,receive side}: controlled by the receiver, can amortize syscall costs - {b,kernel side}: {b green,controlled by the sysadmin}, guarantees bounded forwarding latency, can amortize locking costs during forwarding. The tradeoff is between efficiency and latency - the kernel batching makes sure we can control the forwarding delay (20..150~ns/pkt depending on the size) [Throughput versus receive batch] TX and kernel batch = 1024 packets, single output port - even short receive batches give high speed. @@ctr sz50,img/plot-var-rx-batch.png@@ [Throughput versus transmit batch] kernel batch = 1024, different results depending on rx batch - rx-batch = 1, throughput limited by the receive side - rx-batch = 1024, growth is slower than with variable rx batch because the receiver is faster than the sender @@ctr sz50,img/plot-var-tx-batch.png@@ [Throughput versus kernel batch] TX and RX batch = 1024 - growth is slower than with variable rx batch because even in this case the receiver is faster than the kernel @@ctr sz50,img/plot-var-br-batch.png@@ [Round trip latency] Round trip delay for request-response over VALE - these times are dominated by OS overheads - VALE is slightly better than native bridging @@ctr sz50,img/plot-ping-len.png@@ [VALE and VM performance] @@fr sz50,img/conext-fig-arch.png@@ The switch is no more a bottleneck in any condition, this helps identifying {i,other} bottlenecks: - source/sink (netsend or pkt-gen) - frontend (virtio or e1000) - hypervisor (QEMU or KVM) - backend (TAP or VALE) [VM performance summary] <pre style="color:red; margin-right: 2em; margin-left: 2em; font-size: 90%; background: #ddd;"> Speed, Kpps Configuration TAP VALE TX RX TX RX Raw switch speed 900 900 19700 19700 e1000, QEMU 18 14 23 23 e1000, KVM 20 20 24 24 fast e1000, KVM 200 200 virtio, KVM 1.1.1 370 300 1200 900 pkt-gen e1000 QEMU 490 490 2400 1850 pkt-gen e1000 KVM 550 550 3470 2550 pkt-gen e1000 KVM 500 490 2300 2000 (1514 bytes) vmxnet3 ESX 800 800 (VMware docs) </pre> [Other applications] VALE is not limited to VM interconnection. Other applications include: - performance testing (connecting traffic sources with analyzers, monitors, firewalls...) - support for building packet processing chains (with coarse processing modules) Example: - netmap enabled ipfw+dummynet has been developed and tested using VALE - measured 6.5~Mpps, over 5 times faster than in-kernel version [Conclusions and future work] VALE is extremely fast and can be used to interconnect VMs as well as native packet processing nodes - available for Linux and FreeBSD Future/related work - improve VMM network performance - replace the Learning Bridge algorithm with OVS forwarding code (joint work with EC project OpenLab) See [[http://info.iet.unipi.it/\~luigi/vale]] for code, papers and more details