JustKernel

Objective: Get the maximum performance for VM to VM communication using Infiniband backend utilizing RDMA.

Design:
server1 running VM1—>Usermode app running on server1—–RDMA TRANSPORT—->App2 running on server2—>vm2 running on server 2.

brief description:
vm1 is executed on server1 using emulated virtio-net(frontend) adapter and vhost-user backend. PACKETH runs on vm1 and generates the broadcast packet. App1 running on server1 gets the packet and does RDMA transfer to app2 running on server2. app2 passes the packet to VM2.

A bit of detail:
vhost-user is a different kind of backend in a sense its main task is just to give a third party usermode app direct access to the tx and rx vrings. It sets up all the control tasks necessary to enable this direct access. like setup of MAP tables, mapping of guest physical, virtual memory etc.

vm1 is launched which uses vhost-net use backend and it shares the vring addresses with the app1. VRing addresses , MEMORY_TABLE setup, QUEUE setups is done at the initialization time as soon as you activate virtio_net driver in the guest. (insmod virtio_net.ko). Guest virtio_net driver initiates all this setup and vhost backend in QEMU facilitates it i.e it transfers the information to third party usermode app.

hw/net/vhost_net.c vhost_net_init() { vhost_dev_init()} —-> hw/virtio/vhost.c vhost_dev_init() { vhost_call(VHOST_SET_OWNER), vhost_call(VHOST_SET_FEATURES), vhost_call(VHOST_SET_ADDRESS) etc.) }—-> hw/virtio/vhost-backend.c vhost_user_call( vhost_user_send_fds (msg) // command and fds are sent to my third party app that is listening on a particular port.

When a packet reaches vm1 from PACKETH, its handled by the virtio-net guest drivers(frontend) and is put into the tx vring. Virtio-net driver then issues a kick to inform the backend (vhost-user) about the availability of the packet. During initialization Vhost-user backend driver and app1 has shared the fds that will be kicked. Thus, app1 comes to know about the about the availability of the packetand it does a direct RDMA transfer to app2 running on server2.

I am still working on route packet back from host i.e app2 running in server2 to vm2.

Tags: infiniband, qemu, rdma, vhost-user, virtio-net