Will try to explain the architecture/design behind host guest communication in VBox with an example of display.
You can dynamically configure virtual monitors (attached to guest OS) using SetVideoModeHint command from the host. This involves transfer of parameters from host to guest and vice versa. So, I will take this as the basis to explain the communication part.
Following layers are involved int he communication:
On HOST Side
1) App Layer
2) VMM driver (a virtual PCI device)
On Guest Side
1) VBoxClient on Linux nad VBoxTray on Windows – User mode app
2) VBoxGuest driver
Roles of each layer.
App layer on the HOST side: is responsible for sending the request to the guest to perform the desired action. In our case App layer represents SetVideoModeHint API.
VMM layer on the HOST: catches the request send from the app on the host side. Corresponding to request from the app, it issues an IRQ to the guest. It acts as a medium for communication between host and guest.
VMM layer is represented as a PCI device . The guest sends messages to the host using device I/O on the VMM device, and the host alerts the guest that it wants to send a message using IRQs, whereupon the guest queries the host for the message via device I/O .
The communication takes place via more-or-less fixed format messages (VMM device requests) which are more or less the bottom layer of the communication stack.
The host (VMMDev) sends VMM device event to the guest. It does that by sending an IRQ, and the guest’s IRQ handler (VBoxGuest driver) immediately queries the event code/number. It then notifies any guest process (kernel or user space- Guest side app, VBoxClient or VBoxTray) which is listening for that event number, and that process usually sends a device request to the host to retrieve more information relating to the request.
GUEST Side App: Acts as a listener and continuously polls the guest side VMM driver (VBoxGuest driver) for any events.
GUEST side VMM Driver (VBoxGuest Driver) – the guest VMMDev driver receives the IRQ it checks the event bitmask. and report events to the listeners. Listeners may request additional information about the event from the host using VMMDevReq_* requests.