JustKernel

Ray Of Hope

brain-dump: ballooning bug.

For past 3-4 days I have been working on a problem where after a series of ballooning operation involving in the range of 512 MiB to 30 GiB , the guest (Ubunut 16.04 ) crashes. Logs were: [ 193.432063] Freezing remaining freezable tasks … [ 198.032804] ata2.01: qc timeout (cmd 0xa1) [ 198.032815] ata2.01: failed […]

Posted in Uncategorized | No Comments

Random host crash when running Xen + Braindump pvMMU.

Kernel crash call trace Apr 6 02:50:06 localhost kernel: [ 3107.388117] [] dump_stack+0x19/0x20 Apr 6 02:50:06 localhost kernel: [ 3107.388119] [] warn_slowpath_common+0x70/0xa0 Apr 6 02:50:06 localhost kernel: [ 3107.388122] [] warn_slowpath_null+0x1a/0x20 Apr 6 02:50:06 localhost kernel: [ 3107.388125] [] xen_mc_flush+0x177/0x190 Apr 6 02:50:06 localhost kernel: [ 3107.388128] [] __xen_pgd_pin+0x23c/0x290 Apr 6 02:50:06 localhost kernel: [ […]

Posted in debugging, Xen | 2 Comments

Thinking perspective: debugging a crash

Many times its not possible to reproduce a crash that happened in customer environment, talking specifically of kernel or hypervisor or other low level crashes. So few things to keep in mind: 1) Analyze the log trying to find traces that can hint at what may be happening :Core dumps, debug traces 2) Thinking what […]

Posted in debugging | No Comments

Braindump + Segmentation and Paging And Xen

32 bit PV guest: guest kernel executes in ring 1 and guest user space executes in ring3. 64 bit PV guest: Guest kernel executes in ring 3 and guest user space also executes in ring 3. In 32 bit PV guest Xen used to use Segmentation based protection to protect Xen memory from being accessed […]

Posted in Linux Kernel, Xen | No Comments

Interrupt Path on Xen and on Baremetal system.

I was exploring an issue where qlogic device interrupt vector corresponding to a particular TX ring remained in masked state after the device comes up or is resetted. So the no packet transfer was possible on a port which was using that Tx queue and due to inaction, after some time this ports used to […]

Posted in Linux Kernel, virtualization, Xen | No Comments

GPU Passthrough under ballooning + XEN and role of POD driver.

Its happening only in case HVM guest. I used Win 7 64 bit of reproducing the crash. This article is in continuation with the previous one where I saw a crash in RMRR region in case of GPU passthrough. To work around this problem I introduced a memory whole of 2 GB in the guest […]

Posted in virtualization, Xen | No Comments

RMRR region GPU passthough + xen

Today faced with a issue where one VM is assigned GPU via passthrough mode and other VM doesn’t have GPU assignement. If I swap the max and min dynamic memory assignment of both these VMs (changing the size of balloon for each VM), guest with GPU crashes after 2-3 iterations. E820 Map of Host: (XEN) […]

Posted in virtualization, Xen | No Comments

multipage support for netback driver + Xen

Here are some of my learnings from my work on multipage support for netback driver. Learnings related to how we are making use of multi page and also how netback and netfron communicate. driver/xen/xen_probe.c keeps monitoring the backend and frontend states and calls the appropriate driver.otherend_changed(drivers/net/netfront or driver/net/netback) function. xenbus.c : static DEFINE_XENBUS_DRIVER(netback, , .probe […]

Posted in virtualization, Xen | No Comments

Broadwell + Xen + Classic guest kernel + WRITE_FAULT + SMAP Violation.

Recently faced an issue where classic guest kernel (2.6.*) were faulting on Broadwell hardware. From the day 1 doubt was on SMAP feature that is exclusive to Broadwell but had to prove it. Approach: Compiled the guest kernel with instrumentation but can’t use it as the guest on Broadwell hangs just during reboot. (I compiled […]

Posted in virtualization, Xen | No Comments

QEMU Networking Vhost-user & Transport Infiniband – series 4, recipient path

Finally got the receive path working from app2 running on server2 to VM2. Now I can see the rx stats in “ifconfig eth0” increasing for every packet transmitted to it. What was the problem: The problem was with the handling of the available ring and used ring. Its a rx case and I was treating […]

Posted in vhost-user, vhost-user, virtio-net | No Comments