Any kernel gearheads? irq affinity question?
Any kernel gearheads? irq affinity question?
https://forum.level1techs.com/t/any-kernel-gearheads-irq-affinity-question/154796/5
Operating Systems & Open Source Linux helpdesk vfio Mar 2020 Mar 2020 gordonthree Mar 2020
tl;dr: I’m having network performance issues on my KVM guests, possibly related to SRIOV / vfio / irq handling?
Seeing a lot of these in the logs, well, ever since I started using SRIOV for networking in my guests
[3587751.438487] irq 150: Affinity broken due to vector space exhaustion. [3639410.959080] irq 150: Affinity broken due to vector space exhaustion. [3639410.960207] irq 150: Affinity broken due to vector space exhaustion. [3639783.045026] irq 150: Affinity broken due to vector space exhaustion. [3639783.045639] irq 149: Affinity broken due to vector space exhaustion. [3639783.046873] irq 149: Affinity broken due to vector space exhaustion. [3639783.047425] irq 150: Affinity broken due to vector space exhaustion. [3639830.916299] irq 150: Affinity broken due to vector space exhaustion. [3639830.917404] irq 150: Affinity broken due to vector space exhaustion. [3639865.884657] irq 160: Affinity broken due to vector space exhaustion. [3639865.885711] irq 160: Affinity broken due to vector space exhaustion.
I’m not seeing much by way of a fix, or even explanation as to what’s going on. I tried to make sense of it using this very old (from 2.4) doc, but at this hour, getting lost pretty easy.
https://cs.uwaterloo.ca/~brecht/servers/apic/SMP-affinity.txt 31
The guests seem to “bog down” trying to do much over 20 megabytes/sec despite having a 10 gigabit connection. Heavy network traffic results in those log messages on the host. The IRQs referenced are assigned to the VFIO driver, with specific pci-id’s pointing at my network card’s virtual functions
Any thoughts? Mar 2020
How many virtual functions did you make?
It could also be unrelated to performance issues, is virtio networking an option?
Does irqbalance do anything for you?
(Background CPUs have hardware interrupts where literally they have a pin on the cpu reserved for such purpose, then they have software interrupts in total worth hardware and internal ones there can be max of 255 of them, it’s a limitation of the x86 architecture. With PCIe and modern apic there’s something called msi-x ; that makes interrupt processing slower, but each device gets up to 2048 possible interrupts values for itself. That means in theory a network card could have e.g. 1000 different queues to deliver packets from different ip/ports to different cores each with it’s own interrupt line).
Are you using q35 as the platform for the VM? gordonthree Mar 2020
7 VF, the card supports up to 63. Card is an Intel x520, using the stock Linux ixgbe driver.
Vitrio seems like a kludge compared to SR-IOV. I was happy to move away from using virtio and a host bridge.
Yes, q35 for the guests. Guests are pinned to specific CPU cores. I wonder if that’s part of the problem. I pinned CPUs based on which numa nodes own the GPUs, but the network card is on a different note if I recall correctly.
The host is a 2950x, with numa nodes enabled (?). risk Mar 2020
Hmm, there were a bunch of bugs around interrupt handling when q35-4.0 first came out, but that was a few months back and should be fixed by now. Maybe moving back and forth between different versions here helps.
Can you check if you have msi or msi-x in the guest lspci -vv output?
I don’t understand why it’s even bothering allocating so many legacy interrupts, it seems crazy - it shouldn’t be doing that at all.
I’m assuming both host and guest are running recent 5.4/5.5 mostly vanilla kernels? (for host / guest drivers) gordonthree Mar 2020
Host is Fedora 31, qemu 4.1.1, kernel 5.4.14-200.fc31.x86_64
/proc/interrupts 08:10 and 08:11 are network (this is on the host)
146: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 4468737-edge vfio-msix1 147: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 4468738-edge vfio-msix2 148: 0 6430 0 0 0 177092 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 4472832-edge vfio-msix0 149: 0 0 0 0 0 0 0 0 0 0 0 0 0 50969 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 35127296-edge vfio-msi0 150: 0 0 0 0 0 0 7472 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 186730 0 0 0 0 IR-PCI-MSI 4472833-edge vfio-msix1 151: 0 56317 65 0 0 8228215 122 0 274446 0 0 2306287 0 0 0 39029 0 0 10484193 0 6098138 401 137 0 45943 0 0 18940 0 4884697 0 0 IR-PCI-MSI 4460545-edge vfio-msix1 152: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 48 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 4460546-edge vfio-msix2 155: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 35655680-edge vfio-msi0 156: 0 0 0 0 0 0 4513 0 176208 351079 0 0 26485 0 0 0 0 0 135 0 0 0 0 152332 0 0 0 0 0 0 0 0 IR-PCI-MSI 4464640-edge vfio-msix0 157: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 4464641-edge vfio-msix1 158: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 4464642-edge vfio-msix2 159: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 35657728-edge vfio-msi0 160: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 84 0 4 0 0 0 0 0 0 IR-PCI-MSI 4472834-edge vfio-msix2
Can you check if you have msi or msi-x in the guest lspci -vv output?
From a guest running Ubuntu 18.04, kernel 5.3.0-40-generic
01:00.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
Subsystem: Intel Corporation 82599 Ethernet Controller Virtual Function
Physical Slot: 0
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
gordonthree Mar 2020
Upgraded the host to kernel 5.5, hasn’t helped these errors.