Understanding the iommu Linux grub File Configuration

https://enterprise-support.nvidia.com/s/article/understanding-the-iommu-linux-grub-file-configuration

May 28, 2022 This post discusses the iommu and intel_iommu Linux grub parameters for SR-IOV pass-through (pt) mode. When working in an SR-IOV environment, we need to make sure that kernel enables SR-IOV and that we get good performance.

To enable SR-IOV in the kernel, configure intel_iommu=on in the grub file. To get the best performance, add iommu=pt (pass-through) to the grub file when using SR-IOV. When in pass-through mode, the adapter does not need to use DMA translation to the memory, and this improves the performance. iommu=pt is needed mainly with hypervisor performance is needed.

Watch our free SR-IOV video tutorial and many other configuration tutorials for free on the Mellanox Academy

References Virtualisation Solutions https://www.kernel.org/doc/Documentation/Intel-IOMMU.txt

Setup and Prerequisites One Linux server with Virtualisation environment. You can follow one of the posts below per adapter type and technology used (Ethernet, InfiniBand).

HowTo Configure SR-IOV for ConnectX-3 with KVM (InfiniBand) HowTo Configure SR-IOV for ConnectX-3 with KVM (Ethernet) HowTo Configure SR-IOV for Connect-IB/ConnectX-4 with KVM (InfiniBand) HowTo Configure SR-IOV for ConnectX-4 with KVM (Ethernet)

grub file configuration Depending on the Linux distribution, you may use grub or grub2.

grub files are located by default at the following paths:

/boot/grub/grub.conf /boot/grub2/grub.cfg

grub file example:

Mellanox Technologies Multiboot GRUB Configuration default 0 timeout 10 #Current Os number: 01 title Ub15.10x86_64-4.2.0-42-generic root (hd0,0) kernel /vmlinuz-Ub15.10x86_64-4.2.0-42-generic root=/dev/sda2 console=tty0 console=ttyS0,115200n8 rhgb intel_iommu=on iommu=pt initrd /initrd.img-Ub15.10x86_64-4.2.0-42-generic

Verification

  1. After adding the parameters to the grub file, reboot the server.

  2. After reboot, check dmesg and check that your PCI device isn’t in the list of RMRR setting , if you still see it, you need to contact your server vendor(we saw the issue on HP Gen8).

[ 4.027939] DMAR: Setting RMRR: [ 4.089276] DMAR: Setting identity map for device 0000:02:00.0 [0xe8000 - 0xe8fff] [ 4.097709] DMAR: Setting identity map for device 0000:03:00.0 [0xe8000 - 0xe8fff] Refer to https://www.kernel.org/doc/Documentation/Intel-IOMMU.txt to learn more about DMAR and RMRR functions.

  1. Use the lspci command to confirm you have the list of PCI devices (VFs) as configured.

Updated: