Running DPDK C program without root privileges - dpdk

Question

I have C application that is using DPDK 19.11. Currently, the application is running with root permissions (using sudo command). In addition my application is running with huge pages (1GB).
Network devices using DPDK-compatible driver:
0000:02:00.0 'Ethernet Controller X710 for 10GbE backplane 1581'
drv=igb_uio unused=
I would like to run my application without the root permissions - get rid from "sudo" command.
I change permission for those files/folders:
/sys/class/uio/uio*/device/resource*
/sys/class/uio/uio*/device/config
/dev/uio*
/dev/hugepages/*
when I run my application without "sudo"- I run in a problems with rte_eal_init function. I got this error:
EAL: FATAL: Cannot use IOVA as 'PA' since physical addresses are not available
EAL: Cannot use IOVA as 'PA' since physical addresses are not available
My OS is Ubuntu 18.04, kernel 4.15.0-128-generic I noticed that at DPDK docs there is a remark about Running DPDK Applications Without Root Privileges - "since version 4.0, the kernel does not allow unprivileged processes to read the physical address information from the pagemaps file, making it impossible for those processes to be used by non-privileged users. In such cases, using the VFIO driver is recommended."
After reading comments I tried to use vfio-pci.
I load the module using:
sudo modprobe vfio-pci enable_unsafe_noiommu_mode=1
I also changed permission for /dev/hugepages/* and /dev/vfio/*
running with vfio-pci and sudo was successfully.
when running without sudo i got the same error:
EAL: FATAL: Cannot use IOVA as 'PA' since physical addresses are not available
EAL: Cannot use IOVA as 'PA' since physical addresses are not available
See also: https://doc.dpdk.org/guides/linux_gsg/enable_func.html#running-dpdk-applications-without-root-privileges
I wonder if someone has experience to run DPDK application without root with kernel 4.0 and above?
Also, as an alternative solution is to launch simple DPDK application with root privileges that will init DPDK. In parallel run another application without root privileges - this application will consume the packet and perform the business logic, is it possible?
thanks

Answer 1

First, it makes sense to check if you really need to use the unsafe mode with vfio-pci. Perhaps you just need to add intel_iommu=on iommu=pt to the kernel parameters for making the device work safely, i.e.:
modprobe vfio-pci
I haven't used the unsafe mode so far, perhaps the kernel even unconditionally disallows mappings for the vfio device, if unsafe mode is enabled, for (obvious?) security reasons.
For running a dpdk application without root privileges you need to adjust the permissions of the right vfio device. For example, when the permissions look like this
# ls -l /dev/vfio/
total 0
crw-------. 1 root root 235, 0 2021-08-21 15:13 17
crw-rw-rw-. 1 root root 10, 196 2021-08-21 15:13 vfio
then /dev/vfio/17 is the device you've bound for dpdk, thus adjust its permission like this:
chown juser /dev/vfio/17
A user process doesn't need extra permission for mapping huge pages. You don't even have to mount the hugetblfs, if you supply the --in-memory option to your dpdk program.
However, some hugepages must be reserved by root, e.g. during system boot. Example:
echo 4096 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
echo 8 > /sys/devices/system/node/node0/hugepages/hugepages-1048576kB/nr_hugepages
Besides permissions, the default resource limits likely are too low. Especially, the memlock ones. If it's too low kernel logs something like this when starting the dpdk application:
kernel: vfio_pin_pages_remote: RLIMIT_MEMLOCK (65536) exceeded
And the dpdk application prints:
EAL: cannot set up DMA remapping, error 12 (Cannot allocate memory)
EAL: 0000:05:00.1 DMA remapping failed, error 12 (Cannot allocate memory)
Increasing the limits fixes this issue, e.g.:
cat /etc/security/limits.d/24-memlock.conf
# memlock unit: KiB
juerr hard memlock 16777216
juser soft memlock 1048576

Answer 2

DPDK should detect if you should use IOVA VA or PA. Using the switch enable_unsafe_noiommu_mode=1 is telling DPDK that you have no iommu and that you will use IOVA PA.
The problem is, that running in PA mode requires root privileges as you need access to the physical address.
That dpdk.org document you cited should do the trick. I was able to get DPDK running without root privileges in 20.02 in a docker container. However, there was another problem with the software we were running on top of DPDK and its interaction with the hugepage backing.
In the end, we decided to still run DPDK as root, however, we limited the capabilities of the container to the bare minimum set needed to run DPDK.