I have nodes with an Infiniband connection and a centos 7.9 installed.
when I execute the following
# lspci | grep Mellanox
01:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]
# lspci -vv -s 01:00.0 | grep "Part number" -A 3
[PN] Part number: MCX353A-FCB
I will reinstall the system to rockylinux.
My question is: how to install the Mellanox hardware
Is it enough to use the provided package from the distribution (yum group install Infiniband) or should I also use the package provided by the manufacturers in the following link:
https://www.mellanox.com/products/infiniband-drivers/linux/mlnx_ofed
Related
I have a board with one ethernet interface (eth0) running Linux.
I'm trying to forward all incoming traffic from eth0 to my PMD driver, using dpdk-l2fwd example application.
Here is what I've tried:
./dpdk-l2fwd -c 0x3 --vdev={my_pmd}0 -- -p 0x3 -T 0
I can see that my rx_pkt_burst callback is polled by the application, but that's it.
How can I forward all incoming eth0 packets to my PMD?
I tried to use net_tap, using the following command:
./dpdk-l2fwd -c 0xff --vdev=net_tap0 --vdev={my_pmd}0 -- -p 0x7 -T 0 --portmap="(1,2)"
And my tx_pkt_burst callback is called occasionally, but not when I think it should be called.
For example, if I ping this board from another one, the ping is successful, but the tx_pkt_burst callback is not been called.
I tried to use devbind tool, but no devices are detected:
./usertools/dpdk-devbind.py --status
No 'Network' devices detected
=============================
No 'Baseband' devices detected
==============================
No 'Crypto' devices detected
============================
No 'Eventdev' devices detected
==============================
No 'Mempool' devices detected
=============================
No 'Compress' devices detected
==============================
No 'Misc (rawdev)' devices detected
===================================
No 'Regex' devices detected
===========================
Update
DPDK version - 20.11.
My HW is a embedded device based on NXP's Layerscape.
$ lshw -class network
*-network
description: Ethernet interface
physical id: 3
logical name: eth0
serial: 00:11:22:44:11:44
size: 1Gbit/s
capacity: 1Gbit/s
capabilities: ethernet physical tp mii 10bt-fd 100bt-fd 1000bt-fd autonegotiation
configuration: autonegotiation=on broadcast=yes driver=fsl_dpaa2_eth driverversion=5.10.35-00002-g3434eea0e1e7-dir duplex=full firmware=7.17 ip=192.168.15.157 link=yes multicast=yes port=twisted pair speed=1Gbit/s
I'm trying to bypass all traffic to the PMD I'm currently developing.
Thanks.
[EDIT-1] clarification of using same interface for DPDK and Kernel routing
Answer> as discussed over comments please refer to DPDKD + kernel on same interface
Based on the information shared there are multiple questions to the single query I'm trying to bypass all traffic to the PMD I'm currently developing. Addressing each one separately below
question 1: using dpdk-l2fwd example application
Answer> DPDK application l2fwd application makes use of basic APi with almost no HW offloads. Based on your environment (I have a board with one ethernet interface (eth0)), the right set of parameters should be -p 0x1 --no-mac-updating -T 1. This will configure the application to receive and transmit packet using single DPDK interface (that is eth0 on your board).
Note: DPDK Application can work with DPDK PMD both physical and virtual
question 2: I tried to use net_tap, using the following command:
Answer> If the intend is to intercept the traffic from physical and then forward to tap interface, then one needs modify the eal arguments as ./build/l2fwd --vdev=net_tap0,iface="my_eth0" -- -p 0x3 -T 1 --no-mac-updating. This will allow the application to probe physical NXP interface (eth0) and make use of Linux TAP interface as secondary interface. Thus any traffic from NXP and TAP will be cross connected such as NXP (eth0) <==> TAP (my_eth0)
question 3: ./usertools/dpdk-devbind.py --status returns empty
Answer> Form the dpdk site supported NIC list NXP dpaa, dpaa2, enetc, enetfec, pfe. Cross checking the kernel driver fsl_dpaa2_eth I think it is safe to assume dpaa2 PMD is supported. As you have mentioned the NIC is not enumerated, it looks like there are certain caveats to such model revision, supported board, BSP package, vendor-sub vendor ID check etc. More details can be found Board Support Package, and DPAA2 NIC guide
Debug & Alternative solutions:
To start with use the Kernel Driver to bring in packets
Use extra logging and debug to identify why the NIC is shown in the application
Approach 1:
Make sure the NIC is bind with kernel driver fsl_dpaa2_eth.
ensure NIC is connected and link is up with ethtool eth0
set to promiscous mode with ifconfig eth0 promisc up
start DPDK application with PCAP PMD, ./build/l2fwd --vdev=net_pcap0,iface=eth0 -- -p 1 --no-mac-updating -T 1
Check packet are received and redirected to PCAP eth0 PMD by checking the statistics.
Approach 2:
Ideally the NIC should be categorized under network device to be probed by debind.py.
Check the device details using lshw -c net -businfo for network.
try checking with lspci -Dvmmnnk [PCIe BUS:Slot:Function id] for network details.
If above details does not show up as network device this might be reason for not getting listed.
Suggestions or workaround: You can try to forcefully bind with igb_uio or vfio-pci (I am not much famialr with NXP SoC) by dpdk-devbind -b vfio-pci [PCIe S:B:F]. Then cross check with lspci -ks [PCIe S:B:F]. Once successfully done, one can start dpdk l2fwd in PMD debug mode with ./build/l2fwd -a [PCIe S:B:F] --log-level=pmd,8 -- -p 1 --no-mac-updating | more. Thus by intercepting and interpreting the logs one can identify what is going
Note:
It is assumed the application is build with static libraries and not dynamic. To build with static libraries use make static for l2fwd.
For the described use case recommended application is basicfwd/skeleton rather than l2fwd.
Found the problem.
I had to unbind eth0 from Linux kernel.
Now I can simply run:
./dpdk-l2fwd -c 0x3 --vdev={MY_PMD}0 -- -p 0x3 -T 1
And all traffic in the physical port is forwarded to my PMD.
I use VMware and then install qemu on it.
Create img.
$ sudo qemu-img create -f qcow2 mini3.img 5G
Then create a virtual machine.
$ sudo qemu-system-x86_64 -m 1024 -enable-kvm -hda mini3.img -cdrom ./minix_R3.3.0-588a35b.iso -boot d
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.80000001H:ECX.svm [bit 2]
And I turn off this virtual machine and then start the created virtual machine.
$ sudo qemu-system-x86_64 -m 2048 -enable-kvm -hda mini3.img
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.80000001H:ECX.svm [bit 2]
But no success.
Where is the problem?
I would guess that in the first run your OS is booting from cdrom image (minix_R3.3.0-588a35b.iso). In the second run you removed it, and there is only a hard drive image (mini3.img). You probably failed to properly install the OS in the first run to your hard drive image.
I am trying to bind this nic I40E:
Ethernet Controller X710 for 10GbE backplane 1581
my OS is ubuntu 18.04
kernel: 4.15.0-74-generic
I used dpdk-setup.sh to Insert VFIO module.
i also add iommu=on to grub file.
running devbind command:
sudo ./dpdk-devbind.py -b vfio-pci 02:00.1
i Got this erros:
Error: bind failed for 0000:02:00.1 - Cannot bind to driver vfio-pci
dmesg output:
[ 5091.393436] vfio-pci: probe of 0000:02:00.1 failed with error -22
There is no issue in binding vfio-pci on Ethernet Controller X710. I have followed the following steps with success
DPDK version: dpdk-20.11
NIC: driverversion=2.1.14-k, firmware=6.01
modprobe vfio-pci
confirm via lsmod
bind with sudo ./usertools/dpdk-devbind.py -b vfio-pci [PCIe B:D:F]
[EDIT-1] as per DPDK documentation workaround for VT-d or iommu is suggested.
note:
If VT-D is not enabled in BIOS and kernel command line (grub option), the work around is to use echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode, then follow the above steps.
if the interface is already used in Kernel - Warning: routing table indicates that interface [PCIe B:D:F] is active. Not modifying. Simply disable the interface with ifconfig [interface name] down and bind again.
When I try to start firefox on ec2 instance (centos/7) via X11, I got an error.
Is there a way to solve this?
I'm already confirmed that other GUI application, sucy as xeyes or eog, can be launched.
Env
ec2 side
OS: Centos 7.3.1611 (Core)
SELinux: Enforcing
xauth version: 1.0.9
xterm version: X.Org 7.6.0(295)
xorg-x11-apps.x86_64: 7.7-6.el7
$ sudo cat /etc/ssh/sshd_config
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key
SyslogFacility AUTHPRIV
AuthorizedKeysFile .ssh/authorized_keys
PasswordAuthentication no
ChallengeResponseAuthentication no
GSSAPIAuthentication yes
GSSAPICleanupCredentials no
UsePAM yes
X11Forwarding yes
X11UseLocalhost no
UsePrivilegeSeparation sandbox # Default for new installations.
AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
AcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE
AcceptEnv XMODIFIERS
Subsystem sftp /usr/libexec/openssh/sftp-server
local side
OS: Mac OS X 10.11.6
XQuartz version: 2.7.11 (xorg-server 1.18.4)
$ sudo cat /etc/ssh/sshd_config
# Apple:
Host *
SendEnv LANG LC_*
Host *
XAuthLocation /opt/X11/bin/xauth
Reproduction scenario
# ec2 side
sudo yum -y install xauth xterm xorg-x11-apps xeyes
exit
# local side
ssh -X fhiyo#ec2-xx-xx-xx-xx.ap-northeast-1.compute.amazonaws.com
# ec2 side
xeyes # OK
sudo yum -y install eog
eog path/to/sample.png # OK
sudo yum -y install firefox
$ firefox
[Parent 20819] WARNING: pipe error (36): Connection reset by peer: file /builddir/build/BUILD/firefox-52.2.0/firefox-52.2.0esr/ipc/chromium/src/chrome/common/ipc_channel_posix.cc, line 322
###!!! [Parent][MessageChannel] Error: (msgtype=0x46000E,name=PContent::Msg_NotifyLayerAllocated) Channel error: cannot send/recv
...
[Parent 20819] WARNING: FileDescriptorSet destroyed with unconsumed descriptors: file /builddir/build/BUILD/firefox-52.2.0/firefox-52.2.0esr/ipc/chromium/src/chrome/common/file_descriptor_set_posix.cc, line 22
###!!! [Parent][MessageChannel] Error: (msgtype=0x2C0085,name=PBrowser::Msg_Destroy) Channel error: cannot send/recv
I had a similar problem when launching Firefox version 52.3.0 with X11 forwarding on CentOS 7 :
WARNING: pipe error (35): Connexion ré-initialisée par le correspondant: file /builddir/build/BUILD/firefox-52.3.0/firefox-52.3.0esr/ipc/chromium/src/chrome/common/ipc_channel_posix.cc, line 322
And same error (in french) on firefox "Gah, you tab just crash"
To correct this, i tried first to update all the package without success.
So i combined two solutions i found on the net :
1- I opened the "about:config" page (just type about:config in the url field then enter) and changed to "false" the 2 options browser.tabs.config.autostart and browser.tabs.config.autostart2
2- I opened the "about:support" page and click on "Repair Firefox..."
Then it works ! but i still have the following problem : i need to do this each time i relaunch firefox; So i should not close all tabs.
Check your SELinux configuration. In special check that mozilla_plugin_can_network_connect is enabled. You can do: getsebool mozilla_plugin_can_network_connect if answer is off then do setsebool -P mozilla_plugin_can_network_connect=1. Then try again firefox&
To Configure dhcp ipv6 in my application I am using below mentioned command. Same command is getting executed in dhcp version 4.2.5-P1. But my application has dhclient version 4.3.0. And command cause failure for execution.
So Can any one guide me on this? What difference will be there in both version?
dhclient -6 -pf /var/run/udhcpc6.eth0.pid eth0 -e hostname:"ubuntu" -nw