DPDK failed to configure eth device with err No: -22, tx_queues is greater than one (Ethdev port_id=0 nb_tx_queues=2 > 1) - dpdk

I have an application that uses DPDK for Fast Path. During DPDK initialization, am trying to configure two TX queues for a port, but it failed to configure the eth device.
I am using Intel IGB driver(I350 NIC) on a Bare Metal setup. As per DPDK documentation IGB Poll Mode Driver (https://doc.dpdk.org/guides/nics/igb.html) should support multiple RX/TX queues for a port. Am trying to configure two TX queues for a port(port_id = 0), when invoking the API "rte_eth_dev_configure(portid = 0, nb_rx_queue = 1, nb_tx_queue = 2, &local_port_conf)", it is returning error code: -22, "Ethdev port_id=0 nb_tx_queues=2 > 1".
Does the IGB PMD driver support multiple TX queues for a port? Or do I need to do any configuration changes to support multiple TX queues?

For virtual functions (VFs), the NIC model in question supports only single-queue mode (source).
To test multi-queue support, consider passing a physical function (PF) to the setup instead.

Intel Foundational NIC i350 assign Up to eight queues per port for Physical Funtion driver as per PRODUCT BRIEF IntelĀ® Ethernet Controller I350. With respect to virtual function per port 1 queue for each VF with a total of 8 VF max per port defined under IntelĀ® Ethernet Controller I350 Datasheet.
In Linux, these can be validated by
ethtool: One can check per queue instance with option -S, as statistics is supported per queue.
dpdk: using rte_eth_dev_info_get to get attributes max_rx_queues and max_tx_queues
Note: as VF gets allocated with 1 RX-TX queue pair, the error mentioned -22, Ethdev port_id=0 nb_tx_queues=2 > 1 can be of VF in use. Hence the right way of configuration is first get the DPDK port max rx-tx queues and ensure it is in bounds.
Sample Code Snippet:
/* Initialise each port */
RTE_ETH_FOREACH_DEV(portid) {
struct rte_eth_rxconf rxq_conf;
struct rte_eth_txconf txq_conf;
struct rte_eth_conf local_port_conf = port_conf;
struct rte_eth_dev_info dev_info;
/* skip ports that are not enabled */
if ((l2fwd_enabled_port_mask & (1 << portid)) == 0) {
printf("Skipping disabled port %u\n", portid);
continue;
}
nb_ports_available++;
/* init port */
printf("Initializing port %u... ", portid);
fflush(stdout);
ret = rte_eth_dev_info_get(portid, &dev_info);
if (ret != 0)
rte_exit(EXIT_FAILURE,
"Error during getting device (port %u) info: %s\n",
portid, strerror(-ret));
if (dev_info.tx_offload_capa & RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE)
local_port_conf.txmode.offloads |=
RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE;
/* Configure the number of queues for a port. */
if ((dev_info.max_rx_queues >= user_rx_queues) && (dev_info.max_tx_queues >= user_tx_queues)) {
ret = rte_eth_dev_configure(portid, user_rx_queues, user_tx_queues, &local_port_conf);
if (ret < 0)
rte_exit(EXIT_FAILURE, "Cannot configure device: err=%d, port=%u\n",
ret, portid);
/* >8 End of configuration of the number of queues for a port. */
}

Related

Multiple errors when trying to use DPDK pdump

Goal: To write a DPDK based application that can read UDP packets via a 100gbe ethernet port and write the payload to disk depending on what the destination IP/Port is. At most, each 100gbe link will have two different destination IP addresses and 4 unique destination port numbers. Initial design calls for two unique port numbers.
Hardware
Current Test System
For now, I am testing with the following hardware. The server hardware and NVME drives will be significantly upgraded in the next few weeks. For now, I am using the following hardware to develop a proof of concept (POC). The NIC will remain the same unless recommended otherwise.
2 x Intel Xeon Gold 6348 CPU # 2.6 Ghz
28 cores per socket
Max 3.5 Ghz
Hyperthreading disabled
Ubuntu 22.04.1 LTS
Kernel 5.15.0-53-generic
Cores set to performance governor
4 x Sabrent 2TB Rocket 4 Plus in RAID0 Config
128 GB DDR4 Memory
10 1GB HugePages (Can change to what is required)
1 x Mellanox ConnectX-5 100gbe NIC
31:00.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]
Firmware-version: 16.35.1012
UDP Source:
100 gbe NIC
9000 MTU Packets
ipv4-udp packets
Currently only 4GB/s per port but eventually will be 10GB/s per port.
NIC Information
ethtool output:
Settings for ens7f0np0:
Supported ports: [ Backplane ]
Supported link modes: 1000baseKX/Full
10000baseKR/Full
40000baseKR4/Full
40000baseCR4/Full
40000baseSR4/Full
40000baseLR4/Full
25000baseCR/Full
25000baseKR/Full
25000baseSR/Full
50000baseCR2/Full
50000baseKR2/Full
100000baseKR4/Full
100000baseSR4/Full
100000baseCR4/Full
100000baseLR4_ER4/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Supported FEC modes: None RS BASER
Advertised link modes: 1000baseKX/Full
10000baseKR/Full
40000baseKR4/Full
40000baseCR4/Full
40000baseSR4/Full
40000baseLR4/Full
25000baseCR/Full
25000baseKR/Full
25000baseSR/Full
50000baseCR2/Full
50000baseKR2/Full
100000baseKR4/Full
100000baseSR4/Full
100000baseCR4/Full
100000baseLR4_ER4/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Advertised FEC modes: RS
Speed: 100000Mb/s
Duplex: Full
Auto-negotiation: on
Port: Direct Attach Copper
PHYAD: 0
Transceiver: internal
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000004 (4)
link
Link detected: yes
testpmd info:
[sudo] password for maa:
EAL: Detected CPU lcores: 56
EAL: Detected NUMA nodes: 2
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: No available 2048 kB hugepages reported
EAL: VFIO support initialized
EAL: Probe PCI driver: mlx5_pci (15b3:1017) device: 0000:31:00.0 (socket 0)
EAL: Probe PCI driver: mlx5_pci (15b3:1017) device: 0000:31:00.1 (socket 0)
TELEMETRY: No legacy callbacks, legacy socket not created
Interactive-mode selected
testpmd: create a new mbuf pool <mb_pool_0>: n=171456, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
Configuring Port 0 (socket 0)
Port 0: B8:CE:F6:FB:13:30
Configuring Port 1 (socket 0)
Port 1: B8:CE:F6:FB:13:31
Checking link statuses...
Done
testpmd> show port info 0
********************* Infos for port 0 *********************
MAC address: B8:CE:F6:FB:13:30
Device name: 31:00.0
Driver name: mlx5_pci
Firmware-version: 16.35.1012
Devargs:
Connect to socket: 0
memory allocation on the socket: 0
Link status: up
Link speed: 100 Gbps
Link duplex: full-duplex
Autoneg status: On
MTU: 1500
Promiscuous mode: enabled
Allmulticast mode: disabled
Maximum number of MAC addresses: 128
Maximum number of MAC addresses of hash filtering: 0
VLAN offload:
strip off, filter off, extend off, qinq strip off
Hash key size in bytes: 40
Redirection table size: 1
Supported RSS offload flow types:
ipv4
ipv4-frag
ipv4-tcp
ipv4-udp
ipv4-other
ipv6
ipv6-frag
ipv6-tcp
ipv6-udp
ipv6-other
ipv6-ex
ipv6-tcp-ex
ipv6-udp-ex
l4-dst-only
l4-src-only
l3-dst-only
l3-src-only
Minimum size of RX buffer: 32
Maximum configurable length of RX packet: 65536
Maximum configurable size of LRO aggregated packet: 65280
Current number of RX queues: 1
Max possible RX queues: 1024
Max possible number of RXDs per queue: 65535
Min possible number of RXDs per queue: 0
RXDs number alignment: 1
Current number of TX queues: 1
Max possible TX queues: 1024
Max possible number of TXDs per queue: 65535
Min possible number of TXDs per queue: 0
TXDs number alignment: 1
Max segment number per packet: 40
Max segment number per MTU/TSO: 40
Device capabilities: 0x14( RXQ_SHARE FLOW_SHARED_OBJECT_KEEP )
Switch name: 31:00.0
Switch domain Id: 0
Switch Port Id: 65535
Switch Rx domain: 0
RX offload capabilities:
testpmd> show port 0 rx_offload capabilities
Rx Offloading Capabilities of port 0 :
Per Queue : VLAN_STRIP IPV4_CKSUM UDP_CKSUM TCP_CKSUM TCP_LRO SCATTER TIMESTAMP KEEP_CRC RSS_HASH BUFFER_SPLIT
Per Port : VLAN_FILTER
Physical Layout
For now, I am only attempting to get a single stream on a single port working. My plan is for each queue to be tied to an lcore, and each lcore has its own thread to strip the headers, and then a thread to write to disk. Assume that the hardware is on the same NUMA node. No TX is currently required.
Requirements
Eventually 10 GB/s to disk from one 100gb/s port (Currently trying to get 4GB/s)
Assume that I will have a processor, PCI Lanes, and NVME configuration that can support this, I am only worried about the DPDK side of things for now.
Each individual stream will be ~5GB/s, currently they are ~2GB/s and I am just trying to get that to work for now.
Each stream's headers are stripped.
Each stream has its payload written to disk as a single continous file (.dat format).
Each stream has a unique port number.
If the FCS is bad, drop the packet and replace with blank data (all zeros).
What I have tried:
I am having a similar issue to the poster at this question: DPDK pdump failed to hotplug add device
I attempted the fix there and am still having issues.
I have simply tried to get a pcap file by attaching pdump to the skeleton code.
/* SPDX-License-Identifier: BSD-3-Clause
* Copyright(c) 2010-2015 Intel Corporation
*/
#include <stdint.h>
#include <inttypes.h>
#include <rte_eal.h>
#include <rte_ethdev.h>
#include <rte_cycles.h>
#include <rte_lcore.h>
#include <rte_mbuf.h>
#include <rte_pdump.h>
#define RX_RING_SIZE 1024
#define TX_RING_SIZE 1024
#define NUM_MBUFS 8191
#define MBUF_CACHE_SIZE 250
#define BURST_SIZE 32
/* basicfwd.c: Basic DPDK skeleton forwarding example. */
/*
* Initializes a given port using global settings and with the RX buffers
* coming from the mbuf_pool passed as a parameter.
*/
/* Main functional part of port initialization. 8< */
static inline int
port_init(uint16_t port, struct rte_mempool *mbuf_pool)
{
struct rte_eth_conf port_conf;
const uint16_t rx_rings = 1, tx_rings = 1;
uint16_t nb_rxd = RX_RING_SIZE;
uint16_t nb_txd = TX_RING_SIZE;
int retval;
uint16_t q;
struct rte_eth_dev_info dev_info;
struct rte_eth_txconf txconf;
if (!rte_eth_dev_is_valid_port(port))
return -1;
memset(&port_conf, 0, sizeof(struct rte_eth_conf));
retval = rte_eth_dev_info_get(port, &dev_info);
if (retval != 0) {
printf("Error during getting device (port %u) info: %s\n",
port, strerror(-retval));
return retval;
}
if (dev_info.tx_offload_capa & RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE)
port_conf.txmode.offloads |=
RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE;
/* Configure the Ethernet device. */
retval = rte_eth_dev_configure(port, rx_rings, tx_rings, &port_conf);
if (retval != 0)
return retval;
retval = rte_eth_dev_adjust_nb_rx_tx_desc(port, &nb_rxd, &nb_txd);
if (retval != 0)
return retval;
/* Allocate and set up 1 RX queue per Ethernet port. */
for (q = 0; q < rx_rings; q++) {
retval = rte_eth_rx_queue_setup(port, q, nb_rxd,
rte_eth_dev_socket_id(port), NULL, mbuf_pool);
if (retval < 0)
return retval;
}
txconf = dev_info.default_txconf;
txconf.offloads = port_conf.txmode.offloads;
/* Allocate and set up 1 TX queue per Ethernet port. */
for (q = 0; q < tx_rings; q++) {
retval = rte_eth_tx_queue_setup(port, q, nb_txd,
rte_eth_dev_socket_id(port), &txconf);
if (retval < 0)
return retval;
}
/* Starting Ethernet port. 8< */
retval = rte_eth_dev_start(port);
/* >8 End of starting of ethernet port. */
if (retval < 0)
return retval;
/* Display the port MAC address. */
struct rte_ether_addr addr;
retval = rte_eth_macaddr_get(port, &addr);
if (retval != 0)
return retval;
printf("Port %u MAC: %02" PRIx8 " %02" PRIx8 " %02" PRIx8
" %02" PRIx8 " %02" PRIx8 " %02" PRIx8 "\n",
port, RTE_ETHER_ADDR_BYTES(&addr));
/* Enable RX in promiscuous mode for the Ethernet device. */
retval = rte_eth_promiscuous_enable(port);
/* End of setting RX port in promiscuous mode. */
if (retval != 0)
return retval;
return 0;
}
/* >8 End of main functional part of port initialization. */
/*
* The lcore main. This is the main thread that does the work, reading from
* an input port and writing to an output port.
*/
/* Basic forwarding application lcore. 8< */
static __rte_noreturn void
lcore_main(void)
{
uint16_t port;
int total = 0;
/*
* Check that the port is on the same NUMA node as the polling thread
* for best performance.
*/
RTE_ETH_FOREACH_DEV(port)
if (rte_eth_dev_socket_id(port) >= 0 &&
rte_eth_dev_socket_id(port) !=
(int)rte_socket_id())
printf("WARNING, port %u is on remote NUMA node to "
"polling thread.\n\tPerformance will "
"not be optimal.\n", port);
printf("\nCore %u forwarding packets. [Ctrl+C to quit]\n",
rte_lcore_id());
/* Main work of application loop. 8< */
for (;;) {
/*
* Receive packets on a port and forward them on the paired
* port. The mapping is 0 -> 1, 1 -> 0, 2 -> 3, 3 -> 2, etc.
*/
RTE_ETH_FOREACH_DEV(port) {
/* Get burst of RX packets, from first port of pair. */
struct rte_mbuf *bufs[BURST_SIZE];
const uint16_t nb_rx = rte_eth_rx_burst(port, 0,
bufs, BURST_SIZE);
if (unlikely(nb_rx == 0))
continue;
total += nb_rx;
//rte_pktmbuf_free(bufs);
//printf("\nTotal: %d",total);
}
}
/* >8 End of loop. */
}
/* >8 End Basic forwarding application lcore. */
/*
* The main function, which does initialization and calls the per-lcore
* functions.
*/
int
main(int argc, char *argv[])
{
struct rte_mempool *mbuf_pool;
unsigned nb_ports;
uint16_t portid;
/* Initializion the Environment Abstraction Layer (EAL). 8< */
int ret = rte_eal_init(argc, argv);
rte_pdump_init();
if (ret < 0)
rte_exit(EXIT_FAILURE, "Error with EAL initialization\n");
/* >8 End of initialization the Environment Abstraction Layer (EAL). */
argc -= ret;
argv += ret;
/* Check that there is an even number of ports to send/receive on. */
nb_ports = rte_eth_dev_count_avail();
// if (nb_ports < 2 || (nb_ports & 1))
// rte_exit(EXIT_FAILURE, "Error: number of ports must be even\n");
/* Creates a new mempool in memory to hold the mbufs. */
/* Allocates mempool to hold the mbufs. 8< */
mbuf_pool = rte_pktmbuf_pool_create("MBUF_POOL", NUM_MBUFS * nb_ports,
MBUF_CACHE_SIZE, 0, RTE_MBUF_DEFAULT_BUF_SIZE, rte_socket_id());
/* >8 End of allocating mempool to hold mbuf. */
if (mbuf_pool == NULL)
rte_exit(EXIT_FAILURE, "Cannot create mbuf pool\n");
/* Initializing all ports. 8< */
RTE_ETH_FOREACH_DEV(portid)
if (port_init(portid, mbuf_pool) != 0)
rte_exit(EXIT_FAILURE, "Cannot init port %"PRIu16 "\n",
portid);
/* >8 End of initializing all ports. */
if (rte_lcore_count() > 1)
printf("\nWARNING: Too many lcores enabled. Only 1 used.\n");
/* Call lcore_main on the main core only. Called on single lcore. 8< */
lcore_main();
/* >8 End of called on single lcore. */
/* clean up the EAL */
rte_eal_cleanup();
rte_pdump_uninit();
return 0;
}
Command to run this example that I am using: sudo ./dpdk-skeleton -l 1,2,3,4 -n 4 -a 0000:31:00.0
Command running:
EAL: Detected CPU lcores: 56
EAL: Detected NUMA nodes: 2
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: No available 2048 kB hugepages reported
EAL: VFIO support initialized
EAL: Probe PCI driver: mlx5_pci (15b3:1017) device: 0000:31:00.0 (socket 0)
TELEMETRY: No legacy callbacks, legacy socket not created
Port 0 MAC: b8 ce f6 fb 13 30
WARNING: Too many lcores enabled. Only 1 used.
Core 1 forwarding packets. [Ctrl+C to quit]
I am not really sure what some command line values should be (like -n) so I am just guessing at this point.
For pdump, I am using this command sudo ./dpdk-pdump -l 3,4,5 -a 0000:31:00.0 -- --multi --pdump 'port=0,queue=0,rx-dev=/mnt/md0/rx-1.pcap'
Again I am not sure of some of these values so making a best guess.
The Issue
Now with the primary application running, I attempt to run pdump the first time:
EAL: Detected CPU lcores: 56
EAL: Detected NUMA nodes: 2
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket_57567_26f06531cb8cd
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Probe PCI driver: mlx5_pci (15b3:1017) device: 0000:31:00.0 (socket 0)
Segmentation fault
The Second Time:
EAL: Detected CPU lcores: 56
EAL: Detected NUMA nodes: 2
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket_57601_26f14f88bf7bb
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Probe PCI driver: mlx5_pci (15b3:1017) device: 0000:31:00.0 (socket 0)
EAL: Failed to hotplug add device
EAL: Error - exiting with code: 1
Cause: vdev creation failed:create_mp_ring_vdev:767
And finally, the third time seems to run but has an unfamiliar MAC address:
EAL: Detected CPU lcores: 56
EAL: Detected NUMA nodes: 2
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket_57679_26f28a2a6e9f1
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Probe PCI driver: mlx5_pci (15b3:1017) device: 0000:31:00.0 (socket 0)
Port 1 MAC: 02 70 63 61 70 01
core (4); port 0 device ((null)) queue 0
However, I am not receiving anything and the pcap file is empty (and yes I am sending packets):
Signal 2 received, preparing to exit...
##### PDUMP DEBUG STATS #####
-packets dequeued: 0
-packets transmitted to vdev: 0
-packets freed: 0
Also, on the second run, these messages start appearing on the primary application:
EAL: Failed to send hotplug request to secondary
6: [./dpdk-skeleton(_start+0x25) [0x55f70a909045]]
5: [/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fb90e478e40]]
4: [/lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fb90e478d90]]
3: [./dpdk-skeleton(main+0x1ad) [0x55f70a20a8ad]]
2: [./dpdk-skeleton(+0x1fdde7) [0x55f70a014de7]]
1: [./dpdk-skeleton(rte_dump_stack+0x32) [0x55f70aa973d2]]
lcore 1 called rx_pkt_burst for not ready port 1
6: [./dpdk-skeleton(_start+0x25) [0x55f70a909045]]
5: [/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fb90e478e40]]
4: [/lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fb90e478d90]]
3: [./dpdk-skeleton(main+0x1ad) [0x55f70a20a8ad]]
2: [./dpdk-skeleton(+0x1fdde7) [0x55f70a014de7]]
1: [./dpdk-skeleton(rte_dump_stack+0x32) [0x55f70aa973d2]]
lcore 1 called rx_pkt_burst for not ready port 1
6: [./dpdk-skeleton(_start+0x25) [0x55f70a909045]]
5: [/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fb90e478e40]]
4: [/lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fb90e478d90]]
3: [./dpdk-skeleton(main+0x1ad) [0x55f70a20a8ad]]
2: [./dpdk-skeleton(+0x1fdde7) [0x55f70a014de7]]
1: [./dpdk-skeleton(rte_dump_stack+0x32) [0x55f70aa973d2]]
My Questions
I will handle writing to file etc. later. I am simply trying to get the data into DPDK properly and to the correct queue at this point.
I assume that I need to use RSS or rte_flow to direct each stream to its own queue based on IP/Port. How do I do this? This is my first time doing any kind of advanced network programming so I am not familiar with these concepts, but based on my research they appear to be what I need.
What is the difference between RSS and rte_flow/offloading? Are they the same thing?
Are there any examples out there that are similar to what I am trying to do?
Is what I am trying to do feasible?
First part of question:
If the goal to capture packets at line rate and write the descriptor and met-data into NVME, then please do not use the testpmd-pdump model. this will not scale for your use case intent.
To clarify the above
DPDK allows a multiprocess model - Single primary and multiple secondary to share same hugepage, hw resources and NIC.
DPDK testpmd and dpdk pdump is a prime example of using priamry-secondary model
BUT PDUMP library as of today make use of huge page sharing to clone-copy the packet received from PRIMARY (testpmd) to send it over RING buffer to SECODNARY (PDUMP) and then write to file using PCAP
PDUMP application is enahced to support multiple queues on multiple CPU, but one will still have inherent overhead of packet copy and send over ring buffer.
hence for the above use case mentioned, best approach to have the Primary custom application to receive packets over multiple queues (4 streams - then 4 queues) for UDP. Take as much of help from HW (MLX NIC CX05) for packet filtering using rte_flow
based flow bifurcation and type for packet classification to minimize the CPU overhead in parsing.
The second part of the question:
you are using DPDK Skeleton as primary
there is only 1RX-TX queue configured for the port.
due to the inherent design of CX-5 max that 1 queue can do is around 35Mpps to 40Mpps with 1 queue with the right settings.
for 100Gbps port with 1 queue you can achieve line rate for simple forward for a payload size of 512B onwards
From the code snippet I do find rte_pdump_init invoked but returned not checked (please fix).
I believe MLX CX-5 works with PDUMP with the right firmware, hence please cross-check the same too (information requested over comment).
But the recommendation is not to use PDUMP as secondary due to native overhead to MP_HANDLE. Instead directly consume the paylaod in primary or secondary.
Note:
as mentioned there are 3 or 4 scenarios cramped into a single question, and requires live debug for sorting these out 1 by 1.
please do not club multiple questions and scenario into 1 question

modbus_read_registers() of libmodbus library not able to read RS485 data on Raspbian GNU/Linux 10 (buster) but able to read on Yocto Dizzy Release

I am using below c\c++ sample code to read Modbus RTU data using libmodbus. I have two different linux based gateways one has Raspbian GNU/Linux 10 (buster) while other has Yocto Dizzy Release. Using libmodbus lib, I am able to read modbus tcp data on both gateways. But in case of Raspbian GNU/Linux 10 (buster) for modbus rtu (RS485), I am getting connection timeout while reading buffer. There is one more difference in both gateways, i.e, Raspbian GNU/Linux 10 (buster) uses ttyUSB0 port while Yocto Dizzy Release uses ttymxc2 port of linux.
But when I tried a sample code in python using pymodbus on Raspbian GNU/Linux 10 (buster), I was able to read data.
Can someone help to identify what's going wrong for modbus rtu (RS485) using libmodbus for Raspbian GNU/Linux 10 (buster) in c++.
//Create a new RTU context with proper serial parameters (in this example,
//device name /dev/ttyUSB0, baud rate 9600, no parity bit, 8 data bits, 1 stop bit)
modbus_t *ctx = modbus_new_rtu("/dev/ttyUSB0", 9600, 'N', 8, 1);
if (!ctx) {
printf("Failed to create the context: %s\n", modbus_strerror(errno));
//exit(1);
}
if (modbus_connect(ctx) == -1) {
printf("Unable to connect: %s\n", modbus_strerror(errno));
modbus_free(ctx);
//exit(1);
}
//Set the Modbus address of the remote slave (to 1)
modbus_set_slave(ctx, 1);
uint16_t reg[5];// will store read registers values
//Read 5 holding registers starting from address 10
int num = modbus_read_registers(ctx, 50001, 5, reg);
if (num != 5) {// number of read registers is not the one expected
printf("Failed to read: %s\n", modbus_strerror(errno));
}
else
{
for(int i=0; i < 5; i++)
{
printf("reg %d: %d\n", i,reg[i]);
}
}
modbus_close(ctx);
modbus_free(ctx);
return 0;

DPDK: MPLS packet processing

I am trying to build a multi-RX-queue dpdk program, using RSS to split the incoming traffic into RX queues on a single port. Mellanox ConnectX-5 and DPDK Version 19.11 is used for this purpose. It works fine when I use IP over Ethernet packets as input. However when the packet contains IP over MPLS over Ethernet, RSS does not seems to work. As a result, all packets belonging to various flows (with different src & dst IPs, ports) over MPLS are all sent into the same RX queue.
My queries are
Is there any parameter/techniques in DPDK to distribute MPLS packets to multiple RX queues?
Is there any way to strip off MPLS tags (between Eth and IP) in hardware, something like hw_vlan_strip?
My Port configuration is
const struct rte_eth_conf default_port_conf = {
.rxmode = {
.hw_vlan_strip = 0, /* VLAN strip enabled. */
.header_split = 0, /* Header Split disabled. */
.hw_ip_checksum = 0, /* IP checksum offload disabled. */
.hw_strip_crc = 0, /* CRC stripping by hardware disabled. */
},
.rx_adv_conf = {
.rss_conf = {
.rss_key = NULL,
.rss_key_len = 0,
.rss_hf = ETH_RSS_IP,
},
} };
The requirement of POP_MPLS and RSS on MPLS can be activated via RTE_FLOW for supported NIC PMD. But mellanox mxl5 PMD supports only RTE_FLOW_ACTION_TYPE_OF_PUSH_VLAN & RTE_FLOW_ACTION_TYPE_OF_PUSH_VLAN. Only options supported for tunneled packets by mxl5 PMD are MPLSoGRE, MPLSoUD. Hence POP MPLS in HW via PMD is not possible on MXL5 PMD for DPDK 19.11 LTS
For any PMD RSS is reserved for outer/inner IP address along with TCP/UDP/SCTP port numbers. Hence I have to interpret RSS for MPLS as I would like to distribute/ spread packets with different MPLS to various queues. This can be achieved by again using RTE_FLOW for RTE_FLOW_ITEM_TYPE_MPLS and action field as RTE_FLOW_ACTION_TYPE_QUEUE. Using mask/range fields one can set patterns which can satisfy condition as 2 ^ 20 (MPLS id max value) / number of RX queues. hence the recommendation is to use RTE_FLOW_ITEM_TYPE_MPLS from RTE_FLOW and RTE_FLOW_ACTION_TYPE_QUEUE. But there is no IP/PORT RSS hashing for the same.
to test the same you can use
DPDK testpmd and set the flow rules or
make use of RTE_FLOW code snippet from rte_flow link
note: for POP MPLS I highly recommend to use PTYPES to identify the metadata and use RX-callabck to modify the packet header.

DPDK 18.11 HW checksum support for X722 NIC?

I am running dpdk-stable-18.11.8 on Centos 7, targeting an Intel X722 NIC.
I want ipv4 and udp header checksums to be calculated by hardware, so I set the device configuration to:
struct rte_eth_conf local_port_conf;
memset(&local_port_conf, 0, sizeof(struct rte_eth_conf));
local_port_conf.rxmode.split_hdr_size = 0;
local_port_conf.txmode.mq_mode = ETH_MQ_TX_NONE;
local_port_conf.txmode.offloads = DEV_TX_OFFLOAD_OUTER_UDP_CKSUM | DEV_TX_OFFLOAD_OUTER_IPV4_CKSUM;
rte_eth_dev_configure(0,1,1,&local_port_conf);
rte_eth_dev_configure returns:
0xffffffea (-22)
Does this mean that DPDK 18.11 doesn't support checksum offload to the X722 NIC?
DEV_TX_OFFLOAD_OUTER_IPV4_CKSUM is used for outer tunnelling packet, for which X710 has to be loaded with DDP. If the intent is for normal packet DEV_TX_OFFLOAD_IPV4_CKSUM is to be used.
Note: right way of configuring any DPDK port is to first fetch capability by rte_eth_dev_info_get. Then check dev_info.tx_offload_capa & DEV_TX_OFFLOAD_IPV4_CKSUM, if present configure.

How do I receive raw, layer 2 packets in C/C++?

How do I receive layer 2 packets in POSIXy C++? The packets only have src and dst MAC address, type/length, and custom formatted data. They're not TCP or UDP or IP or IGMP or ARP or whatever - they're a home-brewed format given unto me by the Hardware guys.
My socket(AF_PACKET, SOCK_RAW, IPPROTO_RAW) never returns from its recvfrom().
I can send fine, I just can't receive no matter what options I fling at the network stack.
(Platform is VxWorks, but I can translate POSIX or Linux or whatever...)
receive code (current incarnation):
int s;
if ((s = socket(AF_PACKET, SOCK_RAW, IPPROTO_RAW)) < 0) {
printf("socket create error.");
return -1;
}
struct ifreq _ifr;
strncpy(_ifr.ifr_name, "lltemac0", strlen("lltemac0"));
ioctl(s, IP_SIOCGIFINDEX, &_ifr);
struct sockaddr_ll _sockAttrib;
memset(&_sockAttrib, 0, sizeof(_sockAttrib));
_sockAttrib.sll_len = sizeof(_sockAttrib);
_sockAttrib.sll_family = AF_PACKET;
_sockAttrib.sll_protocol = IFT_ETHER;
_sockAttrib.sll_ifindex = _ifr.ifr_ifindex;
_sockAttrib.sll_hatype = 0xFFFF;
_sockAttrib.sll_pkttype = PACKET_HOST;
_sockAttrib.sll_halen = 6;
_sockAttrib.sll_addr[0] = 0x00;
_sockAttrib.sll_addr[1] = 0x02;
_sockAttrib.sll_addr[2] = 0x03;
_sockAttrib.sll_addr[3] = 0x12;
_sockAttrib.sll_addr[4] = 0x34;
_sockAttrib.sll_addr[5] = 0x56;
int _sockAttribLen = sizeof(_sockAttrib);
char packet[64];
memset(packet, 0, sizeof(packet));
if (recvfrom(s, (char *)packet, sizeof(packet), 0,
(struct sockaddr *)&_sockAttrib, &_sockAttribLen) < 0)
{
printf("packet receive error.");
}
// code never reaches here
I think the way to do this is to write your own Network Service that binds to the MUX layer in the VxWorks network stack. This is reasonably well documented in the VxWorks Network Programmer's Guide and something I have done a number of times.
A custom Network Service can be configured to see all layer 2 packets received on a network interface using the MUX_PROTO_SNARF service type, which is how Wind River's own WDB protocol works, or packets with a specific protocol type.
It is also possible to add a socket interface to your custom Network Service by writing a custom socket back-end that sits between the Network Service and the socket API. This is not required if you are happy to do the application processing in the Network Service.
You haven't said which version of VxWorks you are using but I think the above holds for VxWorks 5.5.x and 6.x
Have you tried setting the socket protocol to htons(ETH_P_ALL) as prescribed in packet(7)? What you're doing doesn't have much to do with IP (although IPPROTO_RAW may be some wildcard value, dunno)
I think this is going to be a bit tougher problem to solve than you expect. Given that it's not IP at all (or apparently any other protocol anything will recognize), I don't think you'll be able to solve your problem(s) entirely with user-level code. On Linux, I think you'd need to write your own device agnostic interface driver (probably using NAPI). Getting it to work under VxWorks will almost certainly be non-trivial (more like a complete rewrite from the ground-up than what most people would think of as a port).
Have you tried confirming via Wireshark that a packet has actually been sent from the other end?
Also, for debugging, ask your hardware guys if they have a debug pin (you can attach to a logic analyzer) that they can assert when it receives a packet. Just to make sure that the hardware is getting the packets fine.
First you need to specify the protocol as ETH_P_ALL so that your interface gets all the packet. Set your socket to be on promiscuous mode. Then bind your RAW socket to an interface before you perform a receive.