Cause: rte_eth_tx_queue_setup: err=-22, port=0 while running QoS sample app - dpdk

While running DPDK QoS sample application for single packet flow configuration as mentioned in Section "122.2. Test Case: 1 pipe, 8 TCs" of https://doc.dpdk.org/dts/test_plans/qos_sched_test_plan.html, I am facing below Tx Queue setup error (-22).
Have used E810-CQDA2 and the same has been bound (PF) with DPDK igb_uio driver.
Also, there is no CONFIG_RTE_SCHED_COLLECT_STATS flag in dpdk-stable-21.11.2, whereas I have seen RTE_SCHED_COLLECT_STATS in config/rte_config.h. So, I have enabled them as #define RTE_SCHED_COLLECT_STATS.
Is this adequate to run DPDK QoS? Or do we need to define anywhere in app side also, dpdk-stable-21.11.2/examples/qos_sched?
~/dpdk-stable-21.11.2/examples/qos_sched/build# ./qos_sched -l 1,2,3 -n 4 -- -i --pfc "0,1,2,3" --cfg ../profile.cfg
EAL: Detected CPU lcores: 128
EAL: Detected NUMA nodes: 2
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: DPDK is running on a NUMA system, but is compiled without NUMA support.
EAL: This will have adverse consequences for performance and usability.
EAL: Please use --legacy-mem option, or recompile with NUMA support.
EAL: Probe PCI driver: net_ice (8086:1592) device: 0000:31:00.0 (socket 0)
ice_load_pkg_type(): Active package is: 1.3.27.0, ICE OS Default Package (single VLAN mode)
EAL: Probe PCI driver: net_ice (8086:1592) device: 0000:31:00.1 (socket 0)
ice_load_pkg_type(): Active package is: 1.3.27.0, ICE OS Default Package (single VLAN mode)
TELEMETRY: No legacy callbacks, legacy socket not created
Interactive-mode selected
APP: Initializing port 0... Ambiguous segment configuration
EAL: Error - exiting with code: 1
Cause: rte_eth_tx_queue_setup: err=-22, port=0
Expectation is to see the below result as mentioned in DPDK qos_sched_test_plan.html,
qos_sched> stats port 1 subport 0 pipe 0
+----+-------+-------------+-------------+-------------+-------------+-------------+
| TC | Queue | Pkts OK |Pkts Dropped | Bytes OK |Bytes Dropped| Length |
+----+-------+-------------+-------------+-------------+-------------+-------------+
| 0 | 0 | 6934 | 2388410 | 457644 | 157635060 | 64 |
+----+-------+-------------+-------------+-------------+-------------+-------------+
| 1 | 0 | 34 | 2395345 | 2244 | 158092770 | 64 |
+----+-------+-------------+-------------+-------------+-------------+-------------+
| 2 | 0 | 0 | 2396210 | 0 | 158149860 | 64 |
+----+-------+-------------+-------------+-------------+-------------+-------------+
| 3 | 0 | 0 | 3613033 | 0 | 238460178 | 64 |
+----+-------+-------------+-------------+-------------+-------------+-------------+
| 4 | 0 | 0 | 2381392 | 0 | 157171872 | 64 |
+----+-------+-------------+-------------+-------------+-------------+-------------+
| 5 | 0 | 0 | 2381377 | 0 | 157170882 | 64 |
+----+-------+-------------+-------------+-------------+-------------+-------------+
| 6 | 0 | 0 | 2381925 | 0 | 157207050 | 64 |
+----+-------+-------------+-------------+-------------+-------------+-------------+
| 7 | 0 | 0 | 2382177 | 0 | 157223682 | 64 |
+----+-------+-------------+-------------+-------------+-------------+-------------+

Related

CMake failling to detect a default CUDA architecture

I'm trying to build this repository and am stuck at building some thirdparty dependencies (note that I'm very new and basically have no knowledge about c++ / cmake)
I'm strictly following the installation guide provided in the repo and am stuck at trying to build ngp with following command:
cmake ./thirdparty/instant-ngp -B build_ngp
I recieve following error message:
CMake Error at /opt/cmake-3.25.2-linux-x86_64/share/cmake-3.25/Modules/CMakeDetermineCUDACompiler.cmake:603 (message):
Failed to detect a default CUDA architecture.
Compiler output:
Call Stack (most recent call first):
CMakeLists.txt:11 (project)
I'm running Ubuntu 20.04 as OS and have a nvidia 2080 super installed in the computer
cmake --version --> 3.25.2
nvidia-smi output:
Tue Feb 14 13:40:27 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.85.12 Driver Version: 525.85.12 CUDA Version: 12.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce ... On | 00000000:2D:00.0 On | N/A |
| 0% 39C P8 20W / 250W | 449MiB / 8192MiB | 1% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 1386 G /usr/lib/xorg/Xorg 53MiB |
| 0 N/A N/A 1971 G /usr/lib/xorg/Xorg 120MiB |
| 0 N/A N/A 2099 G /usr/bin/gnome-shell 36MiB |
| 0 N/A N/A 2542 G /usr/lib/firefox/firefox 226MiB |
+-----------------------------------------------------------------------------+
Would appreciate any kind of help and please let me know if I can provide more valuable info.

CUDNN_STATUS_INTERNAL_ERROR in tensorflow 2.1 c++

I face a problem as the title says when I load a pre-trained model(.pb model of YOLOv3) and infer with this model in tensorflow 2.1 c++. Error messages are as the following:
2020-10-30 21:36:20.245492: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2020-10-30 21:36:20.269906: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
[InferCC] Model infer failed(2 root error(s) found.
(0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[{{node yolov3/yolo_darknet/conv2d/Conv2D}}]]
[[StatefulPartitionedCall/_791]]
(1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[{{node yolov3/yolo_darknet/conv2d/Conv2D}}]]
0 successful operations.
Here is my configuration:
Ubuntu 18.04
Tensorflow 2.1 c++
Cuda 10.1
cuDNN 7.6.5
GPU memory ~6G
$ nvidia-smi
Fri Oct 30 21:50:24 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.57 Driver Version: 450.57 CUDA Version: 11.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 GeForce RTX 2060 Off | 00000000:01:00.0 On | N/A |
| N/A 46C P8 6W / N/A | 553MiB / 5931MiB | 1% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 1242 G /usr/lib/xorg/Xorg 18MiB |
| 0 N/A N/A 1886 G /usr/bin/gnome-shell 50MiB |
| 0 N/A N/A 9708 G /usr/lib/xorg/Xorg 321MiB |
| 0 N/A N/A 9876 G /usr/bin/gnome-shell 128MiB |
| 0 N/A N/A 12591 G /usr/lib/firefox/firefox 3MiB |
| 0 N/A N/A 14848 G ...s/QtCreator/bin/qtcreator 3MiB |
| 0 N/A N/A 15696 G ...AAAAAAAAA= --shared-files 21MiB |
+-----------------------------------------------------------------------------+
I searched it on internet and it seems that my GPU memory is ran out(I'm not sure about it). So I add following codes to set GPU memory growth before load model:
tensorflow::ConfigProto config;
config.mutable_gpu_options()->set_allow_growth(true);
But with no luck, errors are still there.
Or I want to know if it is indeed lack of GPU memory(~6G is not enough for YOLOv3 model)?
Please some one helps me out. Thanks.
Finally, I find a way to correct that error. Just pass the SessionOptions variable(call set_allow_growth with it) into LoadSavedModel function:
if(tensorflow::MaybeSavedModelDirectory(modelName.toStdString()))
{
// set gpu memory growth true here
tensorflow::SessionOptions sessionOpt;
sessionOpt.config.mutable_gpu_options()->set_allow_growth(true);
// then pass that variable into LoadSavedModel
tensorflow::Status status = tensorflow::LoadSavedModel(
sessionOpt,
tensorflow::RunOptions(),
modelName.toStdString(),
{tensorflow::kSavedModelTagServe},
_model);
if(!status.ok())
{
qCritical("[InferCC] Load %s model failed - %s.", modelName.toStdString().c_str(), status.error_message().c_str());
_loaded = false;
return false;
}
}
Then run program while keep calling nvidia-smi in command line, showing that GPU memory is increasing and the last state(used 5278 MiB)as :
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.57 Driver Version: 450.57 CUDA Version: 11.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 GeForce RTX 2060 Off | 00000000:01:00.0 On | N/A |
| N/A 58C P0 63W / N/A | 5832MiB / 5931MiB | 37% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 656 C ...VisTrack/release/VisTrack 5278MiB |
| 0 N/A N/A 1242 G /usr/lib/xorg/Xorg 18MiB |
| 0 N/A N/A 1886 G /usr/bin/gnome-shell 50MiB |
| 0 N/A N/A 4843 G /usr/lib/firefox/firefox 3MiB |
| 0 N/A N/A 5879 G /usr/lib/firefox/firefox 3MiB |
| 0 N/A N/A 9708 G /usr/lib/xorg/Xorg 293MiB |
| 0 N/A N/A 9876 G /usr/bin/gnome-shell 143MiB |
| 0 N/A N/A 12591 G /usr/lib/firefox/firefox 3MiB |
| 0 N/A N/A 13952 G /usr/lib/firefox/firefox 3MiB |
| 0 N/A N/A 14848 G ...s/QtCreator/bin/qtcreator 3MiB |
| 0 N/A N/A 15696 G ...AAAAAAAAA= --shared-files 21MiB |
+-----------------------------------------------------------------------------+

Google Composer - Airflow: My tasks are not scheduled

I am using Airflow and Cloud Composer and as I have some issues with Airflow Scheduler (it is slow or stops)
Version: composer-1.10.4-airflow-1.10.6
I launched a "huge" collect (because I will sometimes need it) with airflow to test the scalability of my pipelines.
The result is that my scheduler apparently only schedule the DAGs with few tasks, and the tasks of the big DAGs are not scheduled. Do you have insights or advices about that?
Here are information about my current configuration:
Cluster config:
10 Cluster nodes, 20 vCPUs, 160Go Memory
airflow config:
core
store_serialized_dags: True
dag_concurrency: 160
store_dag_code: True
min_file_process_interval: 30
max_active_runs_per_dag: 1
dagbag_import_timeout: 900
min_serialized_dag_update_interval: 30
parallelism: 160
scheduler
processor_poll_interval: 1
max_threads: 8
dag_dir_list_interval: 30
celery
worker_concurrency: 16
webserver
default_dag_run_display_number: 5
workers: 2
worker_refresh_interval: 120
airflow scheduler DagBag parsing (airflow list_dags -r):
DagBag loading stats for /home/airflow/gcs/dags
Number of DAGs: 27
Total task number: 32229
DagBag parsing time: 22.468404
---------------+--------------------+---------+----------+-----------------------
file | duration | dag_num | task_num | dags
---------------+--------------------+---------+----------+-----------------------
/folder__dags/dag1 | 1.83547 | 1 | 1554 | dag1
/folder__dags/dag2 | 1.717692 | 1 | 3872 | dag2
/folder__dags/dag3 | 1.53 | 1 | 3872 | dag3
/folder__dags/dag4 | 1.391314 | 1 | 210 | dag4
/folder__dags/dag5 | 1.267788 | 1 | 3872 | dag5
/folder__dags/dag6 | 1.250022 | 1 | 1554 | dag6
/folder__dags/dag7 | 1.0973419999999998 | 1 | 2904 | dag7
/folder__dags/dag8 | 1.081566 | 1 | 3146 | dag8
/folder__dags/dag9 | 1.019032 | 1 | 3872 | dag9
/folder__dags/dag10 | 0.98541 | 1 | 1554 | dag10
/folder__dags/dag11 | 0.959722 | 1 | 160 | dag11
/folder__dags/dag12 | 0.868756 | 1 | 2904 | dag12
/folder__dags/dag13 | 0.81513 | 1 | 160 | dag13
/folder__dags/dag14 | 0.69578 | 1 | 14 | dag14
/folder__dags/dag15 | 0.617646 | 1 | 294 | dag15
/folder__dags/dag16 | 0.588876 | 1 | 210 | dag16
/folder__dags/dag17 | 0.563712 | 1 | 160 | dag17
/folder__dags/dag18 | 0.55615 | 1 | 726 | dag18
/folder__dags/dag19 | 0.553248 | 1 | 140 | dag19
/folder__dags/dag20 | 0.55149 | 1 | 168 | dag20
/folder__dags/dag21 | 0.543682 | 1 | 168 | dag21
/folder__dags/dag22 | 0.530684 | 1 | 168 | dag22
/folder__dags/dag23 | 0.498442 | 1 | 484 | dag23
/folder__dags/dag24 | 0.46574 | 1 | 14 | dag24
/folder__dags/dag25 | 0.454656 | 1 | 28 | dag25
/create_conf | 0.022272 | 1 | 20 | create_conf
/airflow_monitoring | 0.006782 | 1 | 1 | airflow_monitoring
---------------+--------------------+---------+----------+------------------------
Thank you for your help
Airflow scheduler processes files in the DAGs directory in round-robin scheduling algorithm and this can cause long delays between tasks because the scheduler will not be able to enqueue a task whose dependencies recently completed until its round robin returns to the enclosing DAG's module. Multiple DAG objects can be defined in the same Python module, but this is generally discouraged from a fault isolation perspective. It may be necessary to define multiple DAGs per module.
Sometimes the best approach is to restart the scheduler:
Get cluster credentials as described in official documentation
Run the following command to restart the scheduler:
kubectl get deployment airflow-scheduler -o yaml | kubectl replace --force -f -
Additionally, please restart the Airflow web server. Sometimes broken, invalid or resource intensive DAGs can cause webserver crashes, restarts or complete downtime. Once way to do so is remove or upgrade one of the PyPI packages from your environment.
Exceeding API usage limits/quotas
To avoid exceeding API usage limits/quotas or avoid running too many simultaneous processes, you can define Airflow pools in the Airflow web UI and associate tasks with existing pools in your DAGs. Refer to the Airflow documentation.
Check the logs in Logging section -> Cloud Composer Environment and look for any errors or warnings like: cannot import module, DagNotFound in DagModel.
Please, have a look to my earlier answer regarding memory. Referring to the official documentation:
DAG execution is RAM limited. Each task execution starts with two
Airflow processes: task execution and monitoring. Currently, each node
can take up to 6 concurrent tasks. More memory can be consumed,
depending on the size of the DAG.
Moreover, I would like to share with you an interesting article on Medium, regarding calculations for resource requests.
I hope you find the above pieces of information useful.

prevent tensorflow device on multiple gpu c++

I'm using a dynamically linked tensorflow library to run a neural network on a c++ code.
2018-06-07 19:03:10.578031: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1084] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9168 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:17:00.0, compute capability: 6.1)
2018-06-07 19:03:10.615271: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1084] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 9822 MB memory) -> physical GPU (device: 1, name: GeForce GTX 1080 Ti, pci bus id: 0000:65:00.0, compute capability: 6.1)
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 384.90 Driver Version: 384.90 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 108... Off | 00000000:17:00.0 Off | N/A |
| 0% 54C P2 106W / 250W | 10734MiB / 11172MiB | 33% Default |
+-------------------------------+----------------------+----------------------+
| 1 GeForce GTX 108... Off | 00000000:65:00.0 On | N/A |
| 1% 54C P2 65W / 250W | 10655MiB / 11171MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 22230 C ./demo 10723MiB |
| 1 1264 G /usr/lib/xorg/Xorg 309MiB |
| 1 3230 G compiz 235MiB |
| 1 22230 C ./demo 10095MiB |
| 1 25625 G unity-control-center 3MiB |
+-----------------------------------------------------------------------------+
The code
tf::Session* session_ptr;
auto options = tf::SessionOptions();
options.config.mutable_gpu_options()->set_visible_device_list("0");
auto status = NewSession(tf::SessionOptions(), &session_ptr);
session.reset(tensorflow::NewSession(options));
doesn't seem to prevent tensorflow from setting up devices in all of the gpus.
I know you can use the CUDA_VISIBLE_DEVICES environment variable, but I need to do this on runtime. I also need to run several instances of this program probably on the same gpu (4 probably)
Is there any way to do this?
I also tried using
tf::GraphDef graph_def;
auto status1 = ReadBinaryProto(tf::Env::Default(), tf_model, &graph_def);
tensorflow::graph::SetDefaultDevice("0",&graph_def);
also allocates memory to both gpu...

Apache camel-jetty and activemq conflict in karaf?

I am having an issue in a clean karaf (4.0.3) installing both camel-jetty and activemq (for example, activemq-client or activemq-broker) together. It doesn't matter the order the features are installed. The second one hangs during install with no information displayed in the karaf log beyond that the install has begun.
Has anyone seen this before? Is there a workaround? I tried having the activemq-broker in it's own instance but then my app that uses both camel-jetty and jms still need to have the activemq connector initialized and thus I need to load activemq bundles/features.
Here is the output of both install separately, but when performed one after the other the 2nd always hangs the karaf instance. There don't appear to be any bundles in common.
karaf#root()> feature:install activemq-client
karaf#root()> list
START LEVEL 100 , List Threshold: 50
ID | State | Lvl | Version | Name
------------------------------------------------------------------------
52 | Active | 80 | 5.12.1 | activemq-osgi
53 | Active | 80 | 3.3.0 | Commons Net
54 | Active | 80 | 2.4.2 | Apache Commons Pool
55 | Active | 80 | 1.0.1 | geronimo-j2ee-management_1.1_spec
56 | Active | 80 | 1.1.1 | geronimo-jms_1.1_spec
57 | Active | 80 | 1.1.1 | geronimo-jta_1.1_spec
58 | Active | 80 | 3.4.6 | ZooKeeper Bundle
63 | Active | 80 | 2.2.11.1 | Apache ServiceMix :: Bundles :: jaxb-impl
70 | Active | 80 | 3.18.0 | Apache XBean :: Spring
71 | Active | 80 | 0.6.4 | JAXB2 Basics - Runtime
(Performed clean karaf launch in between installs to get bundle listings)
karaf#root()> feature:install camel-jetty
karaf#root()> list
START LEVEL 100 , List Threshold: 50
ID | State | Lvl | Version | Name
--------------------------------------------------------------------------------
55 | Active | 80 | 2.12.2 | camel-core
56 | Active | 80 | 2.12.2 | camel-http
57 | Active | 80 | 2.12.2 | camel-jetty
58 | Active | 80 | 2.12.2 | camel-karaf-commands
59 | Active | 80 | 1.8.0 | Commons Codec
63 | Active | 80 | 3.1.0.7 | Apache ServiceMix :: Bundles :: commons-httpclient