Hello Gert,

I can’t see anything obviously wrong with bringup and hugepage allocation, did you reboot the nodes (just as a sanity check)?

The next steps I will take is to get in touch with SPDK folks and reproduce on our side.

Regards,

Tom Nabarro – DCG/ESAD

M: +44 (0)7786 260986

Skype: tom.nabarro

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Gert Pauwels
Sent: Monday, September 14, 2020 4:30 PM
To: daos@daos.groups.io
Subject: Re: [daos] no dRPC client set on Ubuntu 20.04.1

Hi Tom,

After booting I run the following 2 commands:
root@intel-S2600WFD:~/daos# cat /proc/meminfo | grep Huge
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
FileHugePages:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
root@intel-S2600WFD:~/daos# ls -lah /dev/hugepages/
total 0
drwxr-xr-x  2 root root    0 Sep 14 16:40 .
drwxr-xr-x 20 root root 4,3K Sep 14 16:41 ..

 
At this point followed the steps you described to bind the NVMe drives to the kernel and wiped the pmem[01] devices.
After running "dmg storage format" from another tty the error showed and I stopped daos_server.

The daos_server.log, daos_adim.log and daos_control.log are attached.

At this point I run the two commands again:

root@intel-S2600WFD:~/daos# cat /proc/meminfo | grep Huge
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
FileHugePages:         0 kB
HugePages_Total:    4096
HugePages_Free:     4095
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:         8388608 kB

root@intel-S2600WFD:~/daos# ls -lah /dev/hugepages/spdk_pid4857map_0
-rw------- 1 root root 2,0M Sep 14 16:49 /dev/hugepages/spdk_pid4857map_0

Regards,

Gert,

Hi Anton,

The last one that we’ve tried and worked for us was MOFED 5.0.2 which is what we currently use for our test clusters that use ofi+verbs;ofi_rxm provider.

Thanks,

~~Alex.

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of anton.brekhov@...
Sent: Monday, September 14, 2020 12:45 AM
To: daos@daos.groups.io
Subject: Re: [daos] DAOS_test failed

I'm using Optane PMEM ( there are 4 modules 512GB each, two PMEM modules near each socket). I created /dev/pmem0 and /dev/pmem1 devices using `ipmctl create -goal PersistentMemoryType=AppDirect` command. On daos server I have two mellanox IB interfaces, but only one in use (mlx5_0). Here is ibstat output:

Seems like I didn't install OFED drivers, because ofed_info not found. Which version is better to use? 

Hello Gert,

It’s likely the thread lib init failure is due to a problem with hugepages configuration as spdk_thread_lib_init() only calls spdk_mempool_create().

Can you please do some debug as follows:

• First , some basic verification of hp state e.g.

bash-4.2$ cat /proc/meminfo| grep Huge

AnonHugePages:     73728 kB

HugePages_Total:   16384

HugePages_Free:    16371

HugePages_Rsvd:        0

HugePages_Surp:        0

Hugepagesize:       2048 kB

bash-4.2$ ls -lah /dev/hugepages/spdk_pid217893map_*

-rw------- 1 root daos_admins 2.0M Sep 11 12:33 /dev/hugepages/spdk_pid217893map_0

-rw------- 1 root daos_admins 2.0M Sep 11 12:33 /dev/hugepages/spdk_pid217893map_1

-rw------- 1 root daos_admins 2.0M Sep 11 12:33 /dev/hugepages/spdk_pid217893map_10

-rw------- 1 root daos_admins 2.0M Sep 11 12:33 /dev/hugepages/spdk_pid217893map_11

-rw------- 1 root daos_admins 2.0M Sep 11 12:33 /dev/hugepages/spdk_pid217893map_12

-rw------- 1 root daos_admins 2.0M Sep 11 12:33 /dev/hugepages/spdk_pid217893map_2

...

• Then , enable daos_admin/privileged helper logging by setting "helper_log_file: /tmp/daos_admin.log" with the correct indentation towards the top of daos_server yaml config file.
• Stop daos_server instances/services.
• Wipe pmem with "sudo umount /mnt/daos[01]; sudo wipefs -a /dev/pmem[01]".
• Bind nvme devices back to kernel "sudo daos_server storage prepare -n --reset".
• Optionally reboot (maybe try without first).
• Start daos_server instances/services.
• Wait for reformat prompt "SCM format required...".
• Run "dmg storage format" from different tty.
• Wait for io instances to start/error then send logs including daos_admin.log please.

I realise you don’t need that level of instruction but included for completeness.

Regards,

Tom Nabarro – DCG/ESAD

M: +44 (0)7786 260986

Skype: tom.nabarro

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Gert Pauwels
Sent: Friday, September 11, 2020 5:42 PM
To: daos@daos.groups.io
Subject: Re: [daos] no dRPC client set on Ubuntu 20.04.1

Hi Tom,

After applying the patch you provided the error is now:
    09/11-17:47:17.20 intel-S2600WFD DAOS[11713/11713] bio  ERR  src/bio/bio_xst   09/11-17 ream.c:369 bio_spdk_env_init() Failed to init SPDK thread lib, Unknown error -1003 (-1003)

the original one was:

   DAOS[7642/7642] bio  ERR  src/bio/bio_xstream.c:367 bio_spdk_env_init() Failed to init SPDK thread lib, DER_INVAL(-1003)

The whole /tmp/daos_server looks like:

09/11-17:39:03.32 intel-S2600WFD DAOS[7642/7642] fi   INFO src/gurt/fault_inject.c:481 d_fault_inject_init() No config file, fault injection is OFF.
09/11-17:39:03.35 intel-S2600WFD DAOS[7642/7642] server INFO src/iosrv/init.c:419 set_abt_max_num_xstreams() Setting ABT_MAX_NUM_XSTREAMS to 4
09/11-17:39:03.35 intel-S2600WFD DAOS[7642/7642] server INFO src/iosrv/init.c:490 server_init() Module interface successfully initialized
09/11-17:39:03.36 intel-S2600WFD DAOS[7642/7642] server INFO src/iosrv/init.c:498 server_init() Module vos,rdb,rsvc,security,mgmt,dtx,pool,cont,obj,rebuild successfully loaded
09/11-17:39:03.36 intel-S2600WFD DAOS[7642/7642] crt  INFO src/cart/crt_init.c:269 crt_init_opt() libcart version 4.8.0 initializing
09/11-17:39:03.36 intel-S2600WFD DAOS[7642/7642] crt  WARN src/cart/crt_init.c:161 data_init() FI_UNIVERSE_SIZE was not set; setting to 2048
09/11-17:39:03.36 intel-S2600WFD DAOS[7642/7642] server INFO src/iosrv/init.c:507 server_init() Network successfully initialized
09/11-17:39:03.37 intel-S2600WFD DAOS[7642/7642] server INFO src/iosrv/init.c:516 server_init() Module vos,rdb,rsvc,security,mgmt,dtx,pool,cont,obj,rebuild successfully initialized
09/11-17:39:03.44 intel-S2600WFD DAOS[7642/7642] bio  ERR  src/bio/bio_xstream.c:367 bio_spdk_env_init() Failed to init SPDK thread lib, DER_INVAL(-1003)
09/11-17:39:03.44 intel-S2600WFD DAOS[7642/7642] server ERR  src/iosrv/init.c:521 server_init() DAOS cannot be initialized using the configured path (/mnt/daos).   Please ensure it is on a PMDK compatible file system and writeable by the current user
09/11-17:47:17.09 intel-S2600WFD DAOS[11713/11713] fi   INFO src/gurt/fault_inject.c:481 d_fault_inject_init() No config file, fault injection is OFF.
09/11-17:47:17.13 intel-S2600WFD DAOS[11713/11713] server INFO src/iosrv/init.c:419 set_abt_max_num_xstreams() Setting ABT_MAX_NUM_XSTREAMS to 4
09/11-17:47:17.13 intel-S2600WFD DAOS[11713/11713] server INFO src/iosrv/init.c:490 server_init() Module interface successfully initialized
09/11-17:47:17.13 intel-S2600WFD DAOS[11713/11713] server INFO src/iosrv/init.c:498 server_init() Module vos,rdb,rsvc,security,mgmt,dtx,pool,cont,obj,rebuild successfully loaded
09/11-17:47:17.13 intel-S2600WFD DAOS[11713/11713] crt  INFO src/cart/crt_init.c:269 crt_init_opt() libcart version 4.8.0 initializing
09/11-17:47:17.13 intel-S2600WFD DAOS[11713/11713] crt  WARN src/cart/crt_init.c:161 data_init() FI_UNIVERSE_SIZE was not set; setting to 2048
09/11-17:47:17.13 intel-S2600WFD DAOS[11713/11713] server INFO src/iosrv/init.c:507 server_init() Network successfully initialized
09/11-17:47:17.13 intel-S2600WFD DAOS[11713/11713] server INFO src/iosrv/init.c:516 server_init() Module vos,rdb,rsvc,security,mgmt,dtx,pool,cont,obj,rebuild successfully initialized
09/11-17:47:17.20 intel-S2600WFD DAOS[11713/11713] bio  ERR  src/bio/bio_xstream.c:369 bio_spdk_env_init() Failed to init SPDK thread lib, Unknown error -1003 (-1003)
09/11-17:47:17.21 intel-S2600WFD DAOS[11713/11713] server ERR  src/iosrv/init.c:521 server_init() DAOS cannot be initialized using the configured path (/mnt/daos).   Please ensure it is on a PMDK compatible file system and writeable by the current user

Rgds,

Gert

It sounds like the DAOS server is not able to register memory to initiate a RDMA. Could you please tell me more about the network and storage you use on the server? Optane PMEM or DRAM? Also, what version of OFED do you use?

Cheers,

Johann

This is errors from daos_server.log :

Thanks Gurt,

Yes, that was only to get the identify application running, which looks like it runs fine as expected given the problem looks like initializing the thread library which identify doesn’t do.

Regards,

Tom Nabarro – DCG/ESAD

M: +44 (0)7786 260986

Skype: tom.nabarro

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Gert Pauwels
Sent: Friday, September 11, 2020 1:41 PM
To: daos@daos.groups.io
Subject: Re: [daos] no dRPC client set on Ubuntu 20.04.1

Hi Tom,

I set the variable LD_LIBRARY_PATH as you indicated to spdk library path and that indeed solves the problem for the identify application. Now the identify application returns all the data as expected.

It did not make any difference  when running:

It still the same  error

Rgds,

Gert

Hello,

You and Gurt are experiencing the same issue:

09/10-14:23:55.96 intel-S2600WFD DAOS[260088/260088] bio  ERR  src/bio/bio_xstream.c:367 bio_spdk_env_init() Failed to init SPDK thread lib, DER_INVAL(-1003)

09/10-14:23:55.97 intel-S2600WFD DAOS[260088/260088] server ERR  src/iosrv/init.c:521 server_init() DAOS cannot be initialized using the configured path (/mnt/daos).   Please ensure it is on a PMDK compatible file system and writeable by the current user

Which is that the SPDK thread library cannot be initialised, the thread library is not used in the identify example so that would explain why it runs fine.

spdk_thread_lib_init() is returning an error but unfortunately we wrap it with DER_INVAL and mask the spdk error, which needs fixing.

Can you please apply this patch and rerun to give us a better insight and in the meantime I will find an Ubuntu system to verify:

```

diff --git a/src/bio/bio_xstream.c b/src/bio/bio_xstream.c

index 758ba6d..c888047 100644

--- a/src/bio/bio_xstream.c

+++ b/src/bio/bio_xstream.c

@@ -30,6 +30,7 @@

#include <spdk/env.h>

#include <spdk/nvme.h>

#include <spdk/vmd.h>

+#include <spdk/string.h>

#include <spdk/thread.h>

#include <spdk/bdev.h>

#include <spdk/io_channel.h>

@@ -364,7 +365,8 @@ bio_spdk_env_init(void)

        rc = spdk_thread_lib_init(NULL, 0);

        if (rc != 0) {

                rc = -DER_INVAL;

-               D_ERROR("Failed to init SPDK thread lib, "DF_RC"\n", DP_RC(rc));

+               D_ERROR("Failed to init SPDK thread lib, %s (%d)\n",

+                       spdk_strerror(rc), rc);

                spdk_env_fini();

                return rc;

        }

```

Regards,

Tom Nabarro – DCG/ESAD

M: +44 (0)7786 260986

Skype: tom.nabarro

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Yunjae Lee
Sent: Friday, September 11, 2020 8:34 AM
To: daos@daos.groups.io
Subject: Re: [daos] no dRPC client set on Ubuntu 20.04.1

Same problem here, I used the master branch of daos on ubuntu 20.04.1
I also met this error when starting the daos_server:

ERROR: removing socket file: removing instance 0 socket file: no dRPC client set (data plane not started?)

The same message was printed in /tmp/daos_server.log:

bio ERR src/bio/bio_xstream.c:367 bios_spdk_env_init() Failed to init SPDK thread lib, DER_INVAL(-1003)

The identify error also happened in my case, however, it is now solved thanks to @Tom.
The results of the identify is attached, as well as the server logs and configurations i used.

Looks better. The network stack seems to be properly initialized now.

The failure you see now (“undefined symbol: dmg_pool_create”) is related to the C wrapper of the dmg command which is used to create a pool. I suspect that you are missing something in the LD_LIBRARY_PATH. I will let @Macdonald, Mjmac comment further on that.

Cheers,

Johann

UPDATE: I fixed OFI_DOMAIN to mlx5_0

Here is new error:

[root@sky08 ~]# mpirun -np 1 --allow-run-as-root /tmp/daos_test

--------------------------------------------------------------------------

WARNING: No preset parameters were found for the device that Open MPI

detected:

  Local host:            sky08

  Device name:           mlx5_0

  Device vendor ID:      0x02c9

  Device vendor part ID: 4123

Default device parameters will be used, which may result in lower

performance.  You can edit any of the files specified by the

btl_openib_device_param_files MCA parameter to set values for your

device.

NOTE: You can turn off this warning by setting the MCA parameter

      btl_openib_warn_no_device_params_found to 0.

--------------------------------------------------------------------------

--------------------------------------------------------------------------

No OpenFabrics connection schemes reported that they were able to be

used on a specific port.  As such, the openib BTL (OpenFabrics

support) will be disabled for this port.

  Local host:           sky08

  Local device:         i40iw0

  Local port:           1

  CPCs attempted:       rdmacm, udcm

--------------------------------------------------------------------------

=================

DAOS management tests..

=====================

[==========] Running 5 test(s).

[ RUN      ] MGMT1: create/destroy pool on all tgts

/tmp/daos_test: symbol lookup error: /tmp/daos_test: undefined symbol: dmg_pool_create

creating pool synchronously ... --------------------------------------------------------------------------

Primary job  terminated normally, but 1 process returned

a non-zero exit code. Per user-direction, the job has been aborted.

--------------------------------------------------------------------------

--------------------------------------------------------------------------

mpirun detected that one or more processes exited with non-zero status, thus causing

the job to be terminated. The first process to do so was:

  Process name: [[48263,1],0]

  Exit code:    127

--------------------------------------------------------------------------

[sky08:35443] 2 more processes have sent help message help-mpi-btl-openib.txt / no device params found

[sky08:35443] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

Here is the daos.log:

09/11-13:11:00.14 sky08 DAOS[28284/28284] hg   ERR  src/cart/crt_hg.c:534 crt_hg_init() Could not initialize NA class.

09/11-13:11:00.14 sky08 DAOS[28284/28284] crt  ERR  src/cart/crt_init.c:409 crt_init_opt() crt_hg_init failed rc: -1020.

09/11-13:11:00.14 sky08 DAOS[28284/28284] crt  ERR  src/cart/crt_init.c:477 crt_init_opt() crt_init failed, rc: -1020.

09/11-13:11:00.14 sky08 DAOS[28284/28284] fi   WARN src/gurt/fault_inject.c:685 d_fault_inject_fini() Fault Injection not finalized feature not included in build

09/11-13:11:00.14 sky08 DAOS[28284/28284] client ERR  src/client/api/event.c:93 daos_eq_lib_init() failed to initialize crt: DER_HG(-1020)

09/11-13:11:00.14 sky08 DAOS[28284/28284] client ERR  src/client/api/init.c:160 daos_init() failed to initialize eq_lib: DER_HG(-1020)

09/11-13:11:00.14 sky08 DAOS[28284/28284] fi   WARN src/gurt/fault_inject.c:685 d_fault_inject_fini() Fault Injection not finalized feature not included in build

09/11-13:23:18.51 sky08 DAOS[29621/29621] fi   WARN src/gurt/fault_inject.c:679 d_fault_inject_init() Fault Injection not initialized feature not included in build

09/11-13:23:18.51 sky08 DAOS[29621/29621] fi   WARN src/gurt/fault_inject.c:716 d_fault_attr_set() Fault Injection attr not set feature not included in build

09/11-13:23:18.51 sky08 DAOS[29621/29621] crt  INFO src/cart/crt_init.c:278 crt_init_opt() libcart version 4.8.0 initializing

09/11-13:23:18.51 sky08 DAOS[29621/29621] fi   WARN src/gurt/fault_inject.c:679 d_fault_inject_init() Fault Injection not initialized feature not included in build

09/11-13:23:18.51 sky08 DAOS[29621/29621] crt  WARN src/cart/crt_init.c:170 data_init() FI_UNIVERSE_SIZE was not set; setting to 2048

09/11-13:23:18.51 sky08 DAOS[29621/29621] crt  WARN src/cart/crt_init.c:389 crt_init_opt() FI_OFI_RXM_USE_SRX not set; set=1

09/11-13:23:18.51 sky08 DAOS[29621/29621] hg   ERR  # NA -- Error -- /builddir/build/BUILD/mercury-2.0.0a1/src/na/na_ofi.c:1685

 # na_ofi_domain_open(): No provider found for "verbs;ofi_rxm" provider on domain "mlx_5"

09/11-13:23:18.51 sky08 DAOS[29621/29621] hg   ERR  # NA -- Error -- /builddir/build/BUILD/mercury-2.0.0a1/src/na/na_ofi.c:3150

 # na_ofi_initialize(): Could not open domain for verbs;ofi_rxm, mlx_5

09/11-13:23:18.51 sky08 DAOS[29621/29621] hg   ERR  # NA -- Error -- /builddir/build/BUILD/mercury-2.0.0a1/src/na/na.c:312

 # NA_Initialize_opt(): Could not initialize plugin

09/11-13:23:18.51 sky08 DAOS[29621/29621] hg   ERR  src/cart/crt_hg.c:534 crt_hg_init() Could not initialize NA class.

09/11-13:23:18.51 sky08 DAOS[29621/29621] crt  ERR  src/cart/crt_init.c:409 crt_init_opt() crt_hg_init failed rc: -1020.

09/11-13:23:18.51 sky08 DAOS[29621/29621] crt  ERR  src/cart/crt_init.c:477 crt_init_opt() crt_init failed, rc: -1020.

09/11-13:23:18.51 sky08 DAOS[29621/29621] fi   WARN src/gurt/fault_inject.c:685 d_fault_inject_fini() Fault Injection not finalized feature not included in build

09/11-13:23:18.51 sky08 DAOS[29621/29621] client ERR  src/client/api/event.c:93 daos_eq_lib_init() failed to initialize crt: DER_HG(-1020)

09/11-13:23:18.51 sky08 DAOS[29621/29621] client ERR  src/client/api/init.c:160 daos_init() failed to initialize eq_lib: DER_HG(-1020)

09/11-13:23:18.51 sky08 DAOS[29621/29621] fi   WARN src/gurt/fault_inject.c:685 d_fault_inject_fini() Fault Injection not finalized feature not included in build

09/11-13:26:13.98 sky08 DAOS[29976/29976] fi   WARN src/gurt/fault_inject.c:679 d_fault_inject_init() Fault Injection not initialized feature not included in build

09/11-13:26:13.98 sky08 DAOS[29976/29976] fi   WARN src/gurt/fault_inject.c:716 d_fault_attr_set() Fault Injection attr not set feature not included in build

09/11-13:26:13.98 sky08 DAOS[29976/29976] crt  INFO src/cart/crt_init.c:278 crt_init_opt() libcart version 4.8.0 initializing

09/11-13:26:13.98 sky08 DAOS[29976/29976] fi   WARN src/gurt/fault_inject.c:679 d_fault_inject_init() Fault Injection not initialized feature not included in build

09/11-13:26:13.98 sky08 DAOS[29976/29976] crt  WARN src/cart/crt_init.c:170 data_init() FI_UNIVERSE_SIZE was not set; setting to 2048

09/11-13:26:13.98 sky08 DAOS[29976/29976] crt  WARN src/cart/crt_init.c:389 crt_init_opt() FI_OFI_RXM_USE_SRX not set; set=1

09/11-13:26:13.98 sky08 DAOS[29976/29976] hg   ERR  # NA -- Error -- /builddir/build/BUILD/mercury-2.0.0a1/src/na/na_ofi.c:1685

 # na_ofi_domain_open(): No provider found for "verbs;ofi_rxm" provider on domain "mlx_5"

09/11-13:26:13.98 sky08 DAOS[29976/29976] hg   ERR  # NA -- Error -- /builddir/build/BUILD/mercury-2.0.0a1/src/na/na_ofi.c:3150

 # na_ofi_initialize(): Could not open domain for verbs;ofi_rxm, mlx_5

09/11-13:26:13.98 sky08 DAOS[29976/29976] hg   ERR  # NA -- Error -- /builddir/build/BUILD/mercury-2.0.0a1/src/na/na.c:312

 # NA_Initialize_opt(): Could not initialize plugin

09/11-13:26:13.98 sky08 DAOS[29976/29976] hg   ERR  src/cart/crt_hg.c:534 crt_hg_init() Could not initialize NA class.

09/11-13:26:13.98 sky08 DAOS[29976/29976] crt  ERR  src/cart/crt_init.c:409 crt_init_opt() crt_hg_init failed rc: -1020.

09/11-13:26:13.98 sky08 DAOS[29976/29976] crt  ERR  src/cart/crt_init.c:477 crt_init_opt() crt_init failed, rc: -1020.

09/11-13:26:13.98 sky08 DAOS[29976/29976] fi   WARN src/gurt/fault_inject.c:685 d_fault_inject_fini() Fault Injection not finalized feature not included in build

09/11-13:26:13.98 sky08 DAOS[29976/29976] client ERR  src/client/api/event.c:93 daos_eq_lib_init() failed to initialize crt: DER_HG(-1020)

09/11-13:26:13.98 sky08 DAOS[29976/29976] client ERR  src/client/api/init.c:160 daos_init() failed to initialize eq_lib: DER_HG(-1020)

09/11-13:26:13.98 sky08 DAOS[29976/29976] fi   WARN src/gurt/fault_inject.c:685 d_fault_inject_fini() Fault Injection not finalized feature not included in build

09/11-14:17:47.58 sky08 DAOS[35448/35448] fi   WARN src/gurt/fault_inject.c:679 d_fault_inject_init() Fault Injection not initialized feature not included in build

09/11-14:17:47.58 sky08 DAOS[35448/35448] fi   WARN src/gurt/fault_inject.c:716 d_fault_attr_set() Fault Injection attr not set feature not included in build

09/11-14:17:47.59 sky08 DAOS[35448/35448] crt  INFO src/cart/crt_init.c:278 crt_init_opt() libcart version 4.8.0 initializing

09/11-14:17:47.59 sky08 DAOS[35448/35448] fi   WARN src/gurt/fault_inject.c:679 d_fault_inject_init() Fault Injection not initialized feature not included in build

09/11-14:17:47.59 sky08 DAOS[35448/35448] crt  WARN src/cart/crt_init.c:170 data_init() FI_UNIVERSE_SIZE was not set; setting to 2048

09/11-14:17:47.59 sky08 DAOS[35448/35448] crt  WARN src/cart/crt_init.c:389 crt_init_opt() FI_OFI_RXM_USE_SRX not set; set=1

Hi,

we run 1.0.1 with these settings:

# DAOS 1.0.1 client-side settings for Mellanox InfiniBand:

export OFI_INTERFACE="ib0"

export CRT_PHY_ADDR_STR="ofi+verbs;ofi_rxm"

export OFI_DOMAIN="mlx5_0"

What does “daos_server network scan” say?

Mit freundlichen Grüssen / Best regards,

Michael Hennecke

Chief Technologist - HPC Storage & Networking

mailto:mhennecke@...

--

Lenovo Global Technology (Germany) GmbH * Am Zehnthof 77 * D-45307 Essen * Germany

Geschäftsführung: Colm Gleeson, Christophe Laurent * Sitz der Gesellschaft: Stuttgart * HRB-Nr.: 758298, AG Stuttgart

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of anton.brekhov@...
Sent: Friday, September 11, 2020 1:21 PM
To: daos@daos.groups.io
Subject: [External] Re: [daos] DAOS_test failed

UPDATE: I fixed OFI_DOMAIN to mlx5_0

Here is new error:

[root@sky08 ~]# mpirun -np 1 --allow-run-as-root /tmp/daos_test

--------------------------------------------------------------------------

WARNING: No preset parameters were found for the device that Open MPI

detected:

  Local host:            sky08

  Device name:           mlx5_0

  Device vendor ID:      0x02c9

  Device vendor part ID: 4123

Default device parameters will be used, which may result in lower

performance.  You can edit any of the files specified by the

btl_openib_device_param_files MCA parameter to set values for your

device.

NOTE: You can turn off this warning by setting the MCA parameter

      btl_openib_warn_no_device_params_found to 0.

--------------------------------------------------------------------------

--------------------------------------------------------------------------

No OpenFabrics connection schemes reported that they were able to be

used on a specific port.  As such, the openib BTL (OpenFabrics

support) will be disabled for this port.

  Local host:           sky08

  Local device:         i40iw0

  Local port:           1

  CPCs attempted:       rdmacm, udcm

--------------------------------------------------------------------------

=================

DAOS management tests..

=====================

[==========] Running 5 test(s).

[ RUN      ] MGMT1: create/destroy pool on all tgts

/tmp/daos_test: symbol lookup error: /tmp/daos_test: undefined symbol: dmg_pool_create

creating pool synchronously ... --------------------------------------------------------------------------

Primary job  terminated normally, but 1 process returned

a non-zero exit code. Per user-direction, the job has been aborted.

--------------------------------------------------------------------------

--------------------------------------------------------------------------

mpirun detected that one or more processes exited with non-zero status, thus causing

the job to be terminated. The first process to do so was:

  Process name: [[48263,1],0]

  Exit code:    127

--------------------------------------------------------------------------

[sky08:35443] 2 more processes have sent help message help-mpi-btl-openib.txt / no device params found

[sky08:35443] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

Here is the daos.log:

09/11-13:11:00.14 sky08 DAOS[28284/28284] hg   ERR  src/cart/crt_hg.c:534 crt_hg_init() Could not initialize NA class.

09/11-13:11:00.14 sky08 DAOS[28284/28284] crt  ERR  src/cart/crt_init.c:409 crt_init_opt() crt_hg_init failed rc: -1020.

09/11-13:11:00.14 sky08 DAOS[28284/28284] crt  ERR  src/cart/crt_init.c:477 crt_init_opt() crt_init failed, rc: -1020.

09/11-13:11:00.14 sky08 DAOS[28284/28284] fi   WARN src/gurt/fault_inject.c:685 d_fault_inject_fini() Fault Injection not finalized feature not included in build

09/11-13:11:00.14 sky08 DAOS[28284/28284] client ERR  src/client/api/event.c:93 daos_eq_lib_init() failed to initialize crt: DER_HG(-1020)

09/11-13:11:00.14 sky08 DAOS[28284/28284] client ERR  src/client/api/init.c:160 daos_init() failed to initialize eq_lib: DER_HG(-1020)

09/11-13:11:00.14 sky08 DAOS[28284/28284] fi   WARN src/gurt/fault_inject.c:685 d_fault_inject_fini() Fault Injection not finalized feature not included in build

09/11-13:23:18.51 sky08 DAOS[29621/29621] fi   WARN src/gurt/fault_inject.c:679 d_fault_inject_init() Fault Injection not initialized feature not included in build

09/11-13:23:18.51 sky08 DAOS[29621/29621] fi   WARN src/gurt/fault_inject.c:716 d_fault_attr_set() Fault Injection attr not set feature not included in build

09/11-13:23:18.51 sky08 DAOS[29621/29621] crt  INFO src/cart/crt_init.c:278 crt_init_opt() libcart version 4.8.0 initializing

09/11-13:23:18.51 sky08 DAOS[29621/29621] fi   WARN src/gurt/fault_inject.c:679 d_fault_inject_init() Fault Injection not initialized feature not included in build

09/11-13:23:18.51 sky08 DAOS[29621/29621] crt  WARN src/cart/crt_init.c:170 data_init() FI_UNIVERSE_SIZE was not set; setting to 2048

09/11-13:23:18.51 sky08 DAOS[29621/29621] crt  WARN src/cart/crt_init.c:389 crt_init_opt() FI_OFI_RXM_USE_SRX not set; set=1

09/11-13:23:18.51 sky08 DAOS[29621/29621] hg   ERR  # NA -- Error -- /builddir/build/BUILD/mercury-2.0.0a1/src/na/na_ofi.c:1685

 # na_ofi_domain_open(): No provider found for "verbs;ofi_rxm" provider on domain "mlx_5"

09/11-13:23:18.51 sky08 DAOS[29621/29621] hg   ERR  # NA -- Error -- /builddir/build/BUILD/mercury-2.0.0a1/src/na/na_ofi.c:3150

 # na_ofi_initialize(): Could not open domain for verbs;ofi_rxm, mlx_5

09/11-13:23:18.51 sky08 DAOS[29621/29621] hg   ERR  # NA -- Error -- /builddir/build/BUILD/mercury-2.0.0a1/src/na/na.c:312

 # NA_Initialize_opt(): Could not initialize plugin

09/11-13:23:18.51 sky08 DAOS[29621/29621] hg   ERR  src/cart/crt_hg.c:534 crt_hg_init() Could not initialize NA class.

09/11-13:23:18.51 sky08 DAOS[29621/29621] crt  ERR  src/cart/crt_init.c:409 crt_init_opt() crt_hg_init failed rc: -1020.

09/11-13:23:18.51 sky08 DAOS[29621/29621] crt  ERR  src/cart/crt_init.c:477 crt_init_opt() crt_init failed, rc: -1020.

09/11-13:23:18.51 sky08 DAOS[29621/29621] fi   WARN src/gurt/fault_inject.c:685 d_fault_inject_fini() Fault Injection not finalized feature not included in build

09/11-13:23:18.51 sky08 DAOS[29621/29621] client ERR  src/client/api/event.c:93 daos_eq_lib_init() failed to initialize crt: DER_HG(-1020)

09/11-13:23:18.51 sky08 DAOS[29621/29621] client ERR  src/client/api/init.c:160 daos_init() failed to initialize eq_lib: DER_HG(-1020)

09/11-13:23:18.51 sky08 DAOS[29621/29621] fi   WARN src/gurt/fault_inject.c:685 d_fault_inject_fini() Fault Injection not finalized feature not included in build

09/11-13:26:13.98 sky08 DAOS[29976/29976] fi   WARN src/gurt/fault_inject.c:679 d_fault_inject_init() Fault Injection not initialized feature not included in build

09/11-13:26:13.98 sky08 DAOS[29976/29976] fi   WARN src/gurt/fault_inject.c:716 d_fault_attr_set() Fault Injection attr not set feature not included in build

09/11-13:26:13.98 sky08 DAOS[29976/29976] crt  INFO src/cart/crt_init.c:278 crt_init_opt() libcart version 4.8.0 initializing

09/11-13:26:13.98 sky08 DAOS[29976/29976] fi   WARN src/gurt/fault_inject.c:679 d_fault_inject_init() Fault Injection not initialized feature not included in build

09/11-13:26:13.98 sky08 DAOS[29976/29976] crt  WARN src/cart/crt_init.c:170 data_init() FI_UNIVERSE_SIZE was not set; setting to 2048

09/11-13:26:13.98 sky08 DAOS[29976/29976] crt  WARN src/cart/crt_init.c:389 crt_init_opt() FI_OFI_RXM_USE_SRX not set; set=1

09/11-13:26:13.98 sky08 DAOS[29976/29976] hg   ERR  # NA -- Error -- /builddir/build/BUILD/mercury-2.0.0a1/src/na/na_ofi.c:1685

 # na_ofi_domain_open(): No provider found for "verbs;ofi_rxm" provider on domain "mlx_5"

09/11-13:26:13.98 sky08 DAOS[29976/29976] hg   ERR  # NA -- Error -- /builddir/build/BUILD/mercury-2.0.0a1/src/na/na_ofi.c:3150

 # na_ofi_initialize(): Could not open domain for verbs;ofi_rxm, mlx_5

09/11-13:26:13.98 sky08 DAOS[29976/29976] hg   ERR  # NA -- Error -- /builddir/build/BUILD/mercury-2.0.0a1/src/na/na.c:312

 # NA_Initialize_opt(): Could not initialize plugin

09/11-13:26:13.98 sky08 DAOS[29976/29976] hg   ERR  src/cart/crt_hg.c:534 crt_hg_init() Could not initialize NA class.

09/11-13:26:13.98 sky08 DAOS[29976/29976] crt  ERR  src/cart/crt_init.c:409 crt_init_opt() crt_hg_init failed rc: -1020.

09/11-13:26:13.98 sky08 DAOS[29976/29976] crt  ERR  src/cart/crt_init.c:477 crt_init_opt() crt_init failed, rc: -1020.

09/11-13:26:13.98 sky08 DAOS[29976/29976] fi   WARN src/gurt/fault_inject.c:685 d_fault_inject_fini() Fault Injection not finalized feature not included in build

09/11-13:26:13.98 sky08 DAOS[29976/29976] client ERR  src/client/api/event.c:93 daos_eq_lib_init() failed to initialize crt: DER_HG(-1020)

09/11-13:26:13.98 sky08 DAOS[29976/29976] client ERR  src/client/api/init.c:160 daos_init() failed to initialize eq_lib: DER_HG(-1020)

09/11-13:26:13.98 sky08 DAOS[29976/29976] fi   WARN src/gurt/fault_inject.c:685 d_fault_inject_fini() Fault Injection not finalized feature not included in build

09/11-14:17:47.58 sky08 DAOS[35448/35448] fi   WARN src/gurt/fault_inject.c:679 d_fault_inject_init() Fault Injection not initialized feature not included in build

09/11-14:17:47.58 sky08 DAOS[35448/35448] fi   WARN src/gurt/fault_inject.c:716 d_fault_attr_set() Fault Injection attr not set feature not included in build

09/11-14:17:47.59 sky08 DAOS[35448/35448] crt  INFO src/cart/crt_init.c:278 crt_init_opt() libcart version 4.8.0 initializing

09/11-14:17:47.59 sky08 DAOS[35448/35448] fi   WARN src/gurt/fault_inject.c:679 d_fault_inject_init() Fault Injection not initialized feature not included in build

09/11-14:17:47.59 sky08 DAOS[35448/35448] crt  WARN src/cart/crt_init.c:170 data_init() FI_UNIVERSE_SIZE was not set; setting to 2048

09/11-14:17:47.59 sky08 DAOS[35448/35448] crt  WARN src/cart/crt_init.c:389 crt_init_opt() FI_OFI_RXM_USE_SRX not set; set=1

Could you please try with export OFI_DOMAIN="mlx5_0"?

If it does not work, please collect the the daos logs (i.e. /tmp/daos.log).

Cheers,

Johann

Johann thanks!

I've set export CRT_PHY_ADDR_STR="ofi+verbs;ofi_rxm" both on storage server and client and now I'get an error -1020:

export OFI_INTERFACE=ib0

export CRT_PHY_ADDR_STR="ofi+verbs;ofi_rxm"

export OFI_DOMAIN=mlx_5

mpirun -np 1 --allow-run-as-root /tmp/daos_test

--------------------------------------------------------------------------

WARNING: No preset parameters were found for the device that Open MPI

detected:

  Local host:            sky08

  Device name:           mlx5_0

  Device vendor ID:      0x02c9

  Device vendor part ID: 4123

Default device parameters will be used, which may result in lower

performance.  You can edit any of the files specified by the

btl_openib_device_param_files MCA parameter to set values for your

device.

NOTE: You can turn off this warning by setting the MCA parameter

      btl_openib_warn_no_device_params_found to 0.

--------------------------------------------------------------------------

[sky08:29616] [[17732,0],0] ORTE_ERROR_LOG: Out of resource in file util/show_help.c at line 501

--------------------------------------------------------------------------

No OpenFabrics connection schemes reported that they were able to be

used on a specific port.  As such, the openib BTL (OpenFabrics

support) will be disabled for this port.

  Local host:           sky08

  Local device:         i40iw0

  Local port:           1

  CPCs attempted:       rdmacm, udcm

--------------------------------------------------------------------------

daos_init() failed with -1020

--------------------------------------------------------------------------

Primary job  terminated normally, but 1 process returned

a non-zero exit code. Per user-direction, the job has been aborted.

--------------------------------------------------------------------------

--------------------------------------------------------------------------

mpirun detected that one or more processes exited with non-zero status, thus causing

the job to be terminated. The first process to do so was:

  Process name: [[17732,1],0]

  Exit code:    255

--------------------------------------------------------------------------

[sky08:29616] 1 more process has sent help message help-mpi-btl-openib.txt / no device params found

[sky08:29616] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

Hi Anton,

Since you use 1.0.1, you also need to set CRT_PHY_ADDR_STR  (not required any longer in 1.2/master) as follows:

export CRT_PHY_ADDR_STR="ofi+verbs;ofi_rxm"

Hope this helps.

Cheers,

Johann

Hi everyone!

Im trying to make simple cluster of two nodes connected to IB network. One node is with PMEM and other one is client (for daos agent). I installed rpm for centos 7 v1.0.1

Here is DAOS_server config:

name: daos_server

access_points: ['apache512']

#access_points: ['localhost']

port: 10001

#provider: ofi+sockets

provider: ofi+verbs;ofi_rxm

nr_hugepages: 4096

control_log_file: /tmp/daos_control.log

transport_config:

   allow_insecure: true

servers:

-

  targets: 4

  first_core: 0

  nr_xs_helpers: 0

  fabric_iface: ib0

  fabric_iface_port: 31416

  log_mask: DEBUG

  log_file: /tmp/daos_server.log

  env_vars:

  - DAOS_MD_CAP=1024

  - CRT_CTX_SHARE_ADDR=0

  - CRT_TIMEOUT=30

  - FI_SOCKETS_MAX_CONN_RETRY=1

  - FI_SOCKETS_CONN_TIMEOUT=2000

  #- OFI_INTERFACE=ib0

  #- OFI_DOMAIN=mlx5_0

  #- CRT_PHY_ADDR_STR=ofi+verbs

  # Storage definitions

  # When scm_class is set to ram, tmpfs will be used to emulate SCM.

  # The size of ram is specified by scm_size in GB units.

  scm_mount: /mnt/daos  # map to -s /mnt/daos

  #scm_class: ram

  #scm_size: 8

  scm_class: dcpm

  scm_list: [/dev/pmem0]

  bdev_class: nvme

  bdev_list: ["0000:b1:00.0","0000:b2:00.0","0000:b3:00.0","0000:b4:00.0"]

Here is daos_agent config on client node:

name: daos_server

access_points: ['apache512']

port: 10001

runtime_dir: /var/run/daos_agent

On the client node i'm trying to run daos_test as follow:

export CRT_TIMEOUT=5

export OFI_INTERFACE=ib0

export OFI_DOMAIN=mlx_5
module load mpi/openmpi3-x86_64
mpirun -np 1 --allow-run-as-root /tmp/daos_test

Here is output:

[root@sky08 ~]# mpirun -np 1 --allow-run-as-root /tmp/daos_test

--------------------------------------------------------------------------

WARNING: No preset parameters were found for the device that Open MPI

detected:

  Local host:            sky08

  Device name:           mlx5_0

  Device vendor ID:      0x02c9

  Device vendor part ID: 4123

Default device parameters will be used, which may result in lower

performance.  You can edit any of the files specified by the

btl_openib_device_param_files MCA parameter to set values for your

device.

NOTE: You can turn off this warning by setting the MCA parameter

      btl_openib_warn_no_device_params_found to 0.

--------------------------------------------------------------------------

--------------------------------------------------------------------------

No OpenFabrics connection schemes reported that they were able to be

used on a specific port.  As such, the openib BTL (OpenFabrics

support) will be disabled for this port.

  Local host:           sky08

  Local device:         i40iw0

  Local port:           1

  CPCs attempted:       rdmacm, udcm

--------------------------------------------------------------------------

daos_init() failed with -1003

--------------------------------------------------------------------------

Primary job  terminated normally, but 1 process returned

a non-zero exit code. Per user-direction, the job has been aborted.

--------------------------------------------------------------------------

--------------------------------------------------------------------------

mpirun detected that one or more processes exited with non-zero status, thus causing

the job to be terminated. The first process to do so was:

  Process name: [[27909,1],0]

  Exit code:    255

--------------------------------------------------------------------------

[sky08:23537] 2 more processes have sent help message help-mpi-btl-openib.txt / no device params found

[sky08:23537] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

1003 error is invalid parameter as I understand from here (https://github.com/daos-stack/cart/blob/master/src/include/gurt/errno.h). What can be wrong? fi_pingpong test is OK... 

Thanks!

Anton Brekhov