Date   

Re: [External] Re: [daos] Does DAOS support infiniband now?

Oganezov, Alexander A
 

Hi Shengyu,

 

After talking to mercury developer, it sounds like you still might have a mismatch between versions of cart and mercury that you are using.

Can you do a fresh build of daos by first fully removing install/ and _build_external.Linux/ directories and see if that solves the issue?

 

Thanks,

~~Alex.

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang
Sent: Tuesday, January 21, 2020 12:22 AM
To: daos@daos.groups.io
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Hello Alex,

 

Please see the log attached.

 

Regards,

Shengyu.

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Oganezov, Alexander A
Sent: Tuesday, January 21, 2020 3:26 PM
To: daos@daos.groups.io
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Shengyu,

 

Can you provide full log from start until first occurrence of this error?

 

Thanks,

~~Alex.

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang
Sent: Monday, January 20, 2020 11:02 PM
To: daos@daos.groups.io
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Hello Alex,

 

I have removed mercury, the io_server seems started normally ,however the daos_server.log increasing quickly to eat all my free space. It infinite repeat the three lines:

01/21-01:40:49.28 afa1 DAOS[72964/73009] hg   ERR  src/cart/crt_hg.c:1331 crt_hg_trigger() HG_Trigger failed, hg_ret: 18.
01/21-01:40:49.28 afa1 DAOS[72964/73009] rpc  ERR  src/cart/crt_context.c:1255 crt_progress() crt_hg_progress failed, rc: -1020.
01/21-01:40:49.28 afa1 DAOS[72964/73009] server ERR  src/iosrv/srv.c:469 dss_srv_handler() failed to progress CART context: -1020

 

Regards,

Shengyu,

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Oganezov, Alexander A
Sent: Tuesday, January 21, 2020 4:49 AM
To: daos@daos.groups.io
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

One other thing Shengyu,

 

Can you verify that your mercury build is up to date? Based on domain being used as “mlx5_0/192.168.80.161” it sounds like there is mismatch between what CaRT generates and what mercury consumes; there has been change few weeks ago regarding how domain is provided to mercury level, and it feels as if older mercury is being used.

 

To ensure clean mercury build remove libmercury* libna*  from your install/lib location, remove _build.external-Linux/mercury directory and recompile daos with scons –build-deps=yes –config=force install

 

Thanks,

~~Alex.

 

From: Oganezov, Alexander A
Sent: Monday, January 20, 2020 11:56 AM
To: daos@daos.groups.io
Subject: RE: [External] Re: [daos] Does DAOS support infiniband now?

 

Hi Shengyu,

 

What does this command return to you on the node where you see this strange domain name?

fi_info --provider="verbs"

 

Thanks,

~~Alex.

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang
Sent: Monday, January 20, 2020 12:32 AM
To: daos@daos.groups.io
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Hello Joel,

 

I have some more information,

In the file na_ofi.c + 1609, the io_serve exit there, and on above code, na_ofi_verify_provider function will compare domain names, the domain is “mlx5_0/192.168.80.161”, while prov->domain_attr.name is “mlx5_0”,  it will always return FALSE.

 

Regards,

Shengyu

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang
Sent: Sunday, January 19, 2020 6:15 PM
To: daos@daos.groups.io
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Hello Joel,

 

Network scan:

Scanning fabric for YML specified provider: ofi+verbs;ofi_rxm
Fabric scan found 1 devices matching the provider spec: ofi+verbs;ofi_rxm

fabric_iface: ib0
provider: ofi+verbs;ofi_rxm
pinned_numa_node: 1

 

 

And log attached, it is standalone server.

Additionally, I created two identical virtual machines with IB sr-iov pass-through, however one vm can’t start due to the same problem, while another can start normally. They were using ofi+verbs;ofi_rxm as provider.

 

 

Regards,

Shengyu

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Rosenzweig, Joel B
Sent: Friday, January 17, 2020 11:03 PM
To: daos@daos.groups.io
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Hello Shengyu,

 

It appears that your daos_server.yml is specifying the provider as “ofi+verbs” but I think it should be set to “ofi+verbs;ofi_rxm”.  Can you configure your daos_server.yml with that and try again?  And then, if things still do not work, then please provide:

 

1)     the network scan output again after you make the provider change

2)     the portion of the debug log that shows the environment variables being provided to the daos_io_server.  This will show us what is being set for OFI_INTERFACE, OFI_DOMAIN in the daos_io_server environment.

3)     the daos_server.yml so I can see how you have configured each daos_io_server.

 

Regards,

Joel

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang
Sent: Friday, January 17, 2020 4:55 AM
To: daos@daos.groups.io
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Hello Johann,

 

Here is logs:

 

Scanning fabric for YML specified provider: ofi+verbs
Fabric scan found 4 devices matching the provider spec: ofi+verbs

fabric_iface: ib2
provider: ofi+verbs;ofi_rxm
pinned_numa_node: 1


fabric_iface: ib0
provider: ofi+verbs;ofi_rxm
pinned_numa_node: 1


fabric_iface: ib2
provider: ofi+verbs
pinned_numa_node: 1


fabric_iface: ib0
provider: ofi+verbs
pinned_numa_node: 1

 

And please see my inline comments.

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Lombardi, Johann
Sent: Friday, January 17, 2020 4:51 PM
To: daos@daos.groups.io; Rosenzweig, Joel B <joel.b.rosenzweig@...>
Cc: Macdonald, Mjmac <mjmac.macdonald@...>
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Hi,

 

Cc’ing Joel.

Those errors indicate that you still have some network setup issue. Could you please run daos_server network scan?

 

e.g.:

[root@wolf-118 ~]# daos_server network scan -p "ofi+verbs;ofi_rxm"

Scanning fabric for cmdline specified provider: ofi+verbs;ofi_rxm

Fabric scan found 3 devices matching the provider spec: ofi+verbs;ofi_rxm

 

                fabric_iface: ib0

                provider: ofi+verbs;ofi_rxm

                pinned_numa_node: 0

 

 

                fabric_iface: ib1

                provider: ofi+verbs;ofi_rxm

                pinned_numa_node: 1

 

 

                fabric_iface: eth0

                provider: ofi+verbs;ofi_rxm

                pinned_numa_node: 0

 

As for the NVMe issue, do I understand correctly that:

-        The PCI addresses of the 8 NVMe SSDs show up fine via daos_server storage scan

Yes.

 

-        You have reduced the number of huge pages to 4096 pages (8GB) and all the SPDK errors related to failed huge pages allocation are gone as well as this error from the log:
SPDK prepare: spdk setup failed (): exit status 1: stdout: 0000:00:05.0 (8086 0a54): nvme -> uio_pci_generic

Yes, only reporting pci addr not found and cant start, then reduced nvme to 1 seems pass this step.

-        But the io_server fails to start after dmg storage format?

Yes.

 

I had a second look at daos_control2.log and noticed that you are using 20 targets while you have 8 SSDs and 10 cores. Could you please try with #targets = #SSDs = 8 and set nr_xs_helpers to 0?

This config file was copies from physical machine, maybe some problem, several days ago, it and physical server are working, tried your suggestion, still same while formatting and cant start again.

daos_io_server:0 Using legacy core allocation algorithm
0 target XS(xstream) requested (#cores 10); use (7) target XS
DEBUG 04:51:52.588209 main.go:82: exit status 1
/root/daos/install/bin/daos_io_server (instance 0) exited

 

 

Cheers,

Johann

 

From: <daos@daos.groups.io> on behalf of Shengyu SY19 Zhang <zhangsy19@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Friday 17 January 2020 at 08:21
To: "daos@daos.groups.io" <daos@daos.groups.io>
Cc: "Macdonald, Mjmac" <mjmac.macdonald@...>
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Hello Johann,

 

For the first issue (daos_control.log), it is physical machine and specified 8 NVMes, daos server report cant find PCI address (created the log).

Then I use only one NVMe, daos server can start normally, however it stopped after dmg storage format, just like the second issue (daos_control2.log).

 

DEBUG 02:18:19.457860 config.go:378: Active config saved to /root/daos/install/etc/.daos_server.active.yml (read-only)
01/17-02:18:31.74 afa1 DAOS[185862/185862] hg   ERR  # NA -- Error -- /root/daos/_build.external/mercury/src/na/na_ofi.c:1609
 # na_ofi_domain_open(): No provider found for "verbs;ofi_rxm" provider on domain "mlx5_0/192.168.80.120"
01/17-02:18:31.74 afa1 DAOS[185862/185862] hg   ERR  # NA -- Error -- /root/daos/_build.external/mercury/src/na/na_ofi.c:2975
 # na_ofi_initialize(): Could not open domain for verbs;ofi_rxm, mlx5_0/192.168.80.120
01/17-02:18:31.74 afa1 DAOS[185862/185862] hg   ERR  # NA -- Error -- /root/daos/_build.external/mercury/src/na/na.c:324
 # NA_Initialize_opt(): Could not initialize plugin
01/17-02:18:31.74 afa1 DAOS[185862/185862] hg   ERR  src/cart/crt_hg.c:526 crt_hg_init() Could not initialize NA class.
01/17-02:18:31.74 afa1 DAOS[185862/185862] crt  ERR  src/cart/crt_init.c:317 crt_init_opt() crt_hg_init failed rc: -1020.
01/17-02:18:31.74 afa1 DAOS[185862/185862] crt  ERR  src/cart/crt_init.c:385 crt_init_opt() crt_init failed, rc: -1020.

 

Regards,

Shengyu,

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Lombardi, Johann
Sent: Thursday, January 16, 2020 8:24 PM
To: daos@daos.groups.io
Cc: Macdonald, Mjmac <mjmac.macdonald@...>
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Hi Shengyu,

 

Mike looked into the logs you provided and noticed that:

 

## ERROR: requested 40960 hugepages but only 8585 could be allocated.

## Memory might be heavily fragmented. Please try flushing the system cache, or reboot the machine.

 

Maybe you meant to specify 4096 in the yaml file instead of 40960 for nr_hugepages?  It sounds like you are trying to allocate 82GB of RAM for hugepages.

 

Cheers,

Johann

 

 

From: <daos@daos.groups.io> on behalf of "Nabarro, Tom" <tom.nabarro@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Wednesday 15 January 2020 at 13:17
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

There seems to be multiple messages about not being able to meet the required number of hugepages requested. I was not involved in the work but there have been changes in how huge page allocation is performed for each of the daos_io_server instances and we are now performing automatic storage prepare when starting daos_server.

 

What I would suggest is to reboot the nodes (in case the issues to do with memory fragmentation) and try with a smaller number of drives configured in the config file. Don’t try to do a storage prepare before running daos_server (as it will be performed on start-up anyway). And update to recent master before trying please. You could also try to bump the number of huge pages specified in the server config file to maybe 8192?

 

Regards,

Tom Nabarro – DCG/ESAD

M: +44 (0)7786 260986

Skype: tom.nabarro

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang
Sent: Wednesday, January 15, 2020 3:27 AM
To: daos@daos.groups.io
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Hello

 

Sorry I forgot attachments.

 

Regards,

Shengyu

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang
Sent: Wednesday, January 15, 2020 11:21 AM
To: daos@daos.groups.io
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Hello Johann,

 

I couldnt check it at this time, since I cant start daos server (newest code of master) on my two different machines, I assume there are lots of modification in control path, there could be some new issues:

1.      Daos_server storage prepare -nvme-only, all my NVMe disks switched to uio as expected, then issue storage scan, can see those disks as expected as well.

However, when I start daos server, it report error that reporting cant find PCI address, and all NVMs switched to kernel driver, see daos_control.log

2.      On another machine, it just stopped after being formatted, see doas_control2.log

 

Regards,

Shengyu,

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Lombardi, Johann
Sent: Tuesday, January 14, 2020 5:03 AM
To: daos@daos.groups.io
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Thanks for the logs Shengyu. It does not seem to be related to wrong endpoint addresses. The client did find the server, but this latter returned DER_NONEXIST when connecting to the pool. It might be the same problem fixed recently by PR  #1701. Could you please apply the patch or try with latest master?

 

Cheers,

Johann

 

From: <daos@daos.groups.io> on behalf of Shengyu SY19 Zhang <zhangsy19@...>
Reply-To: "
daos@daos.groups.io" <daos@daos.groups.io>
Date: Friday 10 January 2020 at 07:20
To: "
daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Hello Johann,

 

Here are log files.

My step:

1.      Create fresh environment and start daos server and then format it.

2.      Create pool

3.      Create container.

4.      List container: daos container query -svc=0 path=/tmp/mycontainer, that work great.

5.      CTRl +C to kill daos server

6.      Restart daos server

7.      Repeat 4, daos process will be dead with infinitely loop.

 

Regards,

Shengyu.

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Lombardi, Johann
Sent: Thursday, January 9, 2020 3:45 PM
To: daos@daos.groups.io
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Hm, the issue with server restart might be due to the endpoint address of the servers not being persistent.

Could you please collect full debug logs for the fresh start with reformat and the subsequent restart?

 

Cheers,

Johann

 

From: <daos@daos.groups.io> on behalf of Shengyu SY19 Zhang <zhangsy19@...>
Reply-To: "
daos@daos.groups.io" <daos@daos.groups.io>
Date: Thursday 9 January 2020 at 07:22
To: "
daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Hello,

 

@Johann, Newest version will work on new formatted and created pool, yes, I did set in the yaml.

@Kal, no, I still meet the issue after io_server restart, seems there issues after load existing data.

 

Regards,

Shengyu.

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Lombardi, Johann
Sent: Thursday, January 9, 2020 5:31 AM
To: daos@daos.groups.io
Subject: [External] Re: [daos] Does DAOS support infiniband now?

 

Hi Kevan & Shengyu,

 

Could you please advise what commit hash you use? Also, are you specifying in “fabric_iface_port” in the yaml file?

 

Cheers,

Johann

 

From: <daos@daos.groups.io> on behalf of Kevan Rehm <kevan.rehm@...>
Reply-To: "
daos@daos.groups.io" <daos@daos.groups.io>
Date: Wednesday 8 January 2020 at 21:17
To: "
daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [daos] Does DAOS support infiniband now?

 

I have not had any success, I see the same failure sequence as Shengyu, and due to other commitments I've had to set this aside for a few days.   Hope to get back to it in a week or so.

 

Kevan

 

On 1/8/20, 2:15 PM, "daos@daos.groups.io on behalf of Alfizah, Kurniawan" <daos@daos.groups.io on behalf of kurniawan.alfizah@...> wrote:

 

    Hello Shengyu and Kevan,

    I'm wondering if you have resolved this problem and that DAOS is working well with IB.

    

    Cheers,

    Kal

    

    -----Original Message-----

    From: daos@daos.groups.io [mailto:daos@daos.groups.io] On Behalf Of Shengyu SY19 Zhang

    Sent: Saturday, December 28, 2019 2:46 AM

    To: daos@daos.groups.io

    Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

    

    Hello Kevan,

    

    Yes, its exact the same problem that I meet, the rdma_get_cm_event function will issue write system call to receive completing event from from ib device, however it get nothing at this time, and always return EAGAIN, that caused fi_tsend function infinitely loop.

    

    Regards,

    Shengyu.

    

    -----Original Message-----

    From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Kevan Rehm

    Sent: Saturday, December 28, 2019 12:30 AM

    To: daos@daos.groups.io

    Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

    

    Shengyu,

    

    I think we are going to need help from the experts, I am not familiar with this code.    I tried the same commands that you mentioned in your last email, and they also hang for me.   But I do not see an infinite loop; rather the daos process just hangs forever in a write() request.   Is that what you see as well?

    

    Experts: is there any documentation on CaRT, what it is for, internals?    I have not been able to find anything.

    

    The last entries in the daos_server_srv1.log file are just the ds_mgmt_drpc_get_attach_info() log messages, called from daos_agent.

    

    While the daos command was hung, I sent a kill -6 signal to it to collect the corefile.   It seems like the command has attempted to set up a MSG connection to the daos_io_server, but has not received a completion event.  The dest_addr=0 looks a little suspicious in the fi_tsend() call.  Hopefully others will recognize what the problem is in the backtrace below, otherwise I will keep digging as time permits.

    

    Thanks, Kevan

    

    (gdb) bt

    #0  0x00007fa7fd5749cd in write () from /lib64/libc.so.6

    #1  0x00007fa7fac2d875 in rdma_get_cm_event.part.13 () from /lib64/librdmacm.so.1

    #2  0x00007fa7fb7fd856 in fi_ibv_eq_read () from /home/users/daos/daos/install/lib/libfabric.so.1

    #3  0x00007fa7fb82360f in rxm_eq_read () from /home/users/daos/daos/install/lib/libfabric.so.1

    #4  0x00007fa7fb8252af in rxm_msg_eq_progress () from /home/users/daos/daos/install/lib/libfabric.so.1

    #5  0x00007fa7fb82542d in rxm_cmap_connect () from /home/users/daos/daos/install/lib/libfabric.so.1

    #6  0x00007fa7fb82b5eb in rxm_ep_tsend () from /home/users/daos/daos/install/lib/libfabric.so.1

    #7  0x00007fa7fc59f3e8 in fi_tsend (context=0x1814558, tag=1, dest_addr=0, desc=0x1811ab0, len=332, buf=0x7fa7f6a10038,

        ep=0x17f03c0) at /home/users/daos/daos/install/include/rdma/fi_tagged.h:114

    #8  na_ofi_msg_send_unexpected (na_class=0x17d6250, context=0x180f760, callback=<optimized out>, arg=<optimized out>,

        buf=0x7fa7f6a10038, buf_size=332, plugin_data=0x1811ab0, dest_addr=0x18ba5e0, dest_id=0 '\000', tag=1, op_id=0x1811a48)

        at /home/users/daos/daos/_build.external/mercury/src/na/na_ofi.c:3745

    #9  0x00007fa7fc7b79ff in NA_Msg_send_unexpected (op_id=0x1811a48, tag=<optimized out>, dest_id=<optimized out>,

        dest_addr=<optimized out>, plugin_data=<optimized out>, buf_size=<optimized out>, buf=<optimized out>, arg=0x1811920,

        callback=0x7fa7fc7b9a60 <hg_core_send_input_cb>, context=<optimized out>, na_class=<optimized out>)

        at /home/users/daos/daos/_build.external/mercury/src/na/na.h:1506

    #10 hg_core_forward_na (hg_core_handle=0x1811920) at /home/users/daos/daos/_build.external/mercury/src/mercury_core.c:2076

    #11 0x00007fa7fc7bb5e6 in HG_Core_forward (handle=0x1811920, callback=callback@entry=0x7fa7fc7b0890 <hg_core_forward_cb>,

        arg=arg@entry=0x1814730, flags=<optimized out>, payload_size=<optimized out>)

        at /home/users/daos/daos/_build.external/mercury/src/mercury_core.c:4775

    #12 0x00007fa7fc7b41f7 in HG_Forward (handle=0x1814730, callback=callback@entry=0x7fa7fd8b2980 <crt_hg_req_send_cb>,

        arg=arg@entry=0x18b9190, in_struct=in_struct@entry=0x18b91b0)

        at /home/users/daos/daos/_build.external/mercury/src/mercury.c:2165

    #13 0x00007fa7fd8b9e39 in crt_hg_req_send (rpc_priv=rpc_priv@entry=0x18b9190) at src/cart/crt_hg.c:1191

    #14 0x00007fa7fd90a8ea in crt_req_send_immediately (rpc_priv=<optimized out>) at src/cart/crt_rpc.c:1104

    #15 crt_req_send_internal (rpc_priv=rpc_priv@entry=0x18b9190) at src/cart/crt_rpc.c:1173

    #16 0x00007fa7fd90ef23 in crt_req_hg_addr_lookup_cb (hg_addr=0x18ba590, arg=0x18b9190) at src/cart/crt_rpc.c:569

    #17 0x00007fa7fd8b1062 in crt_hg_addr_lookup_cb (hg_cbinfo=<optimized out>) at src/cart/crt_hg.c:290

    #18 0x00007fa7fc7b0985 in hg_core_addr_lookup_cb (callback_info=<optimized out>)

        at /home/users/daos/daos/_build.external/mercury/src/mercury.c:454

    #19 0x00007fa7fc7bbce2 in hg_core_trigger_lookup_entry (hg_core_op_id=0x18ba530)

        at /home/users/daos/daos/_build.external/mercury/src/mercury_core.c:3444

    #20 hg_core_trigger (context=0x180d590, timeout=<optimized out>, timeout@entry=0, max_count=max_count@entry=4294967295,

        actual_count=actual_count@entry=0x7ffe185f88dc) at /home/users/daos/daos/_build.external/mercury/src/mercury_core.c:3384

    #21 0x00007fa7fc7bca4b in HG_Core_trigger (context=<optimized out>, timeout=timeout@entry=0, max_count=max_count@entry=4294967295,

        actual_count=actual_count@entry=0x7ffe185f88dc) at /home/users/daos/daos/_build.external/mercury/src/mercury_core.c:4900

    #22 0x00007fa7fc7b44ed in HG_Trigger (context=context@entry=0x17d6370, timeout=timeout@entry=0,

        max_count=max_count@entry=4294967295, actual_count=actual_count@entry=0x7ffe185f88dc)

        at /home/users/daos/daos/_build.external/mercury/src/mercury.c:2262

    #23 0x00007fa7fd8b37ea in crt_hg_trigger (hg_ctx=hg_ctx@entry=0x18083d8) at src/cart/crt_hg.c:1327

    #24 0x00007fa7fd8bce5d in crt_hg_progress (hg_ctx=hg_ctx@entry=0x18083d8, timeout=timeout@entry=1000) at src/cart/crt_hg.c:1360

    #25 0x00007fa7fd86dfbb in crt_progress (crt_ctx=0x18083c0, timeout=timeout@entry=-1,

        cond_cb=cond_cb@entry=0x7fa7fe61f7f0 <ev_progress_cb>, arg=arg@entry=0x7ffe185f89d0) at src/cart/crt_context.c:1286

    #26 0x00007fa7fe624bb6 in daos_event_priv_wait () at src/client/api/event.c:1203

    #27 0x00007fa7fe627f96 in dc_task_schedule (task=0x181b4e0, instant=instant@entry=true) at src/client/api/task.c:139

    #28 0x00007fa7fe626eb1 in daos_pool_connect (uuid=uuid@entry=0x7ffe185f8c38 "UXh}E<Kĭ<\035\340\332O", <incomplete sequence \325>,

        grp=0x1805f50 "daos_server", svc=0x1806000, flags=flags@entry=1, poh=poh@entry=0x7ffe185f8c48, info=info@entry=0x0,

        ev=ev@entry=0x0) at src/client/api/pool.c:53

    #29 0x000000000040590d in pool_query_hdlr (ap=0x7ffe185f8c20) at src/utils/daos_hdlr.c:141

    #30 0x0000000000402bc4 in main (argc=5, argv=<optimized out>) at src/utils/daos.c:957

    (gdb)

    

    On 12/25/19, 2:11 AM, "daos@daos.groups.io on behalf of Shengyu SY19 Zhang" <daos@daos.groups.io on behalf of zhangsy19@...> wrote:

    

        Hello Kevan,

        

        Here are the log files.

        Restart server means daos_server restarted, in spite of Ctrl+C to kill process or server reboot, anyhow after restart daos_server, the existing containers can't be touched, it can be 100% reproduced in my environment.

        

        dmg storage query smd -I    ->work.

        daos container query --svc=0 --path=/tmp/mycontainer         ->no response due to infinitely loop

        daos container create... ->->no response due to infinitely loop

        

        with sockets mode, didn't meet this issue.

        

        Regards,

        Shengyu.

        

        -----Original Message-----

        From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Kevan Rehm

        Sent: Wednesday, December 25, 2019 3:30 AM

        To: daos@daos.groups.io

        Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

        

        Shengyu,

        

        When you say "restart server", do you mean that you rebooted the node, or that you just restarted the daos_server process?   Could you send another daos_control.log and daos_server.log from when it fails in this way?

        

        Kevan

        

        On 12/23/19, 11:34 PM, "daos@daos.groups.io on behalf of Shengyu SY19 Zhang" <daos@daos.groups.io on behalf of zhangsy19@...> wrote:

        

            Hello Kevan,

            

            After some more testing, actually the issue is still there, I can get ib to work by following ways:

            Restart subnet

            rm -rf /mnt/daos

            start daos server

            re-format

            create pool

            create container

            

            however once I restart server, will happen the infinitely loop problem, and by any way I can't connect to an existing pool via ib.

            

            Regards,

            Shengyu.

            

            -----Original Message-----

            From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Kevan Rehm

            Sent: Saturday, December 21, 2019 6:57 AM

            To: daos@daos.groups.io

            Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

            

            Shengyu,

            

            As it happens, I also had a case today using infiniband where my daos_test client was in an infinite loop, it generated 200 million lines in daos.log within a minute or so.   It turned out that the IB subnet manager process had died.   I restarted opensm, then re-ran daos_test, and it started to work.    I mention it in case it might be the same problem as yours.   Are you sure your subnet manager is working?    Try a fi_pingpong test; if it works, then your subnet manager is okay, that's not the problem.

            

            Thanks, Kevan

            

            On 12/19/19, 12:48 AM, "daos@daos.groups.io on behalf of Shengyu SY19 Zhang" <daos@daos.groups.io on behalf of zhangsy19@...> wrote:

            

                Hello Joel,

                

                Thanks for your information, those are outputs of the test with and without rxm specified.

                Furthermore, I corrected numa setting, I thought there are mismatch issue with ib devices, I have tried unbind all other devices (ib1,2,3), and also tried remove rxm,

                All same problem happen, daos process just hangs to fi_send due to infinite loop.

                

                After I set FI_LOG_LEVEL=debug, and creating container, I found some interesting, it shows:

                

                libfabric:54383:verbs:core:ofi_check_ep_type():629<info> unsupported endpoint type

                libfabric:54383:verbs:core:ofi_check_ep_type():630<info> Supported: FI_EP_DGRAM

                libfabric:54383:verbs:core:ofi_check_ep_type():630<info> Requested: FI_EP_MSG

                libfabric:54383:verbs:core:ofi_check_domain_attr():525<info> Unknown domain name

                libfabric:54383:verbs:core:ofi_check_domain_attr():526<info> Supported: mlx5_0-xrc

                libfabric:54383:verbs:core:ofi_check_domain_attr():526<info> Requested: mlx5_0

                libfabric:54383:verbs:core:ofi_check_ep_type():629<info> unsupported endpoint type

                libfabric:54383:verbs:core:ofi_check_ep_type():630<info> Supported: FI_EP_DGRAM

                libfabric:54383:verbs:core:ofi_check_ep_type():630<info> Requested: FI_EP_MSG

                

                

                Then I changed environment domain name to mlx5_0-xrc, there will be another message, can't find ofi+verbs provider, just like before. And BTW I think daos app should be failed when connecting pool rather than infinitely loop in fi_send.

                

                Best Regards,

                Shengyu.

                

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Rosenzweig, Joel B

                Sent: Thursday, December 19, 2019 12:15 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hi Shengyu,

                

                Based on your logs, the control plane is successfully matching ib0 with mlx5_0.  It shows "DEBUG 02:06:24.772249 netdetect.go:536: Device alias for ib0 is mlx5_0"  As such, it correctly sets OFI_DOMAIN=mlx5_0.  This matches with your topology data as reported by lstopo.  Your results should not change if you manually are setting OFI_DOMAIN=mlx5_0 or not, because the control plane is already doing the right thing and you are not giving it a conflicting override.  If you find that the behavior changes when you specify OFI_DOMAIN=mlx5_0 in your daos_server.yml, that's a problem we would need to debug.

                

                Your topology shows these 4 interface cards / device combinations (mlx5_0:ib0, mlx5_1:ib1, mlx5_2:ib2 and mlx5_3:ib3).

                

                        PCI 15b3:1013 (P#548864 busid=0000:86:00.0 class=0207(IB) link=15.75GB/s PCISlot=5)

                          Network L#7 (Address=20:00:10:8b:fe:80:00:00:00:00:00:00:b8:59:9f:03:00:05:b5:48 Port=1) "ib0"

                          OpenFabrics L#8 (NodeGUID=b859:9f03:0005:b548 SysImageGUID=b859:9f03:0005:b548 Port1State=4 Port1LID=0x4 Port1LMC=0 Port1GID0=fe80:0000:0000:0000:b859:9f03:0005

                :b548) "mlx5_0"

                

                        PCI 15b3:1013 (P#548865 busid=0000:86:00.1 class=0207(IB) link=15.75GB/s PCISlot=5)

                          Network L#9 (Address=20:00:18:8b:fe:80:00:00:00:00:00:00:b8:59:9f:03:00:05:b5:49 Port=1) "ib1"

                          OpenFabrics L#10 (NodeGUID=b859:9f03:0005:b549 SysImageGUID=b859:9f03:0005:b548 Port1State=1 Port1LID=0xffff Port1LMC=0 Port1GID0=fe80:0000:0000:0000:b859:9f03:

                0005:b549) "mlx5_1"

                

                        PCI 15b3:1013 (P#716800 busid=0000:af:00.0 class=0207(IB) link=15.75GB/s PCISlot=6)

                          Network L#11 (Address=20:00:10:8b:fe:80:00:00:00:00:00:00:b8:59:9f:03:00:05:b5:40 Port=1) "ib2"

                          OpenFabrics L#12 (NodeGUID=b859:9f03:0005:b540 SysImageGUID=b859:9f03:0005:b540 Port1State=4 Port1LID=0x1b Port1LMC=0 Port1GID0=fe80:0000:0000:0000:b859:9f03:00

                05:b540) "mlx5_2"

                

                        PCI 15b3:1013 (P#716801 busid=0000:af:00.1 class=0207(IB) link=15.75GB/s PCISlot=6)

                          Network L#13 (Address=20:00:18:8b:fe:80:00:00:00:00:00:00:b8:59:9f:03:00:05:b5:41 Port=1) "ib3"

                          OpenFabrics L#14 (NodeGUID=b859:9f03:0005:b541 SysImageGUID=b859:9f03:0005:b540 Port1State=1 Port1LID=0xffff Port1LMC=0 Port1GID0=fe80:0000:0000:0000:b859:9f03:

                0005:b541) "mlx5_3"

                

                I agree that fi_info shows that the provider: ofi+verbs;ofi_rxm is valid for ib0, and the control plane agrees with that.  "DEBUG 02:06:20.594910 netdetect.go:764: Device ib0 supports provider: ofi+verbs;ofi_rxm"  This is only a guess, but I have to rule it out.  I am not certain that the provider ofi_rxm is being handled properly.  Can you remove "ofi_rxm" from your provider configuration and try again?

                

                That is, in your daos_server.yml set:

                              provider: ofi+verbs

                

                If you still have an error, then you will want to run the cart diagnostics again that Alex wrote about so we can see the latest results with that.

                

                >> orterun --allow-run-as-root -np 2 -x

                >> CRT_PHY_ADDR_STR="ofi+verbs;ofi_rxm" -x OFI_INTERFACE=ib0 -x

                >> OFI_DOMAIN=mlx5_0 ../bin/crt_launch -e tests/iv_server -v 3

                

                and

                

                >> orterun --allow-run-as-root -np 2 -x CRT_PHY_ADDR_STR="ofi+verbs" -x

                >> OFI_INTERFACE=ib0 -x OFI_DOMAIN=mlx5_0 ../bin/crt_launch -e

                >> tests/iv_server -v 3

                

                One last thing, while this won't generate a runtime error _yet_, you have a mismatch between the pinned_numa_node:0 and the actual NUMA node of your ib0 device.  Your topology data shows it as NUMA node 1.  If you run "daos_server network scan -a" it should show you that the correct pinned_numa_node is 1.  By setting it to the wrong NUMA node, you will have a performance impact once this is running because the daos_io_server will bind the threads to cores in the wrong NUMA node.  The plan is to make a validation error like this a hard error instead of a warning.  There's debug output in your daos_control.log that looks like this:

                

                DEBUG 02:06:20.595012 netdetect.go:901: Validate network config -- given numaNode: 0 DEBUG 02:06:20.872053 netdetect.go:894: ValidateNUMAConfig (device: ib0, NUMA: 0) returned error: The NUMA node for device ib0 does not match the provided value 0. Remove the pinned_numa_node value from daos_server.yml then execute 'daos_server network scan' to see the valid NUMA node associated with the network device

                

                I'll keep working on it with Alex.  Let's see what you find.

                

                Regards,

                Joel

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang

                Sent: Wednesday, December 18, 2019 4:25 AM

                To: daos@daos.groups.io

                Subject: FW: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hello Joel,

                

                I'm wondering if you have any updates for this? I'm also a developer and I very familiar with spdk/dpdk/ibverbs, however I'm not familiar with other projects which depended by the DAOS, if you can give me some hints or guide, I would like to try troubleshooting this issue as well.

                Also if you need environment to reproduce the issue, you may connect to our machine for debug.

                

                Regards,

                Shengyu

                

                

                -----Original Message-----

                From: Shengyu SY19 Zhang

                Sent: Thursday, December 12, 2019 4:05 PM

                To: 'daos@daos.groups.io' <daos@daos.groups.io>

                Subject: RE: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hello Joel,

                

                Yes, please see it in the attachment.

                

                Regards,

                Shengyu

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Rosenzweig, Joel B

                Sent: Thursday, December 12, 2019 11:08 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hi Shengyu,

                

                Can you generate a topology file for me, similar to what I asked Kevan to provide?  You can generate it with with "lstopo --of xml > topology.xml"  I am interested in seeing if your system also has device siblings with same/different ports as the one specified as the OFI_INTERFACE.

                

                Your debug log shows that DAOS chose the sibling of ib0 as mlx5_0.  It's correct that it picked something in the mlxN_N family, but, depending on your topology there could be a better device to choose, possibly one that has a port match.  Your topology file will show if mlx5_0 matches the port or not, and similarly to Kevan's, it will help me develop a better function to find the correct matching sibling.

                

                I don't know why cart failed once it had the OFI_DEVICE that we thought was correct.  Alex's experiment with Kevan will help with that.

                

                Regards,

                Joel

                

                

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang

                Sent: Wednesday, December 11, 2019 2:09 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hello Joel,

                

                Please see those files in attachment, after changed some configurations, daos trapped in (no return from rdma_get_cm_event ) creating container, here is bt log:

                

                (gdb) bt

                #0  0x00007fb8d6ef59cd in write () from /lib64/libc.so.6

                #1  0x00007fb8d45a99ad in rdma_get_cm_event.part.18 () from /lib64/librdmacm.so.1

                #2  0x00007fb8d517f836 in fi_ibv_eq_read () from /root/daos/install/lib/libfabric.so.1

                #3  0x00007fb8d51a556f in rxm_eq_read () from /root/daos/install/lib/libfabric.so.1

                #4  0x00007fb8d51a720f in rxm_msg_eq_progress () from /root/daos/install/lib/libfabric.so.1

                #5  0x00007fb8d51a738d in rxm_cmap_connect () from /root/daos/install/lib/libfabric.so.1

                #6  0x00007fb8d51ad54b in rxm_ep_tsend () from /root/daos/install/lib/libfabric.so.1

                #7  0x00007fb8d5f1e468 in fi_tsend (context=0xbc4018, tag=1, dest_addr=0, desc=0xbaf6f0, len=384, buf=0x7fb8d235c038, ep=0xbb03a0) at /root/daos/install/include/rdma/fi_tagged.h:114

                #8  na_ofi_msg_send_unexpected (na_class=0xacf220, context=0xbbf200, callback=<optimized out>, arg=<optimized out>, buf=0x7fb8d235c038, buf_size=384, plugin_data=0xbaf6f0,

                    dest_addr=0xc6a1b0, dest_id=0 '\000', tag=1, op_id=0xbc14e8) at /root/daos/_build.external/mercury/src/na/na_ofi.c:3622

                #9  0x00007fb8d613885f in NA_Msg_send_unexpected (op_id=0xbc14e8, tag=<optimized out>, dest_id=<optimized out>, dest_addr=<optimized out>, plugin_data=<optimized out>,

                    buf_size=<optimized out>, buf=<optimized out>, arg=0xbc13c0, callback=0x7fb8d613a8c0 <hg_core_send_input_cb>, context=<optimized out>, na_class=<optimized out>)

                    at /root/daos/_build.external/mercury/src/na/na.h:1485

                #10 hg_core_forward_na (hg_core_handle=0xbc13c0) at /root/daos/_build.external/mercury/src/mercury_core.c:2076

                #11 0x00007fb8d613c3a6 in HG_Core_forward (handle=0xbc13c0, callback=callback@entry=0x7fb8d6131740 <hg_core_forward_cb>, arg=arg@entry=0xbc41f0, flags=<optimized out>,

                    payload_size=<optimized out>) at /root/daos/_build.external/mercury/src/mercury_core.c:4748

                #12 0x00007fb8d6135057 in HG_Forward (handle=0xbc41f0, callback=callback@entry=0x7fb8d7233930 <crt_hg_req_send_cb>, arg=arg@entry=0xc68cb0, in_struct=in_struct@entry=0xc68cd0)

                    at /root/daos/_build.external/mercury/src/mercury.c:2147

                #13 0x00007fb8d723ade9 in crt_hg_req_send (rpc_priv=rpc_priv@entry=0xc68cb0) at src/cart/crt_hg.c:1190

                #14 0x00007fb8d728b89a in crt_req_send_immediately (rpc_priv=<optimized out>) at src/cart/crt_rpc.c:1104

                #15 crt_req_send_internal (rpc_priv=rpc_priv@entry=0xc68cb0) at src/cart/crt_rpc.c:1173

                #16 0x00007fb8d728fed3 in crt_req_hg_addr_lookup_cb (hg_addr=0xc6a150, arg=0xc68cb0) at src/cart/crt_rpc.c:569

                #17 0x00007fb8d7232012 in crt_hg_addr_lookup_cb (hg_cbinfo=<optimized out>) at src/cart/crt_hg.c:290

                #18 0x00007fb8d6131835 in hg_core_addr_lookup_cb (callback_info=<optimized out>) at /root/daos/_build.external/mercury/src/mercury.c:454

                #19 0x00007fb8d613caa2 in hg_core_trigger_lookup_entry (hg_core_op_id=0xc6a0f0) at /root/daos/_build.external/mercury/src/mercury_core.c:3444

                #20 hg_core_trigger (context=0xbbd030, timeout=<optimized out>, timeout@entry=0, max_count=max_count@entry=4294967295, actual_count=actual_count@entry=0x7fff3d70388c)

                    at /root/daos/_build.external/mercury/src/mercury_core.c:3384

                #21 0x00007fb8d613d80b in HG_Core_trigger (context=<optimized out>, timeout=timeout@entry=0, max_count=max_count@entry=4294967295, actual_count=actual_count@entry=0x7fff3d70388c)

                    at /root/daos/_build.external/mercury/src/mercury_core.c:4873

                #22 0x00007fb8d613534d in HG_Trigger (context=context@entry=0xacf1e0, timeout=timeout@entry=0, max_count=max_count@entry=4294967295, actual_count=actual_count@entry=0x7fff3d70388c)

                    at /root/daos/_build.external/mercury/src/mercury.c:2244

                #23 0x00007fb8d723479a in crt_hg_trigger (hg_ctx=hg_ctx@entry=0xbb7e78) at src/cart/crt_hg.c:1326

                #24 0x00007fb8d723de0d in crt_hg_progress (hg_ctx=hg_ctx@entry=0xbb7e78, timeout=timeout@entry=1000) at src/cart/crt_hg.c:1359

                #25 0x00007fb8d71eef6b in crt_progress (crt_ctx=0xbb7e60, timeout=timeout@entry=-1, cond_cb=cond_cb@entry=0x7fb8d7fa0230 <ev_progress_cb>, arg=arg@entry=0x7fff3d703980)

                    at src/cart/crt_context.c:1286

                #26 0x00007fb8d7fa55f6 in daos_event_priv_wait () at src/client/api/event.c:1203

                #27 0x00007fb8d7fa89d6 in dc_task_schedule (task=0xbcafa0, instant=instant@entry=true) at src/client/api/task.c:139

                #28 0x00007fb8d7fa78f1 in daos_pool_connect (uuid=uuid@entry=0x7fff3d703b68 "\265\215%f\036AN\354\203\002\317I\035\362\067\273", grp=0xbb59f0 "daos_server", svc=0xbb5aa0,

                    flags=flags@entry=2, poh=poh@entry=0x7fff3d703b78, info=info@entry=0x0, ev=ev@entry=0x0) at src/client/api/pool.c:53

                #29 0x0000000000404eb0 in cont_op_hdlr (ap=ap@entry=0x7fff3d703b50) at src/utils/daos.c:610

                #30 0x0000000000402b94 in main (argc=8, argv=<optimized out>) at src/utils/daos.c:957

                

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Rosenzweig, Joel B

                Sent: Tuesday, December 10, 2019 8:48 PM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hi Shengyu,

                

                It's hard to further diagnose without the logs. Can you share your latest daos_server.yml, full daos_control.log and full server.log? In the daos_server.yml, please set control_log_mask: DEBUG and in the io server section, set log_mask: DEBUG

                

                Regards,

                Joel

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang

                Sent: Tuesday, December 10, 2019 1:42 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hello Joel,

                

                I didn't set OFI_DOMAIN=mlx5_0 before, from the hint of Alex, I set it yesterday, there then there is another error while creating container:

                mgmt ERR  src/mgmt/cli_mgmt.c:325 get_attach_info() GetAttachInfo unsuccessful: 2

                

                

                Regards,

                Shengyu

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Rosenzweig, Joel B

                Sent: Monday, December 9, 2019 11:47 PM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hi Shengyu,

                

                With your latest setup, can you launch DAOS and send your latest daos_control.log?  In particular, I want to see how the daos_io_server environment variables are set.  For example these two lines below show the command line args and the environment variables in use with an ofi+sockets/ib1 config.  I'm looking to see if we are setting OFI_DOMAIN=mlx5_0 in your environment.  I seem to recall that your earlier logs did have this set, but since builds have changed since then, it's worth checking out one more time.

                

                DEBUG 16:33:15.711949 exec.go:112: daos_io_server:1 args: [-t 8 -x 2 -g daos_server -d /tmp/daos_sockets -s /mnt/daos1 -i 1842327892 -p 1 -I 1] DEBUG 16:33:15.711963 exec.go:113: daos_io_server:1 env: [CRT_TIMEOUT=30 FI_SOCKETS_CONN_TIMEOUT=2000 D_LOG_FILE=/tmp/server.log CRT_CTX_SHARE_ADDR=0 FI_SOCKETS_MAX_CONN_RETRY=1 D_LOG_MASK=ERR CRT_PHY_ADDR_STR=ofi+sockets OFI_INTERFACE=ib1 OFI_PORT=31416 DAOS_MD_CAP=1024]

                

                If we are not setting OFI_DOMAIN=mlx5_0 automatically, go ahead and edit your daos_server.yml and add OFI_DOMAIN=mlx5_0 to the env_vars section (per below) and see what you get.  If that doesn't get you going, please send the log for that run, too.

                

                env_vars:

                  - OFI_DOMAIN=mlx5_0

                

                Regards,

                Joel

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Oganezov, Alexander A

                Sent: Monday, December 9, 2019 5:03 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hi Shengyu,

                

                Actually this is really good result - two servers were able to exchange rpcs. That's the end of test -- just ctrl+C out of it.

                

                I'll let Joel take over for daos help on how to make daos use mlx5_0 domain.

                

                Thanks,

                ~~Alex.

                

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang

                Sent: Monday, December 09, 2019 12:57 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hello Alex,

                

                Please see the new log below, and the test seems dead (no more response):

                

                ibv_exp_query_device: invalid comp_mask !!! (comp_mask = 0x7fdd21092400 valid_mask = 0x3) [afa1][[44100,1],1][btl_openib_component.c:1670:init_one_device] error obtaining device attributes for mlx5_0 errno says Invalid argument

                ibv_exp_query_device: invalid comp_mask !!! (comp_mask = 0x195aad0 valid_mask = 0x3) [afa1][[44100,1],0][btl_openib_component.c:1670:init_one_device] error obtaining device attributes for mlx5_0 errno says Invalid argument

                --------------------------------------------------------------------------

                WARNING: There was an error initializing an OpenFabrics device.

                

                  Local host:   afa1

                  Local device: mlx5_0

                --------------------------------------------------------------------------

                12/09-03:53:20.32 afa1 CaRT[101464/101464] crt  WARN src/cart/crt_init.c:270 crt_init_opt() PMIX disabled. Disabling LM automatically

                12/09-03:53:20.32 afa1 CaRT[101465/101465] crt  WARN src/cart/crt_init.c:270 crt_init_opt() PMIX disabled. Disabling LM automatically SRV [rank=1 pid=101465] Server starting, self_rank=1 SRV [rank=0 pid=101464] Server starting, self_rank=0 SRV [rank=1 pid=101465] >>>> Entered iv_set_ivns SRV [rank=1 pid=101465] <<<< Exited iv_set_ivns:773

                

                [afa1:101458] 1 more process has sent help message help-mpi-btl-openib.txt / error in device init [afa1:101458] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

                

                Regards,

                Shengyu

                

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Oganezov, Alexander A

                Sent: Monday, December 9, 2019 4:30 PM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hi Shengyu,

                

                Given your output can you try this now?

                

                orterun --allow-run-as-root -np 2 -x CRT_PHY_ADDR_STR="ofi+verbs;ofi_rxm" -x OFI_INTERFACE=ib0 -x OFI_DOMAIN=mlx5_0 ../bin/crt_launch -e tests/iv_server -v 3

                

                Please note additional OFI_DOMAIN envariable.

                

                Thanks,

                ~~Alex.

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang

                Sent: Monday, December 09, 2019 12:25 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hello Alex,

                

                Here is outputs of fio_info related to verbs:

                

                provider: verbs

                    fabric: IB-0xfe80000000000000

                    domain: mlx5_2

                    version: 1.0

                    type: FI_EP_MSG

                    protocol: FI_PROTO_RDMA_CM_IB_RC

                provider: verbs

                    fabric: IB-0xfe80000000000000

                    domain: mlx5_2-xrc

                    version: 1.0

                    type: FI_EP_MSG

                    protocol: FI_PROTO_RDMA_CM_IB_XRC

                provider: verbs

                    fabric: IB-0xfe80000000000000

                    domain: mlx5_0

                    version: 1.0

                    type: FI_EP_MSG

                    protocol: FI_PROTO_RDMA_CM_IB_RC

                provider: verbs

                    fabric: IB-0xfe80000000000000

                    domain: mlx5_0-xrc

                    version: 1.0

                    type: FI_EP_MSG

                    protocol: FI_PROTO_RDMA_CM_IB_XRC

                provider: verbs

                    fabric: IB-0xfe80000000000000

                    domain: mlx5_0-dgram

                    version: 1.0

                    type: FI_EP_DGRAM

                    protocol: FI_PROTO_IB_UD

                provider: verbs

                    fabric: IB-0xfe80000000000000

                    domain: mlx5_2-dgram

                    version: 1.0

                    type: FI_EP_DGRAM

                    protocol: FI_PROTO_IB_UD

                provider: verbs;ofi_rxm

                    fabric: IB-0xfe80000000000000

                    domain: mlx5_2

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXM

                provider: verbs;ofi_rxm

                    fabric: IB-0xfe80000000000000

                    domain: mlx5_0

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXM

                provider: tcp;ofi_rxm

                    fabric: TCP-IP

                    domain: tcp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXM

                provider: tcp;ofi_rxm

                    fabric: TCP-IP

                    domain: tcp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXM

                provider: tcp;ofi_rxm

                    fabric: TCP-IP

                    domain: tcp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXM

                provider: tcp;ofi_rxm

                    fabric: TCP-IP

                    domain: tcp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXM

                provider: tcp;ofi_rxm

                    fabric: TCP-IP

                    domain: tcp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXM

                provider: tcp;ofi_rxm

                    fabric: TCP-IP

                    domain: tcp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXM

                provider: tcp;ofi_rxm

                    fabric: TCP-IP

                    domain: tcp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXM

                provider: tcp;ofi_rxm

                    fabric: TCP-IP

                    domain: tcp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXM

                provider: tcp;ofi_rxm

                    fabric: TCP-IP

                    domain: tcp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXM

                provider: tcp;ofi_rxm

                    fabric: TCP-IP

                    domain: tcp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXM

                provider: verbs;ofi_rxd

                    fabric: IB-0xfe80000000000000

                    domain: mlx5_2-dgram

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXD

                provider: verbs;ofi_rxd

                    fabric: IB-0xfe80000000000000

                    domain: mlx5_0-dgram

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXD

                provider: UDP;ofi_rxd

                    fabric: UDP-IP

                    domain: udp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXD

                provider: UDP;ofi_rxd

                    fabric: UDP-IP

                    domain: udp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXD

                provider: UDP;ofi_rxd

                    fabric: UDP-IP

                    domain: udp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXD

                provider: UDP;ofi_rxd

                    fabric: UDP-IP

                    domain: udp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXD

                provider: UDP;ofi_rxd

                    fabric: UDP-IP

                    domain: udp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXD

                provider: UDP;ofi_rxd

                    fabric: UDP-IP

                    domain: udp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXD

                provider: UDP;ofi_rxd

                    fabric: UDP-IP

                    domain: udp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXD

                provider: UDP;ofi_rxd

                    fabric: UDP-IP

                    domain: udp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXD

                provider: UDP;ofi_rxd

                    fabric: UDP-IP

                    domain: udp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXD

                provider: UDP;ofi_rxd

                    fabric: UDP-IP

                    domain: udp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXD

                

                Regards,

                Shengyu

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Oganezov, Alexander A

                Sent: Monday, December 9, 2019 4:08 PM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Thanks,

                Can you also provide full fi_info output?

                

                ~~Alex.

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang

                Sent: Monday, December 09, 2019 12:05 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hello Alex,

                

                Here are the outputs:

                

                orterun --allow-run-as-root -np 2 -x CRT_PHY_ADDR_STR="ofi+verbs;ofi_rxm" -x OFI_INTERFACE=ib0 ../bin/crt_launch -e tests/iv_server -v 3

                ibv_exp_query_device: invalid comp_mask !!! (comp_mask = 0x2094f90 valid_mask = 0x3) [afa1][[18544,1],1][btl_openib_component.c:1670:init_one_device] error obtaining device attributes for mlx5_0 errno says Invalid argument

                ibv_exp_query_device: invalid comp_mask !!! (comp_mask = 0xacda60 valid_mask = 0x3) [afa1][[18544,1],0][btl_openib_component.c:1670:init_one_device] error obtaining device attributes for mlx5_0 errno says Invalid argument

                --------------------------------------------------------------------------

                WARNING: There was an error initializing an OpenFabrics device.

                

                  Local host:   afa1

                  Local device: mlx5_0

                --------------------------------------------------------------------------

                12/09-03:01:03.53 afa1 CaRT[92269/92269] hg   ERR  # NA -- Error -- /root/daos/_build.external/mercury/src/na/na_ofi.c:1609

                 # na_ofi_domain_open(): No provider found for "verbs;ofi_rxm" provider on domain "ib0"

                12/09-03:01:03.53 afa1 CaRT[92268/92268] hg   ERR  # NA -- Error -- /root/daos/_build.external/mercury/src/na/na_ofi.c:1609

                 # na_ofi_domain_open(): No provider found for "verbs;ofi_rxm" provider on domain "ib0"

                12/09-03:01:03.53 afa1 CaRT[92269/92269] hg   ERR  # NA -- Error -- /root/daos/_build.external/mercury/src/na/na_ofi.c:2975

                 # na_ofi_initialize(): Could not open domain for verbs;ofi_rxm, ib0

                12/09-03:01:03.53 afa1 CaRT[92269/92269] hg   ERR  # NA -- Error -- /root/daos/_build.external/mercury/src/na/na.c:324

                 # NA_Initialize_opt(): Could not initialize plugin

                12/09-03:01:03.53 afa1 CaRT[92269/92269] hg   ERR  src/cart/crt_hg.c:525 crt_hg_init() Could not initialize NA class.

                12/09-03:01:03.53 afa1 CaRT[92269/92269] crt  ERR  src/cart/crt_init.c:347 crt_init_opt() crt_hg_init failed rc: -1020.

                12/09-03:01:03.53 afa1 CaRT[92269/92269] crt  ERR  src/cart/crt_init.c:421 crt_init_opt() crt_init failed, rc: -1020.

                12/09-03:01:03.53 afa1 CaRT[92268/92268] hg   ERR  # NA -- Error -- /root/daos/_build.external/mercury/src/na/na_ofi.c:2975

                 # na_ofi_initialize(): Could not open domain for verbs;ofi_rxm, ib0

                12/09-03:01:03.53 afa1 CaRT[92268/92268] hg   ERR  # NA -- Error -- /root/daos/_build.external/mercury/src/na/na.c:324

                 # NA_Initialize_opt(): Could not initialize plugin

                12/09-03:01:03.53 afa1 CaRT[92268/92268] hg   ERR  src/cart/crt_hg.c:525 crt_hg_init() Could not initialize NA class.

                12/09-03:01:03.53 afa1 CaRT[92268/92268] crt  ERR  src/cart/crt_init.c:347 crt_init_opt() crt_hg_init failed rc: -1020.

                12/09-03:01:03.53 afa1 CaRT[92268/92268] crt  ERR  src/cart/crt_init.c:421 crt_init_opt() crt_init failed, rc: -1020.

                [afa1:92262] 1 more process has sent help message help-mpi-btl-openib.txt / error in device init [afa1:92262] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

                

                

                Regards,

                Shengyu

                

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Oganezov, Alexander A

                Sent: Saturday, December 7, 2019 1:14 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hi Shengyu,

                

                With latest daos code and mofed 4.6 installed can you rerun this and show what that one gives you?

                

                source scons_local/utils/setup_local.sh

                cd install/Linux/TESTING

                orterun -np 2 -x CRT_PHY_ADDR_STR="ofi+verbs;ofi_rxm" -x OFI_INTERFACE=ib0 ../bin/crt_launch -e tests/iv_server -v 3

                

                Thanks,

                ~~Alex.

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang

                Sent: Thursday, December 05, 2019 10:25 PM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hello Alex,

                

                Yes, however with 4.6 the result is same.

                After I upgraded daos code to newest of master branch, I got some different results, daos io server seems started OK, since I can see lots of fd points to rdma_cm.

                But daos client seems can't connect to server due to same error (can't find efi+verbs provider on ib0) like the log shows, you may find the log in the attachment, that is crated via "create container"

                

                Regards,

                Shengyu.

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Oganezov, Alexander A

                Sent: Wednesday, December 4, 2019 4:02 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hi Shengyu,

                

                Can you try installing MOFED 4.6 packages on your system? In general MOFED is required to get verbs over Mellanox working.

                Those packages can be found at: https://www.mellanox.com/page/mlnx_ofed_matrix?mtag=linux_sw_drivers

                

                There is also 4.7 version available, however there seem to be few longevity issues currently when using 4.7 (according to verbs ofi maintainers).

                

                Thanks,

                ~~Alex.

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang

                Sent: Monday, November 25, 2019 9:55 PM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hello Alex,

                

                Thanks for your suggestion, here is the log:

                

                

                mca_base_component_repository_open: unable to open mca_pml_ucx: libucp.so.0: cannot open shared object file: No such file or directory (ignored)

                11/26-00:40:22.65 afa1 CaRT[365504/365504] hg   ERR  # NA -- Error -- /root/daos/_build.external/mercury/src/na/na_ofi.c:1407

                 # na_ofi_getinfo(): fi_getinfo() failed, rc: -61(No data available)

                11/26-00:40:22.65 afa1 CaRT[365504/365504] hg   ERR  # NA -- Error -- /root/daos/_build.external/mercury/src/na/na_ofi.c:2816

                 # na_ofi_check_protocol(): na_ofi_getinfo() failed

                11/26-00:40:22.65 afa1 CaRT[365504/365504] hg   ERR  # NA -- Error -- /root/daos/_build.external/mercury/src/na/na.c:302

                 # NA_Initialize_opt(): No suitable plugin found that matches ofi+verbs;ofi_rxm://192.168.80.120

                11/26-00:40:22.65 afa1 CaRT[365504/365504] hg   ERR  src/cart/crt_hg.c:521 crt_hg_init() Could not initialize NA class.

                11/26-00:40:22.65 afa1 CaRT[365504/365504] crt  ERR  src/cart/crt_init.c:347 crt_init_opt() crt_hg_init failed rc: -1020.

                11/26-00:40:22.65 afa1 CaRT[365504/365504] crt  ERR  src/cart/crt_init.c:421 crt_init_opt() crt_init failed, rc: -1020.

                

                

                Regards,

                Shengyu

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Oganezov, Alexander A

                Sent: Tuesday, November 26, 2019 12:21 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hi Shengyu,

                

                In order to figure out what is the issue on your system could you run cart standalone test instead and provide the output that you get?

                

                cd daos_dir

                source scons_local/utils/setup_local.sh

                cd install/Linux/TESTING

                orterun -np 2 -x CRT_PHY_ADDR_STR="ofi+verbs;ofi_rxm" -x OFI_INTERFACE=ib0 ../bin/crt_launch -e tests/iv_server -v 3

                

                Note: Depending on how you installed daos your paths might be different, so instead of cd install/Linux/TESTING you might have to cd into different directory first where you have tests/iv_server in. I think in your env it will be   cd /root/daos/install/TESTING/  or cd /root/daos/install/cart/TESTING.

                

                

                Expected output:

                11/25-15:51:48.39 wolf-55 CaRT[53295/53295] crt  WARN src/cart/crt_init.c:270 crt_init_opt() PMIX disabled. Disabling LM automatically

                11/25-15:51:48.40 wolf-55 CaRT[53296/53296] crt  WARN src/cart/crt_init.c:270 crt_init_opt() PMIX disabled. Disabling LM automatically SRV [rank=0 pid=53295]  Server starting, self_rank=0 SRV [rank=1 pid=53296]  Server starting, self_rank=1 SRV [rank=1 pid=53296]  >>>> Entered iv_set_ivns SRV [rank=1 pid=53296]  <<<< Exited iv_set_ivns:773

                

                Thanks,

                ~~Alex.

                

                

                -----Original Message-----

                From: daos@daos.groups.io [mailto:daos@daos.groups.io] On Behalf Of Shengyu SY19 Zhang

                Sent: Monday, November 25, 2019 3:28 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hello Joel,

                

                As the shown in the output.log, there is only one version of libfabrics installed in my machine, and actually I don't nave other software which depends libfabraics installed.

                From you guide to set FI_LOG_LEVEL=debug, I can see the following message, may be helpful:

                

                libfabric:123445:verbs:fabric:fi_ibv_set_default_attr():1263<info> Ignoring provider default value for tx rma_iov_limit as it is greater than the value supported by domain: mlx5_0 libfabric:123445:verbs:fabric:fi_ibv_get_matching_info():1365<info> hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints

                ERROR: daos_io_server:0 libfabric:123445:verbs:core:fi_ibv_check_hints():231<info> Unsupported capabilities libfabric:123445:verbs:core:fi_ibv_check_hints():232<info> Supported: FI_MSG, FI_RECV, FI_SEND, FI_LOCAL_COMM, FI_REMOTE_COMM libfabric:123445:verbs:core:fi_ibv_check_hints():232<info> Requested: FI_MSG, FI_RMA, FI_READ, FI_RECV, FI_SEND, FI_REMOTE_READ

                ERROR: daos_io_server:0 libfabric:123445:verbs:fabric:fi_ibv_get_rai_id():179<info> rdma_bind_addr: No such device(19)

                ERROR: daos_io_server:0 libfabric:123445:verbs:fabric:fi_ibv_get_rai_id():179<info> rdma_bind_addr: No such device(19)

                ERROR: daos_io_server:0 libfabric:123445:verbs:fabric:fi_ibv_get_rai_id():179<info> rdma_bind_addr: Invalid argument(22)

                ERROR: daos_io_server:0 libfabric:123445:verbs:fabric:fi_ibv_get_rai_id():179<info> rdma_bind_addr: Invalid argument(22)

                ERROR: daos_io_server:0 libfabric:123445:verbs:fabric:fi_ibv_get_rai_id():179<info> rdma_bind_addr: Invalid argument(22)

                ERROR: daos_io_server:0 libfabric:123445:core:core:ofi_layering_ok():795<info> Need core provider, skipping ofi_rxd libfabric:123445:core:core:ofi_layering_ok():795<info> Need core provider, skipping ofi_mrail

                

                

                Regards,

                Shengyu.

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Rosenzweig, Joel B

                Sent: Saturday, November 23, 2019 3:20 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hi Shengyu,

                

                The debug output showed me that when daos_server is started via orterun, libfabric is not finding provider support for ofi_rxm at least.  I'm still wondering if you have two different versions of libfabric installed on your machine.

                

                Can you run these commands and provide the output?

                

                1) ldd install/bin/daos_server

                2) modify your orterun command to run ldd on daos_server.   For example, I run this command locally:

                orterun --allow-run-as-root --map-by node --mca btl tcp,self --mca oob tcp -np 1 --hostfile /home/jbrosenz/daos/hostfile --enable-recovery --report-uri /tmp/urifile ldd /home/jbrosenz/daos/install/bin/daos_server

                3) which fi_info

                4) ldd over each version of fi_info found

                

                From the data you provide, I'll understand if the libfabric being used by daos_server when executed directly by you in the shell is the same libfabric being used by daos_server when executed via orterun.  Your original "daos_server network scan" output showed support for ofi+verbs;ofi_rxm but your debug output showed that when daos_server was started (via orterun), libfabric could not find support for the very same providers.  If there are two different versions being used with different configurations, it would explain the failure.  If it's a single installation/configuration, then that will lead the debug in another direction.

                

                Depending on what you find through 1-4, you might find it helpful to export the environment variable FI_LOG_LEVEL=debug which will instruct libfabric to output a good deal of debug info.

                

                Regards,

                Joel

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang

                Sent: Friday, November 22, 2019 12:59 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hello Joel,

                

                Please see those files in attachment.

                I have tried two machines, one have full provider shows in fi_info (verbs and rxm), another doesn't show verbs, but they are same can't start io_server. I found the project conflicts with mellanox drivers, therefor I remove it and use yum package only, however still keep not working.

                

                

                Regards,

                Shengyu

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Rosenzweig, Joel B

                Sent: Friday, November 22, 2019 6:35 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hi Shengyu,

                

                Can you share your daos_server.yml so we can see how you enabled the provider?  And, can you share the log files daos_control.log and server.log so we can see more context?

                

                Thank you,

                Joel

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang

                Sent: Wednesday, November 20, 2019 9:23 PM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hello,

                

                Thank you for your help Alex, Joel and Kevin, I have checked those steps that you provided:

                

                Ibstat:

                State: Active

                Physical state: LinkUp

                

                Ifconfig:

                flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 2044

                

                fi_info:

                verbs:

                    version: 1.0

                ofi_rxm:

                    version: 1.0

                ofi_rxd:

                    version: 1.0

                

                And network is good since I can run SPDK NVMe-oF over Infiniband with good working.

                I also specified "ofi+verbs;ofi_rxm", the same error occurred, the ioserver will be stopped after a while, and print log as I provided previously.

                

                And I noticed, whatever I specify ofi+verbs, ofi_rxm, or ofi+verbs;ofi_rxm, the log keep shows No provider found for "verbs;ofi_rxm" provider on domain "ib0", is it the cause?

                

                BTW: it is working under ofi+sockets.

                

                

                Regards,

                Shengyu.

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Oganezov, Alexander A

                Sent: Thursday, November 21, 2019 7:13 AM

                To: daos@daos.groups.io

                Subject: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hi Shengyu,

                

                > However if I specify either ofi+verbs or ofi_rxm, the same error will happen, and io_server will stop.

                > na_ofi.c:1609

                > # na_ofi_domain_open(): No provider found for "verbs;ofi_rxm" provider on domain "ib0"

                

                To use supported verbs provider you need to have "ofi+verbs;ofi_rxm" in the provider string.

                

                ~~Alex.

                

                -----Original Message-----

                From: daos@daos.groups.io [mailto:daos@daos.groups.io] On Behalf Of Rosenzweig, Joel B

                Sent: Wednesday, November 20, 2019 7:37 AM

                To: daos@daos.groups.io

                Subject: Re: [daos] Does DAOS support infiniband now?

                

                Hi Shengyu,

                

                The daos_server network scan uses information provided by libfabric to determine available devices and providers.  It then cross references that list of devices with device names obtained from hwloc to convert libfabric device names (as necessary) to those you'd find via ifconfig.  Therefore, if "daos_server network scan" displays a device and provider, it means that support for that via libfabric has been provided.   However, as Kevin pointed out, it's possible that the device itself was down, and that could certainly generate an error like what you encountered.  There's another possibility, that you might have more than one version of libfabric installed in your environment.  I have run into this situation in our lab environment.  You might check your target system to see if it has more than one libfabric library with different provider support.

                

                Regards,

                Joel

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Harms, Kevin via Groups.Io

                Sent: Wednesday, November 20, 2019 10:04 AM

                To: daos@daos.groups.io

                Subject: Re: [daos] Does DAOS support infiniband now?

                

                Shengyu,

                

                  I have tried IB and it works. Verify the libfabric verbs provider is available.

                

                  fi_info -l

                

                 you should see these:

                

                ofi\_rxm:

                version: 1.0

                

                verbs:

                version: 1.0

                

                  See here for details:

                

                

                  You might also want to confirm ib0 is in the UP state:

                

                [root@daos01 ~]# ifconfig ib0

                ib0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 4092

                        inet 172.25.6.101  netmask 255.255.0.0  broadcast 172.25.255.255

                

                kevin

                

                ________________________________________

                From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Shengyu SY19 Zhang <zhangsy19@...>

                Sent: Wednesday, November 20, 2019 2:54 AM

                To: daos@daos.groups.io

                Subject: [daos] Does DAOS support infiniband now?

                

                Hello,

                

                I use daos_server network scan, it shows as following:

                fabric_iface: ib0

                       provider: ofi+verbs;ofi_rxm

                       pinned_numa_node: 1

                

                However if I specify either ofi+verbs or ofi_rxm, the same error will happen, and io_server will stop.

                na_ofi.c:1609

                # na_ofi_domain_open(): No provider found for "verbs;ofi_rxm" provider on domain "ib0"

                

                The ib0 is Mellanox nic over Infiniband network.

                

                Regards,

                Shengyu.

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

            

            

            

            

            

            

            

            

        

        

        

        

        

        

        

        

    

    

    

    

    

    

    

    

    

    

    

 

 

 

 

 

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.


Re: does this IB problem look familiar to anyone?

Kevan Rehm
 

All,

 

I figured out the ibv_reg_mr failure; my problem #1 below.   Looking at the MOFED code, the same problem would occur there.   It’s a bit complicated, it will take me a while to think it through in order to open a jira, for now I just want to mention that no one need spend further time thinking about problem #1, I will report back when I have a clear description.

 

As far as problem #2, Mercury not noticing libfabric failures, I will pursue that in a few days.

 

Kevan

 

From: <daos@daos.groups.io> on behalf of "Oganezov, Alexander A" <alexander.a.oganezov@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Monday, January 20, 2020 at 1:51 PM
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [daos] does this IB problem look familiar to anyone?

 

Hi Kevan,

 

I recall similar error (ibv_reg_mr failing on pmem-backed memory) used to happen if MOFED packages weren’t installed and instead default distribution supplied drivers were being used. Have you tried installing mofed 4.6 or above? Those can be found on https://www.mellanox.com/products/infiniband-drivers/linux/mlnx_ofed

 

Thanks,

~~Alex.

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Kevan Rehm
Sent: Monday, January 20, 2020 11:14 AM
To: daos@daos.groups.io
Subject: Re: [daos] does this IB problem look familiar to anyone?

 

Phil,

 

Oddly, setting it to zero had no effect.   And I did a “grep -r ‘FI_MR_CACHE_MAX_COUNT’ daos/_build.external/ofi”, and the only hits were in the fi_mr.3 man pages, no code matches.   Maybe the symbol is no longer used?   The ofi directory in daos is branch master at https://github.com/ofiwg/libfabric, latest commit is 86340704a7c73e924d2d6e3112d2350ad0f83d84 on November 7th 2019.

 

Kevan

 

From: <daos@daos.groups.io> on behalf of "Carns, Philip H. via Groups.Io" <carns@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Monday, January 20, 2020 at 12:44 PM
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [daos] does this IB problem look familiar to anyone?

 

Hi Kevan,

 

Re: that long call sequence to the ibv_reg_mr() failure, you could try setting this environment variable before starting the failing daos process:

 

export FI_MR_CACHE_MAX_COUNT=0

 

That will disable the memory registration cache that the verbs provider is using within libfabric.  I don't expect that to solve your problem, but it might simplify debugging a bit if there is a memory registration issue.

 

thanks,

-Phil


From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Kevan Rehm <kevan.rehm@...>
Sent: Monday, January 20, 2020 1:31 PM
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: [daos] does this IB problem look familiar to anyone?

 

Still no luck getting infiniband to work.  I rebased to commit b5b51a329bdb5afa50eef2a948fa2ed7f7cf49fc, and am now seeing new, different problems.   This post is just in case someone recognizes what I am seeing and could point me in the right direction.

 

I built and installed daos on all the server and client nodes.   The NICs are mlx4.  Servers come up fine, storage (fake SCM, fake NVMe) is formatted, servers are waiting for clients to connect.   When I start daos_test on a client node, I get an immediate infinite loop in daos_test, daos.log grows extremely rapidly (debug-enabled).   I believe there are several cascading problems.   Here are the interesting daos.log bits:

 

daos@hl-d109 /tmp $ grep 'fi_tsend(' daos.log | more

 # na_ofi_msg_send_unexpected(): fi_tsend() unexpected failed, rc: -12(Cannot allocate memory)

 # na_ofi_msg_send_unexpected(): fi_tsend() unexpected failed, rc: -113(No route to host)

 # na_ofi_msg_send_unexpected(): fi_tsend() unexpected failed, rc: -113(No route to host)

 # na_ofi_msg_send_unexpected(): fi_tsend() unexpected failed, rc: -113(No route to host)

 # na_ofi_msg_send_unexpected(): fi_tsend() unexpected failed, rc: -113(No route to host)

 

The first fi_tsend() call in Mercury failed because the ofi_rxm layer first tried to allocate and register memory so that it could post a receive buffer for the results of the connect call to the server.  It calls fi_mr_reg() for this.  Eventually this bubbles down to a ibv_reg_mr() call, which returns NULL, and errno has the value EINVAL.  This problem is my first focus, anyone know what why ibv_reg_mr() might fail?    I can’t breakpoint into it, it appears to be in libibverbs.so.

 

The failure calling sequence is fi_tsend -> rxm_ep_tsend -> rxm_ep_prepare_tx -> rxm_cmap_connect -> rxm_conn_connect -> rxm_msg_ep_open -> rxm_msg_ep_prepost_recv -> rxm_rx_buf_alloc -> ofi_buf_alloc -> ofi_bufpool_grow -> rxm_buf_reg -> rxm_msg_mr_reg_internal -> fi_mr_reg -> fi_ibv_mr_cache_reg -> ofi_mr_cache_reg -> fi_ibv_mr_cache_add_region -> fi_ibv_mr_reg_common -> ibv_reg_mr

 

Next problem: the error bubbles back up to rxm_cmap_connect in the RXM_CMAP_IDLE branch of the switch statement.  ‘ret’ is non-zero, so the handle state is updated to RXM_CMAP_SHUTDOWN by routine rxm_cmap_del_handle, and the handle is scheduled for deletion.   This seems reasonable, but the higher code levels don’t seem to notice the failure, and so fi_tsend gets called again.  

 

This time when rxm_ep_prepare_tx get called, routine rxm_cmap_acquire_handle returns NULL because the handle was deleted, so the routine returns -FI_EHOSTUNREACH.   Mercury keeps calling fi_tsend over and over forever, not recognizing the failure.  Seems like this is a Mercury bug, libfabric failures aren’t being seen and handled?  

 

Regards, Kevan

 


Re: [External] Re: [daos] Does DAOS support infiniband now?

Shengyu SY19 Zhang
 

Hello Alex,

 

Please see the log attached.

 

Regards,

Shengyu.

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Oganezov, Alexander A
Sent: Tuesday, January 21, 2020 3:26 PM
To: daos@daos.groups.io
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Shengyu,

 

Can you provide full log from start until first occurrence of this error?

 

Thanks,

~~Alex.

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang
Sent: Monday, January 20, 2020 11:02 PM
To: daos@daos.groups.io
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Hello Alex,

 

I have removed mercury, the io_server seems started normally ,however the daos_server.log increasing quickly to eat all my free space. It infinite repeat the three lines:

01/21-01:40:49.28 afa1 DAOS[72964/73009] hg   ERR  src/cart/crt_hg.c:1331 crt_hg_trigger() HG_Trigger failed, hg_ret: 18.
01/21-01:40:49.28 afa1 DAOS[72964/73009] rpc  ERR  src/cart/crt_context.c:1255 crt_progress() crt_hg_progress failed, rc: -1020.
01/21-01:40:49.28 afa1 DAOS[72964/73009] server ERR  src/iosrv/srv.c:469 dss_srv_handler() failed to progress CART context: -1020

 

Regards,

Shengyu,

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Oganezov, Alexander A
Sent: Tuesday, January 21, 2020 4:49 AM
To: daos@daos.groups.io
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

One other thing Shengyu,

 

Can you verify that your mercury build is up to date? Based on domain being used as “mlx5_0/192.168.80.161” it sounds like there is mismatch between what CaRT generates and what mercury consumes; there has been change few weeks ago regarding how domain is provided to mercury level, and it feels as if older mercury is being used.

 

To ensure clean mercury build remove libmercury* libna*  from your install/lib location, remove _build.external-Linux/mercury directory and recompile daos with scons –build-deps=yes –config=force install

 

Thanks,

~~Alex.

 

From: Oganezov, Alexander A
Sent: Monday, January 20, 2020 11:56 AM
To: daos@daos.groups.io
Subject: RE: [External] Re: [daos] Does DAOS support infiniband now?

 

Hi Shengyu,

 

What does this command return to you on the node where you see this strange domain name?

fi_info --provider="verbs"

 

Thanks,

~~Alex.

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang
Sent: Monday, January 20, 2020 12:32 AM
To: daos@daos.groups.io
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Hello Joel,

 

I have some more information,

In the file na_ofi.c + 1609, the io_serve exit there, and on above code, na_ofi_verify_provider function will compare domain names, the domain is “mlx5_0/192.168.80.161”, while prov->domain_attr.name is “mlx5_0”,  it will always return FALSE.

 

Regards,

Shengyu

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang
Sent: Sunday, January 19, 2020 6:15 PM
To: daos@daos.groups.io
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Hello Joel,

 

Network scan:

Scanning fabric for YML specified provider: ofi+verbs;ofi_rxm
Fabric scan found 1 devices matching the provider spec: ofi+verbs;ofi_rxm

fabric_iface: ib0
provider: ofi+verbs;ofi_rxm
pinned_numa_node: 1

 

 

And log attached, it is standalone server.

Additionally, I created two identical virtual machines with IB sr-iov pass-through, however one vm can’t start due to the same problem, while another can start normally. They were using ofi+verbs;ofi_rxm as provider.

 

 

Regards,

Shengyu

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Rosenzweig, Joel B
Sent: Friday, January 17, 2020 11:03 PM
To: daos@daos.groups.io
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Hello Shengyu,

 

It appears that your daos_server.yml is specifying the provider as “ofi+verbs” but I think it should be set to “ofi+verbs;ofi_rxm”.  Can you configure your daos_server.yml with that and try again?  And then, if things still do not work, then please provide:

 

1)      the network scan output again after you make the provider change

2)      the portion of the debug log that shows the environment variables being provided to the daos_io_server.  This will show us what is being set for OFI_INTERFACE, OFI_DOMAIN in the daos_io_server environment.

3)      the daos_server.yml so I can see how you have configured each daos_io_server.

 

Regards,

Joel

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang
Sent: Friday, January 17, 2020 4:55 AM
To: daos@daos.groups.io
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Hello Johann,

 

Here is logs:

 

Scanning fabric for YML specified provider: ofi+verbs
Fabric scan found 4 devices matching the provider spec: ofi+verbs

fabric_iface: ib2
provider: ofi+verbs;ofi_rxm
pinned_numa_node: 1


fabric_iface: ib0
provider: ofi+verbs;ofi_rxm
pinned_numa_node: 1


fabric_iface: ib2
provider: ofi+verbs
pinned_numa_node: 1


fabric_iface: ib0
provider: ofi+verbs
pinned_numa_node: 1

 

And please see my inline comments.

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Lombardi, Johann
Sent: Friday, January 17, 2020 4:51 PM
To: daos@daos.groups.io; Rosenzweig, Joel B <joel.b.rosenzweig@...>
Cc: Macdonald, Mjmac <mjmac.macdonald@...>
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Hi,

 

Cc’ing Joel.

Those errors indicate that you still have some network setup issue. Could you please run daos_server network scan?

 

e.g.:

[root@wolf-118 ~]# daos_server network scan -p "ofi+verbs;ofi_rxm"

Scanning fabric for cmdline specified provider: ofi+verbs;ofi_rxm

Fabric scan found 3 devices matching the provider spec: ofi+verbs;ofi_rxm

 

                fabric_iface: ib0

                provider: ofi+verbs;ofi_rxm

                pinned_numa_node: 0

 

 

                fabric_iface: ib1

                provider: ofi+verbs;ofi_rxm

                pinned_numa_node: 1

 

 

                fabric_iface: eth0

                provider: ofi+verbs;ofi_rxm

                pinned_numa_node: 0

 

As for the NVMe issue, do I understand correctly that:

-          The PCI addresses of the 8 NVMe SSDs show up fine via daos_server storage scan

Yes.

 

-          You have reduced the number of huge pages to 4096 pages (8GB) and all the SPDK errors related to failed huge pages allocation are gone as well as this error from the log:
SPDK prepare: spdk setup failed (): exit status 1: stdout: 0000:00:05.0 (8086 0a54): nvme -> uio_pci_generic

Yes, only reporting pci addr not found and cant start, then reduced nvme to 1 seems pass this step.

-          But the io_server fails to start after dmg storage format?

Yes.

 

I had a second look at daos_control2.log and noticed that you are using 20 targets while you have 8 SSDs and 10 cores. Could you please try with #targets = #SSDs = 8 and set nr_xs_helpers to 0?

This config file was copies from physical machine, maybe some problem, several days ago, it and physical server are working, tried your suggestion, still same while formatting and cant start again.

daos_io_server:0 Using legacy core allocation algorithm
0 target XS(xstream) requested (#cores 10); use (7) target XS
DEBUG 04:51:52.588209 main.go:82: exit status 1
/root/daos/install/bin/daos_io_server (instance 0) exited

 

 

Cheers,

Johann

 

From: <daos@daos.groups.io> on behalf of Shengyu SY19 Zhang <zhangsy19@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Friday 17 January 2020 at 08:21
To: "daos@daos.groups.io" <daos@daos.groups.io>
Cc: "Macdonald, Mjmac" <mjmac.macdonald@...>
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Hello Johann,

 

For the first issue (daos_control.log), it is physical machine and specified 8 NVMes, daos server report cant find PCI address (created the log).

Then I use only one NVMe, daos server can start normally, however it stopped after dmg storage format, just like the second issue (daos_control2.log).

 

DEBUG 02:18:19.457860 config.go:378: Active config saved to /root/daos/install/etc/.daos_server.active.yml (read-only)
01/17-02:18:31.74 afa1 DAOS[185862/185862] hg   ERR  # NA -- Error -- /root/daos/_build.external/mercury/src/na/na_ofi.c:1609
 # na_ofi_domain_open(): No provider found for "verbs;ofi_rxm" provider on domain "mlx5_0/192.168.80.120"
01/17-02:18:31.74 afa1 DAOS[185862/185862] hg   ERR  # NA -- Error -- /root/daos/_build.external/mercury/src/na/na_ofi.c:2975
 # na_ofi_initialize(): Could not open domain for verbs;ofi_rxm, mlx5_0/192.168.80.120
01/17-02:18:31.74 afa1 DAOS[185862/185862] hg   ERR  # NA -- Error -- /root/daos/_build.external/mercury/src/na/na.c:324
 # NA_Initialize_opt(): Could not initialize plugin
01/17-02:18:31.74 afa1 DAOS[185862/185862] hg   ERR  src/cart/crt_hg.c:526 crt_hg_init() Could not initialize NA class.
01/17-02:18:31.74 afa1 DAOS[185862/185862] crt  ERR  src/cart/crt_init.c:317 crt_init_opt() crt_hg_init failed rc: -1020.
01/17-02:18:31.74 afa1 DAOS[185862/185862] crt  ERR  src/cart/crt_init.c:385 crt_init_opt() crt_init failed, rc: -1020.

 

Regards,

Shengyu,

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Lombardi, Johann
Sent: Thursday, January 16, 2020 8:24 PM
To: daos@daos.groups.io
Cc: Macdonald, Mjmac <mjmac.macdonald@...>
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Hi Shengyu,

 

Mike looked into the logs you provided and noticed that:

 

## ERROR: requested 40960 hugepages but only 8585 could be allocated.

## Memory might be heavily fragmented. Please try flushing the system cache, or reboot the machine.

 

Maybe you meant to specify 4096 in the yaml file instead of 40960 for nr_hugepages?  It sounds like you are trying to allocate 82GB of RAM for hugepages.

 

Cheers,

Johann

 

 

From: <daos@daos.groups.io> on behalf of "Nabarro, Tom" <tom.nabarro@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Wednesday 15 January 2020 at 13:17
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

There seems to be multiple messages about not being able to meet the required number of hugepages requested. I was not involved in the work but there have been changes in how huge page allocation is performed for each of the daos_io_server instances and we are now performing automatic storage prepare when starting daos_server.

 

What I would suggest is to reboot the nodes (in case the issues to do with memory fragmentation) and try with a smaller number of drives configured in the config file. Don’t try to do a storage prepare before running daos_server (as it will be performed on start-up anyway). And update to recent master before trying please. You could also try to bump the number of huge pages specified in the server config file to maybe 8192?

 

Regards,

Tom Nabarro – DCG/ESAD

M: +44 (0)7786 260986

Skype: tom.nabarro

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang
Sent: Wednesday, January 15, 2020 3:27 AM
To: daos@daos.groups.io
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Hello

 

Sorry I forgot attachments.

 

Regards,

Shengyu

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang
Sent: Wednesday, January 15, 2020 11:21 AM
To: daos@daos.groups.io
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Hello Johann,

 

I couldnt check it at this time, since I cant start daos server (newest code of master) on my two different machines, I assume there are lots of modification in control path, there could be some new issues:

1.       Daos_server storage prepare -nvme-only, all my NVMe disks switched to uio as expected, then issue storage scan, can see those disks as expected as well.

However, when I start daos server, it report error that reporting cant find PCI address, and all NVMs switched to kernel driver, see daos_control.log

2.       On another machine, it just stopped after being formatted, see doas_control2.log

 

Regards,

Shengyu,

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Lombardi, Johann
Sent: Tuesday, January 14, 2020 5:03 AM
To: daos@daos.groups.io
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Thanks for the logs Shengyu. It does not seem to be related to wrong endpoint addresses. The client did find the server, but this latter returned DER_NONEXIST when connecting to the pool. It might be the same problem fixed recently by PR  #1701. Could you please apply the patch or try with latest master?

 

Cheers,

Johann

 

From: <daos@daos.groups.io> on behalf of Shengyu SY19 Zhang <zhangsy19@...>
Reply-To: "
daos@daos.groups.io" <daos@daos.groups.io>
Date: Friday 10 January 2020 at 07:20
To: "
daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Hello Johann,

 

Here are log files.

My step:

1.       Create fresh environment and start daos server and then format it.

2.       Create pool

3.       Create container.

4.       List container: daos container query -svc=0 path=/tmp/mycontainer, that work great.

5.       CTRl +C to kill daos server

6.       Restart daos server

7.       Repeat 4, daos process will be dead with infinitely loop.

 

Regards,

Shengyu.

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Lombardi, Johann
Sent: Thursday, January 9, 2020 3:45 PM
To: daos@daos.groups.io
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Hm, the issue with server restart might be due to the endpoint address of the servers not being persistent.

Could you please collect full debug logs for the fresh start with reformat and the subsequent restart?

 

Cheers,

Johann

 

From: <daos@daos.groups.io> on behalf of Shengyu SY19 Zhang <zhangsy19@...>
Reply-To: "
daos@daos.groups.io" <daos@daos.groups.io>
Date: Thursday 9 January 2020 at 07:22
To: "
daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Hello,

 

@Johann, Newest version will work on new formatted and created pool, yes, I did set in the yaml.

@Kal, no, I still meet the issue after io_server restart, seems there issues after load existing data.

 

Regards,

Shengyu.

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Lombardi, Johann
Sent: Thursday, January 9, 2020 5:31 AM
To: daos@daos.groups.io
Subject: [External] Re: [daos] Does DAOS support infiniband now?

 

Hi Kevan & Shengyu,

 

Could you please advise what commit hash you use? Also, are you specifying in “fabric_iface_port” in the yaml file?

 

Cheers,

Johann

 

From: <daos@daos.groups.io> on behalf of Kevan Rehm <kevan.rehm@...>
Reply-To: "
daos@daos.groups.io" <daos@daos.groups.io>
Date: Wednesday 8 January 2020 at 21:17
To: "
daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [daos] Does DAOS support infiniband now?

 

I have not had any success, I see the same failure sequence as Shengyu, and due to other commitments I've had to set this aside for a few days.   Hope to get back to it in a week or so.

 

Kevan

 

On 1/8/20, 2:15 PM, "daos@daos.groups.io on behalf of Alfizah, Kurniawan" <daos@daos.groups.io on behalf of kurniawan.alfizah@...> wrote:

 

    Hello Shengyu and Kevan,

    I'm wondering if you have resolved this problem and that DAOS is working well with IB.

    

    Cheers,

    Kal

    

    -----Original Message-----

    From: daos@daos.groups.io [mailto:daos@daos.groups.io] On Behalf Of Shengyu SY19 Zhang

    Sent: Saturday, December 28, 2019 2:46 AM

    To: daos@daos.groups.io

    Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

    

    Hello Kevan,

    

    Yes, its exact the same problem that I meet, the rdma_get_cm_event function will issue write system call to receive completing event from from ib device, however it get nothing at this time, and always return EAGAIN, that caused fi_tsend function infinitely loop.

    

    Regards,

    Shengyu.

    

    -----Original Message-----

    From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Kevan Rehm

    Sent: Saturday, December 28, 2019 12:30 AM

    To: daos@daos.groups.io

    Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

    

    Shengyu,

    

    I think we are going to need help from the experts, I am not familiar with this code.    I tried the same commands that you mentioned in your last email, and they also hang for me.   But I do not see an infinite loop; rather the daos process just hangs forever in a write() request.   Is that what you see as well?

    

    Experts: is there any documentation on CaRT, what it is for, internals?    I have not been able to find anything.

    

    The last entries in the daos_server_srv1.log file are just the ds_mgmt_drpc_get_attach_info() log messages, called from daos_agent.

    

    While the daos command was hung, I sent a kill -6 signal to it to collect the corefile.   It seems like the command has attempted to set up a MSG connection to the daos_io_server, but has not received a completion event.  The dest_addr=0 looks a little suspicious in the fi_tsend() call.  Hopefully others will recognize what the problem is in the backtrace below, otherwise I will keep digging as time permits.

    

    Thanks, Kevan

    

    (gdb) bt

    #0  0x00007fa7fd5749cd in write () from /lib64/libc.so.6

    #1  0x00007fa7fac2d875 in rdma_get_cm_event.part.13 () from /lib64/librdmacm.so.1

    #2  0x00007fa7fb7fd856 in fi_ibv_eq_read () from /home/users/daos/daos/install/lib/libfabric.so.1

    #3  0x00007fa7fb82360f in rxm_eq_read () from /home/users/daos/daos/install/lib/libfabric.so.1

    #4  0x00007fa7fb8252af in rxm_msg_eq_progress () from /home/users/daos/daos/install/lib/libfabric.so.1

    #5  0x00007fa7fb82542d in rxm_cmap_connect () from /home/users/daos/daos/install/lib/libfabric.so.1

    #6  0x00007fa7fb82b5eb in rxm_ep_tsend () from /home/users/daos/daos/install/lib/libfabric.so.1

    #7  0x00007fa7fc59f3e8 in fi_tsend (context=0x1814558, tag=1, dest_addr=0, desc=0x1811ab0, len=332, buf=0x7fa7f6a10038,

        ep=0x17f03c0) at /home/users/daos/daos/install/include/rdma/fi_tagged.h:114

    #8  na_ofi_msg_send_unexpected (na_class=0x17d6250, context=0x180f760, callback=<optimized out>, arg=<optimized out>,

        buf=0x7fa7f6a10038, buf_size=332, plugin_data=0x1811ab0, dest_addr=0x18ba5e0, dest_id=0 '\000', tag=1, op_id=0x1811a48)

        at /home/users/daos/daos/_build.external/mercury/src/na/na_ofi.c:3745

    #9  0x00007fa7fc7b79ff in NA_Msg_send_unexpected (op_id=0x1811a48, tag=<optimized out>, dest_id=<optimized out>,

        dest_addr=<optimized out>, plugin_data=<optimized out>, buf_size=<optimized out>, buf=<optimized out>, arg=0x1811920,

        callback=0x7fa7fc7b9a60 <hg_core_send_input_cb>, context=<optimized out>, na_class=<optimized out>)

        at /home/users/daos/daos/_build.external/mercury/src/na/na.h:1506

    #10 hg_core_forward_na (hg_core_handle=0x1811920) at /home/users/daos/daos/_build.external/mercury/src/mercury_core.c:2076

    #11 0x00007fa7fc7bb5e6 in HG_Core_forward (handle=0x1811920, callback=callback@entry=0x7fa7fc7b0890 <hg_core_forward_cb>,

        arg=arg@entry=0x1814730, flags=<optimized out>, payload_size=<optimized out>)

        at /home/users/daos/daos/_build.external/mercury/src/mercury_core.c:4775

    #12 0x00007fa7fc7b41f7 in HG_Forward (handle=0x1814730, callback=callback@entry=0x7fa7fd8b2980 <crt_hg_req_send_cb>,

        arg=arg@entry=0x18b9190, in_struct=in_struct@entry=0x18b91b0)

        at /home/users/daos/daos/_build.external/mercury/src/mercury.c:2165

    #13 0x00007fa7fd8b9e39 in crt_hg_req_send (rpc_priv=rpc_priv@entry=0x18b9190) at src/cart/crt_hg.c:1191

    #14 0x00007fa7fd90a8ea in crt_req_send_immediately (rpc_priv=<optimized out>) at src/cart/crt_rpc.c:1104

    #15 crt_req_send_internal (rpc_priv=rpc_priv@entry=0x18b9190) at src/cart/crt_rpc.c:1173

    #16 0x00007fa7fd90ef23 in crt_req_hg_addr_lookup_cb (hg_addr=0x18ba590, arg=0x18b9190) at src/cart/crt_rpc.c:569

    #17 0x00007fa7fd8b1062 in crt_hg_addr_lookup_cb (hg_cbinfo=<optimized out>) at src/cart/crt_hg.c:290

    #18 0x00007fa7fc7b0985 in hg_core_addr_lookup_cb (callback_info=<optimized out>)

        at /home/users/daos/daos/_build.external/mercury/src/mercury.c:454

    #19 0x00007fa7fc7bbce2 in hg_core_trigger_lookup_entry (hg_core_op_id=0x18ba530)

        at /home/users/daos/daos/_build.external/mercury/src/mercury_core.c:3444

    #20 hg_core_trigger (context=0x180d590, timeout=<optimized out>, timeout@entry=0, max_count=max_count@entry=4294967295,

        actual_count=actual_count@entry=0x7ffe185f88dc) at /home/users/daos/daos/_build.external/mercury/src/mercury_core.c:3384

    #21 0x00007fa7fc7bca4b in HG_Core_trigger (context=<optimized out>, timeout=timeout@entry=0, max_count=max_count@entry=4294967295,

        actual_count=actual_count@entry=0x7ffe185f88dc) at /home/users/daos/daos/_build.external/mercury/src/mercury_core.c:4900

    #22 0x00007fa7fc7b44ed in HG_Trigger (context=context@entry=0x17d6370, timeout=timeout@entry=0,

        max_count=max_count@entry=4294967295, actual_count=actual_count@entry=0x7ffe185f88dc)

        at /home/users/daos/daos/_build.external/mercury/src/mercury.c:2262

    #23 0x00007fa7fd8b37ea in crt_hg_trigger (hg_ctx=hg_ctx@entry=0x18083d8) at src/cart/crt_hg.c:1327

    #24 0x00007fa7fd8bce5d in crt_hg_progress (hg_ctx=hg_ctx@entry=0x18083d8, timeout=timeout@entry=1000) at src/cart/crt_hg.c:1360

    #25 0x00007fa7fd86dfbb in crt_progress (crt_ctx=0x18083c0, timeout=timeout@entry=-1,

        cond_cb=cond_cb@entry=0x7fa7fe61f7f0 <ev_progress_cb>, arg=arg@entry=0x7ffe185f89d0) at src/cart/crt_context.c:1286

    #26 0x00007fa7fe624bb6 in daos_event_priv_wait () at src/client/api/event.c:1203

    #27 0x00007fa7fe627f96 in dc_task_schedule (task=0x181b4e0, instant=instant@entry=true) at src/client/api/task.c:139

    #28 0x00007fa7fe626eb1 in daos_pool_connect (uuid=uuid@entry=0x7ffe185f8c38 "UXh}E<Kĭ<\035\340\332O", <incomplete sequence \325>,

        grp=0x1805f50 "daos_server", svc=0x1806000, flags=flags@entry=1, poh=poh@entry=0x7ffe185f8c48, info=info@entry=0x0,

        ev=ev@entry=0x0) at src/client/api/pool.c:53

    #29 0x000000000040590d in pool_query_hdlr (ap=0x7ffe185f8c20) at src/utils/daos_hdlr.c:141

    #30 0x0000000000402bc4 in main (argc=5, argv=<optimized out>) at src/utils/daos.c:957

    (gdb)

    

    On 12/25/19, 2:11 AM, "daos@daos.groups.io on behalf of Shengyu SY19 Zhang" <daos@daos.groups.io on behalf of zhangsy19@...> wrote:

    

        Hello Kevan,

        

        Here are the log files.

        Restart server means daos_server restarted, in spite of Ctrl+C to kill process or server reboot, anyhow after restart daos_server, the existing containers can't be touched, it can be 100% reproduced in my environment.

        

        dmg storage query smd -I    ->work.

        daos container query --svc=0 --path=/tmp/mycontainer         ->no response due to infinitely loop

        daos container create... ->->no response due to infinitely loop

        

        with sockets mode, didn't meet this issue.

        

        Regards,

        Shengyu.

        

        -----Original Message-----

        From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Kevan Rehm

        Sent: Wednesday, December 25, 2019 3:30 AM

        To: daos@daos.groups.io

        Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

        

        Shengyu,

        

        When you say "restart server", do you mean that you rebooted the node, or that you just restarted the daos_server process?   Could you send another daos_control.log and daos_server.log from when it fails in this way?

        

        Kevan

        

        On 12/23/19, 11:34 PM, "daos@daos.groups.io on behalf of Shengyu SY19 Zhang" <daos@daos.groups.io on behalf of zhangsy19@...> wrote:

        

            Hello Kevan,

            

            After some more testing, actually the issue is still there, I can get ib to work by following ways:

            Restart subnet

            rm -rf /mnt/daos

            start daos server

            re-format

            create pool

            create container

            

            however once I restart server, will happen the infinitely loop problem, and by any way I can't connect to an existing pool via ib.

            

            Regards,

            Shengyu.

            

            -----Original Message-----

            From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Kevan Rehm

            Sent: Saturday, December 21, 2019 6:57 AM

            To: daos@daos.groups.io

            Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

            

            Shengyu,

            

            As it happens, I also had a case today using infiniband where my daos_test client was in an infinite loop, it generated 200 million lines in daos.log within a minute or so.   It turned out that the IB subnet manager process had died.   I restarted opensm, then re-ran daos_test, and it started to work.    I mention it in case it might be the same problem as yours.   Are you sure your subnet manager is working?    Try a fi_pingpong test; if it works, then your subnet manager is okay, that's not the problem.

            

            Thanks, Kevan

            

            On 12/19/19, 12:48 AM, "daos@daos.groups.io on behalf of Shengyu SY19 Zhang" <daos@daos.groups.io on behalf of zhangsy19@...> wrote:

            

                Hello Joel,

                

                Thanks for your information, those are outputs of the test with and without rxm specified.

                Furthermore, I corrected numa setting, I thought there are mismatch issue with ib devices, I have tried unbind all other devices (ib1,2,3), and also tried remove rxm,

                All same problem happen, daos process just hangs to fi_send due to infinite loop.

                

                After I set FI_LOG_LEVEL=debug, and creating container, I found some interesting, it shows:

                

                libfabric:54383:verbs:core:ofi_check_ep_type():629<info> unsupported endpoint type

                libfabric:54383:verbs:core:ofi_check_ep_type():630<info> Supported: FI_EP_DGRAM

                libfabric:54383:verbs:core:ofi_check_ep_type():630<info> Requested: FI_EP_MSG

                libfabric:54383:verbs:core:ofi_check_domain_attr():525<info> Unknown domain name

                libfabric:54383:verbs:core:ofi_check_domain_attr():526<info> Supported: mlx5_0-xrc

                libfabric:54383:verbs:core:ofi_check_domain_attr():526<info> Requested: mlx5_0

                libfabric:54383:verbs:core:ofi_check_ep_type():629<info> unsupported endpoint type

                libfabric:54383:verbs:core:ofi_check_ep_type():630<info> Supported: FI_EP_DGRAM

                libfabric:54383:verbs:core:ofi_check_ep_type():630<info> Requested: FI_EP_MSG

                

                

                Then I changed environment domain name to mlx5_0-xrc, there will be another message, can't find ofi+verbs provider, just like before. And BTW I think daos app should be failed when connecting pool rather than infinitely loop in fi_send.

                

                Best Regards,

                Shengyu.

                

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Rosenzweig, Joel B

                Sent: Thursday, December 19, 2019 12:15 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hi Shengyu,

                

                Based on your logs, the control plane is successfully matching ib0 with mlx5_0.  It shows "DEBUG 02:06:24.772249 netdetect.go:536: Device alias for ib0 is mlx5_0"  As such, it correctly sets OFI_DOMAIN=mlx5_0.  This matches with your topology data as reported by lstopo.  Your results should not change if you manually are setting OFI_DOMAIN=mlx5_0 or not, because the control plane is already doing the right thing and you are not giving it a conflicting override.  If you find that the behavior changes when you specify OFI_DOMAIN=mlx5_0 in your daos_server.yml, that's a problem we would need to debug.

                

                Your topology shows these 4 interface cards / device combinations (mlx5_0:ib0, mlx5_1:ib1, mlx5_2:ib2 and mlx5_3:ib3).

                

                        PCI 15b3:1013 (P#548864 busid=0000:86:00.0 class=0207(IB) link=15.75GB/s PCISlot=5)

                          Network L#7 (Address=20:00:10:8b:fe:80:00:00:00:00:00:00:b8:59:9f:03:00:05:b5:48 Port=1) "ib0"

                          OpenFabrics L#8 (NodeGUID=b859:9f03:0005:b548 SysImageGUID=b859:9f03:0005:b548 Port1State=4 Port1LID=0x4 Port1LMC=0 Port1GID0=fe80:0000:0000:0000:b859:9f03:0005

                :b548) "mlx5_0"

                

                        PCI 15b3:1013 (P#548865 busid=0000:86:00.1 class=0207(IB) link=15.75GB/s PCISlot=5)

                          Network L#9 (Address=20:00:18:8b:fe:80:00:00:00:00:00:00:b8:59:9f:03:00:05:b5:49 Port=1) "ib1"

                          OpenFabrics L#10 (NodeGUID=b859:9f03:0005:b549 SysImageGUID=b859:9f03:0005:b548 Port1State=1 Port1LID=0xffff Port1LMC=0 Port1GID0=fe80:0000:0000:0000:b859:9f03:

                0005:b549) "mlx5_1"

                

                        PCI 15b3:1013 (P#716800 busid=0000:af:00.0 class=0207(IB) link=15.75GB/s PCISlot=6)

                          Network L#11 (Address=20:00:10:8b:fe:80:00:00:00:00:00:00:b8:59:9f:03:00:05:b5:40 Port=1) "ib2"

                          OpenFabrics L#12 (NodeGUID=b859:9f03:0005:b540 SysImageGUID=b859:9f03:0005:b540 Port1State=4 Port1LID=0x1b Port1LMC=0 Port1GID0=fe80:0000:0000:0000:b859:9f03:00

                05:b540) "mlx5_2"

                

                        PCI 15b3:1013 (P#716801 busid=0000:af:00.1 class=0207(IB) link=15.75GB/s PCISlot=6)

                          Network L#13 (Address=20:00:18:8b:fe:80:00:00:00:00:00:00:b8:59:9f:03:00:05:b5:41 Port=1) "ib3"

                          OpenFabrics L#14 (NodeGUID=b859:9f03:0005:b541 SysImageGUID=b859:9f03:0005:b540 Port1State=1 Port1LID=0xffff Port1LMC=0 Port1GID0=fe80:0000:0000:0000:b859:9f03:

                0005:b541) "mlx5_3"

                

                I agree that fi_info shows that the provider: ofi+verbs;ofi_rxm is valid for ib0, and the control plane agrees with that.  "DEBUG 02:06:20.594910 netdetect.go:764: Device ib0 supports provider: ofi+verbs;ofi_rxm"  This is only a guess, but I have to rule it out.  I am not certain that the provider ofi_rxm is being handled properly.  Can you remove "ofi_rxm" from your provider configuration and try again?

                

                That is, in your daos_server.yml set:

                              provider: ofi+verbs

                

                If you still have an error, then you will want to run the cart diagnostics again that Alex wrote about so we can see the latest results with that.

                

                >> orterun --allow-run-as-root -np 2 -x

                >> CRT_PHY_ADDR_STR="ofi+verbs;ofi_rxm" -x OFI_INTERFACE=ib0 -x

                >> OFI_DOMAIN=mlx5_0 ../bin/crt_launch -e tests/iv_server -v 3

                

                and

                

                >> orterun --allow-run-as-root -np 2 -x CRT_PHY_ADDR_STR="ofi+verbs" -x

                >> OFI_INTERFACE=ib0 -x OFI_DOMAIN=mlx5_0 ../bin/crt_launch -e

                >> tests/iv_server -v 3

                

                One last thing, while this won't generate a runtime error _yet_, you have a mismatch between the pinned_numa_node:0 and the actual NUMA node of your ib0 device.  Your topology data shows it as NUMA node 1.  If you run "daos_server network scan -a" it should show you that the correct pinned_numa_node is 1.  By setting it to the wrong NUMA node, you will have a performance impact once this is running because the daos_io_server will bind the threads to cores in the wrong NUMA node.  The plan is to make a validation error like this a hard error instead of a warning.  There's debug output in your daos_control.log that looks like this:

                

                DEBUG 02:06:20.595012 netdetect.go:901: Validate network config -- given numaNode: 0 DEBUG 02:06:20.872053 netdetect.go:894: ValidateNUMAConfig (device: ib0, NUMA: 0) returned error: The NUMA node for device ib0 does not match the provided value 0. Remove the pinned_numa_node value from daos_server.yml then execute 'daos_server network scan' to see the valid NUMA node associated with the network device

                

                I'll keep working on it with Alex.  Let's see what you find.

                

                Regards,

                Joel

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang

                Sent: Wednesday, December 18, 2019 4:25 AM

                To: daos@daos.groups.io

                Subject: FW: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hello Joel,

                

                I'm wondering if you have any updates for this? I'm also a developer and I very familiar with spdk/dpdk/ibverbs, however I'm not familiar with other projects which depended by the DAOS, if you can give me some hints or guide, I would like to try troubleshooting this issue as well.

                Also if you need environment to reproduce the issue, you may connect to our machine for debug.

                

                Regards,

                Shengyu

                

                

                -----Original Message-----

                From: Shengyu SY19 Zhang

                Sent: Thursday, December 12, 2019 4:05 PM

                To: 'daos@daos.groups.io' <daos@daos.groups.io>

                Subject: RE: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hello Joel,

                

                Yes, please see it in the attachment.

                

                Regards,

                Shengyu

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Rosenzweig, Joel B

                Sent: Thursday, December 12, 2019 11:08 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hi Shengyu,

                

                Can you generate a topology file for me, similar to what I asked Kevan to provide?  You can generate it with with "lstopo --of xml > topology.xml"  I am interested in seeing if your system also has device siblings with same/different ports as the one specified as the OFI_INTERFACE.

                

                Your debug log shows that DAOS chose the sibling of ib0 as mlx5_0.  It's correct that it picked something in the mlxN_N family, but, depending on your topology there could be a better device to choose, possibly one that has a port match.  Your topology file will show if mlx5_0 matches the port or not, and similarly to Kevan's, it will help me develop a better function to find the correct matching sibling.

                

                I don't know why cart failed once it had the OFI_DEVICE that we thought was correct.  Alex's experiment with Kevan will help with that.

                

                Regards,

                Joel

                

                

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang

                Sent: Wednesday, December 11, 2019 2:09 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hello Joel,

                

                Please see those files in attachment, after changed some configurations, daos trapped in (no return from rdma_get_cm_event ) creating container, here is bt log:

                

                (gdb) bt

                #0  0x00007fb8d6ef59cd in write () from /lib64/libc.so.6

                #1  0x00007fb8d45a99ad in rdma_get_cm_event.part.18 () from /lib64/librdmacm.so.1

                #2  0x00007fb8d517f836 in fi_ibv_eq_read () from /root/daos/install/lib/libfabric.so.1

                #3  0x00007fb8d51a556f in rxm_eq_read () from /root/daos/install/lib/libfabric.so.1

                #4  0x00007fb8d51a720f in rxm_msg_eq_progress () from /root/daos/install/lib/libfabric.so.1

                #5  0x00007fb8d51a738d in rxm_cmap_connect () from /root/daos/install/lib/libfabric.so.1

                #6  0x00007fb8d51ad54b in rxm_ep_tsend () from /root/daos/install/lib/libfabric.so.1

                #7  0x00007fb8d5f1e468 in fi_tsend (context=0xbc4018, tag=1, dest_addr=0, desc=0xbaf6f0, len=384, buf=0x7fb8d235c038, ep=0xbb03a0) at /root/daos/install/include/rdma/fi_tagged.h:114

                #8  na_ofi_msg_send_unexpected (na_class=0xacf220, context=0xbbf200, callback=<optimized out>, arg=<optimized out>, buf=0x7fb8d235c038, buf_size=384, plugin_data=0xbaf6f0,

                    dest_addr=0xc6a1b0, dest_id=0 '\000', tag=1, op_id=0xbc14e8) at /root/daos/_build.external/mercury/src/na/na_ofi.c:3622

                #9  0x00007fb8d613885f in NA_Msg_send_unexpected (op_id=0xbc14e8, tag=<optimized out>, dest_id=<optimized out>, dest_addr=<optimized out>, plugin_data=<optimized out>,

                    buf_size=<optimized out>, buf=<optimized out>, arg=0xbc13c0, callback=0x7fb8d613a8c0 <hg_core_send_input_cb>, context=<optimized out>, na_class=<optimized out>)

                    at /root/daos/_build.external/mercury/src/na/na.h:1485

                #10 hg_core_forward_na (hg_core_handle=0xbc13c0) at /root/daos/_build.external/mercury/src/mercury_core.c:2076

                #11 0x00007fb8d613c3a6 in HG_Core_forward (handle=0xbc13c0, callback=callback@entry=0x7fb8d6131740 <hg_core_forward_cb>, arg=arg@entry=0xbc41f0, flags=<optimized out>,

                    payload_size=<optimized out>) at /root/daos/_build.external/mercury/src/mercury_core.c:4748

                #12 0x00007fb8d6135057 in HG_Forward (handle=0xbc41f0, callback=callback@entry=0x7fb8d7233930 <crt_hg_req_send_cb>, arg=arg@entry=0xc68cb0, in_struct=in_struct@entry=0xc68cd0)

                    at /root/daos/_build.external/mercury/src/mercury.c:2147

                #13 0x00007fb8d723ade9 in crt_hg_req_send (rpc_priv=rpc_priv@entry=0xc68cb0) at src/cart/crt_hg.c:1190

                #14 0x00007fb8d728b89a in crt_req_send_immediately (rpc_priv=<optimized out>) at src/cart/crt_rpc.c:1104

                #15 crt_req_send_internal (rpc_priv=rpc_priv@entry=0xc68cb0) at src/cart/crt_rpc.c:1173

                #16 0x00007fb8d728fed3 in crt_req_hg_addr_lookup_cb (hg_addr=0xc6a150, arg=0xc68cb0) at src/cart/crt_rpc.c:569

                #17 0x00007fb8d7232012 in crt_hg_addr_lookup_cb (hg_cbinfo=<optimized out>) at src/cart/crt_hg.c:290

                #18 0x00007fb8d6131835 in hg_core_addr_lookup_cb (callback_info=<optimized out>) at /root/daos/_build.external/mercury/src/mercury.c:454

                #19 0x00007fb8d613caa2 in hg_core_trigger_lookup_entry (hg_core_op_id=0xc6a0f0) at /root/daos/_build.external/mercury/src/mercury_core.c:3444

                #20 hg_core_trigger (context=0xbbd030, timeout=<optimized out>, timeout@entry=0, max_count=max_count@entry=4294967295, actual_count=actual_count@entry=0x7fff3d70388c)

                    at /root/daos/_build.external/mercury/src/mercury_core.c:3384

                #21 0x00007fb8d613d80b in HG_Core_trigger (context=<optimized out>, timeout=timeout@entry=0, max_count=max_count@entry=4294967295, actual_count=actual_count@entry=0x7fff3d70388c)

                    at /root/daos/_build.external/mercury/src/mercury_core.c:4873

                #22 0x00007fb8d613534d in HG_Trigger (context=context@entry=0xacf1e0, timeout=timeout@entry=0, max_count=max_count@entry=4294967295, actual_count=actual_count@entry=0x7fff3d70388c)

                    at /root/daos/_build.external/mercury/src/mercury.c:2244

                #23 0x00007fb8d723479a in crt_hg_trigger (hg_ctx=hg_ctx@entry=0xbb7e78) at src/cart/crt_hg.c:1326

                #24 0x00007fb8d723de0d in crt_hg_progress (hg_ctx=hg_ctx@entry=0xbb7e78, timeout=timeout@entry=1000) at src/cart/crt_hg.c:1359

                #25 0x00007fb8d71eef6b in crt_progress (crt_ctx=0xbb7e60, timeout=timeout@entry=-1, cond_cb=cond_cb@entry=0x7fb8d7fa0230 <ev_progress_cb>, arg=arg@entry=0x7fff3d703980)

                    at src/cart/crt_context.c:1286

                #26 0x00007fb8d7fa55f6 in daos_event_priv_wait () at src/client/api/event.c:1203

                #27 0x00007fb8d7fa89d6 in dc_task_schedule (task=0xbcafa0, instant=instant@entry=true) at src/client/api/task.c:139

                #28 0x00007fb8d7fa78f1 in daos_pool_connect (uuid=uuid@entry=0x7fff3d703b68 "\265\215%f\036AN\354\203\002\317I\035\362\067\273", grp=0xbb59f0 "daos_server", svc=0xbb5aa0,

                    flags=flags@entry=2, poh=poh@entry=0x7fff3d703b78, info=info@entry=0x0, ev=ev@entry=0x0) at src/client/api/pool.c:53

                #29 0x0000000000404eb0 in cont_op_hdlr (ap=ap@entry=0x7fff3d703b50) at src/utils/daos.c:610

                #30 0x0000000000402b94 in main (argc=8, argv=<optimized out>) at src/utils/daos.c:957

                

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Rosenzweig, Joel B

                Sent: Tuesday, December 10, 2019 8:48 PM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hi Shengyu,

                

                It's hard to further diagnose without the logs. Can you share your latest daos_server.yml, full daos_control.log and full server.log? In the daos_server.yml, please set control_log_mask: DEBUG and in the io server section, set log_mask: DEBUG

                

                Regards,

                Joel

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang

                Sent: Tuesday, December 10, 2019 1:42 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hello Joel,

                

                I didn't set OFI_DOMAIN=mlx5_0 before, from the hint of Alex, I set it yesterday, there then there is another error while creating container:

                mgmt ERR  src/mgmt/cli_mgmt.c:325 get_attach_info() GetAttachInfo unsuccessful: 2

                

                

                Regards,

                Shengyu

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Rosenzweig, Joel B

                Sent: Monday, December 9, 2019 11:47 PM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hi Shengyu,

                

                With your latest setup, can you launch DAOS and send your latest daos_control.log?  In particular, I want to see how the daos_io_server environment variables are set.  For example these two lines below show the command line args and the environment variables in use with an ofi+sockets/ib1 config.  I'm looking to see if we are setting OFI_DOMAIN=mlx5_0 in your environment.  I seem to recall that your earlier logs did have this set, but since builds have changed since then, it's worth checking out one more time.

                

                DEBUG 16:33:15.711949 exec.go:112: daos_io_server:1 args: [-t 8 -x 2 -g daos_server -d /tmp/daos_sockets -s /mnt/daos1 -i 1842327892 -p 1 -I 1] DEBUG 16:33:15.711963 exec.go:113: daos_io_server:1 env: [CRT_TIMEOUT=30 FI_SOCKETS_CONN_TIMEOUT=2000 D_LOG_FILE=/tmp/server.log CRT_CTX_SHARE_ADDR=0 FI_SOCKETS_MAX_CONN_RETRY=1 D_LOG_MASK=ERR CRT_PHY_ADDR_STR=ofi+sockets OFI_INTERFACE=ib1 OFI_PORT=31416 DAOS_MD_CAP=1024]

                

                If we are not setting OFI_DOMAIN=mlx5_0 automatically, go ahead and edit your daos_server.yml and add OFI_DOMAIN=mlx5_0 to the env_vars section (per below) and see what you get.  If that doesn't get you going, please send the log for that run, too.

                

                env_vars:

                  - OFI_DOMAIN=mlx5_0

                

                Regards,

                Joel

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Oganezov, Alexander A

                Sent: Monday, December 9, 2019 5:03 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hi Shengyu,

                

                Actually this is really good result - two servers were able to exchange rpcs. That's the end of test -- just ctrl+C out of it.

                

                I'll let Joel take over for daos help on how to make daos use mlx5_0 domain.

                

                Thanks,

                ~~Alex.

                

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang

                Sent: Monday, December 09, 2019 12:57 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hello Alex,

                

                Please see the new log below, and the test seems dead (no more response):

                

                ibv_exp_query_device: invalid comp_mask !!! (comp_mask = 0x7fdd21092400 valid_mask = 0x3) [afa1][[44100,1],1][btl_openib_component.c:1670:init_one_device] error obtaining device attributes for mlx5_0 errno says Invalid argument

                ibv_exp_query_device: invalid comp_mask !!! (comp_mask = 0x195aad0 valid_mask = 0x3) [afa1][[44100,1],0][btl_openib_component.c:1670:init_one_device] error obtaining device attributes for mlx5_0 errno says Invalid argument

                --------------------------------------------------------------------------

                WARNING: There was an error initializing an OpenFabrics device.

                

                  Local host:   afa1

                  Local device: mlx5_0

                --------------------------------------------------------------------------

                12/09-03:53:20.32 afa1 CaRT[101464/101464] crt  WARN src/cart/crt_init.c:270 crt_init_opt() PMIX disabled. Disabling LM automatically

                12/09-03:53:20.32 afa1 CaRT[101465/101465] crt  WARN src/cart/crt_init.c:270 crt_init_opt() PMIX disabled. Disabling LM automatically SRV [rank=1 pid=101465] Server starting, self_rank=1 SRV [rank=0 pid=101464] Server starting, self_rank=0 SRV [rank=1 pid=101465] >>>> Entered iv_set_ivns SRV [rank=1 pid=101465] <<<< Exited iv_set_ivns:773

                

                [afa1:101458] 1 more process has sent help message help-mpi-btl-openib.txt / error in device init [afa1:101458] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

                

                Regards,

                Shengyu

                

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Oganezov, Alexander A

                Sent: Monday, December 9, 2019 4:30 PM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hi Shengyu,

                

                Given your output can you try this now?

                

                orterun --allow-run-as-root -np 2 -x CRT_PHY_ADDR_STR="ofi+verbs;ofi_rxm" -x OFI_INTERFACE=ib0 -x OFI_DOMAIN=mlx5_0 ../bin/crt_launch -e tests/iv_server -v 3

                

                Please note additional OFI_DOMAIN envariable.

                

                Thanks,

                ~~Alex.

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang

                Sent: Monday, December 09, 2019 12:25 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hello Alex,

                

                Here is outputs of fio_info related to verbs:

                

                provider: verbs

                    fabric: IB-0xfe80000000000000

                    domain: mlx5_2

                    version: 1.0

                    type: FI_EP_MSG

                    protocol: FI_PROTO_RDMA_CM_IB_RC

                provider: verbs

                    fabric: IB-0xfe80000000000000

                    domain: mlx5_2-xrc

                    version: 1.0

                    type: FI_EP_MSG

                    protocol: FI_PROTO_RDMA_CM_IB_XRC

                provider: verbs

                    fabric: IB-0xfe80000000000000

                    domain: mlx5_0

                    version: 1.0

                    type: FI_EP_MSG

                    protocol: FI_PROTO_RDMA_CM_IB_RC

                provider: verbs

                    fabric: IB-0xfe80000000000000

                    domain: mlx5_0-xrc

                    version: 1.0

                    type: FI_EP_MSG

                    protocol: FI_PROTO_RDMA_CM_IB_XRC

                provider: verbs

                    fabric: IB-0xfe80000000000000

                    domain: mlx5_0-dgram

                    version: 1.0

                    type: FI_EP_DGRAM

                    protocol: FI_PROTO_IB_UD

                provider: verbs

                    fabric: IB-0xfe80000000000000

                    domain: mlx5_2-dgram

                    version: 1.0

                    type: FI_EP_DGRAM

                    protocol: FI_PROTO_IB_UD

                provider: verbs;ofi_rxm

                    fabric: IB-0xfe80000000000000

                    domain: mlx5_2

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXM

                provider: verbs;ofi_rxm

                    fabric: IB-0xfe80000000000000

                    domain: mlx5_0

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXM

                provider: tcp;ofi_rxm

                    fabric: TCP-IP

                    domain: tcp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXM

                provider: tcp;ofi_rxm

                    fabric: TCP-IP

                    domain: tcp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXM

                provider: tcp;ofi_rxm

                    fabric: TCP-IP

                    domain: tcp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXM

                provider: tcp;ofi_rxm

                    fabric: TCP-IP

                    domain: tcp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXM

                provider: tcp;ofi_rxm

                    fabric: TCP-IP

                    domain: tcp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXM

                provider: tcp;ofi_rxm

                    fabric: TCP-IP

                    domain: tcp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXM

                provider: tcp;ofi_rxm

                    fabric: TCP-IP

                    domain: tcp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXM

                provider: tcp;ofi_rxm

                    fabric: TCP-IP

                    domain: tcp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXM

                provider: tcp;ofi_rxm

                    fabric: TCP-IP

                    domain: tcp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXM

                provider: tcp;ofi_rxm

                    fabric: TCP-IP

                    domain: tcp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXM

                provider: verbs;ofi_rxd

                    fabric: IB-0xfe80000000000000

                    domain: mlx5_2-dgram

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXD

                provider: verbs;ofi_rxd

                    fabric: IB-0xfe80000000000000

                    domain: mlx5_0-dgram

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXD

                provider: UDP;ofi_rxd

                    fabric: UDP-IP

                    domain: udp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXD

                provider: UDP;ofi_rxd

                    fabric: UDP-IP

                    domain: udp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXD

                provider: UDP;ofi_rxd

                    fabric: UDP-IP

                    domain: udp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXD

                provider: UDP;ofi_rxd

                    fabric: UDP-IP

                    domain: udp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXD

                provider: UDP;ofi_rxd

                    fabric: UDP-IP

                    domain: udp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXD

                provider: UDP;ofi_rxd

                    fabric: UDP-IP

                    domain: udp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXD

                provider: UDP;ofi_rxd

                    fabric: UDP-IP

                    domain: udp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXD

                provider: UDP;ofi_rxd

                    fabric: UDP-IP

                    domain: udp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXD

                provider: UDP;ofi_rxd

                    fabric: UDP-IP

                    domain: udp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXD

                provider: UDP;ofi_rxd

                    fabric: UDP-IP

                    domain: udp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXD

                

                Regards,

                Shengyu

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Oganezov, Alexander A

                Sent: Monday, December 9, 2019 4:08 PM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Thanks,

                Can you also provide full fi_info output?

                

                ~~Alex.

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang

                Sent: Monday, December 09, 2019 12:05 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hello Alex,

                

                Here are the outputs:

                

                orterun --allow-run-as-root -np 2 -x CRT_PHY_ADDR_STR="ofi+verbs;ofi_rxm" -x OFI_INTERFACE=ib0 ../bin/crt_launch -e tests/iv_server -v 3

                ibv_exp_query_device: invalid comp_mask !!! (comp_mask = 0x2094f90 valid_mask = 0x3) [afa1][[18544,1],1][btl_openib_component.c:1670:init_one_device] error obtaining device attributes for mlx5_0 errno says Invalid argument

                ibv_exp_query_device: invalid comp_mask !!! (comp_mask = 0xacda60 valid_mask = 0x3) [afa1][[18544,1],0][btl_openib_component.c:1670:init_one_device] error obtaining device attributes for mlx5_0 errno says Invalid argument

                --------------------------------------------------------------------------

                WARNING: There was an error initializing an OpenFabrics device.

                

                  Local host:   afa1

                  Local device: mlx5_0

                --------------------------------------------------------------------------

                12/09-03:01:03.53 afa1 CaRT[92269/92269] hg   ERR  # NA -- Error -- /root/daos/_build.external/mercury/src/na/na_ofi.c:1609

                 # na_ofi_domain_open(): No provider found for "verbs;ofi_rxm" provider on domain "ib0"

                12/09-03:01:03.53 afa1 CaRT[92268/92268] hg   ERR  # NA -- Error -- /root/daos/_build.external/mercury/src/na/na_ofi.c:1609

                 # na_ofi_domain_open(): No provider found for "verbs;ofi_rxm" provider on domain "ib0"

                12/09-03:01:03.53 afa1 CaRT[92269/92269] hg   ERR  # NA -- Error -- /root/daos/_build.external/mercury/src/na/na_ofi.c:2975

                 # na_ofi_initialize(): Could not open domain for verbs;ofi_rxm, ib0

                12/09-03:01:03.53 afa1 CaRT[92269/92269] hg   ERR  # NA -- Error -- /root/daos/_build.external/mercury/src/na/na.c:324

                 # NA_Initialize_opt(): Could not initialize plugin

                12/09-03:01:03.53 afa1 CaRT[92269/92269] hg   ERR  src/cart/crt_hg.c:525 crt_hg_init() Could not initialize NA class.

                12/09-03:01:03.53 afa1 CaRT[92269/92269] crt  ERR  src/cart/crt_init.c:347 crt_init_opt() crt_hg_init failed rc: -1020.

                12/09-03:01:03.53 afa1 CaRT[92269/92269] crt  ERR  src/cart/crt_init.c:421 crt_init_opt() crt_init failed, rc: -1020.

                12/09-03:01:03.53 afa1 CaRT[92268/92268] hg   ERR  # NA -- Error -- /root/daos/_build.external/mercury/src/na/na_ofi.c:2975

                 # na_ofi_initialize(): Could not open domain for verbs;ofi_rxm, ib0

                12/09-03:01:03.53 afa1 CaRT[92268/92268] hg   ERR  # NA -- Error -- /root/daos/_build.external/mercury/src/na/na.c:324

                 # NA_Initialize_opt(): Could not initialize plugin

                12/09-03:01:03.53 afa1 CaRT[92268/92268] hg   ERR  src/cart/crt_hg.c:525 crt_hg_init() Could not initialize NA class.

                12/09-03:01:03.53 afa1 CaRT[92268/92268] crt  ERR  src/cart/crt_init.c:347 crt_init_opt() crt_hg_init failed rc: -1020.

                12/09-03:01:03.53 afa1 CaRT[92268/92268] crt  ERR  src/cart/crt_init.c:421 crt_init_opt() crt_init failed, rc: -1020.

                [afa1:92262] 1 more process has sent help message help-mpi-btl-openib.txt / error in device init [afa1:92262] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

                

                

                Regards,

                Shengyu

                

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Oganezov, Alexander A

                Sent: Saturday, December 7, 2019 1:14 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hi Shengyu,

                

                With latest daos code and mofed 4.6 installed can you rerun this and show what that one gives you?

                

                source scons_local/utils/setup_local.sh

                cd install/Linux/TESTING

                orterun -np 2 -x CRT_PHY_ADDR_STR="ofi+verbs;ofi_rxm" -x OFI_INTERFACE=ib0 ../bin/crt_launch -e tests/iv_server -v 3

                

                Thanks,

                ~~Alex.

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang

                Sent: Thursday, December 05, 2019 10:25 PM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hello Alex,

                

                Yes, however with 4.6 the result is same.

                After I upgraded daos code to newest of master branch, I got some different results, daos io server seems started OK, since I can see lots of fd points to rdma_cm.

                But daos client seems can't connect to server due to same error (can't find efi+verbs provider on ib0) like the log shows, you may find the log in the attachment, that is crated via "create container"

                

                Regards,

                Shengyu.

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Oganezov, Alexander A

                Sent: Wednesday, December 4, 2019 4:02 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hi Shengyu,

                

                Can you try installing MOFED 4.6 packages on your system? In general MOFED is required to get verbs over Mellanox working.

                Those packages can be found at: https://www.mellanox.com/page/mlnx_ofed_matrix?mtag=linux_sw_drivers

                

                There is also 4.7 version available, however there seem to be few longevity issues currently when using 4.7 (according to verbs ofi maintainers).

                

                Thanks,

                ~~Alex.

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang

                Sent: Monday, November 25, 2019 9:55 PM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hello Alex,

                

                Thanks for your suggestion, here is the log:

                

                

                mca_base_component_repository_open: unable to open mca_pml_ucx: libucp.so.0: cannot open shared object file: No such file or directory (ignored)

                11/26-00:40:22.65 afa1 CaRT[365504/365504] hg   ERR  # NA -- Error -- /root/daos/_build.external/mercury/src/na/na_ofi.c:1407

                 # na_ofi_getinfo(): fi_getinfo() failed, rc: -61(No data available)

                11/26-00:40:22.65 afa1 CaRT[365504/365504] hg   ERR  # NA -- Error -- /root/daos/_build.external/mercury/src/na/na_ofi.c:2816

                 # na_ofi_check_protocol(): na_ofi_getinfo() failed

                11/26-00:40:22.65 afa1 CaRT[365504/365504] hg   ERR  # NA -- Error -- /root/daos/_build.external/mercury/src/na/na.c:302

                 # NA_Initialize_opt(): No suitable plugin found that matches ofi+verbs;ofi_rxm://192.168.80.120

                11/26-00:40:22.65 afa1 CaRT[365504/365504] hg   ERR  src/cart/crt_hg.c:521 crt_hg_init() Could not initialize NA class.

                11/26-00:40:22.65 afa1 CaRT[365504/365504] crt  ERR  src/cart/crt_init.c:347 crt_init_opt() crt_hg_init failed rc: -1020.

                11/26-00:40:22.65 afa1 CaRT[365504/365504] crt  ERR  src/cart/crt_init.c:421 crt_init_opt() crt_init failed, rc: -1020.

                

                

                Regards,

                Shengyu

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Oganezov, Alexander A

                Sent: Tuesday, November 26, 2019 12:21 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hi Shengyu,

                

                In order to figure out what is the issue on your system could you run cart standalone test instead and provide the output that you get?

                

                cd daos_dir

                source scons_local/utils/setup_local.sh

                cd install/Linux/TESTING

                orterun -np 2 -x CRT_PHY_ADDR_STR="ofi+verbs;ofi_rxm" -x OFI_INTERFACE=ib0 ../bin/crt_launch -e tests/iv_server -v 3

                

                Note: Depending on how you installed daos your paths might be different, so instead of cd install/Linux/TESTING you might have to cd into different directory first where you have tests/iv_server in. I think in your env it will be   cd /root/daos/install/TESTING/  or cd /root/daos/install/cart/TESTING.

                

                

                Expected output:

                11/25-15:51:48.39 wolf-55 CaRT[53295/53295] crt  WARN src/cart/crt_init.c:270 crt_init_opt() PMIX disabled. Disabling LM automatically

                11/25-15:51:48.40 wolf-55 CaRT[53296/53296] crt  WARN src/cart/crt_init.c:270 crt_init_opt() PMIX disabled. Disabling LM automatically SRV [rank=0 pid=53295]  Server starting, self_rank=0 SRV [rank=1 pid=53296]  Server starting, self_rank=1 SRV [rank=1 pid=53296]  >>>> Entered iv_set_ivns SRV [rank=1 pid=53296]  <<<< Exited iv_set_ivns:773

                

                Thanks,

                ~~Alex.

                

                

                -----Original Message-----

                From: daos@daos.groups.io [mailto:daos@daos.groups.io] On Behalf Of Shengyu SY19 Zhang

                Sent: Monday, November 25, 2019 3:28 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hello Joel,

                

                As the shown in the output.log, there is only one version of libfabrics installed in my machine, and actually I don't nave other software which depends libfabraics installed.

                From you guide to set FI_LOG_LEVEL=debug, I can see the following message, may be helpful:

                

                libfabric:123445:verbs:fabric:fi_ibv_set_default_attr():1263<info> Ignoring provider default value for tx rma_iov_limit as it is greater than the value supported by domain: mlx5_0 libfabric:123445:verbs:fabric:fi_ibv_get_matching_info():1365<info> hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints

                ERROR: daos_io_server:0 libfabric:123445:verbs:core:fi_ibv_check_hints():231<info> Unsupported capabilities libfabric:123445:verbs:core:fi_ibv_check_hints():232<info> Supported: FI_MSG, FI_RECV, FI_SEND, FI_LOCAL_COMM, FI_REMOTE_COMM libfabric:123445:verbs:core:fi_ibv_check_hints():232<info> Requested: FI_MSG, FI_RMA, FI_READ, FI_RECV, FI_SEND, FI_REMOTE_READ

                ERROR: daos_io_server:0 libfabric:123445:verbs:fabric:fi_ibv_get_rai_id():179<info> rdma_bind_addr: No such device(19)

                ERROR: daos_io_server:0 libfabric:123445:verbs:fabric:fi_ibv_get_rai_id():179<info> rdma_bind_addr: No such device(19)

                ERROR: daos_io_server:0 libfabric:123445:verbs:fabric:fi_ibv_get_rai_id():179<info> rdma_bind_addr: Invalid argument(22)

                ERROR: daos_io_server:0 libfabric:123445:verbs:fabric:fi_ibv_get_rai_id():179<info> rdma_bind_addr: Invalid argument(22)

                ERROR: daos_io_server:0 libfabric:123445:verbs:fabric:fi_ibv_get_rai_id():179<info> rdma_bind_addr: Invalid argument(22)

                ERROR: daos_io_server:0 libfabric:123445:core:core:ofi_layering_ok():795<info> Need core provider, skipping ofi_rxd libfabric:123445:core:core:ofi_layering_ok():795<info> Need core provider, skipping ofi_mrail

                

                

                Regards,

                Shengyu.

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Rosenzweig, Joel B

                Sent: Saturday, November 23, 2019 3:20 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hi Shengyu,

                

                The debug output showed me that when daos_server is started via orterun, libfabric is not finding provider support for ofi_rxm at least.  I'm still wondering if you have two different versions of libfabric installed on your machine.

                

                Can you run these commands and provide the output?

                

                1) ldd install/bin/daos_server

                2) modify your orterun command to run ldd on daos_server.   For example, I run this command locally:

                orterun --allow-run-as-root --map-by node --mca btl tcp,self --mca oob tcp -np 1 --hostfile /home/jbrosenz/daos/hostfile --enable-recovery --report-uri /tmp/urifile ldd /home/jbrosenz/daos/install/bin/daos_server

                3) which fi_info

                4) ldd over each version of fi_info found

                

                From the data you provide, I'll understand if the libfabric being used by daos_server when executed directly by you in the shell is the same libfabric being used by daos_server when executed via orterun.  Your original "daos_server network scan" output showed support for ofi+verbs;ofi_rxm but your debug output showed that when daos_server was started (via orterun), libfabric could not find support for the very same providers.  If there are two different versions being used with different configurations, it would explain the failure.  If it's a single installation/configuration, then that will lead the debug in another direction.

                

                Depending on what you find through 1-4, you might find it helpful to export the environment variable FI_LOG_LEVEL=debug which will instruct libfabric to output a good deal of debug info.

                

                Regards,

                Joel

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang

                Sent: Friday, November 22, 2019 12:59 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hello Joel,

                

                Please see those files in attachment.

                I have tried two machines, one have full provider shows in fi_info (verbs and rxm), another doesn't show verbs, but they are same can't start io_server. I found the project conflicts with mellanox drivers, therefor I remove it and use yum package only, however still keep not working.

                

                

                Regards,

                Shengyu

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Rosenzweig, Joel B

                Sent: Friday, November 22, 2019 6:35 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hi Shengyu,

                

                Can you share your daos_server.yml so we can see how you enabled the provider?  And, can you share the log files daos_control.log and server.log so we can see more context?

                

                Thank you,

                Joel

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang

                Sent: Wednesday, November 20, 2019 9:23 PM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hello,

                

                Thank you for your help Alex, Joel and Kevin, I have checked those steps that you provided:

                

                Ibstat:

                State: Active

                Physical state: LinkUp

                

                Ifconfig:

                flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 2044

                

                fi_info:

                verbs:

                    version: 1.0

                ofi_rxm:

                    version: 1.0

                ofi_rxd:

                    version: 1.0

                

                And network is good since I can run SPDK NVMe-oF over Infiniband with good working.

                I also specified "ofi+verbs;ofi_rxm", the same error occurred, the ioserver will be stopped after a while, and print log as I provided previously.

                

                And I noticed, whatever I specify ofi+verbs, ofi_rxm, or ofi+verbs;ofi_rxm, the log keep shows No provider found for "verbs;ofi_rxm" provider on domain "ib0", is it the cause?

                

                BTW: it is working under ofi+sockets.

                

                

                Regards,

                Shengyu.

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Oganezov, Alexander A

                Sent: Thursday, November 21, 2019 7:13 AM

                To: daos@daos.groups.io

                Subject: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hi Shengyu,

                

                > However if I specify either ofi+verbs or ofi_rxm, the same error will happen, and io_server will stop.

                > na_ofi.c:1609

                > # na_ofi_domain_open(): No provider found for "verbs;ofi_rxm" provider on domain "ib0"

                

                To use supported verbs provider you need to have "ofi+verbs;ofi_rxm" in the provider string.

                

                ~~Alex.

                

                -----Original Message-----

                From: daos@daos.groups.io [mailto:daos@daos.groups.io] On Behalf Of Rosenzweig, Joel B

                Sent: Wednesday, November 20, 2019 7:37 AM

                To: daos@daos.groups.io

                Subject: Re: [daos] Does DAOS support infiniband now?

                

                Hi Shengyu,

                

                The daos_server network scan uses information provided by libfabric to determine available devices and providers.  It then cross references that list of devices with device names obtained from hwloc to convert libfabric device names (as necessary) to those you'd find via ifconfig.  Therefore, if "daos_server network scan" displays a device and provider, it means that support for that via libfabric has been provided.   However, as Kevin pointed out, it's possible that the device itself was down, and that could certainly generate an error like what you encountered.  There's another possibility, that you might have more than one version of libfabric installed in your environment.  I have run into this situation in our lab environment.  You might check your target system to see if it has more than one libfabric library with different provider support.

                

                Regards,

                Joel

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Harms, Kevin via Groups.Io

                Sent: Wednesday, November 20, 2019 10:04 AM

                To: daos@daos.groups.io

                Subject: Re: [daos] Does DAOS support infiniband now?

                

                Shengyu,

                

                  I have tried IB and it works. Verify the libfabric verbs provider is available.

                

                  fi_info -l

                

                 you should see these:

                

                ofi\_rxm:

                version: 1.0

                

                verbs:

                version: 1.0

                

                  See here for details:

                

                

                  You might also want to confirm ib0 is in the UP state:

                

                [root@daos01 ~]# ifconfig ib0

                ib0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 4092

                        inet 172.25.6.101  netmask 255.255.0.0  broadcast 172.25.255.255

                

                kevin

                

                ________________________________________

                From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Shengyu SY19 Zhang <zhangsy19@...>

                Sent: Wednesday, November 20, 2019 2:54 AM

                To: daos@daos.groups.io

                Subject: [daos] Does DAOS support infiniband now?

                

                Hello,

                

                I use daos_server network scan, it shows as following:

                fabric_iface: ib0

                       provider: ofi+verbs;ofi_rxm

                       pinned_numa_node: 1

                

                However if I specify either ofi+verbs or ofi_rxm, the same error will happen, and io_server will stop.

                na_ofi.c:1609

                # na_ofi_domain_open(): No provider found for "verbs;ofi_rxm" provider on domain "ib0"

                

                The ib0 is Mellanox nic over Infiniband network.

                

                Regards,

                Shengyu.

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

            

            

            

            

            

            

            

            

        

        

        

        

        

        

        

        

    

    

    

    

    

    

    

    

    

    

    

 

 

 

 

 

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.


Re: [External] Re: [daos] Does DAOS support infiniband now?

Oganezov, Alexander A
 

Shengyu,

 

Can you provide full log from start until first occurrence of this error?

 

Thanks,

~~Alex.

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang
Sent: Monday, January 20, 2020 11:02 PM
To: daos@daos.groups.io
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Hello Alex,

 

I have removed mercury, the io_server seems started normally ,however the daos_server.log increasing quickly to eat all my free space. It infinite repeat the three lines:

01/21-01:40:49.28 afa1 DAOS[72964/73009] hg   ERR  src/cart/crt_hg.c:1331 crt_hg_trigger() HG_Trigger failed, hg_ret: 18.
01/21-01:40:49.28 afa1 DAOS[72964/73009] rpc  ERR  src/cart/crt_context.c:1255 crt_progress() crt_hg_progress failed, rc: -1020.
01/21-01:40:49.28 afa1 DAOS[72964/73009] server ERR  src/iosrv/srv.c:469 dss_srv_handler() failed to progress CART context: -1020

 

Regards,

Shengyu,

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Oganezov, Alexander A
Sent: Tuesday, January 21, 2020 4:49 AM
To: daos@daos.groups.io
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

One other thing Shengyu,

 

Can you verify that your mercury build is up to date? Based on domain being used as “mlx5_0/192.168.80.161” it sounds like there is mismatch between what CaRT generates and what mercury consumes; there has been change few weeks ago regarding how domain is provided to mercury level, and it feels as if older mercury is being used.

 

To ensure clean mercury build remove libmercury* libna*  from your install/lib location, remove _build.external-Linux/mercury directory and recompile daos with scons –build-deps=yes –config=force install

 

Thanks,

~~Alex.

 

From: Oganezov, Alexander A
Sent: Monday, January 20, 2020 11:56 AM
To: daos@daos.groups.io
Subject: RE: [External] Re: [daos] Does DAOS support infiniband now?

 

Hi Shengyu,

 

What does this command return to you on the node where you see this strange domain name?

fi_info --provider="verbs"

 

Thanks,

~~Alex.

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang
Sent: Monday, January 20, 2020 12:32 AM
To: daos@daos.groups.io
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Hello Joel,

 

I have some more information,

In the file na_ofi.c + 1609, the io_serve exit there, and on above code, na_ofi_verify_provider function will compare domain names, the domain is “mlx5_0/192.168.80.161”, while prov->domain_attr.name is “mlx5_0”,  it will always return FALSE.

 

Regards,

Shengyu

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang
Sent: Sunday, January 19, 2020 6:15 PM
To: daos@daos.groups.io
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Hello Joel,

 

Network scan:

Scanning fabric for YML specified provider: ofi+verbs;ofi_rxm
Fabric scan found 1 devices matching the provider spec: ofi+verbs;ofi_rxm

fabric_iface: ib0
provider: ofi+verbs;ofi_rxm
pinned_numa_node: 1

 

 

And log attached, it is standalone server.

Additionally, I created two identical virtual machines with IB sr-iov pass-through, however one vm can’t start due to the same problem, while another can start normally. They were using ofi+verbs;ofi_rxm as provider.

 

 

Regards,

Shengyu

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Rosenzweig, Joel B
Sent: Friday, January 17, 2020 11:03 PM
To: daos@daos.groups.io
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Hello Shengyu,

 

It appears that your daos_server.yml is specifying the provider as “ofi+verbs” but I think it should be set to “ofi+verbs;ofi_rxm”.  Can you configure your daos_server.yml with that and try again?  And then, if things still do not work, then please provide:

 

1)     the network scan output again after you make the provider change

2)     the portion of the debug log that shows the environment variables being provided to the daos_io_server.  This will show us what is being set for OFI_INTERFACE, OFI_DOMAIN in the daos_io_server environment.

3)     the daos_server.yml so I can see how you have configured each daos_io_server.

 

Regards,

Joel

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang
Sent: Friday, January 17, 2020 4:55 AM
To: daos@daos.groups.io
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Hello Johann,

 

Here is logs:

 

Scanning fabric for YML specified provider: ofi+verbs
Fabric scan found 4 devices matching the provider spec: ofi+verbs

fabric_iface: ib2
provider: ofi+verbs;ofi_rxm
pinned_numa_node: 1


fabric_iface: ib0
provider: ofi+verbs;ofi_rxm
pinned_numa_node: 1


fabric_iface: ib2
provider: ofi+verbs
pinned_numa_node: 1


fabric_iface: ib0
provider: ofi+verbs
pinned_numa_node: 1

 

And please see my inline comments.

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Lombardi, Johann
Sent: Friday, January 17, 2020 4:51 PM
To: daos@daos.groups.io; Rosenzweig, Joel B <joel.b.rosenzweig@...>
Cc: Macdonald, Mjmac <mjmac.macdonald@...>
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Hi,

 

Cc’ing Joel.

Those errors indicate that you still have some network setup issue. Could you please run daos_server network scan?

 

e.g.:

[root@wolf-118 ~]# daos_server network scan -p "ofi+verbs;ofi_rxm"

Scanning fabric for cmdline specified provider: ofi+verbs;ofi_rxm

Fabric scan found 3 devices matching the provider spec: ofi+verbs;ofi_rxm

 

                fabric_iface: ib0

                provider: ofi+verbs;ofi_rxm

                pinned_numa_node: 0

 

 

                fabric_iface: ib1

                provider: ofi+verbs;ofi_rxm

                pinned_numa_node: 1

 

 

                fabric_iface: eth0

                provider: ofi+verbs;ofi_rxm

                pinned_numa_node: 0

 

As for the NVMe issue, do I understand correctly that:

-        The PCI addresses of the 8 NVMe SSDs show up fine via daos_server storage scan

Yes.

 

-        You have reduced the number of huge pages to 4096 pages (8GB) and all the SPDK errors related to failed huge pages allocation are gone as well as this error from the log:
SPDK prepare: spdk setup failed (): exit status 1: stdout: 0000:00:05.0 (8086 0a54): nvme -> uio_pci_generic

Yes, only reporting pci addr not found and cant start, then reduced nvme to 1 seems pass this step.

-        But the io_server fails to start after dmg storage format?

Yes.

 

I had a second look at daos_control2.log and noticed that you are using 20 targets while you have 8 SSDs and 10 cores. Could you please try with #targets = #SSDs = 8 and set nr_xs_helpers to 0?

This config file was copies from physical machine, maybe some problem, several days ago, it and physical server are working, tried your suggestion, still same while formatting and cant start again.

daos_io_server:0 Using legacy core allocation algorithm
0 target XS(xstream) requested (#cores 10); use (7) target XS
DEBUG 04:51:52.588209 main.go:82: exit status 1
/root/daos/install/bin/daos_io_server (instance 0) exited

 

 

Cheers,

Johann

 

From: <daos@daos.groups.io> on behalf of Shengyu SY19 Zhang <zhangsy19@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Friday 17 January 2020 at 08:21
To: "daos@daos.groups.io" <daos@daos.groups.io>
Cc: "Macdonald, Mjmac" <mjmac.macdonald@...>
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Hello Johann,

 

For the first issue (daos_control.log), it is physical machine and specified 8 NVMes, daos server report cant find PCI address (created the log).

Then I use only one NVMe, daos server can start normally, however it stopped after dmg storage format, just like the second issue (daos_control2.log).

 

DEBUG 02:18:19.457860 config.go:378: Active config saved to /root/daos/install/etc/.daos_server.active.yml (read-only)
01/17-02:18:31.74 afa1 DAOS[185862/185862] hg   ERR  # NA -- Error -- /root/daos/_build.external/mercury/src/na/na_ofi.c:1609
 # na_ofi_domain_open(): No provider found for "verbs;ofi_rxm" provider on domain "mlx5_0/192.168.80.120"
01/17-02:18:31.74 afa1 DAOS[185862/185862] hg   ERR  # NA -- Error -- /root/daos/_build.external/mercury/src/na/na_ofi.c:2975
 # na_ofi_initialize(): Could not open domain for verbs;ofi_rxm, mlx5_0/192.168.80.120
01/17-02:18:31.74 afa1 DAOS[185862/185862] hg   ERR  # NA -- Error -- /root/daos/_build.external/mercury/src/na/na.c:324
 # NA_Initialize_opt(): Could not initialize plugin
01/17-02:18:31.74 afa1 DAOS[185862/185862] hg   ERR  src/cart/crt_hg.c:526 crt_hg_init() Could not initialize NA class.
01/17-02:18:31.74 afa1 DAOS[185862/185862] crt  ERR  src/cart/crt_init.c:317 crt_init_opt() crt_hg_init failed rc: -1020.
01/17-02:18:31.74 afa1 DAOS[185862/185862] crt  ERR  src/cart/crt_init.c:385 crt_init_opt() crt_init failed, rc: -1020.

 

Regards,

Shengyu,

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Lombardi, Johann
Sent: Thursday, January 16, 2020 8:24 PM
To: daos@daos.groups.io
Cc: Macdonald, Mjmac <mjmac.macdonald@...>
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Hi Shengyu,

 

Mike looked into the logs you provided and noticed that:

 

## ERROR: requested 40960 hugepages but only 8585 could be allocated.

## Memory might be heavily fragmented. Please try flushing the system cache, or reboot the machine.

 

Maybe you meant to specify 4096 in the yaml file instead of 40960 for nr_hugepages?  It sounds like you are trying to allocate 82GB of RAM for hugepages.

 

Cheers,

Johann

 

 

From: <daos@daos.groups.io> on behalf of "Nabarro, Tom" <tom.nabarro@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Wednesday 15 January 2020 at 13:17
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

There seems to be multiple messages about not being able to meet the required number of hugepages requested. I was not involved in the work but there have been changes in how huge page allocation is performed for each of the daos_io_server instances and we are now performing automatic storage prepare when starting daos_server.

 

What I would suggest is to reboot the nodes (in case the issues to do with memory fragmentation) and try with a smaller number of drives configured in the config file. Don’t try to do a storage prepare before running daos_server (as it will be performed on start-up anyway). And update to recent master before trying please. You could also try to bump the number of huge pages specified in the server config file to maybe 8192?

 

Regards,

Tom Nabarro – DCG/ESAD

M: +44 (0)7786 260986

Skype: tom.nabarro

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang
Sent: Wednesday, January 15, 2020 3:27 AM
To: daos@daos.groups.io
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Hello

 

Sorry I forgot attachments.

 

Regards,

Shengyu

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang
Sent: Wednesday, January 15, 2020 11:21 AM
To: daos@daos.groups.io
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Hello Johann,

 

I couldnt check it at this time, since I cant start daos server (newest code of master) on my two different machines, I assume there are lots of modification in control path, there could be some new issues:

1.      Daos_server storage prepare -nvme-only, all my NVMe disks switched to uio as expected, then issue storage scan, can see those disks as expected as well.

However, when I start daos server, it report error that reporting cant find PCI address, and all NVMs switched to kernel driver, see daos_control.log

2.      On another machine, it just stopped after being formatted, see doas_control2.log

 

Regards,

Shengyu,

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Lombardi, Johann
Sent: Tuesday, January 14, 2020 5:03 AM
To: daos@daos.groups.io
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Thanks for the logs Shengyu. It does not seem to be related to wrong endpoint addresses. The client did find the server, but this latter returned DER_NONEXIST when connecting to the pool. It might be the same problem fixed recently by PR  #1701. Could you please apply the patch or try with latest master?

 

Cheers,

Johann

 

From: <daos@daos.groups.io> on behalf of Shengyu SY19 Zhang <zhangsy19@...>
Reply-To: "
daos@daos.groups.io" <daos@daos.groups.io>
Date: Friday 10 January 2020 at 07:20
To: "
daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Hello Johann,

 

Here are log files.

My step:

1.      Create fresh environment and start daos server and then format it.

2.      Create pool

3.      Create container.

4.      List container: daos container query -svc=0 path=/tmp/mycontainer, that work great.

5.      CTRl +C to kill daos server

6.      Restart daos server

7.      Repeat 4, daos process will be dead with infinitely loop.

 

Regards,

Shengyu.

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Lombardi, Johann
Sent: Thursday, January 9, 2020 3:45 PM
To: daos@daos.groups.io
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Hm, the issue with server restart might be due to the endpoint address of the servers not being persistent.

Could you please collect full debug logs for the fresh start with reformat and the subsequent restart?

 

Cheers,

Johann

 

From: <daos@daos.groups.io> on behalf of Shengyu SY19 Zhang <zhangsy19@...>
Reply-To: "
daos@daos.groups.io" <daos@daos.groups.io>
Date: Thursday 9 January 2020 at 07:22
To: "
daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Hello,

 

@Johann, Newest version will work on new formatted and created pool, yes, I did set in the yaml.

@Kal, no, I still meet the issue after io_server restart, seems there issues after load existing data.

 

Regards,

Shengyu.

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Lombardi, Johann
Sent: Thursday, January 9, 2020 5:31 AM
To: daos@daos.groups.io
Subject: [External] Re: [daos] Does DAOS support infiniband now?

 

Hi Kevan & Shengyu,

 

Could you please advise what commit hash you use? Also, are you specifying in “fabric_iface_port” in the yaml file?

 

Cheers,

Johann

 

From: <daos@daos.groups.io> on behalf of Kevan Rehm <kevan.rehm@...>
Reply-To: "
daos@daos.groups.io" <daos@daos.groups.io>
Date: Wednesday 8 January 2020 at 21:17
To: "
daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [daos] Does DAOS support infiniband now?

 

I have not had any success, I see the same failure sequence as Shengyu, and due to other commitments I've had to set this aside for a few days.   Hope to get back to it in a week or so.

 

Kevan

 

On 1/8/20, 2:15 PM, "daos@daos.groups.io on behalf of Alfizah, Kurniawan" <daos@daos.groups.io on behalf of kurniawan.alfizah@...> wrote:

 

    Hello Shengyu and Kevan,

    I'm wondering if you have resolved this problem and that DAOS is working well with IB.

    

    Cheers,

    Kal

    

    -----Original Message-----

    From: daos@daos.groups.io [mailto:daos@daos.groups.io] On Behalf Of Shengyu SY19 Zhang

    Sent: Saturday, December 28, 2019 2:46 AM

    To: daos@daos.groups.io

    Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

    

    Hello Kevan,

    

    Yes, its exact the same problem that I meet, the rdma_get_cm_event function will issue write system call to receive completing event from from ib device, however it get nothing at this time, and always return EAGAIN, that caused fi_tsend function infinitely loop.

    

    Regards,

    Shengyu.

    

    -----Original Message-----

    From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Kevan Rehm

    Sent: Saturday, December 28, 2019 12:30 AM

    To: daos@daos.groups.io

    Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

    

    Shengyu,

    

    I think we are going to need help from the experts, I am not familiar with this code.    I tried the same commands that you mentioned in your last email, and they also hang for me.   But I do not see an infinite loop; rather the daos process just hangs forever in a write() request.   Is that what you see as well?

    

    Experts: is there any documentation on CaRT, what it is for, internals?    I have not been able to find anything.

    

    The last entries in the daos_server_srv1.log file are just the ds_mgmt_drpc_get_attach_info() log messages, called from daos_agent.

    

    While the daos command was hung, I sent a kill -6 signal to it to collect the corefile.   It seems like the command has attempted to set up a MSG connection to the daos_io_server, but has not received a completion event.  The dest_addr=0 looks a little suspicious in the fi_tsend() call.  Hopefully others will recognize what the problem is in the backtrace below, otherwise I will keep digging as time permits.

    

    Thanks, Kevan

    

    (gdb) bt

    #0  0x00007fa7fd5749cd in write () from /lib64/libc.so.6

    #1  0x00007fa7fac2d875 in rdma_get_cm_event.part.13 () from /lib64/librdmacm.so.1

    #2  0x00007fa7fb7fd856 in fi_ibv_eq_read () from /home/users/daos/daos/install/lib/libfabric.so.1

    #3  0x00007fa7fb82360f in rxm_eq_read () from /home/users/daos/daos/install/lib/libfabric.so.1

    #4  0x00007fa7fb8252af in rxm_msg_eq_progress () from /home/users/daos/daos/install/lib/libfabric.so.1

    #5  0x00007fa7fb82542d in rxm_cmap_connect () from /home/users/daos/daos/install/lib/libfabric.so.1

    #6  0x00007fa7fb82b5eb in rxm_ep_tsend () from /home/users/daos/daos/install/lib/libfabric.so.1

    #7  0x00007fa7fc59f3e8 in fi_tsend (context=0x1814558, tag=1, dest_addr=0, desc=0x1811ab0, len=332, buf=0x7fa7f6a10038,

        ep=0x17f03c0) at /home/users/daos/daos/install/include/rdma/fi_tagged.h:114

    #8  na_ofi_msg_send_unexpected (na_class=0x17d6250, context=0x180f760, callback=<optimized out>, arg=<optimized out>,

        buf=0x7fa7f6a10038, buf_size=332, plugin_data=0x1811ab0, dest_addr=0x18ba5e0, dest_id=0 '\000', tag=1, op_id=0x1811a48)

        at /home/users/daos/daos/_build.external/mercury/src/na/na_ofi.c:3745

    #9  0x00007fa7fc7b79ff in NA_Msg_send_unexpected (op_id=0x1811a48, tag=<optimized out>, dest_id=<optimized out>,

        dest_addr=<optimized out>, plugin_data=<optimized out>, buf_size=<optimized out>, buf=<optimized out>, arg=0x1811920,

        callback=0x7fa7fc7b9a60 <hg_core_send_input_cb>, context=<optimized out>, na_class=<optimized out>)

        at /home/users/daos/daos/_build.external/mercury/src/na/na.h:1506

    #10 hg_core_forward_na (hg_core_handle=0x1811920) at /home/users/daos/daos/_build.external/mercury/src/mercury_core.c:2076

    #11 0x00007fa7fc7bb5e6 in HG_Core_forward (handle=0x1811920, callback=callback@entry=0x7fa7fc7b0890 <hg_core_forward_cb>,

        arg=arg@entry=0x1814730, flags=<optimized out>, payload_size=<optimized out>)

        at /home/users/daos/daos/_build.external/mercury/src/mercury_core.c:4775

    #12 0x00007fa7fc7b41f7 in HG_Forward (handle=0x1814730, callback=callback@entry=0x7fa7fd8b2980 <crt_hg_req_send_cb>,

        arg=arg@entry=0x18b9190, in_struct=in_struct@entry=0x18b91b0)

        at /home/users/daos/daos/_build.external/mercury/src/mercury.c:2165

    #13 0x00007fa7fd8b9e39 in crt_hg_req_send (rpc_priv=rpc_priv@entry=0x18b9190) at src/cart/crt_hg.c:1191

    #14 0x00007fa7fd90a8ea in crt_req_send_immediately (rpc_priv=<optimized out>) at src/cart/crt_rpc.c:1104

    #15 crt_req_send_internal (rpc_priv=rpc_priv@entry=0x18b9190) at src/cart/crt_rpc.c:1173

    #16 0x00007fa7fd90ef23 in crt_req_hg_addr_lookup_cb (hg_addr=0x18ba590, arg=0x18b9190) at src/cart/crt_rpc.c:569

    #17 0x00007fa7fd8b1062 in crt_hg_addr_lookup_cb (hg_cbinfo=<optimized out>) at src/cart/crt_hg.c:290

    #18 0x00007fa7fc7b0985 in hg_core_addr_lookup_cb (callback_info=<optimized out>)

        at /home/users/daos/daos/_build.external/mercury/src/mercury.c:454

    #19 0x00007fa7fc7bbce2 in hg_core_trigger_lookup_entry (hg_core_op_id=0x18ba530)

        at /home/users/daos/daos/_build.external/mercury/src/mercury_core.c:3444

    #20 hg_core_trigger (context=0x180d590, timeout=<optimized out>, timeout@entry=0, max_count=max_count@entry=4294967295,

        actual_count=actual_count@entry=0x7ffe185f88dc) at /home/users/daos/daos/_build.external/mercury/src/mercury_core.c:3384

    #21 0x00007fa7fc7bca4b in HG_Core_trigger (context=<optimized out>, timeout=timeout@entry=0, max_count=max_count@entry=4294967295,

        actual_count=actual_count@entry=0x7ffe185f88dc) at /home/users/daos/daos/_build.external/mercury/src/mercury_core.c:4900

    #22 0x00007fa7fc7b44ed in HG_Trigger (context=context@entry=0x17d6370, timeout=timeout@entry=0,

        max_count=max_count@entry=4294967295, actual_count=actual_count@entry=0x7ffe185f88dc)

        at /home/users/daos/daos/_build.external/mercury/src/mercury.c:2262

    #23 0x00007fa7fd8b37ea in crt_hg_trigger (hg_ctx=hg_ctx@entry=0x18083d8) at src/cart/crt_hg.c:1327

    #24 0x00007fa7fd8bce5d in crt_hg_progress (hg_ctx=hg_ctx@entry=0x18083d8, timeout=timeout@entry=1000) at src/cart/crt_hg.c:1360

    #25 0x00007fa7fd86dfbb in crt_progress (crt_ctx=0x18083c0, timeout=timeout@entry=-1,

        cond_cb=cond_cb@entry=0x7fa7fe61f7f0 <ev_progress_cb>, arg=arg@entry=0x7ffe185f89d0) at src/cart/crt_context.c:1286

    #26 0x00007fa7fe624bb6 in daos_event_priv_wait () at src/client/api/event.c:1203

    #27 0x00007fa7fe627f96 in dc_task_schedule (task=0x181b4e0, instant=instant@entry=true) at src/client/api/task.c:139

    #28 0x00007fa7fe626eb1 in daos_pool_connect (uuid=uuid@entry=0x7ffe185f8c38 "UXh}E<Kĭ<\035\340\332O", <incomplete sequence \325>,

        grp=0x1805f50 "daos_server", svc=0x1806000, flags=flags@entry=1, poh=poh@entry=0x7ffe185f8c48, info=info@entry=0x0,

        ev=ev@entry=0x0) at src/client/api/pool.c:53

    #29 0x000000000040590d in pool_query_hdlr (ap=0x7ffe185f8c20) at src/utils/daos_hdlr.c:141

    #30 0x0000000000402bc4 in main (argc=5, argv=<optimized out>) at src/utils/daos.c:957

    (gdb)

    

    On 12/25/19, 2:11 AM, "daos@daos.groups.io on behalf of Shengyu SY19 Zhang" <daos@daos.groups.io on behalf of zhangsy19@...> wrote:

    

        Hello Kevan,

        

        Here are the log files.

        Restart server means daos_server restarted, in spite of Ctrl+C to kill process or server reboot, anyhow after restart daos_server, the existing containers can't be touched, it can be 100% reproduced in my environment.

        

        dmg storage query smd -I    ->work.

        daos container query --svc=0 --path=/tmp/mycontainer         ->no response due to infinitely loop

        daos container create... ->->no response due to infinitely loop

        

        with sockets mode, didn't meet this issue.

        

        Regards,

        Shengyu.

        

        -----Original Message-----

        From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Kevan Rehm

        Sent: Wednesday, December 25, 2019 3:30 AM

        To: daos@daos.groups.io

        Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

        

        Shengyu,

        

        When you say "restart server", do you mean that you rebooted the node, or that you just restarted the daos_server process?   Could you send another daos_control.log and daos_server.log from when it fails in this way?

        

        Kevan

        

        On 12/23/19, 11:34 PM, "daos@daos.groups.io on behalf of Shengyu SY19 Zhang" <daos@daos.groups.io on behalf of zhangsy19@...> wrote:

        

            Hello Kevan,

            

            After some more testing, actually the issue is still there, I can get ib to work by following ways:

            Restart subnet

            rm -rf /mnt/daos

            start daos server

            re-format

            create pool

            create container

            

            however once I restart server, will happen the infinitely loop problem, and by any way I can't connect to an existing pool via ib.

            

            Regards,

            Shengyu.

            

            -----Original Message-----

            From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Kevan Rehm

            Sent: Saturday, December 21, 2019 6:57 AM

            To: daos@daos.groups.io

            Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

            

            Shengyu,

            

            As it happens, I also had a case today using infiniband where my daos_test client was in an infinite loop, it generated 200 million lines in daos.log within a minute or so.   It turned out that the IB subnet manager process had died.   I restarted opensm, then re-ran daos_test, and it started to work.    I mention it in case it might be the same problem as yours.   Are you sure your subnet manager is working?    Try a fi_pingpong test; if it works, then your subnet manager is okay, that's not the problem.

            

            Thanks, Kevan

            

            On 12/19/19, 12:48 AM, "daos@daos.groups.io on behalf of Shengyu SY19 Zhang" <daos@daos.groups.io on behalf of zhangsy19@...> wrote:

            

                Hello Joel,

                

                Thanks for your information, those are outputs of the test with and without rxm specified.

                Furthermore, I corrected numa setting, I thought there are mismatch issue with ib devices, I have tried unbind all other devices (ib1,2,3), and also tried remove rxm,

                All same problem happen, daos process just hangs to fi_send due to infinite loop.

                

                After I set FI_LOG_LEVEL=debug, and creating container, I found some interesting, it shows:

                

                libfabric:54383:verbs:core:ofi_check_ep_type():629<info> unsupported endpoint type

                libfabric:54383:verbs:core:ofi_check_ep_type():630<info> Supported: FI_EP_DGRAM

                libfabric:54383:verbs:core:ofi_check_ep_type():630<info> Requested: FI_EP_MSG

                libfabric:54383:verbs:core:ofi_check_domain_attr():525<info> Unknown domain name

                libfabric:54383:verbs:core:ofi_check_domain_attr():526<info> Supported: mlx5_0-xrc

                libfabric:54383:verbs:core:ofi_check_domain_attr():526<info> Requested: mlx5_0

                libfabric:54383:verbs:core:ofi_check_ep_type():629<info> unsupported endpoint type

                libfabric:54383:verbs:core:ofi_check_ep_type():630<info> Supported: FI_EP_DGRAM

                libfabric:54383:verbs:core:ofi_check_ep_type():630<info> Requested: FI_EP_MSG

                

                

                Then I changed environment domain name to mlx5_0-xrc, there will be another message, can't find ofi+verbs provider, just like before. And BTW I think daos app should be failed when connecting pool rather than infinitely loop in fi_send.

                

                Best Regards,

                Shengyu.

                

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Rosenzweig, Joel B

                Sent: Thursday, December 19, 2019 12:15 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hi Shengyu,

                

                Based on your logs, the control plane is successfully matching ib0 with mlx5_0.  It shows "DEBUG 02:06:24.772249 netdetect.go:536: Device alias for ib0 is mlx5_0"  As such, it correctly sets OFI_DOMAIN=mlx5_0.  This matches with your topology data as reported by lstopo.  Your results should not change if you manually are setting OFI_DOMAIN=mlx5_0 or not, because the control plane is already doing the right thing and you are not giving it a conflicting override.  If you find that the behavior changes when you specify OFI_DOMAIN=mlx5_0 in your daos_server.yml, that's a problem we would need to debug.

                

                Your topology shows these 4 interface cards / device combinations (mlx5_0:ib0, mlx5_1:ib1, mlx5_2:ib2 and mlx5_3:ib3).

                

                        PCI 15b3:1013 (P#548864 busid=0000:86:00.0 class=0207(IB) link=15.75GB/s PCISlot=5)

                          Network L#7 (Address=20:00:10:8b:fe:80:00:00:00:00:00:00:b8:59:9f:03:00:05:b5:48 Port=1) "ib0"

                          OpenFabrics L#8 (NodeGUID=b859:9f03:0005:b548 SysImageGUID=b859:9f03:0005:b548 Port1State=4 Port1LID=0x4 Port1LMC=0 Port1GID0=fe80:0000:0000:0000:b859:9f03:0005

                :b548) "mlx5_0"

                

                        PCI 15b3:1013 (P#548865 busid=0000:86:00.1 class=0207(IB) link=15.75GB/s PCISlot=5)

                          Network L#9 (Address=20:00:18:8b:fe:80:00:00:00:00:00:00:b8:59:9f:03:00:05:b5:49 Port=1) "ib1"

                          OpenFabrics L#10 (NodeGUID=b859:9f03:0005:b549 SysImageGUID=b859:9f03:0005:b548 Port1State=1 Port1LID=0xffff Port1LMC=0 Port1GID0=fe80:0000:0000:0000:b859:9f03:

                0005:b549) "mlx5_1"

                

                        PCI 15b3:1013 (P#716800 busid=0000:af:00.0 class=0207(IB) link=15.75GB/s PCISlot=6)

                          Network L#11 (Address=20:00:10:8b:fe:80:00:00:00:00:00:00:b8:59:9f:03:00:05:b5:40 Port=1) "ib2"

                          OpenFabrics L#12 (NodeGUID=b859:9f03:0005:b540 SysImageGUID=b859:9f03:0005:b540 Port1State=4 Port1LID=0x1b Port1LMC=0 Port1GID0=fe80:0000:0000:0000:b859:9f03:00

                05:b540) "mlx5_2"

                

                        PCI 15b3:1013 (P#716801 busid=0000:af:00.1 class=0207(IB) link=15.75GB/s PCISlot=6)

                          Network L#13 (Address=20:00:18:8b:fe:80:00:00:00:00:00:00:b8:59:9f:03:00:05:b5:41 Port=1) "ib3"

                          OpenFabrics L#14 (NodeGUID=b859:9f03:0005:b541 SysImageGUID=b859:9f03:0005:b540 Port1State=1 Port1LID=0xffff Port1LMC=0 Port1GID0=fe80:0000:0000:0000:b859:9f03:

                0005:b541) "mlx5_3"

                

                I agree that fi_info shows that the provider: ofi+verbs;ofi_rxm is valid for ib0, and the control plane agrees with that.  "DEBUG 02:06:20.594910 netdetect.go:764: Device ib0 supports provider: ofi+verbs;ofi_rxm"  This is only a guess, but I have to rule it out.  I am not certain that the provider ofi_rxm is being handled properly.  Can you remove "ofi_rxm" from your provider configuration and try again?

                

                That is, in your daos_server.yml set:

                              provider: ofi+verbs

                

                If you still have an error, then you will want to run the cart diagnostics again that Alex wrote about so we can see the latest results with that.

                

                >> orterun --allow-run-as-root -np 2 -x

                >> CRT_PHY_ADDR_STR="ofi+verbs;ofi_rxm" -x OFI_INTERFACE=ib0 -x

                >> OFI_DOMAIN=mlx5_0 ../bin/crt_launch -e tests/iv_server -v 3

                

                and

                

                >> orterun --allow-run-as-root -np 2 -x CRT_PHY_ADDR_STR="ofi+verbs" -x

                >> OFI_INTERFACE=ib0 -x OFI_DOMAIN=mlx5_0 ../bin/crt_launch -e

                >> tests/iv_server -v 3

                

                One last thing, while this won't generate a runtime error _yet_, you have a mismatch between the pinned_numa_node:0 and the actual NUMA node of your ib0 device.  Your topology data shows it as NUMA node 1.  If you run "daos_server network scan -a" it should show you that the correct pinned_numa_node is 1.  By setting it to the wrong NUMA node, you will have a performance impact once this is running because the daos_io_server will bind the threads to cores in the wrong NUMA node.  The plan is to make a validation error like this a hard error instead of a warning.  There's debug output in your daos_control.log that looks like this:

                

                DEBUG 02:06:20.595012 netdetect.go:901: Validate network config -- given numaNode: 0 DEBUG 02:06:20.872053 netdetect.go:894: ValidateNUMAConfig (device: ib0, NUMA: 0) returned error: The NUMA node for device ib0 does not match the provided value 0. Remove the pinned_numa_node value from daos_server.yml then execute 'daos_server network scan' to see the valid NUMA node associated with the network device

                

                I'll keep working on it with Alex.  Let's see what you find.

                

                Regards,

                Joel

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang

                Sent: Wednesday, December 18, 2019 4:25 AM

                To: daos@daos.groups.io

                Subject: FW: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hello Joel,

                

                I'm wondering if you have any updates for this? I'm also a developer and I very familiar with spdk/dpdk/ibverbs, however I'm not familiar with other projects which depended by the DAOS, if you can give me some hints or guide, I would like to try troubleshooting this issue as well.

                Also if you need environment to reproduce the issue, you may connect to our machine for debug.

                

                Regards,

                Shengyu

                

                

                -----Original Message-----

                From: Shengyu SY19 Zhang

                Sent: Thursday, December 12, 2019 4:05 PM

                To: 'daos@daos.groups.io' <daos@daos.groups.io>

                Subject: RE: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hello Joel,

                

                Yes, please see it in the attachment.

                

                Regards,

                Shengyu

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Rosenzweig, Joel B

                Sent: Thursday, December 12, 2019 11:08 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hi Shengyu,

                

                Can you generate a topology file for me, similar to what I asked Kevan to provide?  You can generate it with with "lstopo --of xml > topology.xml"  I am interested in seeing if your system also has device siblings with same/different ports as the one specified as the OFI_INTERFACE.

                

                Your debug log shows that DAOS chose the sibling of ib0 as mlx5_0.  It's correct that it picked something in the mlxN_N family, but, depending on your topology there could be a better device to choose, possibly one that has a port match.  Your topology file will show if mlx5_0 matches the port or not, and similarly to Kevan's, it will help me develop a better function to find the correct matching sibling.

                

                I don't know why cart failed once it had the OFI_DEVICE that we thought was correct.  Alex's experiment with Kevan will help with that.

                

                Regards,

                Joel

                

                

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang

                Sent: Wednesday, December 11, 2019 2:09 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hello Joel,

                

                Please see those files in attachment, after changed some configurations, daos trapped in (no return from rdma_get_cm_event ) creating container, here is bt log:

                

                (gdb) bt

                #0  0x00007fb8d6ef59cd in write () from /lib64/libc.so.6

                #1  0x00007fb8d45a99ad in rdma_get_cm_event.part.18 () from /lib64/librdmacm.so.1

                #2  0x00007fb8d517f836 in fi_ibv_eq_read () from /root/daos/install/lib/libfabric.so.1

                #3  0x00007fb8d51a556f in rxm_eq_read () from /root/daos/install/lib/libfabric.so.1

                #4  0x00007fb8d51a720f in rxm_msg_eq_progress () from /root/daos/install/lib/libfabric.so.1

                #5  0x00007fb8d51a738d in rxm_cmap_connect () from /root/daos/install/lib/libfabric.so.1

                #6  0x00007fb8d51ad54b in rxm_ep_tsend () from /root/daos/install/lib/libfabric.so.1

                #7  0x00007fb8d5f1e468 in fi_tsend (context=0xbc4018, tag=1, dest_addr=0, desc=0xbaf6f0, len=384, buf=0x7fb8d235c038, ep=0xbb03a0) at /root/daos/install/include/rdma/fi_tagged.h:114

                #8  na_ofi_msg_send_unexpected (na_class=0xacf220, context=0xbbf200, callback=<optimized out>, arg=<optimized out>, buf=0x7fb8d235c038, buf_size=384, plugin_data=0xbaf6f0,

                    dest_addr=0xc6a1b0, dest_id=0 '\000', tag=1, op_id=0xbc14e8) at /root/daos/_build.external/mercury/src/na/na_ofi.c:3622

                #9  0x00007fb8d613885f in NA_Msg_send_unexpected (op_id=0xbc14e8, tag=<optimized out>, dest_id=<optimized out>, dest_addr=<optimized out>, plugin_data=<optimized out>,

                    buf_size=<optimized out>, buf=<optimized out>, arg=0xbc13c0, callback=0x7fb8d613a8c0 <hg_core_send_input_cb>, context=<optimized out>, na_class=<optimized out>)

                    at /root/daos/_build.external/mercury/src/na/na.h:1485

                #10 hg_core_forward_na (hg_core_handle=0xbc13c0) at /root/daos/_build.external/mercury/src/mercury_core.c:2076

                #11 0x00007fb8d613c3a6 in HG_Core_forward (handle=0xbc13c0, callback=callback@entry=0x7fb8d6131740 <hg_core_forward_cb>, arg=arg@entry=0xbc41f0, flags=<optimized out>,

                    payload_size=<optimized out>) at /root/daos/_build.external/mercury/src/mercury_core.c:4748

                #12 0x00007fb8d6135057 in HG_Forward (handle=0xbc41f0, callback=callback@entry=0x7fb8d7233930 <crt_hg_req_send_cb>, arg=arg@entry=0xc68cb0, in_struct=in_struct@entry=0xc68cd0)

                    at /root/daos/_build.external/mercury/src/mercury.c:2147

                #13 0x00007fb8d723ade9 in crt_hg_req_send (rpc_priv=rpc_priv@entry=0xc68cb0) at src/cart/crt_hg.c:1190

                #14 0x00007fb8d728b89a in crt_req_send_immediately (rpc_priv=<optimized out>) at src/cart/crt_rpc.c:1104

                #15 crt_req_send_internal (rpc_priv=rpc_priv@entry=0xc68cb0) at src/cart/crt_rpc.c:1173

                #16 0x00007fb8d728fed3 in crt_req_hg_addr_lookup_cb (hg_addr=0xc6a150, arg=0xc68cb0) at src/cart/crt_rpc.c:569

                #17 0x00007fb8d7232012 in crt_hg_addr_lookup_cb (hg_cbinfo=<optimized out>) at src/cart/crt_hg.c:290

                #18 0x00007fb8d6131835 in hg_core_addr_lookup_cb (callback_info=<optimized out>) at /root/daos/_build.external/mercury/src/mercury.c:454

                #19 0x00007fb8d613caa2 in hg_core_trigger_lookup_entry (hg_core_op_id=0xc6a0f0) at /root/daos/_build.external/mercury/src/mercury_core.c:3444

                #20 hg_core_trigger (context=0xbbd030, timeout=<optimized out>, timeout@entry=0, max_count=max_count@entry=4294967295, actual_count=actual_count@entry=0x7fff3d70388c)

                    at /root/daos/_build.external/mercury/src/mercury_core.c:3384

                #21 0x00007fb8d613d80b in HG_Core_trigger (context=<optimized out>, timeout=timeout@entry=0, max_count=max_count@entry=4294967295, actual_count=actual_count@entry=0x7fff3d70388c)

                    at /root/daos/_build.external/mercury/src/mercury_core.c:4873

                #22 0x00007fb8d613534d in HG_Trigger (context=context@entry=0xacf1e0, timeout=timeout@entry=0, max_count=max_count@entry=4294967295, actual_count=actual_count@entry=0x7fff3d70388c)

                    at /root/daos/_build.external/mercury/src/mercury.c:2244

                #23 0x00007fb8d723479a in crt_hg_trigger (hg_ctx=hg_ctx@entry=0xbb7e78) at src/cart/crt_hg.c:1326

                #24 0x00007fb8d723de0d in crt_hg_progress (hg_ctx=hg_ctx@entry=0xbb7e78, timeout=timeout@entry=1000) at src/cart/crt_hg.c:1359

                #25 0x00007fb8d71eef6b in crt_progress (crt_ctx=0xbb7e60, timeout=timeout@entry=-1, cond_cb=cond_cb@entry=0x7fb8d7fa0230 <ev_progress_cb>, arg=arg@entry=0x7fff3d703980)

                    at src/cart/crt_context.c:1286

                #26 0x00007fb8d7fa55f6 in daos_event_priv_wait () at src/client/api/event.c:1203

                #27 0x00007fb8d7fa89d6 in dc_task_schedule (task=0xbcafa0, instant=instant@entry=true) at src/client/api/task.c:139

                #28 0x00007fb8d7fa78f1 in daos_pool_connect (uuid=uuid@entry=0x7fff3d703b68 "\265\215%f\036AN\354\203\002\317I\035\362\067\273", grp=0xbb59f0 "daos_server", svc=0xbb5aa0,

                    flags=flags@entry=2, poh=poh@entry=0x7fff3d703b78, info=info@entry=0x0, ev=ev@entry=0x0) at src/client/api/pool.c:53

                #29 0x0000000000404eb0 in cont_op_hdlr (ap=ap@entry=0x7fff3d703b50) at src/utils/daos.c:610

                #30 0x0000000000402b94 in main (argc=8, argv=<optimized out>) at src/utils/daos.c:957

                

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Rosenzweig, Joel B

                Sent: Tuesday, December 10, 2019 8:48 PM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hi Shengyu,

                

                It's hard to further diagnose without the logs. Can you share your latest daos_server.yml, full daos_control.log and full server.log? In the daos_server.yml, please set control_log_mask: DEBUG and in the io server section, set log_mask: DEBUG

                

                Regards,

                Joel

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang

                Sent: Tuesday, December 10, 2019 1:42 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hello Joel,

                

                I didn't set OFI_DOMAIN=mlx5_0 before, from the hint of Alex, I set it yesterday, there then there is another error while creating container:

                mgmt ERR  src/mgmt/cli_mgmt.c:325 get_attach_info() GetAttachInfo unsuccessful: 2

                

                

                Regards,

                Shengyu

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Rosenzweig, Joel B

                Sent: Monday, December 9, 2019 11:47 PM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hi Shengyu,

                

                With your latest setup, can you launch DAOS and send your latest daos_control.log?  In particular, I want to see how the daos_io_server environment variables are set.  For example these two lines below show the command line args and the environment variables in use with an ofi+sockets/ib1 config.  I'm looking to see if we are setting OFI_DOMAIN=mlx5_0 in your environment.  I seem to recall that your earlier logs did have this set, but since builds have changed since then, it's worth checking out one more time.

                

                DEBUG 16:33:15.711949 exec.go:112: daos_io_server:1 args: [-t 8 -x 2 -g daos_server -d /tmp/daos_sockets -s /mnt/daos1 -i 1842327892 -p 1 -I 1] DEBUG 16:33:15.711963 exec.go:113: daos_io_server:1 env: [CRT_TIMEOUT=30 FI_SOCKETS_CONN_TIMEOUT=2000 D_LOG_FILE=/tmp/server.log CRT_CTX_SHARE_ADDR=0 FI_SOCKETS_MAX_CONN_RETRY=1 D_LOG_MASK=ERR CRT_PHY_ADDR_STR=ofi+sockets OFI_INTERFACE=ib1 OFI_PORT=31416 DAOS_MD_CAP=1024]

                

                If we are not setting OFI_DOMAIN=mlx5_0 automatically, go ahead and edit your daos_server.yml and add OFI_DOMAIN=mlx5_0 to the env_vars section (per below) and see what you get.  If that doesn't get you going, please send the log for that run, too.

                

                env_vars:

                  - OFI_DOMAIN=mlx5_0

                

                Regards,

                Joel

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Oganezov, Alexander A

                Sent: Monday, December 9, 2019 5:03 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hi Shengyu,

                

                Actually this is really good result - two servers were able to exchange rpcs. That's the end of test -- just ctrl+C out of it.

                

                I'll let Joel take over for daos help on how to make daos use mlx5_0 domain.

                

                Thanks,

                ~~Alex.

                

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang

                Sent: Monday, December 09, 2019 12:57 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hello Alex,

                

                Please see the new log below, and the test seems dead (no more response):

                

                ibv_exp_query_device: invalid comp_mask !!! (comp_mask = 0x7fdd21092400 valid_mask = 0x3) [afa1][[44100,1],1][btl_openib_component.c:1670:init_one_device] error obtaining device attributes for mlx5_0 errno says Invalid argument

                ibv_exp_query_device: invalid comp_mask !!! (comp_mask = 0x195aad0 valid_mask = 0x3) [afa1][[44100,1],0][btl_openib_component.c:1670:init_one_device] error obtaining device attributes for mlx5_0 errno says Invalid argument

                --------------------------------------------------------------------------

                WARNING: There was an error initializing an OpenFabrics device.

                

                  Local host:   afa1

                  Local device: mlx5_0

                --------------------------------------------------------------------------

                12/09-03:53:20.32 afa1 CaRT[101464/101464] crt  WARN src/cart/crt_init.c:270 crt_init_opt() PMIX disabled. Disabling LM automatically

                12/09-03:53:20.32 afa1 CaRT[101465/101465] crt  WARN src/cart/crt_init.c:270 crt_init_opt() PMIX disabled. Disabling LM automatically SRV [rank=1 pid=101465] Server starting, self_rank=1 SRV [rank=0 pid=101464] Server starting, self_rank=0 SRV [rank=1 pid=101465] >>>> Entered iv_set_ivns SRV [rank=1 pid=101465] <<<< Exited iv_set_ivns:773

                

                [afa1:101458] 1 more process has sent help message help-mpi-btl-openib.txt / error in device init [afa1:101458] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

                

                Regards,

                Shengyu

                

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Oganezov, Alexander A

                Sent: Monday, December 9, 2019 4:30 PM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hi Shengyu,

                

                Given your output can you try this now?

                

                orterun --allow-run-as-root -np 2 -x CRT_PHY_ADDR_STR="ofi+verbs;ofi_rxm" -x OFI_INTERFACE=ib0 -x OFI_DOMAIN=mlx5_0 ../bin/crt_launch -e tests/iv_server -v 3

                

                Please note additional OFI_DOMAIN envariable.

                

                Thanks,

                ~~Alex.

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang

                Sent: Monday, December 09, 2019 12:25 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hello Alex,

                

                Here is outputs of fio_info related to verbs:

                

                provider: verbs

                    fabric: IB-0xfe80000000000000

                    domain: mlx5_2

                    version: 1.0

                    type: FI_EP_MSG

                    protocol: FI_PROTO_RDMA_CM_IB_RC

                provider: verbs

                    fabric: IB-0xfe80000000000000

                    domain: mlx5_2-xrc

                    version: 1.0

                    type: FI_EP_MSG

                    protocol: FI_PROTO_RDMA_CM_IB_XRC

                provider: verbs

                    fabric: IB-0xfe80000000000000

                    domain: mlx5_0

                    version: 1.0

                    type: FI_EP_MSG

                    protocol: FI_PROTO_RDMA_CM_IB_RC

                provider: verbs

                    fabric: IB-0xfe80000000000000

                    domain: mlx5_0-xrc

                    version: 1.0

                    type: FI_EP_MSG

                    protocol: FI_PROTO_RDMA_CM_IB_XRC

                provider: verbs

                    fabric: IB-0xfe80000000000000

                    domain: mlx5_0-dgram

                    version: 1.0

                    type: FI_EP_DGRAM

                    protocol: FI_PROTO_IB_UD

                provider: verbs

                    fabric: IB-0xfe80000000000000

                    domain: mlx5_2-dgram

                    version: 1.0

                    type: FI_EP_DGRAM

                    protocol: FI_PROTO_IB_UD

                provider: verbs;ofi_rxm

                    fabric: IB-0xfe80000000000000

                    domain: mlx5_2

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXM

                provider: verbs;ofi_rxm

                    fabric: IB-0xfe80000000000000

                    domain: mlx5_0

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXM

                provider: tcp;ofi_rxm

                    fabric: TCP-IP

                    domain: tcp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXM

                provider: tcp;ofi_rxm

                    fabric: TCP-IP

                    domain: tcp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXM

                provider: tcp;ofi_rxm

                    fabric: TCP-IP

                    domain: tcp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXM

                provider: tcp;ofi_rxm

                    fabric: TCP-IP

                    domain: tcp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXM

                provider: tcp;ofi_rxm

                    fabric: TCP-IP

                    domain: tcp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXM

                provider: tcp;ofi_rxm

                    fabric: TCP-IP

                    domain: tcp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXM

                provider: tcp;ofi_rxm

                    fabric: TCP-IP

                    domain: tcp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXM

                provider: tcp;ofi_rxm

                    fabric: TCP-IP

                    domain: tcp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXM

                provider: tcp;ofi_rxm

                    fabric: TCP-IP

                    domain: tcp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXM

                provider: tcp;ofi_rxm

                    fabric: TCP-IP

                    domain: tcp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXM

                provider: verbs;ofi_rxd

                    fabric: IB-0xfe80000000000000

                    domain: mlx5_2-dgram

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXD

                provider: verbs;ofi_rxd

                    fabric: IB-0xfe80000000000000

                    domain: mlx5_0-dgram

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXD

                provider: UDP;ofi_rxd

                    fabric: UDP-IP

                    domain: udp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXD

                provider: UDP;ofi_rxd

                    fabric: UDP-IP

                    domain: udp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXD

                provider: UDP;ofi_rxd

                    fabric: UDP-IP

                    domain: udp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXD

                provider: UDP;ofi_rxd

                    fabric: UDP-IP

                    domain: udp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXD

                provider: UDP;ofi_rxd

                    fabric: UDP-IP

                    domain: udp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXD

                provider: UDP;ofi_rxd

                    fabric: UDP-IP

                    domain: udp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXD

                provider: UDP;ofi_rxd

                    fabric: UDP-IP

                    domain: udp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXD

                provider: UDP;ofi_rxd

                    fabric: UDP-IP

                    domain: udp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXD

                provider: UDP;ofi_rxd

                    fabric: UDP-IP

                    domain: udp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXD

                provider: UDP;ofi_rxd

                    fabric: UDP-IP

                    domain: udp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXD

                

                Regards,

                Shengyu

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Oganezov, Alexander A

                Sent: Monday, December 9, 2019 4:08 PM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Thanks,

                Can you also provide full fi_info output?

                

                ~~Alex.

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang

                Sent: Monday, December 09, 2019 12:05 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hello Alex,

                

                Here are the outputs:

                

                orterun --allow-run-as-root -np 2 -x CRT_PHY_ADDR_STR="ofi+verbs;ofi_rxm" -x OFI_INTERFACE=ib0 ../bin/crt_launch -e tests/iv_server -v 3

                ibv_exp_query_device: invalid comp_mask !!! (comp_mask = 0x2094f90 valid_mask = 0x3) [afa1][[18544,1],1][btl_openib_component.c:1670:init_one_device] error obtaining device attributes for mlx5_0 errno says Invalid argument

                ibv_exp_query_device: invalid comp_mask !!! (comp_mask = 0xacda60 valid_mask = 0x3) [afa1][[18544,1],0][btl_openib_component.c:1670:init_one_device] error obtaining device attributes for mlx5_0 errno says Invalid argument

                --------------------------------------------------------------------------

                WARNING: There was an error initializing an OpenFabrics device.

                

                  Local host:   afa1

                  Local device: mlx5_0

                --------------------------------------------------------------------------

                12/09-03:01:03.53 afa1 CaRT[92269/92269] hg   ERR  # NA -- Error -- /root/daos/_build.external/mercury/src/na/na_ofi.c:1609

                 # na_ofi_domain_open(): No provider found for "verbs;ofi_rxm" provider on domain "ib0"

                12/09-03:01:03.53 afa1 CaRT[92268/92268] hg   ERR  # NA -- Error -- /root/daos/_build.external/mercury/src/na/na_ofi.c:1609

                 # na_ofi_domain_open(): No provider found for "verbs;ofi_rxm" provider on domain "ib0"

                12/09-03:01:03.53 afa1 CaRT[92269/92269] hg   ERR  # NA -- Error -- /root/daos/_build.external/mercury/src/na/na_ofi.c:2975

                 # na_ofi_initialize(): Could not open domain for verbs;ofi_rxm, ib0

                12/09-03:01:03.53 afa1 CaRT[92269/92269] hg   ERR  # NA -- Error -- /root/daos/_build.external/mercury/src/na/na.c:324

                 # NA_Initialize_opt(): Could not initialize plugin

                12/09-03:01:03.53 afa1 CaRT[92269/92269] hg   ERR  src/cart/crt_hg.c:525 crt_hg_init() Could not initialize NA class.

                12/09-03:01:03.53 afa1 CaRT[92269/92269] crt  ERR  src/cart/crt_init.c:347 crt_init_opt() crt_hg_init failed rc: -1020.

                12/09-03:01:03.53 afa1 CaRT[92269/92269] crt  ERR  src/cart/crt_init.c:421 crt_init_opt() crt_init failed, rc: -1020.

                12/09-03:01:03.53 afa1 CaRT[92268/92268] hg   ERR  # NA -- Error -- /root/daos/_build.external/mercury/src/na/na_ofi.c:2975

                 # na_ofi_initialize(): Could not open domain for verbs;ofi_rxm, ib0

                12/09-03:01:03.53 afa1 CaRT[92268/92268] hg   ERR  # NA -- Error -- /root/daos/_build.external/mercury/src/na/na.c:324

                 # NA_Initialize_opt(): Could not initialize plugin

                12/09-03:01:03.53 afa1 CaRT[92268/92268] hg   ERR  src/cart/crt_hg.c:525 crt_hg_init() Could not initialize NA class.

                12/09-03:01:03.53 afa1 CaRT[92268/92268] crt  ERR  src/cart/crt_init.c:347 crt_init_opt() crt_hg_init failed rc: -1020.

                12/09-03:01:03.53 afa1 CaRT[92268/92268] crt  ERR  src/cart/crt_init.c:421 crt_init_opt() crt_init failed, rc: -1020.

                [afa1:92262] 1 more process has sent help message help-mpi-btl-openib.txt / error in device init [afa1:92262] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

                

                

                Regards,

                Shengyu

                

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Oganezov, Alexander A

                Sent: Saturday, December 7, 2019 1:14 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hi Shengyu,

                

                With latest daos code and mofed 4.6 installed can you rerun this and show what that one gives you?

                

                source scons_local/utils/setup_local.sh

                cd install/Linux/TESTING

                orterun -np 2 -x CRT_PHY_ADDR_STR="ofi+verbs;ofi_rxm" -x OFI_INTERFACE=ib0 ../bin/crt_launch -e tests/iv_server -v 3

                

                Thanks,

                ~~Alex.

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang

                Sent: Thursday, December 05, 2019 10:25 PM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hello Alex,

                

                Yes, however with 4.6 the result is same.

                After I upgraded daos code to newest of master branch, I got some different results, daos io server seems started OK, since I can see lots of fd points to rdma_cm.

                But daos client seems can't connect to server due to same error (can't find efi+verbs provider on ib0) like the log shows, you may find the log in the attachment, that is crated via "create container"

                

                Regards,

                Shengyu.

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Oganezov, Alexander A

                Sent: Wednesday, December 4, 2019 4:02 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hi Shengyu,

                

                Can you try installing MOFED 4.6 packages on your system? In general MOFED is required to get verbs over Mellanox working.

                Those packages can be found at: https://www.mellanox.com/page/mlnx_ofed_matrix?mtag=linux_sw_drivers

                

                There is also 4.7 version available, however there seem to be few longevity issues currently when using 4.7 (according to verbs ofi maintainers).

                

                Thanks,

                ~~Alex.

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang

                Sent: Monday, November 25, 2019 9:55 PM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hello Alex,

                

                Thanks for your suggestion, here is the log:

                

                

                mca_base_component_repository_open: unable to open mca_pml_ucx: libucp.so.0: cannot open shared object file: No such file or directory (ignored)

                11/26-00:40:22.65 afa1 CaRT[365504/365504] hg   ERR  # NA -- Error -- /root/daos/_build.external/mercury/src/na/na_ofi.c:1407

                 # na_ofi_getinfo(): fi_getinfo() failed, rc: -61(No data available)

                11/26-00:40:22.65 afa1 CaRT[365504/365504] hg   ERR  # NA -- Error -- /root/daos/_build.external/mercury/src/na/na_ofi.c:2816

                 # na_ofi_check_protocol(): na_ofi_getinfo() failed

                11/26-00:40:22.65 afa1 CaRT[365504/365504] hg   ERR  # NA -- Error -- /root/daos/_build.external/mercury/src/na/na.c:302

                 # NA_Initialize_opt(): No suitable plugin found that matches ofi+verbs;ofi_rxm://192.168.80.120

                11/26-00:40:22.65 afa1 CaRT[365504/365504] hg   ERR  src/cart/crt_hg.c:521 crt_hg_init() Could not initialize NA class.

                11/26-00:40:22.65 afa1 CaRT[365504/365504] crt  ERR  src/cart/crt_init.c:347 crt_init_opt() crt_hg_init failed rc: -1020.

                11/26-00:40:22.65 afa1 CaRT[365504/365504] crt  ERR  src/cart/crt_init.c:421 crt_init_opt() crt_init failed, rc: -1020.

                

                

                Regards,

                Shengyu

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Oganezov, Alexander A

                Sent: Tuesday, November 26, 2019 12:21 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hi Shengyu,

                

                In order to figure out what is the issue on your system could you run cart standalone test instead and provide the output that you get?

                

                cd daos_dir

                source scons_local/utils/setup_local.sh

                cd install/Linux/TESTING

                orterun -np 2 -x CRT_PHY_ADDR_STR="ofi+verbs;ofi_rxm" -x OFI_INTERFACE=ib0 ../bin/crt_launch -e tests/iv_server -v 3

                

                Note: Depending on how you installed daos your paths might be different, so instead of cd install/Linux/TESTING you might have to cd into different directory first where you have tests/iv_server in. I think in your env it will be   cd /root/daos/install/TESTING/  or cd /root/daos/install/cart/TESTING.

                

                

                Expected output:

                11/25-15:51:48.39 wolf-55 CaRT[53295/53295] crt  WARN src/cart/crt_init.c:270 crt_init_opt() PMIX disabled. Disabling LM automatically

                11/25-15:51:48.40 wolf-55 CaRT[53296/53296] crt  WARN src/cart/crt_init.c:270 crt_init_opt() PMIX disabled. Disabling LM automatically SRV [rank=0 pid=53295]  Server starting, self_rank=0 SRV [rank=1 pid=53296]  Server starting, self_rank=1 SRV [rank=1 pid=53296]  >>>> Entered iv_set_ivns SRV [rank=1 pid=53296]  <<<< Exited iv_set_ivns:773

                

                Thanks,

                ~~Alex.

                

                

                -----Original Message-----

                From: daos@daos.groups.io [mailto:daos@daos.groups.io] On Behalf Of Shengyu SY19 Zhang

                Sent: Monday, November 25, 2019 3:28 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hello Joel,

                

                As the shown in the output.log, there is only one version of libfabrics installed in my machine, and actually I don't nave other software which depends libfabraics installed.

                From you guide to set FI_LOG_LEVEL=debug, I can see the following message, may be helpful:

                

                libfabric:123445:verbs:fabric:fi_ibv_set_default_attr():1263<info> Ignoring provider default value for tx rma_iov_limit as it is greater than the value supported by domain: mlx5_0 libfabric:123445:verbs:fabric:fi_ibv_get_matching_info():1365<info> hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints

                ERROR: daos_io_server:0 libfabric:123445:verbs:core:fi_ibv_check_hints():231<info> Unsupported capabilities libfabric:123445:verbs:core:fi_ibv_check_hints():232<info> Supported: FI_MSG, FI_RECV, FI_SEND, FI_LOCAL_COMM, FI_REMOTE_COMM libfabric:123445:verbs:core:fi_ibv_check_hints():232<info> Requested: FI_MSG, FI_RMA, FI_READ, FI_RECV, FI_SEND, FI_REMOTE_READ

                ERROR: daos_io_server:0 libfabric:123445:verbs:fabric:fi_ibv_get_rai_id():179<info> rdma_bind_addr: No such device(19)

                ERROR: daos_io_server:0 libfabric:123445:verbs:fabric:fi_ibv_get_rai_id():179<info> rdma_bind_addr: No such device(19)

                ERROR: daos_io_server:0 libfabric:123445:verbs:fabric:fi_ibv_get_rai_id():179<info> rdma_bind_addr: Invalid argument(22)

                ERROR: daos_io_server:0 libfabric:123445:verbs:fabric:fi_ibv_get_rai_id():179<info> rdma_bind_addr: Invalid argument(22)

                ERROR: daos_io_server:0 libfabric:123445:verbs:fabric:fi_ibv_get_rai_id():179<info> rdma_bind_addr: Invalid argument(22)

                ERROR: daos_io_server:0 libfabric:123445:core:core:ofi_layering_ok():795<info> Need core provider, skipping ofi_rxd libfabric:123445:core:core:ofi_layering_ok():795<info> Need core provider, skipping ofi_mrail

                

                

                Regards,

                Shengyu.

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Rosenzweig, Joel B

                Sent: Saturday, November 23, 2019 3:20 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hi Shengyu,

                

                The debug output showed me that when daos_server is started via orterun, libfabric is not finding provider support for ofi_rxm at least.  I'm still wondering if you have two different versions of libfabric installed on your machine.

                

                Can you run these commands and provide the output?

                

                1) ldd install/bin/daos_server

                2) modify your orterun command to run ldd on daos_server.   For example, I run this command locally:

                orterun --allow-run-as-root --map-by node --mca btl tcp,self --mca oob tcp -np 1 --hostfile /home/jbrosenz/daos/hostfile --enable-recovery --report-uri /tmp/urifile ldd /home/jbrosenz/daos/install/bin/daos_server

                3) which fi_info

                4) ldd over each version of fi_info found

                

                From the data you provide, I'll understand if the libfabric being used by daos_server when executed directly by you in the shell is the same libfabric being used by daos_server when executed via orterun.  Your original "daos_server network scan" output showed support for ofi+verbs;ofi_rxm but your debug output showed that when daos_server was started (via orterun), libfabric could not find support for the very same providers.  If there are two different versions being used with different configurations, it would explain the failure.  If it's a single installation/configuration, then that will lead the debug in another direction.

                

                Depending on what you find through 1-4, you might find it helpful to export the environment variable FI_LOG_LEVEL=debug which will instruct libfabric to output a good deal of debug info.

                

                Regards,

                Joel

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang

                Sent: Friday, November 22, 2019 12:59 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hello Joel,

                

                Please see those files in attachment.

                I have tried two machines, one have full provider shows in fi_info (verbs and rxm), another doesn't show verbs, but they are same can't start io_server. I found the project conflicts with mellanox drivers, therefor I remove it and use yum package only, however still keep not working.

                

                

                Regards,

                Shengyu

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Rosenzweig, Joel B

                Sent: Friday, November 22, 2019 6:35 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hi Shengyu,

                

                Can you share your daos_server.yml so we can see how you enabled the provider?  And, can you share the log files daos_control.log and server.log so we can see more context?

                

                Thank you,

                Joel

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang

                Sent: Wednesday, November 20, 2019 9:23 PM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hello,

                

                Thank you for your help Alex, Joel and Kevin, I have checked those steps that you provided:

                

                Ibstat:

                State: Active

                Physical state: LinkUp

                

                Ifconfig:

                flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 2044

                

                fi_info:

                verbs:

                    version: 1.0

                ofi_rxm:

                    version: 1.0

                ofi_rxd:

                    version: 1.0

                

                And network is good since I can run SPDK NVMe-oF over Infiniband with good working.

                I also specified "ofi+verbs;ofi_rxm", the same error occurred, the ioserver will be stopped after a while, and print log as I provided previously.

                

                And I noticed, whatever I specify ofi+verbs, ofi_rxm, or ofi+verbs;ofi_rxm, the log keep shows No provider found for "verbs;ofi_rxm" provider on domain "ib0", is it the cause?

                

                BTW: it is working under ofi+sockets.

                

                

                Regards,

                Shengyu.

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Oganezov, Alexander A

                Sent: Thursday, November 21, 2019 7:13 AM

                To: daos@daos.groups.io

                Subject: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hi Shengyu,

                

                > However if I specify either ofi+verbs or ofi_rxm, the same error will happen, and io_server will stop.

                > na_ofi.c:1609

                > # na_ofi_domain_open(): No provider found for "verbs;ofi_rxm" provider on domain "ib0"

                

                To use supported verbs provider you need to have "ofi+verbs;ofi_rxm" in the provider string.

                

                ~~Alex.

                

                -----Original Message-----

                From: daos@daos.groups.io [mailto:daos@daos.groups.io] On Behalf Of Rosenzweig, Joel B

                Sent: Wednesday, November 20, 2019 7:37 AM

                To: daos@daos.groups.io

                Subject: Re: [daos] Does DAOS support infiniband now?

                

                Hi Shengyu,

                

                The daos_server network scan uses information provided by libfabric to determine available devices and providers.  It then cross references that list of devices with device names obtained from hwloc to convert libfabric device names (as necessary) to those you'd find via ifconfig.  Therefore, if "daos_server network scan" displays a device and provider, it means that support for that via libfabric has been provided.   However, as Kevin pointed out, it's possible that the device itself was down, and that could certainly generate an error like what you encountered.  There's another possibility, that you might have more than one version of libfabric installed in your environment.  I have run into this situation in our lab environment.  You might check your target system to see if it has more than one libfabric library with different provider support.

                

                Regards,

                Joel

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Harms, Kevin via Groups.Io

                Sent: Wednesday, November 20, 2019 10:04 AM

                To: daos@daos.groups.io

                Subject: Re: [daos] Does DAOS support infiniband now?

                

                Shengyu,

                

                  I have tried IB and it works. Verify the libfabric verbs provider is available.

                

                  fi_info -l

                

                 you should see these:

                

                ofi\_rxm:

                version: 1.0

                

                verbs:

                version: 1.0

                

                  See here for details:

                

                

                  You might also want to confirm ib0 is in the UP state:

                

                [root@daos01 ~]# ifconfig ib0

                ib0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 4092

                        inet 172.25.6.101  netmask 255.255.0.0  broadcast 172.25.255.255

                

                kevin

                

                ________________________________________

                From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Shengyu SY19 Zhang <zhangsy19@...>

                Sent: Wednesday, November 20, 2019 2:54 AM

                To: daos@daos.groups.io

                Subject: [daos] Does DAOS support infiniband now?

                

                Hello,

                

                I use daos_server network scan, it shows as following:

                fabric_iface: ib0

                       provider: ofi+verbs;ofi_rxm

                       pinned_numa_node: 1

                

                However if I specify either ofi+verbs or ofi_rxm, the same error will happen, and io_server will stop.

                na_ofi.c:1609

                # na_ofi_domain_open(): No provider found for "verbs;ofi_rxm" provider on domain "ib0"

                

                The ib0 is Mellanox nic over Infiniband network.

                

                Regards,

                Shengyu.

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

                

            

            

            

            

            

            

            

            

        

        

        

        

        

        

        

        

    

    

    

    

    

    

    

    

    

    

    

 

 

 

 

 

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.


Re: [External] Re: [daos] Does DAOS support infiniband now?

Shengyu SY19 Zhang
 

Hello Alex,

 

I have removed mercury, the io_server seems started normally ,however the daos_server.log increasing quickly to eat all my free space. It infinite repeat the three lines:

01/21-01:40:49.28 afa1 DAOS[72964/73009] hg   ERR  src/cart/crt_hg.c:1331 crt_hg_trigger() HG_Trigger failed, hg_ret: 18.
01/21-01:40:49.28 afa1 DAOS[72964/73009] rpc  ERR  src/cart/crt_context.c:1255 crt_progress() crt_hg_progress failed, rc: -1020.
01/21-01:40:49.28 afa1 DAOS[72964/73009] server ERR  src/iosrv/srv.c:469 dss_srv_handler() failed to progress CART context: -1020

 

Regards,

Shengyu,

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Oganezov, Alexander A
Sent: Tuesday, January 21, 2020 4:49 AM
To: daos@daos.groups.io
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

One other thing Shengyu,

 

Can you verify that your mercury build is up to date? Based on domain being used as “mlx5_0/192.168.80.161” it sounds like there is mismatch between what CaRT generates and what mercury consumes; there has been change few weeks ago regarding how domain is provided to mercury level, and it feels as if older mercury is being used.

 

To ensure clean mercury build remove libmercury* libna*  from your install/lib location, remove _build.external-Linux/mercury directory and recompile daos with scons –build-deps=yes –config=force install

 

Thanks,

~~Alex.

 

From: Oganezov, Alexander A
Sent: Monday, January 20, 2020 11:56 AM
To: daos@daos.groups.io
Subject: RE: [External] Re: [daos] Does DAOS support infiniband now?

 

Hi Shengyu,

 

What does this command return to you on the node where you see this strange domain name?

fi_info --provider="verbs"

 

Thanks,

~~Alex.

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang
Sent: Monday, January 20, 2020 12:32 AM
To: daos@daos.groups.io
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Hello Joel,

 

I have some more information,

In the file na_ofi.c + 1609, the io_serve exit there, and on above code, na_ofi_verify_provider function will compare domain names, the domain is “mlx5_0/192.168.80.161”, while prov->domain_attr.name is “mlx5_0”,  it will always return FALSE.

 

Regards,

Shengyu

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang
Sent: Sunday, January 19, 2020 6:15 PM
To: daos@daos.groups.io
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Hello Joel,

 

Network scan:

Scanning fabric for YML specified provider: ofi+verbs;ofi_rxm
Fabric scan found 1 devices matching the provider spec: ofi+verbs;ofi_rxm

fabric_iface: ib0
provider: ofi+verbs;ofi_rxm
pinned_numa_node: 1

 

 

And log attached, it is standalone server.

Additionally, I created two identical virtual machines with IB sr-iov pass-through, however one vm can’t start due to the same problem, while another can start normally. They were using ofi+verbs;ofi_rxm as provider.

 

 

Regards,

Shengyu

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Rosenzweig, Joel B
Sent: Friday, January 17, 2020 11:03 PM
To: daos@daos.groups.io
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Hello Shengyu,

 

It appears that your daos_server.yml is specifying the provider as “ofi+verbs” but I think it should be set to “ofi+verbs;ofi_rxm”.  Can you configure your daos_server.yml with that and try again?  And then, if things still do not work, then please provide:

 

1)      the network scan output again after you make the provider change

2)      the portion of the debug log that shows the environment variables being provided to the daos_io_server.  This will show us what is being set for OFI_INTERFACE, OFI_DOMAIN in the daos_io_server environment.

3)      the daos_server.yml so I can see how you have configured each daos_io_server.

 

Regards,

Joel

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang
Sent: Friday, January 17, 2020 4:55 AM
To: daos@daos.groups.io
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Hello Johann,

 

Here is logs:

 

Scanning fabric for YML specified provider: ofi+verbs
Fabric scan found 4 devices matching the provider spec: ofi+verbs

fabric_iface: ib2
provider: ofi+verbs;ofi_rxm
pinned_numa_node: 1


fabric_iface: ib0
provider: ofi+verbs;ofi_rxm
pinned_numa_node: 1


fabric_iface: ib2
provider: ofi+verbs
pinned_numa_node: 1


fabric_iface: ib0
provider: ofi+verbs
pinned_numa_node: 1

 

And please see my inline comments.

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Lombardi, Johann
Sent: Friday, January 17, 2020 4:51 PM
To: daos@daos.groups.io; Rosenzweig, Joel B <joel.b.rosenzweig@...>
Cc: Macdonald, Mjmac <mjmac.macdonald@...>
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Hi,

 

Cc’ing Joel.

Those errors indicate that you still have some network setup issue. Could you please run daos_server network scan?

 

e.g.:

[root@wolf-118 ~]# daos_server network scan -p "ofi+verbs;ofi_rxm"

Scanning fabric for cmdline specified provider: ofi+verbs;ofi_rxm

Fabric scan found 3 devices matching the provider spec: ofi+verbs;ofi_rxm

 

                fabric_iface: ib0

                provider: ofi+verbs;ofi_rxm

                pinned_numa_node: 0

 

 

                fabric_iface: ib1

                provider: ofi+verbs;ofi_rxm

                pinned_numa_node: 1

 

 

                fabric_iface: eth0

                provider: ofi+verbs;ofi_rxm

                pinned_numa_node: 0

 

As for the NVMe issue, do I understand correctly that:

-          The PCI addresses of the 8 NVMe SSDs show up fine via daos_server storage scan

Yes.

 

-          You have reduced the number of huge pages to 4096 pages (8GB) and all the SPDK errors related to failed huge pages allocation are gone as well as this error from the log:
SPDK prepare: spdk setup failed (): exit status 1: stdout: 0000:00:05.0 (8086 0a54): nvme -> uio_pci_generic

Yes, only reporting pci addr not found and cant start, then reduced nvme to 1 seems pass this step.

-          But the io_server fails to start after dmg storage format?

Yes.

 

I had a second look at daos_control2.log and noticed that you are using 20 targets while you have 8 SSDs and 10 cores. Could you please try with #targets = #SSDs = 8 and set nr_xs_helpers to 0?

This config file was copies from physical machine, maybe some problem, several days ago, it and physical server are working, tried your suggestion, still same while formatting and cant start again.

daos_io_server:0 Using legacy core allocation algorithm
0 target XS(xstream) requested (#cores 10); use (7) target XS
DEBUG 04:51:52.588209 main.go:82: exit status 1
/root/daos/install/bin/daos_io_server (instance 0) exited

 

 

Cheers,

Johann

 

From: <daos@daos.groups.io> on behalf of Shengyu SY19 Zhang <zhangsy19@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Friday 17 January 2020 at 08:21
To: "daos@daos.groups.io" <daos@daos.groups.io>
Cc: "Macdonald, Mjmac" <mjmac.macdonald@...>
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Hello Johann,

 

For the first issue (daos_control.log), it is physical machine and specified 8 NVMes, daos server report cant find PCI address (created the log).

Then I use only one NVMe, daos server can start normally, however it stopped after dmg storage format, just like the second issue (daos_control2.log).

 

DEBUG 02:18:19.457860 config.go:378: Active config saved to /root/daos/install/etc/.daos_server.active.yml (read-only)
01/17-02:18:31.74 afa1 DAOS[185862/185862] hg   ERR  # NA -- Error -- /root/daos/_build.external/mercury/src/na/na_ofi.c:1609
 # na_ofi_domain_open(): No provider found for "verbs;ofi_rxm" provider on domain "mlx5_0/192.168.80.120"
01/17-02:18:31.74 afa1 DAOS[185862/185862] hg   ERR  # NA -- Error -- /root/daos/_build.external/mercury/src/na/na_ofi.c:2975
 # na_ofi_initialize(): Could not open domain for verbs;ofi_rxm, mlx5_0/192.168.80.120
01/17-02:18:31.74 afa1 DAOS[185862/185862] hg   ERR  # NA -- Error -- /root/daos/_build.external/mercury/src/na/na.c:324
 # NA_Initialize_opt(): Could not initialize plugin
01/17-02:18:31.74 afa1 DAOS[185862/185862] hg   ERR  src/cart/crt_hg.c:526 crt_hg_init() Could not initialize NA class.
01/17-02:18:31.74 afa1 DAOS[185862/185862] crt  ERR  src/cart/crt_init.c:317 crt_init_opt() crt_hg_init failed rc: -1020.
01/17-02:18:31.74 afa1 DAOS[185862/185862] crt  ERR  src/cart/crt_init.c:385 crt_init_opt() crt_init failed, rc: -1020.

 

Regards,

Shengyu,

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Lombardi, Johann
Sent: Thursday, January 16, 2020 8:24 PM
To: daos@daos.groups.io
Cc: Macdonald, Mjmac <mjmac.macdonald@...>
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Hi Shengyu,

 

Mike looked into the logs you provided and noticed that:

 

## ERROR: requested 40960 hugepages but only 8585 could be allocated.

## Memory might be heavily fragmented. Please try flushing the system cache, or reboot the machine.

 

Maybe you meant to specify 4096 in the yaml file instead of 40960 for nr_hugepages?  It sounds like you are trying to allocate 82GB of RAM for hugepages.

 

Cheers,

Johann

 

 

From: <daos@daos.groups.io> on behalf of "Nabarro, Tom" <tom.nabarro@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Wednesday 15 January 2020 at 13:17
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

There seems to be multiple messages about not being able to meet the required number of hugepages requested. I was not involved in the work but there have been changes in how huge page allocation is performed for each of the daos_io_server instances and we are now performing automatic storage prepare when starting daos_server.

 

What I would suggest is to reboot the nodes (in case the issues to do with memory fragmentation) and try with a smaller number of drives configured in the config file. Don’t try to do a storage prepare before running daos_server (as it will be performed on start-up anyway). And update to recent master before trying please. You could also try to bump the number of huge pages specified in the server config file to maybe 8192?

 

Regards,

Tom Nabarro – DCG/ESAD

M: +44 (0)7786 260986

Skype: tom.nabarro

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang
Sent: Wednesday, January 15, 2020 3:27 AM
To: daos@daos.groups.io
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Hello

 

Sorry I forgot attachments.

 

Regards,

Shengyu

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang
Sent: Wednesday, January 15, 2020 11:21 AM
To: daos@daos.groups.io
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Hello Johann,

 

I couldnt check it at this time, since I cant start daos server (newest code of master) on my two different machines, I assume there are lots of modification in control path, there could be some new issues:

1.       Daos_server storage prepare -nvme-only, all my NVMe disks switched to uio as expected, then issue storage scan, can see those disks as expected as well.

However, when I start daos server, it report error that reporting cant find PCI address, and all NVMs switched to kernel driver, see daos_control.log

2.       On another machine, it just stopped after being formatted, see doas_control2.log

 

Regards,

Shengyu,

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Lombardi, Johann
Sent: Tuesday, January 14, 2020 5:03 AM
To: daos@daos.groups.io
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Thanks for the logs Shengyu. It does not seem to be related to wrong endpoint addresses. The client did find the server, but this latter returned DER_NONEXIST when connecting to the pool. It might be the same problem fixed recently by PR  #1701. Could you please apply the patch or try with latest master?

 

Cheers,

Johann

 

From: <daos@daos.groups.io> on behalf of Shengyu SY19 Zhang <zhangsy19@...>
Reply-To: "
daos@daos.groups.io" <daos@daos.groups.io>
Date: Friday 10 January 2020 at 07:20
To: "
daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Hello Johann,

 

Here are log files.

My step:

1.       Create fresh environment and start daos server and then format it.

2.       Create pool

3.       Create container.

4.       List container: daos container query -svc=0 path=/tmp/mycontainer, that work great.

5.       CTRl +C to kill daos server

6.       Restart daos server

7.       Repeat 4, daos process will be dead with infinitely loop.

 

Regards,

Shengyu.

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Lombardi, Johann
Sent: Thursday, January 9, 2020 3:45 PM
To: daos@daos.groups.io
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Hm, the issue with server restart might be due to the endpoint address of the servers not being persistent.

Could you please collect full debug logs for the fresh start with reformat and the subsequent restart?

 

Cheers,

Johann

 

From: <daos@daos.groups.io> on behalf of Shengyu SY19 Zhang <zhangsy19@...>
Reply-To: "
daos@daos.groups.io" <daos@daos.groups.io>
Date: Thursday 9 January 2020 at 07:22
To: "
daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

 

Hello,

 

@Johann, Newest version will work on new formatted and created pool, yes, I did set in the yaml.

@Kal, no, I still meet the issue after io_server restart, seems there issues after load existing data.

 

Regards,

Shengyu.

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Lombardi, Johann
Sent: Thursday, January 9, 2020 5:31 AM
To: daos@daos.groups.io
Subject: [External] Re: [daos] Does DAOS support infiniband now?

 

Hi Kevan & Shengyu,

 

Could you please advise what commit hash you use? Also, are you specifying in “fabric_iface_port” in the yaml file?

 

Cheers,

Johann

 

From: <daos@daos.groups.io> on behalf of Kevan Rehm <kevan.rehm@...>
Reply-To: "
daos@daos.groups.io" <daos@daos.groups.io>
Date: Wednesday 8 January 2020 at 21:17
To: "
daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [daos] Does DAOS support infiniband now?

 

I have not had any success, I see the same failure sequence as Shengyu, and due to other commitments I've had to set this aside for a few days.   Hope to get back to it in a week or so.

 

Kevan

 

On 1/8/20, 2:15 PM, "daos@daos.groups.io on behalf of Alfizah, Kurniawan" <daos@daos.groups.io on behalf of kurniawan.alfizah@...> wrote:

 

    Hello Shengyu and Kevan,

    I'm wondering if you have resolved this problem and that DAOS is working well with IB.

    

    Cheers,

    Kal

    

    -----Original Message-----

    From: daos@daos.groups.io [mailto:daos@daos.groups.io] On Behalf Of Shengyu SY19 Zhang

    Sent: Saturday, December 28, 2019 2:46 AM

    To: daos@daos.groups.io

    Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

    

    Hello Kevan,

    

    Yes, its exact the same problem that I meet, the rdma_get_cm_event function will issue write system call to receive completing event from from ib device, however it get nothing at this time, and always return EAGAIN, that caused fi_tsend function infinitely loop.

    

    Regards,

    Shengyu.

    

    -----Original Message-----

    From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Kevan Rehm

    Sent: Saturday, December 28, 2019 12:30 AM

    To: daos@daos.groups.io

    Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

    

    Shengyu,

    

    I think we are going to need help from the experts, I am not familiar with this code.    I tried the same commands that you mentioned in your last email, and they also hang for me.   But I do not see an infinite loop; rather the daos process just hangs forever in a write() request.   Is that what you see as well?

    

    Experts: is there any documentation on CaRT, what it is for, internals?    I have not been able to find anything.

    

    The last entries in the daos_server_srv1.log file are just the ds_mgmt_drpc_get_attach_info() log messages, called from daos_agent.

    

    While the daos command was hung, I sent a kill -6 signal to it to collect the corefile.   It seems like the command has attempted to set up a MSG connection to the daos_io_server, but has not received a completion event.  The dest_addr=0 looks a little suspicious in the fi_tsend() call.  Hopefully others will recognize what the problem is in the backtrace below, otherwise I will keep digging as time permits.

    

    Thanks, Kevan

    

    (gdb) bt

    #0  0x00007fa7fd5749cd in write () from /lib64/libc.so.6

    #1  0x00007fa7fac2d875 in rdma_get_cm_event.part.13 () from /lib64/librdmacm.so.1

    #2  0x00007fa7fb7fd856 in fi_ibv_eq_read () from /home/users/daos/daos/install/lib/libfabric.so.1

    #3  0x00007fa7fb82360f in rxm_eq_read () from /home/users/daos/daos/install/lib/libfabric.so.1

    #4  0x00007fa7fb8252af in rxm_msg_eq_progress () from /home/users/daos/daos/install/lib/libfabric.so.1

    #5  0x00007fa7fb82542d in rxm_cmap_connect () from /home/users/daos/daos/install/lib/libfabric.so.1

    #6  0x00007fa7fb82b5eb in rxm_ep_tsend () from /home/users/daos/daos/install/lib/libfabric.so.1

    #7  0x00007fa7fc59f3e8 in fi_tsend (context=0x1814558, tag=1, dest_addr=0, desc=0x1811ab0, len=332, buf=0x7fa7f6a10038,

        ep=0x17f03c0) at /home/users/daos/daos/install/include/rdma/fi_tagged.h:114

    #8  na_ofi_msg_send_unexpected (na_class=0x17d6250, context=0x180f760, callback=<optimized out>, arg=<optimized out>,

        buf=0x7fa7f6a10038, buf_size=332, plugin_data=0x1811ab0, dest_addr=0x18ba5e0, dest_id=0 '\000', tag=1, op_id=0x1811a48)

        at /home/users/daos/daos/_build.external/mercury/src/na/na_ofi.c:3745

    #9  0x00007fa7fc7b79ff in NA_Msg_send_unexpected (op_id=0x1811a48, tag=<optimized out>, dest_id=<optimized out>,

        dest_addr=<optimized out>, plugin_data=<optimized out>, buf_size=<optimized out>, buf=<optimized out>, arg=0x1811920,

        callback=0x7fa7fc7b9a60 <hg_core_send_input_cb>, context=<optimized out>, na_class=<optimized out>)

        at /home/users/daos/daos/_build.external/mercury/src/na/na.h:1506

    #10 hg_core_forward_na (hg_core_handle=0x1811920) at /home/users/daos/daos/_build.external/mercury/src/mercury_core.c:2076

    #11 0x00007fa7fc7bb5e6 in HG_Core_forward (handle=0x1811920, callback=callback@entry=0x7fa7fc7b0890 <hg_core_forward_cb>,

        arg=arg@entry=0x1814730, flags=<optimized out>, payload_size=<optimized out>)

        at /home/users/daos/daos/_build.external/mercury/src/mercury_core.c:4775

    #12 0x00007fa7fc7b41f7 in HG_Forward (handle=0x1814730, callback=callback@entry=0x7fa7fd8b2980 <crt_hg_req_send_cb>,

        arg=arg@entry=0x18b9190, in_struct=in_struct@entry=0x18b91b0)

        at /home/users/daos/daos/_build.external/mercury/src/mercury.c:2165

    #13 0x00007fa7fd8b9e39 in crt_hg_req_send (rpc_priv=rpc_priv@entry=0x18b9190) at src/cart/crt_hg.c:1191

    #14 0x00007fa7fd90a8ea in crt_req_send_immediately (rpc_priv=<optimized out>) at src/cart/crt_rpc.c:1104

    #15 crt_req_send_internal (rpc_priv=rpc_priv@entry=0x18b9190) at src/cart/crt_rpc.c:1173

    #16 0x00007fa7fd90ef23 in crt_req_hg_addr_lookup_cb (hg_addr=0x18ba590, arg=0x18b9190) at src/cart/crt_rpc.c:569

    #17 0x00007fa7fd8b1062 in crt_hg_addr_lookup_cb (hg_cbinfo=<optimized out>) at src/cart/crt_hg.c:290

    #18 0x00007fa7fc7b0985 in hg_core_addr_lookup_cb (callback_info=<optimized out>)

        at /home/users/daos/daos/_build.external/mercury/src/mercury.c:454

    #19 0x00007fa7fc7bbce2 in hg_core_trigger_lookup_entry (hg_core_op_id=0x18ba530)

        at /home/users/daos/daos/_build.external/mercury/src/mercury_core.c:3444

    #20 hg_core_trigger (context=0x180d590, timeout=<optimized out>, timeout@entry=0, max_count=max_count@entry=4294967295,

        actual_count=actual_count@entry=0x7ffe185f88dc) at /home/users/daos/daos/_build.external/mercury/src/mercury_core.c:3384

    #21 0x00007fa7fc7bca4b in HG_Core_trigger (context=<optimized out>, timeout=timeout@entry=0, max_count=max_count@entry=4294967295,

        actual_count=actual_count@entry=0x7ffe185f88dc) at /home/users/daos/daos/_build.external/mercury/src/mercury_core.c:4900

    #22 0x00007fa7fc7b44ed in HG_Trigger (context=context@entry=0x17d6370, timeout=timeout@entry=0,

        max_count=max_count@entry=4294967295, actual_count=actual_count@entry=0x7ffe185f88dc)

        at /home/users/daos/daos/_build.external/mercury/src/mercury.c:2262

    #23 0x00007fa7fd8b37ea in crt_hg_trigger (hg_ctx=hg_ctx@entry=0x18083d8) at src/cart/crt_hg.c:1327

    #24 0x00007fa7fd8bce5d in crt_hg_progress (hg_ctx=hg_ctx@entry=0x18083d8, timeout=timeout@entry=1000) at src/cart/crt_hg.c:1360

    #25 0x00007fa7fd86dfbb in crt_progress (crt_ctx=0x18083c0, timeout=timeout@entry=-1,

        cond_cb=cond_cb@entry=0x7fa7fe61f7f0 <ev_progress_cb>, arg=arg@entry=0x7ffe185f89d0) at src/cart/crt_context.c:1286

    #26 0x00007fa7fe624bb6 in daos_event_priv_wait () at src/client/api/event.c:1203

    #27 0x00007fa7fe627f96 in dc_task_schedule (task=0x181b4e0, instant=instant@entry=true) at src/client/api/task.c:139

    #28 0x00007fa7fe626eb1 in daos_pool_connect (uuid=uuid@entry=0x7ffe185f8c38 "UXh}E<Kĭ<\035\340\332O", <incomplete sequence \325>,

        grp=0x1805f50 "daos_server", svc=0x1806000, flags=flags@entry=1, poh=poh@entry=0x7ffe185f8c48, info=info@entry=0x0,

        ev=ev@entry=0x0) at src/client/api/pool.c:53

    #29 0x000000000040590d in pool_query_hdlr (ap=0x7ffe185f8c20) at src/utils/daos_hdlr.c:141

    #30 0x0000000000402bc4 in main (argc=5, argv=<optimized out>) at src/utils/daos.c:957

    (gdb)

    

    On 12/25/19, 2:11 AM, "daos@daos.groups.io on behalf of Shengyu SY19 Zhang" <daos@daos.groups.io on behalf of zhangsy19@...> wrote:

    

        Hello Kevan,

        

        Here are the log files.

        Restart server means daos_server restarted, in spite of Ctrl+C to kill process or server reboot, anyhow after restart daos_server, the existing containers can't be touched, it can be 100% reproduced in my environment.

        

        dmg storage query smd -I    ->work.

        daos container query --svc=0 --path=/tmp/mycontainer         ->no response due to infinitely loop

        daos container create... ->->no response due to infinitely loop

        

        with sockets mode, didn't meet this issue.

        

        Regards,

        Shengyu.

        

        -----Original Message-----

        From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Kevan Rehm

        Sent: Wednesday, December 25, 2019 3:30 AM

        To: daos@daos.groups.io

        Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

        

        Shengyu,

        

        When you say "restart server", do you mean that you rebooted the node, or that you just restarted the daos_server process?   Could you send another daos_control.log and daos_server.log from when it fails in this way?

        

        Kevan

        

        On 12/23/19, 11:34 PM, "daos@daos.groups.io on behalf of Shengyu SY19 Zhang" <daos@daos.groups.io on behalf of zhangsy19@...> wrote:

        

            Hello Kevan,

            

            After some more testing, actually the issue is still there, I can get ib to work by following ways:

            Restart subnet

            rm -rf /mnt/daos

            start daos server

            re-format

            create pool

            create container

            

            however once I restart server, will happen the infinitely loop problem, and by any way I can't connect to an existing pool via ib.

            

            Regards,

            Shengyu.

            

            -----Original Message-----

            From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Kevan Rehm

            Sent: Saturday, December 21, 2019 6:57 AM

            To: daos@daos.groups.io

            Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

            

            Shengyu,

            

            As it happens, I also had a case today using infiniband where my daos_test client was in an infinite loop, it generated 200 million lines in daos.log within a minute or so.   It turned out that the IB subnet manager process had died.   I restarted opensm, then re-ran daos_test, and it started to work.    I mention it in case it might be the same problem as yours.   Are you sure your subnet manager is working?    Try a fi_pingpong test; if it works, then your subnet manager is okay, that's not the problem.

            

            Thanks, Kevan

            

            On 12/19/19, 12:48 AM, "daos@daos.groups.io on behalf of Shengyu SY19 Zhang" <daos@daos.groups.io on behalf of zhangsy19@...> wrote:

            

                Hello Joel,

                

                Thanks for your information, those are outputs of the test with and without rxm specified.

                Furthermore, I corrected numa setting, I thought there are mismatch issue with ib devices, I have tried unbind all other devices (ib1,2,3), and also tried remove rxm,

                All same problem happen, daos process just hangs to fi_send due to infinite loop.

                

                After I set FI_LOG_LEVEL=debug, and creating container, I found some interesting, it shows:

                

                libfabric:54383:verbs:core:ofi_check_ep_type():629<info> unsupported endpoint type

                libfabric:54383:verbs:core:ofi_check_ep_type():630<info> Supported: FI_EP_DGRAM

                libfabric:54383:verbs:core:ofi_check_ep_type():630<info> Requested: FI_EP_MSG

                libfabric:54383:verbs:core:ofi_check_domain_attr():525<info> Unknown domain name

                libfabric:54383:verbs:core:ofi_check_domain_attr():526<info> Supported: mlx5_0-xrc

                libfabric:54383:verbs:core:ofi_check_domain_attr():526<info> Requested: mlx5_0

                libfabric:54383:verbs:core:ofi_check_ep_type():629<info> unsupported endpoint type

                libfabric:54383:verbs:core:ofi_check_ep_type():630<info> Supported: FI_EP_DGRAM

                libfabric:54383:verbs:core:ofi_check_ep_type():630<info> Requested: FI_EP_MSG

                

                

                Then I changed environment domain name to mlx5_0-xrc, there will be another message, can't find ofi+verbs provider, just like before. And BTW I think daos app should be failed when connecting pool rather than infinitely loop in fi_send.

                

                Best Regards,

                Shengyu.

                

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Rosenzweig, Joel B

                Sent: Thursday, December 19, 2019 12:15 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hi Shengyu,

                

                Based on your logs, the control plane is successfully matching ib0 with mlx5_0.  It shows "DEBUG 02:06:24.772249 netdetect.go:536: Device alias for ib0 is mlx5_0"  As such, it correctly sets OFI_DOMAIN=mlx5_0.  This matches with your topology data as reported by lstopo.  Your results should not change if you manually are setting OFI_DOMAIN=mlx5_0 or not, because the control plane is already doing the right thing and you are not giving it a conflicting override.  If you find that the behavior changes when you specify OFI_DOMAIN=mlx5_0 in your daos_server.yml, that's a problem we would need to debug.

                

                Your topology shows these 4 interface cards / device combinations (mlx5_0:ib0, mlx5_1:ib1, mlx5_2:ib2 and mlx5_3:ib3).

                

                        PCI 15b3:1013 (P#548864 busid=0000:86:00.0 class=0207(IB) link=15.75GB/s PCISlot=5)

                          Network L#7 (Address=20:00:10:8b:fe:80:00:00:00:00:00:00:b8:59:9f:03:00:05:b5:48 Port=1) "ib0"

                          OpenFabrics L#8 (NodeGUID=b859:9f03:0005:b548 SysImageGUID=b859:9f03:0005:b548 Port1State=4 Port1LID=0x4 Port1LMC=0 Port1GID0=fe80:0000:0000:0000:b859:9f03:0005

                :b548) "mlx5_0"

                

                        PCI 15b3:1013 (P#548865 busid=0000:86:00.1 class=0207(IB) link=15.75GB/s PCISlot=5)

                          Network L#9 (Address=20:00:18:8b:fe:80:00:00:00:00:00:00:b8:59:9f:03:00:05:b5:49 Port=1) "ib1"

                          OpenFabrics L#10 (NodeGUID=b859:9f03:0005:b549 SysImageGUID=b859:9f03:0005:b548 Port1State=1 Port1LID=0xffff Port1LMC=0 Port1GID0=fe80:0000:0000:0000:b859:9f03:

                0005:b549) "mlx5_1"

                

                        PCI 15b3:1013 (P#716800 busid=0000:af:00.0 class=0207(IB) link=15.75GB/s PCISlot=6)

                          Network L#11 (Address=20:00:10:8b:fe:80:00:00:00:00:00:00:b8:59:9f:03:00:05:b5:40 Port=1) "ib2"

                          OpenFabrics L#12 (NodeGUID=b859:9f03:0005:b540 SysImageGUID=b859:9f03:0005:b540 Port1State=4 Port1LID=0x1b Port1LMC=0 Port1GID0=fe80:0000:0000:0000:b859:9f03:00

                05:b540) "mlx5_2"

                

                        PCI 15b3:1013 (P#716801 busid=0000:af:00.1 class=0207(IB) link=15.75GB/s PCISlot=6)

                          Network L#13 (Address=20:00:18:8b:fe:80:00:00:00:00:00:00:b8:59:9f:03:00:05:b5:41 Port=1) "ib3"

                          OpenFabrics L#14 (NodeGUID=b859:9f03:0005:b541 SysImageGUID=b859:9f03:0005:b540 Port1State=1 Port1LID=0xffff Port1LMC=0 Port1GID0=fe80:0000:0000:0000:b859:9f03:

                0005:b541) "mlx5_3"

                

                I agree that fi_info shows that the provider: ofi+verbs;ofi_rxm is valid for ib0, and the control plane agrees with that.  "DEBUG 02:06:20.594910 netdetect.go:764: Device ib0 supports provider: ofi+verbs;ofi_rxm"  This is only a guess, but I have to rule it out.  I am not certain that the provider ofi_rxm is being handled properly.  Can you remove "ofi_rxm" from your provider configuration and try again?

                

                That is, in your daos_server.yml set:

                              provider: ofi+verbs

                

                If you still have an error, then you will want to run the cart diagnostics again that Alex wrote about so we can see the latest results with that.

                

                >> orterun --allow-run-as-root -np 2 -x

                >> CRT_PHY_ADDR_STR="ofi+verbs;ofi_rxm" -x OFI_INTERFACE=ib0 -x

                >> OFI_DOMAIN=mlx5_0 ../bin/crt_launch -e tests/iv_server -v 3

                

                and

                

                >> orterun --allow-run-as-root -np 2 -x CRT_PHY_ADDR_STR="ofi+verbs" -x

                >> OFI_INTERFACE=ib0 -x OFI_DOMAIN=mlx5_0 ../bin/crt_launch -e

                >> tests/iv_server -v 3

                

                One last thing, while this won't generate a runtime error _yet_, you have a mismatch between the pinned_numa_node:0 and the actual NUMA node of your ib0 device.  Your topology data shows it as NUMA node 1.  If you run "daos_server network scan -a" it should show you that the correct pinned_numa_node is 1.  By setting it to the wrong NUMA node, you will have a performance impact once this is running because the daos_io_server will bind the threads to cores in the wrong NUMA node.  The plan is to make a validation error like this a hard error instead of a warning.  There's debug output in your daos_control.log that looks like this:

                

                DEBUG 02:06:20.595012 netdetect.go:901: Validate network config -- given numaNode: 0 DEBUG 02:06:20.872053 netdetect.go:894: ValidateNUMAConfig (device: ib0, NUMA: 0) returned error: The NUMA node for device ib0 does not match the provided value 0. Remove the pinned_numa_node value from daos_server.yml then execute 'daos_server network scan' to see the valid NUMA node associated with the network device

                

                I'll keep working on it with Alex.  Let's see what you find.

                

                Regards,

                Joel

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang

                Sent: Wednesday, December 18, 2019 4:25 AM

                To: daos@daos.groups.io

                Subject: FW: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hello Joel,

                

                I'm wondering if you have any updates for this? I'm also a developer and I very familiar with spdk/dpdk/ibverbs, however I'm not familiar with other projects which depended by the DAOS, if you can give me some hints or guide, I would like to try troubleshooting this issue as well.

                Also if you need environment to reproduce the issue, you may connect to our machine for debug.

                

                Regards,

                Shengyu

                

                

                -----Original Message-----

                From: Shengyu SY19 Zhang

                Sent: Thursday, December 12, 2019 4:05 PM

                To: 'daos@daos.groups.io' <daos@daos.groups.io>

                Subject: RE: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hello Joel,

                

                Yes, please see it in the attachment.

                

                Regards,

                Shengyu

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Rosenzweig, Joel B

                Sent: Thursday, December 12, 2019 11:08 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hi Shengyu,

                

                Can you generate a topology file for me, similar to what I asked Kevan to provide?  You can generate it with with "lstopo --of xml > topology.xml"  I am interested in seeing if your system also has device siblings with same/different ports as the one specified as the OFI_INTERFACE.

                

                Your debug log shows that DAOS chose the sibling of ib0 as mlx5_0.  It's correct that it picked something in the mlxN_N family, but, depending on your topology there could be a better device to choose, possibly one that has a port match.  Your topology file will show if mlx5_0 matches the port or not, and similarly to Kevan's, it will help me develop a better function to find the correct matching sibling.

                

                I don't know why cart failed once it had the OFI_DEVICE that we thought was correct.  Alex's experiment with Kevan will help with that.

                

                Regards,

                Joel

                

                

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang

                Sent: Wednesday, December 11, 2019 2:09 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hello Joel,

                

                Please see those files in attachment, after changed some configurations, daos trapped in (no return from rdma_get_cm_event ) creating container, here is bt log:

                

                (gdb) bt

                #0  0x00007fb8d6ef59cd in write () from /lib64/libc.so.6

                #1  0x00007fb8d45a99ad in rdma_get_cm_event.part.18 () from /lib64/librdmacm.so.1

                #2  0x00007fb8d517f836 in fi_ibv_eq_read () from /root/daos/install/lib/libfabric.so.1

                #3  0x00007fb8d51a556f in rxm_eq_read () from /root/daos/install/lib/libfabric.so.1

                #4  0x00007fb8d51a720f in rxm_msg_eq_progress () from /root/daos/install/lib/libfabric.so.1

                #5  0x00007fb8d51a738d in rxm_cmap_connect () from /root/daos/install/lib/libfabric.so.1

                #6  0x00007fb8d51ad54b in rxm_ep_tsend () from /root/daos/install/lib/libfabric.so.1

                #7  0x00007fb8d5f1e468 in fi_tsend (context=0xbc4018, tag=1, dest_addr=0, desc=0xbaf6f0, len=384, buf=0x7fb8d235c038, ep=0xbb03a0) at /root/daos/install/include/rdma/fi_tagged.h:114

                #8  na_ofi_msg_send_unexpected (na_class=0xacf220, context=0xbbf200, callback=<optimized out>, arg=<optimized out>, buf=0x7fb8d235c038, buf_size=384, plugin_data=0xbaf6f0,

                    dest_addr=0xc6a1b0, dest_id=0 '\000', tag=1, op_id=0xbc14e8) at /root/daos/_build.external/mercury/src/na/na_ofi.c:3622

                #9  0x00007fb8d613885f in NA_Msg_send_unexpected (op_id=0xbc14e8, tag=<optimized out>, dest_id=<optimized out>, dest_addr=<optimized out>, plugin_data=<optimized out>,

                    buf_size=<optimized out>, buf=<optimized out>, arg=0xbc13c0, callback=0x7fb8d613a8c0 <hg_core_send_input_cb>, context=<optimized out>, na_class=<optimized out>)

                    at /root/daos/_build.external/mercury/src/na/na.h:1485

                #10 hg_core_forward_na (hg_core_handle=0xbc13c0) at /root/daos/_build.external/mercury/src/mercury_core.c:2076

                #11 0x00007fb8d613c3a6 in HG_Core_forward (handle=0xbc13c0, callback=callback@entry=0x7fb8d6131740 <hg_core_forward_cb>, arg=arg@entry=0xbc41f0, flags=<optimized out>,

                    payload_size=<optimized out>) at /root/daos/_build.external/mercury/src/mercury_core.c:4748

                #12 0x00007fb8d6135057 in HG_Forward (handle=0xbc41f0, callback=callback@entry=0x7fb8d7233930 <crt_hg_req_send_cb>, arg=arg@entry=0xc68cb0, in_struct=in_struct@entry=0xc68cd0)

                    at /root/daos/_build.external/mercury/src/mercury.c:2147

                #13 0x00007fb8d723ade9 in crt_hg_req_send (rpc_priv=rpc_priv@entry=0xc68cb0) at src/cart/crt_hg.c:1190

                #14 0x00007fb8d728b89a in crt_req_send_immediately (rpc_priv=<optimized out>) at src/cart/crt_rpc.c:1104

                #15 crt_req_send_internal (rpc_priv=rpc_priv@entry=0xc68cb0) at src/cart/crt_rpc.c:1173

                #16 0x00007fb8d728fed3 in crt_req_hg_addr_lookup_cb (hg_addr=0xc6a150, arg=0xc68cb0) at src/cart/crt_rpc.c:569

                #17 0x00007fb8d7232012 in crt_hg_addr_lookup_cb (hg_cbinfo=<optimized out>) at src/cart/crt_hg.c:290

                #18 0x00007fb8d6131835 in hg_core_addr_lookup_cb (callback_info=<optimized out>) at /root/daos/_build.external/mercury/src/mercury.c:454

                #19 0x00007fb8d613caa2 in hg_core_trigger_lookup_entry (hg_core_op_id=0xc6a0f0) at /root/daos/_build.external/mercury/src/mercury_core.c:3444

                #20 hg_core_trigger (context=0xbbd030, timeout=<optimized out>, timeout@entry=0, max_count=max_count@entry=4294967295, actual_count=actual_count@entry=0x7fff3d70388c)

                    at /root/daos/_build.external/mercury/src/mercury_core.c:3384

                #21 0x00007fb8d613d80b in HG_Core_trigger (context=<optimized out>, timeout=timeout@entry=0, max_count=max_count@entry=4294967295, actual_count=actual_count@entry=0x7fff3d70388c)

                    at /root/daos/_build.external/mercury/src/mercury_core.c:4873

                #22 0x00007fb8d613534d in HG_Trigger (context=context@entry=0xacf1e0, timeout=timeout@entry=0, max_count=max_count@entry=4294967295, actual_count=actual_count@entry=0x7fff3d70388c)

                    at /root/daos/_build.external/mercury/src/mercury.c:2244

                #23 0x00007fb8d723479a in crt_hg_trigger (hg_ctx=hg_ctx@entry=0xbb7e78) at src/cart/crt_hg.c:1326

                #24 0x00007fb8d723de0d in crt_hg_progress (hg_ctx=hg_ctx@entry=0xbb7e78, timeout=timeout@entry=1000) at src/cart/crt_hg.c:1359

                #25 0x00007fb8d71eef6b in crt_progress (crt_ctx=0xbb7e60, timeout=timeout@entry=-1, cond_cb=cond_cb@entry=0x7fb8d7fa0230 <ev_progress_cb>, arg=arg@entry=0x7fff3d703980)

                    at src/cart/crt_context.c:1286

                #26 0x00007fb8d7fa55f6 in daos_event_priv_wait () at src/client/api/event.c:1203

                #27 0x00007fb8d7fa89d6 in dc_task_schedule (task=0xbcafa0, instant=instant@entry=true) at src/client/api/task.c:139

                #28 0x00007fb8d7fa78f1 in daos_pool_connect (uuid=uuid@entry=0x7fff3d703b68 "\265\215%f\036AN\354\203\002\317I\035\362\067\273", grp=0xbb59f0 "daos_server", svc=0xbb5aa0,

                    flags=flags@entry=2, poh=poh@entry=0x7fff3d703b78, info=info@entry=0x0, ev=ev@entry=0x0) at src/client/api/pool.c:53

                #29 0x0000000000404eb0 in cont_op_hdlr (ap=ap@entry=0x7fff3d703b50) at src/utils/daos.c:610

                #30 0x0000000000402b94 in main (argc=8, argv=<optimized out>) at src/utils/daos.c:957

                

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Rosenzweig, Joel B

                Sent: Tuesday, December 10, 2019 8:48 PM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hi Shengyu,

                

                It's hard to further diagnose without the logs. Can you share your latest daos_server.yml, full daos_control.log and full server.log? In the daos_server.yml, please set control_log_mask: DEBUG and in the io server section, set log_mask: DEBUG

                

                Regards,

                Joel

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang

                Sent: Tuesday, December 10, 2019 1:42 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hello Joel,

                

                I didn't set OFI_DOMAIN=mlx5_0 before, from the hint of Alex, I set it yesterday, there then there is another error while creating container:

                mgmt ERR  src/mgmt/cli_mgmt.c:325 get_attach_info() GetAttachInfo unsuccessful: 2

                

                

                Regards,

                Shengyu

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Rosenzweig, Joel B

                Sent: Monday, December 9, 2019 11:47 PM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hi Shengyu,

                

                With your latest setup, can you launch DAOS and send your latest daos_control.log?  In particular, I want to see how the daos_io_server environment variables are set.  For example these two lines below show the command line args and the environment variables in use with an ofi+sockets/ib1 config.  I'm looking to see if we are setting OFI_DOMAIN=mlx5_0 in your environment.  I seem to recall that your earlier logs did have this set, but since builds have changed since then, it's worth checking out one more time.

                

                DEBUG 16:33:15.711949 exec.go:112: daos_io_server:1 args: [-t 8 -x 2 -g daos_server -d /tmp/daos_sockets -s /mnt/daos1 -i 1842327892 -p 1 -I 1] DEBUG 16:33:15.711963 exec.go:113: daos_io_server:1 env: [CRT_TIMEOUT=30 FI_SOCKETS_CONN_TIMEOUT=2000 D_LOG_FILE=/tmp/server.log CRT_CTX_SHARE_ADDR=0 FI_SOCKETS_MAX_CONN_RETRY=1 D_LOG_MASK=ERR CRT_PHY_ADDR_STR=ofi+sockets OFI_INTERFACE=ib1 OFI_PORT=31416 DAOS_MD_CAP=1024]

                

                If we are not setting OFI_DOMAIN=mlx5_0 automatically, go ahead and edit your daos_server.yml and add OFI_DOMAIN=mlx5_0 to the env_vars section (per below) and see what you get.  If that doesn't get you going, please send the log for that run, too.

                

                env_vars:

                  - OFI_DOMAIN=mlx5_0

                

                Regards,

                Joel

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Oganezov, Alexander A

                Sent: Monday, December 9, 2019 5:03 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hi Shengyu,

                

                Actually this is really good result - two servers were able to exchange rpcs. That's the end of test -- just ctrl+C out of it.

                

                I'll let Joel take over for daos help on how to make daos use mlx5_0 domain.

                

                Thanks,

                ~~Alex.

                

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang

                Sent: Monday, December 09, 2019 12:57 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hello Alex,

                

                Please see the new log below, and the test seems dead (no more response):

                

                ibv_exp_query_device: invalid comp_mask !!! (comp_mask = 0x7fdd21092400 valid_mask = 0x3) [afa1][[44100,1],1][btl_openib_component.c:1670:init_one_device] error obtaining device attributes for mlx5_0 errno says Invalid argument

                ibv_exp_query_device: invalid comp_mask !!! (comp_mask = 0x195aad0 valid_mask = 0x3) [afa1][[44100,1],0][btl_openib_component.c:1670:init_one_device] error obtaining device attributes for mlx5_0 errno says Invalid argument

                --------------------------------------------------------------------------

                WARNING: There was an error initializing an OpenFabrics device.

                

                  Local host:   afa1

                  Local device: mlx5_0

                --------------------------------------------------------------------------

                12/09-03:53:20.32 afa1 CaRT[101464/101464] crt  WARN src/cart/crt_init.c:270 crt_init_opt() PMIX disabled. Disabling LM automatically

                12/09-03:53:20.32 afa1 CaRT[101465/101465] crt  WARN src/cart/crt_init.c:270 crt_init_opt() PMIX disabled. Disabling LM automatically SRV [rank=1 pid=101465] Server starting, self_rank=1 SRV [rank=0 pid=101464] Server starting, self_rank=0 SRV [rank=1 pid=101465] >>>> Entered iv_set_ivns SRV [rank=1 pid=101465] <<<< Exited iv_set_ivns:773

                

                [afa1:101458] 1 more process has sent help message help-mpi-btl-openib.txt / error in device init [afa1:101458] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

                

                Regards,

                Shengyu

                

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Oganezov, Alexander A

                Sent: Monday, December 9, 2019 4:30 PM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hi Shengyu,

                

                Given your output can you try this now?

                

                orterun --allow-run-as-root -np 2 -x CRT_PHY_ADDR_STR="ofi+verbs;ofi_rxm" -x OFI_INTERFACE=ib0 -x OFI_DOMAIN=mlx5_0 ../bin/crt_launch -e tests/iv_server -v 3

                

                Please note additional OFI_DOMAIN envariable.

                

                Thanks,

                ~~Alex.

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang

                Sent: Monday, December 09, 2019 12:25 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hello Alex,

                

                Here is outputs of fio_info related to verbs:

                

                provider: verbs

                    fabric: IB-0xfe80000000000000

                    domain: mlx5_2

                    version: 1.0

                    type: FI_EP_MSG

                    protocol: FI_PROTO_RDMA_CM_IB_RC

                provider: verbs

                    fabric: IB-0xfe80000000000000

                    domain: mlx5_2-xrc

                    version: 1.0

                    type: FI_EP_MSG

                    protocol: FI_PROTO_RDMA_CM_IB_XRC

                provider: verbs

                    fabric: IB-0xfe80000000000000

                    domain: mlx5_0

                    version: 1.0

                    type: FI_EP_MSG

                    protocol: FI_PROTO_RDMA_CM_IB_RC

                provider: verbs

                    fabric: IB-0xfe80000000000000

                    domain: mlx5_0-xrc

                    version: 1.0

                    type: FI_EP_MSG

                    protocol: FI_PROTO_RDMA_CM_IB_XRC

                provider: verbs

                    fabric: IB-0xfe80000000000000

                    domain: mlx5_0-dgram

                    version: 1.0

                    type: FI_EP_DGRAM

                    protocol: FI_PROTO_IB_UD

                provider: verbs

                    fabric: IB-0xfe80000000000000

                    domain: mlx5_2-dgram

                    version: 1.0

                    type: FI_EP_DGRAM

                    protocol: FI_PROTO_IB_UD

                provider: verbs;ofi_rxm

                    fabric: IB-0xfe80000000000000

                    domain: mlx5_2

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXM

                provider: verbs;ofi_rxm

                    fabric: IB-0xfe80000000000000

                    domain: mlx5_0

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXM

                provider: tcp;ofi_rxm

                    fabric: TCP-IP

                    domain: tcp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXM

                provider: tcp;ofi_rxm

                    fabric: TCP-IP

                    domain: tcp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXM

                provider: tcp;ofi_rxm

                    fabric: TCP-IP

                    domain: tcp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXM

                provider: tcp;ofi_rxm

                    fabric: TCP-IP

                    domain: tcp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXM

                provider: tcp;ofi_rxm

                    fabric: TCP-IP

                    domain: tcp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXM

                provider: tcp;ofi_rxm

                    fabric: TCP-IP

                    domain: tcp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXM

                provider: tcp;ofi_rxm

                    fabric: TCP-IP

                    domain: tcp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXM

                provider: tcp;ofi_rxm

                    fabric: TCP-IP

                    domain: tcp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXM

                provider: tcp;ofi_rxm

                    fabric: TCP-IP

                    domain: tcp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXM

                provider: tcp;ofi_rxm

                    fabric: TCP-IP

                    domain: tcp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXM

                provider: verbs;ofi_rxd

                    fabric: IB-0xfe80000000000000

                    domain: mlx5_2-dgram

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXD

                provider: verbs;ofi_rxd

                    fabric: IB-0xfe80000000000000

                    domain: mlx5_0-dgram

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXD

                provider: UDP;ofi_rxd

                    fabric: UDP-IP

                    domain: udp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXD

                provider: UDP;ofi_rxd

                    fabric: UDP-IP

                    domain: udp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXD

                provider: UDP;ofi_rxd

                    fabric: UDP-IP

                    domain: udp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXD

                provider: UDP;ofi_rxd

                    fabric: UDP-IP

                    domain: udp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXD

                provider: UDP;ofi_rxd

                    fabric: UDP-IP

                    domain: udp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXD

                provider: UDP;ofi_rxd

                    fabric: UDP-IP

                    domain: udp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXD

                provider: UDP;ofi_rxd

                    fabric: UDP-IP

                    domain: udp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXD

                provider: UDP;ofi_rxd

                    fabric: UDP-IP

                    domain: udp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXD

                provider: UDP;ofi_rxd

                    fabric: UDP-IP

                    domain: udp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXD

                provider: UDP;ofi_rxd

                    fabric: UDP-IP

                    domain: udp

                    version: 1.0

                    type: FI_EP_RDM

                    protocol: FI_PROTO_RXD

                

                Regards,

                Shengyu

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Oganezov, Alexander A

                Sent: Monday, December 9, 2019 4:08 PM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Thanks,

                Can you also provide full fi_info output?

                

                ~~Alex.

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang

                Sent: Monday, December 09, 2019 12:05 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hello Alex,

                

                Here are the outputs:

                

                orterun --allow-run-as-root -np 2 -x CRT_PHY_ADDR_STR="ofi+verbs;ofi_rxm" -x OFI_INTERFACE=ib0 ../bin/crt_launch -e tests/iv_server -v 3

                ibv_exp_query_device: invalid comp_mask !!! (comp_mask = 0x2094f90 valid_mask = 0x3) [afa1][[18544,1],1][btl_openib_component.c:1670:init_one_device] error obtaining device attributes for mlx5_0 errno says Invalid argument

                ibv_exp_query_device: invalid comp_mask !!! (comp_mask = 0xacda60 valid_mask = 0x3) [afa1][[18544,1],0][btl_openib_component.c:1670:init_one_device] error obtaining device attributes for mlx5_0 errno says Invalid argument

                --------------------------------------------------------------------------

                WARNING: There was an error initializing an OpenFabrics device.

                

                  Local host:   afa1

                  Local device: mlx5_0

                --------------------------------------------------------------------------

                12/09-03:01:03.53 afa1 CaRT[92269/92269] hg   ERR  # NA -- Error -- /root/daos/_build.external/mercury/src/na/na_ofi.c:1609

                 # na_ofi_domain_open(): No provider found for "verbs;ofi_rxm" provider on domain "ib0"

                12/09-03:01:03.53 afa1 CaRT[92268/92268] hg   ERR  # NA -- Error -- /root/daos/_build.external/mercury/src/na/na_ofi.c:1609

                 # na_ofi_domain_open(): No provider found for "verbs;ofi_rxm" provider on domain "ib0"

                12/09-03:01:03.53 afa1 CaRT[92269/92269] hg   ERR  # NA -- Error -- /root/daos/_build.external/mercury/src/na/na_ofi.c:2975

                 # na_ofi_initialize(): Could not open domain for verbs;ofi_rxm, ib0

                12/09-03:01:03.53 afa1 CaRT[92269/92269] hg   ERR  # NA -- Error -- /root/daos/_build.external/mercury/src/na/na.c:324

                 # NA_Initialize_opt(): Could not initialize plugin

                12/09-03:01:03.53 afa1 CaRT[92269/92269] hg   ERR  src/cart/crt_hg.c:525 crt_hg_init() Could not initialize NA class.

                12/09-03:01:03.53 afa1 CaRT[92269/92269] crt  ERR  src/cart/crt_init.c:347 crt_init_opt() crt_hg_init failed rc: -1020.

                12/09-03:01:03.53 afa1 CaRT[92269/92269] crt  ERR  src/cart/crt_init.c:421 crt_init_opt() crt_init failed, rc: -1020.

                12/09-03:01:03.53 afa1 CaRT[92268/92268] hg   ERR  # NA -- Error -- /root/daos/_build.external/mercury/src/na/na_ofi.c:2975

                 # na_ofi_initialize(): Could not open domain for verbs;ofi_rxm, ib0

                12/09-03:01:03.53 afa1 CaRT[92268/92268] hg   ERR  # NA -- Error -- /root/daos/_build.external/mercury/src/na/na.c:324

                 # NA_Initialize_opt(): Could not initialize plugin

                12/09-03:01:03.53 afa1 CaRT[92268/92268] hg   ERR  src/cart/crt_hg.c:525 crt_hg_init() Could not initialize NA class.

                12/09-03:01:03.53 afa1 CaRT[92268/92268] crt  ERR  src/cart/crt_init.c:347 crt_init_opt() crt_hg_init failed rc: -1020.

                12/09-03:01:03.53 afa1 CaRT[92268/92268] crt  ERR  src/cart/crt_init.c:421 crt_init_opt() crt_init failed, rc: -1020.

                [afa1:92262] 1 more process has sent help message help-mpi-btl-openib.txt / error in device init [afa1:92262] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

                

                

                Regards,

                Shengyu

                

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Oganezov, Alexander A

                Sent: Saturday, December 7, 2019 1:14 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hi Shengyu,

                

                With latest daos code and mofed 4.6 installed can you rerun this and show what that one gives you?

                

                source scons_local/utils/setup_local.sh

                cd install/Linux/TESTING

                orterun -np 2 -x CRT_PHY_ADDR_STR="ofi+verbs;ofi_rxm" -x OFI_INTERFACE=ib0 ../bin/crt_launch -e tests/iv_server -v 3

                

                Thanks,

                ~~Alex.

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang

                Sent: Thursday, December 05, 2019 10:25 PM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hello Alex,

                

                Yes, however with 4.6 the result is same.

                After I upgraded daos code to newest of master branch, I got some different results, daos io server seems started OK, since I can see lots of fd points to rdma_cm.

                But daos client seems can't connect to server due to same error (can't find efi+verbs provider on ib0) like the log shows, you may find the log in the attachment, that is crated via "create container"

                

                Regards,

                Shengyu.

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Oganezov, Alexander A

                Sent: Wednesday, December 4, 2019 4:02 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hi Shengyu,

                

                Can you try installing MOFED 4.6 packages on your system? In general MOFED is required to get verbs over Mellanox working.

                Those packages can be found at: https://www.mellanox.com/page/mlnx_ofed_matrix?mtag=linux_sw_drivers

                

                There is also 4.7 version available, however there seem to be few longevity issues currently when using 4.7 (according to verbs ofi maintainers).

                

                Thanks,

                ~~Alex.

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Shengyu SY19 Zhang

                Sent: Monday, November 25, 2019 9:55 PM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hello Alex,

                

                Thanks for your suggestion, here is the log:

                

                

                mca_base_component_repository_open: unable to open mca_pml_ucx: libucp.so.0: cannot open shared object file: No such file or directory (ignored)

                11/26-00:40:22.65 afa1 CaRT[365504/365504] hg   ERR  # NA -- Error -- /root/daos/_build.external/mercury/src/na/na_ofi.c:1407

                 # na_ofi_getinfo(): fi_getinfo() failed, rc: -61(No data available)

                11/26-00:40:22.65 afa1 CaRT[365504/365504] hg   ERR  # NA -- Error -- /root/daos/_build.external/mercury/src/na/na_ofi.c:2816

                 # na_ofi_check_protocol(): na_ofi_getinfo() failed

                11/26-00:40:22.65 afa1 CaRT[365504/365504] hg   ERR  # NA -- Error -- /root/daos/_build.external/mercury/src/na/na.c:302

                 # NA_Initialize_opt(): No suitable plugin found that matches ofi+verbs;ofi_rxm://192.168.80.120

                11/26-00:40:22.65 afa1 CaRT[365504/365504] hg   ERR  src/cart/crt_hg.c:521 crt_hg_init() Could not initialize NA class.

                11/26-00:40:22.65 afa1 CaRT[365504/365504] crt  ERR  src/cart/crt_init.c:347 crt_init_opt() crt_hg_init failed rc: -1020.

                11/26-00:40:22.65 afa1 CaRT[365504/365504] crt  ERR  src/cart/crt_init.c:421 crt_init_opt() crt_init failed, rc: -1020.

                

                

                Regards,

                Shengyu

                

                -----Original Message-----

                From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Oganezov, Alexander A

                Sent: Tuesday, November 26, 2019 12:21 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hi Shengyu,

                

                In order to figure out what is the issue on your system could you run cart standalone test instead and provide the output that you get?

                

                cd daos_dir

                source scons_local/utils/setup_local.sh

                cd install/Linux/TESTING

                orterun -np 2 -x CRT_PHY_ADDR_STR="ofi+verbs;ofi_rxm" -x OFI_INTERFACE=ib0 ../bin/crt_launch -e tests/iv_server -v 3

                

                Note: Depending on how you installed daos your paths might be different, so instead of cd install/Linux/TESTING you might have to cd into different directory first where you have tests/iv_server in. I think in your env it will be   cd /root/daos/install/TESTING/  or cd /root/daos/install/cart/TESTING.

                

                

                Expected output:

                11/25-15:51:48.39 wolf-55 CaRT[53295/53295] crt  WARN src/cart/crt_init.c:270 crt_init_opt() PMIX disabled. Disabling LM automatically

                11/25-15:51:48.40 wolf-55 CaRT[53296/53296] crt  WARN src/cart/crt_init.c:270 crt_init_opt() PMIX disabled. Disabling LM automatically SRV [rank=0 pid=53295]  Server starting, self_rank=0 SRV [rank=1 pid=53296]  Server starting, self_rank=1 SRV [rank=1 pid=53296]  >>>> Entered iv_set_ivns SRV [rank=1 pid=53296]  <<<< Exited iv_set_ivns:773

                

                Thanks,

                ~~Alex.

                

                

                -----Original Message-----

                From: daos@daos.groups.io [mailto:daos@daos.groups.io] On Behalf Of Shengyu SY19 Zhang

                Sent: Monday, November 25, 2019 3:28 AM

                To: daos@daos.groups.io

                Subject: Re: [External] Re: [daos] Does DAOS support infiniband now?

                

                Hello Joel,

                

                As the shown in the output.log, there is only one version of libfabrics installed in my machine, and actually I don't nave other software which depends libfabraics installed.

                From you guide to set FI_LOG_LEVEL=debug, I can see the following message, may be helpful:

                

                libfabric:123445:verbs:fabric:fi_ibv_set_default_attr():1263<info> Ignoring provider default value for tx rma_iov_limit as it is greater than the value supported by domain: mlx5_0 libfabric:123445:verbs:fabric:fi_ibv_get_matching_info():1365<info> hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints

                ERROR: daos_io_server:0 libfabric:123445:verbs:core:fi_ibv_check_hints():231<info> Unsupported capabilities libfabric:123445:verbs:core:fi_ibv_check_hints():232<info> Supported: FI_MSG, FI_RECV, FI_SEND, FI_LOCAL_COMM, FI_REMOTE_COMM libfabric:123445:verbs:core:fi_ibv_check_hints():232<info> Requested: FI_MSG, FI_RMA, FI_READ, FI_RECV, FI_SEND, FI_REMOTE_READ

                ERROR: daos_io_server:0 libfabric:123445:verbs:fabric:fi_ibv_get_rai_id():179<info> rdma_bind_addr: No such device(19)

                ERROR: daos_io_server:0 libfabric:123445:verbs:fabric:fi_ibv_get_rai_id():179<info> rdma_bind_addr: No such device(19)

                ERROR: daos_io_server:0 libfabric:123445:verbs:fabric:fi_ibv_get_rai_id():179<info> rdma_bind_addr: Invalid argument(22)

                ERROR: daos_io_server:0 libfabric:123445:verbs:fabric:fi_ibv_get_rai_id():179<info> rdma_bind_addr: Invalid argument(22)

                ERROR: daos_io_server:0 libfabric:123445:verbs:fabric:fi_ibv_get_rai_id():179<info> rdma_bind_addr: Invalid argument(22)

                ERROR: daos_io_server:0 libfabric:123445:core:core:ofi_layering_ok():795<info> Need core provider, skipping ofi_rxd libfabric:123445:core:core:ofi_layering_ok():795<info> Need core provider, skipping ofi_mrail

                

                

                Regards,