Topics

daos_obj_update crashed IO server

Zhang, Jiafu
 

Hi Guys,

 

I can call daos_obj_update successfully with one iod/sgl. But daos_obj_update crashed IO server when there were two iods/sgls under the same dkeys. From the server log, I see below error. Can someone tell me what’s going on here?

 

“dkey1”, “akey1” -> succeeded.

“dkey1”, “akey1”, “akey2” -> failed.

 

 

ERROR: daos_io_server:0 06/16-12:04:43.41 jiafu-dev DAOS[28902/28911] vos  EMRG src/vos/vos_ts.h:332 vos_ts_alloc() Assertion failed (info->ti_type & 1) == 0

ERROR: daos_io_server:0 daos_io_server: src/vos/vos_ts.h:332: vos_ts_alloc: Assertion `(info->ti_type & 1) == 0' failed.

ERROR: daos_io_server:0 *** Process 28902 received signal 6 ***

ERROR: daos_io_server:0 Associated errno: Success (0)

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7fcc895b4890]

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7fcc88b8ae97]

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7fcc88b8c801]

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(+0x3039a)[0x7fcc88b7c39a]

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(+0x30412)[0x7fcc88b7c412]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(vos_ilog_ts_cache+0x34c)[0x7fcc8a24583c]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(key_tree_prepare+0x251)[0x7fcc8a21ea41]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(vos_update_end+0x159c)[0x7fcc8a1f070c]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(+0x1e8e3)[0x7fcc7e1da8e3]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(+0x20945)[0x7fcc7e1dc945]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(ds_obj_rw_handler+0x3a0)[0x7fcc7e1dd470]

ERROR: daos_io_server:0 /usr/local/daos/lib/libcart.so.4(+0xa25ee)[0x7fcc8a5245ee]

ERROR: daos_io_server:0 /usr/local/daos/lib/libabt.so.0(+0x10ea0)[0x7fcc89393ea0]

ERROR: daos_io_server:0 /usr/local/daos/lib/libabt.so.0(+0x11601)[0x7fcc89394601]

 

Thanks.

Zhang, Jiafu
 

1.     I get this ticket, DAOS-4391. But it’s for duplicated akey in single update. I’ll try it anyway.

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Zhang, Jiafu
Sent: Tuesday, June 16, 2020 8:12 PM
To: daos@daos.groups.io
Subject: [daos] daos_obj_update crashed IO server

 

Hi Guys,

 

I can call daos_obj_update successfully with one iod/sgl. But daos_obj_update crashed IO server when there were two iods/sgls under the same dkeys. From the server log, I see below error. Can someone tell me what’s going on here?

 

“dkey1”, “akey1” -> succeeded.

“dkey1”, “akey1”, “akey2” -> failed.

 

 

ERROR: daos_io_server:0 06/16-12:04:43.41 jiafu-dev DAOS[28902/28911] vos  EMRG src/vos/vos_ts.h:332 vos_ts_alloc() Assertion failed (info->ti_type & 1) == 0

ERROR: daos_io_server:0 daos_io_server: src/vos/vos_ts.h:332: vos_ts_alloc: Assertion `(info->ti_type & 1) == 0' failed.

ERROR: daos_io_server:0 *** Process 28902 received signal 6 ***

ERROR: daos_io_server:0 Associated errno: Success (0)

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7fcc895b4890]

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7fcc88b8ae97]

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7fcc88b8c801]

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(+0x3039a)[0x7fcc88b7c39a]

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(+0x30412)[0x7fcc88b7c412]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(vos_ilog_ts_cache+0x34c)[0x7fcc8a24583c]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(key_tree_prepare+0x251)[0x7fcc8a21ea41]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(vos_update_end+0x159c)[0x7fcc8a1f070c]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(+0x1e8e3)[0x7fcc7e1da8e3]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(+0x20945)[0x7fcc7e1dc945]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(ds_obj_rw_handler+0x3a0)[0x7fcc7e1dd470]

ERROR: daos_io_server:0 /usr/local/daos/lib/libcart.so.4(+0xa25ee)[0x7fcc8a5245ee]

ERROR: daos_io_server:0 /usr/local/daos/lib/libabt.so.0(+0x10ea0)[0x7fcc89393ea0]

ERROR: daos_io_server:0 /usr/local/daos/lib/libabt.so.0(+0x11601)[0x7fcc89394601]

 

Thanks.

Lombardi, Johann
 

Hm, the patch was merged a while ago. How recent is your version of master?

Cheers,

Johann

 

From: <daos@daos.groups.io> on behalf of "Zhang, Jiafu" <jiafu.zhang@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Tuesday 16 June 2020 at 14:49
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [daos] daos_obj_update crashed IO server

 

1.       I get this ticket, DAOS-4391. But it’s for duplicated akey in single update. I’ll try it anyway.

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Zhang, Jiafu
Sent: Tuesday, June 16, 2020 8:12 PM
To: daos@daos.groups.io
Subject: [daos] daos_obj_update crashed IO server

 

Hi Guys,

 

I can call daos_obj_update successfully with one iod/sgl. But daos_obj_update crashed IO server when there were two iods/sgls under the same dkeys. From the server log, I see below error. Can someone tell me what’s going on here?

 

“dkey1”, “akey1” -> succeeded.

“dkey1”, “akey1”, “akey2” -> failed.

 

 

ERROR: daos_io_server:0 06/16-12:04:43.41 jiafu-dev DAOS[28902/28911] vos  EMRG src/vos/vos_ts.h:332 vos_ts_alloc() Assertion failed (info->ti_type & 1) == 0

ERROR: daos_io_server:0 daos_io_server: src/vos/vos_ts.h:332: vos_ts_alloc: Assertion `(info->ti_type & 1) == 0' failed.

ERROR: daos_io_server:0 *** Process 28902 received signal 6 ***

ERROR: daos_io_server:0 Associated errno: Success (0)

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7fcc895b4890]

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7fcc88b8ae97]

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7fcc88b8c801]

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(+0x3039a)[0x7fcc88b7c39a]

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(+0x30412)[0x7fcc88b7c412]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(vos_ilog_ts_cache+0x34c)[0x7fcc8a24583c]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(key_tree_prepare+0x251)[0x7fcc8a21ea41]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(vos_update_end+0x159c)[0x7fcc8a1f070c]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(+0x1e8e3)[0x7fcc7e1da8e3]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(+0x20945)[0x7fcc7e1dc945]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(ds_obj_rw_handler+0x3a0)[0x7fcc7e1dd470]

ERROR: daos_io_server:0 /usr/local/daos/lib/libcart.so.4(+0xa25ee)[0x7fcc8a5245ee]

ERROR: daos_io_server:0 /usr/local/daos/lib/libabt.so.0(+0x10ea0)[0x7fcc89393ea0]

ERROR: daos_io_server:0 /usr/local/daos/lib/libabt.so.0(+0x11601)[0x7fcc89394601]

 

Thanks.

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Zhang, Jiafu
 

Hi Johann,

 

My version is old.

 

I tried to upgrade from ubuntu 18.04 to ubuntu 20.04. And built latest DAOS code as of now from master. But I got below error when started daos server with non-root user. I’ve followed “Privileged Helper Configuration” in https://daos-stack.github.io/admin/predeployment_check/.

 

ERROR: /usr/bin/daos_admin /usr/bin/daos_admin: error while loading shared libraries: libspdk_log.so.2.0: cannot open shared object file: No such file or directory

ERROR: pbin: code = 4 description = "Unable to decode response after 5 attempts"

 

When I ldd /usr/bin/daos_admin and ldd libspdk_log.so.2.0, both worked.

 

jiafu@jiafu-dev:~/daos$ ldd /usr/bin/daos_admin

        linux-vdso.so.1 (0x00007ffd9bfeb000)

        /usr/lib/jvm/java-1.8.0-openjdk-amd64/jre/lib/amd64/libjsig.so (0x00007f63ef4b1000)

        libspdk_env_dpdk.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_env_dpdk.so.2.0 (0x00007f63ef4a0000)

        librte_mempool.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_mempool.so.20.0 (0x00007f63ef497000)

        librte_bus_pci.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_bus_pci.so.20.0 (0x00007f63ef487000)

        librte_ring.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_ring.so.20.0 (0x00007f63ef482000)

        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f63ef444000)

        libspdk_nvme.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_nvme.so.2.0 (0x00007f63ef40a000)

        libipmctl.so.4 => /usr/lib/x86_64-linux-gnu/libipmctl.so.4 (0x00007f63ef049000)

        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f63eee57000)

        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f63eee51000)

        /lib64/ld-linux-x86-64.so.2 (0x00007f63ef4b9000)

        librte_eal.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_eal.so.20.0 (0x00007f63eed6e000)

        librte_pci.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_pci.so.20.0 (0x00007f63eed67000)

        librte_kvargs.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_kvargs.so.20.0 (0x00007f63eed62000)

        libibverbs.so.1 => /usr/lib/x86_64-linux-gnu/libibverbs.so.1 (0x00007f63eed43000)

        librdmacm.so.1 => /usr/lib/x86_64-linux-gnu/librdmacm.so.1 (0x00007f63eed24000)

        libspdk_log.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_log.so.2.0 (0x00007f63eed1f000)

        libspdk_sock.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_sock.so.2.0 (0x00007f63eed1a000)

        libspdk_util.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_util.so.2.0 (0x00007f63eed07000)

        libndctl.so.6 => /usr/lib/x86_64-linux-gnu/libndctl.so.6 (0x00007f63eece4000)

        libnuma.so.1 => /usr/lib/x86_64-linux-gnu/libnuma.so.1 (0x00007f63eecd7000)

        libnl-route-3.so.200 => /usr/lib/x86_64-linux-gnu/libnl-route-3.so.200 (0x00007f63eec5f000)

        libnl-3.so.200 => /lib/x86_64-linux-gnu/libnl-3.so.200 (0x00007f63eec3c000)

        libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007f63eec31000)

        libdaxctl.so.1 => /usr/lib/x86_64-linux-gnu/libdaxctl.so.1 (0x00007f63eec27000)

        libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007f63eebfb000)

        libkmod.so.2 => /usr/lib/x86_64-linux-gnu/libkmod.so.2 (0x00007f63eebe0000)

        liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f63eebb9000)

        libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f63ee8e1000)

jiafu@jiafu-dev:~/daos$ cd /code-repo/daos-master/daos/install/lib/

jiafu@jiafu-dev:/code-repo/daos-master/daos/install/lib$ ldd libspdk_log.so.2.0

        linux-vdso.so.1 (0x00007fffa928a000)

        /usr/lib/jvm/java-1.8.0-openjdk-amd64/jre/lib/amd64/libjsig.so (0x00007fb538648000)

        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb53843d000)

        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fb538437000)

        /lib64/ld-linux-x86-64.so.2 (0x00007fb538655000)

 

Please help.

 

Thanks.

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Lombardi, Johann
Sent: Wednesday, June 17, 2020 5:48 AM
To: daos@daos.groups.io
Subject: Re: [daos] daos_obj_update crashed IO server

 

Hm, the patch was merged a while ago. How recent is your version of master?

Cheers,

Johann

 

From: <daos@daos.groups.io> on behalf of "Zhang, Jiafu" <jiafu.zhang@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Tuesday 16 June 2020 at 14:49
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [daos] daos_obj_update crashed IO server

 

  1. I get this ticket, DAOS-4391. But it’s for duplicated akey in single update. I’ll try it anyway.

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Zhang, Jiafu
Sent: Tuesday, June 16, 2020 8:12 PM
To: daos@daos.groups.io
Subject: [daos] daos_obj_update crashed IO server

 

Hi Guys,

 

I can call daos_obj_update successfully with one iod/sgl. But daos_obj_update crashed IO server when there were two iods/sgls under the same dkeys. From the server log, I see below error. Can someone tell me what’s going on here?

 

“dkey1”, “akey1” -> succeeded.

“dkey1”, “akey1”, “akey2” -> failed.

 

 

ERROR: daos_io_server:0 06/16-12:04:43.41 jiafu-dev DAOS[28902/28911] vos  EMRG src/vos/vos_ts.h:332 vos_ts_alloc() Assertion failed (info->ti_type & 1) == 0

ERROR: daos_io_server:0 daos_io_server: src/vos/vos_ts.h:332: vos_ts_alloc: Assertion `(info->ti_type & 1) == 0' failed.

ERROR: daos_io_server:0 *** Process 28902 received signal 6 ***

ERROR: daos_io_server:0 Associated errno: Success (0)

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7fcc895b4890]

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7fcc88b8ae97]

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7fcc88b8c801]

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(+0x3039a)[0x7fcc88b7c39a]

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(+0x30412)[0x7fcc88b7c412]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(vos_ilog_ts_cache+0x34c)[0x7fcc8a24583c]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(key_tree_prepare+0x251)[0x7fcc8a21ea41]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(vos_update_end+0x159c)[0x7fcc8a1f070c]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(+0x1e8e3)[0x7fcc7e1da8e3]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(+0x20945)[0x7fcc7e1dc945]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(ds_obj_rw_handler+0x3a0)[0x7fcc7e1dd470]

ERROR: daos_io_server:0 /usr/local/daos/lib/libcart.so.4(+0xa25ee)[0x7fcc8a5245ee]

ERROR: daos_io_server:0 /usr/local/daos/lib/libabt.so.0(+0x10ea0)[0x7fcc89393ea0]

ERROR: daos_io_server:0 /usr/local/daos/lib/libabt.so.0(+0x11601)[0x7fcc89394601]

 

Thanks.

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Zhang, Jiafu
 

The so issue is fixed by adding DAOS lib paths to ldconfig. It’s new behavior? I only needed to set LD_LIBRARY_PATH last time.

 

Another issue raised up when format new DAOS server.

 

$ dmg -i -l jiafu-dev:10001 storage format --reformat

processing system reformat request

ERROR: dmg: System-Query command failed: failed to get MS response: rpc error: code = Unknown desc = empty system membership

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Zhang, Jiafu
Sent: Thursday, June 18, 2020 4:00 PM
To: daos@daos.groups.io
Subject: Re: [daos] daos_obj_update crashed IO server

 

Hi Johann,

 

My version is old.

 

I tried to upgrade from ubuntu 18.04 to ubuntu 20.04. And built latest DAOS code as of now from master. But I got below error when started daos server with non-root user. I’ve followed “Privileged Helper Configuration” in https://daos-stack.github.io/admin/predeployment_check/.

 

ERROR: /usr/bin/daos_admin /usr/bin/daos_admin: error while loading shared libraries: libspdk_log.so.2.0: cannot open shared object file: No such file or directory

ERROR: pbin: code = 4 description = "Unable to decode response after 5 attempts"

 

When I ldd /usr/bin/daos_admin and ldd libspdk_log.so.2.0, both worked.

 

jiafu@jiafu-dev:~/daos$ ldd /usr/bin/daos_admin

        linux-vdso.so.1 (0x00007ffd9bfeb000)

        /usr/lib/jvm/java-1.8.0-openjdk-amd64/jre/lib/amd64/libjsig.so (0x00007f63ef4b1000)

        libspdk_env_dpdk.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_env_dpdk.so.2.0 (0x00007f63ef4a0000)

        librte_mempool.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_mempool.so.20.0 (0x00007f63ef497000)

        librte_bus_pci.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_bus_pci.so.20.0 (0x00007f63ef487000)

        librte_ring.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_ring.so.20.0 (0x00007f63ef482000)

        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f63ef444000)

        libspdk_nvme.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_nvme.so.2.0 (0x00007f63ef40a000)

        libipmctl.so.4 => /usr/lib/x86_64-linux-gnu/libipmctl.so.4 (0x00007f63ef049000)

        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f63eee57000)

        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f63eee51000)

        /lib64/ld-linux-x86-64.so.2 (0x00007f63ef4b9000)

        librte_eal.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_eal.so.20.0 (0x00007f63eed6e000)

        librte_pci.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_pci.so.20.0 (0x00007f63eed67000)

        librte_kvargs.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_kvargs.so.20.0 (0x00007f63eed62000)

        libibverbs.so.1 => /usr/lib/x86_64-linux-gnu/libibverbs.so.1 (0x00007f63eed43000)

        librdmacm.so.1 => /usr/lib/x86_64-linux-gnu/librdmacm.so.1 (0x00007f63eed24000)

        libspdk_log.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_log.so.2.0 (0x00007f63eed1f000)

        libspdk_sock.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_sock.so.2.0 (0x00007f63eed1a000)

        libspdk_util.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_util.so.2.0 (0x00007f63eed07000)

        libndctl.so.6 => /usr/lib/x86_64-linux-gnu/libndctl.so.6 (0x00007f63eece4000)

        libnuma.so.1 => /usr/lib/x86_64-linux-gnu/libnuma.so.1 (0x00007f63eecd7000)

        libnl-route-3.so.200 => /usr/lib/x86_64-linux-gnu/libnl-route-3.so.200 (0x00007f63eec5f000)

        libnl-3.so.200 => /lib/x86_64-linux-gnu/libnl-3.so.200 (0x00007f63eec3c000)

        libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007f63eec31000)

        libdaxctl.so.1 => /usr/lib/x86_64-linux-gnu/libdaxctl.so.1 (0x00007f63eec27000)

        libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007f63eebfb000)

        libkmod.so.2 => /usr/lib/x86_64-linux-gnu/libkmod.so.2 (0x00007f63eebe0000)

        liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f63eebb9000)

        libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f63ee8e1000)

jiafu@jiafu-dev:~/daos$ cd /code-repo/daos-master/daos/install/lib/

jiafu@jiafu-dev:/code-repo/daos-master/daos/install/lib$ ldd libspdk_log.so.2.0

        linux-vdso.so.1 (0x00007fffa928a000)

        /usr/lib/jvm/java-1.8.0-openjdk-amd64/jre/lib/amd64/libjsig.so (0x00007fb538648000)

        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb53843d000)

        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fb538437000)

        /lib64/ld-linux-x86-64.so.2 (0x00007fb538655000)

 

Please help.

 

Thanks.

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Lombardi, Johann
Sent: Wednesday, June 17, 2020 5:48 AM
To: daos@daos.groups.io
Subject: Re: [daos] daos_obj_update crashed IO server

 

Hm, the patch was merged a while ago. How recent is your version of master?

Cheers,

Johann

 

From: <daos@daos.groups.io> on behalf of "Zhang, Jiafu" <jiafu.zhang@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Tuesday 16 June 2020 at 14:49
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [daos] daos_obj_update crashed IO server

 

  1. I get this ticket, DAOS-4391. But it’s for duplicated akey in single update. I’ll try it anyway.

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Zhang, Jiafu
Sent: Tuesday, June 16, 2020 8:12 PM
To: daos@daos.groups.io
Subject: [daos] daos_obj_update crashed IO server

 

Hi Guys,

 

I can call daos_obj_update successfully with one iod/sgl. But daos_obj_update crashed IO server when there were two iods/sgls under the same dkeys. From the server log, I see below error. Can someone tell me what’s going on here?

 

“dkey1”, “akey1” -> succeeded.

“dkey1”, “akey1”, “akey2” -> failed.

 

 

ERROR: daos_io_server:0 06/16-12:04:43.41 jiafu-dev DAOS[28902/28911] vos  EMRG src/vos/vos_ts.h:332 vos_ts_alloc() Assertion failed (info->ti_type & 1) == 0

ERROR: daos_io_server:0 daos_io_server: src/vos/vos_ts.h:332: vos_ts_alloc: Assertion `(info->ti_type & 1) == 0' failed.

ERROR: daos_io_server:0 *** Process 28902 received signal 6 ***

ERROR: daos_io_server:0 Associated errno: Success (0)

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7fcc895b4890]

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7fcc88b8ae97]

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7fcc88b8c801]

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(+0x3039a)[0x7fcc88b7c39a]

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(+0x30412)[0x7fcc88b7c412]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(vos_ilog_ts_cache+0x34c)[0x7fcc8a24583c]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(key_tree_prepare+0x251)[0x7fcc8a21ea41]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(vos_update_end+0x159c)[0x7fcc8a1f070c]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(+0x1e8e3)[0x7fcc7e1da8e3]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(+0x20945)[0x7fcc7e1dc945]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(ds_obj_rw_handler+0x3a0)[0x7fcc7e1dd470]

ERROR: daos_io_server:0 /usr/local/daos/lib/libcart.so.4(+0xa25ee)[0x7fcc8a5245ee]

ERROR: daos_io_server:0 /usr/local/daos/lib/libabt.so.0(+0x10ea0)[0x7fcc89393ea0]

ERROR: daos_io_server:0 /usr/local/daos/lib/libabt.so.0(+0x11601)[0x7fcc89394601]

 

Thanks.

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Zhang, Jiafu
 

When I reverted to 6c3eb801d0a9307cdf88b8634ea5a103d0b3542f, the format command worked for me.

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Zhang, Jiafu
Sent: Thursday, June 18, 2020 6:18 PM
To: daos@daos.groups.io
Subject: Re: [daos] daos_obj_update crashed IO server

 

The so issue is fixed by adding DAOS lib paths to ldconfig. It’s new behavior? I only needed to set LD_LIBRARY_PATH last time.

 

Another issue raised up when format new DAOS server.

 

$ dmg -i -l jiafu-dev:10001 storage format --reformat

processing system reformat request

ERROR: dmg: System-Query command failed: failed to get MS response: rpc error: code = Unknown desc = empty system membership

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Zhang, Jiafu
Sent: Thursday, June 18, 2020 4:00 PM
To: daos@daos.groups.io
Subject: Re: [daos] daos_obj_update crashed IO server

 

Hi Johann,

 

My version is old.

 

I tried to upgrade from ubuntu 18.04 to ubuntu 20.04. And built latest DAOS code as of now from master. But I got below error when started daos server with non-root user. I’ve followed “Privileged Helper Configuration” in https://daos-stack.github.io/admin/predeployment_check/.

 

ERROR: /usr/bin/daos_admin /usr/bin/daos_admin: error while loading shared libraries: libspdk_log.so.2.0: cannot open shared object file: No such file or directory

ERROR: pbin: code = 4 description = "Unable to decode response after 5 attempts"

 

When I ldd /usr/bin/daos_admin and ldd libspdk_log.so.2.0, both worked.

 

jiafu@jiafu-dev:~/daos$ ldd /usr/bin/daos_admin

        linux-vdso.so.1 (0x00007ffd9bfeb000)

        /usr/lib/jvm/java-1.8.0-openjdk-amd64/jre/lib/amd64/libjsig.so (0x00007f63ef4b1000)

        libspdk_env_dpdk.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_env_dpdk.so.2.0 (0x00007f63ef4a0000)

        librte_mempool.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_mempool.so.20.0 (0x00007f63ef497000)

        librte_bus_pci.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_bus_pci.so.20.0 (0x00007f63ef487000)

        librte_ring.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_ring.so.20.0 (0x00007f63ef482000)

        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f63ef444000)

        libspdk_nvme.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_nvme.so.2.0 (0x00007f63ef40a000)

        libipmctl.so.4 => /usr/lib/x86_64-linux-gnu/libipmctl.so.4 (0x00007f63ef049000)

        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f63eee57000)

        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f63eee51000)

        /lib64/ld-linux-x86-64.so.2 (0x00007f63ef4b9000)

        librte_eal.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_eal.so.20.0 (0x00007f63eed6e000)

        librte_pci.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_pci.so.20.0 (0x00007f63eed67000)

        librte_kvargs.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_kvargs.so.20.0 (0x00007f63eed62000)

        libibverbs.so.1 => /usr/lib/x86_64-linux-gnu/libibverbs.so.1 (0x00007f63eed43000)

        librdmacm.so.1 => /usr/lib/x86_64-linux-gnu/librdmacm.so.1 (0x00007f63eed24000)

        libspdk_log.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_log.so.2.0 (0x00007f63eed1f000)

        libspdk_sock.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_sock.so.2.0 (0x00007f63eed1a000)

        libspdk_util.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_util.so.2.0 (0x00007f63eed07000)

        libndctl.so.6 => /usr/lib/x86_64-linux-gnu/libndctl.so.6 (0x00007f63eece4000)

        libnuma.so.1 => /usr/lib/x86_64-linux-gnu/libnuma.so.1 (0x00007f63eecd7000)

        libnl-route-3.so.200 => /usr/lib/x86_64-linux-gnu/libnl-route-3.so.200 (0x00007f63eec5f000)

        libnl-3.so.200 => /lib/x86_64-linux-gnu/libnl-3.so.200 (0x00007f63eec3c000)

        libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007f63eec31000)

        libdaxctl.so.1 => /usr/lib/x86_64-linux-gnu/libdaxctl.so.1 (0x00007f63eec27000)

        libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007f63eebfb000)

        libkmod.so.2 => /usr/lib/x86_64-linux-gnu/libkmod.so.2 (0x00007f63eebe0000)

        liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f63eebb9000)

        libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f63ee8e1000)

jiafu@jiafu-dev:~/daos$ cd /code-repo/daos-master/daos/install/lib/

jiafu@jiafu-dev:/code-repo/daos-master/daos/install/lib$ ldd libspdk_log.so.2.0

        linux-vdso.so.1 (0x00007fffa928a000)

        /usr/lib/jvm/java-1.8.0-openjdk-amd64/jre/lib/amd64/libjsig.so (0x00007fb538648000)

        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb53843d000)

        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fb538437000)

        /lib64/ld-linux-x86-64.so.2 (0x00007fb538655000)

 

Please help.

 

Thanks.

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Lombardi, Johann
Sent: Wednesday, June 17, 2020 5:48 AM
To: daos@daos.groups.io
Subject: Re: [daos] daos_obj_update crashed IO server

 

Hm, the patch was merged a while ago. How recent is your version of master?

Cheers,

Johann

 

From: <daos@daos.groups.io> on behalf of "Zhang, Jiafu" <jiafu.zhang@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Tuesday 16 June 2020 at 14:49
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [daos] daos_obj_update crashed IO server

 

  1. I get this ticket, DAOS-4391. But it’s for duplicated akey in single update. I’ll try it anyway.

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Zhang, Jiafu
Sent: Tuesday, June 16, 2020 8:12 PM
To: daos@daos.groups.io
Subject: [daos] daos_obj_update crashed IO server

 

Hi Guys,

 

I can call daos_obj_update successfully with one iod/sgl. But daos_obj_update crashed IO server when there were two iods/sgls under the same dkeys. From the server log, I see below error. Can someone tell me what’s going on here?

 

“dkey1”, “akey1” -> succeeded.

“dkey1”, “akey1”, “akey2” -> failed.

 

 

ERROR: daos_io_server:0 06/16-12:04:43.41 jiafu-dev DAOS[28902/28911] vos  EMRG src/vos/vos_ts.h:332 vos_ts_alloc() Assertion failed (info->ti_type & 1) == 0

ERROR: daos_io_server:0 daos_io_server: src/vos/vos_ts.h:332: vos_ts_alloc: Assertion `(info->ti_type & 1) == 0' failed.

ERROR: daos_io_server:0 *** Process 28902 received signal 6 ***

ERROR: daos_io_server:0 Associated errno: Success (0)

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7fcc895b4890]

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7fcc88b8ae97]

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7fcc88b8c801]

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(+0x3039a)[0x7fcc88b7c39a]

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(+0x30412)[0x7fcc88b7c412]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(vos_ilog_ts_cache+0x34c)[0x7fcc8a24583c]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(key_tree_prepare+0x251)[0x7fcc8a21ea41]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(vos_update_end+0x159c)[0x7fcc8a1f070c]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(+0x1e8e3)[0x7fcc7e1da8e3]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(+0x20945)[0x7fcc7e1dc945]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(ds_obj_rw_handler+0x3a0)[0x7fcc7e1dd470]

ERROR: daos_io_server:0 /usr/local/daos/lib/libcart.so.4(+0xa25ee)[0x7fcc8a5245ee]

ERROR: daos_io_server:0 /usr/local/daos/lib/libabt.so.0(+0x10ea0)[0x7fcc89393ea0]

ERROR: daos_io_server:0 /usr/local/daos/lib/libabt.so.0(+0x11601)[0x7fcc89394601]

 

Thanks.

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Zhang, Jiafu
 

Hi Johann,

 

In recent commit (6c3eb801d0a9307cdf88b8634ea5a103d0b3542f), the daos io server crash issue is gone when update object with two akeys under the same dkey.

 

But I got another issue, DER_NO_PERM, no permission. And the update with single akey still worked.

 

By the way, listing dkeys worked. But listing akeys didn’t work in this commit with error invalid parameter. The error in server is,

 

06/19-00:23:45.58 jiafu-dev DAOS[348411/348420] bulk ERR  src/cart/src/cart/crt_bulk.c:106 crt_bulk_desc_valid() invalid parameter of bulk_desc (remote_hdl:0x7f13c8b76400,local_hdl:0x7f13c8b7edf0, bulk_op:104, len: 0.

06/19-00:23:45.58 jiafu-dev DAOS[348411/348420] bulk ERR  src/cart/src/cart/crt_bulk.c:205 crt_bulk_transfer() invalid parameter of bulk_desc.

06/19-00:23:45.58 jiafu-dev DAOS[348411/348420] object ERR  src/object/srv_obj.c:390 obj_bulk_transfer() crt_bulk_transfer 0 error (-1003).

06/19-00:23:45.58 jiafu-dev DAOS[351611/351612] object ERR  src/object/cli_shard.c:856 dc_enumerate_cb() rpc 0x7f79e4aca470 RPC 3 failed: DER_INVAL(-1003)

06/19-00:24:32.05 jiafu-dev DAOS[348411/348420] vos  ERR  src/vos/vos_obj.c:1596 vos_obj_iter_aggregate() Removing orphaned array value tree

06/19-00:26:02.06 jiafu-dev DAOS[348411/348420] vos  ERR  src/vos/vos_obj.c:1596 vos_obj_iter_aggregate() Removing orphaned array value tree

06/19-00:26:02.06 jiafu-dev DAOS[348411/348424] vos  ERR  src/vos/vos_obj.c:1596 vos_obj_iter_aggregate() Removing orphaned array value tree

 

Thanks.

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Zhang, Jiafu
Sent: Thursday, June 18, 2020 6:24 PM
To: daos@daos.groups.io
Subject: Re: [daos] daos_obj_update crashed IO server

 

When I reverted to 6c3eb801d0a9307cdf88b8634ea5a103d0b3542f, the format command worked for me.

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Zhang, Jiafu
Sent: Thursday, June 18, 2020 6:18 PM
To: daos@daos.groups.io
Subject: Re: [daos] daos_obj_update crashed IO server

 

The so issue is fixed by adding DAOS lib paths to ldconfig. It’s new behavior? I only needed to set LD_LIBRARY_PATH last time.

 

Another issue raised up when format new DAOS server.

 

$ dmg -i -l jiafu-dev:10001 storage format --reformat

processing system reformat request

ERROR: dmg: System-Query command failed: failed to get MS response: rpc error: code = Unknown desc = empty system membership

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Zhang, Jiafu
Sent: Thursday, June 18, 2020 4:00 PM
To: daos@daos.groups.io
Subject: Re: [daos] daos_obj_update crashed IO server

 

Hi Johann,

 

My version is old.

 

I tried to upgrade from ubuntu 18.04 to ubuntu 20.04. And built latest DAOS code as of now from master. But I got below error when started daos server with non-root user. I’ve followed “Privileged Helper Configuration” in https://daos-stack.github.io/admin/predeployment_check/.

 

ERROR: /usr/bin/daos_admin /usr/bin/daos_admin: error while loading shared libraries: libspdk_log.so.2.0: cannot open shared object file: No such file or directory

ERROR: pbin: code = 4 description = "Unable to decode response after 5 attempts"

 

When I ldd /usr/bin/daos_admin and ldd libspdk_log.so.2.0, both worked.

 

jiafu@jiafu-dev:~/daos$ ldd /usr/bin/daos_admin

        linux-vdso.so.1 (0x00007ffd9bfeb000)

        /usr/lib/jvm/java-1.8.0-openjdk-amd64/jre/lib/amd64/libjsig.so (0x00007f63ef4b1000)

        libspdk_env_dpdk.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_env_dpdk.so.2.0 (0x00007f63ef4a0000)

        librte_mempool.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_mempool.so.20.0 (0x00007f63ef497000)

        librte_bus_pci.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_bus_pci.so.20.0 (0x00007f63ef487000)

        librte_ring.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_ring.so.20.0 (0x00007f63ef482000)

        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f63ef444000)

        libspdk_nvme.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_nvme.so.2.0 (0x00007f63ef40a000)

        libipmctl.so.4 => /usr/lib/x86_64-linux-gnu/libipmctl.so.4 (0x00007f63ef049000)

        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f63eee57000)

        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f63eee51000)

        /lib64/ld-linux-x86-64.so.2 (0x00007f63ef4b9000)

        librte_eal.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_eal.so.20.0 (0x00007f63eed6e000)

        librte_pci.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_pci.so.20.0 (0x00007f63eed67000)

        librte_kvargs.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_kvargs.so.20.0 (0x00007f63eed62000)

        libibverbs.so.1 => /usr/lib/x86_64-linux-gnu/libibverbs.so.1 (0x00007f63eed43000)

        librdmacm.so.1 => /usr/lib/x86_64-linux-gnu/librdmacm.so.1 (0x00007f63eed24000)

        libspdk_log.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_log.so.2.0 (0x00007f63eed1f000)

        libspdk_sock.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_sock.so.2.0 (0x00007f63eed1a000)

        libspdk_util.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_util.so.2.0 (0x00007f63eed07000)

        libndctl.so.6 => /usr/lib/x86_64-linux-gnu/libndctl.so.6 (0x00007f63eece4000)

        libnuma.so.1 => /usr/lib/x86_64-linux-gnu/libnuma.so.1 (0x00007f63eecd7000)

        libnl-route-3.so.200 => /usr/lib/x86_64-linux-gnu/libnl-route-3.so.200 (0x00007f63eec5f000)

        libnl-3.so.200 => /lib/x86_64-linux-gnu/libnl-3.so.200 (0x00007f63eec3c000)

        libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007f63eec31000)

        libdaxctl.so.1 => /usr/lib/x86_64-linux-gnu/libdaxctl.so.1 (0x00007f63eec27000)

        libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007f63eebfb000)

        libkmod.so.2 => /usr/lib/x86_64-linux-gnu/libkmod.so.2 (0x00007f63eebe0000)

        liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f63eebb9000)

        libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f63ee8e1000)

jiafu@jiafu-dev:~/daos$ cd /code-repo/daos-master/daos/install/lib/

jiafu@jiafu-dev:/code-repo/daos-master/daos/install/lib$ ldd libspdk_log.so.2.0

        linux-vdso.so.1 (0x00007fffa928a000)

        /usr/lib/jvm/java-1.8.0-openjdk-amd64/jre/lib/amd64/libjsig.so (0x00007fb538648000)

        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb53843d000)

        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fb538437000)

        /lib64/ld-linux-x86-64.so.2 (0x00007fb538655000)

 

Please help.

 

Thanks.

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Lombardi, Johann
Sent: Wednesday, June 17, 2020 5:48 AM
To: daos@daos.groups.io
Subject: Re: [daos] daos_obj_update crashed IO server

 

Hm, the patch was merged a while ago. How recent is your version of master?

Cheers,

Johann

 

From: <daos@daos.groups.io> on behalf of "Zhang, Jiafu" <jiafu.zhang@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Tuesday 16 June 2020 at 14:49
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [daos] daos_obj_update crashed IO server

 

  1. I get this ticket, DAOS-4391. But it’s for duplicated akey in single update. I’ll try it anyway.

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Zhang, Jiafu
Sent: Tuesday, June 16, 2020 8:12 PM
To: daos@daos.groups.io
Subject: [daos] daos_obj_update crashed IO server

 

Hi Guys,

 

I can call daos_obj_update successfully with one iod/sgl. But daos_obj_update crashed IO server when there were two iods/sgls under the same dkeys. From the server log, I see below error. Can someone tell me what’s going on here?

 

“dkey1”, “akey1” -> succeeded.

“dkey1”, “akey1”, “akey2” -> failed.

 

 

ERROR: daos_io_server:0 06/16-12:04:43.41 jiafu-dev DAOS[28902/28911] vos  EMRG src/vos/vos_ts.h:332 vos_ts_alloc() Assertion failed (info->ti_type & 1) == 0

ERROR: daos_io_server:0 daos_io_server: src/vos/vos_ts.h:332: vos_ts_alloc: Assertion `(info->ti_type & 1) == 0' failed.

ERROR: daos_io_server:0 *** Process 28902 received signal 6 ***

ERROR: daos_io_server:0 Associated errno: Success (0)

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7fcc895b4890]

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7fcc88b8ae97]

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7fcc88b8c801]

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(+0x3039a)[0x7fcc88b7c39a]

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(+0x30412)[0x7fcc88b7c412]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(vos_ilog_ts_cache+0x34c)[0x7fcc8a24583c]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(key_tree_prepare+0x251)[0x7fcc8a21ea41]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(vos_update_end+0x159c)[0x7fcc8a1f070c]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(+0x1e8e3)[0x7fcc7e1da8e3]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(+0x20945)[0x7fcc7e1dc945]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(ds_obj_rw_handler+0x3a0)[0x7fcc7e1dd470]

ERROR: daos_io_server:0 /usr/local/daos/lib/libcart.so.4(+0xa25ee)[0x7fcc8a5245ee]

ERROR: daos_io_server:0 /usr/local/daos/lib/libabt.so.0(+0x10ea0)[0x7fcc89393ea0]

ERROR: daos_io_server:0 /usr/local/daos/lib/libabt.so.0(+0x11601)[0x7fcc89394601]

 

Thanks.

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Lombardi, Johann
 

Hi Jiafu,

 

Could you please email me your test case? I will look into it.

Cheers,

Johann

 

From: <daos@daos.groups.io> on behalf of "Zhang, Jiafu" <jiafu.zhang@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Friday 19 June 2020 at 02:52
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [daos] daos_obj_update crashed IO server

 

Hi Johann,

 

In recent commit (6c3eb801d0a9307cdf88b8634ea5a103d0b3542f), the daos io server crash issue is gone when update object with two akeys under the same dkey.

 

But I got another issue, DER_NO_PERM, no permission. And the update with single akey still worked.

 

By the way, listing dkeys worked. But listing akeys didn’t work in this commit with error invalid parameter. The error in server is,

 

06/19-00:23:45.58 jiafu-dev DAOS[348411/348420] bulk ERR  src/cart/src/cart/crt_bulk.c:106 crt_bulk_desc_valid() invalid parameter of bulk_desc (remote_hdl:0x7f13c8b76400,local_hdl:0x7f13c8b7edf0, bulk_op:104, len: 0.

06/19-00:23:45.58 jiafu-dev DAOS[348411/348420] bulk ERR  src/cart/src/cart/crt_bulk.c:205 crt_bulk_transfer() invalid parameter of bulk_desc.

06/19-00:23:45.58 jiafu-dev DAOS[348411/348420] object ERR  src/object/srv_obj.c:390 obj_bulk_transfer() crt_bulk_transfer 0 error (-1003).

06/19-00:23:45.58 jiafu-dev DAOS[351611/351612] object ERR  src/object/cli_shard.c:856 dc_enumerate_cb() rpc 0x7f79e4aca470 RPC 3 failed: DER_INVAL(-1003)

06/19-00:24:32.05 jiafu-dev DAOS[348411/348420] vos  ERR  src/vos/vos_obj.c:1596 vos_obj_iter_aggregate() Removing orphaned array value tree

06/19-00:26:02.06 jiafu-dev DAOS[348411/348420] vos  ERR  src/vos/vos_obj.c:1596 vos_obj_iter_aggregate() Removing orphaned array value tree

06/19-00:26:02.06 jiafu-dev DAOS[348411/348424] vos  ERR  src/vos/vos_obj.c:1596 vos_obj_iter_aggregate() Removing orphaned array value tree

 

Thanks.

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Zhang, Jiafu
Sent: Thursday, June 18, 2020 6:24 PM
To: daos@daos.groups.io
Subject: Re: [daos] daos_obj_update crashed IO server

 

When I reverted to 6c3eb801d0a9307cdf88b8634ea5a103d0b3542f, the format command worked for me.

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Zhang, Jiafu
Sent: Thursday, June 18, 2020 6:18 PM
To: daos@daos.groups.io
Subject: Re: [daos] daos_obj_update crashed IO server

 

The so issue is fixed by adding DAOS lib paths to ldconfig. It’s new behavior? I only needed to set LD_LIBRARY_PATH last time.

 

Another issue raised up when format new DAOS server.

 

$ dmg -i -l jiafu-dev:10001 storage format --reformat

processing system reformat request

ERROR: dmg: System-Query command failed: failed to get MS response: rpc error: code = Unknown desc = empty system membership

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Zhang, Jiafu
Sent: Thursday, June 18, 2020 4:00 PM
To: daos@daos.groups.io
Subject: Re: [daos] daos_obj_update crashed IO server

 

Hi Johann,

 

My version is old.

 

I tried to upgrade from ubuntu 18.04 to ubuntu 20.04. And built latest DAOS code as of now from master. But I got below error when started daos server with non-root user. I’ve followed “Privileged Helper Configuration” in https://daos-stack.github.io/admin/predeployment_check/.

 

ERROR: /usr/bin/daos_admin /usr/bin/daos_admin: error while loading shared libraries: libspdk_log.so.2.0: cannot open shared object file: No such file or directory

ERROR: pbin: code = 4 description = "Unable to decode response after 5 attempts"

 

When I ldd /usr/bin/daos_admin and ldd libspdk_log.so.2.0, both worked.

 

jiafu@jiafu-dev:~/daos$ ldd /usr/bin/daos_admin

        linux-vdso.so.1 (0x00007ffd9bfeb000)

        /usr/lib/jvm/java-1.8.0-openjdk-amd64/jre/lib/amd64/libjsig.so (0x00007f63ef4b1000)

        libspdk_env_dpdk.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_env_dpdk.so.2.0 (0x00007f63ef4a0000)

        librte_mempool.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_mempool.so.20.0 (0x00007f63ef497000)

        librte_bus_pci.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_bus_pci.so.20.0 (0x00007f63ef487000)

        librte_ring.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_ring.so.20.0 (0x00007f63ef482000)

        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f63ef444000)

        libspdk_nvme.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_nvme.so.2.0 (0x00007f63ef40a000)

        libipmctl.so.4 => /usr/lib/x86_64-linux-gnu/libipmctl.so.4 (0x00007f63ef049000)

        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f63eee57000)

        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f63eee51000)

        /lib64/ld-linux-x86-64.so.2 (0x00007f63ef4b9000)

        librte_eal.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_eal.so.20.0 (0x00007f63eed6e000)

        librte_pci.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_pci.so.20.0 (0x00007f63eed67000)

        librte_kvargs.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_kvargs.so.20.0 (0x00007f63eed62000)

        libibverbs.so.1 => /usr/lib/x86_64-linux-gnu/libibverbs.so.1 (0x00007f63eed43000)

        librdmacm.so.1 => /usr/lib/x86_64-linux-gnu/librdmacm.so.1 (0x00007f63eed24000)

        libspdk_log.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_log.so.2.0 (0x00007f63eed1f000)

        libspdk_sock.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_sock.so.2.0 (0x00007f63eed1a000)

        libspdk_util.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_util.so.2.0 (0x00007f63eed07000)

        libndctl.so.6 => /usr/lib/x86_64-linux-gnu/libndctl.so.6 (0x00007f63eece4000)

        libnuma.so.1 => /usr/lib/x86_64-linux-gnu/libnuma.so.1 (0x00007f63eecd7000)

        libnl-route-3.so.200 => /usr/lib/x86_64-linux-gnu/libnl-route-3.so.200 (0x00007f63eec5f000)

        libnl-3.so.200 => /lib/x86_64-linux-gnu/libnl-3.so.200 (0x00007f63eec3c000)

        libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007f63eec31000)

        libdaxctl.so.1 => /usr/lib/x86_64-linux-gnu/libdaxctl.so.1 (0x00007f63eec27000)

        libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007f63eebfb000)

        libkmod.so.2 => /usr/lib/x86_64-linux-gnu/libkmod.so.2 (0x00007f63eebe0000)

        liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f63eebb9000)

        libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f63ee8e1000)

jiafu@jiafu-dev:~/daos$ cd /code-repo/daos-master/daos/install/lib/

jiafu@jiafu-dev:/code-repo/daos-master/daos/install/lib$ ldd libspdk_log.so.2.0

        linux-vdso.so.1 (0x00007fffa928a000)

        /usr/lib/jvm/java-1.8.0-openjdk-amd64/jre/lib/amd64/libjsig.so (0x00007fb538648000)

        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb53843d000)

        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fb538437000)

        /lib64/ld-linux-x86-64.so.2 (0x00007fb538655000)

 

Please help.

 

Thanks.

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Lombardi, Johann
Sent: Wednesday, June 17, 2020 5:48 AM
To: daos@daos.groups.io
Subject: Re: [daos] daos_obj_update crashed IO server

 

Hm, the patch was merged a while ago. How recent is your version of master?

Cheers,

Johann

 

From: <daos@daos.groups.io> on behalf of "Zhang, Jiafu" <jiafu.zhang@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Tuesday 16 June 2020 at 14:49
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [daos] daos_obj_update crashed IO server

 

1.       I get this ticket, DAOS-4391. But it’s for duplicated akey in single update. I’ll try it anyway.

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Zhang, Jiafu
Sent: Tuesday, June 16, 2020 8:12 PM
To: daos@daos.groups.io
Subject: [daos] daos_obj_update crashed IO server

 

Hi Guys,

 

I can call daos_obj_update successfully with one iod/sgl. But daos_obj_update crashed IO server when there were two iods/sgls under the same dkeys. From the server log, I see below error. Can someone tell me what’s going on here?

 

“dkey1”, “akey1” -> succeeded.

“dkey1”, “akey1”, “akey2” -> failed.

 

 

ERROR: daos_io_server:0 06/16-12:04:43.41 jiafu-dev DAOS[28902/28911] vos  EMRG src/vos/vos_ts.h:332 vos_ts_alloc() Assertion failed (info->ti_type & 1) == 0

ERROR: daos_io_server:0 daos_io_server: src/vos/vos_ts.h:332: vos_ts_alloc: Assertion `(info->ti_type & 1) == 0' failed.

ERROR: daos_io_server:0 *** Process 28902 received signal 6 ***

ERROR: daos_io_server:0 Associated errno: Success (0)

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7fcc895b4890]

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7fcc88b8ae97]

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7fcc88b8c801]

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(+0x3039a)[0x7fcc88b7c39a]

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(+0x30412)[0x7fcc88b7c412]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(vos_ilog_ts_cache+0x34c)[0x7fcc8a24583c]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(key_tree_prepare+0x251)[0x7fcc8a21ea41]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(vos_update_end+0x159c)[0x7fcc8a1f070c]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(+0x1e8e3)[0x7fcc7e1da8e3]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(+0x20945)[0x7fcc7e1dc945]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(ds_obj_rw_handler+0x3a0)[0x7fcc7e1dd470]

ERROR: daos_io_server:0 /usr/local/daos/lib/libcart.so.4(+0xa25ee)[0x7fcc8a5245ee]

ERROR: daos_io_server:0 /usr/local/daos/lib/libabt.so.0(+0x10ea0)[0x7fcc89393ea0]

ERROR: daos_io_server:0 /usr/local/daos/lib/libabt.so.0(+0x11601)[0x7fcc89394601]

 

Thanks.

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Nabarro, Tom
 

The bug with the storage reformat command has been fixed by Mike on master.

 

Apologies for the inconvenience.

 

Regards,

Tom

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Lombardi, Johann
Sent: Friday, June 19, 2020 12:30 PM
To: daos@daos.groups.io
Subject: Re: [daos] daos_obj_update crashed IO server

 

Hi Jiafu,

 

Could you please email me your test case? I will look into it.

Cheers,

Johann

 

From: <daos@daos.groups.io> on behalf of "Zhang, Jiafu" <jiafu.zhang@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Friday 19 June 2020 at 02:52
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [daos] daos_obj_update crashed IO server

 

Hi Johann,

 

In recent commit (6c3eb801d0a9307cdf88b8634ea5a103d0b3542f), the daos io server crash issue is gone when update object with two akeys under the same dkey.

 

But I got another issue, DER_NO_PERM, no permission. And the update with single akey still worked.

 

By the way, listing dkeys worked. But listing akeys didn’t work in this commit with error invalid parameter. The error in server is,

 

06/19-00:23:45.58 jiafu-dev DAOS[348411/348420] bulk ERR  src/cart/src/cart/crt_bulk.c:106 crt_bulk_desc_valid() invalid parameter of bulk_desc (remote_hdl:0x7f13c8b76400,local_hdl:0x7f13c8b7edf0, bulk_op:104, len: 0.

06/19-00:23:45.58 jiafu-dev DAOS[348411/348420] bulk ERR  src/cart/src/cart/crt_bulk.c:205 crt_bulk_transfer() invalid parameter of bulk_desc.

06/19-00:23:45.58 jiafu-dev DAOS[348411/348420] object ERR  src/object/srv_obj.c:390 obj_bulk_transfer() crt_bulk_transfer 0 error (-1003).

06/19-00:23:45.58 jiafu-dev DAOS[351611/351612] object ERR  src/object/cli_shard.c:856 dc_enumerate_cb() rpc 0x7f79e4aca470 RPC 3 failed: DER_INVAL(-1003)

06/19-00:24:32.05 jiafu-dev DAOS[348411/348420] vos  ERR  src/vos/vos_obj.c:1596 vos_obj_iter_aggregate() Removing orphaned array value tree

06/19-00:26:02.06 jiafu-dev DAOS[348411/348420] vos  ERR  src/vos/vos_obj.c:1596 vos_obj_iter_aggregate() Removing orphaned array value tree

06/19-00:26:02.06 jiafu-dev DAOS[348411/348424] vos  ERR  src/vos/vos_obj.c:1596 vos_obj_iter_aggregate() Removing orphaned array value tree

 

Thanks.

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Zhang, Jiafu
Sent: Thursday, June 18, 2020 6:24 PM
To: daos@daos.groups.io
Subject: Re: [daos] daos_obj_update crashed IO server

 

When I reverted to 6c3eb801d0a9307cdf88b8634ea5a103d0b3542f, the format command worked for me.

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Zhang, Jiafu
Sent: Thursday, June 18, 2020 6:18 PM
To: daos@daos.groups.io
Subject: Re: [daos] daos_obj_update crashed IO server

 

The so issue is fixed by adding DAOS lib paths to ldconfig. It’s new behavior? I only needed to set LD_LIBRARY_PATH last time.

 

Another issue raised up when format new DAOS server.

 

$ dmg -i -l jiafu-dev:10001 storage format --reformat

processing system reformat request

ERROR: dmg: System-Query command failed: failed to get MS response: rpc error: code = Unknown desc = empty system membership

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Zhang, Jiafu
Sent: Thursday, June 18, 2020 4:00 PM
To: daos@daos.groups.io
Subject: Re: [daos] daos_obj_update crashed IO server

 

Hi Johann,

 

My version is old.

 

I tried to upgrade from ubuntu 18.04 to ubuntu 20.04. And built latest DAOS code as of now from master. But I got below error when started daos server with non-root user. I’ve followed “Privileged Helper Configuration” in https://daos-stack.github.io/admin/predeployment_check/.

 

ERROR: /usr/bin/daos_admin /usr/bin/daos_admin: error while loading shared libraries: libspdk_log.so.2.0: cannot open shared object file: No such file or directory

ERROR: pbin: code = 4 description = "Unable to decode response after 5 attempts"

 

When I ldd /usr/bin/daos_admin and ldd libspdk_log.so.2.0, both worked.

 

jiafu@jiafu-dev:~/daos$ ldd /usr/bin/daos_admin

        linux-vdso.so.1 (0x00007ffd9bfeb000)

        /usr/lib/jvm/java-1.8.0-openjdk-amd64/jre/lib/amd64/libjsig.so (0x00007f63ef4b1000)

        libspdk_env_dpdk.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_env_dpdk.so.2.0 (0x00007f63ef4a0000)

        librte_mempool.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_mempool.so.20.0 (0x00007f63ef497000)

        librte_bus_pci.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_bus_pci.so.20.0 (0x00007f63ef487000)

        librte_ring.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_ring.so.20.0 (0x00007f63ef482000)

        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f63ef444000)

        libspdk_nvme.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_nvme.so.2.0 (0x00007f63ef40a000)

        libipmctl.so.4 => /usr/lib/x86_64-linux-gnu/libipmctl.so.4 (0x00007f63ef049000)

        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f63eee57000)

        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f63eee51000)

        /lib64/ld-linux-x86-64.so.2 (0x00007f63ef4b9000)

        librte_eal.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_eal.so.20.0 (0x00007f63eed6e000)

        librte_pci.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_pci.so.20.0 (0x00007f63eed67000)

        librte_kvargs.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_kvargs.so.20.0 (0x00007f63eed62000)

        libibverbs.so.1 => /usr/lib/x86_64-linux-gnu/libibverbs.so.1 (0x00007f63eed43000)

        librdmacm.so.1 => /usr/lib/x86_64-linux-gnu/librdmacm.so.1 (0x00007f63eed24000)

        libspdk_log.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_log.so.2.0 (0x00007f63eed1f000)

        libspdk_sock.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_sock.so.2.0 (0x00007f63eed1a000)

        libspdk_util.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_util.so.2.0 (0x00007f63eed07000)

        libndctl.so.6 => /usr/lib/x86_64-linux-gnu/libndctl.so.6 (0x00007f63eece4000)

        libnuma.so.1 => /usr/lib/x86_64-linux-gnu/libnuma.so.1 (0x00007f63eecd7000)

        libnl-route-3.so.200 => /usr/lib/x86_64-linux-gnu/libnl-route-3.so.200 (0x00007f63eec5f000)

        libnl-3.so.200 => /lib/x86_64-linux-gnu/libnl-3.so.200 (0x00007f63eec3c000)

        libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007f63eec31000)

        libdaxctl.so.1 => /usr/lib/x86_64-linux-gnu/libdaxctl.so.1 (0x00007f63eec27000)

        libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007f63eebfb000)

        libkmod.so.2 => /usr/lib/x86_64-linux-gnu/libkmod.so.2 (0x00007f63eebe0000)

        liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f63eebb9000)

        libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f63ee8e1000)

jiafu@jiafu-dev:~/daos$ cd /code-repo/daos-master/daos/install/lib/

jiafu@jiafu-dev:/code-repo/daos-master/daos/install/lib$ ldd libspdk_log.so.2.0

        linux-vdso.so.1 (0x00007fffa928a000)

        /usr/lib/jvm/java-1.8.0-openjdk-amd64/jre/lib/amd64/libjsig.so (0x00007fb538648000)

        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb53843d000)

        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fb538437000)

        /lib64/ld-linux-x86-64.so.2 (0x00007fb538655000)

 

Please help.

 

Thanks.

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Lombardi, Johann
Sent: Wednesday, June 17, 2020 5:48 AM
To: daos@daos.groups.io
Subject: Re: [daos] daos_obj_update crashed IO server

 

Hm, the patch was merged a while ago. How recent is your version of master?

Cheers,

Johann

 

From: <daos@daos.groups.io> on behalf of "Zhang, Jiafu" <jiafu.zhang@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Tuesday 16 June 2020 at 14:49
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [daos] daos_obj_update crashed IO server

 

1.       I get this ticket, DAOS-4391. But it’s for duplicated akey in single update. I’ll try it anyway.

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Zhang, Jiafu
Sent: Tuesday, June 16, 2020 8:12 PM
To: daos@daos.groups.io
Subject: [daos] daos_obj_update crashed IO server

 

Hi Guys,

 

I can call daos_obj_update successfully with one iod/sgl. But daos_obj_update crashed IO server when there were two iods/sgls under the same dkeys. From the server log, I see below error. Can someone tell me what’s going on here?

 

“dkey1”, “akey1” -> succeeded.

“dkey1”, “akey1”, “akey2” -> failed.

 

 

ERROR: daos_io_server:0 06/16-12:04:43.41 jiafu-dev DAOS[28902/28911] vos  EMRG src/vos/vos_ts.h:332 vos_ts_alloc() Assertion failed (info->ti_type & 1) == 0

ERROR: daos_io_server:0 daos_io_server: src/vos/vos_ts.h:332: vos_ts_alloc: Assertion `(info->ti_type & 1) == 0' failed.

ERROR: daos_io_server:0 *** Process 28902 received signal 6 ***

ERROR: daos_io_server:0 Associated errno: Success (0)

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7fcc895b4890]

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7fcc88b8ae97]

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7fcc88b8c801]

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(+0x3039a)[0x7fcc88b7c39a]

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(+0x30412)[0x7fcc88b7c412]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(vos_ilog_ts_cache+0x34c)[0x7fcc8a24583c]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(key_tree_prepare+0x251)[0x7fcc8a21ea41]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(vos_update_end+0x159c)[0x7fcc8a1f070c]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(+0x1e8e3)[0x7fcc7e1da8e3]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(+0x20945)[0x7fcc7e1dc945]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(ds_obj_rw_handler+0x3a0)[0x7fcc7e1dd470]

ERROR: daos_io_server:0 /usr/local/daos/lib/libcart.so.4(+0xa25ee)[0x7fcc8a5245ee]

ERROR: daos_io_server:0 /usr/local/daos/lib/libabt.so.0(+0x10ea0)[0x7fcc89393ea0]

ERROR: daos_io_server:0 /usr/local/daos/lib/libabt.so.0(+0x11601)[0x7fcc89394601]

 

Thanks.

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Zhang, Jiafu
 

Hi Johann,

 

It turned out to be my bug which failed the object update and key listing. Sorry for the bothering.

 

Thanks.

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Lombardi, Johann
Sent: Friday, June 19, 2020 7:30 PM
To: daos@daos.groups.io
Subject: Re: [daos] daos_obj_update crashed IO server

 

Hi Jiafu,

 

Could you please email me your test case? I will look into it.

Cheers,

Johann

 

From: <daos@daos.groups.io> on behalf of "Zhang, Jiafu" <jiafu.zhang@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Friday 19 June 2020 at 02:52
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [daos] daos_obj_update crashed IO server

 

Hi Johann,

 

In recent commit (6c3eb801d0a9307cdf88b8634ea5a103d0b3542f), the daos io server crash issue is gone when update object with two akeys under the same dkey.

 

But I got another issue, DER_NO_PERM, no permission. And the update with single akey still worked.

 

By the way, listing dkeys worked. But listing akeys didn’t work in this commit with error invalid parameter. The error in server is,

 

06/19-00:23:45.58 jiafu-dev DAOS[348411/348420] bulk ERR  src/cart/src/cart/crt_bulk.c:106 crt_bulk_desc_valid() invalid parameter of bulk_desc (remote_hdl:0x7f13c8b76400,local_hdl:0x7f13c8b7edf0, bulk_op:104, len: 0.

06/19-00:23:45.58 jiafu-dev DAOS[348411/348420] bulk ERR  src/cart/src/cart/crt_bulk.c:205 crt_bulk_transfer() invalid parameter of bulk_desc.

06/19-00:23:45.58 jiafu-dev DAOS[348411/348420] object ERR  src/object/srv_obj.c:390 obj_bulk_transfer() crt_bulk_transfer 0 error (-1003).

06/19-00:23:45.58 jiafu-dev DAOS[351611/351612] object ERR  src/object/cli_shard.c:856 dc_enumerate_cb() rpc 0x7f79e4aca470 RPC 3 failed: DER_INVAL(-1003)

06/19-00:24:32.05 jiafu-dev DAOS[348411/348420] vos  ERR  src/vos/vos_obj.c:1596 vos_obj_iter_aggregate() Removing orphaned array value tree

06/19-00:26:02.06 jiafu-dev DAOS[348411/348420] vos  ERR  src/vos/vos_obj.c:1596 vos_obj_iter_aggregate() Removing orphaned array value tree

06/19-00:26:02.06 jiafu-dev DAOS[348411/348424] vos  ERR  src/vos/vos_obj.c:1596 vos_obj_iter_aggregate() Removing orphaned array value tree

 

Thanks.

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Zhang, Jiafu
Sent: Thursday, June 18, 2020 6:24 PM
To: daos@daos.groups.io
Subject: Re: [daos] daos_obj_update crashed IO server

 

When I reverted to 6c3eb801d0a9307cdf88b8634ea5a103d0b3542f, the format command worked for me.

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Zhang, Jiafu
Sent: Thursday, June 18, 2020 6:18 PM
To: daos@daos.groups.io
Subject: Re: [daos] daos_obj_update crashed IO server

 

The so issue is fixed by adding DAOS lib paths to ldconfig. It’s new behavior? I only needed to set LD_LIBRARY_PATH last time.

 

Another issue raised up when format new DAOS server.

 

$ dmg -i -l jiafu-dev:10001 storage format --reformat

processing system reformat request

ERROR: dmg: System-Query command failed: failed to get MS response: rpc error: code = Unknown desc = empty system membership

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Zhang, Jiafu
Sent: Thursday, June 18, 2020 4:00 PM
To: daos@daos.groups.io
Subject: Re: [daos] daos_obj_update crashed IO server

 

Hi Johann,

 

My version is old.

 

I tried to upgrade from ubuntu 18.04 to ubuntu 20.04. And built latest DAOS code as of now from master. But I got below error when started daos server with non-root user. I’ve followed “Privileged Helper Configuration” in https://daos-stack.github.io/admin/predeployment_check/.

 

ERROR: /usr/bin/daos_admin /usr/bin/daos_admin: error while loading shared libraries: libspdk_log.so.2.0: cannot open shared object file: No such file or directory

ERROR: pbin: code = 4 description = "Unable to decode response after 5 attempts"

 

When I ldd /usr/bin/daos_admin and ldd libspdk_log.so.2.0, both worked.

 

jiafu@jiafu-dev:~/daos$ ldd /usr/bin/daos_admin

        linux-vdso.so.1 (0x00007ffd9bfeb000)

        /usr/lib/jvm/java-1.8.0-openjdk-amd64/jre/lib/amd64/libjsig.so (0x00007f63ef4b1000)

        libspdk_env_dpdk.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_env_dpdk.so.2.0 (0x00007f63ef4a0000)

        librte_mempool.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_mempool.so.20.0 (0x00007f63ef497000)

        librte_bus_pci.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_bus_pci.so.20.0 (0x00007f63ef487000)

        librte_ring.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_ring.so.20.0 (0x00007f63ef482000)

        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f63ef444000)

        libspdk_nvme.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_nvme.so.2.0 (0x00007f63ef40a000)

        libipmctl.so.4 => /usr/lib/x86_64-linux-gnu/libipmctl.so.4 (0x00007f63ef049000)

        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f63eee57000)

        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f63eee51000)

        /lib64/ld-linux-x86-64.so.2 (0x00007f63ef4b9000)

        librte_eal.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_eal.so.20.0 (0x00007f63eed6e000)

        librte_pci.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_pci.so.20.0 (0x00007f63eed67000)

        librte_kvargs.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_kvargs.so.20.0 (0x00007f63eed62000)

        libibverbs.so.1 => /usr/lib/x86_64-linux-gnu/libibverbs.so.1 (0x00007f63eed43000)

        librdmacm.so.1 => /usr/lib/x86_64-linux-gnu/librdmacm.so.1 (0x00007f63eed24000)

        libspdk_log.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_log.so.2.0 (0x00007f63eed1f000)

        libspdk_sock.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_sock.so.2.0 (0x00007f63eed1a000)

        libspdk_util.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_util.so.2.0 (0x00007f63eed07000)

        libndctl.so.6 => /usr/lib/x86_64-linux-gnu/libndctl.so.6 (0x00007f63eece4000)

        libnuma.so.1 => /usr/lib/x86_64-linux-gnu/libnuma.so.1 (0x00007f63eecd7000)

        libnl-route-3.so.200 => /usr/lib/x86_64-linux-gnu/libnl-route-3.so.200 (0x00007f63eec5f000)

        libnl-3.so.200 => /lib/x86_64-linux-gnu/libnl-3.so.200 (0x00007f63eec3c000)

        libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007f63eec31000)

        libdaxctl.so.1 => /usr/lib/x86_64-linux-gnu/libdaxctl.so.1 (0x00007f63eec27000)

        libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007f63eebfb000)

        libkmod.so.2 => /usr/lib/x86_64-linux-gnu/libkmod.so.2 (0x00007f63eebe0000)

        liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f63eebb9000)

        libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f63ee8e1000)

jiafu@jiafu-dev:~/daos$ cd /code-repo/daos-master/daos/install/lib/

jiafu@jiafu-dev:/code-repo/daos-master/daos/install/lib$ ldd libspdk_log.so.2.0

        linux-vdso.so.1 (0x00007fffa928a000)

        /usr/lib/jvm/java-1.8.0-openjdk-amd64/jre/lib/amd64/libjsig.so (0x00007fb538648000)

        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb53843d000)

        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fb538437000)

        /lib64/ld-linux-x86-64.so.2 (0x00007fb538655000)

 

Please help.

 

Thanks.

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Lombardi, Johann
Sent: Wednesday, June 17, 2020 5:48 AM
To: daos@daos.groups.io
Subject: Re: [daos] daos_obj_update crashed IO server

 

Hm, the patch was merged a while ago. How recent is your version of master?

Cheers,

Johann

 

From: <daos@daos.groups.io> on behalf of "Zhang, Jiafu" <jiafu.zhang@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Tuesday 16 June 2020 at 14:49
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [daos] daos_obj_update crashed IO server

 

1.       I get this ticket, DAOS-4391. But it’s for duplicated akey in single update. I’ll try it anyway.

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Zhang, Jiafu
Sent: Tuesday, June 16, 2020 8:12 PM
To: daos@daos.groups.io
Subject: [daos] daos_obj_update crashed IO server

 

Hi Guys,

 

I can call daos_obj_update successfully with one iod/sgl. But daos_obj_update crashed IO server when there were two iods/sgls under the same dkeys. From the server log, I see below error. Can someone tell me what’s going on here?

 

“dkey1”, “akey1” -> succeeded.

“dkey1”, “akey1”, “akey2” -> failed.

 

 

ERROR: daos_io_server:0 06/16-12:04:43.41 jiafu-dev DAOS[28902/28911] vos  EMRG src/vos/vos_ts.h:332 vos_ts_alloc() Assertion failed (info->ti_type & 1) == 0

ERROR: daos_io_server:0 daos_io_server: src/vos/vos_ts.h:332: vos_ts_alloc: Assertion `(info->ti_type & 1) == 0' failed.

ERROR: daos_io_server:0 *** Process 28902 received signal 6 ***

ERROR: daos_io_server:0 Associated errno: Success (0)

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7fcc895b4890]

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7fcc88b8ae97]

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7fcc88b8c801]

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(+0x3039a)[0x7fcc88b7c39a]

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(+0x30412)[0x7fcc88b7c412]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(vos_ilog_ts_cache+0x34c)[0x7fcc8a24583c]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(key_tree_prepare+0x251)[0x7fcc8a21ea41]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(vos_update_end+0x159c)[0x7fcc8a1f070c]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(+0x1e8e3)[0x7fcc7e1da8e3]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(+0x20945)[0x7fcc7e1dc945]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(ds_obj_rw_handler+0x3a0)[0x7fcc7e1dd470]

ERROR: daos_io_server:0 /usr/local/daos/lib/libcart.so.4(+0xa25ee)[0x7fcc8a5245ee]

ERROR: daos_io_server:0 /usr/local/daos/lib/libabt.so.0(+0x10ea0)[0x7fcc89393ea0]

ERROR: daos_io_server:0 /usr/local/daos/lib/libabt.so.0(+0x11601)[0x7fcc89394601]

 

Thanks.

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Lombardi, Johann
 

Np. Thanks for letting us know.

 

Johann

 

From: <daos@daos.groups.io> on behalf of "Zhang, Jiafu" <jiafu.zhang@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Tuesday 23 June 2020 at 05:54
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [daos] daos_obj_update crashed IO server

 

Hi Johann,

 

It turned out to be my bug which failed the object update and key listing. Sorry for the bothering.

 

Thanks.

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.