daos@daos.groups.io | daos_obj_update crashed IO server

Home Messages Hashtags Subgroups

Date Date 1 - 11 of 11

daos_obj_update crashed IO server

Zhang, Jiafu #1047 Hi Guys, I can call daos_obj_update successfully with one iod/sgl. But daos_obj_update crashed IO server when there were two iods/sgls under the same dkeys. From the server log, I see below error. Can someone tell me what’s going on here? “dkey1”, “akey1” -> succeeded. “dkey1”, “akey1”, “akey2” -> failed. ERROR: daos_io_server:0 06/16-12:04:43.41 jiafu-dev DAOS[28902/28911] vos EMRG src/vos/vos_ts.h:332 vos_ts_alloc() Assertion failed (info->ti_type & 1) == 0 ERROR: daos_io_server:0 daos_io_server: src/vos/vos_ts.h:332: vos_ts_alloc: Assertion `(info->ti_type & 1) == 0' failed. ERROR: daos_io_server:0 * Process 28902 received signal 6 * ERROR: daos_io_server:0 Associated errno: Success (0) ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7fcc895b4890] ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7fcc88b8ae97] ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7fcc88b8c801] ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(+0x3039a)[0x7fcc88b7c39a] ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(+0x30412)[0x7fcc88b7c412] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(vos_ilog_ts_cache+0x34c)[0x7fcc8a24583c] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(key_tree_prepare+0x251)[0x7fcc8a21ea41] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(vos_update_end+0x159c)[0x7fcc8a1f070c] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(+0x1e8e3)[0x7fcc7e1da8e3] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(+0x20945)[0x7fcc7e1dc945] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(ds_obj_rw_handler+0x3a0)[0x7fcc7e1dd470] ERROR: daos_io_server:0 /usr/local/daos/lib/libcart.so.4(+0xa25ee)[0x7fcc8a5245ee] ERROR: daos_io_server:0 /usr/local/daos/lib/libabt.so.0(+0x10ea0)[0x7fcc89393ea0] ERROR: daos_io_server:0 /usr/local/daos/lib/libabt.so.0(+0x11601)[0x7fcc89394601] Thanks.
More All Messages By This Member
Zhang, Jiafu #1050 1. I get this ticket, DAOS-4391. But it’s for duplicated akey in single update. I’ll try it anyway. toggle quoted message Show quoted text From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Zhang, Jiafu Sent: Tuesday, June 16, 2020 8:12 PM To: daos@daos.groups.io Subject: [daos] daos_obj_update crashed IO server Hi Guys, I can call daos_obj_update successfully with one iod/sgl. But daos_obj_update crashed IO server when there were two iods/sgls under the same dkeys. From the server log, I see below error. Can someone tell me what’s going on here? “dkey1”, “akey1” -> succeeded. “dkey1”, “akey1”, “akey2” -> failed. ERROR: daos_io_server:0 06/16-12:04:43.41 jiafu-dev DAOS[28902/28911] vos EMRG src/vos/vos_ts.h:332 vos_ts_alloc() Assertion failed (info->ti_type & 1) == 0 ERROR: daos_io_server:0 daos_io_server: src/vos/vos_ts.h:332: vos_ts_alloc: Assertion `(info->ti_type & 1) == 0' failed. ERROR: daos_io_server:0 * Process 28902 received signal 6 * ERROR: daos_io_server:0 Associated errno: Success (0) ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7fcc895b4890] ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7fcc88b8ae97] ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7fcc88b8c801] ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(+0x3039a)[0x7fcc88b7c39a] ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(+0x30412)[0x7fcc88b7c412] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(vos_ilog_ts_cache+0x34c)[0x7fcc8a24583c] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(key_tree_prepare+0x251)[0x7fcc8a21ea41] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(vos_update_end+0x159c)[0x7fcc8a1f070c] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(+0x1e8e3)[0x7fcc7e1da8e3] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(+0x20945)[0x7fcc7e1dc945] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(ds_obj_rw_handler+0x3a0)[0x7fcc7e1dd470] ERROR: daos_io_server:0 /usr/local/daos/lib/libcart.so.4(+0xa25ee)[0x7fcc8a5245ee] ERROR: daos_io_server:0 /usr/local/daos/lib/libabt.so.0(+0x10ea0)[0x7fcc89393ea0] ERROR: daos_io_server:0 /usr/local/daos/lib/libabt.so.0(+0x11601)[0x7fcc89394601] Thanks.
More All Messages By This Member
Lombardi, Johann #1051 Hm, the patch was merged a while ago. How recent is your version of master? Cheers, Johann From: <daos@daos.groups.io> on behalf of "Zhang, Jiafu" <jiafu.zhang@...> Reply-To: "daos@daos.groups.io" <daos@daos.groups.io> Date: Tuesday 16 June 2020 at 14:49 To: "daos@daos.groups.io" <daos@daos.groups.io> Subject: Re: [daos] daos_obj_update crashed IO server 1. I get this ticket, DAOS-4391. But it’s for duplicated akey in single update. I’ll try it anyway. toggle quoted message Show quoted text From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Zhang, Jiafu Sent: Tuesday, June 16, 2020 8:12 PM To: daos@daos.groups.io Subject: [daos] daos_obj_update crashed IO server Hi Guys, I can call daos_obj_update successfully with one iod/sgl. But daos_obj_update crashed IO server when there were two iods/sgls under the same dkeys. From the server log, I see below error. Can someone tell me what’s going on here? “dkey1”, “akey1” -> succeeded. “dkey1”, “akey1”, “akey2” -> failed. ERROR: daos_io_server:0 06/16-12:04:43.41 jiafu-dev DAOS[28902/28911] vos EMRG src/vos/vos_ts.h:332 vos_ts_alloc() Assertion failed (info->ti_type & 1) == 0 ERROR: daos_io_server:0 daos_io_server: src/vos/vos_ts.h:332: vos_ts_alloc: Assertion `(info->ti_type & 1) == 0' failed. ERROR: daos_io_server:0 * Process 28902 received signal 6 * ERROR: daos_io_server:0 Associated errno: Success (0) ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7fcc895b4890] ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7fcc88b8ae97] ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7fcc88b8c801] ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(+0x3039a)[0x7fcc88b7c39a] ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(+0x30412)[0x7fcc88b7c412] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(vos_ilog_ts_cache+0x34c)[0x7fcc8a24583c] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(key_tree_prepare+0x251)[0x7fcc8a21ea41] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(vos_update_end+0x159c)[0x7fcc8a1f070c] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(+0x1e8e3)[0x7fcc7e1da8e3] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(+0x20945)[0x7fcc7e1dc945] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(ds_obj_rw_handler+0x3a0)[0x7fcc7e1dd470] ERROR: daos_io_server:0 /usr/local/daos/lib/libcart.so.4(+0xa25ee)[0x7fcc8a5245ee] ERROR: daos_io_server:0 /usr/local/daos/lib/libabt.so.0(+0x10ea0)[0x7fcc89393ea0] ERROR: daos_io_server:0 /usr/local/daos/lib/libabt.so.0(+0x11601)[0x7fcc89394601] Thanks. --------------------------------------------------------------------- Intel Corporation SAS (French simplified joint stock company) Registered headquarters: "Les Montalets"- 2, rue de Paris, 92196 Meudon Cedex, France Registration Number: 302 456 199 R.C.S. NANTERRE Capital: 4,572,000 Euros This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
More All Messages By This Member
Zhang, Jiafu #1055 Hi Johann, My version is old. I tried to upgrade from ubuntu 18.04 to ubuntu 20.04. And built latest DAOS code as of now from master. But I got below error when started daos server with non-root user. I’ve followed “Privileged Helper Configuration” in https://daos-stack.github.io/admin/predeployment_check/. ERROR: /usr/bin/daos_admin /usr/bin/daos_admin: error while loading shared libraries: libspdk_log.so.2.0: cannot open shared object file: No such file or directory ERROR: pbin: code = 4 description = "Unable to decode response after 5 attempts" When I ldd /usr/bin/daos_admin and ldd libspdk_log.so.2.0, both worked. jiafu@jiafu-dev:~/daos$ ldd /usr/bin/daos_admin linux-vdso.so.1 (0x00007ffd9bfeb000) /usr/lib/jvm/java-1.8.0-openjdk-amd64/jre/lib/amd64/libjsig.so (0x00007f63ef4b1000) libspdk_env_dpdk.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_env_dpdk.so.2.0 (0x00007f63ef4a0000) librte_mempool.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_mempool.so.20.0 (0x00007f63ef497000) librte_bus_pci.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_bus_pci.so.20.0 (0x00007f63ef487000) librte_ring.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_ring.so.20.0 (0x00007f63ef482000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f63ef444000) libspdk_nvme.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_nvme.so.2.0 (0x00007f63ef40a000) libipmctl.so.4 => /usr/lib/x86_64-linux-gnu/libipmctl.so.4 (0x00007f63ef049000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f63eee57000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f63eee51000) /lib64/ld-linux-x86-64.so.2 (0x00007f63ef4b9000) librte_eal.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_eal.so.20.0 (0x00007f63eed6e000) librte_pci.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_pci.so.20.0 (0x00007f63eed67000) librte_kvargs.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_kvargs.so.20.0 (0x00007f63eed62000) libibverbs.so.1 => /usr/lib/x86_64-linux-gnu/libibverbs.so.1 (0x00007f63eed43000) librdmacm.so.1 => /usr/lib/x86_64-linux-gnu/librdmacm.so.1 (0x00007f63eed24000) libspdk_log.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_log.so.2.0 (0x00007f63eed1f000) libspdk_sock.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_sock.so.2.0 (0x00007f63eed1a000) libspdk_util.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_util.so.2.0 (0x00007f63eed07000) libndctl.so.6 => /usr/lib/x86_64-linux-gnu/libndctl.so.6 (0x00007f63eece4000) libnuma.so.1 => /usr/lib/x86_64-linux-gnu/libnuma.so.1 (0x00007f63eecd7000) libnl-route-3.so.200 => /usr/lib/x86_64-linux-gnu/libnl-route-3.so.200 (0x00007f63eec5f000) libnl-3.so.200 => /lib/x86_64-linux-gnu/libnl-3.so.200 (0x00007f63eec3c000) libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007f63eec31000) libdaxctl.so.1 => /usr/lib/x86_64-linux-gnu/libdaxctl.so.1 (0x00007f63eec27000) libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007f63eebfb000) libkmod.so.2 => /usr/lib/x86_64-linux-gnu/libkmod.so.2 (0x00007f63eebe0000) liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f63eebb9000) libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f63ee8e1000) jiafu@jiafu-dev:~/daos$ cd /code-repo/daos-master/daos/install/lib/ jiafu@jiafu-dev:/code-repo/daos-master/daos/install/lib$ ldd libspdk_log.so.2.0 linux-vdso.so.1 (0x00007fffa928a000) /usr/lib/jvm/java-1.8.0-openjdk-amd64/jre/lib/amd64/libjsig.so (0x00007fb538648000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb53843d000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fb538437000) /lib64/ld-linux-x86-64.so.2 (0x00007fb538655000) Please help. Thanks. toggle quoted message Show quoted text From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Lombardi, Johann Sent: Wednesday, June 17, 2020 5:48 AM To: daos@daos.groups.io Subject: Re: [daos] daos_obj_update crashed IO server Hm, the patch was merged a while ago. How recent is your version of master? Cheers, Johann From: <daos@daos.groups.io> on behalf of "Zhang, Jiafu" <jiafu.zhang@...> Reply-To: "daos@daos.groups.io" <daos@daos.groups.io> Date: Tuesday 16 June 2020 at 14:49 To: "daos@daos.groups.io" <daos@daos.groups.io> Subject: Re: [daos] daos_obj_update crashed IO server I get this ticket, DAOS-4391. But it’s for duplicated akey in single update. I’ll try it anyway. From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Zhang, Jiafu Sent: Tuesday, June 16, 2020 8:12 PM To: daos@daos.groups.io Subject: [daos] daos_obj_update crashed IO server Hi Guys, I can call daos_obj_update successfully with one iod/sgl. But daos_obj_update crashed IO server when there were two iods/sgls under the same dkeys. From the server log, I see below error. Can someone tell me what’s going on here? “dkey1”, “akey1” -> succeeded. “dkey1”, “akey1”, “akey2” -> failed. ERROR: daos_io_server:0 06/16-12:04:43.41 jiafu-dev DAOS[28902/28911] vos EMRG src/vos/vos_ts.h:332 vos_ts_alloc() Assertion failed (info->ti_type & 1) == 0 ERROR: daos_io_server:0 daos_io_server: src/vos/vos_ts.h:332: vos_ts_alloc: Assertion `(info->ti_type & 1) == 0' failed. ERROR: daos_io_server:0 * Process 28902 received signal 6 * ERROR: daos_io_server:0 Associated errno: Success (0) ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7fcc895b4890] ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7fcc88b8ae97] ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7fcc88b8c801] ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(+0x3039a)[0x7fcc88b7c39a] ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(+0x30412)[0x7fcc88b7c412] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(vos_ilog_ts_cache+0x34c)[0x7fcc8a24583c] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(key_tree_prepare+0x251)[0x7fcc8a21ea41] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(vos_update_end+0x159c)[0x7fcc8a1f070c] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(+0x1e8e3)[0x7fcc7e1da8e3] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(+0x20945)[0x7fcc7e1dc945] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(ds_obj_rw_handler+0x3a0)[0x7fcc7e1dd470] ERROR: daos_io_server:0 /usr/local/daos/lib/libcart.so.4(+0xa25ee)[0x7fcc8a5245ee] ERROR: daos_io_server:0 /usr/local/daos/lib/libabt.so.0(+0x10ea0)[0x7fcc89393ea0] ERROR: daos_io_server:0 /usr/local/daos/lib/libabt.so.0(+0x11601)[0x7fcc89394601] Thanks. --------------------------------------------------------------------- Intel Corporation SAS (French simplified joint stock company) Registered headquarters: "Les Montalets"- 2, rue de Paris, 92196 Meudon Cedex, France Registration Number: 302 456 199 R.C.S. NANTERRE Capital: 4,572,000 Euros This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
More All Messages By This Member
Zhang, Jiafu #1056 The so issue is fixed by adding DAOS lib paths to ldconfig. It’s new behavior? I only needed to set LD_LIBRARY_PATH last time. Another issue raised up when format new DAOS server. $ dmg -i -l jiafu-dev:10001 storage format --reformat processing system reformat request ERROR: dmg: System-Query command failed: failed to get MS response: rpc error: code = Unknown desc = empty system membership toggle quoted message Show quoted text From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Zhang, Jiafu Sent: Thursday, June 18, 2020 4:00 PM To: daos@daos.groups.io Subject: Re: [daos] daos_obj_update crashed IO server Hi Johann, My version is old. I tried to upgrade from ubuntu 18.04 to ubuntu 20.04. And built latest DAOS code as of now from master. But I got below error when started daos server with non-root user. I’ve followed “Privileged Helper Configuration” in https://daos-stack.github.io/admin/predeployment_check/. ERROR: /usr/bin/daos_admin /usr/bin/daos_admin: error while loading shared libraries: libspdk_log.so.2.0: cannot open shared object file: No such file or directory ERROR: pbin: code = 4 description = "Unable to decode response after 5 attempts" When I ldd /usr/bin/daos_admin and ldd libspdk_log.so.2.0, both worked. jiafu@jiafu-dev:~/daos$ ldd /usr/bin/daos_admin linux-vdso.so.1 (0x00007ffd9bfeb000) /usr/lib/jvm/java-1.8.0-openjdk-amd64/jre/lib/amd64/libjsig.so (0x00007f63ef4b1000) libspdk_env_dpdk.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_env_dpdk.so.2.0 (0x00007f63ef4a0000) librte_mempool.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_mempool.so.20.0 (0x00007f63ef497000) librte_bus_pci.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_bus_pci.so.20.0 (0x00007f63ef487000) librte_ring.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_ring.so.20.0 (0x00007f63ef482000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f63ef444000) libspdk_nvme.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_nvme.so.2.0 (0x00007f63ef40a000) libipmctl.so.4 => /usr/lib/x86_64-linux-gnu/libipmctl.so.4 (0x00007f63ef049000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f63eee57000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f63eee51000) /lib64/ld-linux-x86-64.so.2 (0x00007f63ef4b9000) librte_eal.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_eal.so.20.0 (0x00007f63eed6e000) librte_pci.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_pci.so.20.0 (0x00007f63eed67000) librte_kvargs.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_kvargs.so.20.0 (0x00007f63eed62000) libibverbs.so.1 => /usr/lib/x86_64-linux-gnu/libibverbs.so.1 (0x00007f63eed43000) librdmacm.so.1 => /usr/lib/x86_64-linux-gnu/librdmacm.so.1 (0x00007f63eed24000) libspdk_log.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_log.so.2.0 (0x00007f63eed1f000) libspdk_sock.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_sock.so.2.0 (0x00007f63eed1a000) libspdk_util.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_util.so.2.0 (0x00007f63eed07000) libndctl.so.6 => /usr/lib/x86_64-linux-gnu/libndctl.so.6 (0x00007f63eece4000) libnuma.so.1 => /usr/lib/x86_64-linux-gnu/libnuma.so.1 (0x00007f63eecd7000) libnl-route-3.so.200 => /usr/lib/x86_64-linux-gnu/libnl-route-3.so.200 (0x00007f63eec5f000) libnl-3.so.200 => /lib/x86_64-linux-gnu/libnl-3.so.200 (0x00007f63eec3c000) libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007f63eec31000) libdaxctl.so.1 => /usr/lib/x86_64-linux-gnu/libdaxctl.so.1 (0x00007f63eec27000) libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007f63eebfb000) libkmod.so.2 => /usr/lib/x86_64-linux-gnu/libkmod.so.2 (0x00007f63eebe0000) liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f63eebb9000) libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f63ee8e1000) jiafu@jiafu-dev:~/daos$ cd /code-repo/daos-master/daos/install/lib/ jiafu@jiafu-dev:/code-repo/daos-master/daos/install/lib$ ldd libspdk_log.so.2.0 linux-vdso.so.1 (0x00007fffa928a000) /usr/lib/jvm/java-1.8.0-openjdk-amd64/jre/lib/amd64/libjsig.so (0x00007fb538648000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb53843d000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fb538437000) /lib64/ld-linux-x86-64.so.2 (0x00007fb538655000) Please help. Thanks. From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Lombardi, Johann Sent: Wednesday, June 17, 2020 5:48 AM To: daos@daos.groups.io Subject: Re: [daos] daos_obj_update crashed IO server Hm, the patch was merged a while ago. How recent is your version of master? Cheers, Johann From: <daos@daos.groups.io> on behalf of "Zhang, Jiafu" <jiafu.zhang@...> Reply-To: "daos@daos.groups.io" <daos@daos.groups.io> Date: Tuesday 16 June 2020 at 14:49 To: "daos@daos.groups.io" <daos@daos.groups.io> Subject: Re: [daos] daos_obj_update crashed IO server I get this ticket, DAOS-4391. But it’s for duplicated akey in single update. I’ll try it anyway. From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Zhang, Jiafu Sent: Tuesday, June 16, 2020 8:12 PM To: daos@daos.groups.io Subject: [daos] daos_obj_update crashed IO server Hi Guys, I can call daos_obj_update successfully with one iod/sgl. But daos_obj_update crashed IO server when there were two iods/sgls under the same dkeys. From the server log, I see below error. Can someone tell me what’s going on here? “dkey1”, “akey1” -> succeeded. “dkey1”, “akey1”, “akey2” -> failed. ERROR: daos_io_server:0 06/16-12:04:43.41 jiafu-dev DAOS[28902/28911] vos EMRG src/vos/vos_ts.h:332 vos_ts_alloc() Assertion failed (info->ti_type & 1) == 0 ERROR: daos_io_server:0 daos_io_server: src/vos/vos_ts.h:332: vos_ts_alloc: Assertion `(info->ti_type & 1) == 0' failed. ERROR: daos_io_server:0 * Process 28902 received signal 6 * ERROR: daos_io_server:0 Associated errno: Success (0) ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7fcc895b4890] ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7fcc88b8ae97] ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7fcc88b8c801] ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(+0x3039a)[0x7fcc88b7c39a] ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(+0x30412)[0x7fcc88b7c412] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(vos_ilog_ts_cache+0x34c)[0x7fcc8a24583c] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(key_tree_prepare+0x251)[0x7fcc8a21ea41] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(vos_update_end+0x159c)[0x7fcc8a1f070c] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(+0x1e8e3)[0x7fcc7e1da8e3] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(+0x20945)[0x7fcc7e1dc945] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(ds_obj_rw_handler+0x3a0)[0x7fcc7e1dd470] ERROR: daos_io_server:0 /usr/local/daos/lib/libcart.so.4(+0xa25ee)[0x7fcc8a5245ee] ERROR: daos_io_server:0 /usr/local/daos/lib/libabt.so.0(+0x10ea0)[0x7fcc89393ea0] ERROR: daos_io_server:0 /usr/local/daos/lib/libabt.so.0(+0x11601)[0x7fcc89394601] Thanks. --------------------------------------------------------------------- Intel Corporation SAS (French simplified joint stock company) Registered headquarters: "Les Montalets"- 2, rue de Paris, 92196 Meudon Cedex, France Registration Number: 302 456 199 R.C.S. NANTERRE Capital: 4,572,000 Euros This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
More All Messages By This Member
Zhang, Jiafu #1057 When I reverted to 6c3eb801d0a9307cdf88b8634ea5a103d0b3542f, the format command worked for me. toggle quoted message Show quoted text From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Zhang, Jiafu Sent: Thursday, June 18, 2020 6:18 PM To: daos@daos.groups.io Subject: Re: [daos] daos_obj_update crashed IO server The so issue is fixed by adding DAOS lib paths to ldconfig. It’s new behavior? I only needed to set LD_LIBRARY_PATH last time. Another issue raised up when format new DAOS server. $ dmg -i -l jiafu-dev:10001 storage format --reformat processing system reformat request ERROR: dmg: System-Query command failed: failed to get MS response: rpc error: code = Unknown desc = empty system membership From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Zhang, Jiafu Sent: Thursday, June 18, 2020 4:00 PM To: daos@daos.groups.io Subject: Re: [daos] daos_obj_update crashed IO server Hi Johann, My version is old. I tried to upgrade from ubuntu 18.04 to ubuntu 20.04. And built latest DAOS code as of now from master. But I got below error when started daos server with non-root user. I’ve followed “Privileged Helper Configuration” in https://daos-stack.github.io/admin/predeployment_check/. ERROR: /usr/bin/daos_admin /usr/bin/daos_admin: error while loading shared libraries: libspdk_log.so.2.0: cannot open shared object file: No such file or directory ERROR: pbin: code = 4 description = "Unable to decode response after 5 attempts" When I ldd /usr/bin/daos_admin and ldd libspdk_log.so.2.0, both worked. jiafu@jiafu-dev:~/daos$ ldd /usr/bin/daos_admin linux-vdso.so.1 (0x00007ffd9bfeb000) /usr/lib/jvm/java-1.8.0-openjdk-amd64/jre/lib/amd64/libjsig.so (0x00007f63ef4b1000) libspdk_env_dpdk.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_env_dpdk.so.2.0 (0x00007f63ef4a0000) librte_mempool.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_mempool.so.20.0 (0x00007f63ef497000) librte_bus_pci.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_bus_pci.so.20.0 (0x00007f63ef487000) librte_ring.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_ring.so.20.0 (0x00007f63ef482000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f63ef444000) libspdk_nvme.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_nvme.so.2.0 (0x00007f63ef40a000) libipmctl.so.4 => /usr/lib/x86_64-linux-gnu/libipmctl.so.4 (0x00007f63ef049000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f63eee57000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f63eee51000) /lib64/ld-linux-x86-64.so.2 (0x00007f63ef4b9000) librte_eal.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_eal.so.20.0 (0x00007f63eed6e000) librte_pci.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_pci.so.20.0 (0x00007f63eed67000) librte_kvargs.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_kvargs.so.20.0 (0x00007f63eed62000) libibverbs.so.1 => /usr/lib/x86_64-linux-gnu/libibverbs.so.1 (0x00007f63eed43000) librdmacm.so.1 => /usr/lib/x86_64-linux-gnu/librdmacm.so.1 (0x00007f63eed24000) libspdk_log.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_log.so.2.0 (0x00007f63eed1f000) libspdk_sock.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_sock.so.2.0 (0x00007f63eed1a000) libspdk_util.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_util.so.2.0 (0x00007f63eed07000) libndctl.so.6 => /usr/lib/x86_64-linux-gnu/libndctl.so.6 (0x00007f63eece4000) libnuma.so.1 => /usr/lib/x86_64-linux-gnu/libnuma.so.1 (0x00007f63eecd7000) libnl-route-3.so.200 => /usr/lib/x86_64-linux-gnu/libnl-route-3.so.200 (0x00007f63eec5f000) libnl-3.so.200 => /lib/x86_64-linux-gnu/libnl-3.so.200 (0x00007f63eec3c000) libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007f63eec31000) libdaxctl.so.1 => /usr/lib/x86_64-linux-gnu/libdaxctl.so.1 (0x00007f63eec27000) libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007f63eebfb000) libkmod.so.2 => /usr/lib/x86_64-linux-gnu/libkmod.so.2 (0x00007f63eebe0000) liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f63eebb9000) libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f63ee8e1000) jiafu@jiafu-dev:~/daos$ cd /code-repo/daos-master/daos/install/lib/ jiafu@jiafu-dev:/code-repo/daos-master/daos/install/lib$ ldd libspdk_log.so.2.0 linux-vdso.so.1 (0x00007fffa928a000) /usr/lib/jvm/java-1.8.0-openjdk-amd64/jre/lib/amd64/libjsig.so (0x00007fb538648000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb53843d000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fb538437000) /lib64/ld-linux-x86-64.so.2 (0x00007fb538655000) Please help. Thanks. From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Lombardi, Johann Sent: Wednesday, June 17, 2020 5:48 AM To: daos@daos.groups.io Subject: Re: [daos] daos_obj_update crashed IO server Hm, the patch was merged a while ago. How recent is your version of master? Cheers, Johann From: <daos@daos.groups.io> on behalf of "Zhang, Jiafu" <jiafu.zhang@...> Reply-To: "daos@daos.groups.io" <daos@daos.groups.io> Date: Tuesday 16 June 2020 at 14:49 To: "daos@daos.groups.io" <daos@daos.groups.io> Subject: Re: [daos] daos_obj_update crashed IO server I get this ticket, DAOS-4391. But it’s for duplicated akey in single update. I’ll try it anyway. From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Zhang, Jiafu Sent: Tuesday, June 16, 2020 8:12 PM To: daos@daos.groups.io Subject: [daos] daos_obj_update crashed IO server Hi Guys, I can call daos_obj_update successfully with one iod/sgl. But daos_obj_update crashed IO server when there were two iods/sgls under the same dkeys. From the server log, I see below error. Can someone tell me what’s going on here? “dkey1”, “akey1” -> succeeded. “dkey1”, “akey1”, “akey2” -> failed. ERROR: daos_io_server:0 06/16-12:04:43.41 jiafu-dev DAOS[28902/28911] vos EMRG src/vos/vos_ts.h:332 vos_ts_alloc() Assertion failed (info->ti_type & 1) == 0 ERROR: daos_io_server:0 daos_io_server: src/vos/vos_ts.h:332: vos_ts_alloc: Assertion `(info->ti_type & 1) == 0' failed. ERROR: daos_io_server:0 * Process 28902 received signal 6 * ERROR: daos_io_server:0 Associated errno: Success (0) ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7fcc895b4890] ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7fcc88b8ae97] ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7fcc88b8c801] ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(+0x3039a)[0x7fcc88b7c39a] ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(+0x30412)[0x7fcc88b7c412] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(vos_ilog_ts_cache+0x34c)[0x7fcc8a24583c] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(key_tree_prepare+0x251)[0x7fcc8a21ea41] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(vos_update_end+0x159c)[0x7fcc8a1f070c] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(+0x1e8e3)[0x7fcc7e1da8e3] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(+0x20945)[0x7fcc7e1dc945] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(ds_obj_rw_handler+0x3a0)[0x7fcc7e1dd470] ERROR: daos_io_server:0 /usr/local/daos/lib/libcart.so.4(+0xa25ee)[0x7fcc8a5245ee] ERROR: daos_io_server:0 /usr/local/daos/lib/libabt.so.0(+0x10ea0)[0x7fcc89393ea0] ERROR: daos_io_server:0 /usr/local/daos/lib/libabt.so.0(+0x11601)[0x7fcc89394601] Thanks. --------------------------------------------------------------------- Intel Corporation SAS (French simplified joint stock company) Registered headquarters: "Les Montalets"- 2, rue de Paris, 92196 Meudon Cedex, France Registration Number: 302 456 199 R.C.S. NANTERRE Capital: 4,572,000 Euros This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
More All Messages By This Member
Zhang, Jiafu #1060 Hi Johann, In recent commit (6c3eb801d0a9307cdf88b8634ea5a103d0b3542f), the daos io server crash issue is gone when update object with two akeys under the same dkey. But I got another issue, DER_NO_PERM, no permission. And the update with single akey still worked. By the way, listing dkeys worked. But listing akeys didn’t work in this commit with error invalid parameter. The error in server is, 06/19-00:23:45.58 jiafu-dev DAOS[348411/348420] bulk ERR src/cart/src/cart/crt_bulk.c:106 crt_bulk_desc_valid() invalid parameter of bulk_desc (remote_hdl:0x7f13c8b76400,local_hdl:0x7f13c8b7edf0, bulk_op:104, len: 0. 06/19-00:23:45.58 jiafu-dev DAOS[348411/348420] bulk ERR src/cart/src/cart/crt_bulk.c:205 crt_bulk_transfer() invalid parameter of bulk_desc. 06/19-00:23:45.58 jiafu-dev DAOS[348411/348420] object ERR src/object/srv_obj.c:390 obj_bulk_transfer() crt_bulk_transfer 0 error (-1003). 06/19-00:23:45.58 jiafu-dev DAOS[351611/351612] object ERR src/object/cli_shard.c:856 dc_enumerate_cb() rpc 0x7f79e4aca470 RPC 3 failed: DER_INVAL(-1003) 06/19-00:24:32.05 jiafu-dev DAOS[348411/348420] vos ERR src/vos/vos_obj.c:1596 vos_obj_iter_aggregate() Removing orphaned array value tree 06/19-00:26:02.06 jiafu-dev DAOS[348411/348420] vos ERR src/vos/vos_obj.c:1596 vos_obj_iter_aggregate() Removing orphaned array value tree 06/19-00:26:02.06 jiafu-dev DAOS[348411/348424] vos ERR src/vos/vos_obj.c:1596 vos_obj_iter_aggregate() Removing orphaned array value tree Thanks. toggle quoted message Show quoted text From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Zhang, Jiafu Sent: Thursday, June 18, 2020 6:24 PM To: daos@daos.groups.io Subject: Re: [daos] daos_obj_update crashed IO server When I reverted to 6c3eb801d0a9307cdf88b8634ea5a103d0b3542f, the format command worked for me. From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Zhang, Jiafu Sent: Thursday, June 18, 2020 6:18 PM To: daos@daos.groups.io Subject: Re: [daos] daos_obj_update crashed IO server The so issue is fixed by adding DAOS lib paths to ldconfig. It’s new behavior? I only needed to set LD_LIBRARY_PATH last time. Another issue raised up when format new DAOS server. $ dmg -i -l jiafu-dev:10001 storage format --reformat processing system reformat request ERROR: dmg: System-Query command failed: failed to get MS response: rpc error: code = Unknown desc = empty system membership From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Zhang, Jiafu Sent: Thursday, June 18, 2020 4:00 PM To: daos@daos.groups.io Subject: Re: [daos] daos_obj_update crashed IO server Hi Johann, My version is old. I tried to upgrade from ubuntu 18.04 to ubuntu 20.04. And built latest DAOS code as of now from master. But I got below error when started daos server with non-root user. I’ve followed “Privileged Helper Configuration” in https://daos-stack.github.io/admin/predeployment_check/. ERROR: /usr/bin/daos_admin /usr/bin/daos_admin: error while loading shared libraries: libspdk_log.so.2.0: cannot open shared object file: No such file or directory ERROR: pbin: code = 4 description = "Unable to decode response after 5 attempts" When I ldd /usr/bin/daos_admin and ldd libspdk_log.so.2.0, both worked. jiafu@jiafu-dev:~/daos$ ldd /usr/bin/daos_admin linux-vdso.so.1 (0x00007ffd9bfeb000) /usr/lib/jvm/java-1.8.0-openjdk-amd64/jre/lib/amd64/libjsig.so (0x00007f63ef4b1000) libspdk_env_dpdk.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_env_dpdk.so.2.0 (0x00007f63ef4a0000) librte_mempool.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_mempool.so.20.0 (0x00007f63ef497000) librte_bus_pci.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_bus_pci.so.20.0 (0x00007f63ef487000) librte_ring.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_ring.so.20.0 (0x00007f63ef482000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f63ef444000) libspdk_nvme.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_nvme.so.2.0 (0x00007f63ef40a000) libipmctl.so.4 => /usr/lib/x86_64-linux-gnu/libipmctl.so.4 (0x00007f63ef049000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f63eee57000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f63eee51000) /lib64/ld-linux-x86-64.so.2 (0x00007f63ef4b9000) librte_eal.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_eal.so.20.0 (0x00007f63eed6e000) librte_pci.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_pci.so.20.0 (0x00007f63eed67000) librte_kvargs.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_kvargs.so.20.0 (0x00007f63eed62000) libibverbs.so.1 => /usr/lib/x86_64-linux-gnu/libibverbs.so.1 (0x00007f63eed43000) librdmacm.so.1 => /usr/lib/x86_64-linux-gnu/librdmacm.so.1 (0x00007f63eed24000) libspdk_log.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_log.so.2.0 (0x00007f63eed1f000) libspdk_sock.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_sock.so.2.0 (0x00007f63eed1a000) libspdk_util.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_util.so.2.0 (0x00007f63eed07000) libndctl.so.6 => /usr/lib/x86_64-linux-gnu/libndctl.so.6 (0x00007f63eece4000) libnuma.so.1 => /usr/lib/x86_64-linux-gnu/libnuma.so.1 (0x00007f63eecd7000) libnl-route-3.so.200 => /usr/lib/x86_64-linux-gnu/libnl-route-3.so.200 (0x00007f63eec5f000) libnl-3.so.200 => /lib/x86_64-linux-gnu/libnl-3.so.200 (0x00007f63eec3c000) libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007f63eec31000) libdaxctl.so.1 => /usr/lib/x86_64-linux-gnu/libdaxctl.so.1 (0x00007f63eec27000) libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007f63eebfb000) libkmod.so.2 => /usr/lib/x86_64-linux-gnu/libkmod.so.2 (0x00007f63eebe0000) liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f63eebb9000) libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f63ee8e1000) jiafu@jiafu-dev:~/daos$ cd /code-repo/daos-master/daos/install/lib/ jiafu@jiafu-dev:/code-repo/daos-master/daos/install/lib$ ldd libspdk_log.so.2.0 linux-vdso.so.1 (0x00007fffa928a000) /usr/lib/jvm/java-1.8.0-openjdk-amd64/jre/lib/amd64/libjsig.so (0x00007fb538648000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb53843d000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fb538437000) /lib64/ld-linux-x86-64.so.2 (0x00007fb538655000) Please help. Thanks. From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Lombardi, Johann Sent: Wednesday, June 17, 2020 5:48 AM To: daos@daos.groups.io Subject: Re: [daos] daos_obj_update crashed IO server Hm, the patch was merged a while ago. How recent is your version of master? Cheers, Johann From: <daos@daos.groups.io> on behalf of "Zhang, Jiafu" <jiafu.zhang@...> Reply-To: "daos@daos.groups.io" <daos@daos.groups.io> Date: Tuesday 16 June 2020 at 14:49 To: "daos@daos.groups.io" <daos@daos.groups.io> Subject: Re: [daos] daos_obj_update crashed IO server I get this ticket, DAOS-4391. But it’s for duplicated akey in single update. I’ll try it anyway. From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Zhang, Jiafu Sent: Tuesday, June 16, 2020 8:12 PM To: daos@daos.groups.io Subject: [daos] daos_obj_update crashed IO server Hi Guys, I can call daos_obj_update successfully with one iod/sgl. But daos_obj_update crashed IO server when there were two iods/sgls under the same dkeys. From the server log, I see below error. Can someone tell me what’s going on here? “dkey1”, “akey1” -> succeeded. “dkey1”, “akey1”, “akey2” -> failed. ERROR: daos_io_server:0 06/16-12:04:43.41 jiafu-dev DAOS[28902/28911] vos EMRG src/vos/vos_ts.h:332 vos_ts_alloc() Assertion failed (info->ti_type & 1) == 0 ERROR: daos_io_server:0 daos_io_server: src/vos/vos_ts.h:332: vos_ts_alloc: Assertion `(info->ti_type & 1) == 0' failed. ERROR: daos_io_server:0 * Process 28902 received signal 6 * ERROR: daos_io_server:0 Associated errno: Success (0) ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7fcc895b4890] ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7fcc88b8ae97] ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7fcc88b8c801] ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(+0x3039a)[0x7fcc88b7c39a] ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(+0x30412)[0x7fcc88b7c412] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(vos_ilog_ts_cache+0x34c)[0x7fcc8a24583c] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(key_tree_prepare+0x251)[0x7fcc8a21ea41] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(vos_update_end+0x159c)[0x7fcc8a1f070c] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(+0x1e8e3)[0x7fcc7e1da8e3] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(+0x20945)[0x7fcc7e1dc945] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(ds_obj_rw_handler+0x3a0)[0x7fcc7e1dd470] ERROR: daos_io_server:0 /usr/local/daos/lib/libcart.so.4(+0xa25ee)[0x7fcc8a5245ee] ERROR: daos_io_server:0 /usr/local/daos/lib/libabt.so.0(+0x10ea0)[0x7fcc89393ea0] ERROR: daos_io_server:0 /usr/local/daos/lib/libabt.so.0(+0x11601)[0x7fcc89394601] Thanks. --------------------------------------------------------------------- Intel Corporation SAS (French simplified joint stock company) Registered headquarters: "Les Montalets"- 2, rue de Paris, 92196 Meudon Cedex, France Registration Number: 302 456 199 R.C.S. NANTERRE Capital: 4,572,000 Euros This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
More All Messages By This Member
Lombardi, Johann #1062 Hi Jiafu, Could you please email me your test case? I will look into it. Cheers, Johann From: <daos@daos.groups.io> on behalf of "Zhang, Jiafu" <jiafu.zhang@...> Reply-To: "daos@daos.groups.io" <daos@daos.groups.io> Date: Friday 19 June 2020 at 02:52 To: "daos@daos.groups.io" <daos@daos.groups.io> Subject: Re: [daos] daos_obj_update crashed IO server Hi Johann, In recent commit (6c3eb801d0a9307cdf88b8634ea5a103d0b3542f), the daos io server crash issue is gone when update object with two akeys under the same dkey. But I got another issue, DER_NO_PERM, no permission. And the update with single akey still worked. By the way, listing dkeys worked. But listing akeys didn’t work in this commit with error invalid parameter. The error in server is, 06/19-00:23:45.58 jiafu-dev DAOS[348411/348420] bulk ERR src/cart/src/cart/crt_bulk.c:106 crt_bulk_desc_valid() invalid parameter of bulk_desc (remote_hdl:0x7f13c8b76400,local_hdl:0x7f13c8b7edf0, bulk_op:104, len: 0. 06/19-00:23:45.58 jiafu-dev DAOS[348411/348420] bulk ERR src/cart/src/cart/crt_bulk.c:205 crt_bulk_transfer() invalid parameter of bulk_desc. 06/19-00:23:45.58 jiafu-dev DAOS[348411/348420] object ERR src/object/srv_obj.c:390 obj_bulk_transfer() crt_bulk_transfer 0 error (-1003). 06/19-00:23:45.58 jiafu-dev DAOS[351611/351612] object ERR src/object/cli_shard.c:856 dc_enumerate_cb() rpc 0x7f79e4aca470 RPC 3 failed: DER_INVAL(-1003) 06/19-00:24:32.05 jiafu-dev DAOS[348411/348420] vos ERR src/vos/vos_obj.c:1596 vos_obj_iter_aggregate() Removing orphaned array value tree 06/19-00:26:02.06 jiafu-dev DAOS[348411/348420] vos ERR src/vos/vos_obj.c:1596 vos_obj_iter_aggregate() Removing orphaned array value tree 06/19-00:26:02.06 jiafu-dev DAOS[348411/348424] vos ERR src/vos/vos_obj.c:1596 vos_obj_iter_aggregate() Removing orphaned array value tree Thanks. toggle quoted message Show quoted text From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Zhang, Jiafu Sent: Thursday, June 18, 2020 6:24 PM To: daos@daos.groups.io Subject: Re: [daos] daos_obj_update crashed IO server When I reverted to 6c3eb801d0a9307cdf88b8634ea5a103d0b3542f, the format command worked for me. From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Zhang, Jiafu Sent: Thursday, June 18, 2020 6:18 PM To: daos@daos.groups.io Subject: Re: [daos] daos_obj_update crashed IO server The so issue is fixed by adding DAOS lib paths to ldconfig. It’s new behavior? I only needed to set LD_LIBRARY_PATH last time. Another issue raised up when format new DAOS server. $ dmg -i -l jiafu-dev:10001 storage format --reformat processing system reformat request ERROR: dmg: System-Query command failed: failed to get MS response: rpc error: code = Unknown desc = empty system membership From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Zhang, Jiafu Sent: Thursday, June 18, 2020 4:00 PM To: daos@daos.groups.io Subject: Re: [daos] daos_obj_update crashed IO server Hi Johann, My version is old. I tried to upgrade from ubuntu 18.04 to ubuntu 20.04. And built latest DAOS code as of now from master. But I got below error when started daos server with non-root user. I’ve followed “Privileged Helper Configuration” in https://daos-stack.github.io/admin/predeployment_check/. ERROR: /usr/bin/daos_admin /usr/bin/daos_admin: error while loading shared libraries: libspdk_log.so.2.0: cannot open shared object file: No such file or directory ERROR: pbin: code = 4 description = "Unable to decode response after 5 attempts" When I ldd /usr/bin/daos_admin and ldd libspdk_log.so.2.0, both worked. jiafu@jiafu-dev:~/daos$ ldd /usr/bin/daos_admin linux-vdso.so.1 (0x00007ffd9bfeb000) /usr/lib/jvm/java-1.8.0-openjdk-amd64/jre/lib/amd64/libjsig.so (0x00007f63ef4b1000) libspdk_env_dpdk.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_env_dpdk.so.2.0 (0x00007f63ef4a0000) librte_mempool.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_mempool.so.20.0 (0x00007f63ef497000) librte_bus_pci.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_bus_pci.so.20.0 (0x00007f63ef487000) librte_ring.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_ring.so.20.0 (0x00007f63ef482000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f63ef444000) libspdk_nvme.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_nvme.so.2.0 (0x00007f63ef40a000) libipmctl.so.4 => /usr/lib/x86_64-linux-gnu/libipmctl.so.4 (0x00007f63ef049000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f63eee57000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f63eee51000) /lib64/ld-linux-x86-64.so.2 (0x00007f63ef4b9000) librte_eal.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_eal.so.20.0 (0x00007f63eed6e000) librte_pci.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_pci.so.20.0 (0x00007f63eed67000) librte_kvargs.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_kvargs.so.20.0 (0x00007f63eed62000) libibverbs.so.1 => /usr/lib/x86_64-linux-gnu/libibverbs.so.1 (0x00007f63eed43000) librdmacm.so.1 => /usr/lib/x86_64-linux-gnu/librdmacm.so.1 (0x00007f63eed24000) libspdk_log.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_log.so.2.0 (0x00007f63eed1f000) libspdk_sock.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_sock.so.2.0 (0x00007f63eed1a000) libspdk_util.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_util.so.2.0 (0x00007f63eed07000) libndctl.so.6 => /usr/lib/x86_64-linux-gnu/libndctl.so.6 (0x00007f63eece4000) libnuma.so.1 => /usr/lib/x86_64-linux-gnu/libnuma.so.1 (0x00007f63eecd7000) libnl-route-3.so.200 => /usr/lib/x86_64-linux-gnu/libnl-route-3.so.200 (0x00007f63eec5f000) libnl-3.so.200 => /lib/x86_64-linux-gnu/libnl-3.so.200 (0x00007f63eec3c000) libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007f63eec31000) libdaxctl.so.1 => /usr/lib/x86_64-linux-gnu/libdaxctl.so.1 (0x00007f63eec27000) libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007f63eebfb000) libkmod.so.2 => /usr/lib/x86_64-linux-gnu/libkmod.so.2 (0x00007f63eebe0000) liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f63eebb9000) libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f63ee8e1000) jiafu@jiafu-dev:~/daos$ cd /code-repo/daos-master/daos/install/lib/ jiafu@jiafu-dev:/code-repo/daos-master/daos/install/lib$ ldd libspdk_log.so.2.0 linux-vdso.so.1 (0x00007fffa928a000) /usr/lib/jvm/java-1.8.0-openjdk-amd64/jre/lib/amd64/libjsig.so (0x00007fb538648000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb53843d000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fb538437000) /lib64/ld-linux-x86-64.so.2 (0x00007fb538655000) Please help. Thanks. From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Lombardi, Johann Sent: Wednesday, June 17, 2020 5:48 AM To: daos@daos.groups.io Subject: Re: [daos] daos_obj_update crashed IO server Hm, the patch was merged a while ago. How recent is your version of master? Cheers, Johann From: <daos@daos.groups.io> on behalf of "Zhang, Jiafu" <jiafu.zhang@...> Reply-To: "daos@daos.groups.io" <daos@daos.groups.io> Date: Tuesday 16 June 2020 at 14:49 To: "daos@daos.groups.io" <daos@daos.groups.io> Subject: Re: [daos] daos_obj_update crashed IO server 1. I get this ticket, DAOS-4391. But it’s for duplicated akey in single update. I’ll try it anyway. From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Zhang, Jiafu Sent: Tuesday, June 16, 2020 8:12 PM To: daos@daos.groups.io Subject: [daos] daos_obj_update crashed IO server Hi Guys, I can call daos_obj_update successfully with one iod/sgl. But daos_obj_update crashed IO server when there were two iods/sgls under the same dkeys. From the server log, I see below error. Can someone tell me what’s going on here? “dkey1”, “akey1” -> succeeded. “dkey1”, “akey1”, “akey2” -> failed. ERROR: daos_io_server:0 06/16-12:04:43.41 jiafu-dev DAOS[28902/28911] vos EMRG src/vos/vos_ts.h:332 vos_ts_alloc() Assertion failed (info->ti_type & 1) == 0 ERROR: daos_io_server:0 daos_io_server: src/vos/vos_ts.h:332: vos_ts_alloc: Assertion `(info->ti_type & 1) == 0' failed. ERROR: daos_io_server:0 * Process 28902 received signal 6 * ERROR: daos_io_server:0 Associated errno: Success (0) ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7fcc895b4890] ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7fcc88b8ae97] ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7fcc88b8c801] ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(+0x3039a)[0x7fcc88b7c39a] ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(+0x30412)[0x7fcc88b7c412] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(vos_ilog_ts_cache+0x34c)[0x7fcc8a24583c] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(key_tree_prepare+0x251)[0x7fcc8a21ea41] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(vos_update_end+0x159c)[0x7fcc8a1f070c] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(+0x1e8e3)[0x7fcc7e1da8e3] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(+0x20945)[0x7fcc7e1dc945] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(ds_obj_rw_handler+0x3a0)[0x7fcc7e1dd470] ERROR: daos_io_server:0 /usr/local/daos/lib/libcart.so.4(+0xa25ee)[0x7fcc8a5245ee] ERROR: daos_io_server:0 /usr/local/daos/lib/libabt.so.0(+0x10ea0)[0x7fcc89393ea0] ERROR: daos_io_server:0 /usr/local/daos/lib/libabt.so.0(+0x11601)[0x7fcc89394601] Thanks. --------------------------------------------------------------------- Intel Corporation SAS (French simplified joint stock company) Registered headquarters: "Les Montalets"- 2, rue de Paris, 92196 Meudon Cedex, France Registration Number: 302 456 199 R.C.S. NANTERRE Capital: 4,572,000 Euros This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. --------------------------------------------------------------------- Intel Corporation SAS (French simplified joint stock company) Registered headquarters: "Les Montalets"- 2, rue de Paris, 92196 Meudon Cedex, France Registration Number: 302 456 199 R.C.S. NANTERRE Capital: 4,572,000 Euros This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
More All Messages By This Member
Nabarro, Tom #1068 The bug with the storage reformat command has been fixed by Mike on master. Apologies for the inconvenience. Regards, Tom toggle quoted message Show quoted text From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Lombardi, Johann Sent: Friday, June 19, 2020 12:30 PM To: daos@daos.groups.io Subject: Re: [daos] daos_obj_update crashed IO server Hi Jiafu, Could you please email me your test case? I will look into it. Cheers, Johann From: <daos@daos.groups.io> on behalf of "Zhang, Jiafu" <jiafu.zhang@...> Reply-To: "daos@daos.groups.io" <daos@daos.groups.io> Date: Friday 19 June 2020 at 02:52 To: "daos@daos.groups.io" <daos@daos.groups.io> Subject: Re: [daos] daos_obj_update crashed IO server Hi Johann, In recent commit (6c3eb801d0a9307cdf88b8634ea5a103d0b3542f), the daos io server crash issue is gone when update object with two akeys under the same dkey. But I got another issue, DER_NO_PERM, no permission. And the update with single akey still worked. By the way, listing dkeys worked. But listing akeys didn’t work in this commit with error invalid parameter. The error in server is, 06/19-00:23:45.58 jiafu-dev DAOS[348411/348420] bulk ERR src/cart/src/cart/crt_bulk.c:106 crt_bulk_desc_valid() invalid parameter of bulk_desc (remote_hdl:0x7f13c8b76400,local_hdl:0x7f13c8b7edf0, bulk_op:104, len: 0. 06/19-00:23:45.58 jiafu-dev DAOS[348411/348420] bulk ERR src/cart/src/cart/crt_bulk.c:205 crt_bulk_transfer() invalid parameter of bulk_desc. 06/19-00:23:45.58 jiafu-dev DAOS[348411/348420] object ERR src/object/srv_obj.c:390 obj_bulk_transfer() crt_bulk_transfer 0 error (-1003). 06/19-00:23:45.58 jiafu-dev DAOS[351611/351612] object ERR src/object/cli_shard.c:856 dc_enumerate_cb() rpc 0x7f79e4aca470 RPC 3 failed: DER_INVAL(-1003) 06/19-00:24:32.05 jiafu-dev DAOS[348411/348420] vos ERR src/vos/vos_obj.c:1596 vos_obj_iter_aggregate() Removing orphaned array value tree 06/19-00:26:02.06 jiafu-dev DAOS[348411/348420] vos ERR src/vos/vos_obj.c:1596 vos_obj_iter_aggregate() Removing orphaned array value tree 06/19-00:26:02.06 jiafu-dev DAOS[348411/348424] vos ERR src/vos/vos_obj.c:1596 vos_obj_iter_aggregate() Removing orphaned array value tree Thanks. From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Zhang, Jiafu Sent: Thursday, June 18, 2020 6:24 PM To: daos@daos.groups.io Subject: Re: [daos] daos_obj_update crashed IO server When I reverted to 6c3eb801d0a9307cdf88b8634ea5a103d0b3542f, the format command worked for me. From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Zhang, Jiafu Sent: Thursday, June 18, 2020 6:18 PM To: daos@daos.groups.io Subject: Re: [daos] daos_obj_update crashed IO server The so issue is fixed by adding DAOS lib paths to ldconfig. It’s new behavior? I only needed to set LD_LIBRARY_PATH last time. Another issue raised up when format new DAOS server. $ dmg -i -l jiafu-dev:10001 storage format --reformat processing system reformat request ERROR: dmg: System-Query command failed: failed to get MS response: rpc error: code = Unknown desc = empty system membership From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Zhang, Jiafu Sent: Thursday, June 18, 2020 4:00 PM To: daos@daos.groups.io Subject: Re: [daos] daos_obj_update crashed IO server Hi Johann, My version is old. I tried to upgrade from ubuntu 18.04 to ubuntu 20.04. And built latest DAOS code as of now from master. But I got below error when started daos server with non-root user. I’ve followed “Privileged Helper Configuration” in https://daos-stack.github.io/admin/predeployment_check/. ERROR: /usr/bin/daos_admin /usr/bin/daos_admin: error while loading shared libraries: libspdk_log.so.2.0: cannot open shared object file: No such file or directory ERROR: pbin: code = 4 description = "Unable to decode response after 5 attempts" When I ldd /usr/bin/daos_admin and ldd libspdk_log.so.2.0, both worked. jiafu@jiafu-dev:~/daos$ ldd /usr/bin/daos_admin linux-vdso.so.1 (0x00007ffd9bfeb000) /usr/lib/jvm/java-1.8.0-openjdk-amd64/jre/lib/amd64/libjsig.so (0x00007f63ef4b1000) libspdk_env_dpdk.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_env_dpdk.so.2.0 (0x00007f63ef4a0000) librte_mempool.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_mempool.so.20.0 (0x00007f63ef497000) librte_bus_pci.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_bus_pci.so.20.0 (0x00007f63ef487000) librte_ring.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_ring.so.20.0 (0x00007f63ef482000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f63ef444000) libspdk_nvme.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_nvme.so.2.0 (0x00007f63ef40a000) libipmctl.so.4 => /usr/lib/x86_64-linux-gnu/libipmctl.so.4 (0x00007f63ef049000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f63eee57000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f63eee51000) /lib64/ld-linux-x86-64.so.2 (0x00007f63ef4b9000) librte_eal.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_eal.so.20.0 (0x00007f63eed6e000) librte_pci.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_pci.so.20.0 (0x00007f63eed67000) librte_kvargs.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_kvargs.so.20.0 (0x00007f63eed62000) libibverbs.so.1 => /usr/lib/x86_64-linux-gnu/libibverbs.so.1 (0x00007f63eed43000) librdmacm.so.1 => /usr/lib/x86_64-linux-gnu/librdmacm.so.1 (0x00007f63eed24000) libspdk_log.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_log.so.2.0 (0x00007f63eed1f000) libspdk_sock.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_sock.so.2.0 (0x00007f63eed1a000) libspdk_util.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_util.so.2.0 (0x00007f63eed07000) libndctl.so.6 => /usr/lib/x86_64-linux-gnu/libndctl.so.6 (0x00007f63eece4000) libnuma.so.1 => /usr/lib/x86_64-linux-gnu/libnuma.so.1 (0x00007f63eecd7000) libnl-route-3.so.200 => /usr/lib/x86_64-linux-gnu/libnl-route-3.so.200 (0x00007f63eec5f000) libnl-3.so.200 => /lib/x86_64-linux-gnu/libnl-3.so.200 (0x00007f63eec3c000) libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007f63eec31000) libdaxctl.so.1 => /usr/lib/x86_64-linux-gnu/libdaxctl.so.1 (0x00007f63eec27000) libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007f63eebfb000) libkmod.so.2 => /usr/lib/x86_64-linux-gnu/libkmod.so.2 (0x00007f63eebe0000) liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f63eebb9000) libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f63ee8e1000) jiafu@jiafu-dev:~/daos$ cd /code-repo/daos-master/daos/install/lib/ jiafu@jiafu-dev:/code-repo/daos-master/daos/install/lib$ ldd libspdk_log.so.2.0 linux-vdso.so.1 (0x00007fffa928a000) /usr/lib/jvm/java-1.8.0-openjdk-amd64/jre/lib/amd64/libjsig.so (0x00007fb538648000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb53843d000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fb538437000) /lib64/ld-linux-x86-64.so.2 (0x00007fb538655000) Please help. Thanks. From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Lombardi, Johann Sent: Wednesday, June 17, 2020 5:48 AM To: daos@daos.groups.io Subject: Re: [daos] daos_obj_update crashed IO server Hm, the patch was merged a while ago. How recent is your version of master? Cheers, Johann From: <daos@daos.groups.io> on behalf of "Zhang, Jiafu" <jiafu.zhang@...> Reply-To: "daos@daos.groups.io" <daos@daos.groups.io> Date: Tuesday 16 June 2020 at 14:49 To: "daos@daos.groups.io" <daos@daos.groups.io> Subject: Re: [daos] daos_obj_update crashed IO server 1. I get this ticket, DAOS-4391. But it’s for duplicated akey in single update. I’ll try it anyway. From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Zhang, Jiafu Sent: Tuesday, June 16, 2020 8:12 PM To: daos@daos.groups.io Subject: [daos] daos_obj_update crashed IO server Hi Guys, I can call daos_obj_update successfully with one iod/sgl. But daos_obj_update crashed IO server when there were two iods/sgls under the same dkeys. From the server log, I see below error. Can someone tell me what’s going on here? “dkey1”, “akey1” -> succeeded. “dkey1”, “akey1”, “akey2” -> failed. ERROR: daos_io_server:0 06/16-12:04:43.41 jiafu-dev DAOS[28902/28911] vos EMRG src/vos/vos_ts.h:332 vos_ts_alloc() Assertion failed (info->ti_type & 1) == 0 ERROR: daos_io_server:0 daos_io_server: src/vos/vos_ts.h:332: vos_ts_alloc: Assertion `(info->ti_type & 1) == 0' failed. ERROR: daos_io_server:0 * Process 28902 received signal 6 * ERROR: daos_io_server:0 Associated errno: Success (0) ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7fcc895b4890] ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7fcc88b8ae97] ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7fcc88b8c801] ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(+0x3039a)[0x7fcc88b7c39a] ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(+0x30412)[0x7fcc88b7c412] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(vos_ilog_ts_cache+0x34c)[0x7fcc8a24583c] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(key_tree_prepare+0x251)[0x7fcc8a21ea41] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(vos_update_end+0x159c)[0x7fcc8a1f070c] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(+0x1e8e3)[0x7fcc7e1da8e3] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(+0x20945)[0x7fcc7e1dc945] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(ds_obj_rw_handler+0x3a0)[0x7fcc7e1dd470] ERROR: daos_io_server:0 /usr/local/daos/lib/libcart.so.4(+0xa25ee)[0x7fcc8a5245ee] ERROR: daos_io_server:0 /usr/local/daos/lib/libabt.so.0(+0x10ea0)[0x7fcc89393ea0] ERROR: daos_io_server:0 /usr/local/daos/lib/libabt.so.0(+0x11601)[0x7fcc89394601] Thanks. --------------------------------------------------------------------- Intel Corporation SAS (French simplified joint stock company) Registered headquarters: "Les Montalets"- 2, rue de Paris, 92196 Meudon Cedex, France Registration Number: 302 456 199 R.C.S. NANTERRE Capital: 4,572,000 Euros This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. --------------------------------------------------------------------- Intel Corporation SAS (French simplified joint stock company) Registered headquarters: "Les Montalets"- 2, rue de Paris, 92196 Meudon Cedex, France Registration Number: 302 456 199 R.C.S. NANTERRE Capital: 4,572,000 Euros This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. --------------------------------------------------------------------- Intel Corporation (UK) Limited Registered No. 1134945 (England) Registered Office: Pipers Way, Swindon SN3 1RJ VAT No: 860 2173 47 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
More All Messages By This Member
Zhang, Jiafu #1071 Hi Johann, It turned out to be my bug which failed the object update and key listing. Sorry for the bothering. Thanks. toggle quoted message Show quoted text From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Lombardi, Johann Sent: Friday, June 19, 2020 7:30 PM To: daos@daos.groups.io Subject: Re: [daos] daos_obj_update crashed IO server Hi Jiafu, Could you please email me your test case? I will look into it. Cheers, Johann From: <daos@daos.groups.io> on behalf of "Zhang, Jiafu" <jiafu.zhang@...> Reply-To: "daos@daos.groups.io" <daos@daos.groups.io> Date: Friday 19 June 2020 at 02:52 To: "daos@daos.groups.io" <daos@daos.groups.io> Subject: Re: [daos] daos_obj_update crashed IO server Hi Johann, In recent commit (6c3eb801d0a9307cdf88b8634ea5a103d0b3542f), the daos io server crash issue is gone when update object with two akeys under the same dkey. But I got another issue, DER_NO_PERM, no permission. And the update with single akey still worked. By the way, listing dkeys worked. But listing akeys didn’t work in this commit with error invalid parameter. The error in server is, 06/19-00:23:45.58 jiafu-dev DAOS[348411/348420] bulk ERR src/cart/src/cart/crt_bulk.c:106 crt_bulk_desc_valid() invalid parameter of bulk_desc (remote_hdl:0x7f13c8b76400,local_hdl:0x7f13c8b7edf0, bulk_op:104, len: 0. 06/19-00:23:45.58 jiafu-dev DAOS[348411/348420] bulk ERR src/cart/src/cart/crt_bulk.c:205 crt_bulk_transfer() invalid parameter of bulk_desc. 06/19-00:23:45.58 jiafu-dev DAOS[348411/348420] object ERR src/object/srv_obj.c:390 obj_bulk_transfer() crt_bulk_transfer 0 error (-1003). 06/19-00:23:45.58 jiafu-dev DAOS[351611/351612] object ERR src/object/cli_shard.c:856 dc_enumerate_cb() rpc 0x7f79e4aca470 RPC 3 failed: DER_INVAL(-1003) 06/19-00:24:32.05 jiafu-dev DAOS[348411/348420] vos ERR src/vos/vos_obj.c:1596 vos_obj_iter_aggregate() Removing orphaned array value tree 06/19-00:26:02.06 jiafu-dev DAOS[348411/348420] vos ERR src/vos/vos_obj.c:1596 vos_obj_iter_aggregate() Removing orphaned array value tree 06/19-00:26:02.06 jiafu-dev DAOS[348411/348424] vos ERR src/vos/vos_obj.c:1596 vos_obj_iter_aggregate() Removing orphaned array value tree Thanks. From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Zhang, Jiafu Sent: Thursday, June 18, 2020 6:24 PM To: daos@daos.groups.io Subject: Re: [daos] daos_obj_update crashed IO server When I reverted to 6c3eb801d0a9307cdf88b8634ea5a103d0b3542f, the format command worked for me. From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Zhang, Jiafu Sent: Thursday, June 18, 2020 6:18 PM To: daos@daos.groups.io Subject: Re: [daos] daos_obj_update crashed IO server The so issue is fixed by adding DAOS lib paths to ldconfig. It’s new behavior? I only needed to set LD_LIBRARY_PATH last time. Another issue raised up when format new DAOS server. $ dmg -i -l jiafu-dev:10001 storage format --reformat processing system reformat request ERROR: dmg: System-Query command failed: failed to get MS response: rpc error: code = Unknown desc = empty system membership From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Zhang, Jiafu Sent: Thursday, June 18, 2020 4:00 PM To: daos@daos.groups.io Subject: Re: [daos] daos_obj_update crashed IO server Hi Johann, My version is old. I tried to upgrade from ubuntu 18.04 to ubuntu 20.04. And built latest DAOS code as of now from master. But I got below error when started daos server with non-root user. I’ve followed “Privileged Helper Configuration” in https://daos-stack.github.io/admin/predeployment_check/. ERROR: /usr/bin/daos_admin /usr/bin/daos_admin: error while loading shared libraries: libspdk_log.so.2.0: cannot open shared object file: No such file or directory ERROR: pbin: code = 4 description = "Unable to decode response after 5 attempts" When I ldd /usr/bin/daos_admin and ldd libspdk_log.so.2.0, both worked. jiafu@jiafu-dev:~/daos$ ldd /usr/bin/daos_admin linux-vdso.so.1 (0x00007ffd9bfeb000) /usr/lib/jvm/java-1.8.0-openjdk-amd64/jre/lib/amd64/libjsig.so (0x00007f63ef4b1000) libspdk_env_dpdk.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_env_dpdk.so.2.0 (0x00007f63ef4a0000) librte_mempool.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_mempool.so.20.0 (0x00007f63ef497000) librte_bus_pci.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_bus_pci.so.20.0 (0x00007f63ef487000) librte_ring.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_ring.so.20.0 (0x00007f63ef482000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f63ef444000) libspdk_nvme.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_nvme.so.2.0 (0x00007f63ef40a000) libipmctl.so.4 => /usr/lib/x86_64-linux-gnu/libipmctl.so.4 (0x00007f63ef049000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f63eee57000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f63eee51000) /lib64/ld-linux-x86-64.so.2 (0x00007f63ef4b9000) librte_eal.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_eal.so.20.0 (0x00007f63eed6e000) librte_pci.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_pci.so.20.0 (0x00007f63eed67000) librte_kvargs.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_kvargs.so.20.0 (0x00007f63eed62000) libibverbs.so.1 => /usr/lib/x86_64-linux-gnu/libibverbs.so.1 (0x00007f63eed43000) librdmacm.so.1 => /usr/lib/x86_64-linux-gnu/librdmacm.so.1 (0x00007f63eed24000) libspdk_log.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_log.so.2.0 (0x00007f63eed1f000) libspdk_sock.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_sock.so.2.0 (0x00007f63eed1a000) libspdk_util.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_util.so.2.0 (0x00007f63eed07000) libndctl.so.6 => /usr/lib/x86_64-linux-gnu/libndctl.so.6 (0x00007f63eece4000) libnuma.so.1 => /usr/lib/x86_64-linux-gnu/libnuma.so.1 (0x00007f63eecd7000) libnl-route-3.so.200 => /usr/lib/x86_64-linux-gnu/libnl-route-3.so.200 (0x00007f63eec5f000) libnl-3.so.200 => /lib/x86_64-linux-gnu/libnl-3.so.200 (0x00007f63eec3c000) libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007f63eec31000) libdaxctl.so.1 => /usr/lib/x86_64-linux-gnu/libdaxctl.so.1 (0x00007f63eec27000) libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007f63eebfb000) libkmod.so.2 => /usr/lib/x86_64-linux-gnu/libkmod.so.2 (0x00007f63eebe0000) liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f63eebb9000) libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f63ee8e1000) jiafu@jiafu-dev:~/daos$ cd /code-repo/daos-master/daos/install/lib/ jiafu@jiafu-dev:/code-repo/daos-master/daos/install/lib$ ldd libspdk_log.so.2.0 linux-vdso.so.1 (0x00007fffa928a000) /usr/lib/jvm/java-1.8.0-openjdk-amd64/jre/lib/amd64/libjsig.so (0x00007fb538648000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb53843d000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fb538437000) /lib64/ld-linux-x86-64.so.2 (0x00007fb538655000) Please help. Thanks. From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Lombardi, Johann Sent: Wednesday, June 17, 2020 5:48 AM To: daos@daos.groups.io Subject: Re: [daos] daos_obj_update crashed IO server Hm, the patch was merged a while ago. How recent is your version of master? Cheers, Johann From: <daos@daos.groups.io> on behalf of "Zhang, Jiafu" <jiafu.zhang@...> Reply-To: "daos@daos.groups.io" <daos@daos.groups.io> Date: Tuesday 16 June 2020 at 14:49 To: "daos@daos.groups.io" <daos@daos.groups.io> Subject: Re: [daos] daos_obj_update crashed IO server 1. I get this ticket, DAOS-4391. But it’s for duplicated akey in single update. I’ll try it anyway. From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Zhang, Jiafu Sent: Tuesday, June 16, 2020 8:12 PM To: daos@daos.groups.io Subject: [daos] daos_obj_update crashed IO server Hi Guys, I can call daos_obj_update successfully with one iod/sgl. But daos_obj_update crashed IO server when there were two iods/sgls under the same dkeys. From the server log, I see below error. Can someone tell me what’s going on here? “dkey1”, “akey1” -> succeeded. “dkey1”, “akey1”, “akey2” -> failed. ERROR: daos_io_server:0 06/16-12:04:43.41 jiafu-dev DAOS[28902/28911] vos EMRG src/vos/vos_ts.h:332 vos_ts_alloc() Assertion failed (info->ti_type & 1) == 0 ERROR: daos_io_server:0 daos_io_server: src/vos/vos_ts.h:332: vos_ts_alloc: Assertion `(info->ti_type & 1) == 0' failed. ERROR: daos_io_server:0 * Process 28902 received signal 6 * ERROR: daos_io_server:0 Associated errno: Success (0) ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7fcc895b4890] ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7fcc88b8ae97] ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7fcc88b8c801] ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(+0x3039a)[0x7fcc88b7c39a] ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(+0x30412)[0x7fcc88b7c412] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(vos_ilog_ts_cache+0x34c)[0x7fcc8a24583c] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(key_tree_prepare+0x251)[0x7fcc8a21ea41] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(vos_update_end+0x159c)[0x7fcc8a1f070c] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(+0x1e8e3)[0x7fcc7e1da8e3] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(+0x20945)[0x7fcc7e1dc945] ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(ds_obj_rw_handler+0x3a0)[0x7fcc7e1dd470] ERROR: daos_io_server:0 /usr/local/daos/lib/libcart.so.4(+0xa25ee)[0x7fcc8a5245ee] ERROR: daos_io_server:0 /usr/local/daos/lib/libabt.so.0(+0x10ea0)[0x7fcc89393ea0] ERROR: daos_io_server:0 /usr/local/daos/lib/libabt.so.0(+0x11601)[0x7fcc89394601] Thanks. --------------------------------------------------------------------- Intel Corporation SAS (French simplified joint stock company) Registered headquarters: "Les Montalets"- 2, rue de Paris, 92196 Meudon Cedex, France Registration Number: 302 456 199 R.C.S. NANTERRE Capital: 4,572,000 Euros This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. --------------------------------------------------------------------- Intel Corporation SAS (French simplified joint stock company) Registered headquarters: "Les Montalets"- 2, rue de Paris, 92196 Meudon Cedex, France Registration Number: 302 456 199 R.C.S. NANTERRE Capital: 4,572,000 Euros This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
More All Messages By This Member
Lombardi, Johann #1072 Np. Thanks for letting us know. Johann From: <daos@daos.groups.io> on behalf of "Zhang, Jiafu" <jiafu.zhang@...> Reply-To: "daos@daos.groups.io" <daos@daos.groups.io> Date: Tuesday 23 June 2020 at 05:54 To: "daos@daos.groups.io" <daos@daos.groups.io> Subject: Re: [daos] daos_obj_update crashed IO server Hi Johann, It turned out to be my bug which failed the object update and key listing. Sorry for the bothering. Thanks. --------------------------------------------------------------------- Intel Corporation SAS (French simplified joint stock company) Registered headquarters: "Les Montalets"- 2, rue de Paris, 92196 Meudon Cedex, France Registration Number: 302 456 199 R.C.S. NANTERRE Capital: 4,572,000 Euros This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
More All Messages By This Member

1 - 11 of 11

Previous Topic Next Topic

Home Hashtags Subgroups About Features Pricing Updates Terms Help Twitter