Date   
Re: Intel Optane SSD mounting problem

Nabarro, Tom
 

We probably need to get further information from NSG folks on product specifics regarding Opal support in 905p SSDs. This seems to be a prerequisite for device management through SPDK.

 

Tom

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Farrell, Patrick Arthur
Sent: Wednesday, June 24, 2020 8:42 PM
To: daos@daos.groups.io
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Gert,

 

We're seeing this when we run identify on our 905p SSDs:

"

NVMe Controller at 0000:1a:00.0 [8086:2700]

[...]

Serial Number:                         PHM29226005S480BGN
Model Number:                          INTEL SSDPE21D480GA

[...]

Admin Command Set Attributes

============================

Security Send/Receive:                 Supported

"

 

However, when I run nvme_manage, I get an error stating that Opal is not supported (the same error DAOS is spitting out, incidentally):

 

"Please Input PCI Address(domain:bus:dev.func):

0000:1a:00.00

Opal General Usage:

 

        [1: scan device]

        [2: init - take ownership and activate locking]

        [3: revert tper]

        [4: setup locking range]

        [5: list locking ranges]

        [6: enable user]

        [7: set new password]

        [8: add user to locking range]

        [9: lock/unlock range]

        [10: erase locking range]

        [0: quit]

1

[2020-06-24 14:37:34.452086] nvme_opal.c: 828:opal_discovery0_end: *ERROR*: Opal Not Supported.

"

 

Any thoughts?

 

Regards,

-Patrick


From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Gert Pauwels (intel) <gert.pauwels@...>
Sent: Wednesday, June 24, 2020 11:54 AM
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Hi Tom, Colin,


I'm running Ubuntu 20.04 LTS  (kernel: 5.4.0-37-generic) and compiled DAOS v1.0.0.

I also compiled the latest master as of yesterday, but it did not make a difference.

Any application that can manage Opal 2.0 can be used to check the status of the drive. I used the sedutil-cli, can be found at https://github.com/Drive-Trust-Alliance/sedutil the executable can be found at https://github.com/Drive-Trust-Alliance/sedutil/wiki/Executable-Distributions 
You can run
# sedutil-cli --scan
and
# sedutil-cli --query <device>
it will return useful information about the Opal status of the device.
sedutil-cli will send NVMe security commands and the tool will only work if your SSD is bound the the NVMe driver.

In case the SSD is not bound to the NVMe driver you can manage the drive with the nvme_manage spdk utility you can found in the /daos/_build.external/dev/spdk
# ./examples/nvme/nvme_manage/nvme_manage 
Select 8 to get into the OPAL NVMe Management Options, enter the PCIe address of your SSD for the list you get prompted, select 1 to scan the device. These steps gives you the same results as running the # sedutil-cli --query <device> in case the SSD is bound to the NVMe driver.

Regards,

Gert,

 

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Re: Link Error with DAOS 1.0

Nabarro, Tom
 

There have been some fixes in master for build issues with ubuntu 20.04 (specifically for spdk, ipmctl and I think hwloc). Can you try building from master?

 

Tom

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Olivier, Jeffrey V
Sent: Thursday, June 25, 2020 2:06 AM
To: daos@daos.groups.io
Subject: Re: [daos] Link Error with DAOS 1.0

 

This small patch gets me past that error:

 

jvolivie-desk1:[~/daos_1.0]$ git diff

diff --git a/src/control/lib/spdk/spdk.go b/src/control/lib/spdk/spdk.go

index 28b55e537..bbc3d43c6 100644

--- a/src/control/lib/spdk/spdk.go

+++ b/src/control/lib/spdk/spdk.go

@@ -30,6 +30,7 @@ package spdk

/*

#cgo LDFLAGS: -lspdk_env_dpdk -lrte_mempool -lrte_mempool_ring -lrte_bus_pci

#cgo LDFLAGS: -lrte_pci -lrte_ring -lrte_mbuf -lrte_eal -lrte_kvargs -ldl -lnuma

+#cgo LDFLAGS: -lisal

 

 #include "stdlib.h"

#include "spdk/stdinc.h"

 

But then it runs into an issue with hwloc.

 

build/src/control/src/github.com/daos-stack/daos/src/control/lib/netdetect/netdetect.go:156:48: could not determine kind of name for C.HWLOC_TOPOLOGY_FLAG_IO_DEVICES

 

It looks like hwloc version 1.5 which should work but perhaps there is something new on Ubuntu 20.04 that we only resolved on master.  

 

-Jeff

 

From: <daos@daos.groups.io> on behalf of "Olivier, Jeffrey V" <jeffrey.v.olivier@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Wednesday, June 24, 2020 at 5:52 PM
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [daos] Link Error with DAOS 1.0

 

Hi Archie,

 

I don’t know if 1.0 has been tried with Ubuntu 20.04 though it does work with 18.04.   I will give it a try locally and see what I can find out.

 

-Jeff

 

From: <daos@daos.groups.io> on behalf of Archie Morningstar <archiemorningstar88@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Wednesday, June 24, 2020 at 12:29 PM
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [daos] Link Error with DAOS 1.0

 

[Edited Message Follows]

Sorry for not having that:
Ubuntu 20.04 LTS

Archie

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Re: Link Error with DAOS 1.0

Olivier, Jeffrey V
 

This small patch gets me past that error:

 

jvolivie-desk1:[~/daos_1.0]$ git diff

diff --git a/src/control/lib/spdk/spdk.go b/src/control/lib/spdk/spdk.go

index 28b55e537..bbc3d43c6 100644

--- a/src/control/lib/spdk/spdk.go

+++ b/src/control/lib/spdk/spdk.go

@@ -30,6 +30,7 @@ package spdk

/*

#cgo LDFLAGS: -lspdk_env_dpdk -lrte_mempool -lrte_mempool_ring -lrte_bus_pci

#cgo LDFLAGS: -lrte_pci -lrte_ring -lrte_mbuf -lrte_eal -lrte_kvargs -ldl -lnuma

+#cgo LDFLAGS: -lisal

 #include "stdlib.h"

#include "spdk/stdinc.h"

 

But then it runs into an issue with hwloc.

 

build/src/control/src/github.com/daos-stack/daos/src/control/lib/netdetect/netdetect.go:156:48: could not determine kind of name for C.HWLOC_TOPOLOGY_FLAG_IO_DEVICES

 

It looks like hwloc version 1.5 which should work but perhaps there is something new on Ubuntu 20.04 that we only resolved on master.  

 

-Jeff

 

From: <daos@daos.groups.io> on behalf of "Olivier, Jeffrey V" <jeffrey.v.olivier@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Wednesday, June 24, 2020 at 5:52 PM
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [daos] Link Error with DAOS 1.0

 

Hi Archie,

 

I don’t know if 1.0 has been tried with Ubuntu 20.04 though it does work with 18.04.   I will give it a try locally and see what I can find out.

 

-Jeff

 

From: <daos@daos.groups.io> on behalf of Archie Morningstar <archiemorningstar88@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Wednesday, June 24, 2020 at 12:29 PM
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [daos] Link Error with DAOS 1.0

 

[Edited Message Follows]

Sorry for not having that:
Ubuntu 20.04 LTS

Archie

Re: Link Error with DAOS 1.0

Olivier, Jeffrey V
 

Hi Archie,

 

I don’t know if 1.0 has been tried with Ubuntu 20.04 though it does work with 18.04.   I will give it a try locally and see what I can find out.

 

-Jeff

 

From: <daos@daos.groups.io> on behalf of Archie Morningstar <archiemorningstar88@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Wednesday, June 24, 2020 at 12:29 PM
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [daos] Link Error with DAOS 1.0

 

[Edited Message Follows]

Sorry for not having that:
Ubuntu 20.04 LTS

Archie

Re: Intel Optane SSD mounting problem

Farrell, Patrick Arthur
 

Gert,

We're seeing this when we run identify on our 905p SSDs:
"
NVMe Controller at 0000:1a:00.0 [8086:2700]
[...]
Serial Number:                         PHM29226005S480BGN
Model Number:                          INTEL SSDPE21D480GA
[...]
Admin Command Set Attributes
============================
Security Send/Receive:                 Supported
"

However, when I run nvme_manage, I get an error stating that Opal is not supported (the same error DAOS is spitting out, incidentally):

"Please Input PCI Address(domain:bus:dev.func):
0000:1a:00.00
Opal General Usage:

        [1: scan device]
        [2: init - take ownership and activate locking]
        [3: revert tper]
        [4: setup locking range]
        [5: list locking ranges]
        [6: enable user]
        [7: set new password]
        [8: add user to locking range]
        [9: lock/unlock range]
        [10: erase locking range]
        [0: quit]
1
[2020-06-24 14:37:34.452086] nvme_opal.c: 828:opal_discovery0_end: *ERROR*: Opal Not Supported.
"

Any thoughts?

Regards,
-Patrick


From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Gert Pauwels (intel) <gert.pauwels@...>
Sent: Wednesday, June 24, 2020 11:54 AM
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem
 
Hi Tom, Colin,

I'm running Ubuntu 20.04 LTS  (kernel: 5.4.0-37-generic) and compiled DAOS v1.0.0.
I also compiled the latest master as of yesterday, but it did not make a difference.

Any application that can manage Opal 2.0 can be used to check the status of the drive. I used the sedutil-cli, can be found at https://github.com/Drive-Trust-Alliance/sedutil the executable can be found at https://github.com/Drive-Trust-Alliance/sedutil/wiki/Executable-Distributions 
You can run
# sedutil-cli --scan
and
# sedutil-cli --query <device>
it will return useful information about the Opal status of the device.
sedutil-cli will send NVMe security commands and the tool will only work if your SSD is bound the the NVMe driver.

In case the SSD is not bound to the NVMe driver you can manage the drive with the nvme_manage spdk utility you can found in the /daos/_build.external/dev/spdk
# ./examples/nvme/nvme_manage/nvme_manage 
Select 8 to get into the OPAL NVMe Management Options, enter the PCIe address of your SSD for the list you get prompted, select 1 to scan the device. These steps gives you the same results as running the # sedutil-cli --query <device> in case the SSD is bound to the NVMe driver.

Regards,

Gert,


 

Re: Link Error with DAOS 1.0

Archie Morningstar
 
Edited

Sorry for not having that:
Ubuntu 20.04 LTS

Archie

Re: Intel Optane SSD mounting problem

Gert Pauwels (intel)
 

Hi Tom, Colin,

I'm running Ubuntu 20.04 LTS  (kernel: 5.4.0-37-generic) and compiled DAOS v1.0.0.
I also compiled the latest master as of yesterday, but it did not make a difference.

Any application that can manage Opal 2.0 can be used to check the status of the drive. I used the sedutil-cli, can be found at https://github.com/Drive-Trust-Alliance/sedutil the executable can be found at https://github.com/Drive-Trust-Alliance/sedutil/wiki/Executable-Distributions 
You can run
# sedutil-cli --scan
and
# sedutil-cli --query <device>
it will return useful information about the Opal status of the device.
sedutil-cli will send NVMe security commands and the tool will only work if your SSD is bound the the NVMe driver.

In case the SSD is not bound to the NVMe driver you can manage the drive with the nvme_manage spdk utility you can found in the /daos/_build.external/dev/spdk
# ./examples/nvme/nvme_manage/nvme_manage 
Select 8 to get into the OPAL NVMe Management Options, enter the PCIe address of your SSD for the list you get prompted, select 1 to scan the device. These steps gives you the same results as running the # sedutil-cli --query <device> in case the SSD is bound to the NVMe driver.

Regards,

Gert,


 

Re: Intel Optane SSD mounting problem

Colin Ngam
 

Hi Gert,

 

You mentioned below Oparl 2.0 “enabled and configured”. What cmd do you use to see and configure these configuration on the SSD?

 

Thanks.

 

Colin

 

From: <daos@daos.groups.io> on behalf of "Nabarro, Tom" <tom.nabarro@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Wednesday, June 24, 2020 at 6:03 AM
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Hello Gert

 

Out of curiosity , what is your operating system and kernel version?

 

Regards,

Tom Nabarro – DCG/ESAD

M: +44 (0)7786 260986

Skype: tom.nabarro

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of gert.pauwels@...
Sent: Monday, June 22, 2020 9:42 AM
To: daos@daos.groups.io
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Hi Colin, Patrick,

 

The ModelNumber : INTEL SSDPE21D480GA is a 480 GB Intel(R) Optane(TM) SSD 905P Series and supports Opal 2.0.

The last letter in the ModelNumber is a reference to the memory media, the 4 characters before the last letter refer to the capacity.

 

The O for Opal in the ModelNumber is only applicable to the Intel 3D NAND SSDs as there are Models with and without Opal 2.0 support.

 

I compiled DAOS v1.0.0 on my system and did not see the problem. I used a P4800X added it to my daos_server.yml when running:

root@intel-S2600WFD:~/daos# dmg -i storage scan –verbose

It returns the following:

---------

localhost

---------

SCM Namespace Socket ID Capacity

------------- --------- --------

pmem0         0         531 GB

pmem1         1         531 GB

 

NVMe PCI     Model                FW Revision Socket ID Capacity

--------     -----                ----------- --------- --------

0000:d8:00.0 INTEL SSDPE2KE016T8  VDV10170    1         1.6 TB

0000:d9:00.0 INTEL SSDPE21K375GA  E2010485    1         375 GB

 

Next I will try the current DAOS master and also try an Opal 2.0 'enabled and configured' P4800X in DAOS.

 

Regards,

 

Gert,

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Re: Intel Optane SSD mounting problem

Nabarro, Tom
 

Hello Gert

 

Out of curiosity , what is your operating system and kernel version?

 

Regards,

Tom Nabarro – DCG/ESAD

M: +44 (0)7786 260986

Skype: tom.nabarro

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of gert.pauwels@...
Sent: Monday, June 22, 2020 9:42 AM
To: daos@daos.groups.io
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Hi Colin, Patrick,

 

The ModelNumber : INTEL SSDPE21D480GA is a 480 GB Intel(R) Optane(TM) SSD 905P Series and supports Opal 2.0.

The last letter in the ModelNumber is a reference to the memory media, the 4 characters before the last letter refer to the capacity.

 

The O for Opal in the ModelNumber is only applicable to the Intel 3D NAND SSDs as there are Models with and without Opal 2.0 support.

 

I compiled DAOS v1.0.0 on my system and did not see the problem. I used a P4800X added it to my daos_server.yml when running:

root@intel-S2600WFD:~/daos# dmg -i storage scan –verbose

It returns the following:

---------

localhost

---------

SCM Namespace Socket ID Capacity

------------- --------- --------

pmem0         0         531 GB

pmem1         1         531 GB

 

NVMe PCI     Model                FW Revision Socket ID Capacity

--------     -----                ----------- --------- --------

0000:d8:00.0 INTEL SSDPE2KE016T8  VDV10170    1         1.6 TB

0000:d9:00.0 INTEL SSDPE21K375GA  E2010485    1         375 GB

 

Next I will try the current DAOS master and also try an Opal 2.0 'enabled and configured' P4800X in DAOS.

 

Regards,

 

Gert,

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Build changes on master

Olivier, Jeffrey V
 

I’ve just landed a patch to DAOS master that affects developer builds of DAOS.   It is an effort to make things simpler in the long run

 

If you use RPMs this does not apply.  Also, it does not affect USE_INSTALLED option.  It only applies to manual builds with –build-deps=yes enabled.

 

Five major things to note

  1. You will need to install patchelf to do a full build with prerequisites
  2. _build.external has moved to build/external
  3. Prerequisites in a developer build are now installed in $PREFIX/prereq/$TARGET_TYPE/[component]
  4. The install is relocatable (with the exception of daos_admin which needs patching if you move daos)
  5. You will notice a change.  On your first build of daos, you will need to rebuild the prerequisites.

 

This is an extention of a recent change to build DAOS components in $BUILD_ROOT/$BUILD_TYPE/$COMPILER

 

This allows us to switch between build types and compiers without a full rebuild of all components.  This recent change extends this to the prerequisite builds.

 

With this change, the following options have been removed

TARGET_PREFIX (it’s default now)

PREBUILT_PREFIX

[component]_PREBUILT

SRC_PREFIX

[component]_SRC

 

Let me know if you run into any issues.

 

-Jeff

Re: daos_obj_update crashed IO server

Lombardi, Johann
 

Np. Thanks for letting us know.

 

Johann

 

From: <daos@daos.groups.io> on behalf of "Zhang, Jiafu" <jiafu.zhang@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Tuesday 23 June 2020 at 05:54
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [daos] daos_obj_update crashed IO server

 

Hi Johann,

 

It turned out to be my bug which failed the object update and key listing. Sorry for the bothering.

 

Thanks.

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Re: daos_obj_update crashed IO server

Zhang, Jiafu
 

Hi Johann,

 

It turned out to be my bug which failed the object update and key listing. Sorry for the bothering.

 

Thanks.

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Lombardi, Johann
Sent: Friday, June 19, 2020 7:30 PM
To: daos@daos.groups.io
Subject: Re: [daos] daos_obj_update crashed IO server

 

Hi Jiafu,

 

Could you please email me your test case? I will look into it.

Cheers,

Johann

 

From: <daos@daos.groups.io> on behalf of "Zhang, Jiafu" <jiafu.zhang@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Friday 19 June 2020 at 02:52
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [daos] daos_obj_update crashed IO server

 

Hi Johann,

 

In recent commit (6c3eb801d0a9307cdf88b8634ea5a103d0b3542f), the daos io server crash issue is gone when update object with two akeys under the same dkey.

 

But I got another issue, DER_NO_PERM, no permission. And the update with single akey still worked.

 

By the way, listing dkeys worked. But listing akeys didn’t work in this commit with error invalid parameter. The error in server is,

 

06/19-00:23:45.58 jiafu-dev DAOS[348411/348420] bulk ERR  src/cart/src/cart/crt_bulk.c:106 crt_bulk_desc_valid() invalid parameter of bulk_desc (remote_hdl:0x7f13c8b76400,local_hdl:0x7f13c8b7edf0, bulk_op:104, len: 0.

06/19-00:23:45.58 jiafu-dev DAOS[348411/348420] bulk ERR  src/cart/src/cart/crt_bulk.c:205 crt_bulk_transfer() invalid parameter of bulk_desc.

06/19-00:23:45.58 jiafu-dev DAOS[348411/348420] object ERR  src/object/srv_obj.c:390 obj_bulk_transfer() crt_bulk_transfer 0 error (-1003).

06/19-00:23:45.58 jiafu-dev DAOS[351611/351612] object ERR  src/object/cli_shard.c:856 dc_enumerate_cb() rpc 0x7f79e4aca470 RPC 3 failed: DER_INVAL(-1003)

06/19-00:24:32.05 jiafu-dev DAOS[348411/348420] vos  ERR  src/vos/vos_obj.c:1596 vos_obj_iter_aggregate() Removing orphaned array value tree

06/19-00:26:02.06 jiafu-dev DAOS[348411/348420] vos  ERR  src/vos/vos_obj.c:1596 vos_obj_iter_aggregate() Removing orphaned array value tree

06/19-00:26:02.06 jiafu-dev DAOS[348411/348424] vos  ERR  src/vos/vos_obj.c:1596 vos_obj_iter_aggregate() Removing orphaned array value tree

 

Thanks.

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Zhang, Jiafu
Sent: Thursday, June 18, 2020 6:24 PM
To: daos@daos.groups.io
Subject: Re: [daos] daos_obj_update crashed IO server

 

When I reverted to 6c3eb801d0a9307cdf88b8634ea5a103d0b3542f, the format command worked for me.

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Zhang, Jiafu
Sent: Thursday, June 18, 2020 6:18 PM
To: daos@daos.groups.io
Subject: Re: [daos] daos_obj_update crashed IO server

 

The so issue is fixed by adding DAOS lib paths to ldconfig. It’s new behavior? I only needed to set LD_LIBRARY_PATH last time.

 

Another issue raised up when format new DAOS server.

 

$ dmg -i -l jiafu-dev:10001 storage format --reformat

processing system reformat request

ERROR: dmg: System-Query command failed: failed to get MS response: rpc error: code = Unknown desc = empty system membership

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Zhang, Jiafu
Sent: Thursday, June 18, 2020 4:00 PM
To: daos@daos.groups.io
Subject: Re: [daos] daos_obj_update crashed IO server

 

Hi Johann,

 

My version is old.

 

I tried to upgrade from ubuntu 18.04 to ubuntu 20.04. And built latest DAOS code as of now from master. But I got below error when started daos server with non-root user. I’ve followed “Privileged Helper Configuration” in https://daos-stack.github.io/admin/predeployment_check/.

 

ERROR: /usr/bin/daos_admin /usr/bin/daos_admin: error while loading shared libraries: libspdk_log.so.2.0: cannot open shared object file: No such file or directory

ERROR: pbin: code = 4 description = "Unable to decode response after 5 attempts"

 

When I ldd /usr/bin/daos_admin and ldd libspdk_log.so.2.0, both worked.

 

jiafu@jiafu-dev:~/daos$ ldd /usr/bin/daos_admin

        linux-vdso.so.1 (0x00007ffd9bfeb000)

        /usr/lib/jvm/java-1.8.0-openjdk-amd64/jre/lib/amd64/libjsig.so (0x00007f63ef4b1000)

        libspdk_env_dpdk.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_env_dpdk.so.2.0 (0x00007f63ef4a0000)

        librte_mempool.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_mempool.so.20.0 (0x00007f63ef497000)

        librte_bus_pci.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_bus_pci.so.20.0 (0x00007f63ef487000)

        librte_ring.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_ring.so.20.0 (0x00007f63ef482000)

        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f63ef444000)

        libspdk_nvme.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_nvme.so.2.0 (0x00007f63ef40a000)

        libipmctl.so.4 => /usr/lib/x86_64-linux-gnu/libipmctl.so.4 (0x00007f63ef049000)

        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f63eee57000)

        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f63eee51000)

        /lib64/ld-linux-x86-64.so.2 (0x00007f63ef4b9000)

        librte_eal.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_eal.so.20.0 (0x00007f63eed6e000)

        librte_pci.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_pci.so.20.0 (0x00007f63eed67000)

        librte_kvargs.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_kvargs.so.20.0 (0x00007f63eed62000)

        libibverbs.so.1 => /usr/lib/x86_64-linux-gnu/libibverbs.so.1 (0x00007f63eed43000)

        librdmacm.so.1 => /usr/lib/x86_64-linux-gnu/librdmacm.so.1 (0x00007f63eed24000)

        libspdk_log.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_log.so.2.0 (0x00007f63eed1f000)

        libspdk_sock.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_sock.so.2.0 (0x00007f63eed1a000)

        libspdk_util.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_util.so.2.0 (0x00007f63eed07000)

        libndctl.so.6 => /usr/lib/x86_64-linux-gnu/libndctl.so.6 (0x00007f63eece4000)

        libnuma.so.1 => /usr/lib/x86_64-linux-gnu/libnuma.so.1 (0x00007f63eecd7000)

        libnl-route-3.so.200 => /usr/lib/x86_64-linux-gnu/libnl-route-3.so.200 (0x00007f63eec5f000)

        libnl-3.so.200 => /lib/x86_64-linux-gnu/libnl-3.so.200 (0x00007f63eec3c000)

        libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007f63eec31000)

        libdaxctl.so.1 => /usr/lib/x86_64-linux-gnu/libdaxctl.so.1 (0x00007f63eec27000)

        libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007f63eebfb000)

        libkmod.so.2 => /usr/lib/x86_64-linux-gnu/libkmod.so.2 (0x00007f63eebe0000)

        liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f63eebb9000)

        libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f63ee8e1000)

jiafu@jiafu-dev:~/daos$ cd /code-repo/daos-master/daos/install/lib/

jiafu@jiafu-dev:/code-repo/daos-master/daos/install/lib$ ldd libspdk_log.so.2.0

        linux-vdso.so.1 (0x00007fffa928a000)

        /usr/lib/jvm/java-1.8.0-openjdk-amd64/jre/lib/amd64/libjsig.so (0x00007fb538648000)

        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb53843d000)

        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fb538437000)

        /lib64/ld-linux-x86-64.so.2 (0x00007fb538655000)

 

Please help.

 

Thanks.

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Lombardi, Johann
Sent: Wednesday, June 17, 2020 5:48 AM
To: daos@daos.groups.io
Subject: Re: [daos] daos_obj_update crashed IO server

 

Hm, the patch was merged a while ago. How recent is your version of master?

Cheers,

Johann

 

From: <daos@daos.groups.io> on behalf of "Zhang, Jiafu" <jiafu.zhang@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Tuesday 16 June 2020 at 14:49
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [daos] daos_obj_update crashed IO server

 

1.       I get this ticket, DAOS-4391. But it’s for duplicated akey in single update. I’ll try it anyway.

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Zhang, Jiafu
Sent: Tuesday, June 16, 2020 8:12 PM
To: daos@daos.groups.io
Subject: [daos] daos_obj_update crashed IO server

 

Hi Guys,

 

I can call daos_obj_update successfully with one iod/sgl. But daos_obj_update crashed IO server when there were two iods/sgls under the same dkeys. From the server log, I see below error. Can someone tell me what’s going on here?

 

“dkey1”, “akey1” -> succeeded.

“dkey1”, “akey1”, “akey2” -> failed.

 

 

ERROR: daos_io_server:0 06/16-12:04:43.41 jiafu-dev DAOS[28902/28911] vos  EMRG src/vos/vos_ts.h:332 vos_ts_alloc() Assertion failed (info->ti_type & 1) == 0

ERROR: daos_io_server:0 daos_io_server: src/vos/vos_ts.h:332: vos_ts_alloc: Assertion `(info->ti_type & 1) == 0' failed.

ERROR: daos_io_server:0 *** Process 28902 received signal 6 ***

ERROR: daos_io_server:0 Associated errno: Success (0)

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7fcc895b4890]

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7fcc88b8ae97]

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7fcc88b8c801]

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(+0x3039a)[0x7fcc88b7c39a]

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(+0x30412)[0x7fcc88b7c412]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(vos_ilog_ts_cache+0x34c)[0x7fcc8a24583c]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(key_tree_prepare+0x251)[0x7fcc8a21ea41]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(vos_update_end+0x159c)[0x7fcc8a1f070c]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(+0x1e8e3)[0x7fcc7e1da8e3]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(+0x20945)[0x7fcc7e1dc945]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(ds_obj_rw_handler+0x3a0)[0x7fcc7e1dd470]

ERROR: daos_io_server:0 /usr/local/daos/lib/libcart.so.4(+0xa25ee)[0x7fcc8a5245ee]

ERROR: daos_io_server:0 /usr/local/daos/lib/libabt.so.0(+0x10ea0)[0x7fcc89393ea0]

ERROR: daos_io_server:0 /usr/local/daos/lib/libabt.so.0(+0x11601)[0x7fcc89394601]

 

Thanks.

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Re: Link Error with DAOS 1.0

Olivier, Jeffrey V
 

What is the Linux distro and version?

Composed on a tiny keyboard

On Jun 18, 2020, at 7:04 PM, Archie Morningstar <archiemorningstar88@...> wrote:


Hi,

I was trying to build a clean install of DAOS v1.0.0 using the instructions on the DAOS admin guide (https://daos-stack.github.io/admin/installation/) with just the simple commands:
git checkout v1.0.0
git submodule init
git submodule update
scons --config=force --build-deps=yes install

But I'm hitting a linker error. Is there something I'm missing?
I think the libraries are in isal but I'm not familiar with scons to modify the build scripts to add them.

scons: Building targets ...
<lambda>(["build/src/control/bin/daos_admin"], ["src/control/cmd/daos_admin/main.go", "build/src/control/lib/spdk/libnvme_control.a"])
go install -v -ldflags "-X main.daosVersion=1.0.0 -X main.configDir=/home/archie/daos_1.0/daos/install/etc -B 0x10c0ffc53672015e4d3e239af2f447c247b05c02" github.com/daos-stack/daos/src/control/cmd/daos_admin
github.com/daos-stack/daos/src/control/lib/spdk
# github.com/daos-stack/daos/src/control/lib/spdk
/usr/bin/ld: /home/archie/daos_1.0/daos/install/lib/libspdk_env_dpdk.so: undefined reference to `rte_eal_get_configuration'
/usr/bin/ld: /home/archie/daos_1.0/daos/install/lib/libspdk_util.so: undefined reference to `crc16_t10dif'
/usr/bin/ld: /home/archie/daos_1.0/daos/install/lib/libspdk_util.so: undefined reference to `crc16_t10dif_copy'
/usr/bin/ld: /home/archie/daos_1.0/daos/install/lib/libspdk_env_dpdk.so: undefined reference to `lcore_config'
/usr/bin/ld: /home/archie/daos_1.0/daos/install/lib/libspdk_util.so: undefined reference to `crc32_iscsi'
collect2: error: ld returned 1 exit status
scons: *** Error 2
scons: *** [build/src/control/bin/daos_admin] Error 2
scons: building terminated because of errors.

Archie

Re: Intel Optane SSD mounting problem

Gert Pauwels (intel)
 

Hi Colin, Patrick,

 

The ModelNumber : INTEL SSDPE21D480GA is a 480 GB Intel(R) Optane(TM) SSD 905P Series and supports Opal 2.0.

The last letter in the ModelNumber is a reference to the memory media, the 4 characters before the last letter refer to the capacity.

 

The O for Opal in the ModelNumber is only applicable to the Intel 3D NAND SSDs as there are Models with and without Opal 2.0 support.

 

I compiled DAOS v1.0.0 on my system and did not see the problem. I used a P4800X added it to my daos_server.yml when running:

root@intel-S2600WFD:~/daos# dmg -i storage scan –verbose

It returns the following:

---------

localhost

---------

SCM Namespace Socket ID Capacity

------------- --------- --------

pmem0         0         531 GB

pmem1         1         531 GB

 

NVMe PCI     Model                FW Revision Socket ID Capacity

--------     -----                ----------- --------- --------

0000:d8:00.0 INTEL SSDPE2KE016T8  VDV10170    1         1.6 TB

0000:d9:00.0 INTEL SSDPE21K375GA  E2010485    1         375 GB

 

Next I will try the current DAOS master and also try an Opal 2.0 'enabled and configured' P4800X in DAOS.

 

Regards,

 

Gert,

Re: daos_obj_update crashed IO server

Nabarro, Tom
 

The bug with the storage reformat command has been fixed by Mike on master.

 

Apologies for the inconvenience.

 

Regards,

Tom

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Lombardi, Johann
Sent: Friday, June 19, 2020 12:30 PM
To: daos@daos.groups.io
Subject: Re: [daos] daos_obj_update crashed IO server

 

Hi Jiafu,

 

Could you please email me your test case? I will look into it.

Cheers,

Johann

 

From: <daos@daos.groups.io> on behalf of "Zhang, Jiafu" <jiafu.zhang@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Friday 19 June 2020 at 02:52
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [daos] daos_obj_update crashed IO server

 

Hi Johann,

 

In recent commit (6c3eb801d0a9307cdf88b8634ea5a103d0b3542f), the daos io server crash issue is gone when update object with two akeys under the same dkey.

 

But I got another issue, DER_NO_PERM, no permission. And the update with single akey still worked.

 

By the way, listing dkeys worked. But listing akeys didn’t work in this commit with error invalid parameter. The error in server is,

 

06/19-00:23:45.58 jiafu-dev DAOS[348411/348420] bulk ERR  src/cart/src/cart/crt_bulk.c:106 crt_bulk_desc_valid() invalid parameter of bulk_desc (remote_hdl:0x7f13c8b76400,local_hdl:0x7f13c8b7edf0, bulk_op:104, len: 0.

06/19-00:23:45.58 jiafu-dev DAOS[348411/348420] bulk ERR  src/cart/src/cart/crt_bulk.c:205 crt_bulk_transfer() invalid parameter of bulk_desc.

06/19-00:23:45.58 jiafu-dev DAOS[348411/348420] object ERR  src/object/srv_obj.c:390 obj_bulk_transfer() crt_bulk_transfer 0 error (-1003).

06/19-00:23:45.58 jiafu-dev DAOS[351611/351612] object ERR  src/object/cli_shard.c:856 dc_enumerate_cb() rpc 0x7f79e4aca470 RPC 3 failed: DER_INVAL(-1003)

06/19-00:24:32.05 jiafu-dev DAOS[348411/348420] vos  ERR  src/vos/vos_obj.c:1596 vos_obj_iter_aggregate() Removing orphaned array value tree

06/19-00:26:02.06 jiafu-dev DAOS[348411/348420] vos  ERR  src/vos/vos_obj.c:1596 vos_obj_iter_aggregate() Removing orphaned array value tree

06/19-00:26:02.06 jiafu-dev DAOS[348411/348424] vos  ERR  src/vos/vos_obj.c:1596 vos_obj_iter_aggregate() Removing orphaned array value tree

 

Thanks.

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Zhang, Jiafu
Sent: Thursday, June 18, 2020 6:24 PM
To: daos@daos.groups.io
Subject: Re: [daos] daos_obj_update crashed IO server

 

When I reverted to 6c3eb801d0a9307cdf88b8634ea5a103d0b3542f, the format command worked for me.

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Zhang, Jiafu
Sent: Thursday, June 18, 2020 6:18 PM
To: daos@daos.groups.io
Subject: Re: [daos] daos_obj_update crashed IO server

 

The so issue is fixed by adding DAOS lib paths to ldconfig. It’s new behavior? I only needed to set LD_LIBRARY_PATH last time.

 

Another issue raised up when format new DAOS server.

 

$ dmg -i -l jiafu-dev:10001 storage format --reformat

processing system reformat request

ERROR: dmg: System-Query command failed: failed to get MS response: rpc error: code = Unknown desc = empty system membership

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Zhang, Jiafu
Sent: Thursday, June 18, 2020 4:00 PM
To: daos@daos.groups.io
Subject: Re: [daos] daos_obj_update crashed IO server

 

Hi Johann,

 

My version is old.

 

I tried to upgrade from ubuntu 18.04 to ubuntu 20.04. And built latest DAOS code as of now from master. But I got below error when started daos server with non-root user. I’ve followed “Privileged Helper Configuration” in https://daos-stack.github.io/admin/predeployment_check/.

 

ERROR: /usr/bin/daos_admin /usr/bin/daos_admin: error while loading shared libraries: libspdk_log.so.2.0: cannot open shared object file: No such file or directory

ERROR: pbin: code = 4 description = "Unable to decode response after 5 attempts"

 

When I ldd /usr/bin/daos_admin and ldd libspdk_log.so.2.0, both worked.

 

jiafu@jiafu-dev:~/daos$ ldd /usr/bin/daos_admin

        linux-vdso.so.1 (0x00007ffd9bfeb000)

        /usr/lib/jvm/java-1.8.0-openjdk-amd64/jre/lib/amd64/libjsig.so (0x00007f63ef4b1000)

        libspdk_env_dpdk.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_env_dpdk.so.2.0 (0x00007f63ef4a0000)

        librte_mempool.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_mempool.so.20.0 (0x00007f63ef497000)

        librte_bus_pci.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_bus_pci.so.20.0 (0x00007f63ef487000)

        librte_ring.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_ring.so.20.0 (0x00007f63ef482000)

        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f63ef444000)

        libspdk_nvme.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_nvme.so.2.0 (0x00007f63ef40a000)

        libipmctl.so.4 => /usr/lib/x86_64-linux-gnu/libipmctl.so.4 (0x00007f63ef049000)

        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f63eee57000)

        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f63eee51000)

        /lib64/ld-linux-x86-64.so.2 (0x00007f63ef4b9000)

        librte_eal.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_eal.so.20.0 (0x00007f63eed6e000)

        librte_pci.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_pci.so.20.0 (0x00007f63eed67000)

        librte_kvargs.so.20.0 => /code-repo/daos-master/daos/install/lib/librte_kvargs.so.20.0 (0x00007f63eed62000)

        libibverbs.so.1 => /usr/lib/x86_64-linux-gnu/libibverbs.so.1 (0x00007f63eed43000)

        librdmacm.so.1 => /usr/lib/x86_64-linux-gnu/librdmacm.so.1 (0x00007f63eed24000)

        libspdk_log.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_log.so.2.0 (0x00007f63eed1f000)

        libspdk_sock.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_sock.so.2.0 (0x00007f63eed1a000)

        libspdk_util.so.2.0 => /code-repo/daos-master/daos/install/lib/libspdk_util.so.2.0 (0x00007f63eed07000)

        libndctl.so.6 => /usr/lib/x86_64-linux-gnu/libndctl.so.6 (0x00007f63eece4000)

        libnuma.so.1 => /usr/lib/x86_64-linux-gnu/libnuma.so.1 (0x00007f63eecd7000)

        libnl-route-3.so.200 => /usr/lib/x86_64-linux-gnu/libnl-route-3.so.200 (0x00007f63eec5f000)

        libnl-3.so.200 => /lib/x86_64-linux-gnu/libnl-3.so.200 (0x00007f63eec3c000)

        libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007f63eec31000)

        libdaxctl.so.1 => /usr/lib/x86_64-linux-gnu/libdaxctl.so.1 (0x00007f63eec27000)

        libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007f63eebfb000)

        libkmod.so.2 => /usr/lib/x86_64-linux-gnu/libkmod.so.2 (0x00007f63eebe0000)

        liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f63eebb9000)

        libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f63ee8e1000)

jiafu@jiafu-dev:~/daos$ cd /code-repo/daos-master/daos/install/lib/

jiafu@jiafu-dev:/code-repo/daos-master/daos/install/lib$ ldd libspdk_log.so.2.0

        linux-vdso.so.1 (0x00007fffa928a000)

        /usr/lib/jvm/java-1.8.0-openjdk-amd64/jre/lib/amd64/libjsig.so (0x00007fb538648000)

        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb53843d000)

        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fb538437000)

        /lib64/ld-linux-x86-64.so.2 (0x00007fb538655000)

 

Please help.

 

Thanks.

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Lombardi, Johann
Sent: Wednesday, June 17, 2020 5:48 AM
To: daos@daos.groups.io
Subject: Re: [daos] daos_obj_update crashed IO server

 

Hm, the patch was merged a while ago. How recent is your version of master?

Cheers,

Johann

 

From: <daos@daos.groups.io> on behalf of "Zhang, Jiafu" <jiafu.zhang@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Tuesday 16 June 2020 at 14:49
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [daos] daos_obj_update crashed IO server

 

1.       I get this ticket, DAOS-4391. But it’s for duplicated akey in single update. I’ll try it anyway.

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Zhang, Jiafu
Sent: Tuesday, June 16, 2020 8:12 PM
To: daos@daos.groups.io
Subject: [daos] daos_obj_update crashed IO server

 

Hi Guys,

 

I can call daos_obj_update successfully with one iod/sgl. But daos_obj_update crashed IO server when there were two iods/sgls under the same dkeys. From the server log, I see below error. Can someone tell me what’s going on here?

 

“dkey1”, “akey1” -> succeeded.

“dkey1”, “akey1”, “akey2” -> failed.

 

 

ERROR: daos_io_server:0 06/16-12:04:43.41 jiafu-dev DAOS[28902/28911] vos  EMRG src/vos/vos_ts.h:332 vos_ts_alloc() Assertion failed (info->ti_type & 1) == 0

ERROR: daos_io_server:0 daos_io_server: src/vos/vos_ts.h:332: vos_ts_alloc: Assertion `(info->ti_type & 1) == 0' failed.

ERROR: daos_io_server:0 *** Process 28902 received signal 6 ***

ERROR: daos_io_server:0 Associated errno: Success (0)

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7fcc895b4890]

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7fcc88b8ae97]

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7fcc88b8c801]

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(+0x3039a)[0x7fcc88b7c39a]

ERROR: daos_io_server:0 /lib/x86_64-linux-gnu/libc.so.6(+0x30412)[0x7fcc88b7c412]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(vos_ilog_ts_cache+0x34c)[0x7fcc8a24583c]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(key_tree_prepare+0x251)[0x7fcc8a21ea41]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libvos_srv.so(vos_update_end+0x159c)[0x7fcc8a1f070c]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(+0x1e8e3)[0x7fcc7e1da8e3]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(+0x20945)[0x7fcc7e1dc945]

ERROR: daos_io_server:0 /usr/local/daos/bin/../lib64/daos_srv/libobj.so(ds_obj_rw_handler+0x3a0)[0x7fcc7e1dd470]

ERROR: daos_io_server:0 /usr/local/daos/lib/libcart.so.4(+0xa25ee)[0x7fcc8a5245ee]

ERROR: daos_io_server:0 /usr/local/daos/lib/libabt.so.0(+0x10ea0)[0x7fcc89393ea0]

ERROR: daos_io_server:0 /usr/local/daos/lib/libabt.so.0(+0x11601)[0x7fcc89394601]

 

Thanks.

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Re: Intel Optane SSD mounting problem

Colin Ngam
 

Greetings Gert,

 

The Model Number is: ModelNumber : INTEL SSDPE21D480GA

 

Is that the Model that supports OPAL? Is that ‘0’ above?

 

Thanks.

 

Colin

 

PS-Is identify same as spdk_nvme_identify?

 

From: <daos@daos.groups.io> on behalf of "Farrell, Patrick Arthur" <patrick.farrell@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Friday, June 19, 2020 at 10:49 AM
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Gert,

 

Thanks - I'll be curious what results you see.  FYI, we are using a very recent version of master, and didn't see this problem a few months back, so if you're using a different branch or older version, you may not see what we're seeing.

 

Regards,

-Patrick


From: daos@daos.groups.io <daos@daos.groups.io> on behalf of gert.pauwels@... <gert.pauwels@...>
Sent: Friday, June 19, 2020 10:34 AM
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Hi Patrick, Sydney,

Both the Intel(r) Optane(TM) SSD 905P as the Intel(r) Optane(TM) SSD P4800X do support Opal 2.0.
You can see this with the identify tool in spdk. I ran the following on the P4800X in my system:
~/daos/_build.external/spdk/examples/nvme/identify# ./identify
...

Admin Command Set Attributes

============================

Security Send/Receive:                 Supported


I ran the following on the Intel(r) SSD DC P4610 in my system

...
~/daos/_build.external/spdk/examples/nvme/identify# ./identify
...

Admin Command Set Attributes

============================

Security Send/Receive:                 Not Supported
...
FYI the P4510 and P4610 exists in two version, one with Opal support and the other without Opal. You can see this in the model string as the one with Opal has an "O" in it, but also by querying the Identify.

I'll add the P4800X to my DAOS v1.0 system and see what happens.

Regards,

Gert


 

Re: Intel Optane SSD mounting problem

Farrell, Patrick Arthur
 

Gert,

Thanks - I'll be curious what results you see.  FYI, we are using a very recent version of master, and didn't see this problem a few months back, so if you're using a different branch or older version, you may not see what we're seeing.

Regards,
-Patrick


From: daos@daos.groups.io <daos@daos.groups.io> on behalf of gert.pauwels@... <gert.pauwels@...>
Sent: Friday, June 19, 2020 10:34 AM
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem
 
Hi Patrick, Sydney,

Both the Intel(r) Optane(TM) SSD 905P as the Intel(r) Optane(TM) SSD P4800X do support Opal 2.0.
You can see this with the identify tool in spdk. I ran the following on the P4800X in my system:
~/daos/_build.external/spdk/examples/nvme/identify# ./identify
...
Admin Command Set Attributes
============================
Security Send/Receive:                 Supported

I ran the following on the Intel(r) SSD DC P4610 in my system
...
~/daos/_build.external/spdk/examples/nvme/identify# ./identify
...
Admin Command Set Attributes
============================
Security Send/Receive:                 Not Supported
...
FYI the P4510 and P4610 exists in two version, one with Opal support and the other without Opal. You can see this in the model string as the one with Opal has an "O" in it, but also by querying the Identify.

I'll add the P4800X to my DAOS v1.0 system and see what happens.

Regards,

Gert


 

Re: Intel Optane SSD mounting problem

Gert Pauwels (intel)
 

Hi Patrick, Sydney,

Both the Intel(r) Optane(TM) SSD 905P as the Intel(r) Optane(TM) SSD P4800X do support Opal 2.0.
You can see this with the identify tool in spdk. I ran the following on the P4800X in my system:
~/daos/_build.external/spdk/examples/nvme/identify# ./identify
...
Admin Command Set Attributes
============================
Security Send/Receive:                 Supported

I ran the following on the Intel(r) SSD DC P4610 in my system
...
~/daos/_build.external/spdk/examples/nvme/identify# ./identify
...
Admin Command Set Attributes
============================
Security Send/Receive:                 Not Supported
...
FYI the P4510 and P4610 exists in two version, one with Opal support and the other without Opal. You can see this in the model string as the one with Opal has an "O" in it, but also by querying the Identify.

I'll add the P4800X to my DAOS v1.0 system and see what happens.

Regards,

Gert


 

Re: Intel Optane SSD mounting problem

Farrell, Patrick Arthur
 

Note also that these are indeed the same drives we used previously, and we have not made any drive firmware or settings changes.  (We're open to trying some if required - Just noting that we have not made any.)

-Patrick


From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Colin Ngam <colin.ngam@...>
Sent: Friday, June 19, 2020 9:26 AM
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem
 

Greetings Sydney,

 

Our drives are:

 

Product : MansionBeach

ProductFamily : Intel(R) Optane(TM) SSD 905P Series

 

I cannot find that these drives support OPAL? Should we even be in this code segment?

 

Is there something in the drive firmware or BIOS that is not correctly set? Any guidance will be most helpful.

 

Thanks.

 

Colin

 

 

From: <daos@daos.groups.io> on behalf of "Vanda, Sydney M" <sydney.m.vanda@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Thursday, June 18, 2020 at 4:36 PM
To: "daos@daos.groups.io" <daos@daos.groups.io>, "Farrell, Patrick Arthur" <patrick.farrell@...>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Hi Patrick,

 

I’m assuming the Intel Optane SSD you are trying to use has Opal support and is enabled?  Is this the same SSD you have ran DAOS successfully with before with a commit previous to the one you mentioned?

 

It seems in reading some of the SPDK code (https://review.spdk.io/gerrit/plugins/gitiles/spdk/spdk/+/f428cd732cf7653adac2c6d321c832112eb81bdd/lib/nvme/nvme_opal.c)

that SPDK only supports certain feature codes. Seems like this is the error that you are getting below:

 

                        switch (feat_code) {

                        case FEATURECODE_TPER:

                                    opal_check_tper(dev, cpos);

                                    break;

                        case FEATURECODE_SINGLEUSER:

                                    single_user = opal_check_sum(dev, cpos);

                                    break;

                        case FEATURECODE_GEOMETRY:

                                    opal_check_geometry(dev, cpos);

                                    break;

                        case FEATURECODE_LOCKING:

                                    opal_check_lock(dev, cpos);

                                    break;

                        case FEATURECODE_DATASTORE:

                                    opal_check_datastore(dev, cpos);

                                    break;

                        case FEATURECODE_OPALV100:

                                    comid = opal_get_comid_v100(dev, cpos);

                                    supported = true;

                                    break;

                        case FEATURECODE_OPALV200:

                                    comid = opal_get_comid_v200(dev, cpos);

                                    supported = true;

                                    break;

                        default:

                                    SPDK_INFOLOG(SPDK_LOG_OPAL, "Unknow feature code: %d\n", feat_code);

                        }

                        cpos += feat_hdr->length + sizeof(*feat_hdr);

            }

            if (supported == false) {

                        SPDK_ERRLOG("Opal Not Supported.\n");

                        return -ENOTSUP;

            }

 

Thanks,

Sydney

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Farrell, Patrick Arthur
Sent: Thursday, June 18, 2020 9:40 AM
To: daos@daos.groups.io
Subject: [daos] Intel Optane SSD mounting problem

 

Good morning,

 

We are attempting to run DAOS using an Intel Optane SSD.  We've done this successfully in the past, but current master (commit 03cefa25cdf94a6c8582b6964d3b395be48fd231) isn't able to use the SSD.

 

It's giving a very strange error, that seems to be confusion in SPDK:

06/18-08:56:27.315649 delphi-003 DAOS[117955/118009] bio  INFO src/bio/bio_xstream.c:1075 bio_xsctxt_alloc() Initialize NVMe context, tgt_id:0, init_thread:(nil)

nvme_opal.c: 922:opal_discovery0_end: *ERROR*: Opal Not Supported.

 

And then fails to do anything with the SSD.  (The server starts using PMEM, but the SSD is not present.)

 

Opal appears to be a reference to the Intel Opal SSDs with encryption support, and seems to just be a matter of confusion on the part of SPDK.

 

Curious if anyone has insight on this.

 

Thanks.

-Patrick Farrell

Re: Intel Optane SSD mounting problem

Colin Ngam
 

Greetings Sydney,

 

Our drives are:

 

Product : MansionBeach

ProductFamily : Intel(R) Optane(TM) SSD 905P Series

 

I cannot find that these drives support OPAL? Should we even be in this code segment?

 

Is there something in the drive firmware or BIOS that is not correctly set? Any guidance will be most helpful.

 

Thanks.

 

Colin

 

 

From: <daos@daos.groups.io> on behalf of "Vanda, Sydney M" <sydney.m.vanda@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Thursday, June 18, 2020 at 4:36 PM
To: "daos@daos.groups.io" <daos@daos.groups.io>, "Farrell, Patrick Arthur" <patrick.farrell@...>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Hi Patrick,

 

I’m assuming the Intel Optane SSD you are trying to use has Opal support and is enabled?  Is this the same SSD you have ran DAOS successfully with before with a commit previous to the one you mentioned?

 

It seems in reading some of the SPDK code (https://review.spdk.io/gerrit/plugins/gitiles/spdk/spdk/+/f428cd732cf7653adac2c6d321c832112eb81bdd/lib/nvme/nvme_opal.c)

that SPDK only supports certain feature codes. Seems like this is the error that you are getting below:

 

                        switch (feat_code) {

                        case FEATURECODE_TPER:

                                    opal_check_tper(dev, cpos);

                                    break;

                        case FEATURECODE_SINGLEUSER:

                                    single_user = opal_check_sum(dev, cpos);

                                    break;

                        case FEATURECODE_GEOMETRY:

                                    opal_check_geometry(dev, cpos);

                                    break;

                        case FEATURECODE_LOCKING:

                                    opal_check_lock(dev, cpos);

                                    break;

                        case FEATURECODE_DATASTORE:

                                    opal_check_datastore(dev, cpos);

                                    break;

                        case FEATURECODE_OPALV100:

                                    comid = opal_get_comid_v100(dev, cpos);

                                    supported = true;

                                    break;

                        case FEATURECODE_OPALV200:

                                    comid = opal_get_comid_v200(dev, cpos);

                                    supported = true;

                                    break;

                        default:

                                    SPDK_INFOLOG(SPDK_LOG_OPAL, "Unknow feature code: %d\n", feat_code);

                        }

                        cpos += feat_hdr->length + sizeof(*feat_hdr);

            }

            if (supported == false) {

                        SPDK_ERRLOG("Opal Not Supported.\n");

                        return -ENOTSUP;

            }

 

Thanks,

Sydney

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Farrell, Patrick Arthur
Sent: Thursday, June 18, 2020 9:40 AM
To: daos@daos.groups.io
Subject: [daos] Intel Optane SSD mounting problem

 

Good morning,

 

We are attempting to run DAOS using an Intel Optane SSD.  We've done this successfully in the past, but current master (commit 03cefa25cdf94a6c8582b6964d3b395be48fd231) isn't able to use the SSD.

 

It's giving a very strange error, that seems to be confusion in SPDK:

06/18-08:56:27.315649 delphi-003 DAOS[117955/118009] bio  INFO src/bio/bio_xstream.c:1075 bio_xsctxt_alloc() Initialize NVMe context, tgt_id:0, init_thread:(nil)

nvme_opal.c: 922:opal_discovery0_end: *ERROR*: Opal Not Supported.

 

And then fails to do anything with the SSD.  (The server starts using PMEM, but the SSD is not present.)

 

Opal appears to be a reference to the Intel Opal SSDs with encryption support, and seems to just be a matter of confusion on the part of SPDK.

 

Curious if anyone has insight on this.

 

Thanks.

-Patrick Farrell