Topics

Intel Optane SSD mounting problem

Farrell, Patrick Arthur
 

Good morning,

We are attempting to run DAOS using an Intel Optane SSD.  We've done this successfully in the past, but current master (commit 03cefa25cdf94a6c8582b6964d3b395be48fd231) isn't able to use the SSD.

It's giving a very strange error, that seems to be confusion in SPDK:
06/18-08:56:27.315649 delphi-003 DAOS[117955/118009] bio  INFO src/bio/bio_xstream.c:1075 bio_xsctxt_alloc() Initialize NVMe context, tgt_id:0, init_thread:(nil)
nvme_opal.c: 922:opal_discovery0_end: *ERROR*: Opal Not Supported.

And then fails to do anything with the SSD.  (The server starts using PMEM, but the SSD is not present.)

Opal appears to be a reference to the Intel Opal SSDs with encryption support, and seems to just be a matter of confusion on the part of SPDK.

Curious if anyone has insight on this.

Thanks.
-Patrick Farrell

Vanda, Sydney M
 

Hi Patrick,

 

I’m assuming the Intel Optane SSD you are trying to use has Opal support and is enabled?  Is this the same SSD you have ran DAOS successfully with before with a commit previous to the one you mentioned?

 

It seems in reading some of the SPDK code (https://review.spdk.io/gerrit/plugins/gitiles/spdk/spdk/+/f428cd732cf7653adac2c6d321c832112eb81bdd/lib/nvme/nvme_opal.c)

that SPDK only supports certain feature codes. Seems like this is the error that you are getting below:

 

                        switch (feat_code) {

                        case FEATURECODE_TPER:

                                    opal_check_tper(dev, cpos);

                                    break;

                        case FEATURECODE_SINGLEUSER:

                                    single_user = opal_check_sum(dev, cpos);

                                    break;

                        case FEATURECODE_GEOMETRY:

                                    opal_check_geometry(dev, cpos);

                                    break;

                        case FEATURECODE_LOCKING:

                                    opal_check_lock(dev, cpos);

                                    break;

                        case FEATURECODE_DATASTORE:

                                    opal_check_datastore(dev, cpos);

                                    break;

                        case FEATURECODE_OPALV100:

                                    comid = opal_get_comid_v100(dev, cpos);

                                    supported = true;

                                    break;

                        case FEATURECODE_OPALV200:

                                    comid = opal_get_comid_v200(dev, cpos);

                                    supported = true;

                                    break;

                        default:

                                    SPDK_INFOLOG(SPDK_LOG_OPAL, "Unknow feature code: %d\n", feat_code);

                        }

                        cpos += feat_hdr->length + sizeof(*feat_hdr);

            }

            if (supported == false) {

                        SPDK_ERRLOG("Opal Not Supported.\n");

                        return -ENOTSUP;

            }

 

Thanks,

Sydney

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Farrell, Patrick Arthur
Sent: Thursday, June 18, 2020 9:40 AM
To: daos@daos.groups.io
Subject: [daos] Intel Optane SSD mounting problem

 

Good morning,

 

We are attempting to run DAOS using an Intel Optane SSD.  We've done this successfully in the past, but current master (commit 03cefa25cdf94a6c8582b6964d3b395be48fd231) isn't able to use the SSD.

 

It's giving a very strange error, that seems to be confusion in SPDK:

06/18-08:56:27.315649 delphi-003 DAOS[117955/118009] bio  INFO src/bio/bio_xstream.c:1075 bio_xsctxt_alloc() Initialize NVMe context, tgt_id:0, init_thread:(nil)

nvme_opal.c: 922:opal_discovery0_end: *ERROR*: Opal Not Supported.

 

And then fails to do anything with the SSD.  (The server starts using PMEM, but the SSD is not present.)

 

Opal appears to be a reference to the Intel Opal SSDs with encryption support, and seems to just be a matter of confusion on the part of SPDK.

 

Curious if anyone has insight on this.

 

Thanks.

-Patrick Farrell

Colin Ngam
 

Greetings Sydney,

 

Our drives are:

 

Product : MansionBeach

ProductFamily : Intel(R) Optane(TM) SSD 905P Series

 

I cannot find that these drives support OPAL? Should we even be in this code segment?

 

Is there something in the drive firmware or BIOS that is not correctly set? Any guidance will be most helpful.

 

Thanks.

 

Colin

 

 

From: <daos@daos.groups.io> on behalf of "Vanda, Sydney M" <sydney.m.vanda@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Thursday, June 18, 2020 at 4:36 PM
To: "daos@daos.groups.io" <daos@daos.groups.io>, "Farrell, Patrick Arthur" <patrick.farrell@...>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Hi Patrick,

 

I’m assuming the Intel Optane SSD you are trying to use has Opal support and is enabled?  Is this the same SSD you have ran DAOS successfully with before with a commit previous to the one you mentioned?

 

It seems in reading some of the SPDK code (https://review.spdk.io/gerrit/plugins/gitiles/spdk/spdk/+/f428cd732cf7653adac2c6d321c832112eb81bdd/lib/nvme/nvme_opal.c)

that SPDK only supports certain feature codes. Seems like this is the error that you are getting below:

 

                        switch (feat_code) {

                        case FEATURECODE_TPER:

                                    opal_check_tper(dev, cpos);

                                    break;

                        case FEATURECODE_SINGLEUSER:

                                    single_user = opal_check_sum(dev, cpos);

                                    break;

                        case FEATURECODE_GEOMETRY:

                                    opal_check_geometry(dev, cpos);

                                    break;

                        case FEATURECODE_LOCKING:

                                    opal_check_lock(dev, cpos);

                                    break;

                        case FEATURECODE_DATASTORE:

                                    opal_check_datastore(dev, cpos);

                                    break;

                        case FEATURECODE_OPALV100:

                                    comid = opal_get_comid_v100(dev, cpos);

                                    supported = true;

                                    break;

                        case FEATURECODE_OPALV200:

                                    comid = opal_get_comid_v200(dev, cpos);

                                    supported = true;

                                    break;

                        default:

                                    SPDK_INFOLOG(SPDK_LOG_OPAL, "Unknow feature code: %d\n", feat_code);

                        }

                        cpos += feat_hdr->length + sizeof(*feat_hdr);

            }

            if (supported == false) {

                        SPDK_ERRLOG("Opal Not Supported.\n");

                        return -ENOTSUP;

            }

 

Thanks,

Sydney

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Farrell, Patrick Arthur
Sent: Thursday, June 18, 2020 9:40 AM
To: daos@daos.groups.io
Subject: [daos] Intel Optane SSD mounting problem

 

Good morning,

 

We are attempting to run DAOS using an Intel Optane SSD.  We've done this successfully in the past, but current master (commit 03cefa25cdf94a6c8582b6964d3b395be48fd231) isn't able to use the SSD.

 

It's giving a very strange error, that seems to be confusion in SPDK:

06/18-08:56:27.315649 delphi-003 DAOS[117955/118009] bio  INFO src/bio/bio_xstream.c:1075 bio_xsctxt_alloc() Initialize NVMe context, tgt_id:0, init_thread:(nil)

nvme_opal.c: 922:opal_discovery0_end: *ERROR*: Opal Not Supported.

 

And then fails to do anything with the SSD.  (The server starts using PMEM, but the SSD is not present.)

 

Opal appears to be a reference to the Intel Opal SSDs with encryption support, and seems to just be a matter of confusion on the part of SPDK.

 

Curious if anyone has insight on this.

 

Thanks.

-Patrick Farrell

Farrell, Patrick Arthur
 

Note also that these are indeed the same drives we used previously, and we have not made any drive firmware or settings changes.  (We're open to trying some if required - Just noting that we have not made any.)

-Patrick


From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Colin Ngam <colin.ngam@...>
Sent: Friday, June 19, 2020 9:26 AM
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem
 

Greetings Sydney,

 

Our drives are:

 

Product : MansionBeach

ProductFamily : Intel(R) Optane(TM) SSD 905P Series

 

I cannot find that these drives support OPAL? Should we even be in this code segment?

 

Is there something in the drive firmware or BIOS that is not correctly set? Any guidance will be most helpful.

 

Thanks.

 

Colin

 

 

From: <daos@daos.groups.io> on behalf of "Vanda, Sydney M" <sydney.m.vanda@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Thursday, June 18, 2020 at 4:36 PM
To: "daos@daos.groups.io" <daos@daos.groups.io>, "Farrell, Patrick Arthur" <patrick.farrell@...>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Hi Patrick,

 

I’m assuming the Intel Optane SSD you are trying to use has Opal support and is enabled?  Is this the same SSD you have ran DAOS successfully with before with a commit previous to the one you mentioned?

 

It seems in reading some of the SPDK code (https://review.spdk.io/gerrit/plugins/gitiles/spdk/spdk/+/f428cd732cf7653adac2c6d321c832112eb81bdd/lib/nvme/nvme_opal.c)

that SPDK only supports certain feature codes. Seems like this is the error that you are getting below:

 

                        switch (feat_code) {

                        case FEATURECODE_TPER:

                                    opal_check_tper(dev, cpos);

                                    break;

                        case FEATURECODE_SINGLEUSER:

                                    single_user = opal_check_sum(dev, cpos);

                                    break;

                        case FEATURECODE_GEOMETRY:

                                    opal_check_geometry(dev, cpos);

                                    break;

                        case FEATURECODE_LOCKING:

                                    opal_check_lock(dev, cpos);

                                    break;

                        case FEATURECODE_DATASTORE:

                                    opal_check_datastore(dev, cpos);

                                    break;

                        case FEATURECODE_OPALV100:

                                    comid = opal_get_comid_v100(dev, cpos);

                                    supported = true;

                                    break;

                        case FEATURECODE_OPALV200:

                                    comid = opal_get_comid_v200(dev, cpos);

                                    supported = true;

                                    break;

                        default:

                                    SPDK_INFOLOG(SPDK_LOG_OPAL, "Unknow feature code: %d\n", feat_code);

                        }

                        cpos += feat_hdr->length + sizeof(*feat_hdr);

            }

            if (supported == false) {

                        SPDK_ERRLOG("Opal Not Supported.\n");

                        return -ENOTSUP;

            }

 

Thanks,

Sydney

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Farrell, Patrick Arthur
Sent: Thursday, June 18, 2020 9:40 AM
To: daos@daos.groups.io
Subject: [daos] Intel Optane SSD mounting problem

 

Good morning,

 

We are attempting to run DAOS using an Intel Optane SSD.  We've done this successfully in the past, but current master (commit 03cefa25cdf94a6c8582b6964d3b395be48fd231) isn't able to use the SSD.

 

It's giving a very strange error, that seems to be confusion in SPDK:

06/18-08:56:27.315649 delphi-003 DAOS[117955/118009] bio  INFO src/bio/bio_xstream.c:1075 bio_xsctxt_alloc() Initialize NVMe context, tgt_id:0, init_thread:(nil)

nvme_opal.c: 922:opal_discovery0_end: *ERROR*: Opal Not Supported.

 

And then fails to do anything with the SSD.  (The server starts using PMEM, but the SSD is not present.)

 

Opal appears to be a reference to the Intel Opal SSDs with encryption support, and seems to just be a matter of confusion on the part of SPDK.

 

Curious if anyone has insight on this.

 

Thanks.

-Patrick Farrell

Gert Pauwels (intel)
 

Hi Patrick, Sydney,

Both the Intel(r) Optane(TM) SSD 905P as the Intel(r) Optane(TM) SSD P4800X do support Opal 2.0.
You can see this with the identify tool in spdk. I ran the following on the P4800X in my system:
~/daos/_build.external/spdk/examples/nvme/identify# ./identify
...
Admin Command Set Attributes
============================
Security Send/Receive:                 Supported

I ran the following on the Intel(r) SSD DC P4610 in my system
...
~/daos/_build.external/spdk/examples/nvme/identify# ./identify
...
Admin Command Set Attributes
============================
Security Send/Receive:                 Not Supported
...
FYI the P4510 and P4610 exists in two version, one with Opal support and the other without Opal. You can see this in the model string as the one with Opal has an "O" in it, but also by querying the Identify.

I'll add the P4800X to my DAOS v1.0 system and see what happens.

Regards,

Gert


 

Farrell, Patrick Arthur
 

Gert,

Thanks - I'll be curious what results you see.  FYI, we are using a very recent version of master, and didn't see this problem a few months back, so if you're using a different branch or older version, you may not see what we're seeing.

Regards,
-Patrick


From: daos@daos.groups.io <daos@daos.groups.io> on behalf of gert.pauwels@... <gert.pauwels@...>
Sent: Friday, June 19, 2020 10:34 AM
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem
 
Hi Patrick, Sydney,

Both the Intel(r) Optane(TM) SSD 905P as the Intel(r) Optane(TM) SSD P4800X do support Opal 2.0.
You can see this with the identify tool in spdk. I ran the following on the P4800X in my system:
~/daos/_build.external/spdk/examples/nvme/identify# ./identify
...
Admin Command Set Attributes
============================
Security Send/Receive:                 Supported

I ran the following on the Intel(r) SSD DC P4610 in my system
...
~/daos/_build.external/spdk/examples/nvme/identify# ./identify
...
Admin Command Set Attributes
============================
Security Send/Receive:                 Not Supported
...
FYI the P4510 and P4610 exists in two version, one with Opal support and the other without Opal. You can see this in the model string as the one with Opal has an "O" in it, but also by querying the Identify.

I'll add the P4800X to my DAOS v1.0 system and see what happens.

Regards,

Gert


 

Colin Ngam
 

Greetings Gert,

 

The Model Number is: ModelNumber : INTEL SSDPE21D480GA

 

Is that the Model that supports OPAL? Is that ‘0’ above?

 

Thanks.

 

Colin

 

PS-Is identify same as spdk_nvme_identify?

 

From: <daos@daos.groups.io> on behalf of "Farrell, Patrick Arthur" <patrick.farrell@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Friday, June 19, 2020 at 10:49 AM
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Gert,

 

Thanks - I'll be curious what results you see.  FYI, we are using a very recent version of master, and didn't see this problem a few months back, so if you're using a different branch or older version, you may not see what we're seeing.

 

Regards,

-Patrick


From: daos@daos.groups.io <daos@daos.groups.io> on behalf of gert.pauwels@... <gert.pauwels@...>
Sent: Friday, June 19, 2020 10:34 AM
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Hi Patrick, Sydney,

Both the Intel(r) Optane(TM) SSD 905P as the Intel(r) Optane(TM) SSD P4800X do support Opal 2.0.
You can see this with the identify tool in spdk. I ran the following on the P4800X in my system:
~/daos/_build.external/spdk/examples/nvme/identify# ./identify
...

Admin Command Set Attributes

============================

Security Send/Receive:                 Supported


I ran the following on the Intel(r) SSD DC P4610 in my system

...
~/daos/_build.external/spdk/examples/nvme/identify# ./identify
...

Admin Command Set Attributes

============================

Security Send/Receive:                 Not Supported
...
FYI the P4510 and P4610 exists in two version, one with Opal support and the other without Opal. You can see this in the model string as the one with Opal has an "O" in it, but also by querying the Identify.

I'll add the P4800X to my DAOS v1.0 system and see what happens.

Regards,

Gert


 

Gert Pauwels (intel)
 

Hi Colin, Patrick,

 

The ModelNumber : INTEL SSDPE21D480GA is a 480 GB Intel(R) Optane(TM) SSD 905P Series and supports Opal 2.0.

The last letter in the ModelNumber is a reference to the memory media, the 4 characters before the last letter refer to the capacity.

 

The O for Opal in the ModelNumber is only applicable to the Intel 3D NAND SSDs as there are Models with and without Opal 2.0 support.

 

I compiled DAOS v1.0.0 on my system and did not see the problem. I used a P4800X added it to my daos_server.yml when running:

root@intel-S2600WFD:~/daos# dmg -i storage scan –verbose

It returns the following:

---------

localhost

---------

SCM Namespace Socket ID Capacity

------------- --------- --------

pmem0         0         531 GB

pmem1         1         531 GB

 

NVMe PCI     Model                FW Revision Socket ID Capacity

--------     -----                ----------- --------- --------

0000:d8:00.0 INTEL SSDPE2KE016T8  VDV10170    1         1.6 TB

0000:d9:00.0 INTEL SSDPE21K375GA  E2010485    1         375 GB

 

Next I will try the current DAOS master and also try an Opal 2.0 'enabled and configured' P4800X in DAOS.

 

Regards,

 

Gert,

Nabarro, Tom
 

Hello Gert

 

Out of curiosity , what is your operating system and kernel version?

 

Regards,

Tom Nabarro – DCG/ESAD

M: +44 (0)7786 260986

Skype: tom.nabarro

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of gert.pauwels@...
Sent: Monday, June 22, 2020 9:42 AM
To: daos@daos.groups.io
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Hi Colin, Patrick,

 

The ModelNumber : INTEL SSDPE21D480GA is a 480 GB Intel(R) Optane(TM) SSD 905P Series and supports Opal 2.0.

The last letter in the ModelNumber is a reference to the memory media, the 4 characters before the last letter refer to the capacity.

 

The O for Opal in the ModelNumber is only applicable to the Intel 3D NAND SSDs as there are Models with and without Opal 2.0 support.

 

I compiled DAOS v1.0.0 on my system and did not see the problem. I used a P4800X added it to my daos_server.yml when running:

root@intel-S2600WFD:~/daos# dmg -i storage scan –verbose

It returns the following:

---------

localhost

---------

SCM Namespace Socket ID Capacity

------------- --------- --------

pmem0         0         531 GB

pmem1         1         531 GB

 

NVMe PCI     Model                FW Revision Socket ID Capacity

--------     -----                ----------- --------- --------

0000:d8:00.0 INTEL SSDPE2KE016T8  VDV10170    1         1.6 TB

0000:d9:00.0 INTEL SSDPE21K375GA  E2010485    1         375 GB

 

Next I will try the current DAOS master and also try an Opal 2.0 'enabled and configured' P4800X in DAOS.

 

Regards,

 

Gert,

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Colin Ngam
 

Hi Gert,

 

You mentioned below Oparl 2.0 “enabled and configured”. What cmd do you use to see and configure these configuration on the SSD?

 

Thanks.

 

Colin

 

From: <daos@daos.groups.io> on behalf of "Nabarro, Tom" <tom.nabarro@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Wednesday, June 24, 2020 at 6:03 AM
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Hello Gert

 

Out of curiosity , what is your operating system and kernel version?

 

Regards,

Tom Nabarro – DCG/ESAD

M: +44 (0)7786 260986

Skype: tom.nabarro

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of gert.pauwels@...
Sent: Monday, June 22, 2020 9:42 AM
To: daos@daos.groups.io
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Hi Colin, Patrick,

 

The ModelNumber : INTEL SSDPE21D480GA is a 480 GB Intel(R) Optane(TM) SSD 905P Series and supports Opal 2.0.

The last letter in the ModelNumber is a reference to the memory media, the 4 characters before the last letter refer to the capacity.

 

The O for Opal in the ModelNumber is only applicable to the Intel 3D NAND SSDs as there are Models with and without Opal 2.0 support.

 

I compiled DAOS v1.0.0 on my system and did not see the problem. I used a P4800X added it to my daos_server.yml when running:

root@intel-S2600WFD:~/daos# dmg -i storage scan –verbose

It returns the following:

---------

localhost

---------

SCM Namespace Socket ID Capacity

------------- --------- --------

pmem0         0         531 GB

pmem1         1         531 GB

 

NVMe PCI     Model                FW Revision Socket ID Capacity

--------     -----                ----------- --------- --------

0000:d8:00.0 INTEL SSDPE2KE016T8  VDV10170    1         1.6 TB

0000:d9:00.0 INTEL SSDPE21K375GA  E2010485    1         375 GB

 

Next I will try the current DAOS master and also try an Opal 2.0 'enabled and configured' P4800X in DAOS.

 

Regards,

 

Gert,

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Gert Pauwels (intel)
 

Hi Tom, Colin,

I'm running Ubuntu 20.04 LTS  (kernel: 5.4.0-37-generic) and compiled DAOS v1.0.0.
I also compiled the latest master as of yesterday, but it did not make a difference.

Any application that can manage Opal 2.0 can be used to check the status of the drive. I used the sedutil-cli, can be found at https://github.com/Drive-Trust-Alliance/sedutil the executable can be found at https://github.com/Drive-Trust-Alliance/sedutil/wiki/Executable-Distributions 
You can run
# sedutil-cli --scan
and
# sedutil-cli --query <device>
it will return useful information about the Opal status of the device.
sedutil-cli will send NVMe security commands and the tool will only work if your SSD is bound the the NVMe driver.

In case the SSD is not bound to the NVMe driver you can manage the drive with the nvme_manage spdk utility you can found in the /daos/_build.external/dev/spdk
# ./examples/nvme/nvme_manage/nvme_manage 
Select 8 to get into the OPAL NVMe Management Options, enter the PCIe address of your SSD for the list you get prompted, select 1 to scan the device. These steps gives you the same results as running the # sedutil-cli --query <device> in case the SSD is bound to the NVMe driver.

Regards,

Gert,


 

Farrell, Patrick Arthur
 

Gert,

We're seeing this when we run identify on our 905p SSDs:
"
NVMe Controller at 0000:1a:00.0 [8086:2700]
[...]
Serial Number:                         PHM29226005S480BGN
Model Number:                          INTEL SSDPE21D480GA
[...]
Admin Command Set Attributes
============================
Security Send/Receive:                 Supported
"

However, when I run nvme_manage, I get an error stating that Opal is not supported (the same error DAOS is spitting out, incidentally):

"Please Input PCI Address(domain:bus:dev.func):
0000:1a:00.00
Opal General Usage:

        [1: scan device]
        [2: init - take ownership and activate locking]
        [3: revert tper]
        [4: setup locking range]
        [5: list locking ranges]
        [6: enable user]
        [7: set new password]
        [8: add user to locking range]
        [9: lock/unlock range]
        [10: erase locking range]
        [0: quit]
1
[2020-06-24 14:37:34.452086] nvme_opal.c: 828:opal_discovery0_end: *ERROR*: Opal Not Supported.
"

Any thoughts?

Regards,
-Patrick


From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Gert Pauwels (intel) <gert.pauwels@...>
Sent: Wednesday, June 24, 2020 11:54 AM
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem
 
Hi Tom, Colin,

I'm running Ubuntu 20.04 LTS  (kernel: 5.4.0-37-generic) and compiled DAOS v1.0.0.
I also compiled the latest master as of yesterday, but it did not make a difference.

Any application that can manage Opal 2.0 can be used to check the status of the drive. I used the sedutil-cli, can be found at https://github.com/Drive-Trust-Alliance/sedutil the executable can be found at https://github.com/Drive-Trust-Alliance/sedutil/wiki/Executable-Distributions 
You can run
# sedutil-cli --scan
and
# sedutil-cli --query <device>
it will return useful information about the Opal status of the device.
sedutil-cli will send NVMe security commands and the tool will only work if your SSD is bound the the NVMe driver.

In case the SSD is not bound to the NVMe driver you can manage the drive with the nvme_manage spdk utility you can found in the /daos/_build.external/dev/spdk
# ./examples/nvme/nvme_manage/nvme_manage 
Select 8 to get into the OPAL NVMe Management Options, enter the PCIe address of your SSD for the list you get prompted, select 1 to scan the device. These steps gives you the same results as running the # sedutil-cli --query <device> in case the SSD is bound to the NVMe driver.

Regards,

Gert,


 

Nabarro, Tom
 

We probably need to get further information from NSG folks on product specifics regarding Opal support in 905p SSDs. This seems to be a prerequisite for device management through SPDK.

 

Tom

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Farrell, Patrick Arthur
Sent: Wednesday, June 24, 2020 8:42 PM
To: daos@daos.groups.io
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Gert,

 

We're seeing this when we run identify on our 905p SSDs:

"

NVMe Controller at 0000:1a:00.0 [8086:2700]

[...]

Serial Number:                         PHM29226005S480BGN
Model Number:                          INTEL SSDPE21D480GA

[...]

Admin Command Set Attributes

============================

Security Send/Receive:                 Supported

"

 

However, when I run nvme_manage, I get an error stating that Opal is not supported (the same error DAOS is spitting out, incidentally):

 

"Please Input PCI Address(domain:bus:dev.func):

0000:1a:00.00

Opal General Usage:

 

        [1: scan device]

        [2: init - take ownership and activate locking]

        [3: revert tper]

        [4: setup locking range]

        [5: list locking ranges]

        [6: enable user]

        [7: set new password]

        [8: add user to locking range]

        [9: lock/unlock range]

        [10: erase locking range]

        [0: quit]

1

[2020-06-24 14:37:34.452086] nvme_opal.c: 828:opal_discovery0_end: *ERROR*: Opal Not Supported.

"

 

Any thoughts?

 

Regards,

-Patrick


From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Gert Pauwels (intel) <gert.pauwels@...>
Sent: Wednesday, June 24, 2020 11:54 AM
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Hi Tom, Colin,


I'm running Ubuntu 20.04 LTS  (kernel: 5.4.0-37-generic) and compiled DAOS v1.0.0.

I also compiled the latest master as of yesterday, but it did not make a difference.

Any application that can manage Opal 2.0 can be used to check the status of the drive. I used the sedutil-cli, can be found at https://github.com/Drive-Trust-Alliance/sedutil the executable can be found at https://github.com/Drive-Trust-Alliance/sedutil/wiki/Executable-Distributions 
You can run
# sedutil-cli --scan
and
# sedutil-cli --query <device>
it will return useful information about the Opal status of the device.
sedutil-cli will send NVMe security commands and the tool will only work if your SSD is bound the the NVMe driver.

In case the SSD is not bound to the NVMe driver you can manage the drive with the nvme_manage spdk utility you can found in the /daos/_build.external/dev/spdk
# ./examples/nvme/nvme_manage/nvme_manage 
Select 8 to get into the OPAL NVMe Management Options, enter the PCIe address of your SSD for the list you get prompted, select 1 to scan the device. These steps gives you the same results as running the # sedutil-cli --query <device> in case the SSD is bound to the NVMe driver.

Regards,

Gert,

 

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Farrell, Patrick Arthur
 

Recall that we have used this SSD with DAOS in the past.  Whatever is preventing this now is a change, perhaps in SPDK but possibly in how DAOS is invoking it.

Regards,
-Patrick


From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Nabarro, Tom <tom.nabarro@...>
Sent: Thursday, June 25, 2020 3:43 AM
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem
 

We probably need to get further information from NSG folks on product specifics regarding Opal support in 905p SSDs. This seems to be a prerequisite for device management through SPDK.

 

Tom

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Farrell, Patrick Arthur
Sent: Wednesday, June 24, 2020 8:42 PM
To: daos@daos.groups.io
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Gert,

 

We're seeing this when we run identify on our 905p SSDs:

"

NVMe Controller at 0000:1a:00.0 [8086:2700]

[...]

Serial Number:                         PHM29226005S480BGN
Model Number:                          INTEL SSDPE21D480GA

[...]

Admin Command Set Attributes

============================

Security Send/Receive:                 Supported

"

 

However, when I run nvme_manage, I get an error stating that Opal is not supported (the same error DAOS is spitting out, incidentally):

 

"Please Input PCI Address(domain:bus:dev.func):

0000:1a:00.00

Opal General Usage:

 

        [1: scan device]

        [2: init - take ownership and activate locking]

        [3: revert tper]

        [4: setup locking range]

        [5: list locking ranges]

        [6: enable user]

        [7: set new password]

        [8: add user to locking range]

        [9: lock/unlock range]

        [10: erase locking range]

        [0: quit]

1

[2020-06-24 14:37:34.452086] nvme_opal.c: 828:opal_discovery0_end: *ERROR*: Opal Not Supported.

"

 

Any thoughts?

 

Regards,

-Patrick


From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Gert Pauwels (intel) <gert.pauwels@...>
Sent: Wednesday, June 24, 2020 11:54 AM
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Hi Tom, Colin,


I'm running Ubuntu 20.04 LTS  (kernel: 5.4.0-37-generic) and compiled DAOS v1.0.0.

I also compiled the latest master as of yesterday, but it did not make a difference.

Any application that can manage Opal 2.0 can be used to check the status of the drive. I used the sedutil-cli, can be found at https://github.com/Drive-Trust-Alliance/sedutil the executable can be found at https://github.com/Drive-Trust-Alliance/sedutil/wiki/Executable-Distributions 
You can run
# sedutil-cli --scan
and
# sedutil-cli --query <device>
it will return useful information about the Opal status of the device.
sedutil-cli will send NVMe security commands and the tool will only work if your SSD is bound the the NVMe driver.

In case the SSD is not bound to the NVMe driver you can manage the drive with the nvme_manage spdk utility you can found in the /daos/_build.external/dev/spdk
# ./examples/nvme/nvme_manage/nvme_manage 
Select 8 to get into the OPAL NVMe Management Options, enter the PCIe address of your SSD for the list you get prompted, select 1 to scan the device. These steps gives you the same results as running the # sedutil-cli --query <device> in case the SSD is bound to the NVMe driver.

Regards,

Gert,

 

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Gert Pauwels (intel)
 

Patrick, Colin,

 

I verified with the NSG Business Unit that builds the 905P SSD.

The product specification lists both Pyrite as Opal support, what is less clear in the documentation is that Opal is not enable. This is causing the behavior you are seeing.

The Identify command correctly returns the Security Send/Receive is supported as this is needed for both Opal and Pyrite.

It is correct that SPDK returns “No Opal support” for the 905P SSD. I verified on my system you do not have this with the Intel Optane SSD P4800X as this support Opal 2.0

 

What I cannot explain right now is why the drive was working with DAOS in the past.

 

Regards,

 

Gert,

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Farrell, Patrick Arthur
Sent: Thursday, June 25, 2020 5:07 PM
To: daos@daos.groups.io
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Recall that we have used this SSD with DAOS in the past.  Whatever is preventing this now is a change, perhaps in SPDK but possibly in how DAOS is invoking it.

 

Regards,

-Patrick


From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Nabarro, Tom <tom.nabarro@...>
Sent: Thursday, June 25, 2020 3:43 AM
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

We probably need to get further information from NSG folks on product specifics regarding Opal support in 905p SSDs. This seems to be a prerequisite for device management through SPDK.

 

Tom

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Farrell, Patrick Arthur
Sent: Wednesday, June 24, 2020 8:42 PM
To: daos@daos.groups.io
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Gert,

 

We're seeing this when we run identify on our 905p SSDs:

"

NVMe Controller at 0000:1a:00.0 [8086:2700]

[...]

Serial Number:                         PHM29226005S480BGN
Model Number:                          INTEL SSDPE21D480GA

[...]

Admin Command Set Attributes

============================

Security Send/Receive:                 Supported

"

 

However, when I run nvme_manage, I get an error stating that Opal is not supported (the same error DAOS is spitting out, incidentally):

 

"Please Input PCI Address(domain:bus:dev.func):

0000:1a:00.00

Opal General Usage:

 

        [1: scan device]

        [2: init - take ownership and activate locking]

        [3: revert tper]

        [4: setup locking range]

        [5: list locking ranges]

        [6: enable user]

        [7: set new password]

        [8: add user to locking range]

        [9: lock/unlock range]

        [10: erase locking range]

        [0: quit]

1

[2020-06-24 14:37:34.452086] nvme_opal.c: 828:opal_discovery0_end: *ERROR*: Opal Not Supported.

"

 

Any thoughts?

 

Regards,

-Patrick


From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Gert Pauwels (intel) <gert.pauwels@...>
Sent: Wednesday, June 24, 2020 11:54 AM
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Hi Tom, Colin,


I'm running Ubuntu 20.04 LTS  (kernel: 5.4.0-37-generic) and compiled DAOS v1.0.0.

I also compiled the latest master as of yesterday, but it did not make a difference.

Any application that can manage Opal 2.0 can be used to check the status of the drive. I used the sedutil-cli, can be found at https://github.com/Drive-Trust-Alliance/sedutil the executable can be found at https://github.com/Drive-Trust-Alliance/sedutil/wiki/Executable-Distributions 
You can run
# sedutil-cli --scan
and
# sedutil-cli --query <device>
it will return useful information about the Opal status of the device.
sedutil-cli will send NVMe security commands and the tool will only work if your SSD is bound the the NVMe driver.

In case the SSD is not bound to the NVMe driver you can manage the drive with the nvme_manage spdk utility you can found in the /daos/_build.external/dev/spdk
# ./examples/nvme/nvme_manage/nvme_manage 
Select 8 to get into the OPAL NVMe Management Options, enter the PCIe address of your SSD for the list you get prompted, select 1 to scan the device. These steps gives you the same results as running the # sedutil-cli --query <device> in case the SSD is bound to the NVMe driver.

Regards,

Gert,

 

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Intel Corporation NV/SA
Kings Square, Veldkant 31
2550 Kontich
RPM (Bruxelles) 0415.497.718.
Citibank, Brussels, account 570/1031255/09

This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

Farrell, Patrick Arthur
 

Gert,

Thanks for the further information.  Something I don't quite follow:

"The product specification lists both Pyrite as Opal support, what is less clear in the documentation is that Opal is not enable."

Then, can Opal be enabled?  Or is it permanently disabled on this model?

A general question for you:
Is SPDK simply totally unable to manage some current Intel SSDs if they don't have the full Opal support?

Separately, I'm going to dig a little, and I also plan to try older versions of DAOS/SPDK to see what I can figure out.

Thanks,
-Patrick


From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Gert Pauwels (intel) <gert.pauwels@...>
Sent: Thursday, June 25, 2020 12:05 PM
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem
 

Patrick, Colin,

 

I verified with the NSG Business Unit that builds the 905P SSD.

The product specification lists both Pyrite as Opal support, what is less clear in the documentation is that Opal is not enable. This is causing the behavior you are seeing.

The Identify command correctly returns the Security Send/Receive is supported as this is needed for both Opal and Pyrite.

It is correct that SPDK returns “No Opal support” for the 905P SSD. I verified on my system you do not have this with the Intel Optane SSD P4800X as this support Opal 2.0

 

What I cannot explain right now is why the drive was working with DAOS in the past.

 

Regards,

 

Gert,

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Farrell, Patrick Arthur
Sent: Thursday, June 25, 2020 5:07 PM
To: daos@daos.groups.io
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Recall that we have used this SSD with DAOS in the past.  Whatever is preventing this now is a change, perhaps in SPDK but possibly in how DAOS is invoking it.

 

Regards,

-Patrick


From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Nabarro, Tom <tom.nabarro@...>
Sent: Thursday, June 25, 2020 3:43 AM
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

We probably need to get further information from NSG folks on product specifics regarding Opal support in 905p SSDs. This seems to be a prerequisite for device management through SPDK.

 

Tom

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Farrell, Patrick Arthur
Sent: Wednesday, June 24, 2020 8:42 PM
To: daos@daos.groups.io
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Gert,

 

We're seeing this when we run identify on our 905p SSDs:

"

NVMe Controller at 0000:1a:00.0 [8086:2700]

[...]

Serial Number:                         PHM29226005S480BGN
Model Number:                          INTEL SSDPE21D480GA

[...]

Admin Command Set Attributes

============================

Security Send/Receive:                 Supported

"

 

However, when I run nvme_manage, I get an error stating that Opal is not supported (the same error DAOS is spitting out, incidentally):

 

"Please Input PCI Address(domain:bus:dev.func):

0000:1a:00.00

Opal General Usage:

 

        [1: scan device]

        [2: init - take ownership and activate locking]

        [3: revert tper]

        [4: setup locking range]

        [5: list locking ranges]

        [6: enable user]

        [7: set new password]

        [8: add user to locking range]

        [9: lock/unlock range]

        [10: erase locking range]

        [0: quit]

1

[2020-06-24 14:37:34.452086] nvme_opal.c: 828:opal_discovery0_end: *ERROR*: Opal Not Supported.

"

 

Any thoughts?

 

Regards,

-Patrick


From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Gert Pauwels (intel) <gert.pauwels@...>
Sent: Wednesday, June 24, 2020 11:54 AM
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Hi Tom, Colin,


I'm running Ubuntu 20.04 LTS  (kernel: 5.4.0-37-generic) and compiled DAOS v1.0.0.

I also compiled the latest master as of yesterday, but it did not make a difference.

Any application that can manage Opal 2.0 can be used to check the status of the drive. I used the sedutil-cli, can be found at https://github.com/Drive-Trust-Alliance/sedutil the executable can be found at https://github.com/Drive-Trust-Alliance/sedutil/wiki/Executable-Distributions 
You can run
# sedutil-cli --scan
and
# sedutil-cli --query <device>
it will return useful information about the Opal status of the device.
sedutil-cli will send NVMe security commands and the tool will only work if your SSD is bound the the NVMe driver.

In case the SSD is not bound to the NVMe driver you can manage the drive with the nvme_manage spdk utility you can found in the /daos/_build.external/dev/spdk
# ./examples/nvme/nvme_manage/nvme_manage 
Select 8 to get into the OPAL NVMe Management Options, enter the PCIe address of your SSD for the list you get prompted, select 1 to scan the device. These steps gives you the same results as running the # sedutil-cli --query <device> in case the SSD is bound to the NVMe driver.

Regards,

Gert,

 

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Intel Corporation NV/SA
Kings Square, Veldkant 31
2550 Kontich
RPM (Bruxelles) 0415.497.718.
Citibank, Brussels, account 570/1031255/09

This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

Nabarro, Tom
 

We upgraded from SPDK v19.04 to v20.01 shortly after releasing DAOS 1.0, that would be my best guess as to the cause (obviously not if you a testing with 1.0 rather than master).

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Farrell, Patrick Arthur
Sent: Thursday, June 25, 2020 4:07 PM
To: daos@daos.groups.io
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Recall that we have used this SSD with DAOS in the past.  Whatever is preventing this now is a change, perhaps in SPDK but possibly in how DAOS is invoking it.

 

Regards,

-Patrick


From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Nabarro, Tom <tom.nabarro@...>
Sent: Thursday, June 25, 2020 3:43 AM
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

We probably need to get further information from NSG folks on product specifics regarding Opal support in 905p SSDs. This seems to be a prerequisite for device management through SPDK.

 

Tom

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Farrell, Patrick Arthur
Sent: Wednesday, June 24, 2020 8:42 PM
To: daos@daos.groups.io
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Gert,

 

We're seeing this when we run identify on our 905p SSDs:

"

NVMe Controller at 0000:1a:00.0 [8086:2700]

[...]

Serial Number:                         PHM29226005S480BGN
Model Number:                          INTEL SSDPE21D480GA

[...]

Admin Command Set Attributes

============================

Security Send/Receive:                 Supported

"

 

However, when I run nvme_manage, I get an error stating that Opal is not supported (the same error DAOS is spitting out, incidentally):

 

"Please Input PCI Address(domain:bus:dev.func):

0000:1a:00.00

Opal General Usage:

 

        [1: scan device]

        [2: init - take ownership and activate locking]

        [3: revert tper]

        [4: setup locking range]

        [5: list locking ranges]

        [6: enable user]

        [7: set new password]

        [8: add user to locking range]

        [9: lock/unlock range]

        [10: erase locking range]

        [0: quit]

1

[2020-06-24 14:37:34.452086] nvme_opal.c: 828:opal_discovery0_end: *ERROR*: Opal Not Supported.

"

 

Any thoughts?

 

Regards,

-Patrick


From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Gert Pauwels (intel) <gert.pauwels@...>
Sent: Wednesday, June 24, 2020 11:54 AM
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Hi Tom, Colin,


I'm running Ubuntu 20.04 LTS  (kernel: 5.4.0-37-generic) and compiled DAOS v1.0.0.

I also compiled the latest master as of yesterday, but it did not make a difference.

Any application that can manage Opal 2.0 can be used to check the status of the drive. I used the sedutil-cli, can be found at https://github.com/Drive-Trust-Alliance/sedutil the executable can be found at https://github.com/Drive-Trust-Alliance/sedutil/wiki/Executable-Distributions 
You can run
# sedutil-cli --scan
and
# sedutil-cli --query <device>
it will return useful information about the Opal status of the device.
sedutil-cli will send NVMe security commands and the tool will only work if your SSD is bound the the NVMe driver.

In case the SSD is not bound to the NVMe driver you can manage the drive with the nvme_manage spdk utility you can found in the /daos/_build.external/dev/spdk
# ./examples/nvme/nvme_manage/nvme_manage 
Select 8 to get into the OPAL NVMe Management Options, enter the PCIe address of your SSD for the list you get prompted, select 1 to scan the device. These steps gives you the same results as running the # sedutil-cli --query <device> in case the SSD is bound to the NVMe driver.

Regards,

Gert,

 

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Lombardi, Johann
 

Hi Patrick,

 

I talked to the SPDK folks and they suggested to try removing line 1464-1470 in module/bdev/nvme/bdev_nvme.c:

 

-              if (spdk_nvme_ctrlr_get_flags(nvme_bdev_ctrlr->ctrlr) &

-                  SPDK_NVME_CTRLR_SECURITY_SEND_RECV_SUPPORTED) {

-                              nvme_bdev_ctrlr->opal_dev = spdk_opal_dev_construct(nvme_bdev_ctrlr->ctrlr);

-                              if (nvme_bdev_ctrlr->opal_dev == NULL) {

-                                              SPDK_ERRLOG("Failed to initialize Opal\n");

-                              }

-              }

 

Could you please give this a spin?

 

Cheers,

Johann

 

From: <daos@daos.groups.io> on behalf of "Farrell, Patrick Arthur" <patrick.farrell@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Thursday 25 June 2020 at 19:11
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Gert,

 

Thanks for the further information.  Something I don't quite follow:

"The product specification lists both Pyrite as Opal support, what is less clear in the documentation is that Opal is not enable."

 

Then, can Opal be enabled?  Or is it permanently disabled on this model?

 

A general question for you:
Is SPDK simply totally unable to manage some current Intel SSDs if they don't have the full Opal support?

 

Separately, I'm going to dig a little, and I also plan to try older versions of DAOS/SPDK to see what I can figure out.

 

Thanks,

-Patrick


From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Gert Pauwels (intel) <gert.pauwels@...>
Sent: Thursday, June 25, 2020 12:05 PM
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Patrick, Colin,

 

I verified with the NSG Business Unit that builds the 905P SSD.

The product specification lists both Pyrite as Opal support, what is less clear in the documentation is that Opal is not enable. This is causing the behavior you are seeing.

The Identify command correctly returns the Security Send/Receive is supported as this is needed for both Opal and Pyrite.

It is correct that SPDK returns “No Opal support” for the 905P SSD. I verified on my system you do not have this with the Intel Optane SSD P4800X as this support Opal 2.0

 

What I cannot explain right now is why the drive was working with DAOS in the past.

 

Regards,

 

Gert,

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Farrell, Patrick Arthur
Sent: Thursday, June 25, 2020 5:07 PM
To: daos@daos.groups.io
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Recall that we have used this SSD with DAOS in the past.  Whatever is preventing this now is a change, perhaps in SPDK but possibly in how DAOS is invoking it.

 

Regards,

-Patrick


From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Nabarro, Tom <tom.nabarro@...>
Sent: Thursday, June 25, 2020 3:43 AM
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

We probably need to get further information from NSG folks on product specifics regarding Opal support in 905p SSDs. This seems to be a prerequisite for device management through SPDK.

 

Tom

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Farrell, Patrick Arthur
Sent: Wednesday, June 24, 2020 8:42 PM
To: daos@daos.groups.io
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Gert,

 

We're seeing this when we run identify on our 905p SSDs:

"

NVMe Controller at 0000:1a:00.0 [8086:2700]

[...]

Serial Number:                         PHM29226005S480BGN
Model Number:                          INTEL SSDPE21D480GA

[...]

Admin Command Set Attributes

============================

Security Send/Receive:                 Supported

"

 

However, when I run nvme_manage, I get an error stating that Opal is not supported (the same error DAOS is spitting out, incidentally):

 

"Please Input PCI Address(domain:bus:dev.func):

0000:1a:00.00

Opal General Usage:

 

        [1: scan device]

        [2: init - take ownership and activate locking]

        [3: revert tper]

        [4: setup locking range]

        [5: list locking ranges]

        [6: enable user]

        [7: set new password]

        [8: add user to locking range]

        [9: lock/unlock range]

        [10: erase locking range]

        [0: quit]

1

[2020-06-24 14:37:34.452086] nvme_opal.c: 828:opal_discovery0_end: *ERROR*: Opal Not Supported.

"

 

Any thoughts?

 

Regards,

-Patrick


From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Gert Pauwels (intel) <gert.pauwels@...>
Sent: Wednesday, June 24, 2020 11:54 AM
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Hi Tom, Colin,


I'm running Ubuntu 20.04 LTS  (kernel: 5.4.0-37-generic) and compiled DAOS v1.0.0.

I also compiled the latest master as of yesterday, but it did not make a difference.

Any application that can manage Opal 2.0 can be used to check the status of the drive. I used the sedutil-cli, can be found at https://github.com/Drive-Trust-Alliance/sedutil the executable can be found at https://github.com/Drive-Trust-Alliance/sedutil/wiki/Executable-Distributions 
You can run
# sedutil-cli --scan
and
# sedutil-cli --query <device>
it will return useful information about the Opal status of the device.
sedutil-cli will send NVMe security commands and the tool will only work if your SSD is bound the the NVMe driver.

In case the SSD is not bound to the NVMe driver you can manage the drive with the nvme_manage spdk utility you can found in the /daos/_build.external/dev/spdk
#
./examples/nvme/nvme_manage/nvme_manage 
Select 8 to get into the OPAL NVMe Management Options, enter the PCIe address of your SSD for the list you get prompted, select 1 to scan the device. These steps gives you the same results as running the 
# sedutil-cli --query <device> in case the SSD is bound to the NVMe driver.

Regards,

Gert,

 

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Intel Corporation NV/SA
Kings Square, Veldkant 31
2550 Kontich
RPM (Bruxelles) 0415.497.718.
Citibank, Brussels, account 570/1031255/09

This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Farrell, Patrick Arthur
 

Heh, yes!  These are among the lines I was planning to try to remove, so will do.

It is interesting to note of course that SEND_RECV is reported by the tool Gert had me using, but that's not a total shock.

-Patrick


From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Lombardi, Johann <johann.lombardi@...>
Sent: Tuesday, June 30, 2020 5:47 AM
To: daos@daos.groups.io <daos@daos.groups.io>
Cc: Liu, Changpeng <changpeng.liu@...>
Subject: Re: [daos] Intel Optane SSD mounting problem
 

Hi Patrick,

 

I talked to the SPDK folks and they suggested to try removing line 1464-1470 in module/bdev/nvme/bdev_nvme.c:

 

-              if (spdk_nvme_ctrlr_get_flags(nvme_bdev_ctrlr->ctrlr) &

-                  SPDK_NVME_CTRLR_SECURITY_SEND_RECV_SUPPORTED) {

-                              nvme_bdev_ctrlr->opal_dev = spdk_opal_dev_construct(nvme_bdev_ctrlr->ctrlr);

-                              if (nvme_bdev_ctrlr->opal_dev == NULL) {

-                                              SPDK_ERRLOG("Failed to initialize Opal\n");

-                              }

-              }

 

Could you please give this a spin?

 

Cheers,

Johann

 

From: <daos@daos.groups.io> on behalf of "Farrell, Patrick Arthur" <patrick.farrell@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Thursday 25 June 2020 at 19:11
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Gert,

 

Thanks for the further information.  Something I don't quite follow:

"The product specification lists both Pyrite as Opal support, what is less clear in the documentation is that Opal is not enable."

 

Then, can Opal be enabled?  Or is it permanently disabled on this model?

 

A general question for you:
Is SPDK simply totally unable to manage some current Intel SSDs if they don't have the full Opal support?

 

Separately, I'm going to dig a little, and I also plan to try older versions of DAOS/SPDK to see what I can figure out.

 

Thanks,

-Patrick


From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Gert Pauwels (intel) <gert.pauwels@...>
Sent: Thursday, June 25, 2020 12:05 PM
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Patrick, Colin,

 

I verified with the NSG Business Unit that builds the 905P SSD.

The product specification lists both Pyrite as Opal support, what is less clear in the documentation is that Opal is not enable. This is causing the behavior you are seeing.

The Identify command correctly returns the Security Send/Receive is supported as this is needed for both Opal and Pyrite.

It is correct that SPDK returns “No Opal support” for the 905P SSD. I verified on my system you do not have this with the Intel Optane SSD P4800X as this support Opal 2.0

 

What I cannot explain right now is why the drive was working with DAOS in the past.

 

Regards,

 

Gert,

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Farrell, Patrick Arthur
Sent: Thursday, June 25, 2020 5:07 PM
To: daos@daos.groups.io
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Recall that we have used this SSD with DAOS in the past.  Whatever is preventing this now is a change, perhaps in SPDK but possibly in how DAOS is invoking it.

 

Regards,

-Patrick


From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Nabarro, Tom <tom.nabarro@...>
Sent: Thursday, June 25, 2020 3:43 AM
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

We probably need to get further information from NSG folks on product specifics regarding Opal support in 905p SSDs. This seems to be a prerequisite for device management through SPDK.

 

Tom

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Farrell, Patrick Arthur
Sent: Wednesday, June 24, 2020 8:42 PM
To: daos@daos.groups.io
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Gert,

 

We're seeing this when we run identify on our 905p SSDs:

"

NVMe Controller at 0000:1a:00.0 [8086:2700]

[...]

Serial Number:                         PHM29226005S480BGN
Model Number:                          INTEL SSDPE21D480GA

[...]

Admin Command Set Attributes

============================

Security Send/Receive:                 Supported

"

 

However, when I run nvme_manage, I get an error stating that Opal is not supported (the same error DAOS is spitting out, incidentally):

 

"Please Input PCI Address(domain:bus:dev.func):

0000:1a:00.00

Opal General Usage:

 

        [1: scan device]

        [2: init - take ownership and activate locking]

        [3: revert tper]

        [4: setup locking range]

        [5: list locking ranges]

        [6: enable user]

        [7: set new password]

        [8: add user to locking range]

        [9: lock/unlock range]

        [10: erase locking range]

        [0: quit]

1

[2020-06-24 14:37:34.452086] nvme_opal.c: 828:opal_discovery0_end: *ERROR*: Opal Not Supported.

"

 

Any thoughts?

 

Regards,

-Patrick


From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Gert Pauwels (intel) <gert.pauwels@...>
Sent: Wednesday, June 24, 2020 11:54 AM
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Hi Tom, Colin,


I'm running Ubuntu 20.04 LTS  (kernel: 5.4.0-37-generic) and compiled DAOS v1.0.0.

I also compiled the latest master as of yesterday, but it did not make a difference.

Any application that can manage Opal 2.0 can be used to check the status of the drive. I used the sedutil-cli, can be found at https://github.com/Drive-Trust-Alliance/sedutil the executable can be found at https://github.com/Drive-Trust-Alliance/sedutil/wiki/Executable-Distributions 
You can run
# sedutil-cli --scan
and
# sedutil-cli --query <device>
it will return useful information about the Opal status of the device.
sedutil-cli will send NVMe security commands and the tool will only work if your SSD is bound the the NVMe driver.

In case the SSD is not bound to the NVMe driver you can manage the drive with the nvme_manage spdk utility you can found in the /daos/_build.external/dev/spdk
#
./examples/nvme/nvme_manage/nvme_manage 
Select 8 to get into the OPAL NVMe Management Options, enter the PCIe address of your SSD for the list you get prompted, select 1 to scan the device. These steps gives you the same results as running the 
# sedutil-cli --query <device> in case the SSD is bound to the NVMe driver.

Regards,

Gert,

 

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Intel Corporation NV/SA
Kings Square, Veldkant 31
2550 Kontich
RPM (Bruxelles) 0415.497.718.
Citibank, Brussels, account 570/1031255/09

This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Farrell, Patrick Arthur
 

As suspected, the SSDs are now coming online normally.

Let me know if there's any further troubleshooting the SPDK team would like.

-Patrick


From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Lombardi, Johann <johann.lombardi@...>
Sent: Tuesday, June 30, 2020 5:47 AM
To: daos@daos.groups.io <daos@daos.groups.io>
Cc: Liu, Changpeng <changpeng.liu@...>
Subject: Re: [daos] Intel Optane SSD mounting problem
 

Hi Patrick,

 

I talked to the SPDK folks and they suggested to try removing line 1464-1470 in module/bdev/nvme/bdev_nvme.c:

 

-              if (spdk_nvme_ctrlr_get_flags(nvme_bdev_ctrlr->ctrlr) &

-                  SPDK_NVME_CTRLR_SECURITY_SEND_RECV_SUPPORTED) {

-                              nvme_bdev_ctrlr->opal_dev = spdk_opal_dev_construct(nvme_bdev_ctrlr->ctrlr);

-                              if (nvme_bdev_ctrlr->opal_dev == NULL) {

-                                              SPDK_ERRLOG("Failed to initialize Opal\n");

-                              }

-              }

 

Could you please give this a spin?

 

Cheers,

Johann

 

From: <daos@daos.groups.io> on behalf of "Farrell, Patrick Arthur" <patrick.farrell@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Thursday 25 June 2020 at 19:11
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Gert,

 

Thanks for the further information.  Something I don't quite follow:

"The product specification lists both Pyrite as Opal support, what is less clear in the documentation is that Opal is not enable."

 

Then, can Opal be enabled?  Or is it permanently disabled on this model?

 

A general question for you:
Is SPDK simply totally unable to manage some current Intel SSDs if they don't have the full Opal support?

 

Separately, I'm going to dig a little, and I also plan to try older versions of DAOS/SPDK to see what I can figure out.

 

Thanks,

-Patrick


From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Gert Pauwels (intel) <gert.pauwels@...>
Sent: Thursday, June 25, 2020 12:05 PM
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Patrick, Colin,

 

I verified with the NSG Business Unit that builds the 905P SSD.

The product specification lists both Pyrite as Opal support, what is less clear in the documentation is that Opal is not enable. This is causing the behavior you are seeing.

The Identify command correctly returns the Security Send/Receive is supported as this is needed for both Opal and Pyrite.

It is correct that SPDK returns “No Opal support” for the 905P SSD. I verified on my system you do not have this with the Intel Optane SSD P4800X as this support Opal 2.0

 

What I cannot explain right now is why the drive was working with DAOS in the past.

 

Regards,

 

Gert,

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Farrell, Patrick Arthur
Sent: Thursday, June 25, 2020 5:07 PM
To: daos@daos.groups.io
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Recall that we have used this SSD with DAOS in the past.  Whatever is preventing this now is a change, perhaps in SPDK but possibly in how DAOS is invoking it.

 

Regards,

-Patrick


From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Nabarro, Tom <tom.nabarro@...>
Sent: Thursday, June 25, 2020 3:43 AM
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

We probably need to get further information from NSG folks on product specifics regarding Opal support in 905p SSDs. This seems to be a prerequisite for device management through SPDK.

 

Tom

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Farrell, Patrick Arthur
Sent: Wednesday, June 24, 2020 8:42 PM
To: daos@daos.groups.io
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Gert,

 

We're seeing this when we run identify on our 905p SSDs:

"

NVMe Controller at 0000:1a:00.0 [8086:2700]

[...]

Serial Number:                         PHM29226005S480BGN
Model Number:                          INTEL SSDPE21D480GA

[...]

Admin Command Set Attributes

============================

Security Send/Receive:                 Supported

"

 

However, when I run nvme_manage, I get an error stating that Opal is not supported (the same error DAOS is spitting out, incidentally):

 

"Please Input PCI Address(domain:bus:dev.func):

0000:1a:00.00

Opal General Usage:

 

        [1: scan device]

        [2: init - take ownership and activate locking]

        [3: revert tper]

        [4: setup locking range]

        [5: list locking ranges]

        [6: enable user]

        [7: set new password]

        [8: add user to locking range]

        [9: lock/unlock range]

        [10: erase locking range]

        [0: quit]

1

[2020-06-24 14:37:34.452086] nvme_opal.c: 828:opal_discovery0_end: *ERROR*: Opal Not Supported.

"

 

Any thoughts?

 

Regards,

-Patrick


From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Gert Pauwels (intel) <gert.pauwels@...>
Sent: Wednesday, June 24, 2020 11:54 AM
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Hi Tom, Colin,


I'm running Ubuntu 20.04 LTS  (kernel: 5.4.0-37-generic) and compiled DAOS v1.0.0.

I also compiled the latest master as of yesterday, but it did not make a difference.

Any application that can manage Opal 2.0 can be used to check the status of the drive. I used the sedutil-cli, can be found at https://github.com/Drive-Trust-Alliance/sedutil the executable can be found at https://github.com/Drive-Trust-Alliance/sedutil/wiki/Executable-Distributions 
You can run
# sedutil-cli --scan
and
# sedutil-cli --query <device>
it will return useful information about the Opal status of the device.
sedutil-cli will send NVMe security commands and the tool will only work if your SSD is bound the the NVMe driver.

In case the SSD is not bound to the NVMe driver you can manage the drive with the nvme_manage spdk utility you can found in the /daos/_build.external/dev/spdk
#
./examples/nvme/nvme_manage/nvme_manage 
Select 8 to get into the OPAL NVMe Management Options, enter the PCIe address of your SSD for the list you get prompted, select 1 to scan the device. These steps gives you the same results as running the 
# sedutil-cli --query <device> in case the SSD is bound to the NVMe driver.

Regards,

Gert,

 

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Intel Corporation NV/SA
Kings Square, Veldkant 31
2550 Kontich
RPM (Bruxelles) 0415.497.718.
Citibank, Brussels, account 570/1031255/09

This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.