1. daos@daos.groups.io
  2. Messages
   Date   
Re: Client IO Request.

Colin Ngam
 

Thanks WangDi.

 

So each target on the Server is actually listening for RPC IO Requests.

 

Thanks.

 

Colin

 

From: <daos@daos.groups.io> on behalf of "Wang, Di" <di.wang@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Saturday, July 11, 2020 at 5:38 PM
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [daos] Client IO Request.

 

Hello,

 

The endpoint is one target of the server, i.e. the client needs to specify both rank and target index (within that server).

 

Thanks

WangDi

 

On 7/11/20, 3:30 PM, "daos@daos.groups.io on behalf of Colin Ngam" <daos@daos.groups.io on behalf of colin.ngam@...> wrote:

 

Greetings,

 

When a Client makes an IO Request, is the endpoint the IO Server or the Target?

 

Thanks.

 

Colin
Re: Client IO Request.

Wang, Di
 

Hello,

 

The endpoint is one target of the server, i.e. the client needs to specify both rank and target index (within that server).

 

Thanks

WangDi

 

On 7/11/20, 3:30 PM, "daos@daos.groups.io on behalf of Colin Ngam" <daos@daos.groups.io on behalf of colin.ngam@...> wrote:

 

Greetings,

 

When a Client makes an IO Request, is the endpoint the IO Server or the Target?

 

Thanks.

 

Colin
Client IO Request.

Colin Ngam
 

Greetings,

 

When a Client makes an IO Request, is the endpoint the IO Server or the Target?

 

Thanks.

 

Colin
Re: dfuse fails mounting container

Ruben Felgenhauer <4felgenh@...>
 

Hi Mohamad,

okay I think that clears up everything, thanks very much!

The second question was regarding using DAOS-VOL to write data into DAOS and afterwards mounting the container with DFS to check the container contents on a POSIX level for convenience. But as I said, you already cleared that up perfectly.

I was focusing mainly on DAOS-VOL, so I'll definitely try out the same task as written above with MPICH instead.

Thanks again,
Ruben

Am 11.07.20 um 20:43 schrieb Chaarawi, Mohamad:

Hi Ruben,

I wouldn't call this requirement "new" as it has been there for a long time now.
But yes when using a DFS container (MPI-IO, dfuse, hdf5 over posix or MPI-IO), the container must be of type POSIX.
Yes I guess we can make this clearer in the documentation.

The HDF5 DAOS-VOL (this is different than running HDF5 with the native vol over MPI or POSIX over DFS DAOS) is a different thing. The HDF5 VOL plugin manages the container creation and it uses type HDF5 not POSIX, since underneath it doesn't use the DFS or POSIX API. So not sure what you mean there by DFS will not work at all. Could you elaborate more please?

As for MPICH, it should work fine still. Not sure what doesn't work. But the container created has to be of type POSIX.
Note that the driver has been integrated into upstream MPICH master repo, and there is no need to use the fork for that anymore.

Thanks,
Mohamad

On 7/11/20, 1:36 PM, "daos@daos.groups.io on behalf of Ruben Felgenhauer" <daos@daos.groups.io on behalf of 4felgenh@...> wrote:

Hi Mohamad,

thanks for the tip, I had not created the container as POSIX-type. I
remember that this wasn't explicitly necessary somewhere in the past, is
this a new behavior? The "POSIX Namespace" page of the user guide is
hinting in this direction, but the necessity isn't really stated
explicitly in my opinion.

Does this also mean that DFS will not work at all, if the container was
automatically created by DAOS-VOL / HDF5V? I figure that DAOS-VOL will
not create the container with the posix type?

And I guess, the forked MPICH with the adio driver wouldn't work either?

Kind regards,
Ruben

Am 11.07.20 um 18:12 schrieb Chaarawi, Mohamad:
>
> Hi Ruben,
>
> I assume you create the container with type POSIX?
>
> daos cont create --pool=$DAOS_POOL --svc=$DAOS_SVCL --type=POSIX
>
> and that succeeds and you pass that cont uuid to the dfuse command?
>
> After you create the container, could you please run a cont query on
> that to verify?
>
> daos cont query --pool=$DAOS_POOL --svc=$DAOS_SVCL --cont=$DAOS_CONT
>
> Thanks,
>
> Mohamad
>
> *From: *<daos@daos.groups.io> on behalf of Ruben Felgenhauer
> <4felgenh@...>
> *Reply-To: *"daos@daos.groups.io" <daos@daos.groups.io>
> *Date: *Saturday, July 11, 2020 at 4:32 AM
> *To: *"daos@daos.groups.io" <daos@daos.groups.io>
> *Subject: *[daos] dfuse fails mounting container
>
> Hi,
>
> I'm still failing to get the DAOS Fuse filesystem running. I'm using
> the daos_server_local.yml config with DAOS v0.9 and have server and
> agent running.
>
> Both the high level and low level Fuse interface are failing on me:
>
> Low level:
> OFI_INTERFACE=eth0 dfuse -S --mountpoint="$DFS_MNT" --svc="$DAOS_SVCL"
> --pool="$DAOS_POOL" --container="$DAOS_CONT" --foreground
>
> This returns immediately without any error message and will log the
> following in daos.log:
>
> 07/11-11:16:41.21 abu2 DAOS[62814/62814] fi INFO
> src/gurt/fault_inject.c:486 d_fault_inject_init() No config file,
> fault injection is OFF.
> 07/11-11:16:41.21 abu2 DAOS[62814/62814] crt INFO
> src/cart/crt_init.c:278 crt_init_opt() libcart version 4.7.0 initializing
> 07/11-11:16:41.21 abu2 DAOS[62814/62814] crt WARN
> src/cart/crt_init.c:170 data_init() FI_UNIVERSE_SIZE was not set;
> setting to 2048
> 07/11-11:16:41.46 abu2 DAOS[62814/62814] daos INFO
> src/common/drpc.c:664 drpc_close() Closing dRPC socket fd=21
> 07/11-11:16:41.48 abu2 DAOS[62814/62814] daos INFO
> src/common/drpc.c:664 drpc_close() Closing dRPC socket fd=21
> 07/11-11:16:41.48 abu2 DAOS[62814/62814] dfuse ERR
> src/client/dfuse/dfuse_main.c:328 main(0x559201d7b010) dfs_mount
> failed (22)
> 07/11-11:16:41.48 abu2 DAOS[62814/62814] dfuse ERR
> src/client/dfuse/dfuse_main.c:362 main(0x559201d7b010) DFP left at the end
> 07/11-11:16:41.48 abu2 DAOS[62814/62814] dfuse ERR
> src/client/dfuse/dfuse_main.c:365 main(0x559201d7af80) DFS left at the end
> 07/11-11:16:41.48 abu2 DAOS[62814/62814] dfuse ERR
> src/client/dfuse/dfuse_main.c:372 main(0x559201d7af80) dfs_umount()
> failed (22)
> 07/11-11:16:41.48 abu2 DAOS[62814/62814] dfuse ERR
> src/client/dfuse/dfuse_main.c:378 main(0x559201d7af80)
> daos_cont_close() failed: (-1002)
> 07/11-11:16:41.48 abu2 DAOS[62814/62814] dfuse ERR
> src/client/dfuse/dfuse_main.c:390 main(0x559201d7b010)
> daos_pool_disconnect() failed: (-1002)
> 07/11-11:16:41.49 abu2 DAOS[62814/62814] dfuse INFO
> src/client/dfuse/dfuse_main.c:404 main() Exiting with status 0
>
> High Level:
> $ OFI_INTERFACE=eth0 dfuse_hl "$DFS_MNT" -s -f -d -p "$DAOS_POOL" -l
> "$DAOS_SVCL" -c "$DAOS_CONT"
> Pool Connect...
> DFS Pool = 0cce90ce-2f6c-4621-836f-b24476acefd0
> DFS SVCL = 0
> DFS Container: 6312c82d-7daf-34a2-edb7-3b45441cdd6f
> Failed dfs mount (22)
>
> This logs the following:
> 07/11-11:21:11.29 abu2 DAOS[62833/62833] fi INFO
> src/gurt/fault_inject.c:486 d_fault_inject_init() No config file,
> fault injection is OFF.
> 07/11-11:21:11.29 abu2 DAOS[62833/62833] crt INFO
> src/cart/crt_init.c:278 crt_init_opt() libcart version 4.7.0 initializing
> 07/11-11:21:11.29 abu2 DAOS[62833/62833] crt WARN
> src/cart/crt_init.c:170 data_init() FI_UNIVERSE_SIZE was not set;
> setting to 2048
> 07/11-11:21:11.54 abu2 DAOS[62833/62833] daos INFO
> src/common/drpc.c:664 drpc_close() Closing dRPC socket fd=21
> 07/11-11:21:11.56 abu2 DAOS[62833/62833] daos INFO
> src/common/drpc.c:664 drpc_close() Closing dRPC socket fd=21
>
> Neither of these are mounting anything in "$DFS_MNT". Interestingly,
> if I leave out the --container at the low level dfuse, it seems to
> work at first, but later fails if I want to do anything with the
> $DFS_MNT directory.
>
> Kind regards,
> Ruben
>
>
Re: dfuse fails mounting container

Chaarawi, Mohamad
 

Hi Ruben,

I wouldn't call this requirement "new" as it has been there for a long time now.
But yes when using a DFS container (MPI-IO, dfuse, hdf5 over posix or MPI-IO), the container must be of type POSIX.
Yes I guess we can make this clearer in the documentation.

The HDF5 DAOS-VOL (this is different than running HDF5 with the native vol over MPI or POSIX over DFS DAOS) is a different thing. The HDF5 VOL plugin manages the container creation and it uses type HDF5 not POSIX, since underneath it doesn't use the DFS or POSIX API. So not sure what you mean there by DFS will not work at all. Could you elaborate more please?

As for MPICH, it should work fine still. Not sure what doesn't work. But the container created has to be of type POSIX.
Note that the driver has been integrated into upstream MPICH master repo, and there is no need to use the fork for that anymore.

Thanks,
Mohamad

On 7/11/20, 1:36 PM, "daos@daos.groups.io on behalf of Ruben Felgenhauer" <daos@daos.groups.io on behalf of 4felgenh@...> wrote:

Hi Mohamad,

thanks for the tip, I had not created the container as POSIX-type. I
remember that this wasn't explicitly necessary somewhere in the past, is
this a new behavior? The "POSIX Namespace" page of the user guide is
hinting in this direction, but the necessity isn't really stated
explicitly in my opinion.

Does this also mean that DFS will not work at all, if the container was
automatically created by DAOS-VOL / HDF5V? I figure that DAOS-VOL will
not create the container with the posix type?

And I guess, the forked MPICH with the adio driver wouldn't work either?

Kind regards,
Ruben

Am 11.07.20 um 18:12 schrieb Chaarawi, Mohamad:

> Hi Ruben,
>
> I assume you create the container with type POSIX?
>
> daos cont create --pool=$DAOS_POOL --svc=$DAOS_SVCL --type=POSIX
>
> and that succeeds and you pass that cont uuid to the dfuse command?
>
> After you create the container, could you please run a cont query on
> that to verify?
>
> daos cont query --pool=$DAOS_POOL --svc=$DAOS_SVCL --cont=$DAOS_CONT
>
> Thanks,
>
> Mohamad
>
> *From: *<daos@daos.groups.io> on behalf of Ruben Felgenhauer
> <4felgenh@...>
> *Reply-To: *"daos@daos.groups.io" <daos@daos.groups.io>
> *Date: *Saturday, July 11, 2020 at 4:32 AM
> *To: *"daos@daos.groups.io" <daos@daos.groups.io>
> *Subject: *[daos] dfuse fails mounting container
>
> Hi,
>
> I'm still failing to get the DAOS Fuse filesystem running. I'm using
> the daos_server_local.yml config with DAOS v0.9 and have server and
> agent running.
>
> Both the high level and low level Fuse interface are failing on me:
>
> Low level:
> OFI_INTERFACE=eth0 dfuse -S --mountpoint="$DFS_MNT" --svc="$DAOS_SVCL"
> --pool="$DAOS_POOL" --container="$DAOS_CONT" --foreground
>
> This returns immediately without any error message and will log the
> following in daos.log:
>
> 07/11-11:16:41.21 abu2 DAOS[62814/62814] fi INFO
> src/gurt/fault_inject.c:486 d_fault_inject_init() No config file,
> fault injection is OFF.
> 07/11-11:16:41.21 abu2 DAOS[62814/62814] crt INFO
> src/cart/crt_init.c:278 crt_init_opt() libcart version 4.7.0 initializing
> 07/11-11:16:41.21 abu2 DAOS[62814/62814] crt WARN
> src/cart/crt_init.c:170 data_init() FI_UNIVERSE_SIZE was not set;
> setting to 2048
> 07/11-11:16:41.46 abu2 DAOS[62814/62814] daos INFO
> src/common/drpc.c:664 drpc_close() Closing dRPC socket fd=21
> 07/11-11:16:41.48 abu2 DAOS[62814/62814] daos INFO
> src/common/drpc.c:664 drpc_close() Closing dRPC socket fd=21
> 07/11-11:16:41.48 abu2 DAOS[62814/62814] dfuse ERR
> src/client/dfuse/dfuse_main.c:328 main(0x559201d7b010) dfs_mount
> failed (22)
> 07/11-11:16:41.48 abu2 DAOS[62814/62814] dfuse ERR
> src/client/dfuse/dfuse_main.c:362 main(0x559201d7b010) DFP left at the end
> 07/11-11:16:41.48 abu2 DAOS[62814/62814] dfuse ERR
> src/client/dfuse/dfuse_main.c:365 main(0x559201d7af80) DFS left at the end
> 07/11-11:16:41.48 abu2 DAOS[62814/62814] dfuse ERR
> src/client/dfuse/dfuse_main.c:372 main(0x559201d7af80) dfs_umount()
> failed (22)
> 07/11-11:16:41.48 abu2 DAOS[62814/62814] dfuse ERR
> src/client/dfuse/dfuse_main.c:378 main(0x559201d7af80)
> daos_cont_close() failed: (-1002)
> 07/11-11:16:41.48 abu2 DAOS[62814/62814] dfuse ERR
> src/client/dfuse/dfuse_main.c:390 main(0x559201d7b010)
> daos_pool_disconnect() failed: (-1002)
> 07/11-11:16:41.49 abu2 DAOS[62814/62814] dfuse INFO
> src/client/dfuse/dfuse_main.c:404 main() Exiting with status 0
>
> High Level:
> $ OFI_INTERFACE=eth0 dfuse_hl "$DFS_MNT" -s -f -d -p "$DAOS_POOL" -l
> "$DAOS_SVCL" -c "$DAOS_CONT"
> Pool Connect...
> DFS Pool = 0cce90ce-2f6c-4621-836f-b24476acefd0
> DFS SVCL = 0
> DFS Container: 6312c82d-7daf-34a2-edb7-3b45441cdd6f
> Failed dfs mount (22)
>
> This logs the following:
> 07/11-11:21:11.29 abu2 DAOS[62833/62833] fi INFO
> src/gurt/fault_inject.c:486 d_fault_inject_init() No config file,
> fault injection is OFF.
> 07/11-11:21:11.29 abu2 DAOS[62833/62833] crt INFO
> src/cart/crt_init.c:278 crt_init_opt() libcart version 4.7.0 initializing
> 07/11-11:21:11.29 abu2 DAOS[62833/62833] crt WARN
> src/cart/crt_init.c:170 data_init() FI_UNIVERSE_SIZE was not set;
> setting to 2048
> 07/11-11:21:11.54 abu2 DAOS[62833/62833] daos INFO
> src/common/drpc.c:664 drpc_close() Closing dRPC socket fd=21
> 07/11-11:21:11.56 abu2 DAOS[62833/62833] daos INFO
> src/common/drpc.c:664 drpc_close() Closing dRPC socket fd=21
>
> Neither of these are mounting anything in "$DFS_MNT". Interestingly,
> if I leave out the --container at the low level dfuse, it seems to
> work at first, but later fails if I want to do anything with the
> $DFS_MNT directory.
>
> Kind regards,
> Ruben
>
>

Re: dfuse fails mounting container

Ruben Felgenhauer <4felgenh@...>
 

Hi Mohamad,

thanks for the tip, I had not created the container as POSIX-type. I remember that this wasn't explicitly necessary somewhere in the past, is this a new behavior? The "POSIX Namespace" page of the user guide is hinting in this direction, but the necessity isn't really stated explicitly in my opinion.

Does this also mean that DFS will not work at all, if the container was automatically created by DAOS-VOL / HDF5V? I figure that DAOS-VOL will not create the container with the posix type?

And I guess, the forked MPICH with the adio driver wouldn't work either?

Kind regards,
Ruben

Am 11.07.20 um 18:12 schrieb Chaarawi, Mohamad:


Hi Ruben,

I assume you create the container with type POSIX?

daos cont create --pool=$DAOS_POOL --svc=$DAOS_SVCL --type=POSIX

and that succeeds and you pass that cont uuid to the dfuse command?

After you create the container, could you please run a cont query on that to verify?

daos cont query --pool=$DAOS_POOL --svc=$DAOS_SVCL --cont=$DAOS_CONT

Thanks,

Mohamad

*From: *<daos@daos.groups.io> on behalf of Ruben Felgenhauer <4felgenh@...>
*Reply-To: *"daos@daos.groups.io" <daos@daos.groups.io>
*Date: *Saturday, July 11, 2020 at 4:32 AM
*To: *"daos@daos.groups.io" <daos@daos.groups.io>
*Subject: *[daos] dfuse fails mounting container

Hi,

I'm still failing to get the DAOS Fuse filesystem running. I'm using the daos_server_local.yml config with DAOS v0.9 and have server and agent running.

Both the high level and low level Fuse interface are failing on me:

Low level:
OFI_INTERFACE=eth0 dfuse -S --mountpoint="$DFS_MNT" --svc="$DAOS_SVCL" --pool="$DAOS_POOL" --container="$DAOS_CONT" --foreground

This returns immediately without any error message and will log the following in daos.log:

07/11-11:16:41.21 abu2 DAOS[62814/62814] fi   INFO src/gurt/fault_inject.c:486 d_fault_inject_init() No config file, fault injection is OFF.
07/11-11:16:41.21 abu2 DAOS[62814/62814] crt  INFO src/cart/crt_init.c:278 crt_init_opt() libcart version 4.7.0 initializing
07/11-11:16:41.21 abu2 DAOS[62814/62814] crt  WARN src/cart/crt_init.c:170 data_init() FI_UNIVERSE_SIZE was not set; setting to 2048
07/11-11:16:41.46 abu2 DAOS[62814/62814] daos INFO src/common/drpc.c:664 drpc_close() Closing dRPC socket fd=21
07/11-11:16:41.48 abu2 DAOS[62814/62814] daos INFO src/common/drpc.c:664 drpc_close() Closing dRPC socket fd=21
07/11-11:16:41.48 abu2 DAOS[62814/62814] dfuse ERR src/client/dfuse/dfuse_main.c:328 main(0x559201d7b010) dfs_mount failed (22)
07/11-11:16:41.48 abu2 DAOS[62814/62814] dfuse ERR src/client/dfuse/dfuse_main.c:362 main(0x559201d7b010) DFP left at the end
07/11-11:16:41.48 abu2 DAOS[62814/62814] dfuse ERR src/client/dfuse/dfuse_main.c:365 main(0x559201d7af80) DFS left at the end
07/11-11:16:41.48 abu2 DAOS[62814/62814] dfuse ERR src/client/dfuse/dfuse_main.c:372 main(0x559201d7af80) dfs_umount() failed (22)
07/11-11:16:41.48 abu2 DAOS[62814/62814] dfuse ERR src/client/dfuse/dfuse_main.c:378 main(0x559201d7af80) daos_cont_close() failed: (-1002)
07/11-11:16:41.48 abu2 DAOS[62814/62814] dfuse ERR src/client/dfuse/dfuse_main.c:390 main(0x559201d7b010) daos_pool_disconnect() failed: (-1002)
07/11-11:16:41.49 abu2 DAOS[62814/62814] dfuse INFO src/client/dfuse/dfuse_main.c:404 main() Exiting with status 0

High Level:
$ OFI_INTERFACE=eth0 dfuse_hl "$DFS_MNT" -s -f -d -p "$DAOS_POOL" -l "$DAOS_SVCL" -c "$DAOS_CONT"
Pool Connect...
DFS Pool = 0cce90ce-2f6c-4621-836f-b24476acefd0
DFS SVCL = 0
DFS Container: 6312c82d-7daf-34a2-edb7-3b45441cdd6f
Failed dfs mount (22)

This logs the following:
07/11-11:21:11.29 abu2 DAOS[62833/62833] fi   INFO src/gurt/fault_inject.c:486 d_fault_inject_init() No config file, fault injection is OFF.
07/11-11:21:11.29 abu2 DAOS[62833/62833] crt  INFO src/cart/crt_init.c:278 crt_init_opt() libcart version 4.7.0 initializing
07/11-11:21:11.29 abu2 DAOS[62833/62833] crt  WARN src/cart/crt_init.c:170 data_init() FI_UNIVERSE_SIZE was not set; setting to 2048
07/11-11:21:11.54 abu2 DAOS[62833/62833] daos INFO src/common/drpc.c:664 drpc_close() Closing dRPC socket fd=21
07/11-11:21:11.56 abu2 DAOS[62833/62833] daos INFO src/common/drpc.c:664 drpc_close() Closing dRPC socket fd=21

Neither of these are mounting anything in "$DFS_MNT". Interestingly, if I leave out the --container at the low level dfuse, it seems to work at first, but later fails if I want to do anything with the $DFS_MNT directory.

Kind regards,
Ruben
Re: dfuse fails mounting container

Chaarawi, Mohamad
 

Hi Ruben,

 

I assume you create the container with type POSIX?

daos cont create --pool=$DAOS_POOL --svc=$DAOS_SVCL --type=POSIX

and that succeeds and you pass that cont uuid to the dfuse command?

After you create the container, could you please run a cont query on that to verify?

daos cont query --pool=$DAOS_POOL --svc=$DAOS_SVCL --cont=$DAOS_CONT

 

Thanks,

Mohamad

 

From: <daos@daos.groups.io> on behalf of Ruben Felgenhauer <4felgenh@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Saturday, July 11, 2020 at 4:32 AM
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: [daos] dfuse fails mounting container

 

Hi,

I'm still failing to get the DAOS Fuse filesystem running. I'm using the daos_server_local.yml config with DAOS v0.9 and have server and agent running.

Both the high level and low level Fuse interface are failing on me:

Low level:
OFI_INTERFACE=eth0 dfuse -S --mountpoint="$DFS_MNT" --svc="$DAOS_SVCL" --pool="$DAOS_POOL" --container="$DAOS_CONT" --foreground

This returns immediately without any error message and will log the following in daos.log:

07/11-11:16:41.21 abu2 DAOS[62814/62814] fi   INFO src/gurt/fault_inject.c:486 d_fault_inject_init() No config file, fault injection is OFF.
07/11-11:16:41.21 abu2 DAOS[62814/62814] crt  INFO src/cart/crt_init.c:278 crt_init_opt() libcart version 4.7.0 initializing
07/11-11:16:41.21 abu2 DAOS[62814/62814] crt  WARN src/cart/crt_init.c:170 data_init() FI_UNIVERSE_SIZE was not set; setting to 2048
07/11-11:16:41.46 abu2 DAOS[62814/62814] daos INFO src/common/drpc.c:664 drpc_close() Closing dRPC socket fd=21
07/11-11:16:41.48 abu2 DAOS[62814/62814] daos INFO src/common/drpc.c:664 drpc_close() Closing dRPC socket fd=21
07/11-11:16:41.48 abu2 DAOS[62814/62814] dfuse ERR  src/client/dfuse/dfuse_main.c:328 main(0x559201d7b010) dfs_mount failed (22)
07/11-11:16:41.48 abu2 DAOS[62814/62814] dfuse ERR  src/client/dfuse/dfuse_main.c:362 main(0x559201d7b010) DFP left at the end
07/11-11:16:41.48 abu2 DAOS[62814/62814] dfuse ERR  src/client/dfuse/dfuse_main.c:365 main(0x559201d7af80) DFS left at the end
07/11-11:16:41.48 abu2 DAOS[62814/62814] dfuse ERR  src/client/dfuse/dfuse_main.c:372 main(0x559201d7af80) dfs_umount() failed (22)
07/11-11:16:41.48 abu2 DAOS[62814/62814] dfuse ERR  src/client/dfuse/dfuse_main.c:378 main(0x559201d7af80) daos_cont_close() failed: (-1002)
07/11-11:16:41.48 abu2 DAOS[62814/62814] dfuse ERR  src/client/dfuse/dfuse_main.c:390 main(0x559201d7b010) daos_pool_disconnect() failed: (-1002)
07/11-11:16:41.49 abu2 DAOS[62814/62814] dfuse INFO src/client/dfuse/dfuse_main.c:404 main() Exiting with status 0

High Level:
$ OFI_INTERFACE=eth0 dfuse_hl "$DFS_MNT" -s -f -d -p "$DAOS_POOL" -l "$DAOS_SVCL" -c "$DAOS_CONT"
Pool Connect...
DFS Pool = 0cce90ce-2f6c-4621-836f-b24476acefd0
DFS SVCL = 0
DFS Container: 6312c82d-7daf-34a2-edb7-3b45441cdd6f
Failed dfs mount (22)

This logs the following:
07/11-11:21:11.29 abu2 DAOS[62833/62833] fi   INFO src/gurt/fault_inject.c:486 d_fault_inject_init() No config file, fault injection is OFF.
07/11-11:21:11.29 abu2 DAOS[62833/62833] crt  INFO src/cart/crt_init.c:278 crt_init_opt() libcart version 4.7.0 initializing
07/11-11:21:11.29 abu2 DAOS[62833/62833] crt  WARN src/cart/crt_init.c:170 data_init() FI_UNIVERSE_SIZE was not set; setting to 2048
07/11-11:21:11.54 abu2 DAOS[62833/62833] daos INFO src/common/drpc.c:664 drpc_close() Closing dRPC socket fd=21
07/11-11:21:11.56 abu2 DAOS[62833/62833] daos INFO src/common/drpc.c:664 drpc_close() Closing dRPC socket fd=21

Neither of these are mounting anything in "$DFS_MNT". Interestingly, if I leave out the --container at the low level dfuse, it seems to work at first, but later fails if I want to do anything with the $DFS_MNT directory.

Kind regards,
Ruben

dfuse fails mounting container

Ruben Felgenhauer <4felgenh@...>
 

Hi,

I'm still failing to get the DAOS Fuse filesystem running. I'm using the daos_server_local.yml config with DAOS v0.9 and have server and agent running.

Both the high level and low level Fuse interface are failing on me:

Low level:
OFI_INTERFACE=eth0 dfuse -S --mountpoint="$DFS_MNT" --svc="$DAOS_SVCL" --pool="$DAOS_POOL" --container="$DAOS_CONT" --foreground

This returns immediately without any error message and will log the following in daos.log:

07/11-11:16:41.21 abu2 DAOS[62814/62814] fi   INFO src/gurt/fault_inject.c:486 d_fault_inject_init() No config file, fault injection is OFF.
07/11-11:16:41.21 abu2 DAOS[62814/62814] crt  INFO src/cart/crt_init.c:278 crt_init_opt() libcart version 4.7.0 initializing
07/11-11:16:41.21 abu2 DAOS[62814/62814] crt  WARN src/cart/crt_init.c:170 data_init() FI_UNIVERSE_SIZE was not set; setting to 2048
07/11-11:16:41.46 abu2 DAOS[62814/62814] daos INFO src/common/drpc.c:664 drpc_close() Closing dRPC socket fd=21
07/11-11:16:41.48 abu2 DAOS[62814/62814] daos INFO src/common/drpc.c:664 drpc_close() Closing dRPC socket fd=21
07/11-11:16:41.48 abu2 DAOS[62814/62814] dfuse ERR  src/client/dfuse/dfuse_main.c:328 main(0x559201d7b010) dfs_mount failed (22)
07/11-11:16:41.48 abu2 DAOS[62814/62814] dfuse ERR  src/client/dfuse/dfuse_main.c:362 main(0x559201d7b010) DFP left at the end
07/11-11:16:41.48 abu2 DAOS[62814/62814] dfuse ERR  src/client/dfuse/dfuse_main.c:365 main(0x559201d7af80) DFS left at the end
07/11-11:16:41.48 abu2 DAOS[62814/62814] dfuse ERR  src/client/dfuse/dfuse_main.c:372 main(0x559201d7af80) dfs_umount() failed (22)
07/11-11:16:41.48 abu2 DAOS[62814/62814] dfuse ERR  src/client/dfuse/dfuse_main.c:378 main(0x559201d7af80) daos_cont_close() failed: (-1002)
07/11-11:16:41.48 abu2 DAOS[62814/62814] dfuse ERR  src/client/dfuse/dfuse_main.c:390 main(0x559201d7b010) daos_pool_disconnect() failed: (-1002)
07/11-11:16:41.49 abu2 DAOS[62814/62814] dfuse INFO src/client/dfuse/dfuse_main.c:404 main() Exiting with status 0

High Level:
$ OFI_INTERFACE=eth0 dfuse_hl "$DFS_MNT" -s -f -d -p "$DAOS_POOL" -l "$DAOS_SVCL" -c "$DAOS_CONT"
Pool Connect...
DFS Pool = 0cce90ce-2f6c-4621-836f-b24476acefd0
DFS SVCL = 0
DFS Container: 6312c82d-7daf-34a2-edb7-3b45441cdd6f
Failed dfs mount (22)

This logs the following:
07/11-11:21:11.29 abu2 DAOS[62833/62833] fi   INFO src/gurt/fault_inject.c:486 d_fault_inject_init() No config file, fault injection is OFF.
07/11-11:21:11.29 abu2 DAOS[62833/62833] crt  INFO src/cart/crt_init.c:278 crt_init_opt() libcart version 4.7.0 initializing
07/11-11:21:11.29 abu2 DAOS[62833/62833] crt  WARN src/cart/crt_init.c:170 data_init() FI_UNIVERSE_SIZE was not set; setting to 2048
07/11-11:21:11.54 abu2 DAOS[62833/62833] daos INFO src/common/drpc.c:664 drpc_close() Closing dRPC socket fd=21
07/11-11:21:11.56 abu2 DAOS[62833/62833] daos INFO src/common/drpc.c:664 drpc_close() Closing dRPC socket fd=21

Neither of these are mounting anything in "$DFS_MNT". Interestingly, if I leave out the --container at the low level dfuse, it seems to work at first, but later fails if I want to do anything with the $DFS_MNT directory.

Kind regards,
Ruben

Re: Intel Optane SSD mounting problem

Lombardi, Johann
 

Thanks a lot Patrick!

 

From: <daos@daos.groups.io> on behalf of "Farrell, Patrick Arthur" <patrick.farrell@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Thursday 2 July 2020 at 18:30
To: "daos@daos.groups.io" <daos@daos.groups.io>
Cc: "Liu, Changpeng" <changpeng.liu@...>, "Harris, James R" <james.r.harris@...>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Johann,

 

Patch worked just fine - able to mount, etc.

 

Thanks much.

 

Regards,

-Patrick

 

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
Re: Intel Optane SSD mounting problem

Farrell, Patrick Arthur <patrick.farrell@...>
 

Johann,

Patch worked just fine - able to mount, etc.

Thanks much.
 
Regards,
-Patrick
From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Lombardi, Johann <johann.lombardi@...>
Sent: Wednesday, July 1, 2020 2:47 PM
To: daos@daos.groups.io <daos@daos.groups.io>
Cc: Liu, Changpeng <changpeng.liu@...>; Harris, James R <james.r.harris@...>
Subject: Re: [daos] Intel Optane SSD mounting problem
 

Thanks Patrick!

 

The SPDK team came up with this patch: https://review.spdk.io/gerrit/c/spdk/spdk/+/3156

Would you mind giving it a try please? If this works, we will get the process started to integrate it into SPDK and update DAOS to latest SPDK.

Thanks in advance.

 

Cheers,

Johann

 

From: <daos@daos.groups.io> on behalf of "Farrell, Patrick Arthur" <patrick.farrell@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Tuesday 30 June 2020 at 22:20
To: "daos@daos.groups.io" <daos@daos.groups.io>
Cc: "Liu, Changpeng" <changpeng.liu@...>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

As suspected, the SSDs are now coming online normally.

 

Let me know if there's any further troubleshooting the SPDK team would like.

 

-Patrick

From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Lombardi, Johann <johann.lombardi@...>
Sent: Tuesday, June 30, 2020 5:47 AM
To: daos@daos.groups.io <daos@daos.groups.io>
Cc: Liu, Changpeng <changpeng.liu@...>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Hi Patrick,

 

I talked to the SPDK folks and they suggested to try removing line 1464-1470 in module/bdev/nvme/bdev_nvme.c:

 

-              if (spdk_nvme_ctrlr_get_flags(nvme_bdev_ctrlr->ctrlr) &

-                  SPDK_NVME_CTRLR_SECURITY_SEND_RECV_SUPPORTED) {

-                              nvme_bdev_ctrlr->opal_dev = spdk_opal_dev_construct(nvme_bdev_ctrlr->ctrlr);

-                              if (nvme_bdev_ctrlr->opal_dev == NULL) {

-                                              SPDK_ERRLOG("Failed to initialize Opal\n");

-                              }

-              }

 

Could you please give this a spin?

 

Cheers,

Johann

 

From: <daos@daos.groups.io> on behalf of "Farrell, Patrick Arthur" <patrick.farrell@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Thursday 25 June 2020 at 19:11
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Gert,

 

Thanks for the further information.  Something I don't quite follow:

"The product specification lists both Pyrite as Opal support, what is less clear in the documentation is that Opal is not enable."

 

Then, can Opal be enabled?  Or is it permanently disabled on this model?

 

A general question for you:
Is SPDK simply totally unable to manage some current Intel SSDs if they don't have the full Opal support?

 

Separately, I'm going to dig a little, and I also plan to try older versions of DAOS/SPDK to see what I can figure out.

 

Thanks,

-Patrick

From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Gert Pauwels (intel) <gert.pauwels@...>
Sent: Thursday, June 25, 2020 12:05 PM
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Patrick, Colin,

 

I verified with the NSG Business Unit that builds the 905P SSD.

The product specification lists both Pyrite as Opal support, what is less clear in the documentation is that Opal is not enable. This is causing the behavior you are seeing.

The Identify command correctly returns the Security Send/Receive is supported as this is needed for both Opal and Pyrite.

It is correct that SPDK returns “No Opal support” for the 905P SSD. I verified on my system you do not have this with the Intel Optane SSD P4800X as this support Opal 2.0

 

What I cannot explain right now is why the drive was working with DAOS in the past.

 

Regards,

 

Gert,

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Farrell, Patrick Arthur
Sent: Thursday, June 25, 2020 5:07 PM
To: daos@daos.groups.io
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Recall that we have used this SSD with DAOS in the past.  Whatever is preventing this now is a change, perhaps in SPDK but possibly in how DAOS is invoking it.

 

Regards,

-Patrick

From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Nabarro, Tom <tom.nabarro@...>
Sent: Thursday, June 25, 2020 3:43 AM
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

We probably need to get further information from NSG folks on product specifics regarding Opal support in 905p SSDs. This seems to be a prerequisite for device management through SPDK.

 

Tom

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Farrell, Patrick Arthur
Sent: Wednesday, June 24, 2020 8:42 PM
To: daos@daos.groups.io
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Gert,

 

We're seeing this when we run identify on our 905p SSDs:

"

NVMe Controller at 0000:1a:00.0 [8086:2700]

[...]

Serial Number:                         PHM29226005S480BGN
Model Number:                          INTEL SSDPE21D480GA

[...]

Admin Command Set Attributes

============================

Security Send/Receive:                 Supported

"

 

However, when I run nvme_manage, I get an error stating that Opal is not supported (the same error DAOS is spitting out, incidentally):

 

"Please Input PCI Address(domain:bus:dev.func):

0000:1a:00.00

Opal General Usage:

 

        [1: scan device]

        [2: init - take ownership and activate locking]

        [3: revert tper]

        [4: setup locking range]

        [5: list locking ranges]

        [6: enable user]

        [7: set new password]

        [8: add user to locking range]

        [9: lock/unlock range]

        [10: erase locking range]

        [0: quit]

1

[2020-06-24 14:37:34.452086] nvme_opal.c: 828:opal_discovery0_end: *ERROR*: Opal Not Supported.

"

 

Any thoughts?

 

Regards,

-Patrick

From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Gert Pauwels (intel) <gert.pauwels@...>
Sent: Wednesday, June 24, 2020 11:54 AM
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Hi Tom, Colin,


I'm running Ubuntu 20.04 LTS  (kernel: 5.4.0-37-generic) and compiled DAOS v1.0.0.

I also compiled the latest master as of yesterday, but it did not make a difference.

Any application that can manage Opal 2.0 can be used to check the status of the drive. I used the sedutil-cli, can be found at https://github.com/Drive-Trust-Alliance/sedutil the executable can be found at https://github.com/Drive-Trust-Alliance/sedutil/wiki/Executable-Distributions 
You can run
# sedutil-cli --scan
and
# sedutil-cli --query <device>
it will return useful information about the Opal status of the device.
sedutil-cli will send NVMe security commands and the tool will only work if your SSD is bound the the NVMe driver.

In case the SSD is not bound to the NVMe driver you can manage the drive with the nvme_manage spdk utility you can found in the /daos/_build.external/dev/spdk
# ./examples/nvme/nvme_manage/nvme_manage 
Select 8 to get into the OPAL NVMe Management Options, enter the PCIe address of your SSD for the list you get prompted, select 1 to scan the device. These steps gives you the same results as running the # sedutil-cli --query <device> in case the SSD is bound to the NVMe driver.

Regards,

Gert,
Re: Intel Optane SSD mounting problem

Lombardi, Johann
 

Thanks Patrick!

 

The SPDK team came up with this patch: https://review.spdk.io/gerrit/c/spdk/spdk/+/3156

Would you mind giving it a try please? If this works, we will get the process started to integrate it into SPDK and update DAOS to latest SPDK.

Thanks in advance.

 

Cheers,

Johann

 

From: <daos@daos.groups.io> on behalf of "Farrell, Patrick Arthur" <patrick.farrell@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Tuesday 30 June 2020 at 22:20
To: "daos@daos.groups.io" <daos@daos.groups.io>
Cc: "Liu, Changpeng" <changpeng.liu@...>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

As suspected, the SSDs are now coming online normally.

 

Let me know if there's any further troubleshooting the SPDK team would like.

 

-Patrick

From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Lombardi, Johann <johann.lombardi@...>
Sent: Tuesday, June 30, 2020 5:47 AM
To: daos@daos.groups.io <daos@daos.groups.io>
Cc: Liu, Changpeng <changpeng.liu@...>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Hi Patrick,

 

I talked to the SPDK folks and they suggested to try removing line 1464-1470 in module/bdev/nvme/bdev_nvme.c:

 

-              if (spdk_nvme_ctrlr_get_flags(nvme_bdev_ctrlr->ctrlr) &

-                  SPDK_NVME_CTRLR_SECURITY_SEND_RECV_SUPPORTED) {

-                              nvme_bdev_ctrlr->opal_dev = spdk_opal_dev_construct(nvme_bdev_ctrlr->ctrlr);

-                              if (nvme_bdev_ctrlr->opal_dev == NULL) {

-                                              SPDK_ERRLOG("Failed to initialize Opal\n");

-                              }

-              }

 

Could you please give this a spin?

 

Cheers,

Johann

 

From: <daos@daos.groups.io> on behalf of "Farrell, Patrick Arthur" <patrick.farrell@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Thursday 25 June 2020 at 19:11
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Gert,

 

Thanks for the further information.  Something I don't quite follow:

"The product specification lists both Pyrite as Opal support, what is less clear in the documentation is that Opal is not enable."

 

Then, can Opal be enabled?  Or is it permanently disabled on this model?

 

A general question for you:
Is SPDK simply totally unable to manage some current Intel SSDs if they don't have the full Opal support?

 

Separately, I'm going to dig a little, and I also plan to try older versions of DAOS/SPDK to see what I can figure out.

 

Thanks,

-Patrick

From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Gert Pauwels (intel) <gert.pauwels@...>
Sent: Thursday, June 25, 2020 12:05 PM
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Patrick, Colin,

 

I verified with the NSG Business Unit that builds the 905P SSD.

The product specification lists both Pyrite as Opal support, what is less clear in the documentation is that Opal is not enable. This is causing the behavior you are seeing.

The Identify command correctly returns the Security Send/Receive is supported as this is needed for both Opal and Pyrite.

It is correct that SPDK returns “No Opal support” for the 905P SSD. I verified on my system you do not have this with the Intel Optane SSD P4800X as this support Opal 2.0

 

What I cannot explain right now is why the drive was working with DAOS in the past.

 

Regards,

 

Gert,

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Farrell, Patrick Arthur
Sent: Thursday, June 25, 2020 5:07 PM
To: daos@daos.groups.io
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Recall that we have used this SSD with DAOS in the past.  Whatever is preventing this now is a change, perhaps in SPDK but possibly in how DAOS is invoking it.

 

Regards,

-Patrick

From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Nabarro, Tom <tom.nabarro@...>
Sent: Thursday, June 25, 2020 3:43 AM
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

We probably need to get further information from NSG folks on product specifics regarding Opal support in 905p SSDs. This seems to be a prerequisite for device management through SPDK.

 

Tom

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Farrell, Patrick Arthur
Sent: Wednesday, June 24, 2020 8:42 PM
To: daos@daos.groups.io
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Gert,

 

We're seeing this when we run identify on our 905p SSDs:

"

NVMe Controller at 0000:1a:00.0 [8086:2700]

[...]

Serial Number:                         PHM29226005S480BGN
Model Number:                          INTEL SSDPE21D480GA

[...]

Admin Command Set Attributes

============================

Security Send/Receive:                 Supported

"

 

However, when I run nvme_manage, I get an error stating that Opal is not supported (the same error DAOS is spitting out, incidentally):

 

"Please Input PCI Address(domain:bus:dev.func):

0000:1a:00.00

Opal General Usage:

 

        [1: scan device]

        [2: init - take ownership and activate locking]

        [3: revert tper]

        [4: setup locking range]

        [5: list locking ranges]

        [6: enable user]

        [7: set new password]

        [8: add user to locking range]

        [9: lock/unlock range]

        [10: erase locking range]

        [0: quit]

1

[2020-06-24 14:37:34.452086] nvme_opal.c: 828:opal_discovery0_end: *ERROR*: Opal Not Supported.

"

 

Any thoughts?

 

Regards,

-Patrick

From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Gert Pauwels (intel) <gert.pauwels@...>
Sent: Wednesday, June 24, 2020 11:54 AM
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Hi Tom, Colin,


I'm running Ubuntu 20.04 LTS  (kernel: 5.4.0-37-generic) and compiled DAOS v1.0.0.

I also compiled the latest master as of yesterday, but it did not make a difference.

Any application that can manage Opal 2.0 can be used to check the status of the drive. I used the sedutil-cli, can be found at https://github.com/Drive-Trust-Alliance/sedutil the executable can be found at https://github.com/Drive-Trust-Alliance/sedutil/wiki/Executable-Distributions 
You can run
# sedutil-cli --scan
and
# sedutil-cli --query <device>
it will return useful information about the Opal status of the device.
sedutil-cli will send NVMe security commands and the tool will only work if your SSD is bound the the NVMe driver.

In case the SSD is not bound to the NVMe driver you can manage the drive with the nvme_manage spdk utility you can found in the /daos/_build.external/dev/spdk
# ./examples/nvme/nvme_manage/nvme_manage 
Select 8 to get into the OPAL NVMe Management Options, enter the PCIe address of your SSD for the list you get prompted, select 1 to scan the device. These steps gives you the same results as running the # sedutil-cli --query <device> in case the SSD is bound to the NVMe driver.

Regards,

Gert,

_._,_._,_

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
Re: Intel Optane SSD mounting problem

Farrell, Patrick Arthur <patrick.farrell@...>
 

As suspected, the SSDs are now coming online normally.

Let me know if there's any further troubleshooting the SPDK team would like.

-Patrick
From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Lombardi, Johann <johann.lombardi@...>
Sent: Tuesday, June 30, 2020 5:47 AM
To: daos@daos.groups.io <daos@daos.groups.io>
Cc: Liu, Changpeng <changpeng.liu@...>
Subject: Re: [daos] Intel Optane SSD mounting problem
 

Hi Patrick,

 

I talked to the SPDK folks and they suggested to try removing line 1464-1470 in module/bdev/nvme/bdev_nvme.c:

 

-              if (spdk_nvme_ctrlr_get_flags(nvme_bdev_ctrlr->ctrlr) &

-                  SPDK_NVME_CTRLR_SECURITY_SEND_RECV_SUPPORTED) {

-                              nvme_bdev_ctrlr->opal_dev = spdk_opal_dev_construct(nvme_bdev_ctrlr->ctrlr);

-                              if (nvme_bdev_ctrlr->opal_dev == NULL) {

-                                              SPDK_ERRLOG("Failed to initialize Opal\n");

-                              }

-              }

 

Could you please give this a spin?

 

Cheers,

Johann

 

From: <daos@daos.groups.io> on behalf of "Farrell, Patrick Arthur" <patrick.farrell@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Thursday 25 June 2020 at 19:11
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Gert,

 

Thanks for the further information.  Something I don't quite follow:

"The product specification lists both Pyrite as Opal support, what is less clear in the documentation is that Opal is not enable."

 

Then, can Opal be enabled?  Or is it permanently disabled on this model?

 

A general question for you:
Is SPDK simply totally unable to manage some current Intel SSDs if they don't have the full Opal support?

 

Separately, I'm going to dig a little, and I also plan to try older versions of DAOS/SPDK to see what I can figure out.

 

Thanks,

-Patrick

From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Gert Pauwels (intel) <gert.pauwels@...>
Sent: Thursday, June 25, 2020 12:05 PM
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Patrick, Colin,

 

I verified with the NSG Business Unit that builds the 905P SSD.

The product specification lists both Pyrite as Opal support, what is less clear in the documentation is that Opal is not enable. This is causing the behavior you are seeing.

The Identify command correctly returns the Security Send/Receive is supported as this is needed for both Opal and Pyrite.

It is correct that SPDK returns “No Opal support” for the 905P SSD. I verified on my system you do not have this with the Intel Optane SSD P4800X as this support Opal 2.0

 

What I cannot explain right now is why the drive was working with DAOS in the past.

 

Regards,

 

Gert,

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Farrell, Patrick Arthur
Sent: Thursday, June 25, 2020 5:07 PM
To: daos@daos.groups.io
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Recall that we have used this SSD with DAOS in the past.  Whatever is preventing this now is a change, perhaps in SPDK but possibly in how DAOS is invoking it.

 

Regards,

-Patrick

From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Nabarro, Tom <tom.nabarro@...>
Sent: Thursday, June 25, 2020 3:43 AM
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

We probably need to get further information from NSG folks on product specifics regarding Opal support in 905p SSDs. This seems to be a prerequisite for device management through SPDK.

 

Tom

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Farrell, Patrick Arthur
Sent: Wednesday, June 24, 2020 8:42 PM
To: daos@daos.groups.io
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Gert,

 

We're seeing this when we run identify on our 905p SSDs:

"

NVMe Controller at 0000:1a:00.0 [8086:2700]

[...]

Serial Number:                         PHM29226005S480BGN
Model Number:                          INTEL SSDPE21D480GA

[...]

Admin Command Set Attributes

============================

Security Send/Receive:                 Supported

"

 

However, when I run nvme_manage, I get an error stating that Opal is not supported (the same error DAOS is spitting out, incidentally):

 

"Please Input PCI Address(domain:bus:dev.func):

0000:1a:00.00

Opal General Usage:

 

        [1: scan device]

        [2: init - take ownership and activate locking]

        [3: revert tper]

        [4: setup locking range]

        [5: list locking ranges]

        [6: enable user]

        [7: set new password]

        [8: add user to locking range]

        [9: lock/unlock range]

        [10: erase locking range]

        [0: quit]

1

[2020-06-24 14:37:34.452086] nvme_opal.c: 828:opal_discovery0_end: *ERROR*: Opal Not Supported.

"

 

Any thoughts?

 

Regards,

-Patrick

From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Gert Pauwels (intel) <gert.pauwels@...>
Sent: Wednesday, June 24, 2020 11:54 AM
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Hi Tom, Colin,


I'm running Ubuntu 20.04 LTS  (kernel: 5.4.0-37-generic) and compiled DAOS v1.0.0.

I also compiled the latest master as of yesterday, but it did not make a difference.

Any application that can manage Opal 2.0 can be used to check the status of the drive. I used the sedutil-cli, can be found at https://github.com/Drive-Trust-Alliance/sedutil the executable can be found at https://github.com/Drive-Trust-Alliance/sedutil/wiki/Executable-Distributions 
You can run
# sedutil-cli --scan
and
# sedutil-cli --query <device>
it will return useful information about the Opal status of the device.
sedutil-cli will send NVMe security commands and the tool will only work if your SSD is bound the the NVMe driver.

In case the SSD is not bound to the NVMe driver you can manage the drive with the nvme_manage spdk utility you can found in the /daos/_build.external/dev/spdk
# ./examples/nvme/nvme_manage/nvme_manage 
Select 8 to get into the OPAL NVMe Management Options, enter the PCIe address of your SSD for the list you get prompted, select 1 to scan the device. These steps gives you the same results as running the # sedutil-cli --query <device> in case the SSD is bound to the NVMe driver.

Regards,

Gert,

 

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Intel Corporation NV/SA
Kings Square, Veldkant 31
2550 Kontich
RPM (Bruxelles) 0415.497.718.
Citibank, Brussels, account 570/1031255/09

This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
Re: Intel Optane SSD mounting problem

Farrell, Patrick Arthur <patrick.farrell@...>
 

Heh, yes!  These are among the lines I was planning to try to remove, so will do.

It is interesting to note of course that SEND_RECV is reported by the tool Gert had me using, but that's not a total shock.

-Patrick
From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Lombardi, Johann <johann.lombardi@...>
Sent: Tuesday, June 30, 2020 5:47 AM
To: daos@daos.groups.io <daos@daos.groups.io>
Cc: Liu, Changpeng <changpeng.liu@...>
Subject: Re: [daos] Intel Optane SSD mounting problem
 

Hi Patrick,

 

I talked to the SPDK folks and they suggested to try removing line 1464-1470 in module/bdev/nvme/bdev_nvme.c:

 

-              if (spdk_nvme_ctrlr_get_flags(nvme_bdev_ctrlr->ctrlr) &

-                  SPDK_NVME_CTRLR_SECURITY_SEND_RECV_SUPPORTED) {

-                              nvme_bdev_ctrlr->opal_dev = spdk_opal_dev_construct(nvme_bdev_ctrlr->ctrlr);

-                              if (nvme_bdev_ctrlr->opal_dev == NULL) {

-                                              SPDK_ERRLOG("Failed to initialize Opal\n");

-                              }

-              }

 

Could you please give this a spin?

 

Cheers,

Johann

 

From: <daos@daos.groups.io> on behalf of "Farrell, Patrick Arthur" <patrick.farrell@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Thursday 25 June 2020 at 19:11
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Gert,

 

Thanks for the further information.  Something I don't quite follow:

"The product specification lists both Pyrite as Opal support, what is less clear in the documentation is that Opal is not enable."

 

Then, can Opal be enabled?  Or is it permanently disabled on this model?

 

A general question for you:
Is SPDK simply totally unable to manage some current Intel SSDs if they don't have the full Opal support?

 

Separately, I'm going to dig a little, and I also plan to try older versions of DAOS/SPDK to see what I can figure out.

 

Thanks,

-Patrick

From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Gert Pauwels (intel) <gert.pauwels@...>
Sent: Thursday, June 25, 2020 12:05 PM
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Patrick, Colin,

 

I verified with the NSG Business Unit that builds the 905P SSD.

The product specification lists both Pyrite as Opal support, what is less clear in the documentation is that Opal is not enable. This is causing the behavior you are seeing.

The Identify command correctly returns the Security Send/Receive is supported as this is needed for both Opal and Pyrite.

It is correct that SPDK returns “No Opal support” for the 905P SSD. I verified on my system you do not have this with the Intel Optane SSD P4800X as this support Opal 2.0

 

What I cannot explain right now is why the drive was working with DAOS in the past.

 

Regards,

 

Gert,

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Farrell, Patrick Arthur
Sent: Thursday, June 25, 2020 5:07 PM
To: daos@daos.groups.io
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Recall that we have used this SSD with DAOS in the past.  Whatever is preventing this now is a change, perhaps in SPDK but possibly in how DAOS is invoking it.

 

Regards,

-Patrick

From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Nabarro, Tom <tom.nabarro@...>
Sent: Thursday, June 25, 2020 3:43 AM
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

We probably need to get further information from NSG folks on product specifics regarding Opal support in 905p SSDs. This seems to be a prerequisite for device management through SPDK.

 

Tom

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Farrell, Patrick Arthur
Sent: Wednesday, June 24, 2020 8:42 PM
To: daos@daos.groups.io
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Gert,

 

We're seeing this when we run identify on our 905p SSDs:

"

NVMe Controller at 0000:1a:00.0 [8086:2700]

[...]

Serial Number:                         PHM29226005S480BGN
Model Number:                          INTEL SSDPE21D480GA

[...]

Admin Command Set Attributes

============================

Security Send/Receive:                 Supported

"

 

However, when I run nvme_manage, I get an error stating that Opal is not supported (the same error DAOS is spitting out, incidentally):

 

"Please Input PCI Address(domain:bus:dev.func):

0000:1a:00.00

Opal General Usage:

 

        [1: scan device]

        [2: init - take ownership and activate locking]

        [3: revert tper]

        [4: setup locking range]

        [5: list locking ranges]

        [6: enable user]

        [7: set new password]

        [8: add user to locking range]

        [9: lock/unlock range]

        [10: erase locking range]

        [0: quit]

1

[2020-06-24 14:37:34.452086] nvme_opal.c: 828:opal_discovery0_end: *ERROR*: Opal Not Supported.

"

 

Any thoughts?

 

Regards,

-Patrick

From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Gert Pauwels (intel) <gert.pauwels@...>
Sent: Wednesday, June 24, 2020 11:54 AM
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Hi Tom, Colin,


I'm running Ubuntu 20.04 LTS  (kernel: 5.4.0-37-generic) and compiled DAOS v1.0.0.

I also compiled the latest master as of yesterday, but it did not make a difference.

Any application that can manage Opal 2.0 can be used to check the status of the drive. I used the sedutil-cli, can be found at https://github.com/Drive-Trust-Alliance/sedutil the executable can be found at https://github.com/Drive-Trust-Alliance/sedutil/wiki/Executable-Distributions 
You can run
# sedutil-cli --scan
and
# sedutil-cli --query <device>
it will return useful information about the Opal status of the device.
sedutil-cli will send NVMe security commands and the tool will only work if your SSD is bound the the NVMe driver.

In case the SSD is not bound to the NVMe driver you can manage the drive with the nvme_manage spdk utility you can found in the /daos/_build.external/dev/spdk
# ./examples/nvme/nvme_manage/nvme_manage 
Select 8 to get into the OPAL NVMe Management Options, enter the PCIe address of your SSD for the list you get prompted, select 1 to scan the device. These steps gives you the same results as running the # sedutil-cli --query <device> in case the SSD is bound to the NVMe driver.

Regards,

Gert,

 

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Intel Corporation NV/SA
Kings Square, Veldkant 31
2550 Kontich
RPM (Bruxelles) 0415.497.718.
Citibank, Brussels, account 570/1031255/09

This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
Re: Intel Optane SSD mounting problem

Lombardi, Johann
 

Hi Patrick,

 

I talked to the SPDK folks and they suggested to try removing line 1464-1470 in module/bdev/nvme/bdev_nvme.c:

 

-              if (spdk_nvme_ctrlr_get_flags(nvme_bdev_ctrlr->ctrlr) &

-                  SPDK_NVME_CTRLR_SECURITY_SEND_RECV_SUPPORTED) {

-                              nvme_bdev_ctrlr->opal_dev = spdk_opal_dev_construct(nvme_bdev_ctrlr->ctrlr);

-                              if (nvme_bdev_ctrlr->opal_dev == NULL) {

-                                              SPDK_ERRLOG("Failed to initialize Opal\n");

-                              }

-              }

 

Could you please give this a spin?

 

Cheers,

Johann

 

From: <daos@daos.groups.io> on behalf of "Farrell, Patrick Arthur" <patrick.farrell@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Thursday 25 June 2020 at 19:11
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Gert,

 

Thanks for the further information.  Something I don't quite follow:

"The product specification lists both Pyrite as Opal support, what is less clear in the documentation is that Opal is not enable."

 

Then, can Opal be enabled?  Or is it permanently disabled on this model?

 

A general question for you:
Is SPDK simply totally unable to manage some current Intel SSDs if they don't have the full Opal support?

 

Separately, I'm going to dig a little, and I also plan to try older versions of DAOS/SPDK to see what I can figure out.

 

Thanks,

-Patrick

From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Gert Pauwels (intel) <gert.pauwels@...>
Sent: Thursday, June 25, 2020 12:05 PM
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Patrick, Colin,

 

I verified with the NSG Business Unit that builds the 905P SSD.

The product specification lists both Pyrite as Opal support, what is less clear in the documentation is that Opal is not enable. This is causing the behavior you are seeing.

The Identify command correctly returns the Security Send/Receive is supported as this is needed for both Opal and Pyrite.

It is correct that SPDK returns “No Opal support” for the 905P SSD. I verified on my system you do not have this with the Intel Optane SSD P4800X as this support Opal 2.0

 

What I cannot explain right now is why the drive was working with DAOS in the past.

 

Regards,

 

Gert,

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Farrell, Patrick Arthur
Sent: Thursday, June 25, 2020 5:07 PM
To: daos@daos.groups.io
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Recall that we have used this SSD with DAOS in the past.  Whatever is preventing this now is a change, perhaps in SPDK but possibly in how DAOS is invoking it.

 

Regards,

-Patrick

From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Nabarro, Tom <tom.nabarro@...>
Sent: Thursday, June 25, 2020 3:43 AM
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

We probably need to get further information from NSG folks on product specifics regarding Opal support in 905p SSDs. This seems to be a prerequisite for device management through SPDK.

 

Tom

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Farrell, Patrick Arthur
Sent: Wednesday, June 24, 2020 8:42 PM
To: daos@daos.groups.io
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Gert,

 

We're seeing this when we run identify on our 905p SSDs:

"

NVMe Controller at 0000:1a:00.0 [8086:2700]

[...]

Serial Number:                         PHM29226005S480BGN
Model Number:                          INTEL SSDPE21D480GA

[...]

Admin Command Set Attributes

============================

Security Send/Receive:                 Supported

"

 

However, when I run nvme_manage, I get an error stating that Opal is not supported (the same error DAOS is spitting out, incidentally):

 

"Please Input PCI Address(domain:bus:dev.func):

0000:1a:00.00

Opal General Usage:

 

        [1: scan device]

        [2: init - take ownership and activate locking]

        [3: revert tper]

        [4: setup locking range]

        [5: list locking ranges]

        [6: enable user]

        [7: set new password]

        [8: add user to locking range]

        [9: lock/unlock range]

        [10: erase locking range]

        [0: quit]

1

[2020-06-24 14:37:34.452086] nvme_opal.c: 828:opal_discovery0_end: *ERROR*: Opal Not Supported.

"

 

Any thoughts?

 

Regards,

-Patrick

From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Gert Pauwels (intel) <gert.pauwels@...>
Sent: Wednesday, June 24, 2020 11:54 AM
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Hi Tom, Colin,


I'm running Ubuntu 20.04 LTS  (kernel: 5.4.0-37-generic) and compiled DAOS v1.0.0.

I also compiled the latest master as of yesterday, but it did not make a difference.

Any application that can manage Opal 2.0 can be used to check the status of the drive. I used the sedutil-cli, can be found at https://github.com/Drive-Trust-Alliance/sedutil the executable can be found at https://github.com/Drive-Trust-Alliance/sedutil/wiki/Executable-Distributions 
You can run
# sedutil-cli --scan
and
# sedutil-cli --query <device>
it will return useful information about the Opal status of the device.
sedutil-cli will send NVMe security commands and the tool will only work if your SSD is bound the the NVMe driver.

In case the SSD is not bound to the NVMe driver you can manage the drive with the nvme_manage spdk utility you can found in the /daos/_build.external/dev/spdk
# ./examples/nvme/nvme_manage/nvme_manage 
Select 8 to get into the OPAL NVMe Management Options, enter the PCIe address of your SSD for the list you get prompted, select 1 to scan the device. These steps gives you the same results as running the # sedutil-cli --query <device> in case the SSD is bound to the NVMe driver.

Regards,

Gert,

 

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Intel Corporation NV/SA
Kings Square, Veldkant 31
2550 Kontich
RPM (Bruxelles) 0415.497.718.
Citibank, Brussels, account 570/1031255/09

This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
Re: GCC Version

Venkatesan, Vishwanath <vishwanath.venkatesan@...>
 

Hi Colin,

 

I think this is coming from FIO with their latest ‘atomics’ patch in the latest master.
You can change the fio version to their latest release (3.20) and recompile to fix this problem.

So from daos source tree

cd _build.external/fio/ && git checkout fio-3.20

and recompile with scons.

 

HTH

 

Cheers,

Vish

--------------------------------------------------

Vishwanath Venkatesan

Software Development Engineer

Extreme-scale Storage Architecture and Development (ESAD)

Intel Corporation

 

From: <daos@daos.groups.io> on behalf of "Olivier, Jeffrey V" <jeffrey.v.olivier@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Monday, June 29, 2020 at 11:32AM
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [daos] GCC Version

 

Hi Colin,

 

What platform are you building on and do you know what prerequisite component is being built when you get that message?   Sounds like a bug in that component’s configure script.    In theory, DAOS should work with and without stdatomic.h

 

-Jeff

 

From: <daos@daos.groups.io> on behalf of Colin Ngam <cngam@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Thursday, June 25, 2020 at 4:01 PM
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: [daos] GCC Version

 

Greetings,

 

What GCC version is required for building daos v1.0.0?

 

Just want to verify as I am getting:

..

In file included from crc/crc32c.h:23:0,

                 from crc/crc32c.c:33:

crc/../arch/arch.h:4:23: fatal error: stdatomic.h: No such file or directory

#include <stdatomic.h>

                       ^

compilation terminated.

 

The log recommends GCC 4.9 and up. We are using 4.8.

 

Thanks.

 

Colin
Re: GCC Version

Olivier, Jeffrey V
 

Hi Colin,

 

What platform are you building on and do you know what prerequisite component is being built when you get that message?   Sounds like a bug in that component’s configure script.    In theory, DAOS should work with and without stdatomic.h

 

-Jeff

 

From: <daos@daos.groups.io> on behalf of Colin Ngam <cngam@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Thursday, June 25, 2020 at 4:01 PM
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: [daos] GCC Version

 

Greetings,

 

What GCC version is required for building daos v1.0.0?

 

Just want to verify as I am getting:

..

In file included from crc/crc32c.h:23:0,

                 from crc/crc32c.c:33:

crc/../arch/arch.h:4:23: fatal error: stdatomic.h: No such file or directory

#include <stdatomic.h>

                       ^

compilation terminated.

 

The log recommends GCC 4.9 and up. We are using 4.8.

 

Thanks.

 

Colin
GCC Version

Colin Ngam <cngam@...>
 

Greetings,

 

What GCC version is required for building daos v1.0.0?

 

Just want to verify as I am getting:

..

In file included from crc/crc32c.h:23:0,

                 from crc/crc32c.c:33:

crc/../arch/arch.h:4:23: fatal error: stdatomic.h: No such file or directory

#include <stdatomic.h>

                       ^

compilation terminated.

 

The log recommends GCC 4.9 and up. We are using 4.8.

 

Thanks.

 

Colin
GCC Version

Colin Ngam
 

Greetings,

 

What GCC version is required for building daos v1.0.0?

 

Just want to verify as I am getting:

..

In file included from crc/crc32c.h:23:0,

                 from crc/crc32c.c:33:

crc/../arch/arch.h:4:23: fatal error: stdatomic.h: No such file or directory

#include <stdatomic.h>

                       ^

compilation terminated.

 

The log recommends GCC 4.9 and up. We are using 4.8.

 

Thanks.

 

Colin

 
Re: Intel Optane SSD mounting problem

Nabarro, Tom
 

We upgraded from SPDK v19.04 to v20.01 shortly after releasing DAOS 1.0, that would be my best guess as to the cause (obviously not if you a testing with 1.0 rather than master).

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Farrell, Patrick Arthur
Sent: Thursday, June 25, 2020 4:07 PM
To: daos@daos.groups.io
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Recall that we have used this SSD with DAOS in the past.  Whatever is preventing this now is a change, perhaps in SPDK but possibly in how DAOS is invoking it.

 

Regards,

-Patrick

From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Nabarro, Tom <tom.nabarro@...>
Sent: Thursday, June 25, 2020 3:43 AM
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

We probably need to get further information from NSG folks on product specifics regarding Opal support in 905p SSDs. This seems to be a prerequisite for device management through SPDK.

 

Tom

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Farrell, Patrick Arthur
Sent: Wednesday, June 24, 2020 8:42 PM
To: daos@daos.groups.io
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Gert,

 

We're seeing this when we run identify on our 905p SSDs:

"

NVMe Controller at 0000:1a:00.0 [8086:2700]

[...]

Serial Number:                         PHM29226005S480BGN
Model Number:                          INTEL SSDPE21D480GA

[...]

Admin Command Set Attributes

============================

Security Send/Receive:                 Supported

"

 

However, when I run nvme_manage, I get an error stating that Opal is not supported (the same error DAOS is spitting out, incidentally):

 

"Please Input PCI Address(domain:bus:dev.func):

0000:1a:00.00

Opal General Usage:

 

        [1: scan device]

        [2: init - take ownership and activate locking]

        [3: revert tper]

        [4: setup locking range]

        [5: list locking ranges]

        [6: enable user]

        [7: set new password]

        [8: add user to locking range]

        [9: lock/unlock range]

        [10: erase locking range]

        [0: quit]

1

[2020-06-24 14:37:34.452086] nvme_opal.c: 828:opal_discovery0_end: *ERROR*: Opal Not Supported.

"

 

Any thoughts?

 

Regards,

-Patrick

From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Gert Pauwels (intel) <gert.pauwels@...>
Sent: Wednesday, June 24, 2020 11:54 AM
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Hi Tom, Colin,


I'm running Ubuntu 20.04 LTS  (kernel: 5.4.0-37-generic) and compiled DAOS v1.0.0.

I also compiled the latest master as of yesterday, but it did not make a difference.

Any application that can manage Opal 2.0 can be used to check the status of the drive. I used the sedutil-cli, can be found at https://github.com/Drive-Trust-Alliance/sedutil the executable can be found at https://github.com/Drive-Trust-Alliance/sedutil/wiki/Executable-Distributions 
You can run
# sedutil-cli --scan
and
# sedutil-cli --query <device>
it will return useful information about the Opal status of the device.
sedutil-cli will send NVMe security commands and the tool will only work if your SSD is bound the the NVMe driver.

In case the SSD is not bound to the NVMe driver you can manage the drive with the nvme_manage spdk utility you can found in the /daos/_build.external/dev/spdk
# ./examples/nvme/nvme_manage/nvme_manage 
Select 8 to get into the OPAL NVMe Management Options, enter the PCIe address of your SSD for the list you get prompted, select 1 to scan the device. These steps gives you the same results as running the # sedutil-cli --query <device> in case the SSD is bound to the NVMe driver.

Regards,

Gert,

 

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
Re: Intel Optane SSD mounting problem

Farrell, Patrick Arthur <patrick.farrell@...>
 

Gert,

Thanks for the further information.  Something I don't quite follow:

"The product specification lists both Pyrite as Opal support, what is less clear in the documentation is that Opal is not enable."

Then, can Opal be enabled?  Or is it permanently disabled on this model?

A general question for you:
Is SPDK simply totally unable to manage some current Intel SSDs if they don't have the full Opal support?

Separately, I'm going to dig a little, and I also plan to try older versions of DAOS/SPDK to see what I can figure out.

Thanks,
-Patrick
From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Gert Pauwels (intel) <gert.pauwels@...>
Sent: Thursday, June 25, 2020 12:05 PM
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem
 

Patrick, Colin,

 

I verified with the NSG Business Unit that builds the 905P SSD.

The product specification lists both Pyrite as Opal support, what is less clear in the documentation is that Opal is not enable. This is causing the behavior you are seeing.

The Identify command correctly returns the Security Send/Receive is supported as this is needed for both Opal and Pyrite.

It is correct that SPDK returns “No Opal support” for the 905P SSD. I verified on my system you do not have this with the Intel Optane SSD P4800X as this support Opal 2.0

 

What I cannot explain right now is why the drive was working with DAOS in the past.

 

Regards,

 

Gert,

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Farrell, Patrick Arthur
Sent: Thursday, June 25, 2020 5:07 PM
To: daos@daos.groups.io
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Recall that we have used this SSD with DAOS in the past.  Whatever is preventing this now is a change, perhaps in SPDK but possibly in how DAOS is invoking it.

 

Regards,

-Patrick

From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Nabarro, Tom <tom.nabarro@...>
Sent: Thursday, June 25, 2020 3:43 AM
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

We probably need to get further information from NSG folks on product specifics regarding Opal support in 905p SSDs. This seems to be a prerequisite for device management through SPDK.

 

Tom

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Farrell, Patrick Arthur
Sent: Wednesday, June 24, 2020 8:42 PM
To: daos@daos.groups.io
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Gert,

 

We're seeing this when we run identify on our 905p SSDs:

"

NVMe Controller at 0000:1a:00.0 [8086:2700]

[...]

Serial Number:                         PHM29226005S480BGN
Model Number:                          INTEL SSDPE21D480GA

[...]

Admin Command Set Attributes

============================

Security Send/Receive:                 Supported

"

 

However, when I run nvme_manage, I get an error stating that Opal is not supported (the same error DAOS is spitting out, incidentally):

 

"Please Input PCI Address(domain:bus:dev.func):

0000:1a:00.00

Opal General Usage:

 

        [1: scan device]

        [2: init - take ownership and activate locking]

        [3: revert tper]

        [4: setup locking range]

        [5: list locking ranges]

        [6: enable user]

        [7: set new password]

        [8: add user to locking range]

        [9: lock/unlock range]

        [10: erase locking range]

        [0: quit]

1

[2020-06-24 14:37:34.452086] nvme_opal.c: 828:opal_discovery0_end: *ERROR*: Opal Not Supported.

"

 

Any thoughts?

 

Regards,

-Patrick

From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Gert Pauwels (intel) <gert.pauwels@...>
Sent: Wednesday, June 24, 2020 11:54 AM
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: Re: [daos] Intel Optane SSD mounting problem

 

Hi Tom, Colin,


I'm running Ubuntu 20.04 LTS  (kernel: 5.4.0-37-generic) and compiled DAOS v1.0.0.

I also compiled the latest master as of yesterday, but it did not make a difference.

Any application that can manage Opal 2.0 can be used to check the status of the drive. I used the sedutil-cli, can be found at https://github.com/Drive-Trust-Alliance/sedutil the executable can be found at https://github.com/Drive-Trust-Alliance/sedutil/wiki/Executable-Distributions 
You can run
# sedutil-cli --scan
and
# sedutil-cli --query <device>
it will return useful information about the Opal status of the device.
sedutil-cli will send NVMe security commands and the tool will only work if your SSD is bound the the NVMe driver.

In case the SSD is not bound to the NVMe driver you can manage the drive with the nvme_manage spdk utility you can found in the /daos/_build.external/dev/spdk
# ./examples/nvme/nvme_manage/nvme_manage 
Select 8 to get into the OPAL NVMe Management Options, enter the PCIe address of your SSD for the list you get prompted, select 1 to scan the device. These steps gives you the same results as running the # sedutil-cli --query <device> in case the SSD is bound to the NVMe driver.

Regards,

Gert,

 

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Intel Corporation NV/SA
Kings Square, Veldkant 31
2550 Kontich
RPM (Bruxelles) 0415.497.718.
Citibank, Brussels, account 570/1031255/09

This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
561 - 580 of 1664