Re: daos_server storage prepare failures


Nabarro, Tom
 

PR landed to master, Commit 0396c0a218c7e63fe54ddb9e2dbb04345a99807d

 

Regards,

Tom Nabarro – DCG/ESAD

M: +44 (0)7786 260986

Skype: tom.nabarro

 

From: daos@daos.groups.io [mailto:daos@daos.groups.io] On Behalf Of Nabarro, Tom
Sent: Saturday, October 26, 2019 4:00 AM
To: daos@daos.groups.io
Subject: Re: [daos] daos_server storage prepare failures

 

Apologies that this regression slipped in

PR for the fix https://github.com/daos-stack/daos/pull/1323

 

Regards,

Tom Nabarro – DCG/ESAD

M: +44 (0)7786 260986

Skype: tom.nabarro

 

From: daos@daos.groups.io [mailto:daos@daos.groups.io] On Behalf Of Kevan Rehm
Sent: Friday, October 25, 2019 9:00 PM
To: daos@daos.groups.io
Subject: [daos] daos_server storage prepare failures

 

Greetings,

 

I’ve been tracking down a problem where “daos_server storage prepare” fails every time.  I am running daos master, up to date as of a few minutes ago.  The command looks like this:

 

/home/users/daos/daos/install/bin/daos_server storage prepare --debug --nvme-only --target-user=daos -p 1024 -o /home/users/daos/daos/utils/config/examples/daos_server_local.yml

 

Output is this:

 

Preparing locally-attached NVMe storage...

DEBUG 14:26:03.463504 storage_nvme.go:104: spdk setup with _NRHUGE=1024

DEBUG 14:26:03.463959 storage_nvme.go:108: spdk setup with _TARGET_USER=daos

DEBUG 14:26:13.369224 main.go:67: scan error(s):

  SPDK setup: spdk setup failed (_NRHUGE=1024, _TARGET_USER=daos, , chown: cannot access ‘libvirt’: No such file or directory

): exit status 123

 

main.concatErrors

       /home/users/daos/daos/build/src/control/src/github.com/daos-stack/daos/src/control/cmd/daos_server/storage.go:94

main.(*storagePrepareCmd).Execute

       /home/users/daos/daos/build/src/control/src/github.com/daos-stack/daos/src/control/cmd/daos_server/storage.go:160

main.parseOpts.func1

       /home/users/daos/daos/build/src/control/src/github.com/daos-stack/daos/src/control/cmd/daos_server/main.go:103

github.com/daos-stack/daos/src/control/vendor/github.com/jessevdk/go-flags.(*Parser).ParseArgs

       /home/users/daos/daos/build/src/control/src/github.com/daos-stack/daos/src/control/vendor/github.com/jessevdk/go-flags/parser.go:314

main.parseOpts

       /home/users/daos/daos/build/src/control/src/github.com/daos-stack/daos/src/control/cmd/daos_server/main.go:111

main.main

       /home/users/daos/daos/build/src/control/src/github.com/daos-stack/daos/src/control/cmd/daos_server/main.go:123

runtime.main

       /usr/lib/golang/src/runtime/proc.go:203

runtime.goexit

       /usr/lib/golang/src/runtime/asm_amd64.s:1357

ERROR: scan error(s):

  SPDK setup: spdk setup failed (_NRHUGE=1024, _TARGET_USER=daos, , chown: cannot access ‘libvirt’: No such file or directory

): exit status 123

 

The problem is in file src/control/server/init/setup_spdk.sh, recently changed.   Code section is this:

 

        # build arglist manually to filter missing directories/files

        # so we don't error on non-existent entities

        for glob in '/dev/hugepages' '/dev/uio*'                \

                '/sys/class/uio/uio*/device/config'     \

                '/sys/class/uio/uio*/device/resource*'; do

 

                if list=$(ls $glob); then

                        echo "RUN: ls $glob | xargs -r chown -R $_TARGET_USER"

                        echo "$list" | xargs -r chown -R "$_TARGET_USER"

                fi

        done

 

On my machine, /dev/hugepages looks like this:

 

[root@hl-d109 ~]# ls -lR /dev/hugepages

/dev/hugepages:

total 0

drwxr-xr-x 3 root root 0 Oct 25 13:26 libvirt

 

/dev/hugepages/libvirt:

total 0

drwxr-xr-x 2 root root 0 Oct 25 13:26 qemu

 

/dev/hugepages/libvirt/qemu:

total 0

 

I think the intent was for the pathnames in the glob variable to be absolute pathnames, but the above code yields relative pathnames, i.e. libvirt instead of /dev/hugepages/libvirt, so the chown in the xargs statement fails because the command is not be executed inside /dev/hugepages.

 

If you need more information, let me know.

 

Thanks, Kevan

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Join daos@daos.groups.io to automatically receive all group messages.