Re: DAOS master error when formatting tmpfs


Jordan Henderson
 

Hi Tom,

I had a little more time to play around with this and it looks like I only encounter the issue if the 'scm_mount_path' or 'scm_mount' that I use in my server configuration file differs from '/mnt/daos'. When it is set to the default '/mnt/daos' I don't have any problems, but if I set it to something else I encounter the error about unformatted storage.


From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Nabarro, Tom via Groups.Io <tom.nabarro@...>
Sent: Friday, October 25, 2019 12:37 PM
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: Re: [daos] DAOS master error when formatting tmpfs
 

Let’s focus on the non-root case to start with, after you have manually created empty /mnt/daos and mounted tmpfs, change permissions to 777 (just for this experiment) on the SCM directory and run daos_server with control_log_mask: DEBUG (please also paste config file). The superblock should be created and the IO server start.

 

Thanks

 

Regards,

Tom Nabarro – DCG/ESAD

M: +44 (0)7786 260986

Skype: tom.nabarro

 

From: daos@daos.groups.io [mailto:daos@daos.groups.io] On Behalf Of Jordan Henderson
Sent: Friday, October 25, 2019 5:00 PM
To: daos@daos.groups.io
Subject: Re: [daos] DAOS master error when formatting tmpfs

 

Hi Tom,

 

in general I don't usually run as root, but in this case it did seem to be the only way that I could get the server to wait for storage formatting. However, even when I started from a clean slate for the tmpfs mount, as per these instructions, it didn't seem to matter whether I manually mounted the tmpfs myself or allowed the storage formatting to do so. When running as root, the storage format appeared to be successful in both cases, but the server still immediately returned an error for the storage not being formatted and then exited. When running as non-root, the server didn't wait for storage formatting in either case.

 

It might be worth noting that after each successful format command, my tmpfs mount still didn't contain a superblock file. I'm guessing that this is probably why the server is returning a formatting error right after the storage format command?

 


From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Nabarro, Tom via Groups.Io <tom.nabarro@...>
Sent: Friday, October 25, 2019 8:58 AM
To: Chaarawi, Mohamad <mohamad.chaarawi@...>; Jordan Henderson <jhenderson@...>
Cc: daos@daos.groups.io <daos@daos.groups.io>
Subject: Re: [daos] DAOS master error when formatting tmpfs

 

Hello Jordan

 

If running as root, could you please try “umount /mnt/daos; rm -rf /mnt/daos” then start server which should wait for format, then format from daos_shell.

 

$  umount /mnt/daos; rm -rf /mnt/daos

$  orterun -N 1 -H localhost --report-uri /tmp/urifile --allow-run-as-root daos_server start -t 1 -o ~tanabarr/projects/daos_m/utils/config/examples/daos_server_sockets.yml -i

$ daos_shell -i storage format

 

If running as non-root, the following works for me:

 

$  sudo umount /mnt/daos; sudo rm -rf /mnt/daos; sudo mkdir /mnt/daos; sudo mount -t tmpfs -o size=64G tmpfs /mnt/daos

$ orterun -np 1 -H localhost --report-uri /tmp/urifile daos_server start -t 1 -d /tmp/ -o ~tanabarr/projects/daos_m/utils/config/examples/daos_server_sockets.yml –i

 

Above tried on commit 15168685005843766c038afff45fd6681c07f341 .

86f730a37d0170fdb733c1b08308162a245e5aea did introduce changes to the formatting code but I haven’t observed any subsequent regressions in my testing, it may just need a clean slate in your situation.

 

Thanks

 

Regards,

Tom Nabarro – DCG/ESAD

M: +44 (0)7786 260986

Skype: tom.nabarro

 

From: Chaarawi, Mohamad
Sent: Friday, October 25, 2019 2:24 PM
To: Jordan Henderson <jhenderson@...>
Cc: Nabarro, Tom <tom.nabarro@...>
Subject: Re: DAOS master error when formatting tmpfs

 

Jordan, it would be great to send such emails to the DAOS user list where there are more people (from the control plane) who can answer. Please subscribe here:

https://daos.groups.io/g/daos

 

did you clear your tmpfs beforehand?

Im not sure why you would be getting this over tmpfs, but I never start the server as root.

 

Mohamad

 

 

From: Jordan Henderson <jhenderson@...>
Date: Thursday, October 24, 2019 at 10:22 PM
To: "Chaarawi, Mohamad" <mohamad.chaarawi@...>
Subject: DAOS master error when formatting tmpfs

 

Hi Mohamad,

 

are you aware of any bugs with the current DAOS master (commit 15168685005843766c038afff45fd6681c07f341) and trying to format storage when the SCM is emulated through a tmpfs? I tried updating my DAOS to the latest master but when I tried to start DAOS I got the error below. If I started the server as root it then waited for me to format the storage, but when I did so the format command returned success and then the server immediately returned the same error as below and exited. Once I switched to the v0.6 tag (which was also ahead of my current install), I was able to run the server as non-root, format the tmpfs and start the server fine.

 

DEBUG 21:03:05.903946 instance.go:199: /home/jhenderson/Work/DAOS_Workspace/daos_fs/ (ram) needs format: true

ERROR: failed to read existing superblock: can't read superblock from unformatted storage

DEBUG 21:03:05.904215 main.go:67: can't read superblock from unformatted storage

github.com/daos-stack/daos/src/control/server.(*IOServerInstance).ReadSuperblock

/home/jhenderson/git/daos/build/src/control/src/github.com/daos-stack/daos/src/control/server/superblock.go:176

github.com/daos-stack/daos/src/control/server.(*IOServerInstance).NeedsSuperblock

/home/jhenderson/git/daos/build/src/control/src/github.com/daos-stack/daos/src/control/server/superblock.go:110

github.com/daos-stack/daos/src/control/server.(*IOServerHarness).CreateSuperblocks

/home/jhenderson/git/daos/build/src/control/src/github.com/daos-stack/daos/src/control/server/harness.go:118

github.com/daos-stack/daos/src/control/server.Start

/home/jhenderson/git/daos/build/src/control/src/github.com/daos-stack/daos/src/control/server/server.go:166

main.(*startCmd).Execute

/home/jhenderson/git/daos/build/src/control/src/github.com/daos-stack/daos/src/control/cmd/daos_server/start.go:167

main.parseOpts.func1

/home/jhenderson/git/daos/build/src/control/src/github.com/daos-stack/daos/src/control/cmd/daos_server/main.go:103

github.com/daos-stack/daos/src/control/vendor/github.com/jessevdk/go-flags.(*Parser).ParseArgs

/home/jhenderson/git/daos/build/src/control/src/github.com/daos-stack/daos/src/control/vendor/github.com/jessevdk/go-flags/parser.go:314

main.parseOpts

/home/jhenderson/git/daos/build/src/control/src/github.com/daos-stack/daos/src/control/cmd/daos_server/main.go:111

main.main

/home/jhenderson/git/daos/build/src/control/src/github.com/daos-stack/daos/src/control/cmd/daos_server/main.go:123

runtime.main

/usr/lib64/go1.11.9/go/src/runtime/proc.go:201

runtime.goexit

/usr/lib64/go1.11.9/go/src/runtime/asm_amd64.s:1333

failed to read existing superblock

github.com/daos-stack/daos/src/control/server.(*IOServerInstance).NeedsSuperblock

/home/jhenderson/git/daos/build/src/control/src/github.com/daos-stack/daos/src/control/server/superblock.go:117

github.com/daos-stack/daos/src/control/server.(*IOServerHarness).CreateSuperblocks

/home/jhenderson/git/daos/build/src/control/src/github.com/daos-stack/daos/src/control/server/harness.go:118

github.com/daos-stack/daos/src/control/server.Start

/home/jhenderson/git/daos/build/src/control/src/github.com/daos-stack/daos/src/control/server/server.go:166

main.(*startCmd).Execute

/home/jhenderson/git/daos/build/src/control/src/github.com/daos-stack/daos/src/control/cmd/daos_server/start.go:167

main.parseOpts.func1

/home/jhenderson/git/daos/build/src/control/src/github.com/daos-stack/daos/src/control/cmd/daos_server/main.go:103

github.com/daos-stack/daos/src/control/vendor/github.com/jessevdk/go-flags.(*Parser).ParseArgs

/home/jhenderson/git/daos/build/src/control/src/github.com/daos-stack/daos/src/control/vendor/github.com/jessevdk/go-flags/parser.go:314

main.parseOpts

/home/jhenderson/git/daos/build/src/control/src/github.com/daos-stack/daos/src/control/cmd/daos_server/main.go:111

main.main

/home/jhenderson/git/daos/build/src/control/src/github.com/daos-stack/daos/src/control/cmd/daos_server/main.go:123

runtime.main

/usr/lib64/go1.11.9/go/src/runtime/proc.go:201

runtime.goexit

/usr/lib64/go1.11.9/go/src/runtime/asm_amd64.s:1333

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Join daos@daos.groups.io to automatically receive all group messages.