Re: daos_io_server fails if daos_server.log is missing


Nabarro, Tom
 

Hello Kevan,

 

Can you supply your server configuration file please, the ownership changes that are being triggered are part of a previous attempt to work around requiring elevated permissions when formatting storage prior to wanting to run daos_io_server as a normal user.

 

This code will be removed in a current WIP PR of mine as this requirement will not exist when the formatting operations are performed by the standalone privileged binary (daos_admin). The requisite work will be completed soon.

 

I am imagining you have “username” set in your config file, try removing that (or groupname) if set.

 

Apologies for any inconvenience.

 

Regards,

Tom Nabarro – DCG/ESAD

M: +44 (0)7786 260986

Skype: tom.nabarro

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Kevan Rehm
Sent: Friday, December 13, 2019 8:40 PM
To: daos@daos.groups.io
Subject: [daos] daos_io_server fails if daos_server.log is missing

 

Greetings,

 

I’d like to mention a bug that I tripped on today, in case I’m the first to notice it.   As background, I followed the instructions from about a week ago on how to chown/chgrp/chmod the binaries daos_admin and daos_server to make them setuid so that daos_server could run as user daos.   If I do try to run daos_server as user daos with a “file” nvme device, it fails in spdk initialization, I’m not sure why yet, but…

 

If I instead start daos_server as root,  it dies if file /tmp/daos_server.log does not exist. 

 

changing file ownership

hl-c507 ERROR 2019/12/13 13:20:14 changing file ownership: recursive chown /tmp/daos_server_svr1.log: accessing path /tmp/daos_server_svr1.log: lstat /tmp/daos_server_svr1.log: no such file or directory

 

Routine chownAll() checks the error code from a failed chown for !os.IsNotExist(err) but routine ChownR is apparently returning some other error, so the test fails, and the daos_io_server exits.

 

I will try touching the file on each server to get around this.  If there’s not a Jira already, let me know.

 

Thanks, Kevan

 

P.S. here is the daos_control.log:

 

DEBUG 13:20:13.942072 start.go:145: Switching control log level to DEBUG

DEBUG 13:20:13.942860 server.go:63: cfg: &server.Configuration{ControlPort:10000, TransportConfig:(*security.TransportConfig)(0xc00018d420), Servers:[]*ioserver.Config{(*ioserver.Config)(0xc000001b00)}, BdevInclude:[]string(nil), BdevExclude:[]string(nil), NrHugepages:4096, ControlLogMask:3, ControlLogFile:"/tmp/daos_control.log", ControlLogJSON:false, HelperLogFile:"", UserName:"daos", GroupName:"daos_grp", RecreateSuperblocks:true, SystemName:"daos_krehm", SocketDir:"/var/run/daos_server", Fabric:ioserver.FabricConfig{Provider:"ofi+verbs;ofi_rxm", Interface:"", InterfacePort:0, PinnedNumaNode:(*uint)(nil)}, Modules:"", AccessPoints:[]string{"hl-c500:10000"}, FaultPath:"", FaultCb:"", Hyperthreads:false, Path:"/home/users/daos/daos/install/etc/daos_server.yml", ext:(*server.ext)(0xc00012e880), NvmeShmID:227167734, validateProviderFn:(server.networkProviderValidation)(0x5e1c00), validateNUMAFn:(server.networkNUMAValidation)(0x5e3110)}

DEBUG 13:20:13.942904 netdetect.go:749: Calling ValidateProviderConfig with ib0, ofi+verbs;ofi_rxm

DEBUG 13:20:13.942945 netdetect.go:793: Input provider string: ofi+verbs;ofi_rxm

DEBUG 13:20:14.077492 netdetect.go:825: There are 0 hfi1 devices in the system

DEBUG 13:20:14.077543 netdetect.go:764: Device ib0 supports provider: ofi+verbs;ofi_rxm

DEBUG 13:20:14.077769 netdetect.go:901: Validate network config -- given numaNode: 0

DEBUG 13:20:14.208993 netdetect.go:926: The NUMA node for device ib0 matches the provided value 0.  Network configuration is valid.

DEBUG 13:20:14.209736 config.go:410: Active config saved to /home/users/daos/daos/install/etc/.daos_server.active.yml (read-only)

hl-c507 INFO 2019/12/13 13:20:14 DAOS control server (pid 24144) listening on 0.0.0.0:10000

hl-c507 INFO 2019/12/13 13:20:14 Waiting for I/O server instance storage to be ready...

DEBUG 13:20:14.309826 instance.go:177: /mnt/daos/svr1: checking formatting

DEBUG 13:20:14.319004 instance.go:195: /mnt/daos/svr1 (ram) needs format: false

DEBUG 13:20:14.319035 superblock.go:107: /mnt/daos/svr1: checking superblock

DEBUG 13:20:14.319052 instance.go:177: /mnt/daos/svr1: checking formatting

DEBUG 13:20:14.327795 instance.go:195: /mnt/daos/svr1 (ram) needs format: false

DEBUG 13:20:14.329552 drpc_server.go:85: Quitting listener

hl-c507 ERROR 2019/12/13 13:20:14 /var/run/daos_server/daos_server.sock: failed to accept connection: accept unixpacket /var/run/daos_server/daos_server.sock: use of closed network connection

DEBUG 13:20:14.329973 main.go:68: lstat /tmp/daos_server_svr1.log: no such file or directory

accessing path /tmp/daos_server_svr1.log

github.com/daos-stack/daos/src/control/server.(*ext).chownR.func1

        /home/users/daos/daos/build/src/control/src/github.com/daos-stack/daos/src/control/server/external.go:101

path/filepath.Walk

        /usr/lib/golang/src/path/filepath/path.go:402

github.com/daos-stack/daos/src/control/server.(*ext).chownR

        /home/users/daos/daos/build/src/control/src/github.com/daos-stack/daos/src/control/server/external.go:99

github.com/daos-stack/daos/src/control/server.chownAll

        /home/users/daos/daos/build/src/control/src/github.com/daos-stack/daos/src/control/server/ownership.go:95

github.com/daos-stack/daos/src/control/server.changeFileOwnership

        /home/users/daos/daos/build/src/control/src/github.com/daos-stack/daos/src/control/server/ownership.go:122

github.com/daos-stack/daos/src/control/server.Start

        /home/users/daos/daos/build/src/control/src/github.com/daos-stack/daos/src/control/server/server.go:207

main.(*startCmd).Execute

        /home/users/daos/daos/build/src/control/src/github.com/daos-stack/daos/src/control/cmd/daos_server/start.go:165

main.parseOpts.func1

        /home/users/daos/daos/build/src/control/src/github.com/daos-stack/daos/src/control/cmd/daos_server/main.go:107

github.com/daos-stack/daos/src/control/vendor/github.com/jessevdk/go-flags.(*Parser).ParseArgs

        /home/users/daos/daos/build/src/control/src/github.com/daos-stack/daos/src/control/vendor/github.com/jessevdk/go-flags/parser.go:314

main.parseOpts

        /home/users/daos/daos/build/src/control/src/github.com/daos-stack/daos/src/control/cmd/daos_server/main.go:115

main.main

        /home/users/daos/daos/build/src/control/src/github.com/daos-stack/daos/src/control/cmd/daos_server/main.go:127

runtime.main

        /usr/lib/golang/src/runtime/proc.go:203

runtime.goexit

        /usr/lib/golang/src/runtime/asm_amd64.s:1357

recursive chown /tmp/daos_server_svr1.log

github.com/daos-stack/daos/src/control/server.chownAll

        /home/users/daos/daos/build/src/control/src/github.com/daos-stack/daos/src/control/server/ownership.go:97

github.com/daos-stack/daos/src/control/server.changeFileOwnership

        /home/users/daos/daos/build/src/control/src/github.com/daos-stack/daos/src/control/server/ownership.go:122

github.com/daos-stack/daos/src/control/server.Start

        /home/users/daos/daos/build/src/control/src/github.com/daos-stack/daos/src/control/server/server.go:207

main.(*startCmd).Execute

        /home/users/daos/daos/build/src/control/src/github.com/daos-stack/daos/src/control/cmd/daos_server/start.go:165

main.parseOpts.func1

        /home/users/daos/daos/build/src/control/src/github.com/daos-stack/daos/src/control/cmd/daos_server/main.go:107

github.com/daos-stack/daos/src/control/vendor/github.com/jessevdk/go-flags.(*Parser).ParseArgs

        /home/users/daos/daos/build/src/control/src/github.com/daos-stack/daos/src/control/vendor/github.com/jessevdk/go-flags/parser.go:314

main.parseOpts

        /home/users/daos/daos/build/src/control/src/github.com/daos-stack/daos/src/control/cmd/daos_server/main.go:115

main.main

        /home/users/daos/daos/build/src/control/src/github.com/daos-stack/daos/src/control/cmd/daos_server/main.go:127

runtime.main

        /usr/lib/golang/src/runtime/proc.go:203

runtime.goexit

        /usr/lib/golang/src/runtime/asm_amd64.s:1357

changing file ownership

hl-c507 ERROR 2019/12/13 13:20:14 changing file ownership: recursive chown /tmp/daos_server_svr1.log: accessing path /tmp/daos_server_svr1.log: lstat /tmp/daos_server_svr1.log: no such file or directory

 

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Join daos@daos.groups.io to automatically receive all group messages.