Hi,

This is the same issue as Gert hit last week, specifically that you’re providing the pool/container UUIDs twice to dfuse, one via the path and once on the command line.  The fix in this case would not be to provide –pool or –container options on the command line.

It would however made sense for dfuse to support this usage where the uuids match, so I’ve filed DAOS-5778 to allow this.

Ashley.

Hello!

I am trying to use dfuse to create a POSIX-enabled mount.

My container is created thusly:
daos cont create --pool=[pool_id] --svc=0 --type=POSIX --path=/tmp/mycontainer

I then try to mount like this:
dfuse --mountpoint=/tmp/mycontainer --svc=0 --pool=[pool_id] --container=[container_id] --foreground


And this is the error I receive:
09/25-14:46:17.18 master-node DAOS[849/849] fi   INFO src/gurt/fault_inject.c:481 d_fault_inject_init() No config file, fault injection is OFF.
09/25-14:46:17.18 master-node DAOS[849/849] daos INFO src/common/drpc.c:717 drpc_close() Closing dRPC socket fd=3
09/25-14:46:17.18 master-node DAOS[849/849] crt  INFO src/cart/crt_init.c:269 crt_init_opt() libcart version 4.8.0 initializing
09/25-14:46:17.18 master-node DAOS[849/849] crt  WARN src/cart/crt_init.c:161 data_init() FI_UNIVERSE_SIZE was not set; setting to 2048
09/25-14:46:17.27 master-node DAOS[849/849] dfuse INFO src/client/dfuse/dfuse_main.c:436 main(0x55c5e3a8c980) duns_resolve_path() returned 0 Success
UNS configured on mount point but pool provided
09/25-14:46:17.27 master-node DAOS[849/849] dfuse ERR  src/client/dfuse/dfuse_main.c:519 main(0x55c5e3b31410) DFP left at the end
09/25-14:46:17.27 master-node DAOS[849/849] dfuse ERR  src/client/dfuse/dfuse_main.c:522 main(0x55c5e3b31510) DFS left at the end
09/25-14:46:17.27 master-node DAOS[849/849] dfuse INFO src/client/dfuse/dfuse_main.c:561 main() Exiting with status -1003

I would much appreciate any guidance in solving this.

Thank you,

Peter

This case is different, the mount point here has not been used to create a container so dfuse is attempting to use the pool and container on the command line.  The error reported is a DAOS error rather than a dfuse one, but I suspect this is a server error, either because the wrong svc value was specified or because on of the servers isn’t running or contactable.

Ashley,

Thanks for the response, I tried again with:
dfuse  --mountpoint=/home/daos/container/ --svc=0  --pool=[pool_id] --container=[container_id] --foreground

This led to the following error:
10/05-15:25:36.12 master-node DAOS[442/442] fi   INFO src/gurt/fault_inject.c:481 d_fault_inject_init() No config file, fault injection is OFF.
10/05-15:25:36.12 master-node DAOS[442/442] daos INFO src/common/drpc.c:717 drpc_close() Closing dRPC socket fd=3
10/05-15:25:36.12 master-node DAOS[442/442] crt  INFO src/cart/crt_init.c:269 crt_init_opt() libcart version 4.8.0 initializing
10/05-15:25:36.12 master-node DAOS[442/442] crt  WARN src/cart/crt_init.c:161 data_init() FI_UNIVERSE_SIZE was not set; setting to 2048
10/05-15:25:36.14 master-node DAOS[442/442] duns INFO src/client/dfs/duns.c:301 duns_resolve_path() Path does not represent a DAOS link
10/05-15:25:36.14 master-node DAOS[442/442] dfuse INFO src/client/dfuse/dfuse_main.c:436 main(0x561ee77ad980) duns_resolve_path() returned 61 No data available
10/05-15:25:36.14 master-node DAOS[442/442] daos INFO src/common/drpc.c:717 drpc_close() Closing dRPC socket fd=24
10/05-15:25:36.15 master-node DAOS[442/442] daos INFO src/common/drpc.c:717 drpc_close() Closing dRPC socket fd=24
10/05-15:25:36.15 master-node DAOS[442/442] common ERR  src/common/rsvc.c:141 rsvc_client_process_error() removed rank 0 from replica list due to DER_NOTREPLICA(-2020): 'Not a service replica'
10/05-15:25:36.15 master-node DAOS[442/442] common WARN src/common/rsvc.c:102 rsvc_client_choose() replica list empty
10/05-15:25:36.15 master-node DAOS[442/442] pool ERR  src/pool/cli.c:471 dc_pool_connect() 30d0e9d9: cannot find pool service: DER_NOTREPLICA(-2020): 'Not a service replica'
Failed to connect to pool (-1005)
10/05-15:25:36.15 master-node DAOS[442/442] dfuse ERR  src/client/dfuse/dfuse_main.c:519 main(0x561ee784ffe0) DFP left at the end
10/05-15:25:36.15 master-node DAOS[442/442] dfuse ERR  src/client/dfuse/dfuse_main.c:522 main(0x561ee78500e0) DFS left at the end
10/05-15:25:36.15 master-node DAOS[442/442] dfuse INFO src/client/dfuse/dfuse_main.c:561 main() Exiting with status 0

This specific issue is different appears to be at the dfuse/fuse kernel module level.  I note you’re using a newer fuse driver than I am (7.31 vs 7.23) and also a newer libfuse3 as evidenced by the “Unknown flags 0x3000000” error.  Purely from the output you’ve included the we are not explicitly disabling SPLICE_READ although we should not be using it, however the error from fuse seems to indicate that it’s use is attempted.

We do have Ubuntu 20.04 in our CI so I know we test on this, I’ll see if I can find any CI results or if not try Ubuntu 20.04 on a test machine here.

Ashley.

I'm experiencing (most likely) the same issue since a week or more on the master branch.

I reproduced it on today's master branch on Ubuntu 20.04. Verified a few days ago on Centos 7.

root@intel-S2600WFD:~/daos-step-by-step# dmg -i pool create --scm-size=30G --nvme-size=300G
Creating DAOS pool with 30 GB SCM and 300 GB NVMe storage (10.00 % ratio)
Pool-create command SUCCEEDED: UUID: fe6475b3-74dc-464f-b8b0-cac50778a9f9, Service replicas: 0

root@intel-S2600WFD:~/daos-step-by-step# daos container create  --svc=0 --path=/mnt/mycontainer --chunk_size=4K --type=POSIX --pool=fe6475b3-74dc-464f-b8b0-cac50778a9f9
fi   INFO src/gurt/fault_inject.c:481 d_fault_inject_init() No config file, fault injection is OFF.
daos INFO src/common/drpc.c:717 drpc_close() Closing dRPC socket fd=3
crt  INFO src/cart/crt_init.c:269 crt_init_opt() libcart version 4.8.0 initializing
crt  WARN src/cart/crt_init.c:161 data_init() FI_UNIVERSE_SIZE was not set; setting to 2048
client INFO src/utils/daos.c:153 cmd_args_print() DAOS system name: daos_server
client INFO src/utils/daos.c:154 cmd_args_print() pool UUID: fe6475b3-74dc-464f-b8b0-cac50778a9f9
client INFO src/utils/daos.c:155 cmd_args_print() cont UUID: 00000000-0000-0000-0000-000000000000
client INFO src/utils/daos.c:157 cmd_args_print() pool svc: parsed 1 ranks from input 0
client INFO src/utils/daos.c:161 cmd_args_print() attr: name=NULL, value=NULL
client INFO src/utils/daos.c:165 cmd_args_print() path=/mnt/mycontainer, type=POSIX, oclass=UNKNOWN, chunk_size=4096
client INFO src/utils/daos.c:168 cmd_args_print() snapshot: name=NULL, epoch=0, epoch range=NULL (0-0)
client INFO src/utils/daos.c:174 cmd_args_print() oid: 0.0
daos INFO src/common/drpc.c:717 drpc_close() Closing dRPC socket fd=18
daos INFO src/common/drpc.c:717 drpc_close() Closing dRPC socket fd=18
Successfully created container 0912dece-00e1-4e8f-8d7f-ac63603f52ac type POSIX

root@intel-S2600WFD:~/daos-step-by-step# dfuse --pool=fe6475b3-74dc-464f-b8b0-cac50778a9f9 --cont=0912dece-00e1-4e8f-8d7f-ac63603f52ac --mountpoint=/mnt/1 --svc=0 --foreground
fi   INFO src/gurt/fault_inject.c:481 d_fault_inject_init() No config file, fault injection is OFF.
daos INFO src/common/drpc.c:717 drpc_close() Closing dRPC socket fd=3
crt  INFO src/cart/crt_init.c:269 crt_init_opt() libcart version 4.8.0 initializing
crt  WARN src/cart/crt_init.c:161 data_init() FI_UNIVERSE_SIZE was not set; setting to 2048
duns INFO src/client/dfs/duns.c:393 duns_resolve_path() Path does not represent a DAOS link
dfuse INFO src/client/dfuse/dfuse_main.c:436 main(0x55639bbfb100) duns_resolve_path() returned 61 No data available
daos INFO src/common/drpc.c:717 drpc_close() Closing dRPC socket fd=18
daos INFO src/common/drpc.c:717 drpc_close() Closing dRPC socket fd=18
dfuse INFO src/client/dfuse/dfuse_fuseops.c:72 dfuse_fuse_init(0x55639bc6eb30) Fuse configuration
dfuse INFO src/client/dfuse/dfuse_fuseops.c:74 dfuse_fuse_init(0x55639bc6eb30) Proto 7 31
dfuse INFO src/client/dfuse/dfuse_fuseops.c:84 dfuse_fuse_init(0x55639bc6eb30) max read 0x400000
dfuse INFO src/client/dfuse/dfuse_fuseops.c:85 dfuse_fuse_init(0x55639bc6eb30) max write 0x400000
dfuse INFO src/client/dfuse/dfuse_fuseops.c:86 dfuse_fuse_init(0x55639bc6eb30) readahead 0x20000
dfuse INFO src/client/dfuse/dfuse_fuseops.c:88 dfuse_fuse_init(0x55639bc6eb30) Capability supported 0x31fffdb
dfuse INFO src/client/dfuse/dfuse_fuseops.c:39 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_ASYNC_READ enabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:40 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_POSIX_LOCKS enabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:41 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_ATOMIC_O_TRUNC enabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:42 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_EXPORT_SUPPORT enabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:43 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_DONT_MASK enabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:44 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_SPLICE_WRITE enabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:45 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_SPLICE_MOVE enabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:46 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_SPLICE_READ enabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:47 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_FLOCK_LOCKS enabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:48 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_IOCTL_DIR enabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:49 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_AUTO_INVAL_DATA enabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:50 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_READDIRPLUS enabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:51 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_READDIRPLUS_AUTO enabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:52 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_ASYNC_DIO enabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:53 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_WRITEBACK_CACHE enabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:54 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_NO_OPEN_SUPPORT enabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:55 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_PARALLEL_DIROPS enabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:56 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_POSIX_ACL enabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:57 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_HANDLE_KILLPRIV enabled
dfuse ERR  src/client/dfuse/dfuse_fuseops.c:60 dfuse_show_flags(0x55639bc6eb30) Unknown flags 0x3000000
dfuse INFO src/client/dfuse/dfuse_fuseops.c:92 dfuse_fuse_init(0x55639bc6eb30) Capability requested 0x149a09
dfuse INFO src/client/dfuse/dfuse_fuseops.c:39 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_ASYNC_READ enabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:40 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_POSIX_LOCKS disabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:41 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_ATOMIC_O_TRUNC enabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:42 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_EXPORT_SUPPORT disabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:43 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_DONT_MASK disabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:44 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_SPLICE_WRITE disabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:45 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_SPLICE_MOVE disabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:46 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_SPLICE_READ enabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:47 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_FLOCK_LOCKS disabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:48 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_IOCTL_DIR enabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:49 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_AUTO_INVAL_DATA enabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:50 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_READDIRPLUS disabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:51 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_READDIRPLUS_AUTO disabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:52 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_ASYNC_DIO enabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:53 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_WRITEBACK_CACHE disabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:54 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_NO_OPEN_SUPPORT disabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:55 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_PARALLEL_DIROPS enabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:56 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_POSIX_ACL disabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:57 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_HANDLE_KILLPRIV enabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:99 dfuse_fuse_init(0x55639bc6eb30) max_background 16
dfuse INFO src/client/dfuse/dfuse_fuseops.c:100 dfuse_fuse_init(0x55639bc6eb30) congestion_threshold 8
fuse: splice from device: Invalid argument
dfuse ERR  src/client/dfuse/dfuse_main.c:191 ll_loop_fn(0x55639bbfb100) Fuse loop exited with return code: -22
dfuse ERR  src/client/dfuse/dfuse_core.c:384 dfuse_start(0x55639bc6eb30) Unable to register FUSE fs
dfuse ERR  src/client/dfuse/dfuse_core.c:399 dfuse_start(0x55639bc6eb30) Failed to start dfuse, rc: -1003
dfuse ERR  src/client/dfuse/dfuse_main.c:519 main(0x55639bc8c2f0) DFP left at the end
dfuse ERR  src/client/dfuse/dfuse_main.c:522 main(0x55639bc8c3f0) DFS left at the end
dfuse INFO src/client/dfuse/dfuse_main.c:561 main() Exiting with status -1003

When --pool and --cont ar not specified and --mountpoint pointing to the path specified when creating the container, you get the same output as above.
Calling dfuse --mountpoint=/mnt/mycontainer --svc=0 --foreground gives the same output as as dfuse --pool=fe6475b3-74dc-464f-b8b0-cac50778a9f9 --cont=0912dece-00e1-4e8f-8d7f-ac63603f52ac --mountpoint=/mnt/1 --svc=0 --foreground

Gert,

Peter,

There have been no changes in status since your email on the 5^th, although there has been a new dependency added to DAOS recently, l4z-devel is now a required package.

Your previous problems were the result of inconsistent command line options, incorrect svc values or non-existent pool rather than anything we could address within dfuse that I’m aware of.

DAOS-5778 has been filed to allow the setting of pool/container uuids in multiple places, so if you specify a path which has them embedded in and the same uuids on the command line then this will in future be allowed but I’ve not started work on this ticket yet.

Ashley.

Hello again,

Has there been any update on the status of dfuse in recent builds? Or is there any foolproof way to mount a DAOS as a POSIX namespace?

Thank you for the help,

Peter

Peter,

Thank you for the update.  I had spoken to Gert privately and we’d looked into the Ubuntu issue a bit but he’d also switched to a different OS version for testing.  As I said before we do have ubuntu 20.04 here in our CI (it looks like it’s got a newer version of the fuse kernel driver) so it’s on my TODO list to investigate this.  I’ll let you know when we have an update.

Ashley.

I'm sorry, I should have been more clear, I had also attempted the same commands as Gert and received the same errors.

I had been using Ubuntu 20.04/centos7 docker images with Ubuntu 20.04 as my host OS. 

Downgrading my host OS to Ubuntu 18.04 and using Ubuntu 20.04 docker images solved my issues.

I would be grateful if you could try this PR on your Ubuntu 20.04 system.  I was able to reproduce the problem here and this fixes it for me, it appears to be additional checks in newer kernels, combined with an existing but previously harmless bug in libfuse itself.

https://github.com/daos-stack/daos/pull/3678

Ashley.

I'm sorry, I should have been more clear, I had also attempted the same commands as Gert and received the same errors.

I had been using Ubuntu 20.04/centos7 docker images with Ubuntu 20.04 as my host OS. 

Downgrading my host OS to Ubuntu 18.04 and using Ubuntu 20.04 docker images solved my issues.