This specific issue is different appears to be at the dfuse/fuse kernel module level.  I note you’re using a newer fuse driver than I am (7.31 vs 7.23) and also a newer libfuse3 as evidenced by the “Unknown flags 0x3000000” error.  Purely from the output you’ve included the we are not explicitly disabling SPLICE_READ although we should not be using it, however the error from fuse seems to indicate that it’s use is attempted.

We do have Ubuntu 20.04 in our CI so I know we test on this, I’ll see if I can find any CI results or if not try Ubuntu 20.04 on a test machine here.

Ashley.

I'm experiencing (most likely) the same issue since a week or more on the master branch.

I reproduced it on today's master branch on Ubuntu 20.04. Verified a few days ago on Centos 7.

root@intel-S2600WFD:~/daos-step-by-step# dmg -i pool create --scm-size=30G --nvme-size=300G
Creating DAOS pool with 30 GB SCM and 300 GB NVMe storage (10.00 % ratio)
Pool-create command SUCCEEDED: UUID: fe6475b3-74dc-464f-b8b0-cac50778a9f9, Service replicas: 0

root@intel-S2600WFD:~/daos-step-by-step# daos container create  --svc=0 --path=/mnt/mycontainer --chunk_size=4K --type=POSIX --pool=fe6475b3-74dc-464f-b8b0-cac50778a9f9
fi   INFO src/gurt/fault_inject.c:481 d_fault_inject_init() No config file, fault injection is OFF.
daos INFO src/common/drpc.c:717 drpc_close() Closing dRPC socket fd=3
crt  INFO src/cart/crt_init.c:269 crt_init_opt() libcart version 4.8.0 initializing
crt  WARN src/cart/crt_init.c:161 data_init() FI_UNIVERSE_SIZE was not set; setting to 2048
client INFO src/utils/daos.c:153 cmd_args_print() DAOS system name: daos_server
client INFO src/utils/daos.c:154 cmd_args_print() pool UUID: fe6475b3-74dc-464f-b8b0-cac50778a9f9
client INFO src/utils/daos.c:155 cmd_args_print() cont UUID: 00000000-0000-0000-0000-000000000000
client INFO src/utils/daos.c:157 cmd_args_print() pool svc: parsed 1 ranks from input 0
client INFO src/utils/daos.c:161 cmd_args_print() attr: name=NULL, value=NULL
client INFO src/utils/daos.c:165 cmd_args_print() path=/mnt/mycontainer, type=POSIX, oclass=UNKNOWN, chunk_size=4096
client INFO src/utils/daos.c:168 cmd_args_print() snapshot: name=NULL, epoch=0, epoch range=NULL (0-0)
client INFO src/utils/daos.c:174 cmd_args_print() oid: 0.0
daos INFO src/common/drpc.c:717 drpc_close() Closing dRPC socket fd=18
daos INFO src/common/drpc.c:717 drpc_close() Closing dRPC socket fd=18
Successfully created container 0912dece-00e1-4e8f-8d7f-ac63603f52ac type POSIX

root@intel-S2600WFD:~/daos-step-by-step# dfuse --pool=fe6475b3-74dc-464f-b8b0-cac50778a9f9 --cont=0912dece-00e1-4e8f-8d7f-ac63603f52ac --mountpoint=/mnt/1 --svc=0 --foreground
fi   INFO src/gurt/fault_inject.c:481 d_fault_inject_init() No config file, fault injection is OFF.
daos INFO src/common/drpc.c:717 drpc_close() Closing dRPC socket fd=3
crt  INFO src/cart/crt_init.c:269 crt_init_opt() libcart version 4.8.0 initializing
crt  WARN src/cart/crt_init.c:161 data_init() FI_UNIVERSE_SIZE was not set; setting to 2048
duns INFO src/client/dfs/duns.c:393 duns_resolve_path() Path does not represent a DAOS link
dfuse INFO src/client/dfuse/dfuse_main.c:436 main(0x55639bbfb100) duns_resolve_path() returned 61 No data available
daos INFO src/common/drpc.c:717 drpc_close() Closing dRPC socket fd=18
daos INFO src/common/drpc.c:717 drpc_close() Closing dRPC socket fd=18
dfuse INFO src/client/dfuse/dfuse_fuseops.c:72 dfuse_fuse_init(0x55639bc6eb30) Fuse configuration
dfuse INFO src/client/dfuse/dfuse_fuseops.c:74 dfuse_fuse_init(0x55639bc6eb30) Proto 7 31
dfuse INFO src/client/dfuse/dfuse_fuseops.c:84 dfuse_fuse_init(0x55639bc6eb30) max read 0x400000
dfuse INFO src/client/dfuse/dfuse_fuseops.c:85 dfuse_fuse_init(0x55639bc6eb30) max write 0x400000
dfuse INFO src/client/dfuse/dfuse_fuseops.c:86 dfuse_fuse_init(0x55639bc6eb30) readahead 0x20000
dfuse INFO src/client/dfuse/dfuse_fuseops.c:88 dfuse_fuse_init(0x55639bc6eb30) Capability supported 0x31fffdb
dfuse INFO src/client/dfuse/dfuse_fuseops.c:39 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_ASYNC_READ enabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:40 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_POSIX_LOCKS enabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:41 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_ATOMIC_O_TRUNC enabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:42 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_EXPORT_SUPPORT enabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:43 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_DONT_MASK enabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:44 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_SPLICE_WRITE enabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:45 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_SPLICE_MOVE enabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:46 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_SPLICE_READ enabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:47 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_FLOCK_LOCKS enabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:48 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_IOCTL_DIR enabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:49 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_AUTO_INVAL_DATA enabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:50 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_READDIRPLUS enabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:51 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_READDIRPLUS_AUTO enabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:52 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_ASYNC_DIO enabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:53 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_WRITEBACK_CACHE enabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:54 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_NO_OPEN_SUPPORT enabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:55 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_PARALLEL_DIROPS enabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:56 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_POSIX_ACL enabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:57 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_HANDLE_KILLPRIV enabled
dfuse ERR  src/client/dfuse/dfuse_fuseops.c:60 dfuse_show_flags(0x55639bc6eb30) Unknown flags 0x3000000
dfuse INFO src/client/dfuse/dfuse_fuseops.c:92 dfuse_fuse_init(0x55639bc6eb30) Capability requested 0x149a09
dfuse INFO src/client/dfuse/dfuse_fuseops.c:39 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_ASYNC_READ enabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:40 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_POSIX_LOCKS disabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:41 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_ATOMIC_O_TRUNC enabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:42 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_EXPORT_SUPPORT disabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:43 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_DONT_MASK disabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:44 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_SPLICE_WRITE disabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:45 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_SPLICE_MOVE disabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:46 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_SPLICE_READ enabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:47 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_FLOCK_LOCKS disabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:48 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_IOCTL_DIR enabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:49 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_AUTO_INVAL_DATA enabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:50 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_READDIRPLUS disabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:51 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_READDIRPLUS_AUTO disabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:52 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_ASYNC_DIO enabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:53 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_WRITEBACK_CACHE disabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:54 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_NO_OPEN_SUPPORT disabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:55 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_PARALLEL_DIROPS enabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:56 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_POSIX_ACL disabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:57 dfuse_show_flags(0x55639bc6eb30) Flag FUSE_CAP_HANDLE_KILLPRIV enabled
dfuse INFO src/client/dfuse/dfuse_fuseops.c:99 dfuse_fuse_init(0x55639bc6eb30) max_background 16
dfuse INFO src/client/dfuse/dfuse_fuseops.c:100 dfuse_fuse_init(0x55639bc6eb30) congestion_threshold 8
fuse: splice from device: Invalid argument
dfuse ERR  src/client/dfuse/dfuse_main.c:191 ll_loop_fn(0x55639bbfb100) Fuse loop exited with return code: -22
dfuse ERR  src/client/dfuse/dfuse_core.c:384 dfuse_start(0x55639bc6eb30) Unable to register FUSE fs
dfuse ERR  src/client/dfuse/dfuse_core.c:399 dfuse_start(0x55639bc6eb30) Failed to start dfuse, rc: -1003
dfuse ERR  src/client/dfuse/dfuse_main.c:519 main(0x55639bc8c2f0) DFP left at the end
dfuse ERR  src/client/dfuse/dfuse_main.c:522 main(0x55639bc8c3f0) DFS left at the end
dfuse INFO src/client/dfuse/dfuse_main.c:561 main() Exiting with status -1003

When --pool and --cont ar not specified and --mountpoint pointing to the path specified when creating the container, you get the same output as above.
Calling dfuse --mountpoint=/mnt/mycontainer --svc=0 --foreground gives the same output as as dfuse --pool=fe6475b3-74dc-464f-b8b0-cac50778a9f9 --cont=0912dece-00e1-4e8f-8d7f-ac63603f52ac --mountpoint=/mnt/1 --svc=0 --foreground

Gert,