mmap support with a dfuse-mounted posix conntainer
|
shmatsuu@...
Hi,
I have a very quick question about the below selection in Posix compliance. From a single DAOS client node, is mmap with MAP_SHARED supported from the client node against a file on a DAOS POSIX container, if it is mounted with dfuse to the client? https://docs.daos.io/v2.0/user/filesystem/ I've tried, but I get an error return from mmap. Below is the portion of my test code and the page size is 4KB. Thanks in advance! ==== #include <stdio.h>
#include <fcntl.h>
#include <sys/mman.h>
#include <string.h>
#define FILE_SIZE 4096
int main(int argc, char *argv[]) {
int fd;
char *map;
size_t map_size;
fd = open(argv[1], O_CREAT | O_RDWR, 0665);
if(fd < 0) {
printf("Error : can't open file\n");
return -1;
}
map_size=FILE_SIZE;
map = (char*)mmap(NULL, map_size, PROT_WRITE, MAP_SHARED, fd, 0);
if(map == MAP_FAILED) {
printf("Error : mmap failed\n");
return -1;
}
=== |
|
|
|
Faccini, Bruno
Well, I just tried your test-case on my test-bed and you program just works as expected, no error from mmap() ! : ============================================================================================= [bfaccini@wolf-5 ~]$ ps -ef | grep daos bfaccini 1632 5549 0 May26 pts/0 00:00:30 daos_agent -i -o /home/bfaccini/daos_agent.yml start bfaccini 36536 5549 0 09:35 pts/0 00:00:00 daos_server --config=/home/bfaccini/daos_server.yml start bfaccini 36603 36536 99 09:36 pts/0 00:21:54 /home/bfaccini/daos/install/bin/daos_engine -t 8 -x 2 -g daos_server -d /var/run/daos_server -T 1 -I 0 -r 0 -H 0 -s /mnt/daos/ bfaccini 37141 5549 0 09:54 pts/0 00:00:00 grep --color=auto daos [bfaccini@wolf-5 ~]$ ps -ef | grep dfuse bfaccini 36714 1 0 09:39 pts/0 00:00:00 dfuse --mountpoint /mnt/dfuse/ --pool 757c8377-7bf9-4cb1-a9d0-05a86901c9dd --cont a76ad1a2-c0f1-4b40-9d8e-6a042204198f bfaccini 37143 5549 0 09:54 pts/0 00:00:00 grep --color=auto dfuse [bfaccini@wolf-5 ~]$ dmg pool list -i -v Label UUID SvcReps SCM Size SCM Used SCM Imbalance NVME Size NVME Used NVME Imbalance Disabled ----- ---- ------- -------- -------- ------------- --------- --------- -------------- -------- foo 757c8377-7bf9-4cb1-a9d0-05a86901c9dd 0 4.0 GB 21 MB 0% 0 B 0 B 0% 0/8
[bfaccini@wolf-5 ~]$ daos pool list-cont foo UUID Label ---- ----- a76ad1a2-c0f1-4b40-9d8e-6a042204198f container_label_not_set [bfaccini@wolf-5 ~]$ [bfaccini@wolf-5 ~]$ cat test_mmap.c #include <stdio.h> #include <fcntl.h> #include <sys/mman.h> #include <string.h>
#define FILE_SIZE 4096
int main(int argc, char *argv[]) { int fd; char *map; size_t map_size;
fd = open(argv[1], O_CREAT | O_RDWR, 0665); if(fd < 0) { printf("Error : can't open file\n"); return -1; }
map_size=FILE_SIZE;
map = (char*)mmap(NULL, map_size, PROT_WRITE, MAP_SHARED, fd, 0);
if(map == MAP_FAILED) { printf("Error : mmap failed\n"); return -1; } }
[bfaccini@wolf-5 ~]$ gcc -o test_mmap test_mmap.c [bfaccini@wolf-5 ~]$ [bfaccini@wolf-5 ~]$ ls -la /mnt/dfuse total 19075 -rwxrwxr-x 1 bfaccini bfaccini 19531936 May 28 15:18 daos_admin [bfaccini@wolf-5 ~]$ df /mnt/dfuse/daos_admin Filesystem 1K-blocks Used Available Use% Mounted on dfuse 3906272 20526 3885747 1% /mnt/dfuse [bfaccini@wolf-5 ~]$ [bfaccini@wolf-5 ~]$ ./test_mmap /mnt/dfuse/daos_admin [bfaccini@wolf-5 ~]$ echo $? 0 [bfaccini@wolf-5 ~]$ [bfaccini@wolf-5 ~]$ strace -o /tmp/test_mmap.strace ./test_mmap /mnt/dfuse/daos_admin [bfaccini@wolf-5 ~]$ [bfaccini@wolf-5 ~]$ tail -10 /tmp/test_mmap.strace access("/etc/sysconfig/strcasecmp-nonascii", F_OK) = -1 ENOENT (No such file or directory) access("/etc/sysconfig/strcasecmp-nonascii", F_OK) = -1 ENOENT (No such file or directory) mprotect(0x7ff27cb54000, 16384, PROT_READ) = 0 mprotect(0x600000, 4096, PROT_READ) = 0 mprotect(0x7ff27cd80000, 4096, PROT_READ) = 0 munmap(0x7ff27cd67000, 97934) = 0 open("/mnt/dfuse/daos_admin", O_RDWR|O_CREAT, 0665) = 3 mmap(NULL, 4096, PROT_WRITE, MAP_SHARED, 3, 0) = 0x7ff27cd7e000 exit_group(2094522368) = ? +++ exited with 0 +++ [bfaccini@wolf-5 ~]$ =============================================================================================
So, can you double-check your setup ?? Also, which DAOS version are you running ? Have a good day! Bruno.
From: <daos@daos.groups.io> on behalf of "shmatsuu@..." <shmatsuu@...>
Hi, #include <stdio.h> #include <fcntl.h> #include <sys/mman.h> #include <string.h>
#define FILE_SIZE 4096
int main(int argc, char *argv[]) { int fd; char *map; size_t map_size;
fd = open(argv[1], O_CREAT | O_RDWR, 0665); if(fd < 0) { printf("Error : can't open file\n"); return -1; }
map_size=FILE_SIZE;
map = (char*)mmap(NULL, map_size, PROT_WRITE, MAP_SHARED, fd, 0);
if(map == MAP_FAILED) { printf("Error : mmap failed\n"); return -1; } === --------------------------------------------------------------------- This e-mail and any attachments may contain confidential material for |
|
|
|
shmatsuu@...
Hi Bruno,
Thank you very much for checking! I'm using DAOS v2.0.1.One thing I forgot to mention previously is that I mounted the POSIX container with dfuse "--disable-caching" and got an error from mmap. Yes, without "--disable-caching" option, mmap succeeds with no errors in my environment as well. If you try to do dfuse mount with --disable-caching option, in your environment, does mmap succeed? I wanted to know if this is an expected behavior and to evaluate the effectiveness of caching in dfuse. Thank you very much for your help, --- Shohei |
|
|
|
Faccini, Bruno
Ok, I can reproduce the behaviour you have reported.
Having a look to the related source code, this may come from the fact if caching is disabled, DFuse automatically/silently switches to direct-io mode, but then we may trigger the current Fuse Kernel module limitation that cause shared mmap() mappings to be unsupported (ENODEV).
From: <daos@daos.groups.io> on behalf of "shmatsuu@..." <shmatsuu@...>
Hi Bruno, --------------------------------------------------------------------- This e-mail and any attachments may contain confidential material for |
|
|
|
Faccini, Bruno
So, you may want to use mmap() without MAP_SHARED when using DFuse "--disable-caching" option ?? What do you think ?
From: <daos@daos.groups.io> on behalf of Bruno Faccini <bruno.faccini@...>
Ok, I can reproduce the behaviour you have reported.
Having a look to the related source code, this may come from the fact if caching is disabled, DFuse automatically/silently switches to direct-io mode, but then we may trigger the current Fuse Kernel module limitation that cause shared mmap() mappings to be unsupported (ENODEV).
From: <daos@daos.groups.io> on behalf of "shmatsuu@..." <shmatsuu@...>
Hi Bruno, --------------------------------------------------------------------- This e-mail and any attachments may contain confidential material for --------------------------------------------------------------------- This e-mail and any attachments may contain confidential material for |
|
|