Could you double check if creating container would cause NVMe free space dropping? If it’s true, please open a ticket for further investigation. I can’t think of why container creation could consume NVMe space for now.

Thanks

-Niu

Hi, Chuck

The reserved space is per pool, it’s not relevant with container creation, so I think the space change you observed after container creation isn’t caused by space reservation.

FYI, we’ve just changed the space reservation a bit, NVMe reservation has been removed from current master and 2.2, only SCM reservation is kept.

As for space query, current client space query reports only total space and free space, I think that’s a common practice for most systems. I think It could be improved to report detailed usage like how much space is used for reservation in the future, thanks for the input.

Thanks

-Niu

Hi there,

Please find below the DAOS community newsletter for June 2022.

Past Events

• Salishan Conference On High Speed Computing (April 26^th)

Accelerating Data-driven Workflows with DAOS
https://salishan.ahsc-nm.org/program4.html

Johann Lombardi (Intel)

• ECP BoF (May 11^th at 11am Eastern time)

DAOS Next Generation Storage

https://www.exascaleproject.org/event/ecp-community-bof-days-2022/ (registration is open)

Kevin Harms (ANL)

Mohamad Chaarawi (Intel)

Johann Lombardi (Intel)

• Intel Vision (May 11^th)

Advanced Storage and Memory Hierarchy in AI and HPC with DAOS Storage

https://reg.oneventseries.intel.com/flow/intel/vision2022/portal-live/page/session-catalog?tab.day=20220511&search=AIML009

Andrey Kudryavtsev (Intel)

• SODACODE 2022 Keynote (May 25^th)

Advanced Storage and Memory Hierarchy in AI and HPC with DAOS Storage

Andrey Kudryavtsev (Intel)

• SODACODE 2022 Breakout Session (May 26^th)

Accelerating AI with DAOS Storage

Johann Lombardi (Intel)

• ISC’22 BoF (May 30^th at 4pm CEST)

Accelerating HPC and AI with DAOS Storage

https://app.swapcard.com/widget/event/isc-high-performance-2022/planning/UGxhbm5pbmdfODYxMTYx

Kevin Harms (ANL)

Adrian Jackson (EPCC)

Michael Hennecke (Intel)

Mohamad Chaarawi (Intel)

Johann Lombardi (Intel)

• ISC’22 IXPUG (June 2^nd at 9am CEST)

DAOS Features for Next Generation Platforms

https://www.ixpug.org/events/isc22-ixpug-workshop

Mohamad Chaarawi (Intel)

• ISC’22 Intel Booth (May 30^th to June 1^st)

DAOS demonstration, Fireside chats, …

Upcoming Events

• ESSA’22: 3^rd Workshop on Extreme-Scale Storage and Analysis (June 3^rd)

DAOS: Nextgen Storage Stack for HPC and AI

https://sites.google.com/view/essa-2022/

Johann Lombardi (Intel)

• EMOSS’22: Emerging Open Storage Systems and Solutions for Data Intensive Computing (July 1^st)

One big happy family: sharing the S3 layer between Ceph, CORTX, and DAOS

https://iosea-project.eu/event/emoss-22-workshop/
Zuhair AlSader (Seagate)
Andriy Tkachuk (Seagate)

Release

• Current stable release is 2.0.2. See https://docs.daos.io/v2.0/ and https://packages.daos.io/v2.0/ for more information.
• Branches:
• release/2.0 is the release branch for the stable 2.0 release. Latest bug fix release is 2.0.2 and 2.0.3 rc1 has been tagged.
• release/2.2 is the development branch for the future 2.2 release. Latest test build is 2.1.102.
• Master is the development branch for the future 2.4 release. Latest test build is 2.3.100.
• Major recent changes on release/2.0 (bugfix release):
• Upgrade to libfabric v1.15.1
• Fill a sem leak on failure path of pool creation
• Several patches to improve error reporting when downgrading from 2.2 to 2.0 after dmg pool upgrade was completed.
• Fix a limitation in the agent code and NIC selection on servers with a lot of NUMA nodes
• Fix reference leak on container destroy path
• Major recent changes on release/2.2 (future 2.2 release):
• All patches listed in the 2.0 section above.
• Evict pool handles on agent shutdown
• Several test fixes to run the CI over UCX
• Add support for virtual network interface
• Disable FUSE metadata caching when using IL
• Split large I/Os into chunk size IOs (8MB by default) to avoid exceeding the SPDK bdev single IO size limit.
• Fix build on aarch64
• Reduce duration of NVMe storage format in the control plane
• Fix huge pages NUMA locality
• Fix available storage with heterogeneous NVMe
• Disable unified mode in UCX
• Major recent changes on master (future 2.4 release):
• All patches listed in the 2.2 section above.
• Add dmg pool query-targets
• Add chgrp support to dfuse
• Check thread ID in IL destructor
• Use pip to install deps on ubuntu
• Remove some compatibility code with DAOS 1.2
• Allow fanout control RPC to report progress
• Add rebuild generation for each rebuild retry
• Add new DFS APIs to retrieve pool and container handles
• What is coming:
• 2.0.3 validation and decision on whether we need another release candidate
• 2.2.0 code freeze
• 2.4.0 feature freeze

R&D

• Major features under development:
• Checksum scrubber
• Improvement to test coverage in progress.
• Should be merged to master soon.
• Branch: feature/csum-scrubbing
• Target release: 2.4
• Multi-user dfuse
• Allow dfuse mountpoint to be accessible by multiple users
• Should be merged to master soon.
• PR: https://github.com/daos-stack/daos/pull/3956
• Target release: 2.4
• More aggressive caching in dfuse for AI APPs
• PR: https://github.com/daos-stack/daos/pull/6776
• Performance results shared at IXPUG conference.
• Should be merged to master soon.
• Target release: 2.4
• I/O steaming function interception
• Add the ability to intercept fopen/fclose/fread/fwrite and other streaming functions in the interception library.
• Should be merged to master soon.
• PR: https://github.com/daos-stack/daos/pull/6939
• Target release: 2.4
• Catastrophic recovery
• Aka distributed fsck or checker
• First passes implemented.
• Testing and debugging in progress.
• Branch: feature/cat_recovery
• Target release: 2.6
• Multi-homed network support
• Aka multi-provider support
• This feature aims at supporting multiple network provider in the engine
• Changes to the engine to support multiple provider implemented.
• Branch: feature/multiprovider
• Target release: 2.6
• Client-side metrics
• API to collect libdaos metrics to be eventually integrated with Darshan
• PR: https://github.com/daos-stack/daos/pull/6497
• Target release: 2.6
• Performance domain
• Extend placement algorithm to be aware of fabric topology
• Branch: feature/perf_dom
• Target release: 2.8 
• LDMS plugin to export DAOS metrics
• Branch: https://github.com/daos-stack/ovis/tree/mjmac/DAOS-sampler/ldms/src/contrib/sampler/daos
• Packaging in progress on https://github.com/daos-stack/ovis-daos-sampler
• To be submitted/integrated upstream
• Pathfinding:
• DAOS Pipeline API for active storage
• Branch: feature/pipeline_api
• Prototyping of server-side find using the pipeline API completed. Initial results shared at IXPUG conference.
• Changes to MariaDB DAOS engine with predicate pushdown to the DAOS storage
• Leveraging the Intel Data Streaming Accelerator (DSA) to accelerate DAOS
• Prototype leveraging DSA for VOS aggregation delivered
• Initial results shared at IXPUG conference.
• DAOS/DFS integration with NFS Ganesha
• Under exploration
• Discussions about NFS handle and readdir+ support
• OPX provider support in collaboration with Cornelis Networks
• DAOS and Mercury patches to add OPX support under review.
• GPU data path optimizations
• I/O Middleware / Framework Support
• TensorFlow-IO plugin for DAOS
• https://github.com/daos-stack/tensorflow-io-daos/blob/devel/docs/daos_tf_docs.md
• https://docs.daos.io/v2.2/user/tensorflow/
• Plugin demonstrated at ISC’22 with CosmoFlow.
• PR submitted upstream by Omar/Brightskies: https://github.com/tensorflow/io/pull/1603
• S3 support via a DAOS backend to Rados Gateway (RGW)
• Support for versioning and multi-part upload added
• Work underway to evaluate performance and move DAOS-specific code to a new library called libds3.
• PR submitted upstream by Zuhair/Seagate: https://github.com/ceph/ceph/pull/45888
• Block interface over DAOS using SPDK DAOS bdev
• PR submitted upstream by DenisB/Croit: https://review.spdk.io/gerrit/c/spdk/spdk/+/12260
• New Jenkins instance is being set up by Croit to integrate with the SPDK CI

News

• Jerome Soumagne joins the Intel DAOS team to work on the network stack.

Hi, Chuck

The “used space” (total – free) is kind of OP (over-provisioning), you know that DAOS server has to reserve some space on both SCM and NVMe to ensure punch, container/object destroy, GC and aggregation not fail for ENOSPACE.

The size of this sys reservation is roughly: SCM: 5% of SCM total space; NVMe: 2% of NVMe total space; and the minimum reservation size is 2GB per pool target (for both SCM and NVMe). The reservation will be disabled if the pool size is tiny  (when each pool target SCM and NVMe size is less than 5GB, we regard it as tiny pool, which is usually used for testing), so the operations I mentioned above could fail on the tiny pool when it’s running short of space.

There is an open ticket for reducing the OP, but it’s not on our schedule yet.

Thanks

-Niu

Hello Chunk

That is strange.  According the output of dmg pool query, 83 objects were deleted after extend, so some space  should be reclaimed.

“dmg pool query kiddie

………

   Rebuild done, 83 objs, 0 recs

“

before extend:

s_total(30000021504,470000000000) s_free(29994849224,434968621056)

after extend:

s_total(60000043008,940000000000) s_free(59994841512,869937672192)

Hmm It seems SCM is ok, only NVME space are doubled after extend.  If you do not see NVME free space  get back after a few mins, probably need create a ticket.

Thanks

WangDi

Hello Chunk

That is strange.  According the output of dmg pool query, 83 objects were deleted after extend, so some space  should be reclaimed.

“dmg pool query kiddie

………

   Rebuild done, 83 objs, 0 recs

“

before extend:

s_total(30000021504,470000000000) s_free(29994849224,434968621056)

after extend:

s_total(60000043008,940000000000) s_free(59994841512,869937672192)

Hmm It seems SCM is ok, only NVME space are doubled after extend.  If you do not see NVME free space  get back after a few mins, probably need create a ticket.

Thanks

WangDi

Hi Kevin,

I see you on slack, so I assume that you eventually got the invite.

Cheers,

Johann

So, you may want to use mmap() without MAP_SHARED when using DFuse "--disable-caching" option ??

What do you think ?

Ok, I can reproduce the behaviour you have reported.

Having a look to the related source code, this may come from the fact if caching is disabled, DFuse automatically/silently switches to direct-io mode, but then we may trigger the current Fuse Kernel module limitation that cause shared mmap() mappings to be  unsupported (ENODEV).

Hi Bruno,

Thank you very much for checking! I'm using DAOS v2.0.1.One thing I forgot to mention previously is that I mounted the POSIX container with dfuse "--disable-caching" and got an error from mmap. Yes, without "--disable-caching" option, mmap succeeds with no errors in my environment as well. If you try to do dfuse mount with --disable-caching option, in your environment, does mmap succeed? I wanted to know if this is an expected behavior and to evaluate the effectiveness of caching in dfuse.

Thank you very much for your help,
---
Shohei

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 5 208 026.16 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Ok, I can reproduce the behaviour you have reported.

Having a look to the related source code, this may come from the fact if caching is disabled, DFuse automatically/silently switches to direct-io mode, but then we may trigger the current Fuse Kernel module limitation that cause shared mmap() mappings to be  unsupported (ENODEV).

Hi Bruno,

Thank you very much for checking! I'm using DAOS v2.0.1.One thing I forgot to mention previously is that I mounted the POSIX container with dfuse "--disable-caching" and got an error from mmap. Yes, without "--disable-caching" option, mmap succeeds with no errors in my environment as well. If you try to do dfuse mount with --disable-caching option, in your environment, does mmap succeed? I wanted to know if this is an expected behavior and to evaluate the effectiveness of caching in dfuse.

Thank you very much for your help,
---
Shohei

Well, I just tried your test-case on my test-bed and you program just works as expected, no error from mmap() ! :

=============================================================================================

[bfaccini@wolf-5 ~]$ ps -ef | grep daos

bfaccini  1632  5549  0 May26 pts/0    00:00:30 daos_agent -i -o /home/bfaccini/daos_agent.yml start

bfaccini 36536  5549  0 09:35 pts/0    00:00:00 daos_server --config=/home/bfaccini/daos_server.yml start

bfaccini 36603 36536 99 09:36 pts/0    00:21:54 /home/bfaccini/daos/install/bin/daos_engine -t 8 -x 2 -g daos_server -d /var/run/daos_server -T 1 -I 0 -r 0 -H 0 -s /mnt/daos/

bfaccini 37141  5549  0 09:54 pts/0    00:00:00 grep --color=auto daos

[bfaccini@wolf-5 ~]$ ps -ef | grep dfuse

bfaccini 36714     1  0 09:39 pts/0    00:00:00 dfuse --mountpoint /mnt/dfuse/ --pool 757c8377-7bf9-4cb1-a9d0-05a86901c9dd --cont a76ad1a2-c0f1-4b40-9d8e-6a042204198f

bfaccini 37143  5549  0 09:54 pts/0    00:00:00 grep --color=auto dfuse

[bfaccini@wolf-5 ~]$ dmg pool list -i -v

Label UUID                                 SvcReps SCM Size SCM Used SCM Imbalance NVME Size NVME Used NVME Imbalance Disabled 

----- ----                                 ------- -------- -------- ------------- --------- --------- -------------- -------- 

foo   757c8377-7bf9-4cb1-a9d0-05a86901c9dd 0       4.0 GB   21 MB    0%            0 B       0 B       0%             0/8      

[bfaccini@wolf-5 ~]$ daos pool list-cont foo

UUID                                 Label                   

----                                 -----                   

a76ad1a2-c0f1-4b40-9d8e-6a042204198f container_label_not_set 

[bfaccini@wolf-5 ~]$ 

[bfaccini@wolf-5 ~]$ cat test_mmap.c

#include <stdio.h>

#include <fcntl.h>

#include <sys/mman.h>

#include <string.h>

#define FILE_SIZE 4096

int main(int argc, char *argv[]) {

    int fd;

    char *map;

    size_t map_size;

    fd = open(argv[1], O_CREAT | O_RDWR, 0665);

    if(fd < 0) {

        printf("Error : can't open file\n");

        return -1;

    }

    map_size=FILE_SIZE;

    map = (char*)mmap(NULL, map_size, PROT_WRITE, MAP_SHARED, fd, 0);

    if(map == MAP_FAILED) {

        printf("Error : mmap failed\n");

        return -1;

    }

}

[bfaccini@wolf-5 ~]$ gcc -o test_mmap test_mmap.c

[bfaccini@wolf-5 ~]$ 

[bfaccini@wolf-5 ~]$ ls -la /mnt/dfuse

total 19075

-rwxrwxr-x 1 bfaccini bfaccini 19531936 May 28 15:18 daos_admin

[bfaccini@wolf-5 ~]$ df /mnt/dfuse/daos_admin

Filesystem     1K-blocks  Used Available Use% Mounted on

dfuse            3906272 20526   3885747   1% /mnt/dfuse

[bfaccini@wolf-5 ~]$ 

[bfaccini@wolf-5 ~]$ ./test_mmap /mnt/dfuse/daos_admin 

[bfaccini@wolf-5 ~]$ echo $?

0

[bfaccini@wolf-5 ~]$ 

[bfaccini@wolf-5 ~]$ strace -o /tmp/test_mmap.strace ./test_mmap /mnt/dfuse/daos_admin 

[bfaccini@wolf-5 ~]$ 

[bfaccini@wolf-5 ~]$ tail -10 /tmp/test_mmap.strace

access("/etc/sysconfig/strcasecmp-nonascii", F_OK) = -1 ENOENT (No such file or directory)

access("/etc/sysconfig/strcasecmp-nonascii", F_OK) = -1 ENOENT (No such file or directory)

mprotect(0x7ff27cb54000, 16384, PROT_READ) = 0

mprotect(0x600000, 4096, PROT_READ)     = 0

mprotect(0x7ff27cd80000, 4096, PROT_READ) = 0

munmap(0x7ff27cd67000, 97934)           = 0

open("/mnt/dfuse/daos_admin", O_RDWR|O_CREAT, 0665) = 3

mmap(NULL, 4096, PROT_WRITE, MAP_SHARED, 3, 0) = 0x7ff27cd7e000

exit_group(2094522368)                  = ?

+++ exited with 0 +++

[bfaccini@wolf-5 ~]$ 

=============================================================================================

So, can you double-check your setup ?? Also, which DAOS version are you running ?

Have a good day!

Bruno.

Hi, 

I have a very quick question about the below selection in Posix compliance. From a single DAOS client node, is mmap with MAP_SHARED supported from the client node against a file on a DAOS POSIX container, if it is mounted with dfuse to the client?  
https://docs.daos.io/v2.0/user/filesystem/
I've tried, but I get an error return from mmap. Below is the portion of my test code and the page size is 4KB.  Thanks in advance! 

====

Hello Chuck

Pool extend might migrate the data to the new pool target, then the original data will be delete asynchronous, so those space might be reclaimed a few mins later if the system is not busy.

You probably should do your daos_pool_query() a bit later. Btw: what are those objects class in your pool?

Thanks

Wangdi