DAOS Community Update / Oct'21


Lombardi, Johann
 

Hi there,

 

Please find below the DAOS community newsletter for October 2021.

 

Past Events (September)

  • 5th EAGE workshop on HPC for Upstream (September 6-8)
    Leveraging DAOS file system for seismic data storage.

Merna Moawad, Amr Nasr, Omar Marzouk, Mohamed AlHarouni (Brightskies)

Johann Lombardi, Mohamad Chaarawi, Philippe Thierry (Intel)

Upcoming Events

 

Release

  • We continue to make good progress towards our 2.0 release and should tag a new test build (see v1.3.106-tb) soon.
  • Major recent changes:
    • Intercept __open_2 in libioil
    • Remove “highest aggregation epoch” field which was a leftover from old concurrency control model
    • Rework VOS epoch range discard
    • Make label for pools mandatory
    • Hyphens are now supported in both pool and container labels
    • Many more erasure code fixes
    • Quiet noisy SPDK/DPDK messages introduced with the upgrade to v21.07
    • Return most recent snapshot in container query
    • Integrate CART self-test with the DAOS agent (no need to dump attach info file any longer)
    • Fix a few memory leaks on error paths
    • Add “enable_hotplug” field to the yaml file to enable SSD hotplug support
    • Fix spurious pool handle eviction from the agent under memory pressure
    • Add several swim improvements to support “high-latency” network
    • Fix race with double RPC completions
    • Bump RDMA threshold for I/Os from 3.5KiB to 19KiB
    • Leverage helper xstreams (when available) for DTX batched commit
  • What is coming:
    • Fixes, fixes and more fixes.
    • Libfabric upgrade to grab some critical rxm/verbs fixes for DAOS.
    • Investigate an issue with NVMe unmap (aka trim) after repeated reformats.

 

R&D

  • Major features under development:
    • Checksum scrubbing
    • LDMS plugin to export DAOS metrics
    • API to collect libdaos metrics to be integrated with Darshan
    • Multi-user dfuse
    • More aggressive caching in dfuse for AI APPs
    • Design for catastrophic recovery / fsck
  • Pathfinding:
    • MariaDB DAOS engine with predicate pushdown to the DAOS storage nodes
    • Leveraging the Intel Data Streaming Accelerator (DSA) to accelerate DAOS

 

News

  • New look for https://docs.daos.io (dark theme)
  • Staffing:
    • We welcome Michael Hennecke to the Intel DAOS team.

 

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.