Will you try appending ${MY_DAOS_INSTALL_PATH}/lib64 to your LD_LIBRARY_PATH and re-run the “make” command?

export LD_LIBRARY_PATH=${MY_DAOS_INSTALL_PATH}/lib64:$LD_LIBRARY_PATH

~Dalton Bohning

Hi, 

And The build command of libmfu is:
git clone https://github.com/mchaarawi/mpifileutils -b pfind_integration "${MY_MFU_SOURCE_PATH}" &&
mkdir -p "${MY_MFU_BUILD_PATH}" &&
cd "${MY_MFU_BUILD_PATH}" &&
CFLAGS="-I${MY_DAOS_INSTALL_PATH}/include" \
LDFLAGS="-L${MY_DAOS_INSTALL_PATH}/lib64/ -luuid -ldaos -ldfs -ldaos_common -lgurt -lpthread" \
cmake "${MY_MFU_SOURCE_PATH}" \
-DENABLE_XATTRS=OFF \
-DWITH_DTCMP_PREFIX=${MY_MFU_INSTALL_PATH} \
-DWITH_LibCircle_PREFIX=${MY_MFU_INSTALL_PATH} \
-DCMAKE_INSTALL_PREFIX=${MY_MFU_INSTALL_PATH} &&
make -j8 install

Interesting. As you said, the libraries are in fact there. And I think cmake is finding the libraries, or else cmake would print something like “/usr/bin/ld: cannot find -ldfs” before getting to the “make” command. Could you please the compile command used to build libmfu?

$ cat ${MY_MFU_BUILD_PATH}/src/common/CMakeFiles/mfu.dir/link.txt

~Dalton Bohning

Hi, yes, "build" is where daos is installed.

Is this directory correct?

/home/daos_agent/git/daos/build/lib64/

Libraries are usually in the “install” directory, not “build”. Is “build” in fact where daos is installed?

Similarly, is ${MY_DAOS_INSTALL_PATH}/include the installed include directory?

~Dalton Bohning

Apologies, my previous command snippet was muddled. Correction:

$ grep -B 2 "CMAKE_SHARED_LINKER_FLAGS:" ${MY_MFU_BUILD_PATH}/CMakeCache.txt

//Flags used by the linker during the creation of shared libraries

// during all build types.

CMAKE_SHARED_LINKER_FLAGS:STRING=-L/home/dbohning/daos/install/lib64/ -luuid -ldaos -ldfs -ldaos_common -lgurt -lpthread

~Dalton Bohning

Hello,

It looks like most  of the libraries specified by “-luuid -ldaos -ldfs -ldaos_common -lgurt -lpthread” are not linked. Something to keep in mind is that “CFLAGS=… LDFLAGS=… cmake…” should be executed as a single command so CFLAGS and LDFLAGS is propagated to the cmake environment. You can check this by running:

$ grep -B 1 "CMAKE_EXE_LINKER_FLAGS:" ${MY_MFU_BUILD_PATH}/CMakeCache.txt

grep -B 2 "CMAKE_SHARED_LINKER_FLAGS:" CMakeCache.txt

//Flags used by the linker during the creation of shared libraries

// during all build types.

CMAKE_SHARED_LINKER_FLAGS:STRING=-L/home/dbohning/daos/install/lib64/ -luuid -ldaos -ldfs -ldaos_common -lgurt -lpthread

Where CMAKE_SHARED_LINKER_FLAGS should be set to the LDFLAGS from the build command.

~Dalton Bohning

Hi,
I built MFU according to https://daosio.atlassian.net/wiki/spaces/DC/pages/4874571083/IO-500+ISC21 but encountered some "undefined reference" errors.
[ 39%] Linking C executable dbcast
/usr/bin/ld: ../common/libmfu.so: undefined reference to `d_hash_rec_insert'
/usr/bin/ld: ../common/libmfu.so: undefined reference to `dfs_lookup'
/usr/bin/ld: ../common/libmfu.so: undefined reference to `d_hash_rec_unlinked'
/usr/bin/ld: ../common/libmfu.so: undefined reference to `dfs_stat'
/usr/bin/ld: ../common/libmfu.so: undefined reference to `dfs_readdir'
/usr/bin/ld: ../common/libmfu.so: undefined reference to `d_hash_table_create'
/usr/bin/ld: ../common/libmfu.so: undefined reference to `dfs_obj_anchor_split'
/usr/bin/ld: ../common/libmfu.so: undefined reference to `dfs_release'
/usr/bin/ld: ../common/libmfu.so: undefined reference to `d_hash_rec_find'
/usr/bin/ld: ../common/libmfu.so: undefined reference to `dfs_obj_anchor_set'

The build command is:
git clone https://github.com/mchaarawi/mpifileutils -b pfind_integration "${MY_MFU_SOURCE_PATH}" &&
mkdir -p "${MY_MFU_BUILD_PATH}" &&
cd "${MY_MFU_BUILD_PATH}" &&
CFLAGS="-I${MY_DAOS_INSTALL_PATH}/include" \
LDFLAGS="-L${MY_DAOS_INSTALL_PATH}/lib64/ -luuid -ldaos -ldfs -ldaos_common -lgurt -lpthread" \
cmake "${MY_MFU_SOURCE_PATH}" \
-DENABLE_XATTRS=OFF \
-DWITH_DTCMP_PREFIX=${MY_MFU_INSTALL_PATH} \
-DWITH_LibCircle_PREFIX=${MY_MFU_INSTALL_PATH} \
-DCMAKE_INSTALL_PREFIX=${MY_MFU_INSTALL_PATH} &&
make -j8 install

And there is libgurt.so in ${MY_DAOS_INSTALL_PATH}/lib64/.
$ nm -D ${MY_DAOS_INSTALL_PATH}/lib64/libgurt.so | grep d_hash_rec_insert
000000000000fb40 T d_hash_rec_insert
0000000000010100 T d_hash_rec_insert_anonym

But looks like libgurt.so is not linked by libmfu.so.
$ ldd src/common/libmfu.so
linux-vdso.so.1 (0x00007ffda13f3000)
libmpi.so.40 => /lib/x86_64-linux-gnu/libmpi.so.40 (0x00007fd75c830000)
libdtcmp.so => /home/daos_agent/install/lib/libdtcmp.so (0x00007fd75c814000)
libcircle.so.2 => /home/daos_agent/install/lib/libcircle.so.2 (0x00007fd75c806000)
libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007fd75c7f3000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd75c601000)
libopen-rte.so.40 => /lib/x86_64-linux-gnu/libopen-rte.so.40 (0x00007fd75c545000)
libopen-pal.so.40 => /lib/x86_64-linux-gnu/libopen-pal.so.40 (0x00007fd75c497000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fd75c348000)
libhwloc.so.15 => /lib/x86_64-linux-gnu/libhwloc.so.15 (0x00007fd75c2f7000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fd75c2d4000)
liblwgrp.so => /home/daos_agent/install/lib/liblwgrp.so (0x00007fd75c2c5000)
/lib64/ld-linux-x86-64.so.2 (0x00007fd75c9b2000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fd75c2a7000)
libevent-2.1.so.7 => /lib/x86_64-linux-gnu/libevent-2.1.so.7 (0x00007fd75c251000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fd75c24b000)
libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007fd75c246000)
libevent_pthreads-2.1.so.7 => /lib/x86_64-linux-gnu/libevent_pthreads-2.1.so.7 (0x00007fd75c241000)
libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007fd75c214000)
libltdl.so.7 => /lib/x86_64-linux-gnu/libltdl.so.7 (0x00007fd75c207000)"

Hello,

It looks like most  of the libraries specified by “-luuid -ldaos -ldfs -ldaos_common -lgurt -lpthread” are not linked. Something to keep in mind is that “CFLAGS=… LDFLAGS=… cmake…” should be executed as a single command so CFLAGS and LDFLAGS is propagated to the cmake environment. You can check this by running:

$ grep -B 1 "CMAKE_EXE_LINKER_FLAGS:" ${MY_MFU_BUILD_PATH}/CMakeCache.txt

grep -B 2 "CMAKE_SHARED_LINKER_FLAGS:" CMakeCache.txt

//Flags used by the linker during the creation of shared libraries

// during all build types.

CMAKE_SHARED_LINKER_FLAGS:STRING=-L/home/dbohning/daos/install/lib64/ -luuid -ldaos -ldfs -ldaos_common -lgurt -lpthread

Where CMAKE_SHARED_LINKER_FLAGS should be set to the LDFLAGS from the build command.

~Dalton Bohning

Hi,
I built MFU according to https://daosio.atlassian.net/wiki/spaces/DC/pages/4874571083/IO-500+ISC21 but encountered some "undefined reference" errors.
[ 39%] Linking C executable dbcast
/usr/bin/ld: ../common/libmfu.so: undefined reference to `d_hash_rec_insert'
/usr/bin/ld: ../common/libmfu.so: undefined reference to `dfs_lookup'
/usr/bin/ld: ../common/libmfu.so: undefined reference to `d_hash_rec_unlinked'
/usr/bin/ld: ../common/libmfu.so: undefined reference to `dfs_stat'
/usr/bin/ld: ../common/libmfu.so: undefined reference to `dfs_readdir'
/usr/bin/ld: ../common/libmfu.so: undefined reference to `d_hash_table_create'
/usr/bin/ld: ../common/libmfu.so: undefined reference to `dfs_obj_anchor_split'
/usr/bin/ld: ../common/libmfu.so: undefined reference to `dfs_release'
/usr/bin/ld: ../common/libmfu.so: undefined reference to `d_hash_rec_find'
/usr/bin/ld: ../common/libmfu.so: undefined reference to `dfs_obj_anchor_set'

The build command is:
git clone https://github.com/mchaarawi/mpifileutils -b pfind_integration "${MY_MFU_SOURCE_PATH}" &&
mkdir -p "${MY_MFU_BUILD_PATH}" &&
cd "${MY_MFU_BUILD_PATH}" &&
CFLAGS="-I${MY_DAOS_INSTALL_PATH}/include" \
LDFLAGS="-L${MY_DAOS_INSTALL_PATH}/lib64/ -luuid -ldaos -ldfs -ldaos_common -lgurt -lpthread" \
cmake "${MY_MFU_SOURCE_PATH}" \
-DENABLE_XATTRS=OFF \
-DWITH_DTCMP_PREFIX=${MY_MFU_INSTALL_PATH} \
-DWITH_LibCircle_PREFIX=${MY_MFU_INSTALL_PATH} \
-DCMAKE_INSTALL_PREFIX=${MY_MFU_INSTALL_PATH} &&
make -j8 install

And there is libgurt.so in ${MY_DAOS_INSTALL_PATH}/lib64/.
$ nm -D ${MY_DAOS_INSTALL_PATH}/lib64/libgurt.so | grep d_hash_rec_insert
000000000000fb40 T d_hash_rec_insert
0000000000010100 T d_hash_rec_insert_anonym

But looks like libgurt.so is not linked by libmfu.so.
$ ldd src/common/libmfu.so
linux-vdso.so.1 (0x00007ffda13f3000)
libmpi.so.40 => /lib/x86_64-linux-gnu/libmpi.so.40 (0x00007fd75c830000)
libdtcmp.so => /home/daos_agent/install/lib/libdtcmp.so (0x00007fd75c814000)
libcircle.so.2 => /home/daos_agent/install/lib/libcircle.so.2 (0x00007fd75c806000)
libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007fd75c7f3000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd75c601000)
libopen-rte.so.40 => /lib/x86_64-linux-gnu/libopen-rte.so.40 (0x00007fd75c545000)
libopen-pal.so.40 => /lib/x86_64-linux-gnu/libopen-pal.so.40 (0x00007fd75c497000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fd75c348000)
libhwloc.so.15 => /lib/x86_64-linux-gnu/libhwloc.so.15 (0x00007fd75c2f7000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fd75c2d4000)
liblwgrp.so => /home/daos_agent/install/lib/liblwgrp.so (0x00007fd75c2c5000)
/lib64/ld-linux-x86-64.so.2 (0x00007fd75c9b2000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fd75c2a7000)
libevent-2.1.so.7 => /lib/x86_64-linux-gnu/libevent-2.1.so.7 (0x00007fd75c251000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fd75c24b000)
libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007fd75c246000)
libevent_pthreads-2.1.so.7 => /lib/x86_64-linux-gnu/libevent_pthreads-2.1.so.7 (0x00007fd75c241000)
libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007fd75c214000)
libltdl.so.7 => /lib/x86_64-linux-gnu/libltdl.so.7 (0x00007fd75c207000)"

Hi,

The design is to ensure that all IO requests from different pools are processed in FIFO order, and space pressure from one pool doesn’t interfere request processing for other pools, but the implementation of policy_fifo_process() (as you pointed out) does have a defect which makes the latter requests (in FIFO queue) are impeded by the former request for a different pool with space pressure.

I’ll  cook a quick fix soon, thanks a lot for spotting this!

Thanks

-Niu  

Hello, Everyone!

Now all update/fetch requests of pools are in sched_info->si_fifo_list. If a pool request exceeds req_kick_limit, will it also block other pool requests?

thanks!

All,

We are pleased to announce that DAOS 2.0 release is now generally available. This release brings support for the following features:

• Erasure code¶
• Telemetry and monitoring¶
• Pool and container labels¶
• Improved usability and management capabilities¶
• Increased flexibility in object layout¶
• mpifileutils integration¶
• Improved stability in the network stack

There are a number of resources available for the release:

• RPM Repositories: https://packages.daos.io/v2.0/
• Release Notes: https://docs.daos.io/v2.0/release/release_notes_v2_0/
• Admin Guide: https://daos-stack.github.io/admin/hardware/
• User Guide: https://docs.daos.io/v2.0/user/workflow/
• Architecture Overview: https://docs.daos.io/v2.0/overview/architecture/
• Source Code: https://github.com/daos-stack/daos/releases/tag/v2.0.0

As always, feel free to use this mailing list for any issues you may find with the release, or our JIRA bug tracking system, available at https://daosio.atlassian.net/jira, or on our Slack channel, available at https://daos-stack.slack.com.

Regards,

Kelsey Prantis

Senior Software Engineering Manager

Extreme Storage Architecture and Development Division

Intel

Hi there,

Please find below the DAOS community newsletter for December 2021.

Past Events (November)

• SC’21 Tutorial (Nov 15)
  Practical Persistent Memory Programming: PMDK and DAOS
  Adrian Jackson (University of Edinburgh)
  Mohamad Chaarawi (Intel)
  Johann Lombardi (Intel)
  https://sc21.supercomputing.org/presentation/?id=tut134&sess=sess210
• SC’21 BoF (Nov 16)
  Object-stores for HPC - a Devonian Explosion or an Extinction Event?
  Philippe Deniel, CEA
  John Bent, Seagate
  Tiago Quinto, ECMWF
  Johann Lombardi, Intel
  https://sc21.supercomputing.org/presentation/?id=bof122&sess=sess368
• DAOS User Group (DUG’21)
  Slide decks and recordings available at http://dug.daos.io
• Intel SC21 Booth
  Dev led talk:
  DAOS Unleashes the Power in HPC Applications: QCT(Quanta) DevCloud Experience Sharing
  A Compact, Scalable, Efficient Data Collection Solution for Edge/IoT to Cloud by Zettar
  Fireside chat:
  Cambridge Service for Data Driven Discovery Enables Real-time Hospital Decision Support Systems to Improve Outcomes
  https://hpcevents.intel.com/devhub

Upcoming Events

• None this month. 

Release

• 2.0.0 RC2 was tagged (v2.0.0-rc2) earlier this week.
• The only difference with 2.0.0 RC1 is the log4j CVE update in the Hadoop connector (not included in the RPMs).
• Master is the development branch for the future 2.2 release.
• Major recent 2.0 changes:
• Integrate libfabric fix for fi_cancel()
• Bump log4j-core version to 2.16.0 in the Hadoop connector
• Fix a race between EC aggregation and reintegrate operation exposed by soak testing
• Many documentation updates that will be on http://docs.daos.io soon
• Add new recipes and scripts to install RPMs and run DAOS in docker containers
• Fix a memory registration issue with verbs & rebuild
• Quiet group version mismatch errors at pool creation
• Fix a SWIM regression introduced recently
• Fix a bulk leak on error path
• Refactor DAOS object ID to encode the number of groups/shards and improve performance
• Add interception for openat
• Lock memory only for VOS pools instead of using mlockall
• Major recent master changes:
• Add new logic for punch propagation to accelerate empty directory removal
• Restructure usage of hwloc in the control plane
• Add CXI support to netdetect
• Several VMD-related fixes
• Rework how we use GitHub actions
• Add new interface to create (and track) a ABT ULT (sched_create_ult())
• Several DTX enhancements
• What is coming:
• rc2 validation

R&D

• Major features under development:
• Checksum scrubbing
• LDMS plugin to export DAOS metrics (targeted for 2.2)
• API to collect libdaos metrics to be integrated with Darshan (targeted for 2.4)
• Multi-user dfuse (targeted for 2.2)
• More aggressive caching in dfuse for AI APPs (targeted for 2.2)
• TensorFlow-IO plugin for DFS
• https://github.com/daos-stack/tensorflow-io-daos/blob/devel/docs/daos_tf_docs.md
• Pathfinding:
• MariaDB DAOS engine with predicate pushdown to the DAOS storage nodes
• Prototyped DAOS MariaDB engine available here: https://github.com/daos-stack/mariadb
• PR for pipeline API (#6238)
• Work in progress to support pipeline API in the engine
• Leveraging the Intel Data Streaming Accelerator (DSA) to accelerate DAOS
• Prototype leveraging DSA for VOS aggregation delivered

News

• The documentation is migrating to use the “mike” plugin from MkDocs for multi-version support
• DAOS landing page now available on intel.com
• https://www.intel.com/content/www/us/en/high-performance-computing/daos.html
• Article on how to use PyDAOS from Eduardo and Eeheet
• https://www.intel.com/content/www/us/en/developer/articles/case-study/unlock-the-power-of-daos-in-python-with-pydaos.html
• Staffing:
• We welcome Cedric Koch-Hofer (France) and Lei Huang (TX, USA) to the Intel DAOS team. Cedric and Lei join us as full-stack engineers.
• We welcome Ray Dennis (NM, USA) to the Intel DAOS. Ray is joining our DevOps team.

Hi, Allen

The log showed it was stuck on creating blobstore. It looks like your device isn’t well supported by SPDK, could you collect some device information by SPDK ‘identify’ tool? Also, there is a ‘blob_cli’ tool from SPDK, could you try it to see if you can create blobstore on the device specified in your server yaml?

Thanks

-Niu

The failure seems to be happening when creating the blobstore but just to reduce variables, can you try with the following env_vars:

  env_vars:

  - CRT_TIMEOUT=300

  - CRT_CREDIT_EP_CTX=0

And simplify by running with

provider: ofi+sockets

Regards,

Tom

Hi Tom,
I noticed an error in the engine log.
DAOS[11610/11614] bio  DBUG src/bio/bio_xstream.c:662 load_blobstore() load blobstore failed -1025
Is it because of this? And what does "-1025" mean?

Some parameters of calling spdk_bs_load are as follows:
bs_dev->blocklen = 512
bs_dev->blockcnt = 7814037168
bs_opts.max_md_ops = 32
bs_opts.max_channel_ops = 4096 
bs_opts.cluster_sz = 1073741824

And the memory information of the server is as follows: