Date   

Re: Post DAOS 2.0 installation and comments and questions

Omer
 

Might take a few days till I'm back working on that setup, but will update. As far as I remember, no output what so ever was printed though.


Re: Post DAOS 2.0 installation and comments and questions

Pittman, Ashley M
 

Omer,

 

I’d be interested in seeing the stdout/stderr and logging from dfuse where this is happening.  As you say the -f option should make it run in the foreground so if you’re not passing this then it should background itself so you can get back a prompt.  Even when it does go into the background it’ll wait until the fuse mount is registered with the kernel before detaching the terminal so that it can report any error to the user so there is code in there to delay and then exit after forking so there is potential for an issue there although I’ve not known one before and we do have tests which use both methods.

 

Ashley.

 

From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Omer via groups.io <omer.caspi@...>
Date: Monday, 17 January 2022 at 14:32
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: Re: [daos] Post DAOS 2.0 installation and comments and questions

Hi,

Thanks for your reply.

1. I saw :). Cool.
2. I used to build DAOS and install it, so I had a script that did that for me, but yes, with RPM based solution this shouldn't be a concern. I fixed my unit file to do what I suggested to overcome this.
3. I tried " dfuse --pool=pool0 --container=cont0 -m /mnt/dfuse" and after a while " dfuse -f --pool=pool0 --container=cont0 -m /mnt/dfuse". I'm guessing the 2nd version is expected to not return as the '-f' suggests, but I didn't get a command prompt back either way.

Omer


Re: Post DAOS 2.0 installation and comments and questions

Omer
 

Hi,

Thanks for your reply.

1. I saw :). Cool.
2. I used to build DAOS and install it, so I had a script that did that for me, but yes, with RPM based solution this shouldn't be a concern. I fixed my unit file to do what I suggested to overcome this.
3. I tried " dfuse --pool=pool0 --container=cont0 -m /mnt/dfuse" and after a while " dfuse -f --pool=pool0 --container=cont0 -m /mnt/dfuse". I'm guessing the 2nd version is expected to not return as the '-f' suggests, but I didn't get a command prompt back either way.

Omer


Re: Post DAOS 2.0 installation and comments and questions

Lombardi, Johann
 

Hi there,

 

Thanks for reaching out.

  1. The online doc has been fixed and now specifies a different URL for CentOS7 and 8.
  2. /var/run/daos_agent should be automatically created after enabling/starting the daos_agent systemd service (i.e. “systemctl enable/start daos_agent”).
    We have some instructions in the documentation on how to start the agent manually (w/o systemd), see https://docs.daos.io/v2.0/admin/predeployment_check/#runtime-directory-setup

That being said, this should not be required if you use RPMs/systemd.

  1. Could you please advise what option(s) you pass to the dfuse command line? Can you set “export D_LOG_MASK=debug” and then run dfuse with the -f option and see what it logs?

 

Cheers,

Johann

 

 

From: <daos@daos.groups.io> on behalf of "Omer via groups.io" <omer.caspi@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Wednesday 12 January 2022 at 12:18
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: [daos] Post DAOS 2.0 installation and comments and questions

 

Some post DAOS 2.0 installation and comments and questions:

1. We use a mix of CentOS 7 and 8 in our setup. However, the DAOS online documentation at "https://docs.daos.io/v2.0/QSG/setup_centos/" states the following URLas the repository to use when installing DAOS:

"https://packages.daos.io/v2.0/CentOS7/packages/x86_64/daos_packages.repo".

Of course, this lead to some installation problems as the is a specific CentOS 7 URL. Fixing the repository URL solved this problem, but only after some confusion about what going on. This probably needs some documentation clarification.

2. The DAOS agent creates a socket in /var/run/daos_agent. This isn't a persistent directory and the service doesn't create it, and it needs to be created on every boot for the agent to work properly.
a. This isn't mentioned in the documentation.
b. A systemd unit addition like "ExecStartPre=mkdir -p /var/run/daos_agent" is probably a good idea

3. Not sure why, but mounting a POSIX container using dfuse doesn't return for me. The process doesn't return. During that time, the mount is working (or so it seems), as a `mount` command does list a dfuse mount in the relevant directory, but the dfuse command didn't return to the CLI prompt.

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.


Re: High latency in metada write

Lombardi, Johann
 

Hi there,

 

Could you please tell us more about your HW configuration and the test case (e.g. I/O size, read or write, object class, …)? We definitely see lower latency in our internal testing.

 

Cheers,

Johann

 

From: <daos@daos.groups.io> on behalf of "shadow_vector@..." <shadow_vector@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Tuesday 11 January 2022 at 12:26
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: [daos] High latency in metada write

 

Hello everyone:
  Recently, I did some testing with daos array interface, found that after finishing nvme bio, it took me a long time for writing metadata and resource release ; The total latency of single channel IO is about 80us, and  vos_update_end taks about 30 us(about 17us in dkey akey update via SCM); I thought that IO via SCM should be very fast, but through the test, it's slower than the nvme IO. Is evtree operation takes too long or are there other reasons?

Best regards!

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.


Post DAOS 2.0 installation and comments and questions

Omer
 

Some post DAOS 2.0 installation and comments and questions:

1. We use a mix of CentOS 7 and 8 in our setup. However, the DAOS online documentation at "https://docs.daos.io/v2.0/QSG/setup_centos/" states the following URLas the repository to use when installing DAOS:

"https://packages.daos.io/v2.0/CentOS7/packages/x86_64/daos_packages.repo".

Of course, this lead to some installation problems as the is a specific CentOS 7 URL. Fixing the repository URL solved this problem, but only after some confusion about what going on. This probably needs some documentation clarification.

2. The DAOS agent creates a socket in /var/run/daos_agent. This isn't a persistent directory and the service doesn't create it, and it needs to be created on every boot for the agent to work properly.
a. This isn't mentioned in the documentation.
b. A systemd unit addition like "ExecStartPre=mkdir -p /var/run/daos_agent" is probably a good idea

3. Not sure why, but mounting a POSIX container using dfuse doesn't return for me. The process doesn't return. During that time, the mount is working (or so it seems), as a `mount` command does list a dfuse mount in the relevant directory, but the dfuse command didn't return to the CLI prompt.


High latency in metada write

shadow_vector@...
 

Hello everyone:
  Recently, I did some testing with daos array interface, found that after finishing nvme bio, it took me a long time for writing metadata and resource release ; The total latency of single channel IO is about 80us, and  vos_update_end taks about 30 us(about 17us in dkey akey update via SCM); I thought that IO via SCM should be very fast, but through the test, it's slower than the nvme IO. Is evtree operation takes too long or are there other reasons?

Best regards!


Re: Questions about ULT Schedule

Niu, Yawei
 

Hi,

 

In your example, the ULT will be tracked in the wait list from “ABT_future” on waiting, once it’s waked up, it’ll be pushed back to a runnable ULT FIFO list (per ABT pool, maintained by Argobots internally), and being executed once server scheduler picked it again.

 

Thanks

-Niu

 

From: daos@daos.groups.io <daos@daos.groups.io> on behalf of 段世博 <duanshibo.d@...>
Date: Sunday, January 2, 2022 at 9:00 PM
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: Re: [daos] Questions about ULT Schedule

I have one more question A ULT is suspended during execution, where it is when it was awakened. For example, dtx_leader_exec_ops_ult hangs while waiting for ABT_future,

When the future meets the conditions, the ULT is awakened. Where is the awakened ULT stored (fifo_list or other list), and when will the ULT continue to execute?


Re: Unable to build libmfu properly

Bohning, Dalton
 

I’m not sure why the libraries aren’t being linked/loaded – maybe someone more knowledgeable on the DAOS team has an idea. The make command does include the libraries, and the libraries are available, so I wouldn’t think there should be an issue.

 

~Dalton Bohning

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of netsurfed
Sent: Thursday, December 30, 2021 5:32 PM
To: daos@daos.groups.io
Subject: Re: [daos] Unable to build libmfu properly

 

Yes, the same errors.

daos_agent@sw2:~/git/mpifileutils/build$ export LD_LIBRARY_PATH=${MY_DAOS_INSTALL_PATH}/lib64:$LD_LIBRARY_PATH

daos_agent@sw2:~/git/mpifileutils/build$ echo $LD_LIBRARY_PATH

/home/daos_agent/git/daos/build/lib64:

daos_agent@sw2:~/git/mpifileutils/build$ make

[ 32%] Built target mfu_o

[ 34%] Built target mfu-static

[ 36%] Built target mfu

[ 37%] Linking C executable dbcast

/usr/bin/ld: ../common/libmfu.so: undefined reference to `d_hash_rec_insert'

/usr/bin/ld: ../common/libmfu.so: undefined reference to `dfs_lookup'

/usr/bin/ld: ../common/libmfu.so: undefined reference to `d_hash_rec_unlinked'

/usr/bin/ld: ../common/libmfu.so: undefined reference to `dfs_stat'

/usr/bin/ld: ../common/libmfu.so: undefined reference to `dfs_readdir'

/usr/bin/ld: ../common/libmfu.so: undefined reference to `d_hash_table_create'

/usr/bin/ld: ../common/libmfu.so: undefined reference to `dfs_obj_anchor_split'

/usr/bin/ld: ../common/libmfu.so: undefined reference to `dfs_release'

/usr/bin/ld: ../common/libmfu.so: undefined reference to `d_hash_rec_find'

/usr/bin/ld: ../common/libmfu.so: undefined reference to `dfs_obj_anchor_set'

collect2: error: ld returned 1 exit status

make[2]: *** [src/dbcast/CMakeFiles/dbcast.dir/build.make:90: src/dbcast/dbcast] Error 1

make[1]: *** [CMakeFiles/Makefile2:539: src/dbcast/CMakeFiles/dbcast.dir/all] Error 2

make: *** [Makefile:130: all] Error 2

 


Re: Unable to build libmfu properly

netsurfed
 

Yes, the same errors.

daos_agent@sw2:~/git/mpifileutils/build$ export LD_LIBRARY_PATH=${MY_DAOS_INSTALL_PATH}/lib64:$LD_LIBRARY_PATH
daos_agent@sw2:~/git/mpifileutils/build$ echo $LD_LIBRARY_PATH
/home/daos_agent/git/daos/build/lib64:
daos_agent@sw2:~/git/mpifileutils/build$ make
[ 32%] Built target mfu_o
[ 34%] Built target mfu-static
[ 36%] Built target mfu
[ 37%] Linking C executable dbcast
/usr/bin/ld: ../common/libmfu.so: undefined reference to `d_hash_rec_insert'
/usr/bin/ld: ../common/libmfu.so: undefined reference to `dfs_lookup'
/usr/bin/ld: ../common/libmfu.so: undefined reference to `d_hash_rec_unlinked'
/usr/bin/ld: ../common/libmfu.so: undefined reference to `dfs_stat'
/usr/bin/ld: ../common/libmfu.so: undefined reference to `dfs_readdir'
/usr/bin/ld: ../common/libmfu.so: undefined reference to `d_hash_table_create'
/usr/bin/ld: ../common/libmfu.so: undefined reference to `dfs_obj_anchor_split'
/usr/bin/ld: ../common/libmfu.so: undefined reference to `dfs_release'
/usr/bin/ld: ../common/libmfu.so: undefined reference to `d_hash_rec_find'
/usr/bin/ld: ../common/libmfu.so: undefined reference to `dfs_obj_anchor_set'
collect2: error: ld returned 1 exit status
make[2]: *** [src/dbcast/CMakeFiles/dbcast.dir/build.make:90: src/dbcast/dbcast] Error 1
make[1]: *** [CMakeFiles/Makefile2:539: src/dbcast/CMakeFiles/dbcast.dir/all] Error 2
make: *** [Makefile:130: all] Error 2
 


Re: Unable to build libmfu properly

Bohning, Dalton
 

Will you try appending ${MY_DAOS_INSTALL_PATH}/lib64 to your LD_LIBRARY_PATH and re-run the “make” command?

export LD_LIBRARY_PATH=${MY_DAOS_INSTALL_PATH}/lib64:$LD_LIBRARY_PATH

 

~Dalton Bohning

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of netsurfed
Sent: Wednesday, December 29, 2021 6:04 PM
To: daos@daos.groups.io
Subject: Re: [daos] Unable to build libmfu properly

 

Hi, 

$ cat ${MY_MFU_BUILD_PATH}/src/common/CMakeFiles/mfu.dir/link.txt

/usr/bin/cc -fPIC -I/home/daos_agent/git/daos/build/include -L/home/daos_agent/git/daos/build/lib64/ -luuid -ldaos -ldfs -ldaos_common -lgurt -lpthread -shared -Wl,-soname,libmfu.so -o libmfu.so CMakeFiles/mfu_o.dir/mfu_bz2.c.o CMakeFiles/mfu_o.dir/mfu_bz2_static.c.o CMakeFiles/mfu_o.dir/mfu_compress_bz2_libcircle.c.o CMakeFiles/mfu_o.dir/mfu_decompress_bz2_libcircle.c.o CMakeFiles/mfu_o.dir/mfu_flist.c.o CMakeFiles/mfu_o.dir/mfu_flist_chunk.c.o CMakeFiles/mfu_o.dir/mfu_flist_copy.c.o CMakeFiles/mfu_o.dir/mfu_flist_io.c.o CMakeFiles/mfu_o.dir/mfu_flist_chmod.c.o CMakeFiles/mfu_o.dir/mfu_flist_create.c.o CMakeFiles/mfu_o.dir/mfu_flist_remove.c.o CMakeFiles/mfu_o.dir/mfu_flist_sort.c.o CMakeFiles/mfu_o.dir/mfu_flist_usrgrp.c.o CMakeFiles/mfu_o.dir/mfu_flist_walk.c.o CMakeFiles/mfu_o.dir/mfu_io.c.o CMakeFiles/mfu_o.dir/mfu_param_path.c.o CMakeFiles/mfu_o.dir/mfu_path.c.o CMakeFiles/mfu_o.dir/mfu_pred.c.o CMakeFiles/mfu_o.dir/mfu_util.c.o CMakeFiles/mfu_o.dir/strmap.c.o  -Wl,-rpath,/usr/lib/x86_64-linux-gnu/openmpi/lib:/home/daos_agent/install/lib: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so /home/daos_agent/install/lib/libdtcmp.so -larchive /home/daos_agent/install/lib/libcircle.so -lbz2

 

And The build command of libmfu is:
git clone 
https://github.com/mchaarawi/mpifileutils -b pfind_integration "${MY_MFU_SOURCE_PATH}" &&
mkdir -p "${MY_MFU_BUILD_PATH}" &&
cd "${MY_MFU_BUILD_PATH}" &&
CFLAGS="-I${MY_DAOS_INSTALL_PATH}/include" \
LDFLAGS="-L${MY_DAOS_INSTALL_PATH}/lib64/ -luuid -ldaos -ldfs -ldaos_common -lgurt -lpthread" \
cmake "${MY_MFU_SOURCE_PATH}" \
-DENABLE_XATTRS=OFF \
-DWITH_DTCMP_PREFIX=${MY_MFU_INSTALL_PATH} \
-DWITH_LibCircle_PREFIX=${MY_MFU_INSTALL_PATH} \
-DCMAKE_INSTALL_PREFIX=${MY_MFU_INSTALL_PATH} &&
make -j8 install


Re: Unable to build libmfu properly

netsurfed
 

Hi, 
$ cat ${MY_MFU_BUILD_PATH}/src/common/CMakeFiles/mfu.dir/link.txt
/usr/bin/cc -fPIC -I/home/daos_agent/git/daos/build/include -L/home/daos_agent/git/daos/build/lib64/ -luuid -ldaos -ldfs -ldaos_common -lgurt -lpthread -shared -Wl,-soname,libmfu.so -o libmfu.so CMakeFiles/mfu_o.dir/mfu_bz2.c.o CMakeFiles/mfu_o.dir/mfu_bz2_static.c.o CMakeFiles/mfu_o.dir/mfu_compress_bz2_libcircle.c.o CMakeFiles/mfu_o.dir/mfu_decompress_bz2_libcircle.c.o CMakeFiles/mfu_o.dir/mfu_flist.c.o CMakeFiles/mfu_o.dir/mfu_flist_chunk.c.o CMakeFiles/mfu_o.dir/mfu_flist_copy.c.o CMakeFiles/mfu_o.dir/mfu_flist_io.c.o CMakeFiles/mfu_o.dir/mfu_flist_chmod.c.o CMakeFiles/mfu_o.dir/mfu_flist_create.c.o CMakeFiles/mfu_o.dir/mfu_flist_remove.c.o CMakeFiles/mfu_o.dir/mfu_flist_sort.c.o CMakeFiles/mfu_o.dir/mfu_flist_usrgrp.c.o CMakeFiles/mfu_o.dir/mfu_flist_walk.c.o CMakeFiles/mfu_o.dir/mfu_io.c.o CMakeFiles/mfu_o.dir/mfu_param_path.c.o CMakeFiles/mfu_o.dir/mfu_path.c.o CMakeFiles/mfu_o.dir/mfu_pred.c.o CMakeFiles/mfu_o.dir/mfu_util.c.o CMakeFiles/mfu_o.dir/strmap.c.o  -Wl,-rpath,/usr/lib/x86_64-linux-gnu/openmpi/lib:/home/daos_agent/install/lib: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so /home/daos_agent/install/lib/libdtcmp.so -larchive /home/daos_agent/install/lib/libcircle.so -lbz2
 
And The build command of libmfu is:
git clone https://github.com/mchaarawi/mpifileutils -b pfind_integration "${MY_MFU_SOURCE_PATH}" &&
mkdir -p "${MY_MFU_BUILD_PATH}" &&
cd "${MY_MFU_BUILD_PATH}" &&
CFLAGS="-I${MY_DAOS_INSTALL_PATH}/include" \
LDFLAGS="-L${MY_DAOS_INSTALL_PATH}/lib64/ -luuid -ldaos -ldfs -ldaos_common -lgurt -lpthread" \
cmake "${MY_MFU_SOURCE_PATH}" \
-DENABLE_XATTRS=OFF \
-DWITH_DTCMP_PREFIX=${MY_MFU_INSTALL_PATH} \
-DWITH_LibCircle_PREFIX=${MY_MFU_INSTALL_PATH} \
-DCMAKE_INSTALL_PREFIX=${MY_MFU_INSTALL_PATH} &&
make -j8 install


Re: Unable to build libmfu properly

Bohning, Dalton
 

Interesting. As you said, the libraries are in fact there. And I think cmake is finding the libraries, or else cmake would print something like “/usr/bin/ld: cannot find -ldfs” before getting to the “make” command. Could you please the compile command used to build libmfu?

$ cat ${MY_MFU_BUILD_PATH}/src/common/CMakeFiles/mfu.dir/link.txt

 

~Dalton Bohning

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of netsurfed
Sent: Tuesday, December 28, 2021 9:40 PM
To: daos@daos.groups.io
Subject: Re: [daos] Unable to build libmfu properly

 

Hi, yes, "build" is where daos is installed.

$ ls -l /home/daos_agent/git/daos/build/lib64/

total 22080

drwxrwxr-x 3 daos_agent daos_agent    4096 Dec 13 08:06 daos

drwxrwxr-x 2 daos_agent daos_agent    4096 Dec 13 10:18 daos_srv

lrwxrwxrwx 1 daos_agent daos_agent      16 Dec 13 10:18 libcart.so -> libcart.so.4.9.0

lrwxrwxrwx 1 daos_agent daos_agent      16 Dec 13 10:18 libcart.so.4 -> libcart.so.4.9.0

-rwxrwxr-x 1 daos_agent daos_agent 4948104 Dec 13 10:10 libcart.so.4.9.0

-rwxrwxr-x 1 daos_agent daos_agent  710680 Dec 13 10:10 libdaos_cmd_hdlrs.so

-rwxrwxr-x 1 daos_agent daos_agent 2955992 Dec 13 10:10 libdaos_common_pmem.so

-rwxrwxr-x 1 daos_agent daos_agent 2905240 Dec 13 10:10 libdaos_common.so

lrwxrwxrwx 1 daos_agent daos_agent      16 Dec 13 10:18 libdaos.so -> libdaos.so.2.0.0

lrwxrwxrwx 1 daos_agent daos_agent      16 Dec 13 10:18 libdaos.so.2 -> libdaos.so.2.0.0

-rwxrwxr-x 1 daos_agent daos_agent 6814552 Dec 13 10:04 libdaos.so.2.0.0

-rwxrwxr-x 1 daos_agent daos_agent  274776 Dec 13 10:10 libdaos_tests.so

-rwxrwxr-x 1 daos_agent daos_agent   27160 Dec 13 10:18 libdfs_internal.so

-rwxrwxr-x 1 daos_agent daos_agent  816728 Dec 13 10:10 libdfs.so

-rw-rw-r-- 1 daos_agent daos_agent  206304 Dec 13 08:07 libdfuse.a

-rwxrwxr-x 1 daos_agent daos_agent  137440 Dec 13 10:18 libdfuse.so

-rwxrwxr-x 1 daos_agent daos_agent  116120 Dec 13 10:10 libdts.so

-rwxrwxr-x 1 daos_agent daos_agent  158712 Dec 13 10:10 libduns.so

lrwxrwxrwx 1 daos_agent daos_agent      16 Dec 13 10:18 libgurt.so -> libgurt.so.4.9.0

lrwxrwxrwx 1 daos_agent daos_agent      16 Dec 13 10:18 libgurt.so.4 -> libgurt.so.4.9.0

-rwxrwxr-x 1 daos_agent daos_agent 1132488 Dec 13 10:03 libgurt.so.4.9.0

-rw-rw-r-- 1 daos_agent daos_agent  777432 Dec 13 08:06 libioil.a

-rwxrwxr-x 1 daos_agent daos_agent  411776 Dec 13 10:18 libioil.so

-rw-rw-r-- 1 daos_agent daos_agent  169190 Dec 13 08:03 libnvme_control.a

drwxrwxr-x 3 daos_agent daos_agent    4096 Dec 13 08:07 python3.8


And "${MY_DAOS_INSTALL_PATH}/include" is the installed include directory.

$ ls -l ${MY_DAOS_INSTALL_PATH}/include

total 332

drwxrwxr-x 2 daos_agent daos_agent  4096 Dec 17 08:11 cart

drwxrwxr-x 2 daos_agent daos_agent  4096 Dec 13 08:06 daos

-rw-r--r-- 1 daos_agent daos_agent  6274 Dec 13 06:39 daos_api.h

-rw-r--r-- 1 daos_agent daos_agent 14770 Dec 13 06:39 daos_array.h

-rw-r--r-- 1 daos_agent daos_agent 27543 Dec 13 06:39 daos_cont.h

-rw-r--r-- 1 daos_agent daos_agent 13947 Dec 13 06:39 daos_errno.h

-rw-r--r-- 1 daos_agent daos_agent  8573 Dec 13 06:39 daos_event.h

-rw-r--r-- 1 daos_agent daos_agent 32290 Dec 13 06:39 daos_fs.h

-rw-r--r-- 1 daos_agent daos_agent 16091 Dec 13 06:39 daos_fs_sys.h

-rw-r--r-- 1 daos_agent daos_agent   974 Dec 13 06:39 daos.h

-rw-r--r-- 1 daos_agent daos_agent  6895 Dec 13 06:39 daos_kv.h

-rw-r--r-- 1 daos_agent daos_agent  2990 Dec 13 06:39 daos_mgmt.h

-rw-r--r-- 1 daos_agent daos_agent 17727 Dec 13 06:39 daos_obj_class.h

-rw-r--r-- 1 daos_agent daos_agent 39305 Dec 13 06:39 daos_obj.h

-rw-r--r-- 1 daos_agent daos_agent 14316 Dec 13 06:39 daos_pool.h

-rw-r--r-- 1 daos_agent daos_agent 16599 Dec 13 06:39 daos_prop.h

-rw-r--r-- 1 daos_agent daos_agent 17364 Dec 13 06:39 daos_security.h

drwxrwxr-x 2 daos_agent daos_agent  4096 Dec 13 08:06 daos_srv

-rw-r--r-- 1 daos_agent daos_agent 27072 Dec 13 06:39 daos_task.h

-rw-r--r-- 1 daos_agent daos_agent  4872 Dec 13 06:39 daos_types.h

-rw-r--r-- 1 daos_agent daos_agent  8607 Dec 13 06:39 daos_uns.h

-rw-rw-r-- 1 daos_agent daos_agent   486 Dec 13 08:02 daos_version.h

drwxrwxr-x 2 daos_agent daos_agent  4096 Dec 20 02:12 gurt

drwxrwxr-x 2 daos_agent daos_agent  4096 Dec 13 08:06 spdk


My command to compile daos is "scons PREFIX=./build install --build-deps=yes --config=force".


Re: Unable to build libmfu properly

netsurfed
 

Hi, yes, "build" is where daos is installed.
$ ls -l /home/daos_agent/git/daos/build/lib64/
total 22080
drwxrwxr-x 3 daos_agent daos_agent    4096 Dec 13 08:06 daos
drwxrwxr-x 2 daos_agent daos_agent    4096 Dec 13 10:18 daos_srv
lrwxrwxrwx 1 daos_agent daos_agent      16 Dec 13 10:18 libcart.so -> libcart.so.4.9.0
lrwxrwxrwx 1 daos_agent daos_agent      16 Dec 13 10:18 libcart.so.4 -> libcart.so.4.9.0
-rwxrwxr-x 1 daos_agent daos_agent 4948104 Dec 13 10:10 libcart.so.4.9.0
-rwxrwxr-x 1 daos_agent daos_agent  710680 Dec 13 10:10 libdaos_cmd_hdlrs.so
-rwxrwxr-x 1 daos_agent daos_agent 2955992 Dec 13 10:10 libdaos_common_pmem.so
-rwxrwxr-x 1 daos_agent daos_agent 2905240 Dec 13 10:10 libdaos_common.so
lrwxrwxrwx 1 daos_agent daos_agent      16 Dec 13 10:18 libdaos.so -> libdaos.so.2.0.0
lrwxrwxrwx 1 daos_agent daos_agent      16 Dec 13 10:18 libdaos.so.2 -> libdaos.so.2.0.0
-rwxrwxr-x 1 daos_agent daos_agent 6814552 Dec 13 10:04 libdaos.so.2.0.0
-rwxrwxr-x 1 daos_agent daos_agent  274776 Dec 13 10:10 libdaos_tests.so
-rwxrwxr-x 1 daos_agent daos_agent   27160 Dec 13 10:18 libdfs_internal.so
-rwxrwxr-x 1 daos_agent daos_agent  816728 Dec 13 10:10 libdfs.so
-rw-rw-r-- 1 daos_agent daos_agent  206304 Dec 13 08:07 libdfuse.a
-rwxrwxr-x 1 daos_agent daos_agent  137440 Dec 13 10:18 libdfuse.so
-rwxrwxr-x 1 daos_agent daos_agent  116120 Dec 13 10:10 libdts.so
-rwxrwxr-x 1 daos_agent daos_agent  158712 Dec 13 10:10 libduns.so
lrwxrwxrwx 1 daos_agent daos_agent      16 Dec 13 10:18 libgurt.so -> libgurt.so.4.9.0
lrwxrwxrwx 1 daos_agent daos_agent      16 Dec 13 10:18 libgurt.so.4 -> libgurt.so.4.9.0
-rwxrwxr-x 1 daos_agent daos_agent 1132488 Dec 13 10:03 libgurt.so.4.9.0
-rw-rw-r-- 1 daos_agent daos_agent  777432 Dec 13 08:06 libioil.a
-rwxrwxr-x 1 daos_agent daos_agent  411776 Dec 13 10:18 libioil.so
-rw-rw-r-- 1 daos_agent daos_agent  169190 Dec 13 08:03 libnvme_control.a
drwxrwxr-x 3 daos_agent daos_agent    4096 Dec 13 08:07 python3.8

And "${MY_DAOS_INSTALL_PATH}/include" is the installed include directory.
$ ls -l ${MY_DAOS_INSTALL_PATH}/include
total 332
drwxrwxr-x 2 daos_agent daos_agent  4096 Dec 17 08:11 cart
drwxrwxr-x 2 daos_agent daos_agent  4096 Dec 13 08:06 daos
-rw-r--r-- 1 daos_agent daos_agent  6274 Dec 13 06:39 daos_api.h
-rw-r--r-- 1 daos_agent daos_agent 14770 Dec 13 06:39 daos_array.h
-rw-r--r-- 1 daos_agent daos_agent 27543 Dec 13 06:39 daos_cont.h
-rw-r--r-- 1 daos_agent daos_agent 13947 Dec 13 06:39 daos_errno.h
-rw-r--r-- 1 daos_agent daos_agent  8573 Dec 13 06:39 daos_event.h
-rw-r--r-- 1 daos_agent daos_agent 32290 Dec 13 06:39 daos_fs.h
-rw-r--r-- 1 daos_agent daos_agent 16091 Dec 13 06:39 daos_fs_sys.h
-rw-r--r-- 1 daos_agent daos_agent   974 Dec 13 06:39 daos.h
-rw-r--r-- 1 daos_agent daos_agent  6895 Dec 13 06:39 daos_kv.h
-rw-r--r-- 1 daos_agent daos_agent  2990 Dec 13 06:39 daos_mgmt.h
-rw-r--r-- 1 daos_agent daos_agent 17727 Dec 13 06:39 daos_obj_class.h
-rw-r--r-- 1 daos_agent daos_agent 39305 Dec 13 06:39 daos_obj.h
-rw-r--r-- 1 daos_agent daos_agent 14316 Dec 13 06:39 daos_pool.h
-rw-r--r-- 1 daos_agent daos_agent 16599 Dec 13 06:39 daos_prop.h
-rw-r--r-- 1 daos_agent daos_agent 17364 Dec 13 06:39 daos_security.h
drwxrwxr-x 2 daos_agent daos_agent  4096 Dec 13 08:06 daos_srv
-rw-r--r-- 1 daos_agent daos_agent 27072 Dec 13 06:39 daos_task.h
-rw-r--r-- 1 daos_agent daos_agent  4872 Dec 13 06:39 daos_types.h
-rw-r--r-- 1 daos_agent daos_agent  8607 Dec 13 06:39 daos_uns.h
-rw-rw-r-- 1 daos_agent daos_agent   486 Dec 13 08:02 daos_version.h
drwxrwxr-x 2 daos_agent daos_agent  4096 Dec 20 02:12 gurt
drwxrwxr-x 2 daos_agent daos_agent  4096 Dec 13 08:06 spdk

My command to compile daos is "scons PREFIX=./build install --build-deps=yes --config=force".


Re: Unable to build libmfu properly

Bohning, Dalton
 

Is this directory correct?

/home/daos_agent/git/daos/build/lib64/

Libraries are usually in the “install” directory, not “build”. Is “build” in fact where daos is installed?

 

Similarly, is ${MY_DAOS_INSTALL_PATH}/include the installed include directory?

 

~Dalton Bohning

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of netsurfed
Sent: Monday, December 27, 2021 9:38 PM
To: daos@daos.groups.io
Subject: Re: [daos] Unable to build libmfu properly

 

On Tue, Dec 28, 2021 at 03:13 AM, Bohning, Dalton wrote:

grep -B 2 "CMAKE_SHARED_LINKER_FLAGS:" ${MY_MFU_BUILD_PATH}/CMakeCache.txt

Hi, I run your command and it prints as follows:
$ grep -B 2 "CMAKE_SHARED_LINKER_FLAGS:" ${MY_MFU_BUILD_PATH}/CMakeCache.txt

//Flags used by the linker during the creation of shared libraries

// during all build types.

CMAKE_SHARED_LINKER_FLAGS:STRING=-L/home/daos_agent/git/daos/build/lib64/ -luuid -ldaos -ldfs -ldaos_common -lgurt -lpthread

 


Re: Unable to build libmfu properly

netsurfed
 

On Tue, Dec 28, 2021 at 03:13 AM, Bohning, Dalton wrote:
grep -B 2 "CMAKE_SHARED_LINKER_FLAGS:" ${MY_MFU_BUILD_PATH}/CMakeCache.txt
Hi, I run your command and it prints as follows:
$ grep -B 2 "CMAKE_SHARED_LINKER_FLAGS:" ${MY_MFU_BUILD_PATH}/CMakeCache.txt
//Flags used by the linker during the creation of shared libraries
// during all build types.
CMAKE_SHARED_LINKER_FLAGS:STRING=-L/home/daos_agent/git/daos/build/lib64/ -luuid -ldaos -ldfs -ldaos_common -lgurt -lpthread
 


Re: Unable to build libmfu properly

Bohning, Dalton
 

Apologies, my previous command snippet was muddled. Correction:

$ grep -B 2 "CMAKE_SHARED_LINKER_FLAGS:" ${MY_MFU_BUILD_PATH}/CMakeCache.txt

//Flags used by the linker during the creation of shared libraries

// during all build types.

CMAKE_SHARED_LINKER_FLAGS:STRING=-L/home/dbohning/daos/install/lib64/ -luuid -ldaos -ldfs -ldaos_common -lgurt -lpthread

 

~Dalton Bohning

 

 

From: Bohning, Dalton
Sent: Monday, December 27, 2021 8:50 AM
To: daos@daos.groups.io
Subject: RE: [daos] Unable to build libmfu properly

 

Hello,

 

It looks like most  of the libraries specified by “-luuid -ldaos -ldfs -ldaos_common -lgurt -lpthread” are not linked. Something to keep in mind is that “CFLAGS=… LDFLAGS=… cmake…” should be executed as a single command so CFLAGS and LDFLAGS is propagated to the cmake environment. You can check this by running:

$ grep -B 1 "CMAKE_EXE_LINKER_FLAGS:" ${MY_MFU_BUILD_PATH}/CMakeCache.txt

grep -B 2 "CMAKE_SHARED_LINKER_FLAGS:" CMakeCache.txt

//Flags used by the linker during the creation of shared libraries

// during all build types.

CMAKE_SHARED_LINKER_FLAGS:STRING=-L/home/dbohning/daos/install/lib64/ -luuid -ldaos -ldfs -ldaos_common -lgurt -lpthread

 

Where CMAKE_SHARED_LINKER_FLAGS should be set to the LDFLAGS from the build command.

 

~Dalton Bohning

 

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of netsurfed
Sent: Thursday, December 23, 2021 7:11 PM
To: daos@daos.groups.io
Subject: [daos] Unable to build libmfu properly

 

Hi,
I built MFU according to https://daosio.atlassian.net/wiki/spaces/DC/pages/4874571083/IO-500+ISC21 but encountered some "undefined reference" errors.
[ 39%] Linking C executable dbcast
/usr/bin/ld: ../common/libmfu.so: undefined reference to `d_hash_rec_insert'
/usr/bin/ld: ../common/libmfu.so: undefined reference to `dfs_lookup'
/usr/bin/ld: ../common/libmfu.so: undefined reference to `d_hash_rec_unlinked'
/usr/bin/ld: ../common/libmfu.so: undefined reference to `dfs_stat'
/usr/bin/ld: ../common/libmfu.so: undefined reference to `dfs_readdir'
/usr/bin/ld: ../common/libmfu.so: undefined reference to `d_hash_table_create'
/usr/bin/ld: ../common/libmfu.so: undefined reference to `dfs_obj_anchor_split'
/usr/bin/ld: ../common/libmfu.so: undefined reference to `dfs_release'
/usr/bin/ld: ../common/libmfu.so: undefined reference to `d_hash_rec_find'
/usr/bin/ld: ../common/libmfu.so: undefined reference to `dfs_obj_anchor_set'

The build command is:
git clone https://github.com/mchaarawi/mpifileutils -b pfind_integration "${MY_MFU_SOURCE_PATH}" &&
mkdir -p "${MY_MFU_BUILD_PATH}" &&
cd "${MY_MFU_BUILD_PATH}" &&
CFLAGS="-I${MY_DAOS_INSTALL_PATH}/include" \
LDFLAGS="-L${MY_DAOS_INSTALL_PATH}/lib64/ -luuid -ldaos -ldfs -ldaos_common -lgurt -lpthread" \
cmake "${MY_MFU_SOURCE_PATH}" \
-DENABLE_XATTRS=OFF \
-DWITH_DTCMP_PREFIX=${MY_MFU_INSTALL_PATH} \
-DWITH_LibCircle_PREFIX=${MY_MFU_INSTALL_PATH} \
-DCMAKE_INSTALL_PREFIX=${MY_MFU_INSTALL_PATH} &&
make -j8 install

And there is libgurt.so in ${MY_DAOS_INSTALL_PATH}/lib64/.
$ nm -D ${MY_DAOS_INSTALL_PATH}/lib64/libgurt.so | grep d_hash_rec_insert
000000000000fb40 T d_hash_rec_insert
0000000000010100 T d_hash_rec_insert_anonym

But looks like libgurt.so is not linked by libmfu.so.
$ ldd src/common/libmfu.so
linux-vdso.so.1 (0x00007ffda13f3000)
libmpi.so.40 => /lib/x86_64-linux-gnu/libmpi.so.40 (0x00007fd75c830000)
libdtcmp.so => /home/daos_agent/install/lib/libdtcmp.so (0x00007fd75c814000)
libcircle.so.2 => /home/daos_agent/install/lib/libcircle.so.2 (0x00007fd75c806000)
libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007fd75c7f3000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd75c601000)
libopen-rte.so.40 => /lib/x86_64-linux-gnu/libopen-rte.so.40 (0x00007fd75c545000)
libopen-pal.so.40 => /lib/x86_64-linux-gnu/libopen-pal.so.40 (0x00007fd75c497000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fd75c348000)
libhwloc.so.15 => /lib/x86_64-linux-gnu/libhwloc.so.15 (0x00007fd75c2f7000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fd75c2d4000)
liblwgrp.so => /home/daos_agent/install/lib/liblwgrp.so (0x00007fd75c2c5000)
/lib64/ld-linux-x86-64.so.2 (0x00007fd75c9b2000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fd75c2a7000)
libevent-2.1.so.7 => /lib/x86_64-linux-gnu/libevent-2.1.so.7 (0x00007fd75c251000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fd75c24b000)
libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007fd75c246000)
libevent_pthreads-2.1.so.7 => /lib/x86_64-linux-gnu/libevent_pthreads-2.1.so.7 (0x00007fd75c241000)
libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007fd75c214000)
libltdl.so.7 => /lib/x86_64-linux-gnu/libltdl.so.7 (0x00007fd75c207000)"


Re: Unable to build libmfu properly

Bohning, Dalton
 

Hello,

 

It looks like most  of the libraries specified by “-luuid -ldaos -ldfs -ldaos_common -lgurt -lpthread” are not linked. Something to keep in mind is that “CFLAGS=… LDFLAGS=… cmake…” should be executed as a single command so CFLAGS and LDFLAGS is propagated to the cmake environment. You can check this by running:

$ grep -B 1 "CMAKE_EXE_LINKER_FLAGS:" ${MY_MFU_BUILD_PATH}/CMakeCache.txt

grep -B 2 "CMAKE_SHARED_LINKER_FLAGS:" CMakeCache.txt

//Flags used by the linker during the creation of shared libraries

// during all build types.

CMAKE_SHARED_LINKER_FLAGS:STRING=-L/home/dbohning/daos/install/lib64/ -luuid -ldaos -ldfs -ldaos_common -lgurt -lpthread

 

Where CMAKE_SHARED_LINKER_FLAGS should be set to the LDFLAGS from the build command.

 

~Dalton Bohning

 

 

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of netsurfed
Sent: Thursday, December 23, 2021 7:11 PM
To: daos@daos.groups.io
Subject: [daos] Unable to build libmfu properly

 

Hi,
I built MFU according to https://daosio.atlassian.net/wiki/spaces/DC/pages/4874571083/IO-500+ISC21 but encountered some "undefined reference" errors.
[ 39%] Linking C executable dbcast
/usr/bin/ld: ../common/libmfu.so: undefined reference to `d_hash_rec_insert'
/usr/bin/ld: ../common/libmfu.so: undefined reference to `dfs_lookup'
/usr/bin/ld: ../common/libmfu.so: undefined reference to `d_hash_rec_unlinked'
/usr/bin/ld: ../common/libmfu.so: undefined reference to `dfs_stat'
/usr/bin/ld: ../common/libmfu.so: undefined reference to `dfs_readdir'
/usr/bin/ld: ../common/libmfu.so: undefined reference to `d_hash_table_create'
/usr/bin/ld: ../common/libmfu.so: undefined reference to `dfs_obj_anchor_split'
/usr/bin/ld: ../common/libmfu.so: undefined reference to `dfs_release'
/usr/bin/ld: ../common/libmfu.so: undefined reference to `d_hash_rec_find'
/usr/bin/ld: ../common/libmfu.so: undefined reference to `dfs_obj_anchor_set'

The build command is:
git clone https://github.com/mchaarawi/mpifileutils -b pfind_integration "${MY_MFU_SOURCE_PATH}" &&
mkdir -p "${MY_MFU_BUILD_PATH}" &&
cd "${MY_MFU_BUILD_PATH}" &&
CFLAGS="-I${MY_DAOS_INSTALL_PATH}/include" \
LDFLAGS="-L${MY_DAOS_INSTALL_PATH}/lib64/ -luuid -ldaos -ldfs -ldaos_common -lgurt -lpthread" \
cmake "${MY_MFU_SOURCE_PATH}" \
-DENABLE_XATTRS=OFF \
-DWITH_DTCMP_PREFIX=${MY_MFU_INSTALL_PATH} \
-DWITH_LibCircle_PREFIX=${MY_MFU_INSTALL_PATH} \
-DCMAKE_INSTALL_PREFIX=${MY_MFU_INSTALL_PATH} &&
make -j8 install

And there is libgurt.so in ${MY_DAOS_INSTALL_PATH}/lib64/.
$ nm -D ${MY_DAOS_INSTALL_PATH}/lib64/libgurt.so | grep d_hash_rec_insert
000000000000fb40 T d_hash_rec_insert
0000000000010100 T d_hash_rec_insert_anonym

But looks like libgurt.so is not linked by libmfu.so.
$ ldd src/common/libmfu.so
linux-vdso.so.1 (0x00007ffda13f3000)
libmpi.so.40 => /lib/x86_64-linux-gnu/libmpi.so.40 (0x00007fd75c830000)
libdtcmp.so => /home/daos_agent/install/lib/libdtcmp.so (0x00007fd75c814000)
libcircle.so.2 => /home/daos_agent/install/lib/libcircle.so.2 (0x00007fd75c806000)
libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007fd75c7f3000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd75c601000)
libopen-rte.so.40 => /lib/x86_64-linux-gnu/libopen-rte.so.40 (0x00007fd75c545000)
libopen-pal.so.40 => /lib/x86_64-linux-gnu/libopen-pal.so.40 (0x00007fd75c497000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fd75c348000)
libhwloc.so.15 => /lib/x86_64-linux-gnu/libhwloc.so.15 (0x00007fd75c2f7000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fd75c2d4000)
liblwgrp.so => /home/daos_agent/install/lib/liblwgrp.so (0x00007fd75c2c5000)
/lib64/ld-linux-x86-64.so.2 (0x00007fd75c9b2000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fd75c2a7000)
libevent-2.1.so.7 => /lib/x86_64-linux-gnu/libevent-2.1.so.7 (0x00007fd75c251000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fd75c24b000)
libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007fd75c246000)
libevent_pthreads-2.1.so.7 => /lib/x86_64-linux-gnu/libevent_pthreads-2.1.so.7 (0x00007fd75c241000)
libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007fd75c214000)
libltdl.so.7 => /lib/x86_64-linux-gnu/libltdl.so.7 (0x00007fd75c207000)"


Re: Questions about ULT Schedule

Niu, Yawei
 

Hi,

 

The design is to ensure that all IO requests from different pools are processed in FIFO order, and space pressure from one pool doesn’t interfere request processing for other pools, but the implementation of policy_fifo_process() (as you pointed out) does have a defect which makes the latter requests (in FIFO queue) are impeded by the former request for a different pool with space pressure.

 

I’ll  cook a quick fix soon, thanks a lot for spotting this!

 

Thanks

-Niu  

 

From: daos@daos.groups.io <daos@daos.groups.io> on behalf of 段世博 <duanshibo.d@...>
Date: Sunday, December 26, 2021 at 10:01 PM
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: [daos] Questions about ULT Schedule

Hello, Everyone!

Now all update/fetch requests of pools are in sched_info->si_fifo_list. If a pool request exceeds req_kick_limit, will it also block other pool requests?

thanks!


Re: failed to create daos cont #chat

hmu102
 

solved at 23 Dec

at the agent.yml, edited the interface numa from "1" to "0", though there is no port under numa 0, but it works at last.

141 - 160 of 1653