Re: no dRPC client set on Ubuntu 20.04.1
Hi Tom,
Thanks for resolving the issue. Thanks to Jim too. I don't see the errors anymore. Rgds, Gert,
|
|
Re: IOR test with dfs
Lombardi, Johann
Could you please change “log_mask: DEBUG” to “log_mask: ERR”, restart the server and rerun ior?
Cheers, Johann
From:
<daos@daos.groups.io> on behalf of "anton.brekhov@..." <anton.brekhov@...>
Johann thanks for help! Here is the information about server: [root@apache512 ior]# lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 112 On-line CPU(s) list: 0-111 Thread(s) per core: 2 Core(s) per socket: 28 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Platinum 8280 CPU @ 2.70GHz Stepping: 7 CPU MHz: 3699.975 CPU max MHz: 4000,0000 CPU min MHz: 1000,0000 BogoMIPS: 5400.00 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 1024K L3 cache: 39424K NUMA node0 CPU(s): 0-27,56-83 NUMA node1 CPU(s): 28-55,84-111
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 invpcid_single intel_pt ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke avx512_vnni md_clear spec_ctrl intel_stibp flush_l1d arch_capabilities
port: 10001 transport_config: allow_insecure: true server_name: server client_cert_dir: .daos/clients ca_cert: .daos/daosCA.crt cert: .daos/daos_server.crt key: .daos/daos_server.key servers: - targets: 4 nr_xs_helpers: 0 first_core: 0 name: daos_server socket_dir: /var/run/daos_server log_mask: DEBUG log_file: /tmp/daos_server.log scm_mount: /mnt/daos scm_class: dcpm scm_list: - /dev/pmem0 bdev_class: nvme bdev_list: - 0000:b1:00.0 - 0000:b2:00.0 - 0000:b3:00.0 - 0000:b4:00.0 provider: ofi+verbs;ofi_rxm fabric_iface: ib0 fabric_iface_port: 31416 env_vars: - DAOS_MD_CAP=1024 - CRT_CTX_SHARE_ADDR=0 - CRT_TIMEOUT=30 - FI_SOCKETS_MAX_CONN_RETRY=1 - FI_SOCKETS_CONN_TIMEOUT=2000 nr_hugepages: 4096 set_hugepages: false control_log_mask: INFO control_log_file: /tmp/daos_control.log helper_log_file: "" recreate_superblocks: false name: daos_server socket_dir: /var/run/daos_server provider: ofi+verbs;ofi_rxm modules: "" access_points: - apache512:10001 fault_path: "" fault_cb: "" hyperthreads: false
path: /root/daos_configs/local_SCM.yml --------------------------------------------------------------------- This e-mail and any attachments may contain confidential material for
|
|
Re: IOR test with dfs
anton.brekhov@...
Here is the screenshot. Some words in Russian, but packages written in English.
yum downgrade boost-random boost-iostreams boost-system boost-thread boost boost-filesystem yum install daos-devel
|
|
Re: no dRPC client set on Ubuntu 20.04.1
Hello Gert and Lee,
https://github.com/daos-stack/daos/pull/3480 has been merged with master to fix this issue.
dpdk (used as the default environment for spdk) is using weak symbols when linking:
"dpdk/lib/librte_eal/common/include/rte_common.h:#define _rte_weak _attribute__((__weak__)) "
and Ubuntu sets --as-needed which means some of the constructor functions in dpdk mempool libs (which handle the access to hugepages) don't get run and cause failure in spdk_thread_lib_init() (called from bio_xstream.c in DAOS).
The fix involves setting --no-as-needed linker flags explicitly during build of the relevant bio module and control-plane libs to pull in all symbols regardless of whether they are explicitly called or not.
Thanks to Jim Harris in the SPDK for help in resolving the issue. Please confirm the fix works if possible.
Regards, Tom Nabarro – DCG/ESAD M: +44 (0)7786 260986 Skype: tom.nabarro
From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of
Gert Pauwels
Hi Tom, The daos_server.log, daos_adim.log and daos_control.log are attached. root@intel-S2600WFD:~/daos#
cat /proc/meminfo | grep Huge root@intel-S2600WFD:~/daos#
ls -lah /dev/hugepages/spdk_pid4857map_0
--------------------------------------------------------------------- This e-mail and any attachments may contain confidential material for
|
|
Re: IOR test with dfs
Murrell, Brian
On Tue, 2020-09-22 at 01:43 -0700, anton.brekhov@... wrote:
*At first I had some issues while installing daos-devel, cause I hadIt would be great if you could paste the errors you got with the newer boost and what version of boost it was. I understand though if those messages are no longer available to you. But then I've downgraded boost packages and installed daos-devel.Do you know which versions of boost you had and then had to downgrade to? "yum history" should be able to tell you if you are not sure from memory. Cheers, b.
|
|
Re: IOR test with dfs
Murrell, Brian
On Mon, 2020-09-21 at 17:34 +0000, Lombardi, Johann wrote:
Thanks Mohamad & Steffen.Why not go further and not even require the use of --with-daos if daos is installed at the default location of /usr and just use it if it's there and don't use it if it's not. That sort of behavior is quite common and usually goes along with a --without-* switch for the case where it's present but not wanted in the build, which are usually the minority of the cases. Typically if somebody installs a feature, they want to use it. Cheers, b.
|
|
Re: IOR test with dfs
anton.brekhov@...
Johann thanks for help! Here is the information about server: [root@apache512 ior]# lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 112 On-line CPU(s) list: 0-111 Thread(s) per core: 2 Core(s) per socket: 28 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Platinum 8280 CPU @ 2.70GHz Stepping: 7 CPU MHz: 3699.975 CPU max MHz: 4000,0000 CPU min MHz: 1000,0000 BogoMIPS: 5400.00 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 1024K L3 cache: 39424K NUMA node0 CPU(s): 0-27,56-83 NUMA node1 CPU(s): 28-55,84-111
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 invpcid_single intel_pt ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke avx512_vnni md_clear spec_ctrl intel_stibp flush_l1d arch_capabilities
port: 10001 transport_config: allow_insecure: true server_name: server client_cert_dir: .daos/clients ca_cert: .daos/daosCA.crt cert: .daos/daos_server.crt key: .daos/daos_server.key servers: - targets: 4 nr_xs_helpers: 0 first_core: 0 name: daos_server socket_dir: /var/run/daos_server log_mask: DEBUG log_file: /tmp/daos_server.log scm_mount: /mnt/daos scm_class: dcpm scm_list: - /dev/pmem0 bdev_class: nvme bdev_list: - 0000:b1:00.0 - 0000:b2:00.0 - 0000:b3:00.0 - 0000:b4:00.0 provider: ofi+verbs;ofi_rxm fabric_iface: ib0 fabric_iface_port: 31416 env_vars: - DAOS_MD_CAP=1024 - CRT_CTX_SHARE_ADDR=0 - CRT_TIMEOUT=30 - FI_SOCKETS_MAX_CONN_RETRY=1 - FI_SOCKETS_CONN_TIMEOUT=2000 nr_hugepages: 4096 set_hugepages: false control_log_mask: INFO control_log_file: /tmp/daos_control.log helper_log_file: "" recreate_superblocks: false name: daos_server socket_dir: /var/run/daos_server provider: ofi+verbs;ofi_rxm modules: "" access_points: - apache512:10001 fault_path: "" fault_cb: "" hyperthreads: false path: /root/daos_configs/local_SCM.yml
|
|
Re: IOR test with dfs
Lombardi, Johann
Hi Anton,
Saw your results on the other email. A few comments:
Cheers, Johann
From:
<daos@daos.groups.io> on behalf of "anton.brekhov@..." <anton.brekhov@...>
At first I had some issues while installing daos-devel, cause I had newer boost-* packages from base repo. So i decided to built it from source. But then I've downgraded boost
packages and installed daos-devel. All headers appeared! Thanks. --------------------------------------------------------------------- This e-mail and any attachments may contain confidential material for
|
|
Re: IOR test with dfs
anton.brekhov@...
At first I had some issues while installing daos-devel, cause I had newer boost-* packages from base repo. So i decided to built it from source. But then I've downgraded boost packages and installed daos-devel. All headers appeared! Thanks.
|
|
Re: IOR test with dfs
Lombardi, Johann
Anton, aren’t the header files in the *-devel RPMs as mentioned by Steffen?
Cheers, Johann
From:
<daos@daos.groups.io> on behalf of "anton.brekhov@..." <anton.brekhov@...>
Thank you everyone for the help! checking for d_hash_murmur64 in -lgurt... yes checking daos_types.h usability... yes checking daos_types.h presence... yes checking for daos_types.h... yes checking for uuid_generate in -luuid... yes checking for daos_sgl_init in -ldaos_common... yes checking for daos_init in -ldaos... yes checking for dfs_mkdir in -ldfs... yes checking for Gfarm file system... no checking that generated files are newer than configure... done configure: creating ./config.status config.status: creating Makefile config.status: creating src/Makefile config.status: creating src/test/Makefile config.status: creating contrib/Makefile config.status: creating doc/Makefile
config.status: creating src/config.h
--------------------------------------------------------------------- This e-mail and any attachments may contain confidential material for
|
|
Re: IOR test with dfs
anton.brekhov@...
I feel like I didn't get high performance through DFS driver:
DFS: mpirun --allow-run-as-root -np 36 --bind-to socket ior -a DFS -b 1G -w -r -o /tmp/dfuse/testFile_dfs -t 128k -k --dfs.pool $dpool --dfs.svcl 0 --dfs.cont $dcont -
-dfs.prefix /tmp/dfuse -G 123 -F
....Command line : ior -a DFS -b 1G -w -r -o /tmp/dfuse/testFile_dfs -t 128k -k --dfs.pool 3f0762ad-23b3-418a-a14d-4f813d122d46 --dfs.svcl 0 --dfs.cont 0e75d729-2164-493e-ba9e-2942df394bca --dfs.prefix /tmp/dfuse -G 123 -F
....
access bw(MiB/s) IOPS Latency(s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) total(s) iter
------ --------- ---- ---------- ---------- --------- -------- -------- -------- -------- ----
write 524.01 4192 0.008581 1048576 128.00 0.015706 70.35 3.73 70.35 0
read 635.64 5085 0.007016 1048576 128.00 0.002732 57.99 2.78 57.99 0
POSIX: mpirun --allow-run-as-root -np 36 --bind-to socket ior -a POSIX -b 1G -w -r -o /tmp/dfuse/testFile_POSIX -t 128k -k -G 123 -F ... Command line : ior -a POSIX -b 1G -w -r -o /tmp/dfuse/testFile_POSIX -t 128k -k -G 123 -F ... access bw(MiB/s) IOPS Latency(s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) total(s) iter
------ --------- ---- ---------- ---------- --------- -------- -------- -------- -------- ----
write 587.98 4704 0.006442 1048576 128.00 0.285280 62.69 11.46 62.70 0
read 155.22 1243.86 0.028893 1048576 128.00 0.425217 237.09 5.65 237.49 0
Is it wrong? I believe that DFS must be much higher
|
|
Re: IOR test with dfs
anton.brekhov@...
Thank you everyone for the help! checking for d_hash_murmur64 in -lgurt... yes checking daos_types.h usability... yes checking daos_types.h presence... yes checking for daos_types.h... yes checking for uuid_generate in -luuid... yes checking for daos_sgl_init in -ldaos_common... yes checking for daos_init in -ldaos... yes checking for dfs_mkdir in -ldfs... yes checking for Gfarm file system... no checking that generated files are newer than configure... done configure: creating ./config.status config.status: creating Makefile config.status: creating src/Makefile config.status: creating src/test/Makefile config.status: creating contrib/Makefile config.status: creating doc/Makefile
config.status: creating src/config.h
|
|
Re: IOR test with dfs
Lombardi, Johann
Thanks Mohamad & Steffen. Can we easily change the IOR build to assume “--with-daos” = “--with-daos=/usr --with-cart=/usr”? TBH, I would also just assume that CART and DAOS libs are located at the same place. Thoughts?
Cheers, Johann
From:
<daos@daos.groups.io> on behalf of "Chaarawi, Mohamad" <mohamad.chaarawi@...>
Yes indeed as Steffen mentioned you need the headers also. Also --with-daos and --with-cart should point to the same location (note that in 1.0, cart is a stand-alone lib, and that's why we had to have 2 paths). (i.e. you need to pass a path to --with-cart=/path/)
Thanks, Mohamad
On 9/21/20, 7:16 AM, "daos@daos.groups.io on behalf of Steffen Christgau" <daos@daos.groups.io on behalf of christgau@...> wrote:
Dear Johann,
On 9/21/20 2:00 PM, Lombardi, Johann wrote: > I was hoping that --with-daoswould be enough if everything is installed > at the default location via RPMs.
Maybe unrelated, but for getting MPICH 3.4a with DAOS ROMIO to compile on a DAOS installation I had to specify both "--with-daos=/usr" and "--with-cart=/usr"
Regards, Steffen
--------------------------------------------------------------------- This e-mail and any attachments may contain confidential material for
|
|
Re: DAOS starting example in C++
adam.abed.abud@...
Hi Johann,
Thanks a lot for your feedback. I compiled the project and I finally managed to get it running on the cluster. Cheers, Adam
|
|
Re: IOR test with dfs
Chaarawi, Mohamad
Yes indeed as Steffen mentioned you need the headers also.
Also --with-daos and --with-cart should point to the same location (note that in 1.0, cart is a stand-alone lib, and that's why we had to have 2 paths). (i.e. you need to pass a path to --with-cart=/path/) Thanks, Mohamad On 9/21/20, 7:16 AM, "daos@daos.groups.io on behalf of Steffen Christgau" <daos@daos.groups.io on behalf of christgau@...> wrote: Dear Johann, On 9/21/20 2:00 PM, Lombardi, Johann wrote: > I was hoping that --with-daoswould be enough if everything is installed > at the default location via RPMs. Maybe unrelated, but for getting MPICH 3.4a with DAOS ROMIO to compile on a DAOS installation I had to specify both "--with-daos=/usr" and "--with-cart=/usr" Regards, Steffen
|
|
Re: IOR test with dfs
Steffen Christgau
Dear Johann,
On 9/21/20 2:00 PM, Lombardi, Johann wrote: I was hoping that --with-daoswould be enough if everything is installed at the default location via RPMs.Maybe unrelated, but for getting MPICH 3.4a with DAOS ROMIO to compile on a DAOS installation I had to specify both "--with-daos=/usr" and "--with-cart=/usr" Regards, Steffen
|
|
Re: IOR test with dfs
Steffen Christgau
Hi Anton,
On 9/21/20 12:44 PM, anton.brekhov@... wrote: *I installed DAOS as rpm v1.0.1.Did you "only" install the daos package? IMO it does not contain the DAOS header files. Try to install daos-devel. Regards, Steffen
|
|
Re: IOR test with dfs
Lombardi, Johann
I was hoping that --with-daos would be enough if everything is installed at the default location via RPMs. @Chaarawi, Mohamad any thoughts?
Cheers, johann
From:
<daos@daos.groups.io> on behalf of "anton.brekhov@..." <anton.brekhov@...>
Johann thanks! I installed DAOS as rpm v1.0.1. Could you please tell where is install folder? I suppose that building with my parameters didn't give right binary: checking for d_hash_murmur64 in -lgurt... no checking daos_types.h usability... no checking daos_types.h presence... no checking for daos_types.h... no checking for uuid_generate in -luuid... no checking for daos_sgl_init in -ldaos_common... yes checking for daos_init in -ldaos... no checking for dfs_mkdir in -ldfs... yes checking for Gfarm file system... no
checking that generated files are newer than configure... done
--------------------------------------------------------------------- This e-mail and any attachments may contain confidential material for
|
|
Re: IOR test with dfs
anton.brekhov@...
I just found tag in source directory and run ior building like this: checking for d_hash_murmur64 in -lgurt... no checking daos_types.h usability... yes checking daos_types.h presence... yes checking for daos_types.h... yes checking for uuid_generate in -luuid... yes checking for daos_sgl_init in -ldaos_common... yes checking for daos_init in -ldaos... yes checking for dfs_mkdir in -ldfs... yes checking for Gfarm file system... no checking that generated files are newer than configure... done configure: creating ./config.status config.status: creating Makefile config.status: creating src/Makefile config.status: creating src/test/Makefile config.status: creating contrib/Makefile config.status: creating doc/Makefile config.status: creating src/config.h config.status: src/config.h is unchanged
config.status: executing depfiles commands [root @ apache512 ior] # make Making all in src make all-recursive Making all in. CCLD ior / usr / bin / ld: ior-aiori-DFS.o: undefined symbol reference "d_hash_table_create" /root/daos/install/lib64/libgurt.so.4: error adding symbols: DSO missing from command line collect2: error: ld execution exited with return code 1 make [3]: *** [ior] Error 1 make [2]: *** [all-recursive] Error 1 make [1]: *** [all] Error 2 make: *** [all-recursive] Error 1
|
|
Re: IOR test with dfs
anton.brekhov@...
Johann thanks! I installed DAOS as rpm v1.0.1. Could you please tell where is install folder? I suppose that building with my parameters didn't give right binary: checking for d_hash_murmur64 in -lgurt... no checking daos_types.h usability... no checking daos_types.h presence... no checking for daos_types.h... no checking for uuid_generate in -luuid... no checking for daos_sgl_init in -ldaos_common... yes checking for daos_init in -ldaos... no checking for dfs_mkdir in -ldfs... yes checking for Gfarm file system... no
checking that generated files are newer than configure... done
|
|