daos@daos.groups.io | IOR test with dfs

Home Messages Hashtags Subgroups

Date Date 1 - 20 of 21

IOR test with dfs

anton.brekhov@... #1185 [root@sky08 ~]# mpirun --allow-run-as-root -np 1 ior -a=DFS -b 1G -w -r -o /mnt/dfuse/testFile_dfs -t 128k -k --dfs.pool $dpool --dfs.svcl 0 --dfs.cont $dc ont --dfs.prefix /tmp/dfuse/ -G 123 Error invalid argument: --dfs.pool Error invalid argument: 3f0762ad-23b3-418a-a14d-4f813d122d46 Error invalid argument: --dfs.svcl Error invalid argument: 0 Error invalid argument: --dfs.cont Error invalid argument: 0e75d729-2164-493e-ba9e-2942df394bca Error invalid argument: --dfs.prefix Error invalid argument: /tmp/dfuse/ Error invalid argument: 123 Invalid options Synopsis ior [root@sky08 ~]# mpirun --allow-run-as-root -np 1 ior -a=DFS -b 1G -w -r -o /mnt/dfuse/testFile_dfs -t 128k -k --dfs.pool $dpool --dfs.svcl 0 --dfs.cont $dcont --dfs.prefix /tmp/dfuse/ -G 123 Error invalid argument: --dfs.pool Error invalid argument: 3f0762ad-23b3-418a-a14d-4f813d122d46 Error invalid argument: --dfs.svcl Error invalid argument: 0 Error invalid argument: --dfs.cont Error invalid argument: 0e75d729-2164-493e-ba9e-2942df394bca Error invalid argument: --dfs.prefix Error invalid argument: /tmp/dfuse/ Error invalid argument: 123 Invalid options Synopsis ior How I need to compile ior to make this work? Thanks!
More All Messages By This Member
Lombardi, Johann #1188 Hey Anton, I assume that you have built ior with the DFS backend support as specified in https://github.com/hpc/ior/blob/master/README_DAOS, right? Although I think it should work, I have never tried “-a=DFS” and always used “-a DFS” on my side. Could you please give this a try just to double check? The prefix is also relative to the absolute file path, so you should be using --dfs.prefix /mnt/dfuse instead of /tmp/dfuse. I assume that $dpool and $dcont match respectively the pool and container UUIDs that you used to mount with dfuse, right? Also, svcl should match what was returned at pool creation time, you got 0 and not 1, right? To summarize, could you please try: ior -a DFS -t 128k -b 1G -r -w -o /mnt/dfuse/testFile_dfs -G 123 -k --dfs.pool $dpool --dfs.svcl 0 --dfs.cont $dcont --dfs.prefix /mnt/dfuse Cheers, Johann From: <daos@daos.groups.io> on behalf of "anton.brekhov@..." <anton.brekhov@...> Reply-To: "daos@daos.groups.io" <daos@daos.groups.io> Date: Sunday 20 September 2020 at 21:01 To: "daos@daos.groups.io" <daos@daos.groups.io> Subject: [daos] IOR test with dfs [root@sky08 ~]# mpirun --allow-run-as-root -np 1 ior -a=DFS -b 1G -w -r -o /mnt/dfuse/testFile_dfs -t 128k -k --dfs.pool $dpool --dfs.svcl 0 --dfs.cont $dc ont --dfs.prefix /tmp/dfuse/ -G 123 Error invalid argument: --dfs.pool Error invalid argument: 3f0762ad-23b3-418a-a14d-4f813d122d46 Error invalid argument: --dfs.svcl Error invalid argument: 0 Error invalid argument: --dfs.cont Error invalid argument: 0e75d729-2164-493e-ba9e-2942df394bca Error invalid argument: --dfs.prefix Error invalid argument: /tmp/dfuse/ Error invalid argument: 123 Invalid options Synopsis ior [root@sky08 ~]# mpirun --allow-run-as-root -np 1 ior -a=DFS -b 1G -w -r -o /mnt/dfuse/testFile_dfs -t 128k -k --dfs.pool $dpool --dfs.svcl 0 --dfs.cont $dcont --dfs.prefix /tmp/dfuse/ -G 123 Error invalid argument: --dfs.pool Error invalid argument: 3f0762ad-23b3-418a-a14d-4f813d122d46 Error invalid argument: --dfs.svcl Error invalid argument: 0 Error invalid argument: --dfs.cont Error invalid argument: 0e75d729-2164-493e-ba9e-2942df394bca Error invalid argument: --dfs.prefix Error invalid argument: /tmp/dfuse/ Error invalid argument: 123 Invalid options Synopsis ior How I need to compile ior to make this work? Thanks! --------------------------------------------------------------------- Intel Corporation SAS (French simplified joint stock company) Registered headquarters: "Les Montalets"- 2, rue de Paris, 92196 Meudon Cedex, France Registration Number: 302 456 199 R.C.S. NANTERRE Capital: 4,572,000 Euros This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
More All Messages By This Member
anton.brekhov@... #1190 Johann thanks! I installed DAOS as rpm v1.0.1. Could you please tell where is install folder? I suppose that building with my parameters didn't give right binary: [root@sky08 ior]# ./configure --with-daos=/usr/ --with-cart ..... checking for d_hash_murmur64 in -lgurt... no checking daos_types.h usability... no checking daos_types.h presence... no checking for daos_types.h... no checking for uuid_generate in -luuid... no checking for daos_sgl_init in -ldaos_common... yes checking for daos_init in -ldaos... no checking for dfs_mkdir in -ldfs... yes checking for Gfarm file system... no checking that generated files are newer than configure... done It seems like I need source code, but there's no tag for 1.0.1. Thanks!
More All Messages By This Member
anton.brekhov@... #1191 I just found tag in source directory and run ior building like this: [root@apache512 ior]# ./configure --with-daos=/root/daos/install --with-cart ... checking for d_hash_murmur64 in -lgurt... no checking daos_types.h usability... yes checking daos_types.h presence... yes checking for daos_types.h... yes checking for uuid_generate in -luuid... yes checking for daos_sgl_init in -ldaos_common... yes checking for daos_init in -ldaos... yes checking for dfs_mkdir in -ldfs... yes checking for Gfarm file system... no checking that generated files are newer than configure... done configure: creating ./config.status config.status: creating Makefile config.status: creating src/Makefile config.status: creating src/test/Makefile config.status: creating contrib/Makefile config.status: creating doc/Makefile config.status: creating src/config.h config.status: src/config.h is unchanged config.status: executing depfiles commands [root @ apache512 ior] # make Making all in src make all-recursive Making all in. CCLD ior / usr / bin / ld: ior-aiori-DFS.o: undefined symbol reference "d_hash_table_create" /root/daos/install/lib64/libgurt.so.4: error adding symbols: DSO missing from command line collect2: error: ld execution exited with return code 1 make [3]: * [ior] Error 1 make [2]: * [all-recursive] Error 1 make [1]: * [all] Error 2 make: * [all-recursive] Error 1
More All Messages By This Member
Lombardi, Johann #1192 I was hoping that --with-daos would be enough if everything is installed at the default location via RPMs. @Chaarawi, Mohamad any thoughts? Cheers, johann From: <daos@daos.groups.io> on behalf of "anton.brekhov@..." <anton.brekhov@...> Reply-To: "daos@daos.groups.io" <daos@daos.groups.io> Date: Monday 21 September 2020 at 12:44 To: "daos@daos.groups.io" <daos@daos.groups.io> Subject: Re: [daos] IOR test with dfs Johann thanks! I installed DAOS as rpm v1.0.1. Could you please tell where is install folder? I suppose that building with my parameters didn't give right binary: [root@sky08 ior]# ./configure --with-daos=/usr/ --with-cart ..... checking for d_hash_murmur64 in -lgurt... no checking daos_types.h usability... no checking daos_types.h presence... no checking for daos_types.h... no checking for uuid_generate in -luuid... no checking for daos_sgl_init in -ldaos_common... yes checking for daos_init in -ldaos... no checking for dfs_mkdir in -ldfs... yes checking for Gfarm file system... no checking that generated files are newer than configure... done It seems like I need source code, but there's no tag for 1.0.1. Thanks! --------------------------------------------------------------------- Intel Corporation SAS (French simplified joint stock company) Registered headquarters: "Les Montalets"- 2, rue de Paris, 92196 Meudon Cedex, France Registration Number: 302 456 199 R.C.S. NANTERRE Capital: 4,572,000 Euros This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
More All Messages By This Member
Steffen Christgau #1193 Hi Anton, On 9/21/20 12:44 PM, anton.brekhov@... wrote: *I installed DAOS as rpm v1.0.1. Did you "only" install the daos package? IMO it does not contain the DAOS header files. Try to install daos-devel. Regards, Steffen
More All Messages By This Member
Steffen Christgau #1194 Dear Johann, On 9/21/20 2:00 PM, Lombardi, Johann wrote: I was hoping that --with-daoswould be enough if everything is installed at the default location via RPMs. Maybe unrelated, but for getting MPICH 3.4a with DAOS ROMIO to compile on a DAOS installation I had to specify both "--with-daos=/usr" and "--with-cart=/usr" Regards, Steffen
More All Messages By This Member
Chaarawi, Mohamad #1195 Yes indeed as Steffen mentioned you need the headers also. Also --with-daos and --with-cart should point to the same location (note that in 1.0, cart is a stand-alone lib, and that's why we had to have 2 paths). (i.e. you need to pass a path to --with-cart=/path/) Thanks, Mohamad On 9/21/20, 7:16 AM, "daos@daos.groups.io on behalf of Steffen Christgau" <daos@daos.groups.io on behalf of christgau@...> wrote: Dear Johann, On 9/21/20 2:00 PM, Lombardi, Johann wrote: > I was hoping that --with-daoswould be enough if everything is installed > at the default location via RPMs. Maybe unrelated, but for getting MPICH 3.4a with DAOS ROMIO to compile on a DAOS installation I had to specify both "--with-daos=/usr" and "--with-cart=/usr" Regards, Steffen
More All Messages By This Member
Lombardi, Johann #1197 Thanks Mohamad & Steffen. Can we easily change the IOR build to assume “--with-daos” = “--with-daos=/usr --with-cart=/usr”? TBH, I would also just assume that CART and DAOS libs are located at the same place. Thoughts? Cheers, Johann From: <daos@daos.groups.io> on behalf of "Chaarawi, Mohamad" <mohamad.chaarawi@...> Reply-To: "daos@daos.groups.io" <daos@daos.groups.io> Date: Monday 21 September 2020 at 15:18 To: "daos@daos.groups.io" <daos@daos.groups.io> Subject: Re: [daos] IOR test with dfs Yes indeed as Steffen mentioned you need the headers also. Also --with-daos and --with-cart should point to the same location (note that in 1.0, cart is a stand-alone lib, and that's why we had to have 2 paths). (i.e. you need to pass a path to --with-cart=/path/) Thanks, Mohamad On 9/21/20, 7:16 AM, "daos@daos.groups.io on behalf of Steffen Christgau" <daos@daos.groups.io on behalf of christgau@...> wrote: Dear Johann, On 9/21/20 2:00 PM, Lombardi, Johann wrote: > I was hoping that --with-daoswould be enough if everything is installed > at the default location via RPMs. Maybe unrelated, but for getting MPICH 3.4a with DAOS ROMIO to compile on a DAOS installation I had to specify both "--with-daos=/usr" and "--with-cart=/usr" Regards, Steffen --------------------------------------------------------------------- Intel Corporation SAS (French simplified joint stock company) Registered headquarters: "Les Montalets"- 2, rue de Paris, 92196 Meudon Cedex, France Registration Number: 302 456 199 R.C.S. NANTERRE Capital: 4,572,000 Euros This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
More All Messages By This Member
anton.brekhov@... #1198 Thank you everyone for the help! It seems that there are no header files while installing with RPM. I had to clone repo, checkout tag 1.0.1 build source and after that all header files was found by IOR. [root@apache512 ior]# ./configure --with-daos=/root/daos/install --with-cart=/root/daos/install .... checking for d_hash_murmur64 in -lgurt... yes checking daos_types.h usability... yes checking daos_types.h presence... yes checking for daos_types.h... yes checking for uuid_generate in -luuid... yes checking for daos_sgl_init in -ldaos_common... yes checking for daos_init in -ldaos... yes checking for dfs_mkdir in -ldfs... yes checking for Gfarm file system... no checking that generated files are newer than configure... done configure: creating ./config.status config.status: creating Makefile config.status: creating src/Makefile config.status: creating src/test/Makefile config.status: creating contrib/Makefile config.status: creating doc/Makefile config.status: creating src/config.h
More All Messages By This Member
anton.brekhov@... #1199 I feel like I didn't get high performance through DFS driver: DFS: mpirun --allow-run-as-root -np 36 --bind-to socket ior -a DFS -b 1G -w -r -o /tmp/dfuse/testFile_dfs -t 128k -k --dfs.pool $dpool --dfs.svcl 0 --dfs.cont $dcont - -dfs.prefix /tmp/dfuse -G 123 -F .... Command line : ior -a DFS -b 1G -w -r -o /tmp/dfuse/testFile_dfs -t 128k -k --dfs.pool 3f0762ad-23b3-418a-a14d-4f813d122d46 --dfs.svcl 0 --dfs.cont 0e75d729-2164-493e-ba9e-2942df394bca --dfs.prefix /tmp/dfuse -G 123 -F .... access bw(MiB/s) IOPS Latency(s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) total(s) iter ------ --------- ---- ---------- ---------- --------- -------- -------- -------- -------- ---- write 524.01 4192 0.008581 1048576 128.00 0.015706 70.35 3.73 70.35 0 read 635.64 5085 0.007016 1048576 128.00 0.002732 57.99 2.78 57.99 0 POSIX: mpirun --allow-run-as-root -np 36 --bind-to socket ior -a POSIX -b 1G -w -r -o /tmp/dfuse/testFile_POSIX -t 128k -k -G 123 -F ... Command line : ior -a POSIX -b 1G -w -r -o /tmp/dfuse/testFile_POSIX -t 128k -k -G 123 -F ... access bw(MiB/s) IOPS Latency(s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) total(s) iter ------ --------- ---- ---------- ---------- --------- -------- -------- -------- -------- ---- write 587.98 4704 0.006442 1048576 128.00 0.285280 62.69 11.46 62.70 0 read 155.22 1243.86 0.028893 1048576 128.00 0.425217 237.09 5.65 237.49 0 Is it wrong? I believe that DFS must be much higher
More All Messages By This Member
Lombardi, Johann #1200 Anton, aren’t the header files in the -devel RPMs as mentioned by Steffen? Cheers, Johann From:* <daos@daos.groups.io> on behalf of "anton.brekhov@..." <anton.brekhov@...> Reply-To: "daos@daos.groups.io" <daos@daos.groups.io> Date: Monday 21 September 2020 at 21:06 To: "daos@daos.groups.io" <daos@daos.groups.io> Subject: Re: [daos] IOR test with dfs Thank you everyone for the help! It seems that there are no header files while installing with RPM. I had to clone repo, checkout tag 1.0.1 build source and after that all header files was found by IOR. [root@apache512 ior]# ./configure --with-daos=/root/daos/install --with-cart=/root/daos/install .... checking for d_hash_murmur64 in -lgurt... yes checking daos_types.h usability... yes checking daos_types.h presence... yes checking for daos_types.h... yes checking for uuid_generate in -luuid... yes checking for daos_sgl_init in -ldaos_common... yes checking for daos_init in -ldaos... yes checking for dfs_mkdir in -ldfs... yes checking for Gfarm file system... no checking that generated files are newer than configure... done configure: creating ./config.status config.status: creating Makefile config.status: creating src/Makefile config.status: creating src/test/Makefile config.status: creating contrib/Makefile config.status: creating doc/Makefile config.status: creating src/config.h --------------------------------------------------------------------- Intel Corporation SAS (French simplified joint stock company) Registered headquarters: "Les Montalets"- 2, rue de Paris, 92196 Meudon Cedex, France Registration Number: 302 456 199 R.C.S. NANTERRE Capital: 4,572,000 Euros This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
More All Messages By This Member
anton.brekhov@... #1201 At first I had some issues while installing daos-devel, cause I had newer boost-* packages from base repo. So i decided to built it from source. But then I've downgraded boost packages and installed daos-devel. All headers appeared! Thanks. Right now I'm testing performance on ior. And I feel like there is no big difference between POSIX and DFS... Maybe it's because a have only one client node or because of using only one IB interface. Any ideas?
More All Messages By This Member
Lombardi, Johann #1202 Hi Anton, Saw your results on the other email. A few comments: We indeed do expect DFS to perform better than dfuse. Could you please try to run some more tests with different tsize and process counts? It might also be great to increase the block size if possible since each process is only issuing 10x I/O requests. I recall from another email thread that you had 4x Optane PMEM DIMMs distributed across 2x sockets. We have never tested such a configuration (we usually have 6x per socket) and I am wondering how many targets you are using? Also, do you have any SSDs in the configuration? If so, could you please advise what type of SSD? It would be great if you could share your server yaml file and output of lscpu on the server node. Cheers, Johann From: <daos@daos.groups.io> on behalf of "anton.brekhov@..." <anton.brekhov@...> Reply-To: "daos@daos.groups.io" <daos@daos.groups.io> Date: Tuesday 22 September 2020 at 10:43 To: "daos@daos.groups.io" <daos@daos.groups.io> Subject: Re: [daos] IOR test with dfs At first I had some issues while installing daos-devel, cause I had newer boost-* packages from base repo. So i decided to built it from source. But then I've downgraded boost packages and installed daos-devel. All headers appeared! Thanks. Right now I'm testing performance on ior. And I feel like there is no big difference between POSIX and DFS... Maybe it's because a have only one client node or because of using only one IB interface. Any ideas? --------------------------------------------------------------------- Intel Corporation SAS (French simplified joint stock company) Registered headquarters: "Les Montalets"- 2, rue de Paris, 92196 Meudon Cedex, France Registration Number: 302 456 199 R.C.S. NANTERRE Capital: 4,572,000 Euros This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
More All Messages By This Member
anton.brekhov@... #1203 Johann thanks for help! Here is the information about server: [root@apache512 ior]# lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 112 On-line CPU(s) list: 0-111 Thread(s) per core: 2 Core(s) per socket: 28 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Platinum 8280 CPU @ 2.70GHz Stepping: 7 CPU MHz: 3699.975 CPU max MHz: 4000,0000 CPU min MHz: 1000,0000 BogoMIPS: 5400.00 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 1024K L3 cache: 39424K NUMA node0 CPU(s): 0-27,56-83 NUMA node1 CPU(s): 28-55,84-111 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 invpcid_single intel_pt ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke avx512_vnni md_clear spec_ctrl intel_stibp flush_l1d arch_capabilities I've set 4 target on one server. And 4 NVMe disk: NVME SSD INTEL SSDPELKX010T8 1TB. Here is my server.yml file: [root@apache512 ior]# cat ../daos_configs/.daos_server.active.yml port: 10001 transport_config: allow_insecure: true server_name: server client_cert_dir: .daos/clients ca_cert: .daos/daosCA.crt cert: .daos/daos_server.crt key: .daos/daos_server.key servers: - targets: 4 nr_xs_helpers: 0 first_core: 0 name: daos_server socket_dir: /var/run/daos_server log_mask: DEBUG log_file: /tmp/daos_server.log scm_mount: /mnt/daos scm_class: dcpm scm_list: - /dev/pmem0 bdev_class: nvme bdev_list: - 0000:b1:00.0 - 0000:b2:00.0 - 0000:b3:00.0 - 0000:b4:00.0 provider: ofi+verbs;ofi_rxm fabric_iface: ib0 fabric_iface_port: 31416 env_vars: - DAOS_MD_CAP=1024 - CRT_CTX_SHARE_ADDR=0 - CRT_TIMEOUT=30 - FI_SOCKETS_MAX_CONN_RETRY=1 - FI_SOCKETS_CONN_TIMEOUT=2000 nr_hugepages: 4096 set_hugepages: false control_log_mask: INFO control_log_file: /tmp/daos_control.log helper_log_file: "" recreate_superblocks: false name: daos_server socket_dir: /var/run/daos_server provider: ofi+verbs;ofi_rxm modules: "" access_points: - apache512:10001 fault_path: "" fault_cb: "" hyperthreads: false path: /root/daos_configs/local_SCM.yml
More All Messages By This Member
Murrell, Brian #1204 On Mon, 2020-09-21 at 17:34 +0000, Lombardi, Johann wrote: Thanks Mohamad & Steffen. Can we easily change the IOR build to assume “--with-daos” = “--with- daos=/usr --with-cart=/usr”? TBH, I would also just assume that CART and DAOS libs are located at the same place. Thoughts? Why not go further and not even require the use of --with-daos if daos is installed at the default location of /usr and just use it if it's there and don't use it if it's not. That sort of behavior is quite common and usually goes along with a --without-* switch for the case where it's present but not wanted in the build, which are usually the minority of the cases. Typically if somebody installs a feature, they want to use it. Cheers, b.
More All Messages By This Member
Murrell, Brian #1205 On Tue, 2020-09-22 at 01:43 -0700, anton.brekhov@... wrote: At first I had some issues while installing daos-devel, cause I had newer boost- packages from base repo. It would be great if you could paste the errors you got with the newer boost and what version of boost it was. I understand though if those messages are no longer available to you. But then I've downgraded boost packages and installed daos-devel. Do you know which versions of boost you had and then had to downgrade to? "yum history" should be able to tell you if you are not sure from memory. Cheers, b.
More All Messages By This Member
anton.brekhov@... #1207 Here is the screenshot. Some words in Russian, but packages written in English. I've solved the issue like this: yum downgrade boost-random boost-iostreams boost-system boost-thread boost boost-filesystem yum install daos-devel
More All Messages By This Member
Lombardi, Johann #1208 Could you please change “log_mask: DEBUG” to “log_mask: ERR”, restart the server and rerun ior? Cheers, Johann From: <daos@daos.groups.io> on behalf of "anton.brekhov@..." <anton.brekhov@...> Reply-To: "daos@daos.groups.io" <daos@daos.groups.io> Date: Tuesday 22 September 2020 at 13:36 To: "daos@daos.groups.io" <daos@daos.groups.io> Subject: Re: [daos] IOR test with dfs Johann thanks for help! Here is the information about server: [root@apache512 ior]# lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 112 On-line CPU(s) list: 0-111 Thread(s) per core: 2 Core(s) per socket: 28 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Platinum 8280 CPU @ 2.70GHz Stepping: 7 CPU MHz: 3699.975 CPU max MHz: 4000,0000 CPU min MHz: 1000,0000 BogoMIPS: 5400.00 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 1024K L3 cache: 39424K NUMA node0 CPU(s): 0-27,56-83 NUMA node1 CPU(s): 28-55,84-111 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 invpcid_single intel_pt ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke avx512_vnni md_clear spec_ctrl intel_stibp flush_l1d arch_capabilities I've set 4 target on one server. And 4 NVMe disk: NVME SSD INTEL SSDPELKX010T8 1TB. Here is my server.yml file: [root@apache512 ior]# cat ../daos_configs/.daos_server.active.yml port: 10001 transport_config: allow_insecure: true server_name: server client_cert_dir: .daos/clients ca_cert: .daos/daosCA.crt cert: .daos/daos_server.crt key: .daos/daos_server.key servers: - targets: 4 nr_xs_helpers: 0 first_core: 0 name: daos_server socket_dir: /var/run/daos_server log_mask: DEBUG log_file: /tmp/daos_server.log scm_mount: /mnt/daos scm_class: dcpm scm_list: - /dev/pmem0 bdev_class: nvme bdev_list: - 0000:b1:00.0 - 0000:b2:00.0 - 0000:b3:00.0 - 0000:b4:00.0 provider: ofi+verbs;ofi_rxm fabric_iface: ib0 fabric_iface_port: 31416 env_vars: - DAOS_MD_CAP=1024 - CRT_CTX_SHARE_ADDR=0 - CRT_TIMEOUT=30 - FI_SOCKETS_MAX_CONN_RETRY=1 - FI_SOCKETS_CONN_TIMEOUT=2000 nr_hugepages: 4096 set_hugepages: false control_log_mask: INFO control_log_file: /tmp/daos_control.log helper_log_file: "" recreate_superblocks: false name: daos_server socket_dir: /var/run/daos_server provider: ofi+verbs;ofi_rxm modules: "" access_points: - apache512:10001 fault_path: "" fault_cb: "" hyperthreads: false path: /root/daos_configs/local_SCM.yml --------------------------------------------------------------------- Intel Corporation SAS (French simplified joint stock company) Registered headquarters: "Les Montalets"- 2, rue de Paris, 92196 Meudon Cedex, France Registration Number: 302 456 199 R.C.S. NANTERRE Capital: 4,572,000 Euros This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
More All Messages By This Member
anton.brekhov@... #1210 Johann thank you so much! I've set log_mask to err and i had 10x better performance even on one socket. Here is my results: POSIX: mpirun --allow-run-as-root -np 18 --bind-to socket ior -a POSIX -b 1G -w -r -o /tmp/dfuse/testFile_POSIX -t 128k -k -G 123 access bw(MiB/s) IOPS Latency(s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) total(s) iter ------ --------- ---- ---------- ---------- --------- -------- -------- -------- -------- ---- write 352.75 2822.04 0.004846 1048576 128.00 1.94 52.25 32.15 52.25 0 read 796.54 8688 0.002055 1048576 128.00 6.17 16.97 0.162640 23.14 0 DFS: mpirun --allow-run-as-root -np 18 --bind-to socket ior -a DFS -b 1G -w -r -o /tmp/dfuse/testFile_dfs -t 128k -k --dfs.pool $dpool --dfs.svcl 0 --dfs.cont $dcont --dfs.prefix /tmp/dfuse -G 123 ... access bw(MiB/s) IOPS Latency(s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) total(s) iter ------ --------- ---- ---------- ---------- --------- -------- -------- -------- -------- ---- write 3307.90 26464 0.000654 1048576 128.00 0.038082 5.57 0.220024 5.57 0 read 3726 29811 0.000602 1048576 128.00 0.000218 4.95 0.208901 4.95 0 I'll try to change transfer size as you recommended. Hope it will perform much better! Thanks!
More All Messages By This Member

1 - 20 of 21

Previous Topic Next Topic

Home Hashtags Subgroups About Features Pricing Updates Terms Help Twitter