Topics

IOR test with dfs


anton.brekhov@...
 

[root@sky08 ~]# mpirun --allow-run-as-root  -np 1  ior -a=DFS -b 1G -w -r  -o /mnt/dfuse/testFile_dfs -t 128k -k --dfs.pool $dpool --dfs.svcl 0 --dfs.cont $dc

ont --dfs.prefix /tmp/dfuse/ -G 123

Error invalid argument: --dfs.pool

Error invalid argument: 3f0762ad-23b3-418a-a14d-4f813d122d46

Error invalid argument: --dfs.svcl

Error invalid argument: 0

Error invalid argument: --dfs.cont

Error invalid argument: 0e75d729-2164-493e-ba9e-2942df394bca

Error invalid argument: --dfs.prefix

Error invalid argument: /tmp/dfuse/

Error invalid argument: 123

Invalid options

Synopsis ior

[root@sky08 ~]# mpirun --allow-run-as-root  -np 1  ior -a=DFS -b 1G -w -r  -o /mnt/dfuse/testFile_dfs -t 128k -k --dfs.pool $dpool --dfs.svcl 0 --dfs.cont $dcont --dfs.prefix /tmp/dfuse/ -G 123

Error invalid argument: --dfs.pool

Error invalid argument: 3f0762ad-23b3-418a-a14d-4f813d122d46

Error invalid argument: --dfs.svcl

Error invalid argument: 0

Error invalid argument: --dfs.cont

Error invalid argument: 0e75d729-2164-493e-ba9e-2942df394bca

Error invalid argument: --dfs.prefix

Error invalid argument: /tmp/dfuse/

Error invalid argument: 123

Invalid options

Synopsis ior

How I need to compile ior to make this work?

Thanks!


Lombardi, Johann
 

Hey Anton,

 

I assume that you have built ior with the DFS backend support as specified in https://github.com/hpc/ior/blob/master/README_DAOS, right?

 

Although I think it should work, I have never tried “-a=DFS” and always used “-a DFS” on my side. Could you please give this a try just to double check?

The prefix is also relative to the absolute file path, so you should be using --dfs.prefix /mnt/dfuse instead of /tmp/dfuse. I assume that $dpool and $dcont match respectively the pool and container UUIDs that you used to mount with dfuse, right? Also, svcl should match what was returned at pool creation time, you got 0 and not 1, right?

 

To summarize, could you please try:

ior -a DFS -t 128k -b 1G -r -w -o /mnt/dfuse/testFile_dfs -G 123 -k --dfs.pool $dpool --dfs.svcl 0 --dfs.cont $dcont --dfs.prefix /mnt/dfuse

 

Cheers,

Johann

 

From: <daos@daos.groups.io> on behalf of "anton.brekhov@..." <anton.brekhov@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Sunday 20 September 2020 at 21:01
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: [daos] IOR test with dfs

 

[root@sky08 ~]# mpirun --allow-run-as-root  -np 1  ior -a=DFS -b 1G -w -r  -o /mnt/dfuse/testFile_dfs -t 128k -k --dfs.pool $dpool --dfs.svcl 0 --dfs.cont $dc

ont --dfs.prefix /tmp/dfuse/ -G 123

Error invalid argument: --dfs.pool

Error invalid argument: 3f0762ad-23b3-418a-a14d-4f813d122d46

Error invalid argument: --dfs.svcl

Error invalid argument: 0

Error invalid argument: --dfs.cont

Error invalid argument: 0e75d729-2164-493e-ba9e-2942df394bca

Error invalid argument: --dfs.prefix

Error invalid argument: /tmp/dfuse/

Error invalid argument: 123

Invalid options

Synopsis ior

[root@sky08 ~]# mpirun --allow-run-as-root  -np 1  ior -a=DFS -b 1G -w -r  -o /mnt/dfuse/testFile_dfs -t 128k -k --dfs.pool $dpool --dfs.svcl 0 --dfs.cont $dcont --dfs.prefix /tmp/dfuse/ -G 123

Error invalid argument: --dfs.pool

Error invalid argument: 3f0762ad-23b3-418a-a14d-4f813d122d46

Error invalid argument: --dfs.svcl

Error invalid argument: 0

Error invalid argument: --dfs.cont

Error invalid argument: 0e75d729-2164-493e-ba9e-2942df394bca

Error invalid argument: --dfs.prefix

Error invalid argument: /tmp/dfuse/

Error invalid argument: 123

Invalid options

Synopsis ior

How I need to compile ior to make this work?

Thanks!

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.


anton.brekhov@...
 

Johann thanks!

I installed DAOS as rpm v1.0.1. Could you please tell where is install folder? I suppose that building with my parameters didn't give right binary:

[root@sky08 ior]# ./configure --with-daos=/usr/ --with-cart
.....

checking for d_hash_murmur64 in -lgurt... no

checking daos_types.h usability... no

checking daos_types.h presence... no

checking for daos_types.h... no

checking for uuid_generate in -luuid... no

checking for daos_sgl_init in -ldaos_common... yes

checking for daos_init in -ldaos... no

checking for dfs_mkdir in -ldfs... yes

checking for Gfarm file system... no

 

checking that generated files are newer than configure... done

It seems like I need source code, but there's no tag for 1.0.1.
Thanks!

 
 
 
 
 


anton.brekhov@...
 

I just found tag in source directory and run ior building like this:
[root@apache512 ior]# ./configure --with-daos=/root/daos/install --with-cart
...

checking for d_hash_murmur64 in -lgurt... no

checking daos_types.h usability... yes

checking daos_types.h presence... yes

checking for daos_types.h... yes

checking for uuid_generate in -luuid... yes

checking for daos_sgl_init in -ldaos_common... yes

checking for daos_init in -ldaos... yes

checking for dfs_mkdir in -ldfs... yes

checking for Gfarm file system... no

checking that generated files are newer than configure... done

configure: creating ./config.status

config.status: creating Makefile

config.status: creating src/Makefile

config.status: creating src/test/Makefile

config.status: creating contrib/Makefile

config.status: creating doc/Makefile

config.status: creating src/config.h

config.status: src/config.h is unchanged

 

config.status: executing depfiles commands

[root @ apache512 ior] # make

Making all in src

make all-recursive

Making all in.

   CCLD ior

/ usr / bin / ld: ior-aiori-DFS.o: undefined symbol reference "d_hash_table_create"

/root/daos/install/lib64/libgurt.so.4: error adding symbols: DSO missing from command line

collect2: error: ld execution exited with return code 1

make [3]: *** [ior] Error 1

make [2]: *** [all-recursive] Error 1

make [1]: *** [all] Error 2

make: *** [all-recursive] Error 1

 


Lombardi, Johann
 

I was hoping that --with-daos would be enough if everything is installed at the default location via RPMs.

@Chaarawi, Mohamad any thoughts?

 

Cheers,

johann

 

From: <daos@daos.groups.io> on behalf of "anton.brekhov@..." <anton.brekhov@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Monday 21 September 2020 at 12:44
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [daos] IOR test with dfs

 

Johann thanks!

I installed DAOS as rpm v1.0.1. Could you please tell where is install folder? I suppose that building with my parameters didn't give right binary:

[root@sky08 ior]# ./configure --with-daos=/usr/ --with-cart
.....

checking for d_hash_murmur64 in -lgurt... no

checking daos_types.h usability... no

checking daos_types.h presence... no

checking for daos_types.h... no

checking for uuid_generate in -luuid... no

checking for daos_sgl_init in -ldaos_common... yes

checking for daos_init in -ldaos... no

checking for dfs_mkdir in -ldfs... yes

checking for Gfarm file system... no

 

checking that generated files are newer than configure... done

It seems like I need source code, but there's no tag for 1.0.1.
Thanks!

 

 

 

 

 

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.


Steffen Christgau
 

Hi Anton,

On 9/21/20 12:44 PM, anton.brekhov@... wrote:
*I installed DAOS as rpm v1.0.1.
Did you "only" install the daos package? IMO it does not contain the DAOS header files. Try to install daos-devel.

Regards, Steffen


Steffen Christgau
 

Dear Johann,

On 9/21/20 2:00 PM, Lombardi, Johann wrote:
I was hoping that --with-daoswould be enough if everything is installed at the default location via RPMs.
Maybe unrelated, but for getting MPICH 3.4a with DAOS ROMIO to compile on a DAOS installation I had to specify both "--with-daos=/usr" and "--with-cart=/usr"

Regards, Steffen


Chaarawi, Mohamad
 

Yes indeed as Steffen mentioned you need the headers also.
Also --with-daos and --with-cart should point to the same location (note that in 1.0, cart is a stand-alone lib, and that's why we had to have 2 paths).
(i.e. you need to pass a path to --with-cart=/path/)

Thanks,
Mohamad

On 9/21/20, 7:16 AM, "daos@daos.groups.io on behalf of Steffen Christgau" <daos@daos.groups.io on behalf of christgau@...> wrote:

Dear Johann,

On 9/21/20 2:00 PM, Lombardi, Johann wrote:
> I was hoping that --with-daoswould be enough if everything is installed
> at the default location via RPMs.

Maybe unrelated, but for getting MPICH 3.4a with DAOS ROMIO to compile
on a DAOS installation I had to specify both "--with-daos=/usr" and
"--with-cart=/usr"

Regards, Steffen


Lombardi, Johann
 

Thanks Mohamad & Steffen.

Can we easily change the IOR build to assume “--with-daos” = “--with-daos=/usr --with-cart=/usr”? TBH, I would also just assume that CART and DAOS libs are located at the same place. Thoughts?

 

Cheers,

Johann

 

From: <daos@daos.groups.io> on behalf of "Chaarawi, Mohamad" <mohamad.chaarawi@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Monday 21 September 2020 at 15:18
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [daos] IOR test with dfs

 

Yes indeed as Steffen mentioned you need the headers also.

Also --with-daos and --with-cart should point to the same location (note that in 1.0, cart is a stand-alone lib, and that's why we had to have 2 paths).

(i.e. you need to pass a path to --with-cart=/path/)

 

Thanks,

Mohamad

 

On 9/21/20, 7:16 AM, "daos@daos.groups.io on behalf of Steffen Christgau" <daos@daos.groups.io on behalf of christgau@...> wrote:

 

    Dear Johann,

 

    On 9/21/20 2:00 PM, Lombardi, Johann wrote:

    > I was hoping that --with-daoswould be enough if everything is installed

    > at the default location via RPMs.

 

    Maybe unrelated, but for getting MPICH 3.4a with DAOS ROMIO to compile

    on a DAOS installation I had to specify both "--with-daos=/usr" and

    "--with-cart=/usr"

 

    Regards, Steffen

 

 

 

 

 

 

 

 

 

 

 

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.


anton.brekhov@...
 

Thank you everyone for the help!
It seems that there are no header files while installing with RPM. I had to clone repo, checkout tag 1.0.1 build source and after that all header files was found by IOR. 
[root@apache512 ior]# ./configure --with-daos=/root/daos/install --with-cart=/root/daos/install
....

checking for d_hash_murmur64 in -lgurt... yes

checking daos_types.h usability... yes

checking daos_types.h presence... yes

checking for daos_types.h... yes

checking for uuid_generate in -luuid... yes

checking for daos_sgl_init in -ldaos_common... yes

checking for daos_init in -ldaos... yes

checking for dfs_mkdir in -ldfs... yes

checking for Gfarm file system... no

checking that generated files are newer than configure... done

configure: creating ./config.status

config.status: creating Makefile

config.status: creating src/Makefile

config.status: creating src/test/Makefile

config.status: creating contrib/Makefile

config.status: creating doc/Makefile

 

config.status: creating src/config.h


 
 
 
 


anton.brekhov@...
 

I feel like I didn't get high performance through DFS driver:
 
DFS: mpirun --allow-run-as-root -np 36 --bind-to socket ior -a DFS -b 1G -w -r  -o /tmp/dfuse/testFile_dfs -t 128k -k --dfs.pool $dpool --dfs.svcl 0 --dfs.cont $dcont -
-dfs.prefix /tmp/dfuse -G 123 -F
....
Command line        : ior -a DFS -b 1G -w -r -o /tmp/dfuse/testFile_dfs -t 128k -k --dfs.pool 3f0762ad-23b3-418a-a14d-4f813d122d46 --dfs.svcl 0 --dfs.cont 0e75d729-2164-493e-ba9e-2942df394bca --dfs.prefix /tmp/dfuse -G 123 -F
....
access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
write     524.01     4192       0.008581    1048576    128.00     0.015706   70.35      3.73       70.35      0
read      635.64     5085       0.007016    1048576    128.00     0.002732   57.99      2.78       57.99      0

POSIX: mpirun --allow-run-as-root -np 36 --bind-to socket ior -a POSIX -b 1G -w -r  -o /tmp/dfuse/testFile_POSIX -t 128k -k -G 123 -F
...
Command line        : ior -a POSIX -b 1G -w -r -o /tmp/dfuse/testFile_POSIX -t 128k -k -G 123 -F
...
access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
write     587.98     4704       0.006442    1048576    128.00     0.285280   62.69      11.46      62.70      0
read      155.22     1243.86    0.028893    1048576    128.00     0.425217   237.09     5.65       237.49     0

Is it wrong? I believe that DFS must be much higher


Lombardi, Johann
 

Anton, aren’t the header files in the *-devel RPMs as mentioned by Steffen?

 

Cheers,

Johann

 

From: <daos@daos.groups.io> on behalf of "anton.brekhov@..." <anton.brekhov@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Monday 21 September 2020 at 21:06
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [daos] IOR test with dfs

 

Thank you everyone for the help!
It seems that there are no header files while installing with RPM. I had to clone repo, checkout tag 1.0.1 build source and after that all header files was found by IOR. 
[root@apache512 ior]# ./configure --with-daos=/root/daos/install --with-cart=/root/daos/install
....

checking for d_hash_murmur64 in -lgurt... yes

checking daos_types.h usability... yes

checking daos_types.h presence... yes

checking for daos_types.h... yes

checking for uuid_generate in -luuid... yes

checking for daos_sgl_init in -ldaos_common... yes

checking for daos_init in -ldaos... yes

checking for dfs_mkdir in -ldfs... yes

checking for Gfarm file system... no

checking that generated files are newer than configure... done

configure: creating ./config.status

config.status: creating Makefile

config.status: creating src/Makefile

config.status: creating src/test/Makefile

config.status: creating contrib/Makefile

config.status: creating doc/Makefile

 

config.status: creating src/config.h

 

 

 

 

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.


anton.brekhov@...
 

At first I had some issues while installing daos-devel, cause I had newer boost-* packages from base repo. So i decided to built it from source. But then I've downgraded boost packages and installed daos-devel. All headers appeared! Thanks. 

Right now I'm testing performance on ior. And I feel like there is no big difference between POSIX and DFS... Maybe it's because a have only one client node or because of using only one IB interface.

Any ideas?


Lombardi, Johann
 

Hi Anton,

 

Saw your results on the other email. A few comments:

  1. We indeed do expect DFS to perform better than dfuse. Could you please try to run some more tests with different tsize and process counts?  It might also be great to increase the block size if possible since each process is only issuing 10x I/O requests.
  2. I recall from another email thread that you had 4x Optane PMEM DIMMs distributed across 2x sockets. We have never tested such a configuration (we usually have 6x per socket) and I am wondering how many targets you are using? Also, do you have any SSDs in the configuration? If so, could you please advise what type of SSD?
  3. It would be great if you could share your server yaml file and output of lscpu on the server node.

 

Cheers,

Johann

 

From: <daos@daos.groups.io> on behalf of "anton.brekhov@..." <anton.brekhov@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Tuesday 22 September 2020 at 10:43
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [daos] IOR test with dfs

 

At first I had some issues while installing daos-devel, cause I had newer boost-* packages from base repo. So i decided to built it from source. But then I've downgraded boost packages and installed daos-devel. All headers appeared! Thanks. 

Right now I'm testing performance on ior. And I feel like there is no big difference between POSIX and DFS... Maybe it's because a have only one client node or because of using only one IB interface.

Any ideas?

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.


anton.brekhov@...
 

Johann thanks for help! Here is the information about server:

[root@apache512 ior]# lscpu

Architecture:          x86_64

CPU op-mode(s):        32-bit, 64-bit

Byte Order:            Little Endian

CPU(s):                112

On-line CPU(s) list:   0-111

Thread(s) per core:    2

Core(s) per socket:    28

Socket(s):             2

NUMA node(s):          2

Vendor ID:             GenuineIntel

CPU family:            6

Model:                 85

Model name:            Intel(R) Xeon(R) Platinum 8280 CPU @ 2.70GHz

Stepping:              7

CPU MHz:               3699.975

CPU max MHz:           4000,0000

CPU min MHz:           1000,0000

BogoMIPS:              5400.00

Virtualization:        VT-x

L1d cache:             32K

L1i cache:             32K

L2 cache:              1024K

L3 cache:              39424K

NUMA node0 CPU(s):     0-27,56-83

NUMA node1 CPU(s):     28-55,84-111

 

Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 invpcid_single intel_pt ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke avx512_vnni md_clear spec_ctrl intel_stibp flush_l1d arch_capabilities


I've set 4 target on one server. And 4 NVMe diskNVME SSD INTEL SSDPELKX010T8 1TB. Here is my server.yml file:

[root@apache512 ior]# cat ../daos_configs/.daos_server.active.yml

port: 10001

transport_config:

  allow_insecure: true

  server_name: server

  client_cert_dir: .daos/clients

  ca_cert: .daos/daosCA.crt

  cert: .daos/daos_server.crt

  key: .daos/daos_server.key

servers:

- targets: 4

  nr_xs_helpers: 0

  first_core: 0

  name: daos_server

  socket_dir: /var/run/daos_server

  log_mask: DEBUG

  log_file: /tmp/daos_server.log

  scm_mount: /mnt/daos

  scm_class: dcpm

  scm_list:

  - /dev/pmem0

  bdev_class: nvme

  bdev_list:

  - 0000:b1:00.0

  - 0000:b2:00.0

  - 0000:b3:00.0

  - 0000:b4:00.0

  provider: ofi+verbs;ofi_rxm

  fabric_iface: ib0

  fabric_iface_port: 31416

  env_vars:

  - DAOS_MD_CAP=1024

  - CRT_CTX_SHARE_ADDR=0

  - CRT_TIMEOUT=30

  - FI_SOCKETS_MAX_CONN_RETRY=1

  - FI_SOCKETS_CONN_TIMEOUT=2000

nr_hugepages: 4096

set_hugepages: false

control_log_mask: INFO

control_log_file: /tmp/daos_control.log

helper_log_file: ""

recreate_superblocks: false

name: daos_server

socket_dir: /var/run/daos_server

provider: ofi+verbs;ofi_rxm

modules: ""

access_points:

- apache512:10001

fault_path: ""

fault_cb: ""

hyperthreads: false

path: /root/daos_configs/local_SCM.yml



Murrell, Brian
 

On Mon, 2020-09-21 at 17:34 +0000, Lombardi, Johann wrote:
Thanks Mohamad & Steffen.
Can we easily change the IOR build to assume “--with-daos” = “--with-
daos=/usr --with-cart=/usr”? TBH, I would also just assume that CART
and DAOS libs are located at the same place. Thoughts?
Why not go further and not even require the use of --with-daos if daos
is installed at the default location of /usr and just use it if it's
there and don't use it if it's not. That sort of behavior is quite
common and usually goes along with a --without-* switch for the case
where it's present but not wanted in the build, which are usually the
minority of the cases. Typically if somebody installs a feature, they
want to use it.

Cheers,
b.


Murrell, Brian
 

On Tue, 2020-09-22 at 01:43 -0700, anton.brekhov@... wrote:
*At first I had some issues while installing daos-devel, cause I had
newer boost-* packages from base repo.
It would be great if you could paste the errors you got with the newer
boost and what version of boost it was. I understand though if those
messages are no longer available to you.

But then I've downgraded boost packages and installed daos-devel.
Do you know which versions of boost you had and then had to downgrade
to? "yum history" should be able to tell you if you are not sure from
memory.

Cheers,
b.


anton.brekhov@...
 

Here is the screenshot. Some words in Russian, but packages written in English. 



I've solved the issue like this:

yum downgrade boost-random boost-iostreams boost-system boost-thread boost boost-filesystem

yum install daos-devel

 

 
 
 
 
 


Lombardi, Johann
 

Could you please change “log_mask: DEBUG” to “log_mask: ERR”, restart the server and rerun ior?

 

Cheers,

Johann

 

From: <daos@daos.groups.io> on behalf of "anton.brekhov@..." <anton.brekhov@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Tuesday 22 September 2020 at 13:36
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [daos] IOR test with dfs

 

Johann thanks for help! Here is the information about server:

[root@apache512 ior]# lscpu

Architecture:          x86_64

CPU op-mode(s):        32-bit, 64-bit

Byte Order:            Little Endian

CPU(s):                112

On-line CPU(s) list:   0-111

Thread(s) per core:    2

Core(s) per socket:    28

Socket(s):             2

NUMA node(s):          2

Vendor ID:             GenuineIntel

CPU family:            6

Model:                 85

Model name:            Intel(R) Xeon(R) Platinum 8280 CPU @ 2.70GHz

Stepping:              7

CPU MHz:               3699.975

CPU max MHz:           4000,0000

CPU min MHz:           1000,0000

BogoMIPS:              5400.00

Virtualization:        VT-x

L1d cache:             32K

L1i cache:             32K

L2 cache:              1024K

L3 cache:              39424K

NUMA node0 CPU(s):     0-27,56-83

NUMA node1 CPU(s):     28-55,84-111

 

Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 invpcid_single intel_pt ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke avx512_vnni md_clear spec_ctrl intel_stibp flush_l1d arch_capabilities


I've set 4 target on one server. And 4 NVMe diskNVME SSD INTEL SSDPELKX010T8 1TB. Here is my server.yml file:

[root@apache512 ior]# cat ../daos_configs/.daos_server.active.yml

port: 10001

transport_config:

  allow_insecure: true

  server_name: server

  client_cert_dir: .daos/clients

  ca_cert: .daos/daosCA.crt

  cert: .daos/daos_server.crt

  key: .daos/daos_server.key

servers:

- targets: 4

  nr_xs_helpers: 0

  first_core: 0

  name: daos_server

  socket_dir: /var/run/daos_server

  log_mask: DEBUG

  log_file: /tmp/daos_server.log

  scm_mount: /mnt/daos

  scm_class: dcpm

  scm_list:

  - /dev/pmem0

  bdev_class: nvme

  bdev_list:

  - 0000:b1:00.0

  - 0000:b2:00.0

  - 0000:b3:00.0

  - 0000:b4:00.0

  provider: ofi+verbs;ofi_rxm

  fabric_iface: ib0

  fabric_iface_port: 31416

  env_vars:

  - DAOS_MD_CAP=1024

  - CRT_CTX_SHARE_ADDR=0

  - CRT_TIMEOUT=30

  - FI_SOCKETS_MAX_CONN_RETRY=1

  - FI_SOCKETS_CONN_TIMEOUT=2000

nr_hugepages: 4096

set_hugepages: false

control_log_mask: INFO

control_log_file: /tmp/daos_control.log

helper_log_file: ""

recreate_superblocks: false

name: daos_server

socket_dir: /var/run/daos_server

provider: ofi+verbs;ofi_rxm

modules: ""

access_points:

- apache512:10001

fault_path: ""

fault_cb: ""

hyperthreads: false

path: /root/daos_configs/local_SCM.yml

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.


anton.brekhov@...
 

Johann thank you so much! I've set log_mask to err and i had 10x better performance even on one socket. Here is my results:
POSIX:
mpirun --allow-run-as-root -np 18 --bind-to socket ior -a POSIX -b 1G -w -r  -o /tmp/dfuse/testFile_POSIX -t 128k -k -G 123
access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
write     352.75     2822.04    0.004846    1048576    128.00     1.94       52.25      32.15      52.25      0
read      796.54     8688       0.002055    1048576    128.00     6.17       16.97      0.162640   23.14      0

DFS:
mpirun --allow-run-as-root -np 18 --bind-to socket ior -a DFS -b 1G -w -r  -o /tmp/dfuse/testFile_dfs -t 128k -k --dfs.pool
 $dpool --dfs.svcl 0 --dfs.cont $dcont --dfs.prefix /tmp/dfuse -G 123
...
access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
write     3307.90    26464      0.000654    1048576    128.00     0.038082   5.57       0.220024   5.57       0
read      3726       29811      0.000602    1048576    128.00     0.000218   4.95       0.208901   4.95       0

I'll try to change transfer size as you recommended. Hope it will perform much better!
Thanks!