Re: FIO Results & Running IO500


JACKSON Adrian
 

Actually, it's been pointed out to me I was confusing engines and
targets. So ignore me. :)

On 29/06/2021 10:09, JACKSON Adrian wrote:
It would be sensible to increase the number of engines per node. For our
system, where we have 48 cores per node, we're running 12 engines per
socket, 24 per node. This night be too many, but I think 1 engine per
node is too few.

cheers

adrianj

On 29/06/2021 08:32, Peter wrote:
This email was sent to you by someone outside the University.
You should only click on links or attachments if you are certain that
the email is genuine and the content is safe.
Johann, my configuration is as follows:


4 nodes, 4 engines in total (1 per node)
2 x 128 GB Optane per socket, 2 x sockets per node (DAOS is currently
only using 1 socket per node)
(We also have 1x 1.5 TB NVMe drive / node, that we plan to eventually
configure DAOS to use)
Nodes are using Mellanox EDR Infiniband
Cent OS 7.9, have tried various MPI distributions.

The yaml file (was generated via the auto conf)

***********
port: 10001
transport_config:
allow_insecure: true
server_name: server
client_cert_dir: /etc/daos/certs/clients
ca_cert: /etc/daos/certs/daosCA.crt
cert: /etc/daos/certs/server.crt
key: /etc/daos/certs/server.key
servers: []
engines:
- targets: 16
nr_xs_helpers: 3
first_core: 0
name: daos_server
socket_dir: /var/run/daos_server
log_file: /tmp/daos_engine.0.log
scm_mount: /mnt/daos0
scm_class: dcpm
scm_list:
- /dev/pmem0
bdev_class: nvme
provider: ofi+verbs;ofi_rxm
fabric_iface: ib0
fabric_iface_port: 31416
pinned_numa_node: 0
disable_vfio: false
disable_vmd: true
nr_hugepages: 0
set_hugepages: false
control_log_mask: INFO
control_log_file: /tmp/daos_server.log
helper_log_file: ""
firmware_helper_log_file: ""
recreate_superblocks: false
fault_path: ""
name: daos_server
socket_dir: /var/run/daos_server
provider: ofi+verbs;ofi_rxm
modules: ""
access_points:
- 172.23.7.3:10001
fault_cb: ""
hyperthreads: false
path: ../etc/daos_server.yml
*******

And yes, I am running FIO from one of the nodes.

Is there anything you see that I should modify or investigate? Thank you
very much for the help!
--
Tel: +44 131 6506470 skype: remoteadrianj
The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Is e buidheann carthannais a th’ ann an Oilthigh Dhùn Èideann, clàraichte an Alba, àireamh clàraidh SC005336.




--
Tel: +44 131 6506470 skype: remoteadrianj

Join daos@daos.groups.io to automatically receive all group messages.