Hi,

A DAOS update operation actually results in several sequential latency-sensitive operations over SCM to locate the object/dkey/akey (and create the associated trees if those don’t exist already) that you want to update. Once this is resolved, the 4K data buffer will be stored in either SCM or NVMe. DAOS uses SCM internally to accelerate the sequential metadata operations.

Cheers,

Johann

Hi Johann:

Tansks for the information. I have checked the latency in the online documentation, about 12 us for update, much lower than that in my test. But I think SCM would be much faster while the 4K write latency is about 10us using NVMe. Does the SCM just resolve the WA problem?

Best Regards!

Hi there,

I don’t think that we have ever run any benchmarks with only 2x Optane DIMMs. Maybe you could start by running vos_perf and compare to the results we have in the online documentation: https://docs.daos.io/v2.0/admin/performance_tuning/#daos_perf-vos_perf

Cheers,

Johann

Thansks for reponseing. And can you share your latency in you internal testing?

Best Regards!

When I replied last week I forgot to mention, we do have a control API written in Go, which you could also use for this purpose: https://github.com/daos-stack/daos/tree/master/src/control/lib/control

Hopefully either that or calling dmg will be sufficient. 😊 Just avoid calling the gRPC methods directly, as we don’t guarantee those for external developers.

Kris

You should use dmg storage format. As things stand, we don’t enable users to call the administrative gRPC methods directly. The API is internal and may change.

Thanks,

Kris

Omer,

I’d be interested in seeing the stdout/stderr and logging from dfuse where this is happening.  As you say the -f option should make it run in the foreground so if you’re not passing this then it should background itself so you can get back a prompt.  Even when it does go into the background it’ll wait until the fuse mount is registered with the kernel before detaching the terminal so that it can report any error to the user so there is code in there to delay and then exit after forking so there is potential for an issue there although I’ve not known one before and we do have tests which use both methods.

Ashley.

Hi,

Thanks for your reply.

1. I saw :). Cool.
2. I used to build DAOS and install it, so I had a script that did that for me, but yes, with RPM based solution this shouldn't be a concern. I fixed my unit file to do what I suggested to overcome this.
3. I tried " dfuse --pool=pool0 --container=cont0 -m /mnt/dfuse" and after a while " dfuse -f --pool=pool0 --container=cont0 -m /mnt/dfuse". I'm guessing the 2nd version is expected to not return as the '-f' suggests, but I didn't get a command prompt back either way.

Omer

Hi there,

Thanks for reaching out.

1. The online doc has been fixed and now specifies a different URL for CentOS7 and 8.
2. /var/run/daos_agent should be automatically created after enabling/starting the daos_agent systemd service (i.e. “systemctl enable/start daos_agent”).
   We have some instructions in the documentation on how to start the agent manually (w/o systemd), see https://docs.daos.io/v2.0/admin/predeployment_check/#runtime-directory-setup

That being said, this should not be required if you use RPMs/systemd.

3. Could you please advise what option(s) you pass to the dfuse command line? Can you set “export D_LOG_MASK=debug” and then run dfuse with the -f option and see what it logs?

Cheers,

Johann

Some post DAOS 2.0 installation and comments and questions:

1. We use a mix of CentOS 7 and 8 in our setup. However, the DAOS online documentation at "https://docs.daos.io/v2.0/QSG/setup_centos/" states the following URLas the repository to use when installing DAOS:

"https://packages.daos.io/v2.0/CentOS7/packages/x86_64/daos_packages.repo".

Of course, this lead to some installation problems as the is a specific CentOS 7 URL. Fixing the repository URL solved this problem, but only after some confusion about what going on. This probably needs some documentation clarification.

2. The DAOS agent creates a socket in /var/run/daos_agent. This isn't a persistent directory and the service doesn't create it, and it needs to be created on every boot for the agent to work properly.
a. This isn't mentioned in the documentation.
b. A systemd unit addition like "ExecStartPre=mkdir -p /var/run/daos_agent" is probably a good idea

3. Not sure why, but mounting a POSIX container using dfuse doesn't return for me. The process doesn't return. During that time, the mount is working (or so it seems), as a `mount` command does list a dfuse mount in the relevant directory, but the dfuse command didn't return to the CLI prompt.

Hi there,

Could you please tell us more about your HW configuration and the test case (e.g. I/O size, read or write, object class, …)? We definitely see lower latency in our internal testing.

Cheers,

Johann

Hello everyone:
  Recently, I did some testing with daos array interface, found that after finishing nvme bio, it took me a long time for writing metadata and resource release ; The total latency of single channel IO is about 80us, and  vos_update_end taks about 30 us(about 17us in dkey akey update via SCM); I thought that IO via SCM should be very fast, but through the test, it's slower than the nvme IO. Is evtree operation takes too long or are there other reasons?

Best regards!

Hi,

In your example, the ULT will be tracked in the wait list from “ABT_future” on waiting, once it’s waked up, it’ll be pushed back to a runnable ULT FIFO list (per ABT pool, maintained by Argobots internally), and being executed once server scheduler picked it again.

Thanks

-Niu

I’m not sure why the libraries aren’t being linked/loaded – maybe someone more knowledgeable on the DAOS team has an idea. The make command does include the libraries, and the libraries are available, so I wouldn’t think there should be an issue.

~Dalton Bohning

Will you try appending ${MY_DAOS_INSTALL_PATH}/lib64 to your LD_LIBRARY_PATH and re-run the “make” command?

export LD_LIBRARY_PATH=${MY_DAOS_INSTALL_PATH}/lib64:$LD_LIBRARY_PATH

~Dalton Bohning