Hi there,

Could you please tell us more about your HW configuration and the test case (e.g. I/O size, read or write, object class, …)? We definitely see lower latency in our internal testing.

Cheers,

Johann

Hello everyone:
  Recently, I did some testing with daos array interface, found that after finishing nvme bio, it took me a long time for writing metadata and resource release ; The total latency of single channel IO is about 80us, and  vos_update_end taks about 30 us(about 17us in dkey akey update via SCM); I thought that IO via SCM should be very fast, but through the test, it's slower than the nvme IO. Is evtree operation takes too long or are there other reasons?

Best regards!

Hi there,

I don’t think that we have ever run any benchmarks with only 2x Optane DIMMs. Maybe you could start by running vos_perf and compare to the results we have in the online documentation: https://docs.daos.io/v2.0/admin/performance_tuning/#daos_perf-vos_perf

Cheers,

Johann

Thansks for reponseing. And can you share your latency in you internal testing?

Best Regards!

Hi,

A DAOS update operation actually results in several sequential latency-sensitive operations over SCM to locate the object/dkey/akey (and create the associated trees if those don’t exist already) that you want to update. Once this is resolved, the 4K data buffer will be stored in either SCM or NVMe. DAOS uses SCM internally to accelerate the sequential metadata operations.

Cheers,

Johann

Hi Johann:

Tansks for the information. I have checked the latency in the online documentation, about 12 us for update, much lower than that in my test. But I think SCM would be much faster while the 4K write latency is about 10us using NVMe. Does the SCM just resolve the WA problem?

Best Regards!

Hi, in the test, it writes 1K to DAOS server, the engine actually does:

1. Search the dkey in SCM
2. Create index for the dkey if it does not exist (b+tree stored in SCM)
3. Do the same for akey
4. Copy 1K data to SCM
5. All the above writes to SCM are in the same PMDK transaction which has its own cost.

This is the reason that VOS write latency is higher than one SCM write. Array write latency should roughly be the same as single value, we will do some benchmark and check if there is any issue.

Liang

Hi Johann：

So you mean that although the operation over SCM is fast， but several sequential operations over SCM may result in the high latency？
Here is the test in my server with vos_perf：


Using array type， the latency is much higher(wrose than my test). The type "array" here means to the array in DAOS interface? Is there something wrong?

Another question, DAOS write the data buffer before the SCM operation in VOS.  I think the leader would waite，until follower complete all the operation and reply and this result in a higher latency.  Is there some problem with my understanding?
I'm confuesd with the array update flow now.


Best Regards!

Jan