Client application single value KV Put high latency using multiple threads (pthread)


ping.wong@...
 

Hi all,

To evaluate replication performance, I write a client application with multiple pthreads (schedule to run on different cores if possible) using daos_kv_put with async event API in a 2-servers cluster.

One server has 44 cores and the other server has 88 cores.
Client is running on a different node with 48 cores.  The leader server replicates to the replica server.  I notice that the leader role switches between the two servers.

To find out why the client has high latency, I added some timing counters to track the duration of KV Puts in the io servers (please refer to the third column of the table below).

The client test application calls daos_kv_put(oh, DAOS_TX_NONE, 0, key, buf_size, buf, &ev) with async event 
then calls daos_event_test(&ev, DAOS_EQ_WAIT, &ev_flag) for IO completion

Each pthread writes to different key for the same object for 1000 values (4K each); hence 1000 calls to daos_kv_put
Increasing number of threads in the client application, I observed a higher latency from the client's perspective (see table below)

In the 100 threads test case, io server has higher latency as well.  

Am I missing something critical?
Is the overhead caused by the daos library?
Is there a way to improve the application latency overhead?

Thanks
Ping

Number client threads | Number of daos_kv_put | daos_io_server average put duration | client average put duration |
-----------------------------------------------------------------------------------------------------------------------------------------------------------
        5                          |      1,000                         |           0.28 ms                                     |                   1.05 ms             |
-----------------------------------------------------------------------------------------------------------------------------------------------------------
        10                        |      1,000                         |           0.27 ms                                     |                   1.65 ms             |
-----------------------------------------------------------------------------------------------------------------------------------------------------------
        15                        |      1,000                         |           0.41 ms                                     |                   2.20 ms             |
-----------------------------------------------------------------------------------------------------------------------------------------------------------
        20                        |      1,000                         |           0.48 ms                                     |                   2.86 ms             |
-----------------------------------------------------------------------------------------------------------------------------------------------------------
        100                      |      1,000                         |           7.02 ms                                     |                 11.45 ms             |
-----------------------------------------------------------------------------------------------------------------------------------------------------------





 

Join daos@daos.groups.io to automatically receive all group messages.