Re: Client application single value KV Put high latency using multiple threads (pthread)


Lombardi, Johann
 

Hi Ping,

 

Those latency numbers are indeed way higher than what we expect/see. Could you please advise what type of network and network provider you are using?
If you are on a recent master, could you please create a pool and give a try to “daos pool autotest --pool $PUUID”?

 

Cheers,

Johann

 

From: <daos@daos.groups.io> on behalf of "ping.wong via groups.io" <ping.wong@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Sunday 31 January 2021 at 22:16
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: [daos] Client application single value KV Put high latency using multiple threads (pthread)

 

Hi all,

To evaluate replication performance, I write a client application with multiple pthreads (schedule to run on different cores if possible) using daos_kv_put with async event API in a 2-servers cluster.

One server has 44 cores and the other server has 88 cores.
Client is running on a different node with 48 cores.  The leader server replicates to the replica server.  I notice that the leader role switches between the two servers.

To find out why the client has high latency, I added some timing counters to track the duration of KV Puts in the io servers (please refer to the third column of the table below).

The client test application calls daos_kv_put(oh, DAOS_TX_NONE, 0, key, buf_size, buf, &ev) with async event 
then calls daos_event_test(&ev, DAOS_EQ_WAIT, &ev_flag) for IO completion

Each pthread writes to different key for the same object for 1000 values (4K each); hence 1000 calls to daos_kv_put
Increasing number of threads in the client application, I observed a higher latency from the client's perspective (see table below)

In the 100 threads test case, io server has higher latency as well.  

Am I missing something critical?
Is the overhead caused by the daos library?
Is there a way to improve the application latency overhead?

Thanks
Ping


Number client threads | Number of daos_kv_put | daos_io_server average put duration | client average put duration |

-----------------------------------------------------------------------------------------------------------------------------------------------------------

        5                          |      1,000                         |           0.28 ms                                     |                   1.05 ms             |

-----------------------------------------------------------------------------------------------------------------------------------------------------------

        10                        |      1,000                         |           0.27 ms                                     |                   1.65 ms             |

-----------------------------------------------------------------------------------------------------------------------------------------------------------

        15                        |      1,000                         |           0.41 ms                                     |                   2.20 ms             |

-----------------------------------------------------------------------------------------------------------------------------------------------------------

        20                        |      1,000                         |           0.48 ms                                     |                   2.86 ms             |

-----------------------------------------------------------------------------------------------------------------------------------------------------------

        100                      |      1,000                         |           7.02 ms                                     |                 11.45 ms             |

-----------------------------------------------------------------------------------------------------------------------------------------------------------



 

 

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Join daos@daos.groups.io to automatically receive all group messages.