daos@daos.groups.io | Questions on DTX and KV Puts values visibility

ping.wong@...

#1365

Hi All,

In a two nodes cluster (one master/leader and one replica), I wrote a client application with multiple writer and reader threads against same object with different keys and values:

1. Writers issue daos_kv_put(oh, DAOS_TX_NONE, ..., &ev); call daos_event_test(&ev, DAOS_EQ_WAIT, &ev_flag); // wait for IO completion

2. Readers issue daos_kv_get(oh, DAOS_TX_NONE, 0, key, &size, buf, &ev); call daos_event_test(&ev, DAOS_EQ_WAIT, &ev_flag); // wait for IO completion

I observed that daos_kv_get returns older version occasionally. I have a few questions regarding DTX and the visibility of values obtained from the client's perspective.

1. On the master/leader server, when it sends RPC reply to the client for the daos_kv_put();

1.1 Has the put been committed before returning to the client? If not, what is the typical asynchronous commit threshold to make the put values visible to other clients?
1.2 Under what condition(s), does the leader commit the transaction?
1.3 Does the leader send RPC to replica to commit the transaction synchronously or asynchronously after replying to the client?

2. On the master server, how long does it wait (timeout) for the replica to reply?
2.1 Does the master server send RPC to replica if the leader does not get reply from replica (e.g. when replica dies)?
2.2 Under what condition(s) does the master server commit synchronously?

The following is a partial output from the client application:

ReadThreadFunc thread 2 key=keyabcde00000006 buf=Value00000001

ReadThreadFunc thread 0 key=keyabcde00000004 buf=Value00000001

ReadThreadFunc thread 1 key=keyabcde00000005 buf=Value00000005

WriteThreadFunc thread 0 put key=keyabcde00000004 buf=Value00000003 Put older version <--------+

WriteThreadFunc thread 1 put key=keyabcde00000006 buf=Value00000003 |

WriteThreadFunc thread 2 put key=keyabcde00000007 buf=Value00000004 |

ReadThreadFunc thread 2 key=keyabcde00000006 buf=Value00000002 |

ReadThreadFunc thread 0 key=keyabcde00000004 buf=Value00000002 |

ReadThreadFunc thread 1 key=keyabcde00000005 buf=Value00000005 |

WriteThreadFunc thread 0 put key=keyabcde00000004 buf=Value00000004 Put newer version |

WriteThreadFunc thread 1 put key=keyabcde00000006 buf=Value00000004 |

WriteThreadFunc thread 2 put key=keyabcde00000007 buf=Value00000005 |

ReadThreadFunc thread 2 key=keyabcde00000006 buf=Value00000003 |

ReadThreadFunc thread 0 key=keyabcde00000004 buf=Value00000003 Get older version <--------+

ReadThreadFunc thread 0 key=keyabcde00000004 buf=Value00000003

...

WriteThreadFunc thread 2 put key=keyabcde00000003 buf=Value00000005 Put older version <---------------------------+

Event Write thread 0 started buf_size=4096 start=1 end=5 |

WriteThreadFunc thread 1 put key=keyabcde00000002 buf=Value00000004 |

Event Read thread 2 started buf_size=4096 start=3 end=7 |

WriteThreadFunc thread 2 put key=keyabcde00000004 buf=Value00000001 |

Event Read thread 1 started buf_size=4096 start=2 end=6 |

WriteThreadFunc thread 1 put key=keyabcde00000002 buf=Value00000005 |

WriteThreadFunc thread 1 put key=keyabcde00000003 buf=Value00000001 Put older version <----+ Put newer version |

ReadThreadFunc thread 2 key=keyabcde00000003 buf=Value00000005 | Get older version <-------------+

ReadThreadFunc thread 2 key=keyabcde00000003 buf=Value00000005 |

ReadThreadFunc thread 0 key=keyabcde00000001 buf=Value00000001 |

ReadThreadFunc thread 1 key=keyabcde00000002 buf=Value00000005 |

WriteThreadFunc thread 0 put key=keyabcde00000001 buf=Value00000002 |

WriteThreadFunc thread 2 put key=keyabcde00000004 buf=Value00000003 |

WriteThreadFunc thread 1 put key=keyabcde00000003 buf=Value00000002 Put newer version |

ReadThreadFunc thread 2 key=keyabcde00000003 buf=Value00000001 Get older version ----+

Thanks

Ping

Yong, Fan

#1366

Hi Ping,

Does “ReadThreadFunc thread 0” is the same thread as “WriteThreadFunc thread 0”? Or they are two different threads? Is there any concurrency control among the read threads and write threads? Or all the threads run in parallel without any control?

More inline comments.

--

Cheers,

Nasf

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of ping.wong via groups.io
Sent: Tuesday, February 9, 2021 1:50 PM
To: daos@daos.groups.io
Subject: [daos] Questions on DTX and KV Puts values visibility

Hi All,

In a two nodes cluster (one master/leader and one replica), I wrote a client application with multiple writer and reader threads against same object with different keys and values:

1. Writers issue daos_kv_put(oh, DAOS_TX_NONE, ..., &ev); call daos_event_test(&ev, DAOS_EQ_WAIT, &ev_flag); // wait for IO completion

2. Readers issue daos_kv_get(oh, DAOS_TX_NONE, 0, key, &size, buf, &ev); call daos_event_test(&ev, DAOS_EQ_WAIT, &ev_flag); // wait for IO completion

I observed that daos_kv_get returns older version occasionally. I have a few questions regarding DTX and the visibility of values obtained from the client's perspective.

1. On the master/leader server, when it sends RPC reply to the client for the daos_kv_put();

1.1 Has the put been committed before returning to the client? If not, what is the typical asynchronous commit threshold to make the put values visible to other clients?

[Nasf] By default, when the leader server reply the client, related update (put) is committable on server, but not really committed to the persistent storage. That is the typical asynchronously commit. But even if it is async commit, related update (put) is still visible to other clients as long as the leader server replied to the sponsored client. That is nothing related with the real commit to persistent storage. But if there is not communication between clients, then it is not easy to guarantee that the read on one client is after the write on another client.

1.2 Under what condition(s), does the leader commit the transaction?

[Nasf] For async commit, there is dedicated ULT that will batched commit the committable DTX entries periodically. Two conditions: the committable DTX entries count exceeds the threshold or some DTX entries become too old.

1.3 Does the leader send RPC to replica to commit the transaction synchronously or asynchronously after replying to the client?

[Nasf] The DTX batched commit ULT (not the IO handler) on the leader will async commit DTX entries after the leader replied related clients.

2. On the master server, how long does it wait (timeout) for the replica to reply?

[Nasf] It is the CaRT timeout, 60 seconds by default.

2.1 Does the master server send RPC to replica if the leader does not get reply from replica (e.g. when replica dies)?

[Nasf] If replica (non-leader) dead, then the leader will get timeout, then related update (put) will fail.

2.2 Under what condition(s) does the master server commit synchronously?

[Nasf] The application can require the leader to synchronously commit related update (put) via some RPC flags. Or when the leader cannot make async commit, it will synchronously commit related DTX.