Request Fan out in DAOS clients.


Colin Ngam <cngam@...>
 

Greetings,

 

Given a request to DAOS Object Array, if the request span all the targets, wraps around and ended up with multiple chunks per Shard, does each request to the Shard contain all the Chunks needed? Or does each request in the fanout contains only 1 Chunk?

 

Example:

 

12 Targets, 1 byte record, 1K Chunks, IO size 48K. Does this result in 48 requests to the servers or just 12 requests and each asking for 4 chunks? What are the rules, if any, that will break convention? Is there a debug or monitor capability for such observation?

 

We are trying to figure out tuning and performance and observed behavior etc.

 

Thanks.

 

Colin


Chaarawi, Mohamad
 

Hi,

 

If you are using the array API or the DFS API, in your example below, this will translate to 48 parallel inflight IOs to 48 different dkeys.

Each chunk maps to a single dkey.

 

Now the distribution to targets is a different question. If you have 12 targets that doesn’t mean all 12 targets are used, and it doesn’t mean either there is any round-robin distribution of chunks to targets.

The layout depends on the object class. Let’s assume you are using widely striped that spans all targets (OC_SX), but still, each dkey is placed on a target using consisting hashing. So this does not guarantee 100% that all targets are used, but we observe that the hashing algorithm does well in distributing the workload to all targets.

 

Thanks,

Mohamad

 

From: <daos@daos.groups.io> on behalf of Colin Ngam <cngam@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Monday, April 20, 2020 at 11:35 AM
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: [daos] Request Fan out in DAOS clients.

 

Greetings,

 

Given a request to DAOS Object Array, if the request span all the targets, wraps around and ended up with multiple chunks per Shard, does each request to the Shard contain all the Chunks needed? Or does each request in the fanout contains only 1 Chunk?

 

Example:

 

12 Targets, 1 byte record, 1K Chunks, IO size 48K. Does this result in 48 requests to the servers or just 12 requests and each asking for 4 chunks? What are the rules, if any, that will break convention? Is there a debug or monitor capability for such observation?

 

We are trying to figure out tuning and performance and observed behavior etc.

 

Thanks.

 

Colin


Colin Ngam
 

Thanks! I did forget about the fact that each chunk is a separate Dkey!

 

From: <daos@daos.groups.io> on behalf of "Chaarawi, Mohamad" <mohamad.chaarawi@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Monday, April 20, 2020 at 11:59 AM
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [daos] Request Fan out in DAOS clients.

 

Hi,

 

If you are using the array API or the DFS API, in your example below, this will translate to 48 parallel inflight IOs to 48 different dkeys.

Each chunk maps to a single dkey.

 

Now the distribution to targets is a different question. If you have 12 targets that doesn’t mean all 12 targets are used, and it doesn’t mean either there is any round-robin distribution of chunks to targets.

The layout depends on the object class. Let’s assume you are using widely striped that spans all targets (OC_SX), but still, each dkey is placed on a target using consisting hashing. So this does not guarantee 100% that all targets are used, but we observe that the hashing algorithm does well in distributing the workload to all targets.

 

Thanks,

Mohamad

 

From: <daos@daos.groups.io> on behalf of Colin Ngam <cngam@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Monday, April 20, 2020 at 11:35 AM
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: [daos] Request Fan out in DAOS clients.

 

Greetings,

 

Given a request to DAOS Object Array, if the request span all the targets, wraps around and ended up with multiple chunks per Shard, does each request to the Shard contain all the Chunks needed? Or does each request in the fanout contains only 1 Chunk?

 

Example:

 

12 Targets, 1 byte record, 1K Chunks, IO size 48K. Does this result in 48 requests to the servers or just 12 requests and each asking for 4 chunks? What are the rules, if any, that will break convention? Is there a debug or monitor capability for such observation?

 

We are trying to figure out tuning and performance and observed behavior etc.

 

Thanks.

 

Colin