Re: DPI_SPACE query after extending pool

Tuffli, Chuck


When I checked this morning, the pool had been idle four days, but the values from daos_pool_query() have not changed.

As for object class, I'm not sure. Pool creation didn't specify a class. Here is the container query output:
# daos cont query
ERROR: daos: pool and container ID must be specified if --path not used
]# daos cont query kiddie whiz
  Container UUID             : 5c61770a-2b56-4922-b95b-d025fa4d0527
  Container Label            : whiz
  Container Type             : POSIX
  Pool UUID                  : 700bf1b6-38b8-467e-9f91-7131138210ba
  Number of snapshots        : 0
  Latest Persistent Snapshot : 0x0
  Container redundancy factor: 0
  Object Class               : UNKNOWN
  Chunk Size                 : 1.0 MiB

From: <> on behalf of Wang, Di <>
Sent: Wednesday, May 25, 2022 4:33 PM
To: <>
Subject: Re: [daos] DPI_SPACE query after extending pool

Hello Chuck


Pool extend might migrate the data to the new pool target, then the original data will be delete asynchronous, so those space might be reclaimed a few mins later if the system is not busy.


You probably should do your daos_pool_query() a bit later. Btw: what are those objects class in your pool?





On 5/25/22, 2:54 PM, " on behalf of Tuffli, Chuck" < on behalf of chuck.tuffli@...> wrote:


# dmg pool query kiddie

Pool 700bf1b6-38b8-467e-9f91-7131138210ba, ntarget=32, disabled=0, leader=0, version=18

Pool space info:

- Target(VOS) count:32

- Storage tier 0 (SCM):

  Total size: 60 GB

  Free: 60 GB, min:1.9 GB, max:1.9 GB, mean:1.9 GB

- Storage tier 1 (NVMe):

  Total size: 940 GB

  Free: 870 GB, min:27 GB, max:27 GB, mean:27 GB

Rebuild done, 83 objs, 0 recs

From: <> on behalf of Nabarro, Tom <tom.nabarro@...>
Sent: Tuesday, May 24, 2022 2:24 PM
To: <>
Subject: Re: [daos] DPI_SPACE query after extending pool


Hello Chuck,


Could you please run `dmg pool query` on the pool and show the results, this will give you a bit more info on pool usage.





From: <> On Behalf Of Tuffli, Chuck
Sent: Tuesday, May 24, 2022 9:05 PM
Subject: [daos] DPI_SPACE query after extending pool


I've been experimenting with extending a pool but don't quite understand the results. Any insights would be most appreciated.


The cluster is running with DAOS v2.0.2 and consists of a client and a pair of servers/storage nodes. To simulate adding a server to the cluster, I created a pool by specifying the ranks associated with one of the servers. I.e.:

# dmg system query --verbose

Rank UUID                                 Control Address  Fault Domain State  Reason

---- ----                                 ---------------  ------------ -----  ------

0    654345f9-249c-48b1-b6dc-ec08dbf2aded x.150.0.3:10001  /d006        Joined

1    b384771a-ddbc-491a-8807-8d86544d7c2f x.150.0.4:10001  /d010        Joined

2    01c672cf-3365-476f-87ec-41a15a44e946 x.150.0.4:10001  /d010        Joined

3    93a8d382-b970-408a-9c21-e01c35265e77 x.150.0.3:10001  /d006        Joined

# dmg pool create --ranks=0,3 --size=500G kiddie


I used the pool extend command to simulate adding a server:

# dmg pool extend --ranks=1,2 kiddie
Extend command succeeded


My application queried the pool size before and after the extension using daos_pool_query( ... DPI_SPACE ...). The numbers below are the info.pi_space.ps_space values for (DAOS_MEDIA_SCM, DAOS_MEDIA_NVME).

before extend:

s_total(30000021504,470000000000) s_free(29994849224,434968621056)

after extend:

s_total(60000043008,940000000000) s_free(59994841512,869937672192)


The total pool sized doubled (good), but the used space (i.e., s_total - s_free) also doubled. Naively, I expected the used space to remain the same as the pool has a redundancy factor of zero. Doing some arithmetic on the above works out to the used space being 35.036 GB before the expansion and 70.068 GB after. Note that, for the moment, I'm choosing to ignore that the used size is several orders of magnitude bigger that the data written (~600 KB).


Where did I goof in this methodology? TIA.




Join { to automatically receive all group messages.