Dear all (again),

sorry for "flooding" the list, but there currently some questions rising
up in my DAOS project.

I know that there is a HDF5 VOL plugin that supports DAOS. From what I
read in the docs it appears to me that an application that uses HDF5 has
to be aware of the DAOS VOL plugin as well. I suspect that netCDF is has
not (yet) been adopted to that plugin. At least I found no clues in the
netCDF 4.7.0 source code.

Is it correct that simply linking netCDF against an HDF5 library with
DAOS VOL plugin would not use DAOS at all?

In [1] netCDF has been adopted to HDF5/DAOS. However, I was not able to
find a version of that modified library. Is there any chance to get this
prototype implementation? I'm sure there are some authors of the paper
on that mailing list. ;-)

Thanks in advance

Steffen

[1] M. Scot Breitenfeld, Neil Fortner, Jordan Henderson, Jerome
Soumagne, Mohamad Chaarawi, Johann Lombardi, Quincey Koziol: DAOS for
Extreme-scale Systems in Scientific Applications
https://arxiv.org/abs/1712.00423

Hi Steffen,

To give you a brief history of this, the HDF group were working on a DAOS VOL prototype in the ESSIO project (ended 2016/17 I believe). During that, we ported NetCDF4 on top of HDF5 DAOS VOL, on top of DAOS. This included some changes to the NetCDF4 source since some functionality that NetCDF4 relies on from the HDF5 API was not supported with the DAOS VOL. I believe someone from the HDF group can point you to where those changes to NetCDF4 reside.

The HDF group is working today on productizing the HDF5 DAOS VOL connector to support several of the missing features and a lot of stabilization work from the prototype that was developed before. They are not at a point where NetCDF4 is supported with the DAOS VOL without any changes, and I'm not sure if the modified version will function today as it hasn't been tested yet with the DAOS connector. We do want to get to a point where any application or middleware that uses HDF5 will function with the DAOS VOL connector, simply by dynamically loading the plugin (like HDF5 filters if you are available with those), hopefully without any application change or maybe minimal for VOL connector selection (which can be done through code or env settings, or external tools).

A stable HDF5 VOL connector will be available later this year, and integration with more apps & libraries like NetCDF4 will follow after.

Thanks,
Mohamad


On 8/15/19, 10:26 AM, "daos@daos.groups.io on behalf of Steffen Christgau" <daos@daos.groups.io on behalf of christgau@...> wrote:

Dear all (again),

sorry for "flooding" the list, but there currently some questions rising
up in my DAOS project.

I know that there is a HDF5 VOL plugin that supports DAOS. From what I
read in the docs it appears to me that an application that uses HDF5 has
to be aware of the DAOS VOL plugin as well. I suspect that netCDF is has
not (yet) been adopted to that plugin. At least I found no clues in the
netCDF 4.7.0 source code.

Is it correct that simply linking netCDF against an HDF5 library with
DAOS VOL plugin would not use DAOS at all?

In [1] netCDF has been adopted to HDF5/DAOS. However, I was not able to
find a version of that modified library. Is there any chance to get this
prototype implementation? I'm sure there are some authors of the paper
on that mailing list. ;-)

Thanks in advance

Steffen

[1] M. Scot Breitenfeld, Neil Fortner, Jordan Henderson, Jerome
Soumagne, Mohamad Chaarawi, Johann Lombardi, Quincey Koziol: DAOS for
Extreme-scale Systems in Scientific Applications
https://arxiv.org/abs/1712.00423