Date   

Re: IOR test with dfs

Chaarawi, Mohamad
 

Yes indeed as Steffen mentioned you need the headers also.
Also --with-daos and --with-cart should point to the same location (note that in 1.0, cart is a stand-alone lib, and that's why we had to have 2 paths).
(i.e. you need to pass a path to --with-cart=/path/)

Thanks,
Mohamad

On 9/21/20, 7:16 AM, "daos@daos.groups.io on behalf of Steffen Christgau" <daos@daos.groups.io on behalf of christgau@...> wrote:

Dear Johann,

On 9/21/20 2:00 PM, Lombardi, Johann wrote:
> I was hoping that --with-daoswould be enough if everything is installed
> at the default location via RPMs.

Maybe unrelated, but for getting MPICH 3.4a with DAOS ROMIO to compile
on a DAOS installation I had to specify both "--with-daos=/usr" and
"--with-cart=/usr"

Regards, Steffen


Re: IOR test with dfs

Steffen Christgau
 

Dear Johann,

On 9/21/20 2:00 PM, Lombardi, Johann wrote:
I was hoping that --with-daoswould be enough if everything is installed at the default location via RPMs.
Maybe unrelated, but for getting MPICH 3.4a with DAOS ROMIO to compile on a DAOS installation I had to specify both "--with-daos=/usr" and "--with-cart=/usr"

Regards, Steffen


Re: IOR test with dfs

Steffen Christgau
 

Hi Anton,

On 9/21/20 12:44 PM, anton.brekhov@... wrote:
*I installed DAOS as rpm v1.0.1.
Did you "only" install the daos package? IMO it does not contain the DAOS header files. Try to install daos-devel.

Regards, Steffen


Re: IOR test with dfs

Lombardi, Johann
 

I was hoping that --with-daos would be enough if everything is installed at the default location via RPMs.

@Chaarawi, Mohamad any thoughts?

 

Cheers,

johann

 

From: <daos@daos.groups.io> on behalf of "anton.brekhov@..." <anton.brekhov@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Monday 21 September 2020 at 12:44
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [daos] IOR test with dfs

 

Johann thanks!

I installed DAOS as rpm v1.0.1. Could you please tell where is install folder? I suppose that building with my parameters didn't give right binary:

[root@sky08 ior]# ./configure --with-daos=/usr/ --with-cart
.....

checking for d_hash_murmur64 in -lgurt... no

checking daos_types.h usability... no

checking daos_types.h presence... no

checking for daos_types.h... no

checking for uuid_generate in -luuid... no

checking for daos_sgl_init in -ldaos_common... yes

checking for daos_init in -ldaos... no

checking for dfs_mkdir in -ldfs... yes

checking for Gfarm file system... no

 

checking that generated files are newer than configure... done

It seems like I need source code, but there's no tag for 1.0.1.
Thanks!

 

 

 

 

 

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.


Re: IOR test with dfs

anton.brekhov@...
 

I just found tag in source directory and run ior building like this:
[root@apache512 ior]# ./configure --with-daos=/root/daos/install --with-cart
...

checking for d_hash_murmur64 in -lgurt... no

checking daos_types.h usability... yes

checking daos_types.h presence... yes

checking for daos_types.h... yes

checking for uuid_generate in -luuid... yes

checking for daos_sgl_init in -ldaos_common... yes

checking for daos_init in -ldaos... yes

checking for dfs_mkdir in -ldfs... yes

checking for Gfarm file system... no

checking that generated files are newer than configure... done

configure: creating ./config.status

config.status: creating Makefile

config.status: creating src/Makefile

config.status: creating src/test/Makefile

config.status: creating contrib/Makefile

config.status: creating doc/Makefile

config.status: creating src/config.h

config.status: src/config.h is unchanged

 

config.status: executing depfiles commands

[root @ apache512 ior] # make

Making all in src

make all-recursive

Making all in.

   CCLD ior

/ usr / bin / ld: ior-aiori-DFS.o: undefined symbol reference "d_hash_table_create"

/root/daos/install/lib64/libgurt.so.4: error adding symbols: DSO missing from command line

collect2: error: ld execution exited with return code 1

make [3]: *** [ior] Error 1

make [2]: *** [all-recursive] Error 1

make [1]: *** [all] Error 2

make: *** [all-recursive] Error 1

 


Re: IOR test with dfs

anton.brekhov@...
 

Johann thanks!

I installed DAOS as rpm v1.0.1. Could you please tell where is install folder? I suppose that building with my parameters didn't give right binary:

[root@sky08 ior]# ./configure --with-daos=/usr/ --with-cart
.....

checking for d_hash_murmur64 in -lgurt... no

checking daos_types.h usability... no

checking daos_types.h presence... no

checking for daos_types.h... no

checking for uuid_generate in -luuid... no

checking for daos_sgl_init in -ldaos_common... yes

checking for daos_init in -ldaos... no

checking for dfs_mkdir in -ldfs... yes

checking for Gfarm file system... no

 

checking that generated files are newer than configure... done

It seems like I need source code, but there's no tag for 1.0.1.
Thanks!

 
 
 
 
 


Re: DAOS_test failed

Lombardi, Johann
 

Ok, glad that it works now for dfuse.

 

Cheers,

Johann

 

From: <daos@daos.groups.io> on behalf of "anton.brekhov@..." <anton.brekhov@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Monday 21 September 2020 at 11:27
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [daos] DAOS_test failed

 

Hi Johann!

I'm sorry I mixed up with mount point and POSIX is working fine, so I've deleted my previous messages. Right now I'm facing issues with dfs API in ior test. Heres new thread

https://daos.groups.io/g/daos/topic/ior_test_with_dfs/76976428?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,76976428

Thank you!

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.


Re: IOR test with dfs

Lombardi, Johann
 

Hey Anton,

 

I assume that you have built ior with the DFS backend support as specified in https://github.com/hpc/ior/blob/master/README_DAOS, right?

 

Although I think it should work, I have never tried “-a=DFS” and always used “-a DFS” on my side. Could you please give this a try just to double check?

The prefix is also relative to the absolute file path, so you should be using --dfs.prefix /mnt/dfuse instead of /tmp/dfuse. I assume that $dpool and $dcont match respectively the pool and container UUIDs that you used to mount with dfuse, right? Also, svcl should match what was returned at pool creation time, you got 0 and not 1, right?

 

To summarize, could you please try:

ior -a DFS -t 128k -b 1G -r -w -o /mnt/dfuse/testFile_dfs -G 123 -k --dfs.pool $dpool --dfs.svcl 0 --dfs.cont $dcont --dfs.prefix /mnt/dfuse

 

Cheers,

Johann

 

From: <daos@daos.groups.io> on behalf of "anton.brekhov@..." <anton.brekhov@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Sunday 20 September 2020 at 21:01
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: [daos] IOR test with dfs

 

[root@sky08 ~]# mpirun --allow-run-as-root  -np 1  ior -a=DFS -b 1G -w -r  -o /mnt/dfuse/testFile_dfs -t 128k -k --dfs.pool $dpool --dfs.svcl 0 --dfs.cont $dc

ont --dfs.prefix /tmp/dfuse/ -G 123

Error invalid argument: --dfs.pool

Error invalid argument: 3f0762ad-23b3-418a-a14d-4f813d122d46

Error invalid argument: --dfs.svcl

Error invalid argument: 0

Error invalid argument: --dfs.cont

Error invalid argument: 0e75d729-2164-493e-ba9e-2942df394bca

Error invalid argument: --dfs.prefix

Error invalid argument: /tmp/dfuse/

Error invalid argument: 123

Invalid options

Synopsis ior

[root@sky08 ~]# mpirun --allow-run-as-root  -np 1  ior -a=DFS -b 1G -w -r  -o /mnt/dfuse/testFile_dfs -t 128k -k --dfs.pool $dpool --dfs.svcl 0 --dfs.cont $dcont --dfs.prefix /tmp/dfuse/ -G 123

Error invalid argument: --dfs.pool

Error invalid argument: 3f0762ad-23b3-418a-a14d-4f813d122d46

Error invalid argument: --dfs.svcl

Error invalid argument: 0

Error invalid argument: --dfs.cont

Error invalid argument: 0e75d729-2164-493e-ba9e-2942df394bca

Error invalid argument: --dfs.prefix

Error invalid argument: /tmp/dfuse/

Error invalid argument: 123

Invalid options

Synopsis ior

How I need to compile ior to make this work?

Thanks!

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.


Re: DAOS_test failed

anton.brekhov@...
 

Hi Johann!

I'm sorry I mixed up with mount point and POSIX is working fine, so I've deleted my previous messages. Right now I'm facing issues with dfs API in ior test. Heres new thread

https://daos.groups.io/g/daos/topic/ior_test_with_dfs/76976428?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,76976428

Thank you!


Re: DAOS_test failed

Lombardi, Johann
 

Hi Anton,

 

Are you able to create files manually under the dfuse mountpoint? Does ior work with file per process (-F)?

 

Cheers,

Johann

 

From: <daos@daos.groups.io> on behalf of "anton.brekhov@..." <anton.brekhov@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Sunday 20 September 2020 at 20:17
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [daos] DAOS_test failed

 

Hi everyone! 

I tried to launch ior test on client node and got the error:

[root@sky08 ~]# mpirun --allow-run-as-root  -np 32  ior -a POSIX -b 1G -w -r  -o /mnt/dfuse/testFile_POSIX -t 128k -k -G 123

IOR-3.4.0+dev: MPI Coordinated Test of Parallel I/O

Began               : Sun Sep 20 21:16:24 2020

Command line        : ior -a POSIX -b 1G -w -r -o /mnt/dfuse/testFile_POSIX -t 128k -k -G 123

Machine             : Linux sky08

TestID              : 0

StartTime           : Sun Sep 20 21:16:24 2020

WARNING: Backend returned error during statfs.

POSIX couldn't call statvfs: No such file or directory

 

Options:

api                 : POSIX

apiVersion          :

test filename       : /mnt/dfuse/testFile_POSIX

access              : single-shared-file

type                : independent

segments            : 1

ordering in a file  : sequential
ordering inter file : no tasks offsets

nodes               : 1

tasks               : 32

clients per node    : 32

repetitions         : 1

xfersize            : 131072 bytes

blocksize           : 1 GiB

aggregate filesize  : 32 GiB

 

Results:

 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter

------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----

--------------------------------------------------------------------------

MPI_ABORT was invoked on rank 30 in communicator MPI_COMM_WORLD

with errorcode -1.

 

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.

You may or may not see output from other processes, depending on

exactly when Open MPI kills them.

--------------------------------------------------------------------------

ERROR: open64("/mnt/dfuse/testFile_POSIX", 66, 0664) failed, (aiori-POSIX.c:440)

ERROR: open64("/mnt/dfuse/testFile_POSIX", 66, 0664) failed, (aiori-POSIX.c:440)

ERROR: open64("/mnt/dfuse/testFile_POSIX", 66, 0664) failed, (aiori-POSIX.c:440)

ERROR: open64("/mnt/dfuse/testFile_POSIX", 66, 0664) failed, (aiori-POSIX.c:440)

ERROR: open64("/mnt/dfuse/testFile_POSIX", 66, 0664) failed, (aiori-POSIX.c:440)

ERROR: open64("/mnt/dfuse/testFile_POSIX", 66, 0664) failed, (aiori-POSIX.c:440)

ERROR: open64("/mnt/dfuse/testFile_POSIX", 66, 0664) failed, (aiori-POSIX.c:440)

ERROR: open64("/mnt/dfuse/testFile_POSIX", 66, 0664) failed, (aiori-POSIX.c:440)

ERROR: open64("/mnt/dfuse/testFile_POSIX", 66, 0664) failed, (aiori-POSIX.c:440)

ERROR: open64("/mnt/dfuse/testFile_POSIX", 66, 0664) failed, (aiori-POSIX.c:440)

ERROR: open64("/mnt/dfuse/testFile_POSIX", 66, 0664) failed, (aiori-POSIX.c:440)

ERROR: open64("/mnt/dfuse/testFile_POSIX", 66, 0664) failed, (aiori-POSIX.c:440)

ERROR: open64("/mnt/dfuse/testFile_POSIX", 66, 0664) failed, (aiori-POSIX.c:440)

ERROR: open64("/mnt/dfuse/testFile_POSIX", 66, 0664) failed, (aiori-POSIX.c:440)

ERROR: open64("/mnt/dfuse/testFile_POSIX", 66, 0664) failed, (aiori-POSIX.c:440)

ERROR: open64("/mnt/dfuse/testFile_POSIX", 66, 0664) failed, (aiori-POSIX.c:440)

ERROR: open64("/mnt/dfuse/testFile_POSIX", 66, 0664) failed, (aiori-POSIX.c:440)

ERROR: open64("/mnt/dfuse/testFile_POSIX", 66, 0664) failed, (aiori-POSIX.c:440)

ERROR: open64("/mnt/dfuse/testFile_POSIX", 66, 0664) failed, (aiori-POSIX.c:440)

ERROR: open64("/mnt/dfuse/testFile_POSIX", 66, 0664) failed, (aiori-POSIX.c:440)

ERROR: open64("/mnt/dfuse/testFile_POSIX", 66, 0664) failed, (aiori-POSIX.c:440)

ERROR: open64("/mnt/dfuse/testFile_POSIX", 66, 0664) failed, (aiori-POSIX.c:440)

ERROR: open64("/mnt/dfuse/testFile_POSIX", 66, 0664) failed, (aiori-POSIX.c:440)

ERROR: open64("/mnt/dfuse/testFile_POSIX", 66, 0664) failed, (aiori-POSIX.c:440)

ERROR: open64("/mnt/dfuse/testFile_POSIX", 66, 0664) failed, (aiori-POSIX.c:440)

ERROR: open64("/mnt/dfuse/testFile_POSIX", 66, 0664) failed, (aiori-POSIX.c:440)

ERROR: open64("/mnt/dfuse/testFile_POSIX", 66, 0664) failed, (aiori-POSIX.c:440)

ERROR: open64("/mnt/dfuse/testFile_POSIX", 66, 0664) failed, (aiori-POSIX.c:440)

ERROR: open64("/mnt/dfuse/testFile_POSIX", 66, 0664) failed, (aiori-POSIX.c:440)

ERROR: open64("/mnt/dfuse/testFile_POSIX", 66, 0664) failed, (aiori-POSIX.c:440)

ERROR: open64("/mnt/dfuse/testFile_POSIX", 66, 0664) failed, (aiori-POSIX.c:440)

ERROR: open64("/mnt/dfuse/testFile_POSIX", 66, 0664) failed, (aiori-POSIX.c:440)

[sky08:04378] PMIX ERROR: UNREACHABLE in file server/pmix_server.c at line 2193

[sky08:04378] PMIX ERROR: UNREACHABLE in file server/pmix_server.c at line 2193

[sky08:04378] PMIX ERROR: UNREACHABLE in file server/pmix_server.c at line 2193

[sky08:04378] PMIX ERROR: UNREACHABLE in file server/pmix_server.c at line 2193

[sky08:04378] PMIX ERROR: UNREACHABLE in file server/pmix_server.c at line 2193

[sky08:04378] 31 more processes have sent help message help-mpi-api.txt / mpi-abort

[sky08:04378] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

What can it be?

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.


IOR test with dfs

anton.brekhov@...
 

[root@sky08 ~]# mpirun --allow-run-as-root  -np 1  ior -a=DFS -b 1G -w -r  -o /mnt/dfuse/testFile_dfs -t 128k -k --dfs.pool $dpool --dfs.svcl 0 --dfs.cont $dc

ont --dfs.prefix /tmp/dfuse/ -G 123

Error invalid argument: --dfs.pool

Error invalid argument: 3f0762ad-23b3-418a-a14d-4f813d122d46

Error invalid argument: --dfs.svcl

Error invalid argument: 0

Error invalid argument: --dfs.cont

Error invalid argument: 0e75d729-2164-493e-ba9e-2942df394bca

Error invalid argument: --dfs.prefix

Error invalid argument: /tmp/dfuse/

Error invalid argument: 123

Invalid options

Synopsis ior

[root@sky08 ~]# mpirun --allow-run-as-root  -np 1  ior -a=DFS -b 1G -w -r  -o /mnt/dfuse/testFile_dfs -t 128k -k --dfs.pool $dpool --dfs.svcl 0 --dfs.cont $dcont --dfs.prefix /tmp/dfuse/ -G 123

Error invalid argument: --dfs.pool

Error invalid argument: 3f0762ad-23b3-418a-a14d-4f813d122d46

Error invalid argument: --dfs.svcl

Error invalid argument: 0

Error invalid argument: --dfs.cont

Error invalid argument: 0e75d729-2164-493e-ba9e-2942df394bca

Error invalid argument: --dfs.prefix

Error invalid argument: /tmp/dfuse/

Error invalid argument: 123

Invalid options

Synopsis ior

How I need to compile ior to make this work?

Thanks!


Re: DAOS starting example in C++

Lombardi, Johann
 

Hi there,

 

Please find attached a small test that you can compile as follows:

$ gcc -o test -L${DAOS_PATH}/lib64 -I${DAOS_PATH}/include -ldaos -lgurt -ldaos_common -luuid ./test.c

 

And then run:

$ D_LOG_MASK=ERR ./test

Initializing DAOS               OK

Parsing svcl                        OK

Connecting to pool           OK

Opening container            OK

Object open                       OK

Writing to object               OK

Closing object                    OK

Closing container              OK

Disconnecting from pool OK

Tearing down DAOS          OK

 

It provides the basics to connect to a pool, open a container and write some KV entries in parallel.

 

Regarding the online Doxygen documentation, it seems to go away each time I update the GitHub I/O pages with mkdocs. I need to spend some time to understand and fix this.

 

HTH

 

Cheers,

Johann

 

From: <daos@daos.groups.io> on behalf of Steffen Christgau <christgau@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Friday 18 September 2020 at 17:00
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [daos] DAOS starting example in C++

 

Hi Adam,

 

On 18/09/2020 16.34, adam.abed.abud@... wrote:

Is there a starting C++ example (with its relative CMake file) that

writes and reads data with the DAOS API?

 

AFAIK there is no C++ binding but one for C ;-) [1] Unfortunately, the

link to the Doxygen documentation does not work, but I used the native

API successfully. Since the header files [2] contain the Doxygen input

you find valuable information there.

 

Concerning compilation, there's nothing more to do than setting the

include directory and the library name (and path) which is daos. In

addition, you may need additional libs: like daos_common, gurt, and uuid.

 

Regards, Steffen

 

 

 

 

 

 

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.


Re: DAOS starting example in C++

Steffen Christgau
 

Hi Adam,

On 18/09/2020 16.34, adam.abed.abud@... wrote:
Is there a starting C++ example (with its relative CMake file) that
writes and reads data with the DAOS API?
AFAIK there is no C++ binding but one for C ;-) [1] Unfortunately, the
link to the Doxygen documentation does not work, but I used the native
API successfully. Since the header files [2] contain the Doxygen input
you find valuable information there.

Concerning compilation, there's nothing more to do than setting the
include directory and the library name (and path) which is daos. In
addition, you may need additional libs: like daos_common, gurt, and uuid.

Regards, Steffen

[1] https://daos-stack.github.io/user/interface/
[2] https://github.com/daos-stack/daos/tree/v1.0.1/src/include


DAOS starting example in C++

adam.abed.abud@...
 

Hello everyone,

Is there a starting C++ example (with its relative CMake file) that writes and reads data with the DAOS API?

Thanks a lot.

Cheers,
Adam

 


Re: DAOS with NVMe-over-Fabrics

anton.brekhov@...
 

Johann and Niu thanks a lot!

I'll try to test it and send result to this topic!


Re: DAOS with NVMe-over-Fabrics

Lombardi, Johann
 

As pointed out by Niu, the main gap is really on the DAOS control plane (i.e. dmg storage scan & support in the yaml config file). SPDK has built-in support for a NVMe-oF initiator/host (see https://spdk.io/doc/nvme.html#nvme_fabrics_host), so we “just” need to generate a SPDK config file (i.e. available under $SCM_MOUNTPOINT/daos_nvme.conf) with the right parameters.

 

This is an example from my local set up:

 

$ cat /mnt/daos0/daos_nvme.conf

[Nvme]

    TransportID "trtype:PCIe traddr:0000:81:00.0" Nvme_wolf-118.wolf.hpdd.intel.com_0

    TransportID "trtype:PCIe traddr:0000:da:00.0" Nvme_wolf-118.wolf.hpdd.intel.com_1

    RetryCount 4

    TimeoutUsec 0

    ActionOnTimeout None

    AdminPollRate 100000

    HotplugEnable No

    HotplugPollRate 0

 

For NVMe-oF, the TransportID should be modified to something like this:

    TransportId "trtype:rdma adrfam:IPv4 traddr:10.9.1.118 trsvcid:4420 subnqn:test" Nvme0_wolf-118.wolf.hpdd.intel.com_0

 

Ideally, we would just support a new “nvmeof” bdev_class in the DAOS yaml file that will automatically generate a daos_nvme.conf with NVMe-oF support, e.g.:

 

bdev_class: nvmeof

bdev_list: ["IPv4:10.9.1.118:4420:test”]

 

Any patches to support this in the control plane would be welcomed 😊

 

Another other option is to use the Linux kernel NVMe-oF driver that gives you a block device that you can then use in the DAOS yaml file via “bdev_class: kdev”. That’s definitely not the most optimal path regarding performance, but for sure the most straightforward way to use NVMe-oF with DAOS until the control plane supports it.

 

HTH

 

Cheers,

Johann

 

From: <daos@daos.groups.io> on behalf of "Niu, Yawei" <yawei.niu@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Thursday 17 September 2020 at 03:02
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: Re: [daos] DAOS with NVMe-over-Fabrics

 

So far DAOS server supports only local PCIe attached NVMe, but it won’t be very difficult to support NVMeOF in the future, it requires only server configuration changes, all others are transparent to DAOS.

If you are talking about client directly connects to NVMe disks through NVMeOF, that’ll require exposing block level protocol to client, we don’t have plan for this in the near term.

 

Thanks

-Niu

 

From: <daos@daos.groups.io> on behalf of "anton.brekhov@..." <anton.brekhov@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Thursday, September 17, 2020 at 1:08 AM
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: [daos] DAOS with NVMe-over-Fabrics

 

[Edited Message Follows]

Our partners at Intel said that it's possible to connect NVMe disks by NVMeOF with DAOS. Could you explain how to do this? 

Thanks!

 

 

 

 

 

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.


Re: DAOS with NVMe-over-Fabrics

Niu, Yawei
 

So far DAOS server supports only local PCIe attached NVMe, but it won’t be very difficult to support NVMeOF in the future, it requires only server configuration changes, all others are transparent to DAOS.

If you are talking about client directly connects to NVMe disks through NVMeOF, that’ll require exposing block level protocol to client, we don’t have plan for this in the near term.

 

Thanks

-Niu

 

From: <daos@daos.groups.io> on behalf of "anton.brekhov@..." <anton.brekhov@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Thursday, September 17, 2020 at 1:08 AM
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: [daos] DAOS with NVMe-over-Fabrics

 

[Edited Message Follows]

Our partners at Intel said that it's possible to connect NVMe disks by NVMeOF with DAOS. Could you explain how to do this? 

Thanks!

 

 

 

 

 


DAOS with NVMe-over-Fabrics

anton.brekhov@...
 
Edited

Our partners at Intel said that it's possible to connect NVMe disks by NVMeOF with DAOS. Could you explain how to do this? 

Thanks!

 
 
 
 
 


Re: DAOS_test failed

anton.brekhov@...
 

Thanks everyone for the help!

I've created container and pool, mounted with dfuse, stored file, unmounted, mounted again and everything is fine! Thanks!


Re: DAOS_test failed

anton.brekhov@...
 

daos_server.log full of this debug:

09/16-15:04:12.07 apache512 DAOS[13572/13608] vos  DBUG src/vos/vos_iterator.c:279 vos_iter_probe() probing iterator
09/16-15:04:12.07 apache512 DAOS[13572/13608] vos  DBUG src/vos/vos_iterator.c:288 vos_iter_probe() done probing iterator rc = DER_NONEXIST(-1005)