Hi,

I assume that you have run dmg storage format after starting the server and before creating the pool, right? If you don’t want to emulate any SSD, you should also comment out the bdev_* options in the yaml file.

Cheers,

Johann

Hello,

My setup and issues are very similar to what has been described in https://daos.groups.io/g/daos/message/317. I have tried every suggested fix from the corresponding thread and so far, none have worked.

I'd like to setup a very simple daos server for testing purposes on a server with 120 GB of regular RAM, no NVMe SSDs attached, and only one single server instance. However, I'm not using docker.

Hence, I installed daos as described in the admin guide and used the config file from daos/utils/config/examples/daos_server_local.yml which would use a ram disk to emulate scm.

After I have started the daos server with, "daos_server --debug --config=$basepath/daos_server_local.yml start", I'd like to create a pool in a second terminal with "dmg -i -l localhost:10001 pool create -s 1G". This fails with:

localhost:10001: connected
Pool-create command FAILED: rpc error: code = Unknown desc = instance is not an access point
ERROR: dmg: rpc error: code = Unknown desc = instance is not an access point

Did I miss some step that I have to execute beforehand?

Hi there,

From your log:

snode3: Starting I/O server instance 0: /usr/bin/daos_io_server

snode2: daos_io_server:0  05/12-06:28:47.96 snode2 Using legacy core allocation algorithm

snode2: instance 0 exited: instance 0 exited prematurely: /usr/bin/daos_io_server (instance 0) exited: exit status 1

After format, the I/O engine failed to be started. Could you please look into the server logs under /tmp (i.e. /tmp/server.log)?

Cheers,

Johann

Greetings,

Executing the command:

daos pool list-cont --pool a68b3845-fe78-481e-aa84-164e851d5f52 --svc 40

Note that 40 does not exist.

We did not get an error from the daos command.

In the log:

05/13-11:57:11.02 delphi-006 DAOS[26509/26552] pool WARN src/pool/srv_target.c:1020 ds_pool_tgt_map_update() Ignore update pool a68b3845 1 -> 1

05/13-11:57:11.02 delphi-006 DAOS[26509/26552] pool WARN src/pool/srv_target.c:1020 ds_pool_tgt_map_update() Ignore update pool a68b3845 1 -> 1

05/13-11:57:11.02 delphi-006 DAOS[26509/26552] pool WARN src/pool/srv_target.c:1020 ds_pool_tgt_map_update() Ignore update pool a68b3845 1 -> 1

My guess is that ds_pool_tgt_map_update() should not even be called?

Cheers,

Colin

Thanks you, Nabarro.

That’s the one . thanks

This one?

doc/dev/development.md

So apologies but that link is dead, anyone know where the contents got moved to? I can’t find any references to relevant content in the docs.

The protobuf-c package is installed during build (I’m using master) and gets pulled in from <daos>/utils/sl/components/__init__.py but we don’t build the protoc* binaries/full compiler to avoid overhead for something that is only required very occasionally for developer purposes (we supply the –disable-protoc configure option in the build).

For the moment you will need to build the compiler and plug-in yourself but we plan on providing a build flag for developers that’s need it.

Tom

This is a development tool so maybe it’s not pulled in by the default build.

I installed as follows:

git clone https://github.com/protobuf-c/protobuf-c

cd protobuf-c/

./autogen.sh

./configure --prefix=/home/tanabarr/protobuf/install PKG_CONFIG_PATH=/home/tanabarr/protobuf/install/lib/pkgconfig

make && make install

it is not a plug-in that ships with the stock protobuf compiler package (which ships instead with C++ plugin).

See the <daos>/src/proto/Makefile for some details and this will point you to the following doc if compiler is missing.

https://github.com/daos-stack/daos/blob/master/doc/development.md#protobuf-compiler

Hi Guys,

In https://github.com/daos-stack/daos/tree/master/src/proto/README.md, it uses “protoc -I mgmt --c_out=../mgmt mgmt/srv.proto --plugin=/opt/potobuf/install/bin/protoc-gen-c” to generate C code. In recent DAOS, I cannot find “protoc-gen-c”.

Do you know why?

Thanks.

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Greetings,

struct cont_snap_destroy_in {

    struct cont_op_in cei_op;

    daos_epoch_t cei_epoch;

};

struct cont_snap_destroy_out {

    struct cont_op_out ceo_op;

    daos_epoch_t ceo_epoch;

};

Looks dead as the routine ds_cont_snap_destroy() seems to use:

struct cont_epoch_op_in {

    struct cont_op_in cei_op;

    daos_epoch_t cei_epoch;

};

struct cont_epoch_op_out {

    struct cont_op_out ceo_op;

    daos_epoch_t ceo_epoch;

};

Thanks.

Colin

Broken links fixed on master https://github.com/daos-stack/daos/pull/2654

That’s the one . thanks

This one?

doc/dev/development.md

So apologies but that link is dead, anyone know where the contents got moved to? I can’t find any references to relevant content in the docs.

The protobuf-c package is installed during build (I’m using master) and gets pulled in from <daos>/utils/sl/components/__init__.py but we don’t build the protoc* binaries/full compiler to avoid overhead for something that is only required very occasionally for developer purposes (we supply the –disable-protoc configure option in the build).

For the moment you will need to build the compiler and plug-in yourself but we plan on providing a build flag for developers that’s need it.

Tom

This is a development tool so maybe it’s not pulled in by the default build.

I installed as follows:

git clone https://github.com/protobuf-c/protobuf-c

cd protobuf-c/

./autogen.sh

./configure --prefix=/home/tanabarr/protobuf/install PKG_CONFIG_PATH=/home/tanabarr/protobuf/install/lib/pkgconfig

make && make install

it is not a plug-in that ships with the stock protobuf compiler package (which ships instead with C++ plugin).

See the <daos>/src/proto/Makefile for some details and this will point you to the following doc if compiler is missing.

https://github.com/daos-stack/daos/blob/master/doc/development.md#protobuf-compiler

Hi Guys,

In https://github.com/daos-stack/daos/tree/master/src/proto/README.md, it uses “protoc -I mgmt --c_out=../mgmt mgmt/srv.proto --plugin=/opt/potobuf/install/bin/protoc-gen-c” to generate C code. In recent DAOS, I cannot find “protoc-gen-c”.

Do you know why?

Thanks.

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

That’s the one . thanks

This one?

doc/dev/development.md

So apologies but that link is dead, anyone know where the contents got moved to? I can’t find any references to relevant content in the docs.

The protobuf-c package is installed during build (I’m using master) and gets pulled in from <daos>/utils/sl/components/__init__.py but we don’t build the protoc* binaries/full compiler to avoid overhead for something that is only required very occasionally for developer purposes (we supply the –disable-protoc configure option in the build).

For the moment you will need to build the compiler and plug-in yourself but we plan on providing a build flag for developers that’s need it.

Tom

This is a development tool so maybe it’s not pulled in by the default build.

I installed as follows:

git clone https://github.com/protobuf-c/protobuf-c

cd protobuf-c/

./autogen.sh

./configure --prefix=/home/tanabarr/protobuf/install PKG_CONFIG_PATH=/home/tanabarr/protobuf/install/lib/pkgconfig

make && make install

it is not a plug-in that ships with the stock protobuf compiler package (which ships instead with C++ plugin).

See the <daos>/src/proto/Makefile for some details and this will point you to the following doc if compiler is missing.

https://github.com/daos-stack/daos/blob/master/doc/development.md#protobuf-compiler

Hi Guys,

In https://github.com/daos-stack/daos/tree/master/src/proto/README.md, it uses “protoc -I mgmt --c_out=../mgmt mgmt/srv.proto --plugin=/opt/potobuf/install/bin/protoc-gen-c” to generate C code. In recent DAOS, I cannot find “protoc-gen-c”.

Do you know why?

Thanks.

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

This one?

doc/dev/development.md

So apologies but that link is dead, anyone know where the contents got moved to? I can’t find any references to relevant content in the docs.

The protobuf-c package is installed during build (I’m using master) and gets pulled in from <daos>/utils/sl/components/__init__.py but we don’t build the protoc* binaries/full compiler to avoid overhead for something that is only required very occasionally for developer purposes (we supply the –disable-protoc configure option in the build).

For the moment you will need to build the compiler and plug-in yourself but we plan on providing a build flag for developers that’s need it.

Tom

This is a development tool so maybe it’s not pulled in by the default build.

I installed as follows:

git clone https://github.com/protobuf-c/protobuf-c

cd protobuf-c/

./autogen.sh

./configure --prefix=/home/tanabarr/protobuf/install PKG_CONFIG_PATH=/home/tanabarr/protobuf/install/lib/pkgconfig

make && make install

it is not a plug-in that ships with the stock protobuf compiler package (which ships instead with C++ plugin).

See the <daos>/src/proto/Makefile for some details and this will point you to the following doc if compiler is missing.

https://github.com/daos-stack/daos/blob/master/doc/development.md#protobuf-compiler

Hi Guys,

In https://github.com/daos-stack/daos/tree/master/src/proto/README.md, it uses “protoc -I mgmt --c_out=../mgmt mgmt/srv.proto --plugin=/opt/potobuf/install/bin/protoc-gen-c” to generate C code. In recent DAOS, I cannot find “protoc-gen-c”.

Do you know why?

Thanks.

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

So apologies but that link is dead, anyone know where the contents got moved to? I can’t find any references to relevant content in the docs.

The protobuf-c package is installed during build (I’m using master) and gets pulled in from <daos>/utils/sl/components/__init__.py but we don’t build the protoc* binaries/full compiler to avoid overhead for something that is only required very occasionally for developer purposes (we supply the –disable-protoc configure option in the build).

For the moment you will need to build the compiler and plug-in yourself but we plan on providing a build flag for developers that’s need it.

Tom

This is a development tool so maybe it’s not pulled in by the default build.

I installed as follows:

git clone https://github.com/protobuf-c/protobuf-c

cd protobuf-c/

./autogen.sh

./configure --prefix=/home/tanabarr/protobuf/install PKG_CONFIG_PATH=/home/tanabarr/protobuf/install/lib/pkgconfig

make && make install

it is not a plug-in that ships with the stock protobuf compiler package (which ships instead with C++ plugin).

See the <daos>/src/proto/Makefile for some details and this will point you to the following doc if compiler is missing.

https://github.com/daos-stack/daos/blob/master/doc/development.md#protobuf-compiler

Hi Guys,

In https://github.com/daos-stack/daos/tree/master/src/proto/README.md, it uses “protoc -I mgmt --c_out=../mgmt mgmt/srv.proto --plugin=/opt/potobuf/install/bin/protoc-gen-c” to generate C code. In recent DAOS, I cannot find “protoc-gen-c”.

Do you know why?

Thanks.

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

This is a development tool so maybe it’s not pulled in by the default build.

I installed as follows:

git clone https://github.com/protobuf-c/protobuf-c

cd protobuf-c/

./autogen.sh

./configure --prefix=/home/tanabarr/protobuf/install PKG_CONFIG_PATH=/home/tanabarr/protobuf/install/lib/pkgconfig

make && make install

it is not a plug-in that ships with the stock protobuf compiler package (which ships instead with C++ plugin).

See the <daos>/src/proto/Makefile for some details and this will point you to the following doc if compiler is missing.

https://github.com/daos-stack/daos/blob/master/doc/development.md#protobuf-compiler

Hi Guys,

In https://github.com/daos-stack/daos/tree/master/src/proto/README.md, it uses “protoc -I mgmt --c_out=../mgmt mgmt/srv.proto --plugin=/opt/potobuf/install/bin/protoc-gen-c” to generate C code. In recent DAOS, I cannot find “protoc-gen-c”.

Do you know why?

Thanks.

Hi Guys,

In https://github.com/daos-stack/daos/tree/master/src/proto/README.md, it uses “protoc -I mgmt --c_out=../mgmt mgmt/srv.proto --plugin=/opt/potobuf/install/bin/protoc-gen-c” to generate C code. In recent DAOS, I cannot find “protoc-gen-c”.

Do you know why?

Thanks.

Hi,

As the comments of co_post_reply (in cart’s api.h):

               /**

                * Collective RPC post-reply callback.

                * This is an optional callback. If specified, it will execute after

                * reply is sent to parent (after co_aggregate executes).

                */

For example, bcast traverse a sub-tree like node_a (parent) -> node_b (self) -> node_c and node_d (children).

At request forward phase -- node_b gets request from node_a and forward to node_c+node_d.

At reply aggregate phase – node_b will send reply to node_a, but the reply will be delayed until gets and aggregates  replies from its children (co_aggregate is for this purpose). When the aggregated reply sent to parent, it will call the co_post_reply to do some cleanup work if it is provided (free memory or whatever else).

Thanks,

Xuezhao

Greetings,

Is “co_aggeregate” operation called before or after “ca_post_reply” operation?

Thanks.

Colin

Greetings,

Is “co_aggeregate” operation called before or after “ca_post_reply” operation?

Thanks.

Colin

I am not sure, unfortunately, we didn’t notice when it first happened.   We will keep an eye out if it happens again.

Kevan

It has been a while since I last saw a kernel backtrace 😊

Never seen this before. I assume that this happens during pool deletion or disconnect?

Cheers,

Johann

We had an oops occur today, backtrace is below.   When I started looking, I see there are 6 of these over the last month, the backtrace is always the same.   It seems to be happening in DAX.  Does this look familiar?

Thanks, Kevan

# cat backtrace

WARNING: CPU: 18 PID: 43720 at fs/dax.c:419 dax_disassociate_entry+0xdb/0x130

Modules linked in: ext4 mbcache jbd2 vfio_pci virtio_pci virtio_ring virtio nfsv3 nfs_acl socwatch2_11(OE) vtsspp(OE) sep5(OE) socperf3(OE) pax(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) sunrpc iTCO_wdt iTCO_vendor_support skx_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm vfat fat irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev mei_me mei lpc_ich i2c_i801 wmi ipmi_si ipmi_devintf ipmi_msghandler dax_pmem device_dax acpi_power_meter acpi_pad knem(OE) ip_tables xfs libcrc32c nd_pmem nd_btt ast i2c_algo_bit

drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci crct10dif_pclmul mlx5_core(OE) crct10dif_common crc32c_intel drm libahci mlxfw(OE) ptp pps_core libata vfio_mdev(OE) vfio_iommu_type1 vfio nvme mdev(OE) devlink nvme_core mlx_compat(OE) drm_panel_orientation_quirks nfit libnvdimm dm_mirror dm_region_hash dm_log dm_mod

CPU: 18 PID: 43720 Comm: daos_sys_0 Tainted: G           OE  ------------ T 3.10.0-1127.el7.x86_64 #1

Hardware name: Cray Inc. SYS-2029UZ-TN20R25M/X11DPU-Z+, BIOS 3.2 10/22/2019

Call Trace:

[<ffffffffb5b7ff85>] dump_stack+0x19/0x1b

[<ffffffffb549bd18>] __warn+0xd8/0x100

[<ffffffffb549be5d>] warn_slowpath_null+0x1d/0x20

[<ffffffffb56a6d0b>] dax_disassociate_entry+0xdb/0x130

[<ffffffffb56a7868>] __dax_invalidate_mapping_entry+0x68/0x120

[<ffffffffb56a9f47>] dax_delete_mapping_entry+0x17/0x50

[<ffffffffb55ce2fc>] truncate_exceptional_entry.part.13+0x1c/0x40

[<ffffffffb55ceaa2>] truncate_inode_pages_range+0x192/0x750

[<ffffffffb55cf0cf>] truncate_inode_pages_final+0x4f/0x60

[<ffffffffc0eeea0f>] ext4_evict_inode+0x10f/0x470 [ext4]

[<ffffffffb566b674>] evict+0xb4/0x180

[<ffffffffb566ba9c>] iput+0xfc/0x190

[<ffffffffb5666438>] __dentry_kill+0x158/0x1d0

[<ffffffffb5666ad5>] dput+0xb5/0x1a0

[<ffffffffb564f4dd>] __fput+0x18d/0x230

[<ffffffffb564f66e>] ____fput+0xe/0x10

[<ffffffffb54c31cb>] task_work_run+0xbb/0xe0

[<ffffffffb542cc65>] do_notify_resume+0xa5/0xc0

[<ffffffffb5b9322f>] int_signal+0x12/0x17

[root@delphi-002 oops-2020-05-05-11:59:34-44218-0]# 

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.