Greetings,

 

Recently I updated my daos repo to master top of tree, and now any attempt to create a container causes the access-point daos_io_server to segfault.   Before I dig deeply, is this a known issue?  My config is one client node plus one server node with dual daos_io_servers.  Before running this test the server storage was reformatted.

 

Commands on the client:

 

[root@delphi-005 tmp]# dmg -i -l delphi-004 system list-pools

delphi-004:10001: connected

No pools in system

[root@delphi-005 tmp]# dmg -i -l delphi-004 pool create --scm-size=768G --nvme-size=10T

delphi-004:10001: connected

Pool-create command SUCCEEDED: UUID: 9acb0a19-2ecf-4d3f-8f7a-2afcec26128f, Service replicas: 0

[root@delphi-005 tmp]# dmg -i -l delphi-004 system list-pools

delphi-004:10001: connected

Pool UUID                            Svc Replicas 

---------                            ------------ 

9acb0a19-2ecf-4d3f-8f7a-2afcec26128f 0            

[root@delphi-005 tmp]# daos container create --pool=9acb0a19-2ecf-4d3f-8f7a-2afcec26128f --svc=0

 

At the point the client window hangs, and the daos_io_server setfaults.  Back trace collected via gdb is:

 

Program received signal SIGSEGV, Segmentation fault.

[Switching to Thread 0x7f37bcdfd700 (LWP 22203)]

0x00007f37cdd2a0a8 in field_is_zeroish (member=member@entry=0x2aaab0423678, field=<optimized out>) at protobuf-c/protobuf-c.c:559

559                  ret = (NULL == *(const char * const *) member) ||

(gdb) bt

#0  0x00007f37cdd2a0a8 in field_is_zeroish (member=member@entry=0x2aaab0423678, field=<optimized out>) at protobuf-c/protobuf-c.c:559

#1  0x00007f37cdd2aa53 in unlabeled_field_get_packed_size (member=0x2aaab0423678, field=0x7f37cf1d4e18 <mgmt__get_attach_info_resp__field_descriptors+216>)

    at protobuf-c/protobuf-c.c:591

#2  protobuf_c_message_get_packed_size (message=message@entry=0x2aaab0423640) at protobuf-c/protobuf-c.c:739

#3  0x00007f37cef93d31 in mgmt__get_attach_info_resp__get_packed_size (message=message@entry=0x2aaab0423640) at src/mgmt/srv.pb-c.c:296

#4  0x00007f37c6998d4a in ds_mgmt_drpc_get_attach_info (drpc_req=<optimized out>, drpc_resp=0x7f3770026a10) at src/mgmt/srv_drpc.c:239

#5  0x000000000040beb5 in drpc_handler_ult (call_ctx=0x7f3770026990) at src/iosrv/drpc_progress.c:297

#6  0x00007f37ce3c317b in ABTD_thread_func_wrapper_thread () from /home/users/daos/daos/install/lib/libabt.so.0

#7  0x00007f37ce3c3851 in make_fcontext () from /home/users/daos/daos/install/lib/libabt.so.0

#8  0x0000000000000000 in ?? ()

(gdb) p member

$1 = (const void *) 0x2aaab0423678

(gdb) p *(const char * const *) member

$3 = 0xb801e74ea7845500 <Address 0xb801e74ea7845500 out of bounds>

 

Is this a known problem?

 

Thanks, Kevan

Sigh.   Please ignore this, one of my compatriots with the same hardware config was able to create this pool and container without error.   So the problem is obviously in my setup.

 

Kevan

 

Greetings,

 

Recently I updated my daos repo to master top of tree, and now any attempt to create a container causes the access-point daos_io_server to segfault.   Before I dig deeply, is this a known issue?  My config is one client node plus one server node with dual daos_io_servers.  Before running this test the server storage was reformatted.

 

Commands on the client:

 

[root@delphi-005 tmp]# dmg -i -l delphi-004 system list-pools

delphi-004:10001: connected

No pools in system

[root@delphi-005 tmp]# dmg -i -l delphi-004 pool create --scm-size=768G --nvme-size=10T

delphi-004:10001: connected

Pool-create command SUCCEEDED: UUID: 9acb0a19-2ecf-4d3f-8f7a-2afcec26128f, Service replicas: 0

[root@delphi-005 tmp]# dmg -i -l delphi-004 system list-pools

delphi-004:10001: connected

Pool UUID                            Svc Replicas 

---------                            ------------ 

9acb0a19-2ecf-4d3f-8f7a-2afcec26128f 0            

[root@delphi-005 tmp]# daos container create --pool=9acb0a19-2ecf-4d3f-8f7a-2afcec26128f --svc=0

 

At the point the client window hangs, and the daos_io_server setfaults.  Back trace collected via gdb is:

 

Program received signal SIGSEGV, Segmentation fault.

[Switching to Thread 0x7f37bcdfd700 (LWP 22203)]

0x00007f37cdd2a0a8 in field_is_zeroish (member=member@entry=0x2aaab0423678, field=<optimized out>) at protobuf-c/protobuf-c.c:559

559                  ret = (NULL == *(const char * const *) member) ||

(gdb) bt

#0  0x00007f37cdd2a0a8 in field_is_zeroish (member=member@entry=0x2aaab0423678, field=<optimized out>) at protobuf-c/protobuf-c.c:559

#1  0x00007f37cdd2aa53 in unlabeled_field_get_packed_size (member=0x2aaab0423678, field=0x7f37cf1d4e18 <mgmt__get_attach_info_resp__field_descriptors+216>)

    at protobuf-c/protobuf-c.c:591

#2  protobuf_c_message_get_packed_size (message=message@entry=0x2aaab0423640) at protobuf-c/protobuf-c.c:739

#3  0x00007f37cef93d31 in mgmt__get_attach_info_resp__get_packed_size (message=message@entry=0x2aaab0423640) at src/mgmt/srv.pb-c.c:296

#4  0x00007f37c6998d4a in ds_mgmt_drpc_get_attach_info (drpc_req=<optimized out>, drpc_resp=0x7f3770026a10) at src/mgmt/srv_drpc.c:239

#5  0x000000000040beb5 in drpc_handler_ult (call_ctx=0x7f3770026990) at src/iosrv/drpc_progress.c:297

#6  0x00007f37ce3c317b in ABTD_thread_func_wrapper_thread () from /home/users/daos/daos/install/lib/libabt.so.0

#7  0x00007f37ce3c3851 in make_fcontext () from /home/users/daos/daos/install/lib/libabt.so.0

#8  0x0000000000000000 in ?? ()

(gdb) p member

$1 = (const void *) 0x2aaab0423678

(gdb) p *(const char * const *) member

$3 = 0xb801e74ea7845500 <Address 0xb801e74ea7845500 out of bounds>

 

Is this a known problem?

 

Thanks, Kevan

Joel,

 

I am still chasing this.   Problem is occurring in the server in routine ds_mgmt_drpc_get_attach_info.  Routine ds_mgmt_get_attach_info_handler() fills in ‘resp’ with nsprs and the psrs array.  Then this routine fills in resp.status and calls mgmg__get_attach_info_resp___get_packed_size().  It is in that routine that the segfault occurs.   The struct is _Mgmt__GetAttachInfoResp, there are other fields that are not being filled in, and the segfault occurs on one of these, ‘interface’.   The MGMT__GET_ATTACH_INFO_RESP__INIT macro at the beginning of function ds_mgmt_drpc_get_attach_info appears to set all the string fields to “”, but by the time the code gets to the ‘interface’ parameter in mgmt__get_attach_info_resp__get_packed_size it contains some out-of-range value that causes the segfault.

 

I don’t really understand the packing code, just giving you these tidbits until I can dig further tomorrow.

 

Kevan

 

 

Joel,

 

I’m curious; how does it help a client to know the interface and domain names of this server?   I can’t see how the client could possibly use them.

 

Anyway, back to the problem.   I am breakpointed in ds_mgmt_drpc_get_attach_info().   At the top of the routine is this:

 

        Mgmt__GetAttachInfoResp  resp = MGMT__GET_ATTACH_INFO_RESP__INIT;

 

If I look in the code at the definition of Mgmt__GetAttachInfoResp it has the 7 data fields including your new interface field, etc.  And the value of MGMT__GET_ATTACH_INFO_RESP__INIT appears to initialize all 7 of those fields.   But if I use gdb to look at that structure you can see that the code doesn’t actually know about any of the new fields, it is only aware of status and n_psrs/psrs:

 

(gdb) p resp

$7 = {base = {descriptor = 0x7f9756b5dcc0 <mgmt__get_attach_info_resp__descriptor>, n_unknown_fields = 0, unknown_fields = 0x0}, status = 0, n_psrs = 0, psrs = 0x0}

(gdb) p resp.status

$8 = 0

(gdb) p resp.n_psrs

$9 = 0

(gdb) p resp.psrs

$10 = (Mgmt__GetAttachInfoResp__Psr **) 0x0

(gdb) p resp.provider

There is no member named provider.

(gdb) p resp.interface

There is no member named interface.

(gdb) p resp.domain

There is no member named domain.

 

(gdb) p sizeof(resp)

$13 = 48

 

If you do the math, you can see that the size of ‘resp’ is correct if the struct ends with field psrs, there is no room in the struct for the new fields.

 

If I then step forward and enter routine mgmt__get_attach_info_resp__get_packed_size(), that routine DOES know about 7 fields and tries to reference all of them, but of course the resp structure on the stack isn’t big enough to hold the 7 fields, so this routine is looking at other junk on the stack past the end of the structure:

 

239           len = mgmt__get_attach_info_resp__get_packed_size(&resp);

(gdb) s

mgmt__get_attach_info_resp__get_packed_size (message=message@entry=0x2aaab041f740) at src/mgmt/srv.pb-c.c:295

295      assert(message->base.descriptor == &mgmt__get_attach_info_resp__descriptor);

(gdb) n

296      return protobuf_c_message_get_packed_size ((const ProtobufCMessage*)(message));

(gdb) p message

$11 = (const Mgmt__GetAttachInfoResp *) 0x2aaab041f740

(gdb) p *message

$12 = {base = {descriptor = 0x7f9756b5dcc0 <mgmt__get_attach_info_resp__descriptor>, n_unknown_fields = 0, unknown_fields = 0x0}, status = 0, n_psrs = 1, 

  psrs = 0x7f9396113e00, provider = 0x0, interface = 0xad26fa6a89442100 <Address 0xad26fa6a89442100 out of bounds>, domain = 0x7f96f4026a10 "\340\230VW\227\177", 

  crtctxshareaddr = 4093798800, crttimeout = 32662}

 

Happy hunting,

 

Kevan

 

 

That’s good information.  I added the “interface” field and some others last week as we are expanding the capabilities of the GetAttachInfo message to help automatically configure the clients. It’s not clear why adding the fields would cause any of the unpacking code to fail, especially when it’s auto generated based on the protobuf definition.  However, I did build those files with newer versions of the protobuf compiler, and it’s possible that there’s a subtle incompatibility that I wasn’t aware of. 

Hi Kevan,

 

You are right that it won’t help a client to know the interface and domain names of the server.  In this case, we’re not actually sending the server’s interface and domain in the server’s response.  These fields are left empty until they are populated by the agent.  On the update I am working on now, the agent scans the client machine for network interfaces that support the server’s provider (based on the GetAttachInfo provider data) and populates the interface and domain fields in the response sent to the client.  In an update after that, the libdaos library then gets some rework to generate a GetAttachInfo prior to initializing CaRT so that it can use the interface and domain data that’s returned to it.  I’m working on getting this through review now.

 

Thanks for the additional debug log.  I appreciate your insight and help.  I will work on replicating the problem locally so I can fix it.

 

Joel

 

 

Joel,

 

I’m curious; how does it help a client to know the interface and domain names of this server?   I can’t see how the client could possibly use them.

 

Anyway, back to the problem.   I am breakpointed in ds_mgmt_drpc_get_attach_info().   At the top of the routine is this:

 

        Mgmt__GetAttachInfoResp  resp = MGMT__GET_ATTACH_INFO_RESP__INIT;

 

If I look in the code at the definition of Mgmt__GetAttachInfoResp it has the 7 data fields including your new interface field, etc.  And the value of MGMT__GET_ATTACH_INFO_RESP__INIT appears to initialize all 7 of those fields.   But if I use gdb to look at that structure you can see that the code doesn’t actually know about any of the new fields, it is only aware of status and n_psrs/psrs:

 

(gdb) p resp

$7 = {base = {descriptor = 0x7f9756b5dcc0 <mgmt__get_attach_info_resp__descriptor>, n_unknown_fields = 0, unknown_fields = 0x0}, status = 0, n_psrs = 0, psrs = 0x0}

(gdb) p resp.status

$8 = 0

(gdb) p resp.n_psrs

$9 = 0

(gdb) p resp.psrs

$10 = (Mgmt__GetAttachInfoResp__Psr **) 0x0

(gdb) p resp.provider

There is no member named provider.

(gdb) p resp.interface

There is no member named interface.

(gdb) p resp.domain

There is no member named domain.

 

(gdb) p sizeof(resp)

$13 = 48

 

If you do the math, you can see that the size of ‘resp’ is correct if the struct ends with field psrs, there is no room in the struct for the new fields.

 

If I then step forward and enter routine mgmt__get_attach_info_resp__get_packed_size(), that routine DOES know about 7 fields and tries to reference all of them, but of course the resp structure on the stack isn’t big enough to hold the 7 fields, so this routine is looking at other junk on the stack past the end of the structure:

 

239           len = mgmt__get_attach_info_resp__get_packed_size(&resp);

(gdb) s

mgmt__get_attach_info_resp__get_packed_size (message=message@entry=0x2aaab041f740) at src/mgmt/srv.pb-c.c:295

295      assert(message->base.descriptor == &mgmt__get_attach_info_resp__descriptor);

(gdb) n

296      return protobuf_c_message_get_packed_size ((const ProtobufCMessage*)(message));

(gdb) p message

$11 = (const Mgmt__GetAttachInfoResp *) 0x2aaab041f740

(gdb) p *message

$12 = {base = {descriptor = 0x7f9756b5dcc0 <mgmt__get_attach_info_resp__descriptor>, n_unknown_fields = 0, unknown_fields = 0x0}, status = 0, n_psrs = 1, 

  psrs = 0x7f9396113e00, provider = 0x0, interface = 0xad26fa6a89442100 <Address 0xad26fa6a89442100 out of bounds>, domain = 0x7f96f4026a10 "\340\230VW\227\177", 

  crtctxshareaddr = 4093798800, crttimeout = 32662}

 

Happy hunting,

 

Kevan

 

 

That’s good information.  I added the “interface” field and some others last week as we are expanding the capabilities of the GetAttachInfo message to help automatically configure the clients. It’s not clear why adding the fields would cause any of the unpacking code to fail, especially when it’s auto generated based on the protobuf definition.  However, I did build those files with newer versions of the protobuf compiler, and it’s possible that there’s a subtle incompatibility that I wasn’t aware of. 

Joel,

 

Thanks for the explanation below, makes sense.   Can’t wait for that code to land.

 

Back to the problem at hand;  now I am even more confused….   I borrowed one of my compatriot’s machines, breakpointed his daos_io_server in routine ds_mgmt_drpc_get_attach_info, in his daemon the resp structure has all 7 fields, and so he doesn’t get a segfault.   We are building with the same commit  point.   ????? 

 

Do you have any ideas on what could be different in my machine?   Same centos 7 release.   I will keep debugging, it again appears to be related to my environment somehow.

 

Kevan

 

Hi Kevan,

 

You are right that it won’t help a client to know the interface and domain names of the server.  In this case, we’re not actually sending the server’s interface and domain in the server’s response.  These fields are left empty until they are populated by the agent.  On the update I am working on now, the agent scans the client machine for network interfaces that support the server’s provider (based on the GetAttachInfo provider data) and populates the interface and domain fields in the response sent to the client.  In an update after that, the libdaos library then gets some rework to generate a GetAttachInfo prior to initializing CaRT so that it can use the interface and domain data that’s returned to it.  I’m working on getting this through review now.

 

Thanks for the additional debug log.  I appreciate your insight and help.  I will work on replicating the problem locally so I can fix it.

 

Joel

 

 

Joel,

 

I’m curious; how does it help a client to know the interface and domain names of this server?   I can’t see how the client could possibly use them.

 

Anyway, back to the problem.   I am breakpointed in ds_mgmt_drpc_get_attach_info().   At the top of the routine is this:

 

        Mgmt__GetAttachInfoResp  resp = MGMT__GET_ATTACH_INFO_RESP__INIT;

 

If I look in the code at the definition of Mgmt__GetAttachInfoResp it has the 7 data fields including your new interface field, etc.  And the value of MGMT__GET_ATTACH_INFO_RESP__INIT appears to initialize all 7 of those fields.   But if I use gdb to look at that structure you can see that the code doesn’t actually know about any of the new fields, it is only aware of status and n_psrs/psrs:

 

(gdb) p resp

$7 = {base = {descriptor = 0x7f9756b5dcc0 <mgmt__get_attach_info_resp__descriptor>, n_unknown_fields = 0, unknown_fields = 0x0}, status = 0, n_psrs = 0, psrs = 0x0}

(gdb) p resp.status

$8 = 0

(gdb) p resp.n_psrs

$9 = 0

(gdb) p resp.psrs

$10 = (Mgmt__GetAttachInfoResp__Psr **) 0x0

(gdb) p resp.provider

There is no member named provider.

(gdb) p resp.interface

There is no member named interface.

(gdb) p resp.domain

There is no member named domain.

 

(gdb) p sizeof(resp)

$13 = 48

 

If you do the math, you can see that the size of ‘resp’ is correct if the struct ends with field psrs, there is no room in the struct for the new fields.

 

If I then step forward and enter routine mgmt__get_attach_info_resp__get_packed_size(), that routine DOES know about 7 fields and tries to reference all of them, but of course the resp structure on the stack isn’t big enough to hold the 7 fields, so this routine is looking at other junk on the stack past the end of the structure:

 

239           len = mgmt__get_attach_info_resp__get_packed_size(&resp);

(gdb) s

mgmt__get_attach_info_resp__get_packed_size (message=message@entry=0x2aaab041f740) at src/mgmt/srv.pb-c.c:295

295      assert(message->base.descriptor == &mgmt__get_attach_info_resp__descriptor);

(gdb) n

296      return protobuf_c_message_get_packed_size ((const ProtobufCMessage*)(message));

(gdb) p message

$11 = (const Mgmt__GetAttachInfoResp *) 0x2aaab041f740

(gdb) p *message

$12 = {base = {descriptor = 0x7f9756b5dcc0 <mgmt__get_attach_info_resp__descriptor>, n_unknown_fields = 0, unknown_fields = 0x0}, status = 0, n_psrs = 1, 

  psrs = 0x7f9396113e00, provider = 0x0, interface = 0xad26fa6a89442100 <Address 0xad26fa6a89442100 out of bounds>, domain = 0x7f96f4026a10 "\340\230VW\227\177", 

  crtctxshareaddr = 4093798800, crttimeout = 32662}

 

Happy hunting,

 

Kevan

 

 

That’s good information.  I added the “interface” field and some others last week as we are expanding the capabilities of the GetAttachInfo message to help automatically configure the clients. It’s not clear why adding the fields would cause any of the unpacking code to fail, especially when it’s auto generated based on the protobuf definition.  However, I did build those files with newer versions of the protobuf compiler, and it’s possible that there’s a subtle incompatibility that I wasn’t aware of. 

Hi Kevan,

 

I ran your test locally in my environment on master and it encountered no issues.  Figures, right?   I spent some time looking at the auto-generated code to see what it is doing.  I don’t have any particular expertise in that.  But, it’s clear that if any of the parts are out of sync, it will not work.  I’m wondering if you have any stale protobuf files on your machine.  Can you diff the protobuf files on your machine against 1) daos master, and 2) your coworker’s machine to see if all three sets are equal?

 

There is a full list in src/proto/Makefile.  Of particular interest are these three:

 

src/mgmt/srv.pb-c.c

src/mgmt/srv.pb-c.h

src/control/common/proto/mgmt/srv.pb.go

 

Joel

 

 

Joel,

 

Thanks for the explanation below, makes sense.   Can’t wait for that code to land.

 

Back to the problem at hand;  now I am even more confused….   I borrowed one of my compatriot’s machines, breakpointed his daos_io_server in routine ds_mgmt_drpc_get_attach_info, in his daemon the resp structure has all 7 fields, and so he doesn’t get a segfault.   We are building with the same commit  point.   ????? 

 

Do you have any ideas on what could be different in my machine?   Same centos 7 release.   I will keep debugging, it again appears to be related to my environment somehow.

 

Kevan

 

Hi Kevan,

 

You are right that it won’t help a client to know the interface and domain names of the server.  In this case, we’re not actually sending the server’s interface and domain in the server’s response.  These fields are left empty until they are populated by the agent.  On the update I am working on now, the agent scans the client machine for network interfaces that support the server’s provider (based on the GetAttachInfo provider data) and populates the interface and domain fields in the response sent to the client.  In an update after that, the libdaos library then gets some rework to generate a GetAttachInfo prior to initializing CaRT so that it can use the interface and domain data that’s returned to it.  I’m working on getting this through review now.

 

Thanks for the additional debug log.  I appreciate your insight and help.  I will work on replicating the problem locally so I can fix it.

 

Joel

 

 

Joel,

 

I’m curious; how does it help a client to know the interface and domain names of this server?   I can’t see how the client could possibly use them.

 

Anyway, back to the problem.   I am breakpointed in ds_mgmt_drpc_get_attach_info().   At the top of the routine is this:

 

        Mgmt__GetAttachInfoResp  resp = MGMT__GET_ATTACH_INFO_RESP__INIT;

 

If I look in the code at the definition of Mgmt__GetAttachInfoResp it has the 7 data fields including your new interface field, etc.  And the value of MGMT__GET_ATTACH_INFO_RESP__INIT appears to initialize all 7 of those fields.   But if I use gdb to look at that structure you can see that the code doesn’t actually know about any of the new fields, it is only aware of status and n_psrs/psrs:

 

(gdb) p resp

$7 = {base = {descriptor = 0x7f9756b5dcc0 <mgmt__get_attach_info_resp__descriptor>, n_unknown_fields = 0, unknown_fields = 0x0}, status = 0, n_psrs = 0, psrs = 0x0}

(gdb) p resp.status

$8 = 0

(gdb) p resp.n_psrs

$9 = 0

(gdb) p resp.psrs

$10 = (Mgmt__GetAttachInfoResp__Psr **) 0x0

(gdb) p resp.provider

There is no member named provider.

(gdb) p resp.interface

There is no member named interface.

(gdb) p resp.domain

There is no member named domain.

 

(gdb) p sizeof(resp)

$13 = 48

 

If you do the math, you can see that the size of ‘resp’ is correct if the struct ends with field psrs, there is no room in the struct for the new fields.

 

If I then step forward and enter routine mgmt__get_attach_info_resp__get_packed_size(), that routine DOES know about 7 fields and tries to reference all of them, but of course the resp structure on the stack isn’t big enough to hold the 7 fields, so this routine is looking at other junk on the stack past the end of the structure:

 

239           len = mgmt__get_attach_info_resp__get_packed_size(&resp);

(gdb) s

mgmt__get_attach_info_resp__get_packed_size (message=message@entry=0x2aaab041f740) at src/mgmt/srv.pb-c.c:295

295      assert(message->base.descriptor == &mgmt__get_attach_info_resp__descriptor);

(gdb) n

296      return protobuf_c_message_get_packed_size ((const ProtobufCMessage*)(message));

(gdb) p message

$11 = (const Mgmt__GetAttachInfoResp *) 0x2aaab041f740

(gdb) p *message

$12 = {base = {descriptor = 0x7f9756b5dcc0 <mgmt__get_attach_info_resp__descriptor>, n_unknown_fields = 0, unknown_fields = 0x0}, status = 0, n_psrs = 1, 

  psrs = 0x7f9396113e00, provider = 0x0, interface = 0xad26fa6a89442100 <Address 0xad26fa6a89442100 out of bounds>, domain = 0x7f96f4026a10 "\340\230VW\227\177", 

  crtctxshareaddr = 4093798800, crttimeout = 32662}

 

Happy hunting,

 

Kevan

 

 

That’s good information.  I added the “interface” field and some others last week as we are expanding the capabilities of the GetAttachInfo message to help automatically configure the clients. It’s not clear why adding the fields would cause any of the unpacking code to fail, especially when it’s auto generated based on the protobuf definition.  However, I did build those files with newer versions of the protobuf compiler, and it’s possible that there’s a subtle incompatibility that I wasn’t aware of. 

Joel,

 

I came to the same conclusion, sorry for wasting your time.  

 

There is currently an issue where the daos_io_server dies immediately because it can’t find its own librdb.so module, which got moved into lib64/daos_srv.   If I move librdb.so to lib then it complains about other modules.  What is the correct way to configure for this?  

 

04/13-04:20:00.47 delphi-004 DAOS[70088/70088] server ERR  src/iosrv/module.c:105 dss_module_load() cannot load librdb.so: librdb.so: cannot open shared object file: No such file or directory

04/13-04:20:00.47 delphi-004 DAOS[70088/70088] server ERR  src/iosrv/init.c:195 modules_load() Failed to load module rdb: -1003

 

To work around this, I set LD_LIBRARY_PATH in the environ section of daos_server.yml to include all library-related subdirectories within the install tree.   And to get the install_dir pushed out to all the server nodes I use rsync.   By default rsync doesn’t delete files at the destination if they are not in the source, so as libraries move around in the install tree over time, I eventually ended up with two copies of the same .so in different directories, and the LD_LIBRARY_PATH resulted in the wrong one being picked.

 

Sorry, Kevan

 

Hi Kevan,

 

I ran your test locally in my environment on master and it encountered no issues.  Figures, right?   I spent some time looking at the auto-generated code to see what it is doing.  I don’t have any particular expertise in that.  But, it’s clear that if any of the parts are out of sync, it will not work.  I’m wondering if you have any stale protobuf files on your machine.  Can you diff the protobuf files on your machine against 1) daos master, and 2) your coworker’s machine to see if all three sets are equal?

 

There is a full list in src/proto/Makefile.  Of particular interest are these three:

 

src/mgmt/srv.pb-c.c

src/mgmt/srv.pb-c.h

src/control/common/proto/mgmt/srv.pb.go

 

Joel

 

 

Joel,

 

Thanks for the explanation below, makes sense.   Can’t wait for that code to land.

 

Back to the problem at hand;  now I am even more confused….   I borrowed one of my compatriot’s machines, breakpointed his daos_io_server in routine ds_mgmt_drpc_get_attach_info, in his daemon the resp structure has all 7 fields, and so he doesn’t get a segfault.   We are building with the same commit  point.   ????? 

 

Do you have any ideas on what could be different in my machine?   Same centos 7 release.   I will keep debugging, it again appears to be related to my environment somehow.

 

Kevan

 

Hi Kevan,

 

You are right that it won’t help a client to know the interface and domain names of the server.  In this case, we’re not actually sending the server’s interface and domain in the server’s response.  These fields are left empty until they are populated by the agent.  On the update I am working on now, the agent scans the client machine for network interfaces that support the server’s provider (based on the GetAttachInfo provider data) and populates the interface and domain fields in the response sent to the client.  In an update after that, the libdaos library then gets some rework to generate a GetAttachInfo prior to initializing CaRT so that it can use the interface and domain data that’s returned to it.  I’m working on getting this through review now.

 

Thanks for the additional debug log.  I appreciate your insight and help.  I will work on replicating the problem locally so I can fix it.

 

Joel

 

 

Joel,

 

I’m curious; how does it help a client to know the interface and domain names of this server?   I can’t see how the client could possibly use them.

 

Anyway, back to the problem.   I am breakpointed in ds_mgmt_drpc_get_attach_info().   At the top of the routine is this:

 

        Mgmt__GetAttachInfoResp  resp = MGMT__GET_ATTACH_INFO_RESP__INIT;

 

If I look in the code at the definition of Mgmt__GetAttachInfoResp it has the 7 data fields including your new interface field, etc.  And the value of MGMT__GET_ATTACH_INFO_RESP__INIT appears to initialize all 7 of those fields.   But if I use gdb to look at that structure you can see that the code doesn’t actually know about any of the new fields, it is only aware of status and n_psrs/psrs:

 

(gdb) p resp

$7 = {base = {descriptor = 0x7f9756b5dcc0 <mgmt__get_attach_info_resp__descriptor>, n_unknown_fields = 0, unknown_fields = 0x0}, status = 0, n_psrs = 0, psrs = 0x0}

(gdb) p resp.status

$8 = 0

(gdb) p resp.n_psrs

$9 = 0

(gdb) p resp.psrs

$10 = (Mgmt__GetAttachInfoResp__Psr **) 0x0

(gdb) p resp.provider

There is no member named provider.

(gdb) p resp.interface

There is no member named interface.

(gdb) p resp.domain

There is no member named domain.

 

(gdb) p sizeof(resp)

$13 = 48

 

If you do the math, you can see that the size of ‘resp’ is correct if the struct ends with field psrs, there is no room in the struct for the new fields.

 

If I then step forward and enter routine mgmt__get_attach_info_resp__get_packed_size(), that routine DOES know about 7 fields and tries to reference all of them, but of course the resp structure on the stack isn’t big enough to hold the 7 fields, so this routine is looking at other junk on the stack past the end of the structure:

 

239           len = mgmt__get_attach_info_resp__get_packed_size(&resp);

(gdb) s

mgmt__get_attach_info_resp__get_packed_size (message=message@entry=0x2aaab041f740) at src/mgmt/srv.pb-c.c:295

295      assert(message->base.descriptor == &mgmt__get_attach_info_resp__descriptor);

(gdb) n

296      return protobuf_c_message_get_packed_size ((const ProtobufCMessage*)(message));

(gdb) p message

$11 = (const Mgmt__GetAttachInfoResp *) 0x2aaab041f740

(gdb) p *message

$12 = {base = {descriptor = 0x7f9756b5dcc0 <mgmt__get_attach_info_resp__descriptor>, n_unknown_fields = 0, unknown_fields = 0x0}, status = 0, n_psrs = 1, 

  psrs = 0x7f9396113e00, provider = 0x0, interface = 0xad26fa6a89442100 <Address 0xad26fa6a89442100 out of bounds>, domain = 0x7f96f4026a10 "\340\230VW\227\177", 

  crtctxshareaddr = 4093798800, crttimeout = 32662}

 

Happy hunting,

 

Kevan

 

 

That’s good information.  I added the “interface” field and some others last week as we are expanding the capabilities of the GetAttachInfo message to help automatically configure the clients. It’s not clear why adding the fields would cause any of the unpacking code to fail, especially when it’s auto generated based on the protobuf definition.  However, I did build those files with newer versions of the protobuf compiler, and it’s possible that there’s a subtle incompatibility that I wasn’t aware of. 

Hi Kevan,

 

Are you building daos from scratch?

 

You should not need to set LD_LIBRARY_PATH to run the IO server in such a setup because daos_io_server is built using RPATH.   Can you do an ldd on daos_io_server ?   If you do use LD_LIBRARY_PATH, make sure lib64 comes before any occurrence of lib

 

-Jeff

 

Joel,

 

I came to the same conclusion, sorry for wasting your time.  

 

There is currently an issue where the daos_io_server dies immediately because it can’t find its own librdb.so module, which got moved into lib64/daos_srv.   If I move librdb.so to lib then it complains about other modules.  What is the correct way to configure for this?  

 

04/13-04:20:00.47 delphi-004 DAOS[70088/70088] server ERR  src/iosrv/module.c:105 dss_module_load() cannot load librdb.so: librdb.so: cannot open shared object file: No such file or directory

04/13-04:20:00.47 delphi-004 DAOS[70088/70088] server ERR  src/iosrv/init.c:195 modules_load() Failed to load module rdb: -1003

 

To work around this, I set LD_LIBRARY_PATH in the environ section of daos_server.yml to include all library-related subdirectories within the install tree.   And to get the install_dir pushed out to all the server nodes I use rsync.   By default rsync doesn’t delete files at the destination if they are not in the source, so as libraries move around in the install tree over time, I eventually ended up with two copies of the same .so in different directories, and the LD_LIBRARY_PATH resulted in the wrong one being picked.

 

Sorry, Kevan

 

Hi Kevan,

 

I ran your test locally in my environment on master and it encountered no issues.  Figures, right?   I spent some time looking at the auto-generated code to see what it is doing.  I don’t have any particular expertise in that.  But, it’s clear that if any of the parts are out of sync, it will not work.  I’m wondering if you have any stale protobuf files on your machine.  Can you diff the protobuf files on your machine against 1) daos master, and 2) your coworker’s machine to see if all three sets are equal?

 

There is a full list in src/proto/Makefile.  Of particular interest are these three:

 

src/mgmt/srv.pb-c.c

src/mgmt/srv.pb-c.h

src/control/common/proto/mgmt/srv.pb.go

 

Joel

 

 

Joel,

 

Thanks for the explanation below, makes sense.   Can’t wait for that code to land.

 

Back to the problem at hand;  now I am even more confused….   I borrowed one of my compatriot’s machines, breakpointed his daos_io_server in routine ds_mgmt_drpc_get_attach_info, in his daemon the resp structure has all 7 fields, and so he doesn’t get a segfault.   We are building with the same commit  point.   ????? 

 

Do you have any ideas on what could be different in my machine?   Same centos 7 release.   I will keep debugging, it again appears to be related to my environment somehow.

 

Kevan

 

Hi Kevan,

 

You are right that it won’t help a client to know the interface and domain names of the server.  In this case, we’re not actually sending the server’s interface and domain in the server’s response.  These fields are left empty until they are populated by the agent.  On the update I am working on now, the agent scans the client machine for network interfaces that support the server’s provider (based on the GetAttachInfo provider data) and populates the interface and domain fields in the response sent to the client.  In an update after that, the libdaos library then gets some rework to generate a GetAttachInfo prior to initializing CaRT so that it can use the interface and domain data that’s returned to it.  I’m working on getting this through review now.

 

Thanks for the additional debug log.  I appreciate your insight and help.  I will work on replicating the problem locally so I can fix it.

 

Joel

 

 

Joel,

 

I’m curious; how does it help a client to know the interface and domain names of this server?   I can’t see how the client could possibly use them.

 

Anyway, back to the problem.   I am breakpointed in ds_mgmt_drpc_get_attach_info().   At the top of the routine is this:

 

        Mgmt__GetAttachInfoResp  resp = MGMT__GET_ATTACH_INFO_RESP__INIT;

 

If I look in the code at the definition of Mgmt__GetAttachInfoResp it has the 7 data fields including your new interface field, etc.  And the value of MGMT__GET_ATTACH_INFO_RESP__INIT appears to initialize all 7 of those fields.   But if I use gdb to look at that structure you can see that the code doesn’t actually know about any of the new fields, it is only aware of status and n_psrs/psrs:

 

(gdb) p resp

$7 = {base = {descriptor = 0x7f9756b5dcc0 <mgmt__get_attach_info_resp__descriptor>, n_unknown_fields = 0, unknown_fields = 0x0}, status = 0, n_psrs = 0, psrs = 0x0}

(gdb) p resp.status

$8 = 0

(gdb) p resp.n_psrs

$9 = 0

(gdb) p resp.psrs

$10 = (Mgmt__GetAttachInfoResp__Psr **) 0x0

(gdb) p resp.provider

There is no member named provider.

(gdb) p resp.interface

There is no member named interface.

(gdb) p resp.domain

There is no member named domain.

 

(gdb) p sizeof(resp)

$13 = 48

 

If you do the math, you can see that the size of ‘resp’ is correct if the struct ends with field psrs, there is no room in the struct for the new fields.

 

If I then step forward and enter routine mgmt__get_attach_info_resp__get_packed_size(), that routine DOES know about 7 fields and tries to reference all of them, but of course the resp structure on the stack isn’t big enough to hold the 7 fields, so this routine is looking at other junk on the stack past the end of the structure:

 

239           len = mgmt__get_attach_info_resp__get_packed_size(&resp);

(gdb) s

mgmt__get_attach_info_resp__get_packed_size (message=message@entry=0x2aaab041f740) at src/mgmt/srv.pb-c.c:295

295      assert(message->base.descriptor == &mgmt__get_attach_info_resp__descriptor);

(gdb) n

296      return protobuf_c_message_get_packed_size ((const ProtobufCMessage*)(message));

(gdb) p message

$11 = (const Mgmt__GetAttachInfoResp *) 0x2aaab041f740

(gdb) p *message

$12 = {base = {descriptor = 0x7f9756b5dcc0 <mgmt__get_attach_info_resp__descriptor>, n_unknown_fields = 0, unknown_fields = 0x0}, status = 0, n_psrs = 1, 

  psrs = 0x7f9396113e00, provider = 0x0, interface = 0xad26fa6a89442100 <Address 0xad26fa6a89442100 out of bounds>, domain = 0x7f96f4026a10 "\340\230VW\227\177", 

  crtctxshareaddr = 4093798800, crttimeout = 32662}

 

Happy hunting,

 

Kevan

 

 

That’s good information.  I added the “interface” field and some others last week as we are expanding the capabilities of the GetAttachInfo message to help automatically configure the clients. It’s not clear why adding the fields would cause any of the unpacking code to fail, especially when it’s auto generated based on the protobuf definition.  However, I did build those files with newer versions of the protobuf compiler, and it’s possible that there’s a subtle incompatibility that I wasn’t aware of. 

Jeff,

 

Yes I am building from scratch, and RPATH used to work.   Here is the output, note that librdb doesn’t show up in the ldd output.

 

Thanks, Kevan

 

[root@delphi-004 ~]# cd ~daos/daos/install/bin

[root@delphi-004 bin]# ls

acl_dump_test  daosctl           daos_test        drpc_test    igzip                pmreorder           spdk_nvme_perf

agent_tests    daos_gen_io_conf  dcont            eq_tests     io_conf              rdbt                spdk_tgt

cart_ctl       daos_io_server    dfuse            evt_ctl      jump_pl_map          ring_pl_map         vea_ut

crt_launch     daos_perf         dfuse_hl         fi_info      nvme_control_ctests  security_test       vos_size

daos           daos_racer        dmg              fi_pingpong  obj_ctl              self_test           vos_size.py

daos_admin     daos_run_io_conf  dmg_old          fi_strerror  pl_bench             smd_ut              vos_tests

daos_agent     daos_server       drpc_iosrv_test  hello_drpc   pmempool             spdk_nvme_identify

[root@delphi-004 bin]# ldd daos_io_server

       linux-vdso.so.1 =>  (0x00007ffc47293000)

       libdaos_common.so => /home/users/daos/daos/install/lib64/libdaos_common.so (0x00007f69369cd000)

       libgurt.so.4 => /home/users/daos/daos/install/lib64/libgurt.so.4 (0x00007f69367aa000)

       libcart.so.4 => /home/users/daos/daos/install/lib64/libcart.so.4 (0x00007f69364c9000)

       libvos_srv.so => /home/users/daos/daos/install/bin/./../lib64/daos_srv/libvos_srv.so (0x00007f693622c000)

       libdaos.so.0 => /home/users/daos/daos/install/lib64/libdaos.so.0 (0x00007f6935f51000)

       libbio.so => /home/users/daos/daos/install/bin/./../lib64/daos_srv/libbio.so (0x00007f6935d26000)

       libdl.so.2 => /lib64/libdl.so.2 (0x00007f6935b22000)

       libuuid.so.1 => /lib64/libuuid.so.1 (0x00007f693591d000)

       libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f6935701000)

       libabt.so.0 => /home/users/daos/daos/install/lib/libabt.so.0 (0x00007f69354df000)

       libhwloc.so.5 => /lib64/libhwloc.so.5 (0x00007f69352a2000)

       libpmemobj.so.1 => /home/users/daos/daos/install/lib/libpmemobj.so.1 (0x00007f6935061000)

       libprotobuf-c.so.1 => /home/users/daos/daos/install/lib/libprotobuf-c.so.1 (0x00007f6934e58000)

       libc.so.6 => /lib64/libc.so.6 (0x00007f6934a8a000)

       libisal.so.2 => /home/users/daos/daos/install/lib/libisal.so.2 (0x00007f693484c000)

       /lib64/ld-linux-x86-64.so.2 (0x00007f6936c3c000)

       libyaml-0.so.2 => /lib64/libyaml-0.so.2 (0x00007f693462c000)

       libmercury.so.2 => /home/users/daos/daos/install/lib/libmercury.so.2 (0x00007f6934412000)

       libna.so.2 => /home/users/daos/daos/install/lib/libna.so.2 (0x00007f69341f4000)

       libmercury_util.so.2 => /home/users/daos/daos/install/lib/libmercury_util.so.2 (0x00007f6933fed000)

       libspdk_env_dpdk.so.2.0 => /home/users/daos/daos/install/lib/libspdk_env_dpdk.so.2.0 (0x00007f6933dde000)

       libspdk_thread.so.2.0 => /home/users/daos/daos/install/lib/libspdk_thread.so.2.0 (0x00007f6933bd8000)

       libspdk_bdev.so.2.0 => /home/users/daos/daos/install/lib/libspdk_bdev.so.2.0 (0x00007f69339c2000)

       libspdk_copy.so.2.0 => /home/users/daos/daos/install/lib/libspdk_copy.so.2.0 (0x00007f69337bf000)

       librte_mempool.so.20.0 => /home/users/daos/daos/install/lib/librte_mempool.so.20.0 (0x00007f69335b9000)

       librte_mempool_ring.so.20.0 => /home/users/daos/daos/install/lib/librte_mempool_ring.so.20.0 (0x00007f69333b6000)

       librte_bus_pci.so.20.0 => /home/users/daos/daos/install/lib/librte_bus_pci.so.20.0 (0x00007f69331ab000)

       librte_pci.so.20.0 => /home/users/daos/daos/install/lib/librte_pci.so.20.0 (0x00007f6932fa8000)

       librte_ring.so.20.0 => /home/users/daos/daos/install/lib/librte_ring.so.20.0 (0x00007f6932da5000)

       librte_mbuf.so.20.0 => /home/users/daos/daos/install/lib/librte_mbuf.so.20.0 (0x00007f6932b9a000)

       librte_eal.so.20.0 => /home/users/daos/daos/install/lib/librte_eal.so.20.0 (0x00007f69328bf000)

       librte_kvargs.so.20.0 => /home/users/daos/daos/install/lib/librte_kvargs.so.20.0 (0x00007f69326bc000)

       libspdk_bdev_aio.so.2.0 => /home/users/daos/daos/install/lib/libspdk_bdev_aio.so.2.0 (0x00007f69324b6000)

       libspdk_bdev_nvme.so.2.0 => /home/users/daos/daos/install/lib/libspdk_bdev_nvme.so.2.0 (0x00007f693229d000)

       libspdk_bdev_malloc.so.2.0 => /home/users/daos/daos/install/lib/libspdk_bdev_malloc.so.2.0 (0x00007f6932098000)

       libspdk_conf.so.2.0 => /home/users/daos/daos/install/lib/libspdk_conf.so.2.0 (0x00007f6931e94000)

       libspdk_blob.so.2.0 => /home/users/daos/daos/install/lib/libspdk_blob.so.2.0 (0x00007f6931c80000)

       libspdk_nvme.so.2.0 => /home/users/daos/daos/install/lib/libspdk_nvme.so.2.0 (0x00007f6931a3d000)

       libspdk_util.so.2.0 => /home/users/daos/daos/install/lib/libspdk_util.so.2.0 (0x00007f693182f000)

       libspdk_json.so.2.0 => /home/users/daos/daos/install/lib/libspdk_json.so.2.0 (0x00007f6931627000)

       libspdk_jsonrpc.so.2.0 => /home/users/daos/daos/install/lib/libspdk_jsonrpc.so.2.0 (0x00007f6931420000)

       libspdk_rpc.so.2.0 => /home/users/daos/daos/install/lib/libspdk_rpc.so.2.0 (0x00007f693121c000)

       libspdk_trace.so.2.0 => /home/users/daos/daos/install/lib/libspdk_trace.so.2.0 (0x00007f6931017000)

       libspdk_sock.so.2.0 => /home/users/daos/daos/install/lib/libspdk_sock.so.2.0 (0x00007f6930e13000)

       libspdk_log.so.2.0 => /home/users/daos/daos/install/lib/libspdk_log.so.2.0 (0x00007f6930c10000)

       libspdk_notify.so.2.0 => /home/users/daos/daos/install/lib/libspdk_notify.so.2.0 (0x00007f69309cd000)

       libspdk_blob_bdev.so.2.0 => /home/users/daos/daos/install/lib/libspdk_blob_bdev.so.2.0 (0x00007f69307c9000)

       libnuma.so.1 => /lib64/libnuma.so.1 (0x00007f69305be000)

       libm.so.6 => /lib64/libm.so.6 (0x00007f69302bc000)

       libltdl.so.7 => /lib64/libltdl.so.7 (0x00007f69300b2000)

       libpmem.so.1 => /home/users/daos/daos/install/lib/libpmem.so.1 (0x00007f692fe89000)

       librt.so.1 => /lib64/librt.so.1 (0x00007f692fc81000)

       libfabric.so.1 => /home/users/daos/daos/install/lib/libfabric.so.1 (0x00007f692f8a7000)

       libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f692f691000)

       libaio.so.1 => /lib64/libaio.so.1 (0x00007f692f48f000)

       libibverbs.so.1 => /lib64/libibverbs.so.1 (0x00007f692f274000)

       librdmacm.so.1 => /lib64/librdmacm.so.1 (0x00007f692f05b000)

       libnl-3.so.200 => /lib64/libnl-3.so.200 (0x00007f692ee3a000)

       libnl-route-3.so.200 => /lib64/libnl-route-3.so.200 (0x00007f692ebcd000)

       libpsm2.so.2 => /home/users/daos/daos/install/lib64/libpsm2.so.2 (0x00007f692e96a000)

 

 

Hi Kevan,

 

Are you building daos from scratch?

 

You should not need to set LD_LIBRARY_PATH to run the IO server in such a setup because daos_io_server is built using RPATH.   Can you do an ldd on daos_io_server ?   If you do use LD_LIBRARY_PATH, make sure lib64 comes before any occurrence of lib

 

-Jeff

 

Joel,

 

I came to the same conclusion, sorry for wasting your time.  

 

There is currently an issue where the daos_io_server dies immediately because it can’t find its own librdb.so module, which got moved into lib64/daos_srv.   If I move librdb.so to lib then it complains about other modules.  What is the correct way to configure for this?  

 

04/13-04:20:00.47 delphi-004 DAOS[70088/70088] server ERR  src/iosrv/module.c:105 dss_module_load() cannot load librdb.so: librdb.so: cannot open shared object file: No such file or directory

04/13-04:20:00.47 delphi-004 DAOS[70088/70088] server ERR  src/iosrv/init.c:195 modules_load() Failed to load module rdb: -1003

 

To work around this, I set LD_LIBRARY_PATH in the environ section of daos_server.yml to include all library-related subdirectories within the install tree.   And to get the install_dir pushed out to all the server nodes I use rsync.   By default rsync doesn’t delete files at the destination if they are not in the source, so as libraries move around in the install tree over time, I eventually ended up with two copies of the same .so in different directories, and the LD_LIBRARY_PATH resulted in the wrong one being picked.

 

Sorry, Kevan

 

Hi Kevan,

 

I ran your test locally in my environment on master and it encountered no issues.  Figures, right?   I spent some time looking at the auto-generated code to see what it is doing.  I don’t have any particular expertise in that.  But, it’s clear that if any of the parts are out of sync, it will not work.  I’m wondering if you have any stale protobuf files on your machine.  Can you diff the protobuf files on your machine against 1) daos master, and 2) your coworker’s machine to see if all three sets are equal?

 

There is a full list in src/proto/Makefile.  Of particular interest are these three:

 

src/mgmt/srv.pb-c.c

src/mgmt/srv.pb-c.h

src/control/common/proto/mgmt/srv.pb.go

 

Joel

 

 

Joel,

 

Thanks for the explanation below, makes sense.   Can’t wait for that code to land.

 

Back to the problem at hand;  now I am even more confused….   I borrowed one of my compatriot’s machines, breakpointed his daos_io_server in routine ds_mgmt_drpc_get_attach_info, in his daemon the resp structure has all 7 fields, and so he doesn’t get a segfault.   We are building with the same commit  point.   ????? 

 

Do you have any ideas on what could be different in my machine?   Same centos 7 release.   I will keep debugging, it again appears to be related to my environment somehow.

 

Kevan

 

Hi Kevan,

 

You are right that it won’t help a client to know the interface and domain names of the server.  In this case, we’re not actually sending the server’s interface and domain in the server’s response.  These fields are left empty until they are populated by the agent.  On the update I am working on now, the agent scans the client machine for network interfaces that support the server’s provider (based on the GetAttachInfo provider data) and populates the interface and domain fields in the response sent to the client.  In an update after that, the libdaos library then gets some rework to generate a GetAttachInfo prior to initializing CaRT so that it can use the interface and domain data that’s returned to it.  I’m working on getting this through review now.

 

Thanks for the additional debug log.  I appreciate your insight and help.  I will work on replicating the problem locally so I can fix it.

 

Joel

 

 

Joel,

 

I’m curious; how does it help a client to know the interface and domain names of this server?   I can’t see how the client could possibly use them.

 

Anyway, back to the problem.   I am breakpointed in ds_mgmt_drpc_get_attach_info().   At the top of the routine is this:

 

        Mgmt__GetAttachInfoResp  resp = MGMT__GET_ATTACH_INFO_RESP__INIT;

 

If I look in the code at the definition of Mgmt__GetAttachInfoResp it has the 7 data fields including your new interface field, etc.  And the value of MGMT__GET_ATTACH_INFO_RESP__INIT appears to initialize all 7 of those fields.   But if I use gdb to look at that structure you can see that the code doesn’t actually know about any of the new fields, it is only aware of status and n_psrs/psrs:

 

(gdb) p resp

$7 = {base = {descriptor = 0x7f9756b5dcc0 <mgmt__get_attach_info_resp__descriptor>, n_unknown_fields = 0, unknown_fields = 0x0}, status = 0, n_psrs = 0, psrs = 0x0}

(gdb) p resp.status

$8 = 0

(gdb) p resp.n_psrs

$9 = 0

(gdb) p resp.psrs

$10 = (Mgmt__GetAttachInfoResp__Psr **) 0x0

(gdb) p resp.provider

There is no member named provider.

(gdb) p resp.interface

There is no member named interface.

(gdb) p resp.domain

There is no member named domain.

 

(gdb) p sizeof(resp)

$13 = 48

 

If you do the math, you can see that the size of ‘resp’ is correct if the struct ends with field psrs, there is no room in the struct for the new fields.

 

If I then step forward and enter routine mgmt__get_attach_info_resp__get_packed_size(), that routine DOES know about 7 fields and tries to reference all of them, but of course the resp structure on the stack isn’t big enough to hold the 7 fields, so this routine is looking at other junk on the stack past the end of the structure:

 

239           len = mgmt__get_attach_info_resp__get_packed_size(&resp);

(gdb) s

mgmt__get_attach_info_resp__get_packed_size (message=message@entry=0x2aaab041f740) at src/mgmt/srv.pb-c.c:295

295      assert(message->base.descriptor == &mgmt__get_attach_info_resp__descriptor);

(gdb) n

296      return protobuf_c_message_get_packed_size ((const ProtobufCMessage*)(message));

(gdb) p message

$11 = (const Mgmt__GetAttachInfoResp *) 0x2aaab041f740

(gdb) p *message

$12 = {base = {descriptor = 0x7f9756b5dcc0 <mgmt__get_attach_info_resp__descriptor>, n_unknown_fields = 0, unknown_fields = 0x0}, status = 0, n_psrs = 1, 

  psrs = 0x7f9396113e00, provider = 0x0, interface = 0xad26fa6a89442100 <Address 0xad26fa6a89442100 out of bounds>, domain = 0x7f96f4026a10 "\340\230VW\227\177", 

  crtctxshareaddr = 4093798800, crttimeout = 32662}

 

Happy hunting,

 

Kevan

 

 

That’s good information.  I added the “interface” field and some others last week as we are expanding the capabilities of the GetAttachInfo message to help automatically configure the clients. It’s not clear why adding the fields would cause any of the unpacking code to fail, especially when it’s auto generated based on the protobuf definition.  However, I did build those files with newer versions of the protobuf compiler, and it’s possible that there’s a subtle incompatibility that I wasn’t aware of. 

Can you also dump the contents of lib, lib64, and lib64/daos_srv ?

 

Jeff,

 

Yes I am building from scratch, and RPATH used to work.   Here is the output, note that librdb doesn’t show up in the ldd output.

 

Thanks, Kevan

 

[root@delphi-004 ~]# cd ~daos/daos/install/bin

[root@delphi-004 bin]# ls

acl_dump_test  daosctl           daos_test        drpc_test    igzip                pmreorder           spdk_nvme_perf

agent_tests    daos_gen_io_conf  dcont            eq_tests     io_conf              rdbt                spdk_tgt

cart_ctl       daos_io_server    dfuse            evt_ctl      jump_pl_map          ring_pl_map         vea_ut

crt_launch     daos_perf         dfuse_hl         fi_info      nvme_control_ctests  security_test       vos_size

daos           daos_racer        dmg              fi_pingpong  obj_ctl              self_test           vos_size.py

daos_admin     daos_run_io_conf  dmg_old          fi_strerror  pl_bench             smd_ut              vos_tests

daos_agent     daos_server       drpc_iosrv_test  hello_drpc   pmempool             spdk_nvme_identify

[root@delphi-004 bin]# ldd daos_io_server

       linux-vdso.so.1 =>  (0x00007ffc47293000)

       libdaos_common.so => /home/users/daos/daos/install/lib64/libdaos_common.so (0x00007f69369cd000)

       libgurt.so.4 => /home/users/daos/daos/install/lib64/libgurt.so.4 (0x00007f69367aa000)

       libcart.so.4 => /home/users/daos/daos/install/lib64/libcart.so.4 (0x00007f69364c9000)

       libvos_srv.so => /home/users/daos/daos/install/bin/./../lib64/daos_srv/libvos_srv.so (0x00007f693622c000)

       libdaos.so.0 => /home/users/daos/daos/install/lib64/libdaos.so.0 (0x00007f6935f51000)

       libbio.so => /home/users/daos/daos/install/bin/./../lib64/daos_srv/libbio.so (0x00007f6935d26000)

       libdl.so.2 => /lib64/libdl.so.2 (0x00007f6935b22000)

       libuuid.so.1 => /lib64/libuuid.so.1 (0x00007f693591d000)

       libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f6935701000)

       libabt.so.0 => /home/users/daos/daos/install/lib/libabt.so.0 (0x00007f69354df000)

       libhwloc.so.5 => /lib64/libhwloc.so.5 (0x00007f69352a2000)

       libpmemobj.so.1 => /home/users/daos/daos/install/lib/libpmemobj.so.1 (0x00007f6935061000)

       libprotobuf-c.so.1 => /home/users/daos/daos/install/lib/libprotobuf-c.so.1 (0x00007f6934e58000)

       libc.so.6 => /lib64/libc.so.6 (0x00007f6934a8a000)

       libisal.so.2 => /home/users/daos/daos/install/lib/libisal.so.2 (0x00007f693484c000)

       /lib64/ld-linux-x86-64.so.2 (0x00007f6936c3c000)

       libyaml-0.so.2 => /lib64/libyaml-0.so.2 (0x00007f693462c000)

       libmercury.so.2 => /home/users/daos/daos/install/lib/libmercury.so.2 (0x00007f6934412000)

       libna.so.2 => /home/users/daos/daos/install/lib/libna.so.2 (0x00007f69341f4000)

       libmercury_util.so.2 => /home/users/daos/daos/install/lib/libmercury_util.so.2 (0x00007f6933fed000)

       libspdk_env_dpdk.so.2.0 => /home/users/daos/daos/install/lib/libspdk_env_dpdk.so.2.0 (0x00007f6933dde000)

       libspdk_thread.so.2.0 => /home/users/daos/daos/install/lib/libspdk_thread.so.2.0 (0x00007f6933bd8000)

       libspdk_bdev.so.2.0 => /home/users/daos/daos/install/lib/libspdk_bdev.so.2.0 (0x00007f69339c2000)

       libspdk_copy.so.2.0 => /home/users/daos/daos/install/lib/libspdk_copy.so.2.0 (0x00007f69337bf000)

       librte_mempool.so.20.0 => /home/users/daos/daos/install/lib/librte_mempool.so.20.0 (0x00007f69335b9000)

       librte_mempool_ring.so.20.0 => /home/users/daos/daos/install/lib/librte_mempool_ring.so.20.0 (0x00007f69333b6000)

       librte_bus_pci.so.20.0 => /home/users/daos/daos/install/lib/librte_bus_pci.so.20.0 (0x00007f69331ab000)

       librte_pci.so.20.0 => /home/users/daos/daos/install/lib/librte_pci.so.20.0 (0x00007f6932fa8000)

       librte_ring.so.20.0 => /home/users/daos/daos/install/lib/librte_ring.so.20.0 (0x00007f6932da5000)

       librte_mbuf.so.20.0 => /home/users/daos/daos/install/lib/librte_mbuf.so.20.0 (0x00007f6932b9a000)

       librte_eal.so.20.0 => /home/users/daos/daos/install/lib/librte_eal.so.20.0 (0x00007f69328bf000)

       librte_kvargs.so.20.0 => /home/users/daos/daos/install/lib/librte_kvargs.so.20.0 (0x00007f69326bc000)

       libspdk_bdev_aio.so.2.0 => /home/users/daos/daos/install/lib/libspdk_bdev_aio.so.2.0 (0x00007f69324b6000)

       libspdk_bdev_nvme.so.2.0 => /home/users/daos/daos/install/lib/libspdk_bdev_nvme.so.2.0 (0x00007f693229d000)

       libspdk_bdev_malloc.so.2.0 => /home/users/daos/daos/install/lib/libspdk_bdev_malloc.so.2.0 (0x00007f6932098000)

       libspdk_conf.so.2.0 => /home/users/daos/daos/install/lib/libspdk_conf.so.2.0 (0x00007f6931e94000)

       libspdk_blob.so.2.0 => /home/users/daos/daos/install/lib/libspdk_blob.so.2.0 (0x00007f6931c80000)

       libspdk_nvme.so.2.0 => /home/users/daos/daos/install/lib/libspdk_nvme.so.2.0 (0x00007f6931a3d000)

       libspdk_util.so.2.0 => /home/users/daos/daos/install/lib/libspdk_util.so.2.0 (0x00007f693182f000)

       libspdk_json.so.2.0 => /home/users/daos/daos/install/lib/libspdk_json.so.2.0 (0x00007f6931627000)

       libspdk_jsonrpc.so.2.0 => /home/users/daos/daos/install/lib/libspdk_jsonrpc.so.2.0 (0x00007f6931420000)

       libspdk_rpc.so.2.0 => /home/users/daos/daos/install/lib/libspdk_rpc.so.2.0 (0x00007f693121c000)

       libspdk_trace.so.2.0 => /home/users/daos/daos/install/lib/libspdk_trace.so.2.0 (0x00007f6931017000)

       libspdk_sock.so.2.0 => /home/users/daos/daos/install/lib/libspdk_sock.so.2.0 (0x00007f6930e13000)

       libspdk_log.so.2.0 => /home/users/daos/daos/install/lib/libspdk_log.so.2.0 (0x00007f6930c10000)

       libspdk_notify.so.2.0 => /home/users/daos/daos/install/lib/libspdk_notify.so.2.0 (0x00007f69309cd000)

       libspdk_blob_bdev.so.2.0 => /home/users/daos/daos/install/lib/libspdk_blob_bdev.so.2.0 (0x00007f69307c9000)

       libnuma.so.1 => /lib64/libnuma.so.1 (0x00007f69305be000)

       libm.so.6 => /lib64/libm.so.6 (0x00007f69302bc000)

       libltdl.so.7 => /lib64/libltdl.so.7 (0x00007f69300b2000)

       libpmem.so.1 => /home/users/daos/daos/install/lib/libpmem.so.1 (0x00007f692fe89000)

       librt.so.1 => /lib64/librt.so.1 (0x00007f692fc81000)

       libfabric.so.1 => /home/users/daos/daos/install/lib/libfabric.so.1 (0x00007f692f8a7000)

       libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f692f691000)

       libaio.so.1 => /lib64/libaio.so.1 (0x00007f692f48f000)

       libibverbs.so.1 => /lib64/libibverbs.so.1 (0x00007f692f274000)

       librdmacm.so.1 => /lib64/librdmacm.so.1 (0x00007f692f05b000)

       libnl-3.so.200 => /lib64/libnl-3.so.200 (0x00007f692ee3a000)

       libnl-route-3.so.200 => /lib64/libnl-route-3.so.200 (0x00007f692ebcd000)

       libpsm2.so.2 => /home/users/daos/daos/install/lib64/libpsm2.so.2 (0x00007f692e96a000)

 

 

Hi Kevan,

 

Are you building daos from scratch?

 

You should not need to set LD_LIBRARY_PATH to run the IO server in such a setup because daos_io_server is built using RPATH.   Can you do an ldd on daos_io_server ?   If you do use LD_LIBRARY_PATH, make sure lib64 comes before any occurrence of lib

 

-Jeff

 

Joel,

 

I came to the same conclusion, sorry for wasting your time.  

 

There is currently an issue where the daos_io_server dies immediately because it can’t find its own librdb.so module, which got moved into lib64/daos_srv.   If I move librdb.so to lib then it complains about other modules.  What is the correct way to configure for this?  

 

04/13-04:20:00.47 delphi-004 DAOS[70088/70088] server ERR  src/iosrv/module.c:105 dss_module_load() cannot load librdb.so: librdb.so: cannot open shared object file: No such file or directory

04/13-04:20:00.47 delphi-004 DAOS[70088/70088] server ERR  src/iosrv/init.c:195 modules_load() Failed to load module rdb: -1003

 

To work around this, I set LD_LIBRARY_PATH in the environ section of daos_server.yml to include all library-related subdirectories within the install tree.   And to get the install_dir pushed out to all the server nodes I use rsync.   By default rsync doesn’t delete files at the destination if they are not in the source, so as libraries move around in the install tree over time, I eventually ended up with two copies of the same .so in different directories, and the LD_LIBRARY_PATH resulted in the wrong one being picked.

 

Sorry, Kevan

 

Hi Kevan,

 

I ran your test locally in my environment on master and it encountered no issues.  Figures, right?   I spent some time looking at the auto-generated code to see what it is doing.  I don’t have any particular expertise in that.  But, it’s clear that if any of the parts are out of sync, it will not work.  I’m wondering if you have any stale protobuf files on your machine.  Can you diff the protobuf files on your machine against 1) daos master, and 2) your coworker’s machine to see if all three sets are equal?

 

There is a full list in src/proto/Makefile.  Of particular interest are these three:

 

src/mgmt/srv.pb-c.c

src/mgmt/srv.pb-c.h

src/control/common/proto/mgmt/srv.pb.go

 

Joel

 

 

Joel,

 

Thanks for the explanation below, makes sense.   Can’t wait for that code to land.

 

Back to the problem at hand;  now I am even more confused….   I borrowed one of my compatriot’s machines, breakpointed his daos_io_server in routine ds_mgmt_drpc_get_attach_info, in his daemon the resp structure has all 7 fields, and so he doesn’t get a segfault.   We are building with the same commit  point.   ????? 

 

Do you have any ideas on what could be different in my machine?   Same centos 7 release.   I will keep debugging, it again appears to be related to my environment somehow.

 

Kevan

 

Hi Kevan,

 

You are right that it won’t help a client to know the interface and domain names of the server.  In this case, we’re not actually sending the server’s interface and domain in the server’s response.  These fields are left empty until they are populated by the agent.  On the update I am working on now, the agent scans the client machine for network interfaces that support the server’s provider (based on the GetAttachInfo provider data) and populates the interface and domain fields in the response sent to the client.  In an update after that, the libdaos library then gets some rework to generate a GetAttachInfo prior to initializing CaRT so that it can use the interface and domain data that’s returned to it.  I’m working on getting this through review now.

 

Thanks for the additional debug log.  I appreciate your insight and help.  I will work on replicating the problem locally so I can fix it.

 

Joel

 

 

Joel,

 

I’m curious; how does it help a client to know the interface and domain names of this server?   I can’t see how the client could possibly use them.

 

Anyway, back to the problem.   I am breakpointed in ds_mgmt_drpc_get_attach_info().   At the top of the routine is this:

 

        Mgmt__GetAttachInfoResp  resp = MGMT__GET_ATTACH_INFO_RESP__INIT;

 

If I look in the code at the definition of Mgmt__GetAttachInfoResp it has the 7 data fields including your new interface field, etc.  And the value of MGMT__GET_ATTACH_INFO_RESP__INIT appears to initialize all 7 of those fields.   But if I use gdb to look at that structure you can see that the code doesn’t actually know about any of the new fields, it is only aware of status and n_psrs/psrs:

 

(gdb) p resp

$7 = {base = {descriptor = 0x7f9756b5dcc0 <mgmt__get_attach_info_resp__descriptor>, n_unknown_fields = 0, unknown_fields = 0x0}, status = 0, n_psrs = 0, psrs = 0x0}

(gdb) p resp.status

$8 = 0

(gdb) p resp.n_psrs

$9 = 0

(gdb) p resp.psrs

$10 = (Mgmt__GetAttachInfoResp__Psr **) 0x0

(gdb) p resp.provider

There is no member named provider.

(gdb) p resp.interface

There is no member named interface.

(gdb) p resp.domain

There is no member named domain.

 

(gdb) p sizeof(resp)

$13 = 48

 

If you do the math, you can see that the size of ‘resp’ is correct if the struct ends with field psrs, there is no room in the struct for the new fields.

 

If I then step forward and enter routine mgmt__get_attach_info_resp__get_packed_size(), that routine DOES know about 7 fields and tries to reference all of them, but of course the resp structure on the stack isn’t big enough to hold the 7 fields, so this routine is looking at other junk on the stack past the end of the structure:

 

239           len = mgmt__get_attach_info_resp__get_packed_size(&resp);

(gdb) s

mgmt__get_attach_info_resp__get_packed_size (message=message@entry=0x2aaab041f740) at src/mgmt/srv.pb-c.c:295

295      assert(message->base.descriptor == &mgmt__get_attach_info_resp__descriptor);

(gdb) n

296      return protobuf_c_message_get_packed_size ((const ProtobufCMessage*)(message));

(gdb) p message

$11 = (const Mgmt__GetAttachInfoResp *) 0x2aaab041f740

(gdb) p *message

$12 = {base = {descriptor = 0x7f9756b5dcc0 <mgmt__get_attach_info_resp__descriptor>, n_unknown_fields = 0, unknown_fields = 0x0}, status = 0, n_psrs = 1, 

  psrs = 0x7f9396113e00, provider = 0x0, interface = 0xad26fa6a89442100 <Address 0xad26fa6a89442100 out of bounds>, domain = 0x7f96f4026a10 "\340\230VW\227\177", 

  crtctxshareaddr = 4093798800, crttimeout = 32662}

 

Happy hunting,

 

Kevan

 

 

That’s good information.  I added the “interface” field and some others last week as we are expanding the capabilities of the GetAttachInfo message to help automatically configure the clients. It’s not clear why adding the fields would cause any of the unpacking code to fail, especially when it’s auto generated based on the protobuf definition.  However, I did build those files with newer versions of the protobuf compiler, and it’s possible that there’s a subtle incompatibility that I wasn’t aware of. 

Yes, here you go.   In talking to the others on our team, they don’t use LD_LIBRARY_PATH, they just move the entire contents of lib64/daos_srv up one directory into lib64 and that works.   I used LD_LIBRARY_PATH so that I didn’t have to keep remembering to do the mv command every time.

 

Kevan

 

daos@hl-d109 ~/daos/install $ find lib lib64 -type f

lib/libabt.so.0.0.0

lib/libabt.la

lib/libabt.a

lib/pkgconfig/argobots.pc

lib/pkgconfig/libpmem.pc

lib/pkgconfig/libpmemobj.pc

lib/pkgconfig/libpmemlog.pc

lib/pkgconfig/libpmemblk.pc

lib/pkgconfig/libpmempool.pc

lib/pkgconfig/libfabric.pc

lib/pkgconfig/openpa.pc

lib/pkgconfig/mercury.pc

lib/pkgconfig/libprotobuf-c.pc

lib/libpmem.so.1.0.0

lib/libpmem.a

lib/pmdk_debug/libpmem.so.1.0.0

lib/pmdk_debug/libpmem.a

lib/pmdk_debug/libpmemlog.so.1.0.0

lib/pmdk_debug/libpmemlog.a

lib/pmdk_debug/libpmemblk.so.1.0.0

lib/pmdk_debug/libpmemblk.a

lib/pmdk_debug/libpmemobj.so.1.0.0

lib/pmdk_debug/libpmemobj.a

lib/pmdk_debug/libpmempool.so.1.0.0

lib/pmdk_debug/libpmempool.a

lib/libpmemlog.so.1.0.0

lib/libpmemlog.a

lib/libpmemblk.so.1.0.0

lib/libpmemblk.a

lib/libpmemobj.so.1.0.0

lib/libpmemobj.a

lib/libpmempool.so.1.0.0

lib/libpmempool.a

lib/libpsm2/libpsm2-compat.cmds

lib/libfabric.so.1.12.0

lib/libfabric.la

lib/libfabric.a

lib/libopa.la

lib/libopa.a

lib/libmercury.so.2.0.0

lib/libmercury_hl.so.2.0.0

lib/libmercury_util.so.2.0.0

lib/libna.so.2.0.0

lib/libprotobuf-c.so.1.0.0

lib/libprotobuf-c.la

lib/libprotobuf-c.a

lib/libspdk_log.a

lib/libspdk_log.so.2.0

lib/libspdk_util.a

lib/libspdk_util.so.2.0

lib/libspdk_conf.a

lib/libspdk_conf.so.2.0

lib/libspdk_thread.a

lib/libspdk_thread.so.2.0

lib/libspdk_json.a

lib/libspdk_json.so.2.0

lib/libspdk_jsonrpc.a

lib/libspdk_jsonrpc.so.2.0

lib/libspdk_rpc.a

lib/libspdk_rpc.so.2.0

lib/libspdk_notify.a

lib/libspdk_notify.so.2.0

lib/libspdk_trace.a

lib/libspdk_trace.so.2.0

lib/libspdk_bdev.a

lib/libspdk_bdev.so.2.0

lib/libspdk_blob.a

lib/libspdk_blob.so.2.0

lib/libspdk_blobfs.a

lib/libspdk_blobfs.so.2.0

lib/libspdk_copy.a

lib/libspdk_copy.so.2.0

lib/libspdk_event.a

lib/libspdk_event.so.2.0

lib/libspdk_log_rpc.a

lib/libspdk_log_rpc.so.2.0

lib/libspdk_lvol.a

lib/libspdk_lvol.so.2.0

lib/libspdk_net.a

lib/libspdk_net.so.2.0

lib/libspdk_sock.a

lib/libspdk_sock.so.2.0

lib/libspdk_nvme.a

lib/libspdk_nvme.so.2.0

lib/libspdk_vmd.a

lib/libspdk_vmd.so.2.0

lib/libspdk_nvmf.a

lib/libspdk_nvmf.so.2.0

lib/libspdk_scsi.a

lib/libspdk_scsi.so.2.0

lib/libspdk_ioat.a

lib/libspdk_ioat.so.2.0

lib/libspdk_ut_mock.a

lib/libspdk_ut_mock.so.2.0

lib/libspdk_iscsi.a

lib/libspdk_iscsi.so.2.0

lib/libspdk_nbd.a

lib/libspdk_nbd.so.2.0

lib/libspdk_ftl.a

lib/libspdk_ftl.so.2.0

lib/libspdk_virtio.a

lib/libspdk_virtio.so.2.0

lib/libspdk_env_dpdk.a

lib/libspdk_env_dpdk.so.2.0

lib/libspdk_blob_bdev.a

lib/libspdk_blob_bdev.so.2.0

lib/libspdk_bdev_delay.a

lib/libspdk_bdev_delay.so.2.0

lib/libspdk_bdev_error.a

lib/libspdk_bdev_error.so.2.0

lib/libspdk_bdev_gpt.a

lib/libspdk_bdev_gpt.so.2.0

lib/libspdk_bdev_lvol.a

lib/libspdk_bdev_lvol.so.2.0

lib/libspdk_bdev_malloc.a

lib/libspdk_bdev_malloc.so.2.0

lib/libspdk_bdev_null.a

lib/libspdk_bdev_null.so.2.0

lib/libspdk_bdev_nvme.a

lib/libspdk_bdev_nvme.so.2.0

lib/libspdk_bdev_passthru.a

lib/libspdk_bdev_passthru.so.2.0

lib/libspdk_bdev_raid.a

lib/libspdk_bdev_raid.so.2.0

lib/libspdk_bdev_rpc.a

lib/libspdk_bdev_rpc.so.2.0

lib/libspdk_bdev_split.a

lib/libspdk_bdev_split.so.2.0

lib/libspdk_bdev_zone_block.a

lib/libspdk_bdev_zone_block.so.2.0

lib/libspdk_bdev_aio.a

lib/libspdk_bdev_aio.so.2.0

lib/libspdk_bdev_ftl.a

lib/libspdk_bdev_ftl.so.2.0

lib/libspdk_bdev_virtio.a

lib/libspdk_bdev_virtio.so.2.0

lib/libspdk_blobfs_bdev.a

lib/libspdk_blobfs_bdev.so.2.0

lib/libspdk_copy_ioat.a

lib/libspdk_copy_ioat.so.2.0

lib/libspdk_event_copy.a

lib/libspdk_event_copy.so.2.0

lib/libspdk_event_vmd.a

lib/libspdk_event_vmd.so.2.0

lib/libspdk_event_bdev.a

lib/libspdk_event_bdev.so.2.0

lib/libspdk_event_scsi.a

lib/libspdk_event_scsi.so.2.0

lib/libspdk_event_iscsi.a

lib/libspdk_event_iscsi.so.2.0

lib/libspdk_event_net.a

lib/libspdk_event_net.so.2.0

lib/libspdk_event_nvmf.a

lib/libspdk_event_nvmf.so.2.0

lib/libspdk_event_nbd.a

lib/libspdk_event_nbd.so.2.0

lib/libspdk_app_rpc.a

lib/libspdk_app_rpc.so.2.0

lib/libspdk_sock_posix.a

lib/libspdk_sock_posix.so.2.0

lib/libspdk_env_dpdk_rpc.a

lib/libspdk_env_dpdk_rpc.so.2.0

lib/libspdk.so.2.0

lib/libdpdk.so

lib/librte_bus_pci.so

lib/librte_bus_pci.so.20.0

lib/librte_bus_vdev.so

lib/librte_bus_vdev.so.20.0

lib/librte_compressdev.so

lib/librte_compressdev.so.0.200

lib/librte_cryptodev.so

lib/librte_cryptodev.so.20.0

lib/librte_eal.so

lib/librte_eal.so.20.0

lib/cart/TESTING/corpc/cart_corpc_five_node.py

lib/cart/TESTING/corpc/cart_corpc_five_node.yaml

lib/cart/TESTING/corpc/cart_corpc_one_node.py

lib/cart/TESTING/corpc/cart_corpc_one_node.yaml

lib/cart/TESTING/corpc/cart_corpc_two_node.py

lib/cart/TESTING/corpc/cart_corpc_two_node.yaml

lib/cart/TESTING/ctl/cart_ctl_five_node.py

lib/cart/TESTING/ctl/cart_ctl_five_node.yaml

lib/cart/TESTING/ctl/cart_ctl_one_node.py

lib/cart/TESTING/ctl/cart_ctl_one_node.yaml

lib/cart/TESTING/group_test/cart_ghost_rank_prc_one_node.py

lib/cart/TESTING/group_test/cart_ghost_rank_prc_one_node.yaml

lib/cart/TESTING/group_test/group_test.py

lib/cart/TESTING/group_test/group_test.yaml

lib/cart/TESTING/group_tiers/cart_group_tiers_three_node.py

lib/cart/TESTING/group_tiers/cart_group_tiers_three_node.yaml

lib/cart/TESTING/iv/cart_iv_one_node.py

lib/cart/TESTING/iv/cart_iv_one_node.yaml

lib/cart/TESTING/iv/cart_iv_two_node.py

lib/cart/TESTING/iv/cart_iv_two_node.yaml

lib/cart/TESTING/launch.py

lib/cart/TESTING/no_pmix/cart_nopmix_multictx_one_node.py

lib/cart/TESTING/no_pmix/cart_nopmix_multictx_one_node.yaml

lib/cart/TESTING/nopmix_launcher/cart_nopmix_launcher_one_node.py

lib/cart/TESTING/nopmix_launcher/cart_nopmix_launcher_one_node.yaml

lib/cart/TESTING/rpc/cart_rpc_one_node.py

lib/cart/TESTING/rpc/cart_rpc_one_node.yaml

lib/cart/TESTING/rpc/cart_rpc_two_node.py

lib/cart/TESTING/rpc/cart_rpc_two_node.yaml

lib/cart/TESTING/selftest/cart_selftest_three_node.py

lib/cart/TESTING/selftest/cart_selftest_three_node.yaml

lib/cart/TESTING/util/cart_logparse.py

lib/cart/TESTING/util/cart_logtest.py

lib/cart/TESTING/util/cart_utils.py

lib/cart/TESTING/tests/no_pmix_corpc_errors

lib/cart/TESTING/tests/test_swim

lib/cart/TESTING/tests/test_group_np_cli

lib/cart/TESTING/tests/test_hlc_net

lib/cart/TESTING/tests/no_pmix_multi_ctx

lib/cart/TESTING/tests/test_corpc_exclusive

lib/cart/TESTING/tests/iv_client

lib/cart/TESTING/tests/no_pmix_launcher_client

lib/cart/TESTING/tests/test_proto_client

lib/cart/TESTING/tests/threaded_server

lib/cart/TESTING/tests/test_no_timeout

lib/cart/TESTING/tests/test_ep_cred_server

lib/cart/TESTING/tests/test_rpc_to_ghost_rank

lib/cart/TESTING/tests/iv_server

lib/cart/TESTING/tests/test_ep_cred_client

lib/cart/TESTING/tests/test_swim_net

lib/cart/TESTING/tests/no_pmix_group_version

lib/cart/TESTING/tests/test_group_np_srv

lib/cart/TESTING/tests/test_corpc_prefwd

lib/cart/TESTING/tests/threaded_client

lib/cart/TESTING/tests/test_proto_server

lib/cart/TESTING/tests/no_pmix_launcher_server

lib/cart/TESTING/tests/no_pmix_group_test

lib/librte_fib.so

lib/librte_fib.so.0.200

lib/librte_kvargs.so

lib/librte_kvargs.so.20.0

lib/librte_mbuf.so

lib/librte_mbuf.so.20.0

lib/librte_mempool_bucket.so

lib/librte_mempool_bucket.so.20.0

lib/librte_mempool_ring.so

lib/librte_mempool_ring.so.20.0

lib/librte_mempool.so

lib/librte_mempool.so.20.0

lib/librte_net.so

lib/librte_net.so.20.0

lib/librte_pci.so

lib/librte_pci.so.20.0

lib/librte_rib.so

lib/librte_rib.so.0.200

lib/librte_ring.so

lib/librte_ring.so.20.0

lib/libisal.so.2.0.26

lib/libisal.la

lib/libisal.a

lib/daos/.build_vars.json

lib/daos/.build_vars.sh

lib/daos/TESTING/ftest/checksum/basic_checksum.py

lib/daos/TESTING/ftest/checksum/basic_checksum.yaml

lib/daos/TESTING/ftest/checksum/csum_error_logging.py

lib/daos/TESTING/ftest/checksum/csum_error_logging.yaml

lib/daos/TESTING/ftest/container/attribute.py

lib/daos/TESTING/ftest/container/attribute.yaml

lib/daos/TESTING/ftest/container/basic_snapshot.py

lib/daos/TESTING/ftest/container/basic_snapshot.yaml

lib/daos/TESTING/ftest/container/basic_tx_test.py

lib/daos/TESTING/ftest/container/basic_tx_test.yaml

lib/daos/TESTING/ftest/container/container_async.py

lib/daos/TESTING/ftest/container/container_async.yaml

lib/daos/TESTING/ftest/container/create.py

lib/daos/TESTING/ftest/container/create.yaml

lib/daos/TESTING/ftest/container/delete.py

lib/daos/TESTING/ftest/container/delete.yaml

lib/daos/TESTING/ftest/container/full_pool_container_create.py

lib/daos/TESTING/ftest/container/full_pool_container_create.yaml

lib/daos/TESTING/ftest/container/global_handle.py

lib/daos/TESTING/ftest/container/global_handle.yaml

lib/daos/TESTING/ftest/container/open.py

lib/daos/TESTING/ftest/container/open.yaml

lib/daos/TESTING/ftest/container/open_close.py

lib/daos/TESTING/ftest/container/open_close.yaml

lib/daos/TESTING/ftest/container/simple_create_delete_test.py

lib/daos/TESTING/ftest/container/simple_create_delete_test.yaml

lib/daos/TESTING/ftest/container/snapshot.py

lib/daos/TESTING/ftest/container/snapshot.yaml

lib/daos/TESTING/ftest/control/daos_admin_privileged.py

lib/daos/TESTING/ftest/control/daos_admin_privileged.yaml

lib/daos/TESTING/ftest/control/dmg_nvme_scan_test.py

lib/daos/TESTING/ftest/control/dmg_nvme_scan_test.yaml

lib/daos/TESTING/ftest/control/super_block_versioning.py

lib/daos/TESTING/ftest/control/super_block_versioning.yaml

lib/daos/TESTING/ftest/daos_test/daos_core_test-rebuild.py

lib/daos/TESTING/ftest/daos_test/daos_core_test-rebuild.yaml

lib/daos/TESTING/ftest/daos_test/daos_core_test.py

lib/daos/TESTING/ftest/daos_test/daos_core_test.yaml

lib/daos/TESTING/ftest/data/daos_agent_baseline.yaml

lib/daos/TESTING/ftest/data/daos_server_baseline.yaml

lib/daos/TESTING/ftest/io/daos_perf.py

lib/daos/TESTING/ftest/io/daos_perf.yaml

lib/daos/TESTING/ftest/io/daos_perf_large.py

lib/daos/TESTING/ftest/io/daos_perf_large.yaml

lib/daos/TESTING/ftest/io/daos_racer.py

lib/daos/TESTING/ftest/io/daos_racer.yaml

lib/daos/TESTING/ftest/io/fio_small.py

lib/daos/TESTING/ftest/io/fio_small.yaml

lib/daos/TESTING/ftest/io/hdf5.py

lib/daos/TESTING/ftest/io/hdf5.yaml

lib/daos/TESTING/ftest/io/ior_intercept_basic.py

lib/daos/TESTING/ftest/io/ior_intercept_basic.yaml

lib/daos/TESTING/ftest/io/ior_intercept_dfuse_mix.py

lib/daos/TESTING/ftest/io/ior_intercept_dfuse_mix.yaml

lib/daos/TESTING/ftest/io/ior_intercept_multi_client.py

lib/daos/TESTING/ftest/io/ior_intercept_multi_client.yaml

lib/daos/TESTING/ftest/io/ior_intercept_verify_data_integrity.py

lib/daos/TESTING/ftest/io/ior_intercept_verify_data_integrity.yaml

lib/daos/TESTING/ftest/io/ior_large.py

lib/daos/TESTING/ftest/io/ior_large.yaml

lib/daos/TESTING/ftest/io/ior_small.py

lib/daos/TESTING/ftest/io/ior_small.yaml

lib/daos/TESTING/ftest/io/llnl_mpi4py.py

lib/daos/TESTING/ftest/io/llnl_mpi4py.yaml

lib/daos/TESTING/ftest/io/mdtest_large.py

lib/daos/TESTING/ftest/io/mdtest_large.yaml

lib/daos/TESTING/ftest/io/mdtest_small.py

lib/daos/TESTING/ftest/io/mdtest_small.yaml

lib/daos/TESTING/ftest/io/nvme_fragmentation.py

lib/daos/TESTING/ftest/io/nvme_fragmentation.yaml

lib/daos/TESTING/ftest/io/nvme_io.py

lib/daos/TESTING/ftest/io/nvme_io.yaml

lib/daos/TESTING/ftest/io/nvme_io_verification.py

lib/daos/TESTING/ftest/io/nvme_io_verification.yaml

lib/daos/TESTING/ftest/io/nvme_object.py

lib/daos/TESTING/ftest/io/nvme_object.yaml

lib/daos/TESTING/ftest/io/romio.py

lib/daos/TESTING/ftest/io/romio.yaml

lib/daos/TESTING/ftest/io/seg_count.py

lib/daos/TESTING/ftest/io/seg_count.yaml

lib/daos/TESTING/ftest/io/unaligned_io.py

lib/daos/TESTING/ftest/io/unaligned_io.yaml

lib/daos/TESTING/ftest/launch.py

lib/daos/TESTING/ftest/metrics.py

lib/daos/TESTING/ftest/network/cart_self_test.py

lib/daos/TESTING/ftest/network/cart_self_test.yaml

lib/daos/TESTING/ftest/object/array_obj_test.py

lib/daos/TESTING/ftest/object/array_obj_test.yaml

lib/daos/TESTING/ftest/object/create_many_dkeys.py

lib/daos/TESTING/ftest/object/create_many_dkeys.yaml

lib/daos/TESTING/ftest/object/obj_fetch_bad_param.py

lib/daos/TESTING/ftest/object/obj_fetch_bad_param.yaml

lib/daos/TESTING/ftest/object/obj_open_bad_param.py

lib/daos/TESTING/ftest/object/obj_open_bad_param.yaml

lib/daos/TESTING/ftest/object/obj_update_bad_param.py

lib/daos/TESTING/ftest/object/obj_update_bad_param.yaml

lib/daos/TESTING/ftest/object/object_integrity.py

lib/daos/TESTING/ftest/object/object_integrity.yaml

lib/daos/TESTING/ftest/object/punch_test.py

lib/daos/TESTING/ftest/object/punch_test.yaml

lib/daos/TESTING/ftest/object/same_key_different_value.py

lib/daos/TESTING/ftest/object/same_key_different_value.yaml

lib/daos/TESTING/ftest/pool/attribute.py

lib/daos/TESTING/ftest/pool/attribute.yaml

lib/daos/TESTING/ftest/pool/bad_connect.py

lib/daos/TESTING/ftest/pool/bad_connect.yaml

lib/daos/TESTING/ftest/pool/bad_create.py

lib/daos/TESTING/ftest/pool/bad_create.yaml

lib/daos/TESTING/ftest/pool/bad_evict.py

lib/daos/TESTING/ftest/pool/bad_evict.yaml

lib/daos/TESTING/ftest/pool/bad_exclude.py

lib/daos/TESTING/ftest/pool/bad_exclude.yaml

lib/daos/TESTING/ftest/pool/bad_query.py

lib/daos/TESTING/ftest/pool/bad_query.yaml

lib/daos/TESTING/ftest/pool/connect_test.py

lib/daos/TESTING/ftest/pool/connect_test.yaml

lib/daos/TESTING/ftest/pool/destroy_rebuild.py

lib/daos/TESTING/ftest/pool/destroy_rebuild.yaml

lib/daos/TESTING/ftest/pool/destroy_tests.py

lib/daos/TESTING/ftest/pool/destroy_tests.yaml

lib/daos/TESTING/ftest/pool/evict_test.py

lib/daos/TESTING/ftest/pool/evict_test.yaml

lib/daos/TESTING/ftest/pool/global_handle.py

lib/daos/TESTING/ftest/pool/global_handle.yaml

lib/daos/TESTING/ftest/pool/info_tests.py

lib/daos/TESTING/ftest/pool/info_tests.yaml

lib/daos/TESTING/ftest/pool/multi_server_create_delete_test.py

lib/daos/TESTING/ftest/pool/multi_server_create_delete_test.yaml

lib/daos/TESTING/ftest/pool/multiple_creates_test.py

lib/daos/TESTING/ftest/pool/multiple_creates_test.yaml

lib/daos/TESTING/ftest/pool/permission.py

lib/daos/TESTING/ftest/pool/permission.yaml

lib/daos/TESTING/ftest/pool/pool_svc.py

lib/daos/TESTING/ftest/pool/pool_svc.yaml

lib/daos/TESTING/ftest/pool/rebuild_no_cap.py

lib/daos/TESTING/ftest/pool/rebuild_no_cap.yaml

lib/daos/TESTING/ftest/pool/rebuild_tests.py

lib/daos/TESTING/ftest/pool/rebuild_tests.yaml

lib/daos/TESTING/ftest/pool/rebuild_with_io.py

lib/daos/TESTING/ftest/pool/rebuild_with_io.yaml

lib/daos/TESTING/ftest/pool/rebuild_with_ior.py

lib/daos/TESTING/ftest/pool/rebuild_with_ior.yaml

lib/daos/TESTING/ftest/pool/simple_create_delete_test.py

lib/daos/TESTING/ftest/pool/simple_create_delete_test.yaml

lib/daos/TESTING/ftest/rebuild/cascading_failures.py

lib/daos/TESTING/ftest/rebuild/cascading_failures.yaml

lib/daos/TESTING/ftest/rebuild/container_create.py

lib/daos/TESTING/ftest/rebuild/container_create.yaml

lib/daos/TESTING/ftest/rebuild/delete_objects.py

lib/daos/TESTING/ftest/rebuild/delete_objects.yaml

lib/daos/TESTING/ftest/rebuild/io_conf_run.py

lib/daos/TESTING/ftest/rebuild/io_conf_run.yaml

lib/daos/TESTING/ftest/rebuild/read_array.py

lib/daos/TESTING/ftest/rebuild/read_array.yaml

lib/daos/TESTING/ftest/security/pool_connect_init.py

lib/daos/TESTING/ftest/security/pool_connect_init.yaml

lib/daos/TESTING/ftest/security/pool_security_acl.py

lib/daos/TESTING/ftest/security/pool_security_acl.yaml

lib/daos/TESTING/ftest/security/pool_security_groups.py

lib/daos/TESTING/ftest/security/pool_security_groups.yaml

lib/daos/TESTING/ftest/server/daos_server_config.py

lib/daos/TESTING/ftest/server/daos_server_config.yaml

lib/daos/TESTING/ftest/server/metadata.py

lib/daos/TESTING/ftest/server/metadata.yaml

lib/daos/TESTING/ftest/slurm_setup.py

lib/daos/TESTING/ftest/soak/soak.py

lib/daos/TESTING/ftest/soak/soak.yaml

lib/daos/TESTING/ftest/unittest/unittest.py

lib/daos/TESTING/ftest/unittest/unittest.yaml

lib/daos/TESTING/ftest/util/__init__.py

lib/daos/TESTING/ftest/util/agent_utils.py

lib/daos/TESTING/ftest/util/apricot/VERSION

lib/daos/TESTING/ftest/util/apricot/apricot/__init__.py

lib/daos/TESTING/ftest/util/apricot/apricot/test.py

lib/daos/TESTING/ftest/util/apricot/setup.py

lib/daos/TESTING/ftest/util/check_for_pool.py

lib/daos/TESTING/ftest/util/command_utils.py

lib/daos/TESTING/ftest/util/configuration_utils.py

lib/daos/TESTING/ftest/util/daos_core_base.py

lib/daos/TESTING/ftest/util/daos_io_conf.py

lib/daos/TESTING/ftest/util/daos_perf_utils.py

lib/daos/TESTING/ftest/util/daos_racer_utils.py

lib/daos/TESTING/ftest/util/daos_utils.py

lib/daos/TESTING/ftest/util/dfuse_utils.py

lib/daos/TESTING/ftest/util/dmg_utils.py

lib/daos/TESTING/ftest/util/env_modules.py

lib/daos/TESTING/ftest/util/fault_config_utils.py

lib/daos/TESTING/ftest/util/fio_test_base.py

lib/daos/TESTING/ftest/util/fio_utils.py

lib/daos/TESTING/ftest/util/general_utils.py

lib/daos/TESTING/ftest/util/get_hosts_from_file.py

lib/daos/TESTING/ftest/util/io_utilities.py

lib/daos/TESTING/ftest/util/ior_test_base.py

lib/daos/TESTING/ftest/util/ior_utils.py

lib/daos/TESTING/ftest/util/mdtest_test_base.py

lib/daos/TESTING/ftest/util/mdtest_utils.py

lib/daos/TESTING/ftest/util/mpio_test_base.py

lib/daos/TESTING/ftest/util/mpio_utils.py

lib/daos/TESTING/ftest/util/pool_security_test_base.py

lib/daos/TESTING/ftest/util/rebuild_test_base.py

lib/daos/TESTING/ftest/util/server_utils.py

lib/daos/TESTING/ftest/util/slurm_utils.py

lib/daos/TESTING/ftest/util/test_utils_base.py

lib/daos/TESTING/ftest/util/test_utils_container.py

lib/daos/TESTING/ftest/util/test_utils_pool.py

lib/daos/TESTING/ftest/util/write_host_file.py

lib/daos/TESTING/ftest/util/write_some_data.py

lib/daos/TESTING/scripts/check_ioil_syms

lib/daos/TESTING/tests/lf_s_test_ioil

lib/daos/TESTING/tests/s_test_ioil

lib64/psm2-compat/libpsm_infinipath.so.1

lib64/libpsm2.so.2.1

lib64/libpsm2.a

lib64/libfuse3.so.3.5.0

lib64/pkgconfig/fuse3.pc

lib64/libvos.so

lib64/daos/API_VERSION

lib64/daos/VERSION

lib64/daos/certgen/admin.cnf

lib64/daos/certgen/agent.cnf

lib64/daos/certgen/ca.cnf

lib64/daos/certgen/gen_certificates.sh

lib64/daos/certgen/server.cnf

lib64/daos_srv/libbio.so

lib64/daos_srv/libplacement.so

lib64/daos_srv/libdtx.so

lib64/daos_srv/libvos_srv.so

lib64/daos_srv/libcont.so

lib64/daos_srv/libsecurity.so

lib64/daos_srv/libmgmt.so

lib64/daos_srv/librdbt.so

lib64/daos_srv/librsvc.so

lib64/daos_srv/libobj.so

lib64/daos_srv/librebuild.so

lib64/daos_srv/librdb.so

lib64/daos_srv/libpool.so

lib64/libioil.a

lib64/libgurt.so.4.7.0

lib64/libnvme_control.a

lib64/python2.7/site-packages/pydaos/__init__.py

lib64/python2.7/site-packages/pydaos/dbm_daos.py

lib64/python2.7/site-packages/pydaos/pydaos_core.py

lib64/python2.7/site-packages/pydaos/raw/__init__.py

lib64/python2.7/site-packages/pydaos/raw/conversion.py

lib64/python2.7/site-packages/pydaos/raw/daos_api.py

lib64/python2.7/site-packages/pydaos/raw/daos_cref.py

lib64/python2.7/site-packages/pydaos/raw/daos_io.py

lib64/python2.7/site-packages/pydaos/pydaos_shim_27.so

lib64/python3/site-packages/pydaos/__init__.py

lib64/python3/site-packages/pydaos/pydaos_core.py

lib64/python3/site-packages/pydaos/dbm_daos.py

lib64/python3/site-packages/pydaos/raw/__init__.py

lib64/python3/site-packages/pydaos/raw/conversion.py

lib64/python3/site-packages/pydaos/raw/daos_api.py

lib64/python3/site-packages/pydaos/raw/daos_cref.py

lib64/python3/site-packages/pydaos/raw/daos_io.py

lib64/python3/site-packages/pydaos/pydaos_shim_3.so

lib64/libdfuse.a

lib64/libcart.so.4.7.0

lib64/libdaos_common.so

lib64/libdaos_tests.so

lib64/libdaos.so.0.9.0

lib64/libioil.so

lib64/libdfs.so

lib64/libdfuse.so

lib64/libduns.so

daos@hl-d109 ~/daos/install $ 

 

 

Can you also dump the contents of lib, lib64, and lib64/daos_srv ?

 

Jeff,

 

Yes I am building from scratch, and RPATH used to work.   Here is the output, note that librdb doesn’t show up in the ldd output.

 

Thanks, Kevan

 

[root@delphi-004 ~]# cd ~daos/daos/install/bin

[root@delphi-004 bin]# ls

acl_dump_test  daosctl           daos_test        drpc_test    igzip                pmreorder           spdk_nvme_perf

agent_tests    daos_gen_io_conf  dcont            eq_tests     io_conf              rdbt                spdk_tgt

cart_ctl       daos_io_server    dfuse            evt_ctl      jump_pl_map          ring_pl_map         vea_ut

crt_launch     daos_perf         dfuse_hl         fi_info      nvme_control_ctests  security_test       vos_size

daos           daos_racer        dmg              fi_pingpong  obj_ctl              self_test           vos_size.py

daos_admin     daos_run_io_conf  dmg_old          fi_strerror  pl_bench             smd_ut              vos_tests

daos_agent     daos_server       drpc_iosrv_test  hello_drpc   pmempool             spdk_nvme_identify

[root@delphi-004 bin]# ldd daos_io_server

       linux-vdso.so.1 =>  (0x00007ffc47293000)

       libdaos_common.so => /home/users/daos/daos/install/lib64/libdaos_common.so (0x00007f69369cd000)

       libgurt.so.4 => /home/users/daos/daos/install/lib64/libgurt.so.4 (0x00007f69367aa000)

       libcart.so.4 => /home/users/daos/daos/install/lib64/libcart.so.4 (0x00007f69364c9000)

       libvos_srv.so => /home/users/daos/daos/install/bin/./../lib64/daos_srv/libvos_srv.so (0x00007f693622c000)

       libdaos.so.0 => /home/users/daos/daos/install/lib64/libdaos.so.0 (0x00007f6935f51000)

       libbio.so => /home/users/daos/daos/install/bin/./../lib64/daos_srv/libbio.so (0x00007f6935d26000)

       libdl.so.2 => /lib64/libdl.so.2 (0x00007f6935b22000)

       libuuid.so.1 => /lib64/libuuid.so.1 (0x00007f693591d000)

       libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f6935701000)

       libabt.so.0 => /home/users/daos/daos/install/lib/libabt.so.0 (0x00007f69354df000)

       libhwloc.so.5 => /lib64/libhwloc.so.5 (0x00007f69352a2000)

       libpmemobj.so.1 => /home/users/daos/daos/install/lib/libpmemobj.so.1 (0x00007f6935061000)

       libprotobuf-c.so.1 => /home/users/daos/daos/install/lib/libprotobuf-c.so.1 (0x00007f6934e58000)

       libc.so.6 => /lib64/libc.so.6 (0x00007f6934a8a000)

       libisal.so.2 => /home/users/daos/daos/install/lib/libisal.so.2 (0x00007f693484c000)

       /lib64/ld-linux-x86-64.so.2 (0x00007f6936c3c000)

       libyaml-0.so.2 => /lib64/libyaml-0.so.2 (0x00007f693462c000)

       libmercury.so.2 => /home/users/daos/daos/install/lib/libmercury.so.2 (0x00007f6934412000)

       libna.so.2 => /home/users/daos/daos/install/lib/libna.so.2 (0x00007f69341f4000)

       libmercury_util.so.2 => /home/users/daos/daos/install/lib/libmercury_util.so.2 (0x00007f6933fed000)

       libspdk_env_dpdk.so.2.0 => /home/users/daos/daos/install/lib/libspdk_env_dpdk.so.2.0 (0x00007f6933dde000)

       libspdk_thread.so.2.0 => /home/users/daos/daos/install/lib/libspdk_thread.so.2.0 (0x00007f6933bd8000)

       libspdk_bdev.so.2.0 => /home/users/daos/daos/install/lib/libspdk_bdev.so.2.0 (0x00007f69339c2000)

       libspdk_copy.so.2.0 => /home/users/daos/daos/install/lib/libspdk_copy.so.2.0 (0x00007f69337bf000)

       librte_mempool.so.20.0 => /home/users/daos/daos/install/lib/librte_mempool.so.20.0 (0x00007f69335b9000)

       librte_mempool_ring.so.20.0 => /home/users/daos/daos/install/lib/librte_mempool_ring.so.20.0 (0x00007f69333b6000)

       librte_bus_pci.so.20.0 => /home/users/daos/daos/install/lib/librte_bus_pci.so.20.0 (0x00007f69331ab000)

       librte_pci.so.20.0 => /home/users/daos/daos/install/lib/librte_pci.so.20.0 (0x00007f6932fa8000)

       librte_ring.so.20.0 => /home/users/daos/daos/install/lib/librte_ring.so.20.0 (0x00007f6932da5000)

       librte_mbuf.so.20.0 => /home/users/daos/daos/install/lib/librte_mbuf.so.20.0 (0x00007f6932b9a000)

       librte_eal.so.20.0 => /home/users/daos/daos/install/lib/librte_eal.so.20.0 (0x00007f69328bf000)

       librte_kvargs.so.20.0 => /home/users/daos/daos/install/lib/librte_kvargs.so.20.0 (0x00007f69326bc000)

       libspdk_bdev_aio.so.2.0 => /home/users/daos/daos/install/lib/libspdk_bdev_aio.so.2.0 (0x00007f69324b6000)

       libspdk_bdev_nvme.so.2.0 => /home/users/daos/daos/install/lib/libspdk_bdev_nvme.so.2.0 (0x00007f693229d000)

       libspdk_bdev_malloc.so.2.0 => /home/users/daos/daos/install/lib/libspdk_bdev_malloc.so.2.0 (0x00007f6932098000)

       libspdk_conf.so.2.0 => /home/users/daos/daos/install/lib/libspdk_conf.so.2.0 (0x00007f6931e94000)

       libspdk_blob.so.2.0 => /home/users/daos/daos/install/lib/libspdk_blob.so.2.0 (0x00007f6931c80000)

       libspdk_nvme.so.2.0 => /home/users/daos/daos/install/lib/libspdk_nvme.so.2.0 (0x00007f6931a3d000)

       libspdk_util.so.2.0 => /home/users/daos/daos/install/lib/libspdk_util.so.2.0 (0x00007f693182f000)

       libspdk_json.so.2.0 => /home/users/daos/daos/install/lib/libspdk_json.so.2.0 (0x00007f6931627000)

       libspdk_jsonrpc.so.2.0 => /home/users/daos/daos/install/lib/libspdk_jsonrpc.so.2.0 (0x00007f6931420000)

       libspdk_rpc.so.2.0 => /home/users/daos/daos/install/lib/libspdk_rpc.so.2.0 (0x00007f693121c000)

       libspdk_trace.so.2.0 => /home/users/daos/daos/install/lib/libspdk_trace.so.2.0 (0x00007f6931017000)

       libspdk_sock.so.2.0 => /home/users/daos/daos/install/lib/libspdk_sock.so.2.0 (0x00007f6930e13000)

       libspdk_log.so.2.0 => /home/users/daos/daos/install/lib/libspdk_log.so.2.0 (0x00007f6930c10000)

       libspdk_notify.so.2.0 => /home/users/daos/daos/install/lib/libspdk_notify.so.2.0 (0x00007f69309cd000)

       libspdk_blob_bdev.so.2.0 => /home/users/daos/daos/install/lib/libspdk_blob_bdev.so.2.0 (0x00007f69307c9000)

       libnuma.so.1 => /lib64/libnuma.so.1 (0x00007f69305be000)

       libm.so.6 => /lib64/libm.so.6 (0x00007f69302bc000)

       libltdl.so.7 => /lib64/libltdl.so.7 (0x00007f69300b2000)

       libpmem.so.1 => /home/users/daos/daos/install/lib/libpmem.so.1 (0x00007f692fe89000)

       librt.so.1 => /lib64/librt.so.1 (0x00007f692fc81000)

       libfabric.so.1 => /home/users/daos/daos/install/lib/libfabric.so.1 (0x00007f692f8a7000)

       libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f692f691000)

       libaio.so.1 => /lib64/libaio.so.1 (0x00007f692f48f000)

       libibverbs.so.1 => /lib64/libibverbs.so.1 (0x00007f692f274000)

       librdmacm.so.1 => /lib64/librdmacm.so.1 (0x00007f692f05b000)

       libnl-3.so.200 => /lib64/libnl-3.so.200 (0x00007f692ee3a000)

       libnl-route-3.so.200 => /lib64/libnl-route-3.so.200 (0x00007f692ebcd000)

       libpsm2.so.2 => /home/users/daos/daos/install/lib64/libpsm2.so.2 (0x00007f692e96a000)

 

 

Hi Kevan,

 

Are you building daos from scratch?

 

You should not need to set LD_LIBRARY_PATH to run the IO server in such a setup because daos_io_server is built using RPATH.   Can you do an ldd on daos_io_server ?   If you do use LD_LIBRARY_PATH, make sure lib64 comes before any occurrence of lib

 

-Jeff

 

Joel,

 

I came to the same conclusion, sorry for wasting your time.  

 

There is currently an issue where the daos_io_server dies immediately because it can’t find its own librdb.so module, which got moved into lib64/daos_srv.   If I move librdb.so to lib then it complains about other modules.  What is the correct way to configure for this?  

 

04/13-04:20:00.47 delphi-004 DAOS[70088/70088] server ERR  src/iosrv/module.c:105 dss_module_load() cannot load librdb.so: librdb.so: cannot open shared object file: No such file or directory

04/13-04:20:00.47 delphi-004 DAOS[70088/70088] server ERR  src/iosrv/init.c:195 modules_load() Failed to load module rdb: -1003

 

To work around this, I set LD_LIBRARY_PATH in the environ section of daos_server.yml to include all library-related subdirectories within the install tree.   And to get the install_dir pushed out to all the server nodes I use rsync.   By default rsync doesn’t delete files at the destination if they are not in the source, so as libraries move around in the install tree over time, I eventually ended up with two copies of the same .so in different directories, and the LD_LIBRARY_PATH resulted in the wrong one being picked.

 

Sorry, Kevan

 

Hi Kevan,

 

I ran your test locally in my environment on master and it encountered no issues.  Figures, right?   I spent some time looking at the auto-generated code to see what it is doing.  I don’t have any particular expertise in that.  But, it’s clear that if any of the parts are out of sync, it will not work.  I’m wondering if you have any stale protobuf files on your machine.  Can you diff the protobuf files on your machine against 1) daos master, and 2) your coworker’s machine to see if all three sets are equal?

 

There is a full list in src/proto/Makefile.  Of particular interest are these three:

 

src/mgmt/srv.pb-c.c

src/mgmt/srv.pb-c.h

src/control/common/proto/mgmt/srv.pb.go

 

Joel

 

 

Joel,

 

Thanks for the explanation below, makes sense.   Can’t wait for that code to land.

 

Back to the problem at hand;  now I am even more confused….   I borrowed one of my compatriot’s machines, breakpointed his daos_io_server in routine ds_mgmt_drpc_get_attach_info, in his daemon the resp structure has all 7 fields, and so he doesn’t get a segfault.   We are building with the same commit  point.   ????? 

 

Do you have any ideas on what could be different in my machine?   Same centos 7 release.   I will keep debugging, it again appears to be related to my environment somehow.

 

Kevan

 

Hi Kevan,

 

You are right that it won’t help a client to know the interface and domain names of the server.  In this case, we’re not actually sending the server’s interface and domain in the server’s response.  These fields are left empty until they are populated by the agent.  On the update I am working on now, the agent scans the client machine for network interfaces that support the server’s provider (based on the GetAttachInfo provider data) and populates the interface and domain fields in the response sent to the client.  In an update after that, the libdaos library then gets some rework to generate a GetAttachInfo prior to initializing CaRT so that it can use the interface and domain data that’s returned to it.  I’m working on getting this through review now.

 

Thanks for the additional debug log.  I appreciate your insight and help.  I will work on replicating the problem locally so I can fix it.

 

Joel

 

 

Joel,

 

I’m curious; how does it help a client to know the interface and domain names of this server?   I can’t see how the client could possibly use them.

 

Anyway, back to the problem.   I am breakpointed in ds_mgmt_drpc_get_attach_info().   At the top of the routine is this:

 

        Mgmt__GetAttachInfoResp  resp = MGMT__GET_ATTACH_INFO_RESP__INIT;

 

If I look in the code at the definition of Mgmt__GetAttachInfoResp it has the 7 data fields including your new interface field, etc.  And the value of MGMT__GET_ATTACH_INFO_RESP__INIT appears to initialize all 7 of those fields.   But if I use gdb to look at that structure you can see that the code doesn’t actually know about any of the new fields, it is only aware of status and n_psrs/psrs:

 

(gdb) p resp

$7 = {base = {descriptor = 0x7f9756b5dcc0 <mgmt__get_attach_info_resp__descriptor>, n_unknown_fields = 0, unknown_fields = 0x0}, status = 0, n_psrs = 0, psrs = 0x0}

(gdb) p resp.status

$8 = 0

(gdb) p resp.n_psrs

$9 = 0

(gdb) p resp.psrs

$10 = (Mgmt__GetAttachInfoResp__Psr **) 0x0

(gdb) p resp.provider

There is no member named provider.

(gdb) p resp.interface

There is no member named interface.

(gdb) p resp.domain

There is no member named domain.

 

(gdb) p sizeof(resp)

$13 = 48

 

If you do the math, you can see that the size of ‘resp’ is correct if the struct ends with field psrs, there is no room in the struct for the new fields.

 

If I then step forward and enter routine mgmt__get_attach_info_resp__get_packed_size(), that routine DOES know about 7 fields and tries to reference all of them, but of course the resp structure on the stack isn’t big enough to hold the 7 fields, so this routine is looking at other junk on the stack past the end of the structure:

 

239           len = mgmt__get_attach_info_resp__get_packed_size(&resp);

(gdb) s

mgmt__get_attach_info_resp__get_packed_size (message=message@entry=0x2aaab041f740) at src/mgmt/srv.pb-c.c:295

295      assert(message->base.descriptor == &mgmt__get_attach_info_resp__descriptor);

(gdb) n

296      return protobuf_c_message_get_packed_size ((const ProtobufCMessage*)(message));

(gdb) p message

$11 = (const Mgmt__GetAttachInfoResp *) 0x2aaab041f740

(gdb) p *message

$12 = {base = {descriptor = 0x7f9756b5dcc0 <mgmt__get_attach_info_resp__descriptor>, n_unknown_fields = 0, unknown_fields = 0x0}, status = 0, n_psrs = 1, 

  psrs = 0x7f9396113e00, provider = 0x0, interface = 0xad26fa6a89442100 <Address 0xad26fa6a89442100 out of bounds>, domain = 0x7f96f4026a10 "\340\230VW\227\177", 

  crtctxshareaddr = 4093798800, crttimeout = 32662}

 

Happy hunting,

 

Kevan

 

 

That’s good information.  I added the “interface” field and some others last week as we are expanding the capabilities of the GetAttachInfo message to help automatically configure the clients. It’s not clear why adding the fields would cause any of the unpacking code to fail, especially when it’s auto generated based on the protobuf definition.  However, I did build those files with newer versions of the protobuf compiler, and it’s possible that there’s a subtle incompatibility that I wasn’t aware of. 

What about readelf -d install/bin/daos_io_server ?

I get this:

 

0x000000000000001d (RUNPATH)            Library runpath: [build/src/cart/src/cart:build/src/cart/src/gurt:/home/jvolivie/prebuilt/pmdk/lib:/home/jvolivie/prebuilt/isal/lib:/usr/lib:/home/jvolivie/prebuilt/argobots/lib:/home/jvolivie/prebuilt/protobufc/lib:$ORIGIN/../lib64/daos_srv:$ORIGIN/../lib64]

 

 

The $ORIGIN/../lib64 is used to get librdb.so….is your directory layout the same?   Do you have that entry in your RUNPATH ?

 

-Jeff

 

Yes, here you go.   In talking to the others on our team, they don’t use LD_LIBRARY_PATH, they just move the entire contents of lib64/daos_srv up one directory into lib64 and that works.   I used LD_LIBRARY_PATH so that I didn’t have to keep remembering to do the mv command every time.

 

Kevan

 

daos@hl-d109 ~/daos/install $ find lib lib64 -type f

lib/libabt.so.0.0.0

lib/libabt.la

lib/libabt.a

lib/pkgconfig/argobots.pc

lib/pkgconfig/libpmem.pc

lib/pkgconfig/libpmemobj.pc

lib/pkgconfig/libpmemlog.pc

lib/pkgconfig/libpmemblk.pc

lib/pkgconfig/libpmempool.pc

lib/pkgconfig/libfabric.pc

lib/pkgconfig/openpa.pc

lib/pkgconfig/mercury.pc

lib/pkgconfig/libprotobuf-c.pc

lib/libpmem.so.1.0.0

lib/libpmem.a

lib/pmdk_debug/libpmem.so.1.0.0

lib/pmdk_debug/libpmem.a

lib/pmdk_debug/libpmemlog.so.1.0.0

lib/pmdk_debug/libpmemlog.a

lib/pmdk_debug/libpmemblk.so.1.0.0

lib/pmdk_debug/libpmemblk.a

lib/pmdk_debug/libpmemobj.so.1.0.0

lib/pmdk_debug/libpmemobj.a

lib/pmdk_debug/libpmempool.so.1.0.0

lib/pmdk_debug/libpmempool.a

lib/libpmemlog.so.1.0.0

lib/libpmemlog.a

lib/libpmemblk.so.1.0.0

lib/libpmemblk.a

lib/libpmemobj.so.1.0.0

lib/libpmemobj.a

lib/libpmempool.so.1.0.0

lib/libpmempool.a

lib/libpsm2/libpsm2-compat.cmds

lib/libfabric.so.1.12.0

lib/libfabric.la

lib/libfabric.a

lib/libopa.la

lib/libopa.a

lib/libmercury.so.2.0.0

lib/libmercury_hl.so.2.0.0

lib/libmercury_util.so.2.0.0

lib/libna.so.2.0.0

lib/libprotobuf-c.so.1.0.0

lib/libprotobuf-c.la

lib/libprotobuf-c.a

lib/libspdk_log.a

lib/libspdk_log.so.2.0

lib/libspdk_util.a

lib/libspdk_util.so.2.0

lib/libspdk_conf.a

lib/libspdk_conf.so.2.0

lib/libspdk_thread.a

lib/libspdk_thread.so.2.0

lib/libspdk_json.a

lib/libspdk_json.so.2.0

lib/libspdk_jsonrpc.a

lib/libspdk_jsonrpc.so.2.0

lib/libspdk_rpc.a

lib/libspdk_rpc.so.2.0

lib/libspdk_notify.a

lib/libspdk_notify.so.2.0

lib/libspdk_trace.a

lib/libspdk_trace.so.2.0

lib/libspdk_bdev.a

lib/libspdk_bdev.so.2.0

lib/libspdk_blob.a

lib/libspdk_blob.so.2.0

lib/libspdk_blobfs.a

lib/libspdk_blobfs.so.2.0

lib/libspdk_copy.a

lib/libspdk_copy.so.2.0

lib/libspdk_event.a

lib/libspdk_event.so.2.0

lib/libspdk_log_rpc.a

lib/libspdk_log_rpc.so.2.0

lib/libspdk_lvol.a

lib/libspdk_lvol.so.2.0

lib/libspdk_net.a

lib/libspdk_net.so.2.0

lib/libspdk_sock.a

lib/libspdk_sock.so.2.0

lib/libspdk_nvme.a

lib/libspdk_nvme.so.2.0

lib/libspdk_vmd.a

lib/libspdk_vmd.so.2.0

lib/libspdk_nvmf.a

lib/libspdk_nvmf.so.2.0

lib/libspdk_scsi.a

lib/libspdk_scsi.so.2.0

lib/libspdk_ioat.a

lib/libspdk_ioat.so.2.0

lib/libspdk_ut_mock.a

lib/libspdk_ut_mock.so.2.0

lib/libspdk_iscsi.a

lib/libspdk_iscsi.so.2.0

lib/libspdk_nbd.a

lib/libspdk_nbd.so.2.0

lib/libspdk_ftl.a

lib/libspdk_ftl.so.2.0

lib/libspdk_virtio.a

lib/libspdk_virtio.so.2.0

lib/libspdk_env_dpdk.a

lib/libspdk_env_dpdk.so.2.0

lib/libspdk_blob_bdev.a

lib/libspdk_blob_bdev.so.2.0

lib/libspdk_bdev_delay.a

lib/libspdk_bdev_delay.so.2.0

lib/libspdk_bdev_error.a

lib/libspdk_bdev_error.so.2.0

lib/libspdk_bdev_gpt.a

lib/libspdk_bdev_gpt.so.2.0

lib/libspdk_bdev_lvol.a

lib/libspdk_bdev_lvol.so.2.0

lib/libspdk_bdev_malloc.a

lib/libspdk_bdev_malloc.so.2.0

lib/libspdk_bdev_null.a

lib/libspdk_bdev_null.so.2.0

lib/libspdk_bdev_nvme.a

lib/libspdk_bdev_nvme.so.2.0

lib/libspdk_bdev_passthru.a

lib/libspdk_bdev_passthru.so.2.0

lib/libspdk_bdev_raid.a

lib/libspdk_bdev_raid.so.2.0

lib/libspdk_bdev_rpc.a

lib/libspdk_bdev_rpc.so.2.0

lib/libspdk_bdev_split.a

lib/libspdk_bdev_split.so.2.0

lib/libspdk_bdev_zone_block.a

lib/libspdk_bdev_zone_block.so.2.0

lib/libspdk_bdev_aio.a

lib/libspdk_bdev_aio.so.2.0

lib/libspdk_bdev_ftl.a

lib/libspdk_bdev_ftl.so.2.0

lib/libspdk_bdev_virtio.a

lib/libspdk_bdev_virtio.so.2.0

lib/libspdk_blobfs_bdev.a

lib/libspdk_blobfs_bdev.so.2.0

lib/libspdk_copy_ioat.a

lib/libspdk_copy_ioat.so.2.0

lib/libspdk_event_copy.a

lib/libspdk_event_copy.so.2.0

lib/libspdk_event_vmd.a

lib/libspdk_event_vmd.so.2.0

lib/libspdk_event_bdev.a

lib/libspdk_event_bdev.so.2.0

lib/libspdk_event_scsi.a

lib/libspdk_event_scsi.so.2.0

lib/libspdk_event_iscsi.a

lib/libspdk_event_iscsi.so.2.0

lib/libspdk_event_net.a

lib/libspdk_event_net.so.2.0

lib/libspdk_event_nvmf.a

lib/libspdk_event_nvmf.so.2.0

lib/libspdk_event_nbd.a

lib/libspdk_event_nbd.so.2.0

lib/libspdk_app_rpc.a

lib/libspdk_app_rpc.so.2.0

lib/libspdk_sock_posix.a

lib/libspdk_sock_posix.so.2.0

lib/libspdk_env_dpdk_rpc.a

lib/libspdk_env_dpdk_rpc.so.2.0

lib/libspdk.so.2.0

lib/libdpdk.so

lib/librte_bus_pci.so

lib/librte_bus_pci.so.20.0

lib/librte_bus_vdev.so

lib/librte_bus_vdev.so.20.0

lib/librte_compressdev.so

lib/librte_compressdev.so.0.200

lib/librte_cryptodev.so

lib/librte_cryptodev.so.20.0

lib/librte_eal.so

lib/librte_eal.so.20.0

lib/cart/TESTING/corpc/cart_corpc_five_node.py

lib/cart/TESTING/corpc/cart_corpc_five_node.yaml

lib/cart/TESTING/corpc/cart_corpc_one_node.py

lib/cart/TESTING/corpc/cart_corpc_one_node.yaml

lib/cart/TESTING/corpc/cart_corpc_two_node.py

lib/cart/TESTING/corpc/cart_corpc_two_node.yaml

lib/cart/TESTING/ctl/cart_ctl_five_node.py

lib/cart/TESTING/ctl/cart_ctl_five_node.yaml

lib/cart/TESTING/ctl/cart_ctl_one_node.py

lib/cart/TESTING/ctl/cart_ctl_one_node.yaml

lib/cart/TESTING/group_test/cart_ghost_rank_prc_one_node.py

lib/cart/TESTING/group_test/cart_ghost_rank_prc_one_node.yaml

lib/cart/TESTING/group_test/group_test.py

lib/cart/TESTING/group_test/group_test.yaml

lib/cart/TESTING/group_tiers/cart_group_tiers_three_node.py

lib/cart/TESTING/group_tiers/cart_group_tiers_three_node.yaml

lib/cart/TESTING/iv/cart_iv_one_node.py

lib/cart/TESTING/iv/cart_iv_one_node.yaml

lib/cart/TESTING/iv/cart_iv_two_node.py

lib/cart/TESTING/iv/cart_iv_two_node.yaml

lib/cart/TESTING/launch.py

lib/cart/TESTING/no_pmix/cart_nopmix_multictx_one_node.py

lib/cart/TESTING/no_pmix/cart_nopmix_multictx_one_node.yaml

lib/cart/TESTING/nopmix_launcher/cart_nopmix_launcher_one_node.py

lib/cart/TESTING/nopmix_launcher/cart_nopmix_launcher_one_node.yaml

lib/cart/TESTING/rpc/cart_rpc_one_node.py

lib/cart/TESTING/rpc/cart_rpc_one_node.yaml

lib/cart/TESTING/rpc/cart_rpc_two_node.py

lib/cart/TESTING/rpc/cart_rpc_two_node.yaml

lib/cart/TESTING/selftest/cart_selftest_three_node.py

lib/cart/TESTING/selftest/cart_selftest_three_node.yaml

lib/cart/TESTING/util/cart_logparse.py

lib/cart/TESTING/util/cart_logtest.py

lib/cart/TESTING/util/cart_utils.py

lib/cart/TESTING/tests/no_pmix_corpc_errors

lib/cart/TESTING/tests/test_swim

lib/cart/TESTING/tests/test_group_np_cli

lib/cart/TESTING/tests/test_hlc_net

lib/cart/TESTING/tests/no_pmix_multi_ctx

lib/cart/TESTING/tests/test_corpc_exclusive

lib/cart/TESTING/tests/iv_client

lib/cart/TESTING/tests/no_pmix_launcher_client

lib/cart/TESTING/tests/test_proto_client

lib/cart/TESTING/tests/threaded_server

lib/cart/TESTING/tests/test_no_timeout

lib/cart/TESTING/tests/test_ep_cred_server

lib/cart/TESTING/tests/test_rpc_to_ghost_rank

lib/cart/TESTING/tests/iv_server

lib/cart/TESTING/tests/test_ep_cred_client

lib/cart/TESTING/tests/test_swim_net

lib/cart/TESTING/tests/no_pmix_group_version

lib/cart/TESTING/tests/test_group_np_srv

lib/cart/TESTING/tests/test_corpc_prefwd

lib/cart/TESTING/tests/threaded_client

lib/cart/TESTING/tests/test_proto_server

lib/cart/TESTING/tests/no_pmix_launcher_server

lib/cart/TESTING/tests/no_pmix_group_test

lib/librte_fib.so

lib/librte_fib.so.0.200

lib/librte_kvargs.so

lib/librte_kvargs.so.20.0

lib/librte_mbuf.so

lib/librte_mbuf.so.20.0

lib/librte_mempool_bucket.so

lib/librte_mempool_bucket.so.20.0

lib/librte_mempool_ring.so

lib/librte_mempool_ring.so.20.0

lib/librte_mempool.so

lib/librte_mempool.so.20.0

lib/librte_net.so

lib/librte_net.so.20.0

lib/librte_pci.so

lib/librte_pci.so.20.0

lib/librte_rib.so

lib/librte_rib.so.0.200

lib/librte_ring.so

lib/librte_ring.so.20.0

lib/libisal.so.2.0.26

lib/libisal.la

lib/libisal.a

lib/daos/.build_vars.json

lib/daos/.build_vars.sh

lib/daos/TESTING/ftest/checksum/basic_checksum.py

lib/daos/TESTING/ftest/checksum/basic_checksum.yaml

lib/daos/TESTING/ftest/checksum/csum_error_logging.py

lib/daos/TESTING/ftest/checksum/csum_error_logging.yaml

lib/daos/TESTING/ftest/container/attribute.py

lib/daos/TESTING/ftest/container/attribute.yaml

lib/daos/TESTING/ftest/container/basic_snapshot.py

lib/daos/TESTING/ftest/container/basic_snapshot.yaml

lib/daos/TESTING/ftest/container/basic_tx_test.py

lib/daos/TESTING/ftest/container/basic_tx_test.yaml

lib/daos/TESTING/ftest/container/container_async.py

lib/daos/TESTING/ftest/container/container_async.yaml

lib/daos/TESTING/ftest/container/create.py

lib/daos/TESTING/ftest/container/create.yaml

lib/daos/TESTING/ftest/container/delete.py

lib/daos/TESTING/ftest/container/delete.yaml

lib/daos/TESTING/ftest/container/full_pool_container_create.py

lib/daos/TESTING/ftest/container/full_pool_container_create.yaml

lib/daos/TESTING/ftest/container/global_handle.py

lib/daos/TESTING/ftest/container/global_handle.yaml

lib/daos/TESTING/ftest/container/open.py

lib/daos/TESTING/ftest/container/open.yaml

lib/daos/TESTING/ftest/container/open_close.py

lib/daos/TESTING/ftest/container/open_close.yaml

lib/daos/TESTING/ftest/container/simple_create_delete_test.py

lib/daos/TESTING/ftest/container/simple_create_delete_test.yaml

lib/daos/TESTING/ftest/container/snapshot.py

lib/daos/TESTING/ftest/container/snapshot.yaml

lib/daos/TESTING/ftest/control/daos_admin_privileged.py

lib/daos/TESTING/ftest/control/daos_admin_privileged.yaml

lib/daos/TESTING/ftest/control/dmg_nvme_scan_test.py

lib/daos/TESTING/ftest/control/dmg_nvme_scan_test.yaml

lib/daos/TESTING/ftest/control/super_block_versioning.py

lib/daos/TESTING/ftest/control/super_block_versioning.yaml

lib/daos/TESTING/ftest/daos_test/daos_core_test-rebuild.py

lib/daos/TESTING/ftest/daos_test/daos_core_test-rebuild.yaml

lib/daos/TESTING/ftest/daos_test/daos_core_test.py

lib/daos/TESTING/ftest/daos_test/daos_core_test.yaml

lib/daos/TESTING/ftest/data/daos_agent_baseline.yaml

lib/daos/TESTING/ftest/data/daos_server_baseline.yaml

lib/daos/TESTING/ftest/io/daos_perf.py

lib/daos/TESTING/ftest/io/daos_perf.yaml

lib/daos/TESTING/ftest/io/daos_perf_large.py

lib/daos/TESTING/ftest/io/daos_perf_large.yaml

lib/daos/TESTING/ftest/io/daos_racer.py

lib/daos/TESTING/ftest/io/daos_racer.yaml

lib/daos/TESTING/ftest/io/fio_small.py

lib/daos/TESTING/ftest/io/fio_small.yaml

lib/daos/TESTING/ftest/io/hdf5.py

lib/daos/TESTING/ftest/io/hdf5.yaml

lib/daos/TESTING/ftest/io/ior_intercept_basic.py

lib/daos/TESTING/ftest/io/ior_intercept_basic.yaml

lib/daos/TESTING/ftest/io/ior_intercept_dfuse_mix.py

lib/daos/TESTING/ftest/io/ior_intercept_dfuse_mix.yaml

lib/daos/TESTING/ftest/io/ior_intercept_multi_client.py

lib/daos/TESTING/ftest/io/ior_intercept_multi_client.yaml

lib/daos/TESTING/ftest/io/ior_intercept_verify_data_integrity.py

lib/daos/TESTING/ftest/io/ior_intercept_verify_data_integrity.yaml

lib/daos/TESTING/ftest/io/ior_large.py

lib/daos/TESTING/ftest/io/ior_large.yaml

lib/daos/TESTING/ftest/io/ior_small.py

lib/daos/TESTING/ftest/io/ior_small.yaml

lib/daos/TESTING/ftest/io/llnl_mpi4py.py

lib/daos/TESTING/ftest/io/llnl_mpi4py.yaml

lib/daos/TESTING/ftest/io/mdtest_large.py

lib/daos/TESTING/ftest/io/mdtest_large.yaml

lib/daos/TESTING/ftest/io/mdtest_small.py

lib/daos/TESTING/ftest/io/mdtest_small.yaml

lib/daos/TESTING/ftest/io/nvme_fragmentation.py

lib/daos/TESTING/ftest/io/nvme_fragmentation.yaml

lib/daos/TESTING/ftest/io/nvme_io.py

lib/daos/TESTING/ftest/io/nvme_io.yaml

lib/daos/TESTING/ftest/io/nvme_io_verification.py

lib/daos/TESTING/ftest/io/nvme_io_verification.yaml

lib/daos/TESTING/ftest/io/nvme_object.py

lib/daos/TESTING/ftest/io/nvme_object.yaml

lib/daos/TESTING/ftest/io/romio.py

lib/daos/TESTING/ftest/io/romio.yaml

lib/daos/TESTING/ftest/io/seg_count.py

lib/daos/TESTING/ftest/io/seg_count.yaml

lib/daos/TESTING/ftest/io/unaligned_io.py

lib/daos/TESTING/ftest/io/unaligned_io.yaml

lib/daos/TESTING/ftest/launch.py

lib/daos/TESTING/ftest/metrics.py

lib/daos/TESTING/ftest/network/cart_self_test.py

lib/daos/TESTING/ftest/network/cart_self_test.yaml

lib/daos/TESTING/ftest/object/array_obj_test.py

lib/daos/TESTING/ftest/object/array_obj_test.yaml

lib/daos/TESTING/ftest/object/create_many_dkeys.py

lib/daos/TESTING/ftest/object/create_many_dkeys.yaml

lib/daos/TESTING/ftest/object/obj_fetch_bad_param.py

lib/daos/TESTING/ftest/object/obj_fetch_bad_param.yaml

lib/daos/TESTING/ftest/object/obj_open_bad_param.py

lib/daos/TESTING/ftest/object/obj_open_bad_param.yaml

lib/daos/TESTING/ftest/object/obj_update_bad_param.py

lib/daos/TESTING/ftest/object/obj_update_bad_param.yaml

lib/daos/TESTING/ftest/object/object_integrity.py

lib/daos/TESTING/ftest/object/object_integrity.yaml

lib/daos/TESTING/ftest/object/punch_test.py

lib/daos/TESTING/ftest/object/punch_test.yaml

lib/daos/TESTING/ftest/object/same_key_different_value.py

lib/daos/TESTING/ftest/object/same_key_different_value.yaml

lib/daos/TESTING/ftest/pool/attribute.py

lib/daos/TESTING/ftest/pool/attribute.yaml

lib/daos/TESTING/ftest/pool/bad_connect.py

lib/daos/TESTING/ftest/pool/bad_connect.yaml

lib/daos/TESTING/ftest/pool/bad_create.py

lib/daos/TESTING/ftest/pool/bad_create.yaml

lib/daos/TESTING/ftest/pool/bad_evict.py

lib/daos/TESTING/ftest/pool/bad_evict.yaml

lib/daos/TESTING/ftest/pool/bad_exclude.py

lib/daos/TESTING/ftest/pool/bad_exclude.yaml

lib/daos/TESTING/ftest/pool/bad_query.py

lib/daos/TESTING/ftest/pool/bad_query.yaml

lib/daos/TESTING/ftest/pool/connect_test.py

lib/daos/TESTING/ftest/pool/connect_test.yaml

lib/daos/TESTING/ftest/pool/destroy_rebuild.py

lib/daos/TESTING/ftest/pool/destroy_rebuild.yaml

lib/daos/TESTING/ftest/pool/destroy_tests.py

lib/daos/TESTING/ftest/pool/destroy_tests.yaml

lib/daos/TESTING/ftest/pool/evict_test.py

lib/daos/TESTING/ftest/pool/evict_test.yaml

lib/daos/TESTING/ftest/pool/global_handle.py

lib/daos/TESTING/ftest/pool/global_handle.yaml

lib/daos/TESTING/ftest/pool/info_tests.py

lib/daos/TESTING/ftest/pool/info_tests.yaml

lib/daos/TESTING/ftest/pool/multi_server_create_delete_test.py

lib/daos/TESTING/ftest/pool/multi_server_create_delete_test.yaml

lib/daos/TESTING/ftest/pool/multiple_creates_test.py

lib/daos/TESTING/ftest/pool/multiple_creates_test.yaml

lib/daos/TESTING/ftest/pool/permission.py

lib/daos/TESTING/ftest/pool/permission.yaml

lib/daos/TESTING/ftest/pool/pool_svc.py

lib/daos/TESTING/ftest/pool/pool_svc.yaml

lib/daos/TESTING/ftest/pool/rebuild_no_cap.py

lib/daos/TESTING/ftest/pool/rebuild_no_cap.yaml

lib/daos/TESTING/ftest/pool/rebuild_tests.py

lib/daos/TESTING/ftest/pool/rebuild_tests.yaml

lib/daos/TESTING/ftest/pool/rebuild_with_io.py

lib/daos/TESTING/ftest/pool/rebuild_with_io.yaml

lib/daos/TESTING/ftest/pool/rebuild_with_ior.py

lib/daos/TESTING/ftest/pool/rebuild_with_ior.yaml

lib/daos/TESTING/ftest/pool/simple_create_delete_test.py

lib/daos/TESTING/ftest/pool/simple_create_delete_test.yaml

lib/daos/TESTING/ftest/rebuild/cascading_failures.py

lib/daos/TESTING/ftest/rebuild/cascading_failures.yaml

lib/daos/TESTING/ftest/rebuild/container_create.py

lib/daos/TESTING/ftest/rebuild/container_create.yaml

lib/daos/TESTING/ftest/rebuild/delete_objects.py

lib/daos/TESTING/ftest/rebuild/delete_objects.yaml

lib/daos/TESTING/ftest/rebuild/io_conf_run.py

lib/daos/TESTING/ftest/rebuild/io_conf_run.yaml

lib/daos/TESTING/ftest/rebuild/read_array.py

lib/daos/TESTING/ftest/rebuild/read_array.yaml

lib/daos/TESTING/ftest/security/pool_connect_init.py

lib/daos/TESTING/ftest/security/pool_connect_init.yaml

lib/daos/TESTING/ftest/security/pool_security_acl.py

lib/daos/TESTING/ftest/security/pool_security_acl.yaml

lib/daos/TESTING/ftest/security/pool_security_groups.py

lib/daos/TESTING/ftest/security/pool_security_groups.yaml

lib/daos/TESTING/ftest/server/daos_server_config.py

lib/daos/TESTING/ftest/server/daos_server_config.yaml

lib/daos/TESTING/ftest/server/metadata.py

lib/daos/TESTING/ftest/server/metadata.yaml

lib/daos/TESTING/ftest/slurm_setup.py

lib/daos/TESTING/ftest/soak/soak.py

lib/daos/TESTING/ftest/soak/soak.yaml

lib/daos/TESTING/ftest/unittest/unittest.py

lib/daos/TESTING/ftest/unittest/unittest.yaml

lib/daos/TESTING/ftest/util/__init__.py

lib/daos/TESTING/ftest/util/agent_utils.py

lib/daos/TESTING/ftest/util/apricot/VERSION

lib/daos/TESTING/ftest/util/apricot/apricot/__init__.py

lib/daos/TESTING/ftest/util/apricot/apricot/test.py

lib/daos/TESTING/ftest/util/apricot/setup.py

lib/daos/TESTING/ftest/util/check_for_pool.py

lib/daos/TESTING/ftest/util/command_utils.py

lib/daos/TESTING/ftest/util/configuration_utils.py

lib/daos/TESTING/ftest/util/daos_core_base.py

lib/daos/TESTING/ftest/util/daos_io_conf.py

lib/daos/TESTING/ftest/util/daos_perf_utils.py

lib/daos/TESTING/ftest/util/daos_racer_utils.py

lib/daos/TESTING/ftest/util/daos_utils.py

lib/daos/TESTING/ftest/util/dfuse_utils.py

lib/daos/TESTING/ftest/util/dmg_utils.py

lib/daos/TESTING/ftest/util/env_modules.py

lib/daos/TESTING/ftest/util/fault_config_utils.py

lib/daos/TESTING/ftest/util/fio_test_base.py

lib/daos/TESTING/ftest/util/fio_utils.py

lib/daos/TESTING/ftest/util/general_utils.py

lib/daos/TESTING/ftest/util/get_hosts_from_file.py

lib/daos/TESTING/ftest/util/io_utilities.py

lib/daos/TESTING/ftest/util/ior_test_base.py

lib/daos/TESTING/ftest/util/ior_utils.py

lib/daos/TESTING/ftest/util/mdtest_test_base.py

lib/daos/TESTING/ftest/util/mdtest_utils.py

lib/daos/TESTING/ftest/util/mpio_test_base.py

lib/daos/TESTING/ftest/util/mpio_utils.py

lib/daos/TESTING/ftest/util/pool_security_test_base.py

lib/daos/TESTING/ftest/util/rebuild_test_base.py

lib/daos/TESTING/ftest/util/server_utils.py

lib/daos/TESTING/ftest/util/slurm_utils.py

lib/daos/TESTING/ftest/util/test_utils_base.py

lib/daos/TESTING/ftest/util/test_utils_container.py

lib/daos/TESTING/ftest/util/test_utils_pool.py

lib/daos/TESTING/ftest/util/write_host_file.py

lib/daos/TESTING/ftest/util/write_some_data.py

lib/daos/TESTING/scripts/check_ioil_syms

lib/daos/TESTING/tests/lf_s_test_ioil

lib/daos/TESTING/tests/s_test_ioil

lib64/psm2-compat/libpsm_infinipath.so.1

lib64/libpsm2.so.2.1

lib64/libpsm2.a

lib64/libfuse3.so.3.5.0

lib64/pkgconfig/fuse3.pc

lib64/libvos.so

lib64/daos/API_VERSION

lib64/daos/VERSION

lib64/daos/certgen/admin.cnf

lib64/daos/certgen/agent.cnf

lib64/daos/certgen/ca.cnf

lib64/daos/certgen/gen_certificates.sh

lib64/daos/certgen/server.cnf

lib64/daos_srv/libbio.so

lib64/daos_srv/libplacement.so

lib64/daos_srv/libdtx.so

lib64/daos_srv/libvos_srv.so

lib64/daos_srv/libcont.so

lib64/daos_srv/libsecurity.so

lib64/daos_srv/libmgmt.so

lib64/daos_srv/librdbt.so

lib64/daos_srv/librsvc.so

lib64/daos_srv/libobj.so

lib64/daos_srv/librebuild.so

lib64/daos_srv/librdb.so

lib64/daos_srv/libpool.so

lib64/libioil.a

lib64/libgurt.so.4.7.0

lib64/libnvme_control.a

lib64/python2.7/site-packages/pydaos/__init__.py

lib64/python2.7/site-packages/pydaos/dbm_daos.py

lib64/python2.7/site-packages/pydaos/pydaos_core.py

lib64/python2.7/site-packages/pydaos/raw/__init__.py

lib64/python2.7/site-packages/pydaos/raw/conversion.py

lib64/python2.7/site-packages/pydaos/raw/daos_api.py

lib64/python2.7/site-packages/pydaos/raw/daos_cref.py

lib64/python2.7/site-packages/pydaos/raw/daos_io.py

lib64/python2.7/site-packages/pydaos/pydaos_shim_27.so

lib64/python3/site-packages/pydaos/__init__.py

lib64/python3/site-packages/pydaos/pydaos_core.py

lib64/python3/site-packages/pydaos/dbm_daos.py

lib64/python3/site-packages/pydaos/raw/__init__.py

lib64/python3/site-packages/pydaos/raw/conversion.py

lib64/python3/site-packages/pydaos/raw/daos_api.py

lib64/python3/site-packages/pydaos/raw/daos_cref.py

lib64/python3/site-packages/pydaos/raw/daos_io.py

lib64/python3/site-packages/pydaos/pydaos_shim_3.so

lib64/libdfuse.a

lib64/libcart.so.4.7.0

lib64/libdaos_common.so

lib64/libdaos_tests.so

lib64/libdaos.so.0.9.0

lib64/libioil.so

lib64/libdfs.so

lib64/libdfuse.so

lib64/libduns.so

daos@hl-d109 ~/daos/install $ 

 

 

Can you also dump the contents of lib, lib64, and lib64/daos_srv ?

 

Jeff,

 

Yes I am building from scratch, and RPATH used to work.   Here is the output, note that librdb doesn’t show up in the ldd output.

 

Thanks, Kevan

 

[root@delphi-004 ~]# cd ~daos/daos/install/bin

[root@delphi-004 bin]# ls

acl_dump_test  daosctl           daos_test        drpc_test    igzip                pmreorder           spdk_nvme_perf

agent_tests    daos_gen_io_conf  dcont            eq_tests     io_conf              rdbt                spdk_tgt

cart_ctl       daos_io_server    dfuse            evt_ctl      jump_pl_map          ring_pl_map         vea_ut

crt_launch     daos_perf         dfuse_hl         fi_info      nvme_control_ctests  security_test       vos_size

daos           daos_racer        dmg              fi_pingpong  obj_ctl              self_test           vos_size.py

daos_admin     daos_run_io_conf  dmg_old          fi_strerror  pl_bench             smd_ut              vos_tests

daos_agent     daos_server       drpc_iosrv_test  hello_drpc   pmempool             spdk_nvme_identify

[root@delphi-004 bin]# ldd daos_io_server

       linux-vdso.so.1 =>  (0x00007ffc47293000)

       libdaos_common.so => /home/users/daos/daos/install/lib64/libdaos_common.so (0x00007f69369cd000)

       libgurt.so.4 => /home/users/daos/daos/install/lib64/libgurt.so.4 (0x00007f69367aa000)

       libcart.so.4 => /home/users/daos/daos/install/lib64/libcart.so.4 (0x00007f69364c9000)

       libvos_srv.so => /home/users/daos/daos/install/bin/./../lib64/daos_srv/libvos_srv.so (0x00007f693622c000)

       libdaos.so.0 => /home/users/daos/daos/install/lib64/libdaos.so.0 (0x00007f6935f51000)

       libbio.so => /home/users/daos/daos/install/bin/./../lib64/daos_srv/libbio.so (0x00007f6935d26000)

       libdl.so.2 => /lib64/libdl.so.2 (0x00007f6935b22000)

       libuuid.so.1 => /lib64/libuuid.so.1 (0x00007f693591d000)

       libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f6935701000)

       libabt.so.0 => /home/users/daos/daos/install/lib/libabt.so.0 (0x00007f69354df000)

       libhwloc.so.5 => /lib64/libhwloc.so.5 (0x00007f69352a2000)

       libpmemobj.so.1 => /home/users/daos/daos/install/lib/libpmemobj.so.1 (0x00007f6935061000)

       libprotobuf-c.so.1 => /home/users/daos/daos/install/lib/libprotobuf-c.so.1 (0x00007f6934e58000)

       libc.so.6 => /lib64/libc.so.6 (0x00007f6934a8a000)

       libisal.so.2 => /home/users/daos/daos/install/lib/libisal.so.2 (0x00007f693484c000)

       /lib64/ld-linux-x86-64.so.2 (0x00007f6936c3c000)

       libyaml-0.so.2 => /lib64/libyaml-0.so.2 (0x00007f693462c000)

       libmercury.so.2 => /home/users/daos/daos/install/lib/libmercury.so.2 (0x00007f6934412000)

       libna.so.2 => /home/users/daos/daos/install/lib/libna.so.2 (0x00007f69341f4000)

       libmercury_util.so.2 => /home/users/daos/daos/install/lib/libmercury_util.so.2 (0x00007f6933fed000)

       libspdk_env_dpdk.so.2.0 => /home/users/daos/daos/install/lib/libspdk_env_dpdk.so.2.0 (0x00007f6933dde000)

       libspdk_thread.so.2.0 => /home/users/daos/daos/install/lib/libspdk_thread.so.2.0 (0x00007f6933bd8000)

       libspdk_bdev.so.2.0 => /home/users/daos/daos/install/lib/libspdk_bdev.so.2.0 (0x00007f69339c2000)

       libspdk_copy.so.2.0 => /home/users/daos/daos/install/lib/libspdk_copy.so.2.0 (0x00007f69337bf000)

       librte_mempool.so.20.0 => /home/users/daos/daos/install/lib/librte_mempool.so.20.0 (0x00007f69335b9000)

       librte_mempool_ring.so.20.0 => /home/users/daos/daos/install/lib/librte_mempool_ring.so.20.0 (0x00007f69333b6000)

       librte_bus_pci.so.20.0 => /home/users/daos/daos/install/lib/librte_bus_pci.so.20.0 (0x00007f69331ab000)

       librte_pci.so.20.0 => /home/users/daos/daos/install/lib/librte_pci.so.20.0 (0x00007f6932fa8000)

       librte_ring.so.20.0 => /home/users/daos/daos/install/lib/librte_ring.so.20.0 (0x00007f6932da5000)

       librte_mbuf.so.20.0 => /home/users/daos/daos/install/lib/librte_mbuf.so.20.0 (0x00007f6932b9a000)

       librte_eal.so.20.0 => /home/users/daos/daos/install/lib/librte_eal.so.20.0 (0x00007f69328bf000)

       librte_kvargs.so.20.0 => /home/users/daos/daos/install/lib/librte_kvargs.so.20.0 (0x00007f69326bc000)

       libspdk_bdev_aio.so.2.0 => /home/users/daos/daos/install/lib/libspdk_bdev_aio.so.2.0 (0x00007f69324b6000)

       libspdk_bdev_nvme.so.2.0 => /home/users/daos/daos/install/lib/libspdk_bdev_nvme.so.2.0 (0x00007f693229d000)

       libspdk_bdev_malloc.so.2.0 => /home/users/daos/daos/install/lib/libspdk_bdev_malloc.so.2.0 (0x00007f6932098000)

       libspdk_conf.so.2.0 => /home/users/daos/daos/install/lib/libspdk_conf.so.2.0 (0x00007f6931e94000)

       libspdk_blob.so.2.0 => /home/users/daos/daos/install/lib/libspdk_blob.so.2.0 (0x00007f6931c80000)

       libspdk_nvme.so.2.0 => /home/users/daos/daos/install/lib/libspdk_nvme.so.2.0 (0x00007f6931a3d000)

       libspdk_util.so.2.0 => /home/users/daos/daos/install/lib/libspdk_util.so.2.0 (0x00007f693182f000)

       libspdk_json.so.2.0 => /home/users/daos/daos/install/lib/libspdk_json.so.2.0 (0x00007f6931627000)

       libspdk_jsonrpc.so.2.0 => /home/users/daos/daos/install/lib/libspdk_jsonrpc.so.2.0 (0x00007f6931420000)

       libspdk_rpc.so.2.0 => /home/users/daos/daos/install/lib/libspdk_rpc.so.2.0 (0x00007f693121c000)

       libspdk_trace.so.2.0 => /home/users/daos/daos/install/lib/libspdk_trace.so.2.0 (0x00007f6931017000)

       libspdk_sock.so.2.0 => /home/users/daos/daos/install/lib/libspdk_sock.so.2.0 (0x00007f6930e13000)

       libspdk_log.so.2.0 => /home/users/daos/daos/install/lib/libspdk_log.so.2.0 (0x00007f6930c10000)

       libspdk_notify.so.2.0 => /home/users/daos/daos/install/lib/libspdk_notify.so.2.0 (0x00007f69309cd000)

       libspdk_blob_bdev.so.2.0 => /home/users/daos/daos/install/lib/libspdk_blob_bdev.so.2.0 (0x00007f69307c9000)

       libnuma.so.1 => /lib64/libnuma.so.1 (0x00007f69305be000)

       libm.so.6 => /lib64/libm.so.6 (0x00007f69302bc000)

       libltdl.so.7 => /lib64/libltdl.so.7 (0x00007f69300b2000)

       libpmem.so.1 => /home/users/daos/daos/install/lib/libpmem.so.1 (0x00007f692fe89000)

       librt.so.1 => /lib64/librt.so.1 (0x00007f692fc81000)

       libfabric.so.1 => /home/users/daos/daos/install/lib/libfabric.so.1 (0x00007f692f8a7000)

       libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f692f691000)

       libaio.so.1 => /lib64/libaio.so.1 (0x00007f692f48f000)

       libibverbs.so.1 => /lib64/libibverbs.so.1 (0x00007f692f274000)

       librdmacm.so.1 => /lib64/librdmacm.so.1 (0x00007f692f05b000)

       libnl-3.so.200 => /lib64/libnl-3.so.200 (0x00007f692ee3a000)

       libnl-route-3.so.200 => /lib64/libnl-route-3.so.200 (0x00007f692ebcd000)

       libpsm2.so.2 => /home/users/daos/daos/install/lib64/libpsm2.so.2 (0x00007f692e96a000)

 

 

Hi Kevan,

 

Are you building daos from scratch?

 

You should not need to set LD_LIBRARY_PATH to run the IO server in such a setup because daos_io_server is built using RPATH.   Can you do an ldd on daos_io_server ?   If you do use LD_LIBRARY_PATH, make sure lib64 comes before any occurrence of lib

 

-Jeff

 

Joel,

 

I came to the same conclusion, sorry for wasting your time.  

 

There is currently an issue where the daos_io_server dies immediately because it can’t find its own librdb.so module, which got moved into lib64/daos_srv.   If I move librdb.so to lib then it complains about other modules.  What is the correct way to configure for this?  

 

04/13-04:20:00.47 delphi-004 DAOS[70088/70088] server ERR  src/iosrv/module.c:105 dss_module_load() cannot load librdb.so: librdb.so: cannot open shared object file: No such file or directory

04/13-04:20:00.47 delphi-004 DAOS[70088/70088] server ERR  src/iosrv/init.c:195 modules_load() Failed to load module rdb: -1003

 

To work around this, I set LD_LIBRARY_PATH in the environ section of daos_server.yml to include all library-related subdirectories within the install tree.   And to get the install_dir pushed out to all the server nodes I use rsync.   By default rsync doesn’t delete files at the destination if they are not in the source, so as libraries move around in the install tree over time, I eventually ended up with two copies of the same .so in different directories, and the LD_LIBRARY_PATH resulted in the wrong one being picked.

 

Sorry, Kevan

 

Hi Kevan,

 

I ran your test locally in my environment on master and it encountered no issues.  Figures, right?   I spent some time looking at the auto-generated code to see what it is doing.  I don’t have any particular expertise in that.  But, it’s clear that if any of the parts are out of sync, it will not work.  I’m wondering if you have any stale protobuf files on your machine.  Can you diff the protobuf files on your machine against 1) daos master, and 2) your coworker’s machine to see if all three sets are equal?

 

There is a full list in src/proto/Makefile.  Of particular interest are these three:

 

src/mgmt/srv.pb-c.c

src/mgmt/srv.pb-c.h

src/control/common/proto/mgmt/srv.pb.go

 

Joel

 

 

Joel,

 

Thanks for the explanation below, makes sense.   Can’t wait for that code to land.

 

Back to the problem at hand;  now I am even more confused….   I borrowed one of my compatriot’s machines, breakpointed his daos_io_server in routine ds_mgmt_drpc_get_attach_info, in his daemon the resp structure has all 7 fields, and so he doesn’t get a segfault.   We are building with the same commit  point.   ????? 

 

Do you have any ideas on what could be different in my machine?   Same centos 7 release.   I will keep debugging, it again appears to be related to my environment somehow.

 

Kevan

 

Hi Kevan,

 

You are right that it won’t help a client to know the interface and domain names of the server.  In this case, we’re not actually sending the server’s interface and domain in the server’s response.  These fields are left empty until they are populated by the agent.  On the update I am working on now, the agent scans the client machine for network interfaces that support the server’s provider (based on the GetAttachInfo provider data) and populates the interface and domain fields in the response sent to the client.  In an update after that, the libdaos library then gets some rework to generate a GetAttachInfo prior to initializing CaRT so that it can use the interface and domain data that’s returned to it.  I’m working on getting this through review now.

 

Thanks for the additional debug log.  I appreciate your insight and help.  I will work on replicating the problem locally so I can fix it.

 

Joel

 

 

Joel,

 

I’m curious; how does it help a client to know the interface and domain names of this server?   I can’t see how the client could possibly use them.

 

Anyway, back to the problem.   I am breakpointed in ds_mgmt_drpc_get_attach_info().   At the top of the routine is this:

 

        Mgmt__GetAttachInfoResp  resp = MGMT__GET_ATTACH_INFO_RESP__INIT;

 

If I look in the code at the definition of Mgmt__GetAttachInfoResp it has the 7 data fields including your new interface field, etc.  And the value of MGMT__GET_ATTACH_INFO_RESP__INIT appears to initialize all 7 of those fields.   But if I use gdb to look at that structure you can see that the code doesn’t actually know about any of the new fields, it is only aware of status and n_psrs/psrs:

 

(gdb) p resp

$7 = {base = {descriptor = 0x7f9756b5dcc0 <mgmt__get_attach_info_resp__descriptor>, n_unknown_fields = 0, unknown_fields = 0x0}, status = 0, n_psrs = 0, psrs = 0x0}

(gdb) p resp.status

$8 = 0

(gdb) p resp.n_psrs

$9 = 0

(gdb) p resp.psrs

$10 = (Mgmt__GetAttachInfoResp__Psr **) 0x0

(gdb) p resp.provider

There is no member named provider.

(gdb) p resp.interface

There is no member named interface.

(gdb) p resp.domain

There is no member named domain.

 

(gdb) p sizeof(resp)

$13 = 48

 

If you do the math, you can see that the size of ‘resp’ is correct if the struct ends with field psrs, there is no room in the struct for the new fields.

 

If I then step forward and enter routine mgmt__get_attach_info_resp__get_packed_size(), that routine DOES know about 7 fields and tries to reference all of them, but of course the resp structure on the stack isn’t big enough to hold the 7 fields, so this routine is looking at other junk on the stack past the end of the structure:

 

239           len = mgmt__get_attach_info_resp__get_packed_size(&resp);

(gdb) s

mgmt__get_attach_info_resp__get_packed_size (message=message@entry=0x2aaab041f740) at src/mgmt/srv.pb-c.c:295

295      assert(message->base.descriptor == &mgmt__get_attach_info_resp__descriptor);

(gdb) n

296      return protobuf_c_message_get_packed_size ((const ProtobufCMessage*)(message));

(gdb) p message

$11 = (const Mgmt__GetAttachInfoResp *) 0x2aaab041f740

(gdb) p *message

$12 = {base = {descriptor = 0x7f9756b5dcc0 <mgmt__get_attach_info_resp__descriptor>, n_unknown_fields = 0, unknown_fields = 0x0}, status = 0, n_psrs = 1, 

  psrs = 0x7f9396113e00, provider = 0x0, interface = 0xad26fa6a89442100 <Address 0xad26fa6a89442100 out of bounds>, domain = 0x7f96f4026a10 "\340\230VW\227\177", 

  crtctxshareaddr = 4093798800, crttimeout = 32662}

 

Happy hunting,

 

Kevan

 

 

That’s good information.  I added the “interface” field and some others last week as we are expanding the capabilities of the GetAttachInfo message to help automatically configure the clients. It’s not clear why adding the fields would cause any of the unpacking code to fail, especially when it’s auto generated based on the protobuf definition.  However, I did build those files with newer versions of the protobuf compiler, and it’s possible that there’s a subtle incompatibility that I wasn’t aware of.