daos@daos.groups.io

Re: dfuse mount error in centos 7

Groot

OK, thanks. And I want to know if I must mount the dfuse before using the libioil interception library?

Thanks a lot.
Groot

Re: dfuse mount error in centos 7

Hennecke, Michael

#1577

Hi,

please make sure that you are on CentOS 7.9, your kernel level indicates that you may be on an older level?

Best,

Michael

Show quoted text

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of Groot
Sent: Saturday, 9 April 2022 16:57
To: daos@daos.groups.io
Subject: [daos] dfuse mount error in centos 7

I create the pool and container with POSIX type successfully. But I mount the dfs by dfuse failed and I get the error message as follow:
fuse: error: filesystem requested capabilities 0x10000 that are not supported by kernel, aborting.
My system is Centos 7 and the kernel is 3.10.0-957. I install the daos services by the yum repo.

Thanks a lot.

Intel Deutschland GmbH
Registered Address: Am Campeon 10, 85579 Neubiberg, Germany
Tel: +49 89 99 8853-0, www.intel.de
Managing Directors: Christin Eisenschmid, Sharon Heck, Tiffany Doon Silva
Chairperson of the Supervisory Board: Nicole Lau
Registered Office: Munich
Commercial Register: Amtsgericht Muenchen HRB 186928

dfuse mount error in centos 7

Groot

#1576

I create the pool and container with POSIX type successfully. But I mount the dfs by dfuse failed and I get the error message as follow:
fuse: error: filesystem requested capabilities 0x10000 that are not supported by kernel, aborting.
My system is Centos 7 and the kernel is 3.10.0-957. I install the daos services by the yum repo.

Thanks a lot.

DAOS Community Update / Apr'22

Lombardi, Johann

#1575

Hi there,

Please find below the DAOS community newsletter for April 2022.

Past Events

Dell Technologies HPC Community Event (February 9^th)
DAOS: High Performance Storage for All HPC Workloads and Scales
Kelsey Prantis (Intel)
https://www.dellhpc.org/event_2-9-2022.html
2022 Energy HPC Conference (March 3^rd)
DAOS Use at Argonne
Kevin Harms (Argonne)
Johann Lombardi (Intel)
Mohamad Chaarawi (Intel)
https://energyhpc.rice.edu/program/

Upcoming Events

Salishan Conference On High Speed Computing (April 26^th)
Accelerating Data-driven Workflows with DAOS
Johann Lombardi (Intel)
ECP BoF (May 11^th)
DAOS Next Generation Storage
Kevin Harms (ANL)
Mohamad Chaarawi (Intel)
Johann Lombardi (Intel)
ISC BoF (May 30^th or June 1^st)
Accelerating HPC and AI with DAOS Storage
Kevin Harms (ANL)
Michael Hennecke (Intel)
Johann Lombardi (Intel)

Release

Current stable release is 2.0.2. See https://docs.daos.io/v2.0/ and https://packages.daos.io/v2.0/ for more information.
Branches:

release/2.0 is the release branch for the stable 2.0 release. Latest bug fix release is 2.0.2.
release/2.2 has been created earlier this week for the future 2.2 release. Latest test build is 2.1.101.
Master is the development branch for the future 2.4 release. Latest test build is 2.3.100.

Major recent changes on release/2.0:

Several documentation fixes
Several test fixes
Avoid holding CPU for too long in DTX
Fix intermittent failure that can occur when a node loses MS leadership and regains it

Major recent changes on release/2.2 and master (same for now):

Add UCX support to the CART and the control plane
Add UCX support to the RPM
Update dmg query command to show the list of excluded engines in a pool.
Add support to core_dump_filter to the server yaml file
Several aggregation/reclaim improvements
Add pool versioning and upgrade capability for interoperability with 2.0.
Add interoperability check to the control plane
Rewrite netdetect in the control plane
Report accurate mtime via libdfs/dfuse by using DAOS internal versioning scheme
Several coverity fixes
Remove CentOS 8 support and move build/testing to Rocky Linux 8.5
Add tensorflow-IO documentation
Use strict CPU binding for the engine
Improve response time of dmg pool list

What is coming:

2.2.0 testing and code freeze
2.4 features to land to master

R&D

Major features under development:

Checksum scrubber (targeted for 2.4)
Catastrophic recovery / distributed fsck
API to collect libdaos metrics to be integrated with Darshan (targeted for 2.4)
Multi-user dfuse (targeted for 2.4)
More aggressive caching in dfuse for AI APPs (targeted for 2.4)
TensorFlow-IO plugin for DFS

LDMS plugin to export DAOS metrics

Pathfinding:

DAOS Pipeline API for active storage

Work in progress to support pipeline API in the engine
Use cases under development:

MariaDB DAOS engine with predicate pushdown to the DAOS storage nodes https://github.com/daos-stack/mariadb
Accelerated find

Leveraging the Intel Data Streaming Accelerator (DSA) to accelerate DAOS

Prototype leveraging DSA for VOS aggregation delivered

S3 support via a DAOS backend to Rados Gateway (RGW)
Block interface over DAOS using SPDK DAOS bdev
GPU data path optimizations

News

Ivan Poddubnyy joins the Intel DAOS team as the new Community Support and Enablement manager.
Sodacode 2022 hackathon is still underway. The list of DAOS activities offered in this hackathon has been created in jira: https://daosio.atlassian.net/browse/DAOS-10010?jql=labels%20%3D%20%22SODACODE2022%22%20and%20statusCategory%20!%3D%20Done

See https://events.linuxfoundation.org/sodacode/ for more information.

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number: 302 456 199 R.C.S. NANTERRE
Capital: 5 208 026.16 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Re: dfs_lookup behavior for non-existent files?

Tuffli, Chuck

#1574

Mohamad

Thank you for the sanity check regarding dfs_lookup. After a little sleuthing, the application (evidently) was modifying the effective UID/GID around the time of that lookup. And it was this *ID change that made networking fail. With those calls changed, DFS is now doing what I expected/thought/hoped 🙂

--chuck

Show quoted text

From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Chaarawi, Mohamad <mohamad.chaarawi@...>
Sent: Tuesday, April 5, 2022 5:21 AM
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: Re: [daos] dfs_lookup behavior for non-existent files?

Hi Chuck,

Neither dfs_lookup nor dfs_stat do set the st_ino in the stat buf.

The reason being is that files are uniquely identified by the daos object ID which is 128 bits (64 hi, 64 lo).

You can retrieve that using dfs_obj2id():

https://github.com/daos-stack/daos/blob/master/src/include/daos_fs.h#L316

now for the other error, that seems weird. The errors are coming from the network layer. At that point, are there any servers that are down or were killed (specifically the engine with rank 1)? This would explain the errors.

When I try this myself, I get ENOENT for lookup on “//.Trash” as expected.

Thanks,

Mohamad

From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Tuffli, Chuck <chuck.tuffli@...>
Date: Tuesday, April 5, 2022 at 12:58 AM
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: [daos] dfs_lookup behavior for non-existent files?

I'm porting an existing application to use DFS (DAOS v2.0.2) instead of POSIX and need help understanding the error messages printed to the console.

The code is using dfs_lookup() to retrieve the struct stat of a file. Note the implementation cannot use dfs_stat() as it requires valid values for fields such as st_ino that dfs_stat() does not provide. The code in question is:

int

d_lstat(const char * restrict path, struct stat * restrict sb)

{

int rc;

dfs_obj_t *obj = NULL;

rc = dfs_lookup(dfs, path, O_RDONLY, &obj, NULL, sb);

...

If the file path exists (e.g. "/"), this works. But if the path, doesn't exist (e.g. "//.Trash"), the call to dfs_lookup() does not return. Instead, the console endlessly prints messages like:

04/04-16:28:50.90 xxxxx DAOS[1178648/1178648/0] external ERR # [6937851.329315] mercury->msg: [error] /builddir/build/BUILD/mercury-2.1.0rc4/src/na/na_ofi.c:2972

# na_ofi_msg_send(): fi_tsend() failed, rc: -13 (Permission denied)

04/04-16:28:50.90 xxxxx DAOS[1178648/1178648/0] external ERR # [6937851.329374] mercury->hg: [error] /builddir/build/BUILD/mercury-2.1.0rc4/src/mercury_core.c:2727

# hg_core_forward_na(): Could not post send for input buffer (NA_ACCESS)

04/04-16:28:50.90 xxxxx DAOS[1178648/1178648/0] hg ERR src/cart/crt_hg.c:1104 crt_hg_req_send_cb(0x1d0cd40) [opc=0x4070001 (DAOS) rpcid=0x63f8133700000008 rank:tag=1:2] RPC failed; rc: DER_HG(-1020): 'Transport layer mercury error'

04/04-16:28:50.90 xxxxx DAOS[1178648/1178648/0] object ERR src/object/cli_shard.c:889 dc_rw_cb() RPC 1 failed, DER_HG(-1020): 'Transport layer mercury error'

Am I mis-using dfs_lookup() or using it incorrectly?

--chuck

Re: dfs_lookup behavior for non-existent files?

Chaarawi, Mohamad

#1573

Hi Chuck,

Neither dfs_lookup nor dfs_stat do set the st_ino in the stat buf.

The reason being is that files are uniquely identified by the daos object ID which is 128 bits (64 hi, 64 lo).

You can retrieve that using dfs_obj2id():

https://github.com/daos-stack/daos/blob/master/src/include/daos_fs.h#L316

now for the other error, that seems weird. The errors are coming from the network layer. At that point, are there any servers that are down or were killed (specifically the engine with rank 1)? This would explain the errors.

When I try this myself, I get ENOENT for lookup on “//.Trash” as expected.

Thanks,

Mohamad

From: daos@daos.groups.io <daos@daos.groups.io> on behalf of Tuffli, Chuck <chuck.tuffli@...>
Date: Tuesday, April 5, 2022 at 12:58 AM
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: [daos] dfs_lookup behavior for non-existent files?

I'm porting an existing application to use DFS (DAOS v2.0.2) instead of POSIX and need help understanding the error messages printed to the console.

The code is using dfs_lookup() to retrieve the struct stat of a file. Note the implementation cannot use dfs_stat() as it requires valid values for fields such as st_ino that dfs_stat() does not provide. The code in question is:

int

d_lstat(const char * restrict path, struct stat * restrict sb)

{

int rc;

dfs_obj_t *obj = NULL;

rc = dfs_lookup(dfs, path, O_RDONLY, &obj, NULL, sb);

...

If the file path exists (e.g. "/"), this works. But if the path, doesn't exist (e.g. "//.Trash"), the call to dfs_lookup() does not return. Instead, the console endlessly prints messages like:

04/04-16:28:50.90 xxxxx DAOS[1178648/1178648/0] external ERR # [6937851.329315] mercury->msg: [error] /builddir/build/BUILD/mercury-2.1.0rc4/src/na/na_ofi.c:2972

# na_ofi_msg_send(): fi_tsend() failed, rc: -13 (Permission denied)

04/04-16:28:50.90 xxxxx DAOS[1178648/1178648/0] external ERR # [6937851.329374] mercury->hg: [error] /builddir/build/BUILD/mercury-2.1.0rc4/src/mercury_core.c:2727

# hg_core_forward_na(): Could not post send for input buffer (NA_ACCESS)

04/04-16:28:50.90 xxxxx DAOS[1178648/1178648/0] hg ERR src/cart/crt_hg.c:1104 crt_hg_req_send_cb(0x1d0cd40) [opc=0x4070001 (DAOS) rpcid=0x63f8133700000008 rank:tag=1:2] RPC failed; rc: DER_HG(-1020): 'Transport layer mercury error'

04/04-16:28:50.90 xxxxx DAOS[1178648/1178648/0] object ERR src/object/cli_shard.c:889 dc_rw_cb() RPC 1 failed, DER_HG(-1020): 'Transport layer mercury error'

Am I mis-using dfs_lookup() or using it incorrectly?

--chuck

Re: What will happen to DAOS if all SCM space is consumed?

Lombardi, Johann

#1572

Hi there,

Applications will get a DER_NOSPACE error. We currently don’t support serialization of metadata to SSDs.

Cheers,

Johann

From: <daos@daos.groups.io> on behalf of "bob@..." <bob@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Friday 1 April 2022 at 09:12
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: [daos] What will happen to DAOS if all SCM space is consumed?

Hi all

What will happen to DAOS when the meta data and small I/Os have consumed all the SCM capacity meanwhile the SSDs still have enough space to hold data ? what is the strategy ? Does it move some 4K values or meta data (keys) to the SSDs and then reclaim their space for incoming data , or just simply stop accepting I/O request?

Regards

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number: 302 456 199 R.C.S. NANTERRE
Capital: 5 208 026.16 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Re: the fault domain setting of the daos container

Lombardi, Johann

#1571

Hi there,

There is a gap in the support of rf_lvl which is not used by the placement algorithm yet. Please see https://daosio.atlassian.net/browse/DAOS-10215 to track progress on this.

Cheers,

Johann

From: <daos@daos.groups.io> on behalf of "dagouxiong2015@..." <dagouxiong2015@...>
Reply-To: "daos@daos.groups.io" <daos@daos.groups.io>
Date: Thursday 24 March 2022 at 05:02
To: "daos@daos.groups.io" <daos@daos.groups.io>
Subject: [daos] the fault domain setting of the daos container

Hello everyone:

Recently, I am studying the fault domain setting of the daos container, hoping to deploy multiple engines on a single physical node,

and the data of the object is distributed on different physical nodes , prevent the failure of a single physical node to cause data loss.

But when I configure

[root@server-1 ~]# daos cont create pool --label cont12 --type POSIX --properties rf:1 --properties rf_lvl:0

ERROR: daos: "rf_lvl" is not a settable property (valid: cksum,cksum_size,compression,dedup,dedup_threshold,ec_cell,encryption,label,rf,srv_cksum,status)

Code analysis can only support one fault domain (by rank) layout.

/**

* Level of fault-domain to use for object allocation

* rank is hardcoded to 1, [2-254] are defined by the admin

*/

enum {

DAOS_PROP_CO_REDUN_MIN = 1,

DAOS_PROP_CO_REDUN_RANK = 1, /** hard-coded */

DAOS_PROP_CO_REDUN_MAX = 254,

};

In the current situation, if you want data redundancy on different physical nodes, do you have any good suggestions?

Does daos plan to support configurable in fault domain in the future?

Best regards!

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number: 302 456 199 R.C.S. NANTERRE
Capital: 5 208 026.16 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

dfs_lookup behavior for non-existent files?

Tuffli, Chuck

#1570

I'm porting an existing application to use DFS (DAOS v2.0.2) instead of POSIX and need help understanding the error messages printed to the console.

The code is using dfs_lookup() to retrieve the struct stat of a file. Note the implementation cannot use dfs_stat() as it requires valid values for fields such as st_ino that dfs_stat() does not provide. The code in question is:

int

d_lstat(const char * restrict path, struct stat * restrict sb)

{

int rc;

dfs_obj_t *obj = NULL;

rc = dfs_lookup(dfs, path, O_RDONLY, &obj, NULL, sb);

...

If the file path exists (e.g. "/"), this works. But if the path, doesn't exist (e.g. "//.Trash"), the call to dfs_lookup() does not return. Instead, the console endlessly prints messages like:

04/04-16:28:50.90 xxxxx DAOS[1178648/1178648/0] external ERR # [6937851.329315] mercury->msg: [error] /builddir/build/BUILD/mercury-2.1.0rc4/src/na/na_ofi.c:2972

# na_ofi_msg_send(): fi_tsend() failed, rc: -13 (Permission denied)

04/04-16:28:50.90 xxxxx DAOS[1178648/1178648/0] external ERR # [6937851.329374] mercury->hg: [error] /builddir/build/BUILD/mercury-2.1.0rc4/src/mercury_core.c:2727

# hg_core_forward_na(): Could not post send for input buffer (NA_ACCESS)

04/04-16:28:50.90 xxxxx DAOS[1178648/1178648/0] hg ERR src/cart/crt_hg.c:1104 crt_hg_req_send_cb(0x1d0cd40) [opc=0x4070001 (DAOS) rpcid=0x63f8133700000008 rank:tag=1:2] RPC failed; rc: DER_HG(-1020): 'Transport layer mercury error'

04/04-16:28:50.90 xxxxx DAOS[1178648/1178648/0] object ERR src/object/cli_shard.c:889 dc_rw_cb() RPC 1 failed, DER_HG(-1020): 'Transport layer mercury error'

Am I mis-using dfs_lookup() or using it incorrectly?

--chuck

What will happen to DAOS if all SCM space is consumed?

bob@...

#1569

Hi all

What will happen to DAOS when the meta data and small I/Os have consumed all the SCM capacity meanwhile the SSDs still have enough space to hold data ? what is the strategy ? Does it move some 4K values or meta data (keys) to the SSDs and then reclaim their space for incoming data , or just simply stop accepting I/O request?

Regards

the fault domain setting of the daos container

dagouxiong2015@...

#1568

Hello everyone:

Recently, I am studying the fault domain setting of the daos container, hoping to deploy multiple engines on a single physical node,
and the data of the object is distributed on different physical nodes , prevent the failure of a single physical node to cause data loss.
But when I configure
[root@server-1 ~]# daos cont create pool --label cont12 --type POSIX --properties rf:1  --properties rf_lvl:0
ERROR: daos: "rf_lvl" is not a settable property (valid: cksum,cksum_size,compression,dedup,dedup_threshold,ec_cell,encryption,label,rf,srv_cksum,status)

Code analysis can only support one fault domain (by rank) layout.
/**
 * Level of fault-domain to use for object allocation
 * rank is hardcoded to 1, [2-254] are defined by the admin
 */
enum {
        DAOS_PROP_CO_REDUN_MIN        = 1,
        DAOS_PROP_CO_REDUN_RANK        = 1, /** hard-coded */
        DAOS_PROP_CO_REDUN_MAX        = 254,
};

In the current situation, if you want data redundancy on different physical nodes, do you have any good suggestions?
Does daos plan to support configurable in fault domain in the future?

Best regards!

Announcement: DAOS 2.0.2 is generally available

Prantis, Kelsey

#1567

All,

We are pleased to announce that DAOS 2.0.2 release is now generally available. Notable changes in this maintenance release includes following updates on top of DAOS 2.0.1:

mercury has been updated from 2.1.0~rc4-3 to 2.1.0~rc4-5. This fixes multiple issues with dmg pool destroy, including DAOS-9725 and DAOS-9006.
dfuse readahead caching has been disabled when write-through caching is enabled DAOS-9738.
An issue with EC aggregation has been fixed where it was running too frequently and consuming CPU cycles even when EC is not used DAOS-9926.
Go dependency has been updated to >= 1.17 DAOS-9908.
Hadoop dependency has been updated to 3.3.2 DAOS-10068.

There are a number of resources available for the release:

RPM Repositories: https://packages.daos.io/v2.0/
Release Notes: https://docs.daos.io/v2.0/release/release_notes/
Admin Guide: https://docs.daos.io/v2.0/admin/hardware/
User Guide: https://docs.daos.io/v2.0/user/workflow/
Architecture Overview: https://docs.daos.io/v2.0/overview/architecture/
Source Code: https://github.com/daos-stack/daos/releases/

As always, feel free to use this mailing list for any issues you may find with the release, or our JIRA bug tracking system, available at https://daosio.atlassian.net/jira, or on our Slack channel, available at https://daos-stack.slack.com.

Regards,

Kelsey Prantis

Senior Software Engineering Manager

Super Compute Storage Architecture and Development Division

Intel

Re: High latency in metada write

shadow_vector@...

#1566

Hi Liang:

Is there any result of array write test ? Is there something wrong with my test?

Best Regards!

Re: Jenkins test

Murrell, Brian

#1565

On Wed, 2022-03-02 at 17:29 -0800, dongfeier wrote:
Scripts not permitted to use staticMethod
org.codehaus.groovy.runtime.DefaultGroovyMethods getAt
java.lang.Object java.lang.String.Ultimately this means that some code in an untrusted shared library is
trying to access a non-whitelisted groovy function.

Administrators can decide whether to approve or reject this
signature. ( http://172.20.18.132:8080/scriptApproval )You *could* do the above with the security implications it involves,
but the correct solution is to use whitelisted methods.

Error when executing unsuccessful post condition:
org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException:
Scripts not permitted to use staticMethod
org.codehaus.groovy.runtime.DefaultGroovyMethods getAt
java.lang.Object java.lang.StringThis is the method that is not whitelisted.

        at
org.jenkinsci.plugins.scriptsecurity.sandbox.whitelists.StaticWhiteli
st.rejectStaticMethod(StaticWhitelist.java:279)
        at
org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxIntercepto
r.onGetArray(SandboxInterceptor.java:476)
        at
org.kohsuke.groovy.sandbox.impl.Checker$11.call(Checker.java:484)
        at
org.kohsuke.groovy.sandbox.impl.Checker.checkedGetArray(Checker.java:
489)
        at
com.cloudbees.groovy.cps.sandbox.SandboxInvoker.getArray(SandboxInvok
er.java:45)
        at
com.cloudbees.groovy.cps.impl.ArrayAccessBlock.rawGet(ArrayAccessBloc
k.java:21)
        at notifyBrokenBranch.call(notifyBrokenBranch.groovy:37)And this is where it's being called from.  It's here:

https://github.com/daos-stack/pipeline-lib/blob/03a6dd8f16808094e2ba2971e839707cd690c0a5/vars/notifyBrokenBranch.groovy#L37

It's the use of env[] that is the problem.  One solution here is to
move that function to the trusted library at:

https://github.com/daos-stack/trusted-pipeline-lib

But it seems a more correct solution is to replace the env[NAME]
accesses to env."NAME" such as this (completely untested) PR does:

https://github.com/daos-stack/pipeline-lib/pull/291

Cheers,
b.

Re: Jenkins test

Pittman, Ashley M

#1564

I suspect that this is because you haven’t configured Jenkins to use a build user and it’s building your code as root. Our dockerfiles use the uid of the caller to own the files so that Jenkins can copy files in/out of the container, and we didn’t think about the case that docker would be run as root. In theory we could probably workaround this in the dockerfiles but I recommend you look at your Jenkins config first.

Ashley.

From: daos@daos.groups.io <daos@daos.groups.io> on behalf of dongfeier <15735154041@...>
Date: Thursday, 3 March 2022 at 03:16
To: daos@daos.groups.io <daos@daos.groups.io>
Subject: Re: [daos] Jenkins test

[Edited Message Follows]

Hello,

useradd: UID 0 is not unique

The command '/bin/sh -c useradd --no-log-init --uid $UID --user-group --create-home --shell /bin/bash             --home /home/daos daos_server' returned a non-zero code: 4

Jenkins not configured to notify users of failed builds.

Scripts not permitted to use staticMethod org.codehaus.groovy.runtime.DefaultGroovyMethods getAt java.lang.Object java.lang.String. Administrators can decide whether to approve or reject this signature.

Error when executing unsuccessful post condition:

org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException: Scripts not permitted to use staticMethod org.codehaus.groovy.runtime.DefaultGroovyMethods getAt java.lang.Object java.lang.String

  at org.jenkinsci.plugins.scriptsecurity.sandbox.whitelists.StaticWhitelist.rejectStaticMethod(StaticWhitelist.java:279)

  at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onGetArray(SandboxInterceptor.java:476)

  at org.kohsuke.groovy.sandbox.impl.Checker$11.call(Checker.java:484)

  at org.kohsuke.groovy.sandbox.impl.Checker.checkedGetArray(Checker.java:489)

  at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.getArray(SandboxInvoker.java:45)

  at com.cloudbees.groovy.cps.impl.ArrayAccessBlock.rawGet(ArrayAccessBlock.java:21)

  at notifyBrokenBranch.call(notifyBrokenBranch.groovy:37)

  at WorkflowScript.run(WorkflowScript:1049)

  at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.delegateAndExecute(ModelInterpreter.groovy:137)

  at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.runPostConditions(ModelInterpreter.groovy:761)

  at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.catchRequiredContextForNode(ModelInterpreter.groovy:395)

  at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.catchRequiredContextForNode(ModelInterpreter.groovy:393)

  at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.runPostConditions(ModelInterpreter.groovy:760)

  at com.cloudbees.groovy.cps.CpsDefaultGroovyMethods.each(CpsDefaultGroovyMethods:2030)

  at com.cloudbees.groovy.cps.CpsDefaultGroovyMethods.each(CpsDefaultGroovyMethods:2015)

  at com.cloudbees.groovy.cps.CpsDefaultGroovyMethods.each(CpsDefaultGroovyMethods:2056)

  at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.runPostConditions(ModelInterpreter.groovy:750)

  at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.runPostConditions(ModelInterpreter.groovy)

  at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.executePostBuild(ModelInterpreter.groovy:728)

  at ___cps.transform___(Native Method)

  at com.cloudbees.groovy.cps.impl.PropertyishBlock$ContinuationImpl.get(PropertyishBlock.java:74)

  at com.cloudbees.groovy.cps.LValueBlock$GetAdapter.receive(LValueBlock.java:30)

  at com.cloudbees.groovy.cps.impl.PropertyishBlock$ContinuationImpl.fixName(PropertyishBlock.java:66)

  at sun.reflect.GeneratedMethodAccessor430.invoke(Unknown Source)

  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

  at java.lang.reflect.Method.invoke(Method.java:498)

  at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)

  at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)

  at com.cloudbees.groovy.cps.Next.step(Next.java:83)

  at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)

  at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)

  at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)

  at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)

  at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)

  at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)

  at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)

  at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:185)

  at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:402)

  at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:96)

  at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:314)

  at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:278)

  at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67)

  at java.util.concurrent.FutureTask.run(FutureTask.java:266)

  at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139)

  at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)

  at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)

  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

  at java.util.concurrent.FutureTask.run(FutureTask.java:266)

  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

  at java.lang.Thread.run(Thread.java:748)

I have found the cause of the problem because the uid is repeated when adding users. Thank you very much

Re: Storage Usage

d.korekovcev@...

#1563

If I manually remove hosts from conf

Re: Storage Usage

Nabarro, Tom

#1562

This should probably return a range in this case i.e. n04p00[1-9], I will create a ticket.

Regards,

Tom

Show quoted text

From: daos@daos.groups.io <daos@daos.groups.io> On Behalf Of d.korekovcev@...
Sent: Thursday, March 3, 2022 12:13 PM
To: daos@daos.groups.io
Subject: [daos] Storage Usage

Hi all!

dmg storage query usage return error argument list too long

Storage Usage

d.korekovcev@...

#1561

Hi all!

dmg storage query usage return error argument list too long

Re: Jenkins test

dongfeier

#1560
Edited

Hello,

useradd: UID 0 is not unique
The command '/bin/sh -c useradd --no-log-init --uid $UID --user-group --create-home --shell /bin/bash             --home /home/daos daos_server' returned a non-zero code: 4

Jenkins not configured to notify users of failed builds.
Scripts not permitted to use staticMethod org.codehaus.groovy.runtime.DefaultGroovyMethods getAt java.lang.Object java.lang.String. Administrators can decide whether to approve or reject this signature.
Error when executing unsuccessful post condition:
org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException: Scripts not permitted to use staticMethod org.codehaus.groovy.runtime.DefaultGroovyMethods getAt java.lang.Object java.lang.String
	at org.jenkinsci.plugins.scriptsecurity.sandbox.whitelists.StaticWhitelist.rejectStaticMethod(StaticWhitelist.java:279)
	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onGetArray(SandboxInterceptor.java:476)
	at org.kohsuke.groovy.sandbox.impl.Checker$11.call(Checker.java:484)
	at org.kohsuke.groovy.sandbox.impl.Checker.checkedGetArray(Checker.java:489)
	at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.getArray(SandboxInvoker.java:45)
	at com.cloudbees.groovy.cps.impl.ArrayAccessBlock.rawGet(ArrayAccessBlock.java:21)
	at notifyBrokenBranch.call(notifyBrokenBranch.groovy:37)
	at WorkflowScript.run(WorkflowScript:1049)
	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.delegateAndExecute(ModelInterpreter.groovy:137)
	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.runPostConditions(ModelInterpreter.groovy:761)
	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.catchRequiredContextForNode(ModelInterpreter.groovy:395)
	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.catchRequiredContextForNode(ModelInterpreter.groovy:393)
	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.runPostConditions(ModelInterpreter.groovy:760)
	at com.cloudbees.groovy.cps.CpsDefaultGroovyMethods.each(CpsDefaultGroovyMethods:2030)
	at com.cloudbees.groovy.cps.CpsDefaultGroovyMethods.each(CpsDefaultGroovyMethods:2015)
	at com.cloudbees.groovy.cps.CpsDefaultGroovyMethods.each(CpsDefaultGroovyMethods:2056)
	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.runPostConditions(ModelInterpreter.groovy:750)
	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.runPostConditions(ModelInterpreter.groovy)
	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.executePostBuild(ModelInterpreter.groovy:728)
	at ___cps.transform___(Native Method)
	at com.cloudbees.groovy.cps.impl.PropertyishBlock$ContinuationImpl.get(PropertyishBlock.java:74)
	at com.cloudbees.groovy.cps.LValueBlock$GetAdapter.receive(LValueBlock.java:30)
	at com.cloudbees.groovy.cps.impl.PropertyishBlock$ContinuationImpl.fixName(PropertyishBlock.java:66)
	at sun.reflect.GeneratedMethodAccessor430.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
	at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
	at com.cloudbees.groovy.cps.Next.step(Next.java:83)
	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
	at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)
	at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)
	at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)
	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:185)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:402)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:96)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:314)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:278)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

I have found the cause of the problem because the uid is repeated when adding users. Thank you very much

DAOS Community Update / Mar'22

Lombardi, Johann

#1559

Hi there,

Please find below the DAOS community newsletter for March 2022.

Past Events (February)

Dell Technologies HPC Community Event (February 9^th)
DAOS: High Performance Storage for All HPC Workloads and Scales
Kelsey Prantis (Intel)
https://www.dellhpc.org/event_2-9-2022.html

Upcoming Events

2022 Energy HPC Conference (March 3^rd)
DAOS Use at Argonne
Kevin Harms (Argonne)
Johann Lombardi (Intel)
Mohamad Chaarawi (Intel)
https://energyhpc.rice.edu/program/

Release

Current stable release is 2.0.1 and 2.0.2 is under development.
Master is the development branch for the future 2.2 release.
Major recent 2.0 changes to be available in 2.0.2:

Build go code shipped in RPM with go >= 1.17
Fix a bug with key enumeration and multiple shards
Many documentation fixes (e.g. fix broken links)
Use larger ABT stack for NVMe poll ULT
Reject integer keys that are smaller than 8 bytes
Update the C-based raft implementation to pick a membership fix
Disable dfuse-level readahead when FUSE is configured in writeback mode and already drive readahead from the kernel.

Major recent master changes:

Fix a problem between mercury and libfabric causing some bulk transfers to be stuck indefinitely
Reduce CPU load caused by the engine when DAOS system is idle
Add support for owner/group to libdfs
Automatically calculate the required number of huge pages
Several VMD and SSD hotplug fixes
Document the need to disable shared received context for Intel E810-C cards
Rewrite of the netdetect module
Add UCX support to the manual (non RPM) build infrastructure

What is coming:

2.0.2 validation
2.2.0 feature freeze

R&D

Major features under development:

Checksum scrubbing
UCX support
LDMS plugin to export DAOS metrics (targeted for 2.2)
API to collect libdaos metrics to be integrated with Darshan (targeted for 2.4)
Multi-user dfuse (targeted for 2.4)
More aggressive caching in dfuse for AI APPs (targeted for 2.4)
TensorFlow-IO plugin for DFS

https://github.com/daos-stack/tensorflow-io-daos/blob/devel/docs/daos_tf_docs.md

Pathfinding:

MariaDB DAOS engine with predicate pushdown to the DAOS storage nodes

Prototyped DAOS MariaDB engine available here: https://github.com/daos-stack/mariadb
PR for pipeline API (#6238)
Work in progress to support pipeline API in the engine

Leveraging the Intel Data Streaming Accelerator (DSA) to accelerate DAOS

Prototype leveraging DSA for VOS aggregation delivered

S3 support via a DAOS backend to Rados Gateway (RGW)
GPU data path optimizations

News

Sodacode 2022 hackathon started today with participation of the DAOS team.
The list of DAOS activities offered in this hackathon has been created in jira: https://daosio.atlassian.net/browse/DAOS-10010?jql=labels%20%3D%20%22SODACODE2022%22%20and%20statusCategory%20!%3D%20Done

See https://events.linuxfoundation.org/sodacode/ for more information.

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris,
92196 Meudon Cedex, France
Registration Number: 302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.