Hi Alex,

On 3/26/21 4:49 PM, Oganezov, Alexander A wrote:

Could you enable OFI level logs by setting FI_LOG_LEVEL=warn on the client side and provide stdout/stderr output from runs that result in mercury erorrs/timeouts?

Thanks for that input, we'll try to reproduce the issue with those settings and provide them ASAP.

Also can you tell us what your ulimit -a reports on client/server nodes?

Sure.

client $ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 1541126
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) 370688000
open files (-n) 65536
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) unlimited
cpu time (seconds, -t) unlimited
max user processes (-u) 4096
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

On the server side pending signals is lower: 761096.

Regards, Steffen