<font size=3 face="sans-serif">On recent releases you can accomplish the
same with the command, "mmlsnode -N waiters -L".</font><br><br><font size=3 face="sans-serif">Fred<br>__________________________________________________<br>Fred Stock | IBM Pittsburgh Lab | 720-430-8821<br>stockf@us.ibm.com</font><br><br><br><br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">valdis.kletnieks@vt.edu</font><br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">gpfsug main discussion
list <gpfsug-discuss@spectrumscale.org></font><br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">06/06/2017 12:46 PM</font><br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">Re: [gpfsug-discuss]
gpfs waiters debugging</font><br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr noshade><br><br><br><tt><font size=2>On Tue, 06 Jun 2017 15:06:57 +0200, Stijn De Weirdt
said:<br>> oh sure, i meant waiters that last > 300 seconds or so (something
that<br>> could trigger deadlock). obviously we're not interested in debugging
the<br>> short ones, it's not that gpfs doesn't work or anything ;)<br><br>At least at one time, a lot of the mm(whatever) administrative commands<br>would leave one dangling waiter for the duration of the command - which<br>could be a while if the command was mmdeldisk or mmrestripefs. I admit<br>not having specifically checked for gpfs 4.2, but it was true for 3.2 through<br>4.1....<br><br>And my addition to the collective debugging knowledge: A bash one-liner
to<br>dump all the waiters across a cluster, sorted by wait time. Note
that<br>our clusters tend to be 5-8 servers, this may be painful for those of you<br>who have 400+ node clusters. :)<br><br>##!/bin/bash<br>for i in ` mmlsnode | tail -1 | sed 's/^[ ]*[^ ]*[ ]*//'`; do ssh
$i /usr/lpp/mmfs/bin/mmfsadm dump waiters | sed "s/^/$i /"; done
| sort -n -r -k 3 -t' '<br><br>We've found it useful - if you have 1 waiter on one node that's 1278 seconds<br>old, and 3 other nodes have waiters that are 1275 seconds old, it's a good<br>chance the other 3 nodes waiters are waiting on the first node's waiter
to<br>resolve itself....<br>[attachment "attltepl.dat" deleted by Frederick Stock/Pittsburgh/IBM]
_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br></font></tt><br><br><BR>