<div dir="ltr"><div><div><div>Well we got it into the down state using mmsdrrestore -p to recover stuff into /var/mmfs/gen to cl004.<br><br></div>Anyhow we ended up unknown for cl004 when it powered off. Short of removing node, unknown is the state you get.<br><br></div>Unknown seems stable for a hopefully short outage of cl004.<br><br><br></div>Thanks<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Feb 2, 2017 at 4:28 PM, Olaf Weiser <span dir="ltr"><<a href="mailto:olaf.weiser@de.ibm.com" target="_blank">olaf.weiser@de.ibm.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><font size="2" face="sans-serif">many ways lead to Rome .. and I agree ..
mmexpelnode is a nice command .. </font><br><font size="2" face="sans-serif">another approach... </font><br><font size="2" face="sans-serif">power it off .. (not reachable by ping)
.. mmdelnode ... power on/boot ... mmaddnode .. </font><br><br><br><br><font size="1" face="sans-serif" color="#5f5f5f">From:
</font><font size="1" face="sans-serif">Aaron Knister <<a href="mailto:aaron.s.knister@nasa.gov" target="_blank">aaron.s.knister@nasa.gov</a>></font><br><font size="1" face="sans-serif" color="#5f5f5f">To:
</font><font size="1" face="sans-serif"><<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@<wbr>spectrumscale.org</a>></font><br><font size="1" face="sans-serif" color="#5f5f5f">Date:
</font><font size="1" face="sans-serif">02/02/2017 08:37 PM</font><br><font size="1" face="sans-serif" color="#5f5f5f">Subject:
</font><font size="1" face="sans-serif">Re: [gpfsug-discuss]
proper gpfs shutdown when node disappears</font><br><font size="1" face="sans-serif" color="#5f5f5f">Sent by:
</font><font size="1" face="sans-serif"><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">gpfsug-discuss-bounces@<wbr>spectrumscale.org</a></font><br><hr noshade><br><br><div class="HOEnZb"><div class="h5"><br><tt><font size="2">You could forcibly expel the node (one of my favorite
GPFS commands):<br><br>mmexpelnode -N $nodename<br><br>and then power it off after the expulsion is complete and then do<br><br>mmepelenode -r -N $nodename<br><br>which will allow it to join the cluster next time you try and start up
<br>GPFS on it. You'll still likely have to go through recovery but you'll
<br>skip the part where GPFS wonders where the node went prior to it <br>expelling it.<br><br>-Aaron<br><br>On 2/2/17 2:28 PM, <a href="mailto:valdis.kletnieks@vt.edu" target="_blank">valdis.kletnieks@vt.edu</a> wrote:<br>> On Thu, 02 Feb 2017 18:28:22 +0100, "Olaf Weiser" said:<br>><br>>> but the /var/mmfs DIR is obviously damaged/empty .. what ever..
that's why you<br>>> see a message like this..<br>>> have you reinstalled that node / any backup/restore thing ?<br>><br>> The internal RAID controller died a horrid death and basically took<br>> all the OS partitions with it. So the node was just sort of
limping along,<br>> where the mmfsd process was still coping because it wasn't doing any<br>> I/O to the OS partitions - but 'ssh bad-node mmshutdown' wouldn't
work<br>> because that requires accessing stuff in /var.<br>><br>> At that point, it starts getting tempting to just use ipmitool from<br>> another node to power the comatose one down - but that often causes<br>> a cascade of other issues while things are stuck waiting for timeouts.<br>><br>><br>> ______________________________<wbr>_________________<br>> gpfsug-discuss mailing list<br>> gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank">spectrumscale.org</a><br>> </font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank"><tt><font size="2">http://gpfsug.org/mailman/<wbr>listinfo/gpfsug-discuss</font></tt></a><tt><font size="2"><br>><br><br>-- <br>Aaron Knister<br>NASA Center for Climate Simulation (Code 606.2)<br>Goddard Space Flight Center<br><a href="tel:(301)%20286-2776" value="+13012862776" target="_blank">(301) 286-2776</a><br>______________________________<wbr>_________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank">spectrumscale.org</a><br></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank"><tt><font size="2">http://gpfsug.org/mailman/<wbr>listinfo/gpfsug-discuss</font></tt></a><tt><font size="2"><br><br></font></tt><br><br><br>
</div></div><br>______________________________<wbr>_________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/<wbr>listinfo/gpfsug-discuss</a><br>
<br></blockquote></div><br></div>