<html><body><p><font size="2">Hi Aaron,</font><br><br><font size="2">The header dump shows all zeroes were received for the header. So no valid magic, version, originator, etc. The "512 more bytes" would have been the meat after the header. Very unexpected hence the shutdown.</font><br><br><font size="2">Logs around that event involving the machines noted in that trace would be required to evaluate further. This is not common.</font><br><br><font size="2">Regards, The Spectrum Scale (GPFS) team<br><br>------------------------------------------------------------------------------------------------------------------<br>If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at <a href="https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479">https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479</a>. <br><br>If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. <br><br>The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team.</font><br><br><img width="16" height="16" src="cid:1__=0ABB09D1DFF12F2E8f9e8a93df938690918c0AB@" border="0" alt="Inactive hide details for Aaron Knister ---11/07/2018 06:38:36 PM---We're experiencing client nodes falling out of the cluster "><font size="2" color="#424282">Aaron Knister ---11/07/2018 06:38:36 PM---We're experiencing client nodes falling out of the cluster with errors that look like this:</font><br><br><font size="2" color="#5F5F5F">From: </font><font size="2">Aaron Knister <aaron.s.knister@nasa.gov></font><br><font size="2" color="#5F5F5F">To: </font><font size="2">gpfsug main discussion list <gpfsug-discuss@spectrumscale.org></font><br><font size="2" color="#5F5F5F">Date: </font><font size="2">11/07/2018 06:38 PM</font><br><font size="2" color="#5F5F5F">Subject: </font><font size="2">[gpfsug-discuss] Unexpected data in message/Bad message</font><br><font size="2" color="#5F5F5F">Sent by: </font><font size="2">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr width="100%" size="2" align="left" noshade style="color:#8091A5; "><br><br><br><tt><font size="2">We're experiencing client nodes falling out of the cluster with errors <br>that look like this:<br><br>Tue Nov 6 15:10:34.939 2018: [E] Unexpected data in message. Header <br>dump: 00000000 0000 0000 00000047 00000000 00 00 0000 00000000 00000000 <br>0000 0000<br>Tue Nov 6 15:10:34.942 2018: [E] [0/0] 512 more bytes were available:<br>Tue Nov 6 15:10:34.965 2018: [N] Close connection to 10.100.X.X <br>nsdserver1 <c0n71> (Unexpected error 120)<br>Tue Nov 6 15:10:34.966 2018: [E] Network error on 10.100.X.X nsdserver1 <br><c0n71>, Check connectivity<br>Tue Nov 6 15:10:36.726 2018: [N] Restarting mmsdrserv<br>Tue Nov 6 15:10:38.850 2018: [E] Bad message<br>Tue Nov 6 15:10:38.851 2018: [X] The mmfs daemon is shutting down <br>abnormally.<br>Tue Nov 6 15:10:38.852 2018: [N] mmfsd is shutting down.<br>Tue Nov 6 15:10:38.853 2018: [N] Reason for shutdown: LOGSHUTDOWN called<br><br>The cluster is running various PTF Levels of 4.1.1.<br><br>Has anyone seen this before? I'm struggling to understand what it means <br>from a technical point of view. Was GPFS expecting a larger message than <br>it received? Did it receive all of the bytes it expected and some of it <br>was corrupt? It says "512 more bytes were available" but then doesn't <br>show any additional bytes.<br><br>Thanks!<br><br>-Aaron<br><br>-- <br>Aaron Knister<br>NASA Center for Climate Simulation (Code 606.2)<br>Goddard Space Flight Center<br>(301) 286-2776<br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><tt><font size="2"><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></font></tt><tt><font size="2"><br><br></font></tt><br><br><BR>
</body></html>