<font size=2 face="sans-serif">Simon,</font><br><br><font size=2 face="sans-serif">Take a look at </font><a href="http://files.gpfsug.org/presentations/2018/USA/Scale_Network_Flow-0.8.pdf"><font size=2 color=blue face="sans-serif">http://files.gpfsug.org/presentations/2018/USA/Scale_Network_Flow-0.8.pdf</font></a><font size=2 face="sans-serif"> slide 13.</font><br><br><font size=2 face="sans-serif"><br>Regards,<br><br>Tomer Perry<br>Scalable I/O Development (Spectrum Scale)<br>email: tomp@il.ibm.com<br>1 Azrieli Center, Tel Aviv 67021, Israel<br>Global Tel: +1 720 3422758<br>Israel Tel: +972 3 9188625<br>Mobile: +972 52 2554625<br></font><br><br><br><br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">Simon Thompson <S.J.Thompson@bham.ac.uk></font><br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">"gpfsug-discuss@spectrumscale.org"
<gpfsug-discuss@spectrumscale.org></font><br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">17/01/2019 13:35</font><br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">[gpfsug-discuss]
Node expels</font><br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr noshade><br><br><br><font size=2 face="Calibri">We’ve recently been seeing quite a few
node expels with messages of the form:</font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri">2019-01-17_11:19:30.882+0000: [W] The TCP
connection to IP address 10.20.0.58 proto-pg-pf01.bear.cluster <c0n236>
(socket 153) state is unexpected: state=1 ca_state=4 snd_cwnd=1 snd_ssthresh=5
unacked=5 probes=0 backoff=7 retransmits=7 rto=26496000 rcv_ssthresh=102828
rtt=6729 rttvar=12066 sacked=0 retrans=1 reordering=3 lost=5</font><br><font size=2 face="Calibri">2019-01-17_11:19:30.882+0000: [I] tscCheckTcpConn:
Sending debug data collection request to node 10.20.0.58 proto-pg-pf01.bear.cluster</font><br><font size=2 face="Calibri">2019-01-17_11:19:30.882+0000: Sending request
to collect TCP debug data to proto-pg-pf01.bear.cluster localNode</font><br><font size=2 face="Calibri">2019-01-17_11:19:30.882+0000: [I] Calling
user exit script gpfsSendRequestToNodes: event sendRequestToNodes, Async
command /usr/lpp/mmfs/bin/mmcommon.</font><br><font size=2 face="Calibri">2019-01-17_11:24:52.611+0000: [E] Timed
out in 300 seconds waiting for a commMsgCheckMessages reply from node 10.20.0.58
proto-pg-pf01.bear.cluster. Sending expel message.</font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri">On the client node, we see messages of
the form:</font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri">2019-01-17_11:19:31.101+0000: [N] sdrServ:
Received Tcp data collection request from 10.10.0.33</font><br><font size=2 face="Calibri">2019-01-17_11:19:31.102+0000: [N] GPFS
will attempt to collect Tcp debug data on this node.</font><br><font size=2 face="Calibri">2019-01-17_11:24:52.838+0000: [N] sdrServ:
Received expel data collection request from 10.10.0.33</font><br><font size=2 face="Calibri">2019-01-17_11:24:52.838+0000: [N] GPFS
will attempt to collect debug data on this node.</font><br><font size=2 face="Calibri">2019-01-17_11:25:02.741+0000: [N] This
node will be expelled from cluster rds.gpfs.servers due to expel msg from
10.10.12.41 (b</font><br><font size=2 face="Calibri">ber-les-nsd01-data.bb2.cluster in rds.gpfs.server</font><br><font size=2 face="Calibri">2019-01-17_11:25:03.160+0000: [N] sdrServ:
Received expel data collection request from 10.20.0.56</font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri">They always appear to be to a specific
type of hardware with the same Ethernet controller, though the nodes are
split across three data centres and we aren’t seeing link congestion on
the links between them.</font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri">On the node I listed above, it’s not actually
doing anything either as the software on it is still being installed (i.e.
it’s not doing GPFS or any other IO other than a couple of home directories).</font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri">Any suggestions on what “(socket 153)
state is unexpected” means?</font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri">Thanks</font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri">Simon</font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri"> </font><tt><font size=2>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br></font></tt><br><br><BR>