<font size=2 face="sans-serif">Hi Kevin,</font><br><br><font size=2 face="sans-serif">The message below indicates that the
mmfsd code had a pending message on a socket, and, when it looked at the
low level socket statistics, GPFS found indications that the TCP connection
was in a 'bad state'. GPFS determines a connection to be a 'bad state'
if:</font><br><br><font size=2 face="sans-serif">1) the CA_STATE for the socket is not
in 0 (or open) state, which means the state must be disorder, recovery,
or loss. See this paper for more details on CA_STATE: </font><br><a href="https://wiki.aalto.fi/download/attachments/69901948/TCP-CongestionControlFinal.pdf"><font size=2 color=blue face="sans-serif">https://wiki.aalto.fi/download/attachments/69901948/TCP-CongestionControlFinal.pdf</font></a><br><br><font size=2 face="sans-serif">or</font><br><br><font size=2 face="sans-serif">2) the RTO is greater than 10 seconds
and there are unacknowledged messages pending on the socket (unacked >
0). </font><br><br><font size=2 face="sans-serif">In the example below we see that <b>rto=27008000,
</b>which means that the non-fast path TCP retransmission timeout is about
27 seconds, and that probably means the connection has experienced significant
packet loss. If there was no expel following this message, I would
suspect there was some transient packet loss that was recovered from.</font><br><br><font size=2 face="sans-serif">There are plenty of places in which
to find more details on RTO, but you might want to start with wikipedia
(</font><a href="https://en.wikipedia.org/wiki/Transmission_Control_Protocol"><font size=2 color=blue face="sans-serif">https://en.wikipedia.org/wiki/Transmission_Control_Protocol</font></a><font size=2 face="sans-serif">)
which states:</font><br><br><font size=3>In addition, senders employ a <i>retransmission timeout</i>(RTO) that is based on the estimated </font><a href="https://en.wikipedia.org/wiki/Round-trip_time"><font size=3 color=blue><u>round-trip
time</u></font></a><font size=3> (or RTT) between the sender and receiver,
as well as the variance in this round trip time. The behavior of this timer
is specified in </font><a href="https://tools.ietf.org/html/rfc6298"><font size=3 color=blue><u>RFC
6298</u></font></a><font size=3>. There are subtleties in the estimation
of RTT. For example, senders must be careful when calculating RTT samples
for retransmitted packets; typically they use </font><a href="https://en.wikipedia.org/wiki/Karn%27s_Algorithm"><font size=3 color=blue><u>Karn's
Algorithm</u></font></a><font size=3> or TCP timestamps (see </font><a href="https://tools.ietf.org/html/rfc1323"><font size=3 color=blue><u>RFC
1323</u></font></a><font size=3>). These individual RTT samples are then
averaged over time to create a Smoothed Round Trip Time (SRTT) using </font><a href="https://en.wikipedia.org/wiki/Van_Jacobson"><font size=3 color=blue><u>Jacobson</u></font></a><font size=3>'s
algorithm. This SRTT value is what is finally used as the round-trip time
estimate. </font><br><font size=2 face="sans-serif">[. . .]</font><br><font size=3>Reliability is achieved by the sender detecting lost data
and retransmitting it. TCP uses two primary techniques to identify loss.
Retransmission timeout (abbreviated as RTO) and duplicate cumulative acknowledgements
(DupAcks). </font><font size=2 face="sans-serif"><br></font><br><br><font size=2 face="sans-serif">Note that older versions of the Spectrum
Scale code had a third criteria in checking for 'bad state', which included
checking if unacked was greater than 8, but that check would sometimes
call-out a socket that was working fine, so this third check has been removed
via the APAR IJ02566. All Spectrum Scale V5 code has this fix and
the 4.2.X code stream picked up this fix in PTF 7 (4.2.3.7 ships APAR IJ02566).</font><br><br><font size=2 face="sans-serif">More details on debugging expels using
these TCP connection messages are in the presentation you referred to,
which I posted here:</font><a href="https://www.ibm.com/developerworks/community/wikis/home?lang=en_us#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/DEBUG%20Expels"><font size=2 color=blue face="sans-serif">https://www.ibm.com/developerworks/community/wikis/home?lang=en_us#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/DEBUG%20Expels</font></a><br><br><font size=2 face="sans-serif">Regards,<br>John Lewars <br>Technical Computing Development, IBM Poughkeepsie<br></font><br><br><font size=1 color=#800080 face="sans-serif">----- Forwarded by Lyle
Gayne/Poughkeepsie/IBM on 09/27/2018 11:15 AM -----</font><br><br><br><font size=3>Hi All, </font><br><br><font size=3>2018-09-27_09:48:50.923-0500: [E] The TCP connection to
IP address 1.2.3.4 some client <c0n509> (socket 442) state is unexpected:
ca_state=1 unacked=3 <b>rto=27008000</b></font><br><br><font size=3>Seeing errors like the above and trying to track down
the root cause. I know that at last weeks\u2019 GPFS User Group meeting
at ORNL this very error message was discussed, but I don\u2019t recall the
details and the slides haven\u2019t been posted to the website yet. IIRC,
the \u201crto\u201d is significant \u2026 </font><br><br><font size=3>I\u2019ve Googled, but haven\u2019t gotten any hits, nor have
I found anything in the GPFS 4.2.2 Problem Determination Guide.</font><br><br><font size=3>Thanks in advance\u2026</font><br><br><font size=3>\u2014</font><br><font size=3>Kevin Buterbaugh - Senior System Administrator</font><br><font size=3>Vanderbilt University - Advanced Computing Center for
Research and Education</font><br><a href="mailto:Kevin.Buterbaugh@vanderbilt.edu"><font size=3 color=blue><u>Kevin.Buterbaugh@vanderbilt.edu</u></font></a><font size=3>- (615)875-9633</font><br><br><br><tt><font size=2>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><br><br><BR>