<div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:12pt" ><div dir="ltr" >Colleagues of mine have communicated that this has been seen in the past due to interaction between the Spectrum Scale performance monitor (zimon) and Grafana. Are you using Grafana? Normally zimon is configured to use local port 9094 so if that is the port which the CLOSE_WAIT is attached then it would seem to be an instance of this problem. You can use the following to check for this condition.</div>
<div dir="ltr" > </div>
<div dir="ltr" >netstat -ntp | grep <span style="font-size:12pt;" >"\:9094 .*CLOSE_WAIT" | wc -l</span></div>
<div dir="ltr" > </div>
<div dir="ltr" > </div>
<div dir="ltr" ><font size="2" face="Default Sans Serif,Verdana,Arial,Helvetica,sans-serif" ><span style="font-size:1.143em;" >Fred</span></font>
<div><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div dir="ltr" ><font size="2" face="Default Sans Serif,Verdana,Arial,Helvetica,sans-serif" ><span style="font-size:1.143em;" >__________________________________________________<br>Fred Stock | IBM Pittsburgh Lab | 720-430-8821<br>stockf@us.ibm.com</span></font></div></div></div>
<div> </div>
<div> </div>
<blockquote data-history-content-modified="1" style="border-left:solid #aaaaaa 2px; margin-left:5px; padding-left:5px; direction:ltr; margin-right:0px" >----- Original message -----<br>From: Leonardo Sala <leonardo.sala@psi.ch><br>Sent by: gpfsug-discuss-bounces@spectrumscale.org<br>To: "gpfsug-discuss@spectrumscale.org" <gpfsug-discuss@spectrumscale.org><br>Cc:<br>Subject: [EXTERNAL] [gpfsug-discuss] Thousands of CLOSE_WAIT IPV6 connections on CES<br>Date: Fri, Feb 21, 2020 9:30 AM<br> <br><!--Notes ACF
<meta http-equiv="content-type" content="text/html; charset=utf8" >-->
<p><tt>Dear all,</tt></p>
<p><tt>I was wondering if anybody recently encountered a similar issue (I found a related thread from 2018, but it was inconclusive). I just found that one of our production CES nodes have 28k CLOSE_WAIT tcp6 connections, I do not understand why... the second node in the same cluster does not have this issue. Both are:</tt></p>
<p><tt>- GPFS 5.0.4.2</tt></p>
<p><tt>- RHEL 7.4</tt></p>
<p><tt>has anybody else encountered anything similar? In the last few days it seems it happened once on one node, and twice on the other, but never on both... </tt></p>
<p><tt>Thanks for any feedback!</tt></p>
<p><tt>cheers</tt></p>
<p><tt>leo</tt></p>
<div><font size="2" face="Default Monospace,Courier New,Courier,monospace" >--<br>Paul Scherrer Institut<br>Dr. Leonardo Sala<br>Group Leader High Performance Computing<br>Deputy Section Head Science IT<br>Science IT<br>WHGA/036<br>Forschungstrasse 111<br>5232 Villigen PSI<br>Switzerland<br><br>Phone: +41 56 310 3369<br><a href="mailto:leonardo.sala@psi.ch" target="_blank">leonardo.sala@psi.ch</a><br><a href="http://www.psi.ch" target="_blank">www.psi.ch</a></font></div>
<div><font size="2" face="Default Monospace,Courier New,Courier,monospace" >_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a> </font></div></blockquote>
<div> </div></div></div><BR>