<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Hi All,
<div class=""><br class="">
</div>
<div class="">Time for a daily update on this saga\u2026</div>
<div class=""><br class="">
</div>
<div class="">First off, responses to those who have responded to me:</div>
<div class=""><br class="">
</div>
<div class="">Yaron - we have QLogic switches, but I\u2019ll RTFM and figure out how to clear the counters \u2026 with a quick look via the CLI interface to one of them I don\u2019t see how to even look at those counters, must less clear them, but I\u2019ll do some digging. &nbsp;QLogic
 does have a GUI app, but given that the Mac version is PowerPC only I think that\u2019s a dead end! &nbsp;:-O</div>
<div class=""><br class="">
</div>
<div class="">Jonathan - understood. &nbsp;We were just wanting to eliminate as much hardware as potential culprits as we could. &nbsp;The storage arrays will all get a power-cycle this Sunday when we take a downtime to do firmware upgrades on them \u2026 the vendor is basically
 refusing to assist further until we get on the latest firmware.</div>
<div class=""><br class="">
</div>
<div class="">So \u2026 we had noticed that things seem to calm down starting Friday evening and continuing throughout the weekend. &nbsp;We have a script that runs every half hour and if there\u2019s any NSD servers where \u201cmmdiag \u2014iohist\u201d shows an I/O &gt; 1,000 ms, we get
 an alert (again, designed to alert us of a CBM failure). &nbsp;We only got three all weekend long (as opposed to last week, when the alerts were coming every half hour round the clock).</div>
<div class=""><br class="">
</div>
<div class="">Then, this morning I repeated the \u201cdd\u201d test that I had run before and after replacing the FC cables going to \u201ceon34\u201d and which had showed very typical I/O rates for all the NSDs except for the 4 in eon34, which were quite poor (~1.5 - 10 MB/sec).
 &nbsp;I ran the new tests this morning from different NSD servers and with a higher \u201ccount\u201d passed to dd to eliminate any potential caching effects. &nbsp;I ran the test twice from two different NSD servers and this morning all NSDs - including those on eon34 - showed
 normal I/O rates!</div>
<div class=""><br class="">
</div>
<div class="">Argh - so do we have a hardware problem or not?!?</div>
<div class=""><br class="">
</div>
<div class="">I still think we do, but am taking *nothing* for granted at this point! &nbsp; So today we also used another script we\u2019ve written to do some investigation \u2026 basically we took the script which runs \u201cmmdiag \u2014iohist\u201d and added some options to it so that
 for every I/O greater than the threshold it will see which client issued the I/O. &nbsp;It then queries SLURM to see what jobs are running on that client.</div>
<div class=""><br class="">
</div>
<div class="">Interestingly enough, one user showed up waaaayyyyyy more often than anybody else. &nbsp;And many times she was on a node with only one other user who we know doesn\u2019t access the GPFS filesystem and other times she was the only user on the node. &nbsp;</div>
<div class=""><br class="">
</div>
<div class="">We certainly recognize that correlation is not causation (she could be a victim and not the culprit), but she was on so many of the reported clients that we decided to investigate further \u2026 but her jobs seem to have fairly modest I/O requirements.
 &nbsp;Each one processes 4 input files, which are basically just gzip\u2019d text files of 1.5 - 5 GB in size. &nbsp;This is what, however, prompted my other query to the list about determining which NSDs a given file has its\u2019 blocks on. &nbsp;I couldn\u2019t see how files of that
 size could have all their blocks on only a couple of NSDs in the pool (out of 19 total!) but wanted to verify that. &nbsp;The files that I have looked at are evenly spread out across the NSDs.</div>
<div class=""><br class="">
</div>
<div class="">So given that her files are spread across all 19 NSDs in the pool and the high I/O wait times are almost always only on LUNs in eon34 (and, more specifically, on two of the four LUNs in eon34) I\u2019m pretty well convinced it\u2019s not her jobs causing
 the problems \u2026 I\u2019m back to thinking a weird hardware issue.</div>
<div class=""><br class="">
</div>
<div class="">But if anyone wants to try to convince me otherwise, I\u2019ll listen\u2026</div>
<div class=""><br class="">
</div>
<div class="">Thanks!</div>
<div class=""><br class="">
</div>
<div class="">Kevin<br class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Jul 8, 2018, at 12:32 PM, Yaron Daniel &lt;<a href="mailto:YARD@il.ibm.com" class="">YARD@il.ibm.com</a>&gt; wrote:</div>
<br class="Apple-interchange-newline">
<div class=""><span style=" font-size:10pt;font-family:sans-serif" class="">Hi</span><br class="">
<br class="">
<span style=" font-size:10pt;font-family:sans-serif" class="">Clean all counters on the FC switches and see which port have errors .</span><br class="">
<br class="">
<span style=" font-size:10pt;font-family:sans-serif" class="">For brocade run :</span><br class="">
<br class="">
<span style=" font-size:10pt;font-family:sans-serif" class="">slotstatsclear</span><br class="">
<span style=" font-size:10pt;font-family:sans-serif" class="">statsclear</span><br class="">
<span style=" font-size:10pt;font-family:sans-serif" class="">porterrshow</span><br class="">
<br class="">
<span style=" font-size:10pt;font-family:sans-serif" class="">For cisco run:</span><br class="">
<span style=" font-size:9pt;font-family:Arial" class="">&nbsp;</span><br class="">
<span style=" font-size:9pt;font-family:Arial" class="">clear countersall</span><br class="">
<br class="">
<span style=" font-size:9pt;font-family:Arial" class="">There might be bad gbic/cable/Storage gbic, which can affect the performance, if there is something like that - u can see which ports have errors grow over time.</span><br class="">
<span style=" font-size:10pt;font-family:Arial" class="">Regards</span><br class="">
<span style=" font-size:9pt;font-family:Arial" class="">&nbsp;</span><br class="">
<table width="780" style="border-collapse:collapse;" class="">
<tbody class="">
<tr height="8" class="">
<td width="780" colspan="4" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" class="">
<div align="center" class="">
<hr noshade="" class="">
</div>
<br class="">
<span style=" font-size:1pt;font-family:Arial" class="">&nbsp;</span></td>
</tr>
<tr height="8" class="">
<td width="780" colspan="4" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" class="">
<span style=" font-size:1pt;font-family:Arial" class="">&nbsp;</span></td>
</tr>
<tr height="8" class="">
<td width="516" colspan="2" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" class="">
<span style=" font-size:10pt;color:blue;font-family:Arial" class=""><b class="">Yaron Daniel</b></span></td>
<td width="168" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" class="">
<span style=" font-size:10pt;color:#5f5f5f;font-family:Arial" class="">&nbsp;94 Em Ha'Moshavot Rd</span></td>
<td width="96" rowspan="3" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" class="">
<div align="right" class=""><span id="cid:_1_0DF9D8440DF9D290006059D7C22582C4">&lt;ATT00001.gif&gt;</span></div>
</td>
</tr>
<tr height="8" class="">
<td width="516" colspan="2" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" class="">
<span style=" font-size:10pt;color:blue;font-family:Arial" class=""><b class="">Storage Architect \u2013 IL Lab Services (Storage)</b></span></td>
<td width="168" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" class="">
<span style=" font-size:10pt;color:#5f5f5f;font-family:Arial" class="">&nbsp;Petach Tiqva, 49527</span></td>
</tr>
<tr height="8" class="">
<td width="516" colspan="2" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" class="">
<span style=" font-size:10pt;color:blue;font-family:Arial" class=""><b class="">IBM Global Markets, Systems HW Sales</b></span></td>
<td width="168" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" class="">
<span style=" font-size:10pt;color:#5f5f5f;font-family:Arial" class="">&nbsp;Israel</span></td>
</tr>
<tr height="8" class="">
<td width="516" colspan="2" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" class="">
<span style=" font-size:10pt;color:blue;font-family:Arial" class=""><b class="">&nbsp;</b></span></td>
<td width="168" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" class="">
<span style=" font-size:10pt;color:#5f5f5f;font-family:Arial" class="">&nbsp;</span></td>
<td width="96" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" class="">
<span style=" font-size:9pt;font-family:Arial" class="">&nbsp;</span></td>
</tr>
<tr height="8" class="">
<td width="90" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" class="">
<span style=" font-size:10pt;color:#5f5f5f;font-family:Arial" class="">Phone:</span></td>
<td width="426" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" class="">
<span style=" font-size:10pt;color:#5f5f5f;font-family:Arial" class="">&#43;972-3-916-5672</span></td>
<td width="168" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" class="">
<span style=" font-size:10pt;color:#5f5f5f;font-family:Arial" class="">&nbsp;</span></td>
<td width="96" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" class="">
<span style=" font-size:10pt" class="">&nbsp;</span></td>
</tr>
<tr height="8" class="">
<td width="90" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" class="">
<span style=" font-size:10pt;color:#5f5f5f;font-family:Arial" class="">Fax:</span></td>
<td width="426" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" class="">
<span style=" font-size:10pt;color:#5f5f5f;font-family:Arial" class="">&#43;972-3-916-5672</span></td>
<td width="168" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" class="">
<span style=" font-size:10pt;color:#5f5f5f;font-family:Arial" class="">&nbsp; </span></td>
<td width="96" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" class="">
<span style=" font-size:10pt" class="">&nbsp;</span></td>
</tr>
<tr height="8" class="">
<td width="90" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" class="">
<span style=" font-size:10pt;color:#5f5f5f;font-family:Arial" class="">Mobile:</span></td>
<td width="426" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" class="">
<span style=" font-size:10pt;color:#5f5f5f;font-family:Arial" class="">&#43;972-52-8395593</span></td>
<td width="168" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" class="">
<span style=" font-size:10pt;color:#5f5f5f;font-family:Arial" class="">&nbsp; </span></td>
<td width="96" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" class="">
<span style=" font-size:10pt" class="">&nbsp;</span></td>
</tr>
<tr height="8" class="">
<td width="90" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" class="">
<span style=" font-size:10pt;color:#5f5f5f;font-family:Arial" class="">e-mail:</span></td>
<td width="426" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" class="">
<span style=" font-size:10pt;color:#5f5f5f;font-family:Arial" class=""><a href="mailto:yard@il.ibm.com" class="">yard@il.ibm.com</a></span></td>
<td width="168" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" class="">
<span style=" font-size:10pt;color:#5f5f5f;font-family:Arial" class="">&nbsp; </span></td>
<td width="96" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" class="">
<span style=" font-size:10pt" class="">&nbsp;</span></td>
</tr>
<tr height="8" class="">
<td width="516" colspan="2" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" class="">
<a href="https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.ibm.com%2Fil%2Fhe%2F&amp;data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C7c1ced16f6d44055c63408d5e4fa7d2e%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636666686866046739&amp;sdata=fpkPC3%2FjrhpFp1iLq3THOlRQTCGFdAInnRjsIs9zFEc%3D&amp;reserved=0" class=""><span style=" font-size:10pt;color:blue;font-family:Arial" class=""><u class="">IBM
 Israel</u></span></a></td>
<td width="168" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" class="">
<span style=" font-size:10pt;color:#5f5f5f;font-family:Arial" class="">&nbsp; </span></td>
<td width="96" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" class="">
<span style=" font-size:10pt" class="">&nbsp;</span></td>
</tr>
<tr height="8" class="">
<td width="780" colspan="4" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" class="">
<span style=" font-size:9pt;color:#5f5f5f;font-family:Arial" class="">&nbsp;</span></td>
</tr>
<tr height="8" class="">
<td width="780" colspan="4" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" class="">
<span style=" font-size:9pt;color:#5f5f5f;font-family:Arial" class="">&nbsp;</span></td>
</tr>
</tbody>
</table>
<p style="margin-top:0px;margin-Bottom:0px" class=""></p>
<br class="">
<span id="cid:_1_0DF969980DF96588006059D7C22582C4">&lt;ATT00002.gif&gt;</span><span id="cid:_1_0DF96BA00DF96588006059D7C22582C4">&lt;ATT00003.gif&gt;</span><span id="cid:_1_0DF96DA80DF96588006059D7C22582C4">&lt;ATT00004.gif&gt;</span><span id="cid:_1_0DF96FB00DF96588006059D7C22582C4">&lt;ATT00005.gif&gt;</span><span style=" font-size:12pt" class="">&nbsp;</span><span id="cid:_1_0DF971D00DF96588006059D7C22582C4">&lt;ATT00006.gif&gt;</span><span style=" font-size:12pt" class="">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
</span><span id="cid:_2_0DF9741C0DF96588006059D7C22582C4">&lt;ATT00007.jpeg&gt;</span><br class="">
<br class="">
<br class="">
<br class="">
<span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif" class="">From: &nbsp; &nbsp; &nbsp; &nbsp;</span><span style=" font-size:9pt;font-family:sans-serif" class="">Jonathan Buzzard &lt;<a href="mailto:jonathan.buzzard@strath.ac.uk" class="">jonathan.buzzard@strath.ac.uk</a>&gt;</span><br class="">
<span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif" class="">To: &nbsp; &nbsp; &nbsp; &nbsp;</span><span style=" font-size:9pt;font-family:sans-serif" class=""><a href="mailto:gpfsug-discuss@spectrumscale.org" class="">gpfsug-discuss@spectrumscale.org</a></span><br class="">
<span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif" class="">Date: &nbsp; &nbsp; &nbsp; &nbsp;</span><span style=" font-size:9pt;font-family:sans-serif" class="">07/07/2018 11:43 AM</span><br class="">
<span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif" class="">Subject: &nbsp; &nbsp; &nbsp; &nbsp;</span><span style=" font-size:9pt;font-family:sans-serif" class="">Re: [gpfsug-discuss] High I/O wait times</span><br class="">
<span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif" class="">Sent by: &nbsp; &nbsp; &nbsp; &nbsp;</span><span style=" font-size:9pt;font-family:sans-serif" class=""><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" class="">gpfsug-discuss-bounces@spectrumscale.org</a></span><br class="">
<hr noshade="" class="">
<br class="">
<br class="">
<br class="">
<tt class=""><span style=" font-size:10pt" class="">On 07/07/18 01:28, Buterbaugh, Kevin L wrote:<br class="">
<br class="">
[SNIP]<br class="">
<br class="">
&gt; <br class="">
&gt; So, to try to rule out everything but the storage array we replaced the <br class="">
&gt; FC cables going from the SAN switches to the array, plugging the new <br class="">
&gt; cables into different ports on the SAN switches. &nbsp;Then we repeated the <br class="">
&gt; dd tests from a different NSD server, which both eliminated the NSD <br class="">
&gt; server and its\u2019 FC cables as a potential cause \u2026 and saw results <br class="">
&gt; virtually identical to the previous test. &nbsp;Therefore, we feel pretty <br class="">
&gt; confident that it is the storage array and have let the vendor know all <br class="">
&gt; of this.<br class="">
<br class="">
I was not thinking of doing anything quite as drastic as replacing <br class="">
stuff, more look into the logs on the switches in the FC network and <br class="">
examine them for packet errors. The above testing didn't eliminate bad <br class="">
optics in the storage array itself for example, though it does appear to <br class="">
be the storage arrays themselves. Sounds like they could do with a power <br class="">
cycle...<br class="">
<br class="">
JAB.<br class="">
<br class="">
-- <br class="">
Jonathan A. Buzzard &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; Tel: &#43;44141-5483420<br class="">
HPC System Administrator, ARCHIE-WeSt.<br class="">
University of Strathclyde, John Anderson Building, Glasgow. G4 0NG<br class="">
_______________________________________________<br class="">
gpfsug-discuss mailing list<br class="">
gpfsug-discuss at <a href="http://spectrumscale.org" class="">spectrumscale.org</a><br class="">
</span></tt><a href="https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttp-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss%26d%3DDwIGaQ%26c%3Djf_iaSHvJObTbx-siA1ZOg%26r%3DBn1XE9uK2a9CZQ8qKnJE3Q%26m%3DTM-kJsvzTX9cq_xmR5ITHclBCfO4FDvZ3ZxyugfJCfQ%26s%3DAss164qVEhb9fC4_VCmzfZeYd_BLOv9cZsfkrzqi8pM%26e%3D&amp;data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C7c1ced16f6d44055c63408d5e4fa7d2e%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636666686866046739&amp;sdata=B%2F2Q9L1bwUvPHv858hLhTzt1hFT%2BMhCIOVeqGvLv3Rg%3D&amp;reserved=0" originalsrc="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&amp;d=DwIGaQ&amp;c=jf_iaSHvJObTbx-siA1ZOg&amp;r=Bn1XE9uK2a9CZQ8qKnJE3Q&amp;m=TM-kJsvzTX9cq_xmR5ITHclBCfO4FDvZ3ZxyugfJCfQ&amp;s=Ass164qVEhb9fC4_VCmzfZeYd_BLOv9cZsfkrzqi8pM&amp;e=" shash="bn6QhCapPNWgL4/t6rqwzZiOSdZ25Vvz5eXW6n1SyuHqq9ux&#43;hKTXJOuUtgnuoP4KkCjDpbrH6SXN5y8rMiM5EKuVqxzNYHVmA0EUPaaPITt7VHgz07kEKG7xT2Wvc/vypw2FtTh461y4/CH7/uQQa4M42wQZuOnZIabLpXDTNQ=" class=""><tt class=""><span style=" font-size:10pt" class="">https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&amp;d=DwIGaQ&amp;c=jf_iaSHvJObTbx-siA1ZOg&amp;r=Bn1XE9uK2a9CZQ8qKnJE3Q&amp;m=TM-kJsvzTX9cq_xmR5ITHclBCfO4FDvZ3ZxyugfJCfQ&amp;s=Ass164qVEhb9fC4_VCmzfZeYd_BLOv9cZsfkrzqi8pM&amp;e=</span></tt></a><tt class=""><span style=" font-size:10pt" class=""><br class="">
<br class="">
</span></tt><br class="">
<br class="">
<br class="">
_______________________________________________<br class="">
gpfsug-discuss mailing list<br class="">
gpfsug-discuss at <a href="http://spectrumscale.org" class="">spectrumscale.org</a><br class="">
<a href="https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&amp;amp;data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C7c1ced16f6d44055c63408d5e4fa7d2e%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636666686866066749&amp;amp;sdata=Viltitj3L9aScuuVKCLSp9FKkj7xdzWxsvvPVDSUqHw%3D&amp;amp;reserved=0" class="">https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&amp;amp;data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C7c1ced16f6d44055c63408d5e4fa7d2e%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636666686866066749&amp;amp;sdata=Viltitj3L9aScuuVKCLSp9FKkj7xdzWxsvvPVDSUqHw%3D&amp;amp;reserved=0</a><br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>