<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Hi All,
<div class=""><br class="">
</div>
<div class="">My thanks to Aaron, Sven, Steve, and whoever responded for the GPFS team. You confirmed what I suspected \u2026 my example 10 second I/O was _from an NSD server_ \u2026 and since we\u2019re in a 8 Gb FC SAN environment, it therefore means - correct me if I\u2019m
wrong about this someone - that I\u2019ve got a problem somewhere in one (or more) of the following 3 components:</div>
<div class=""><br class="">
</div>
<div class="">1) the NSD servers</div>
<div class="">2) the SAN fabric</div>
<div class="">3) the storage arrays</div>
<div class=""><br class="">
</div>
<div class="">I\u2019ve been looking at all of the above and none of them are showing any obvious problems. I\u2019ve actually got a techie from the storage array vendor stopping by on Thursday, so I\u2019ll see if he can spot anything there. Our FC switches are QLogic\u2019s,
so I\u2019m kinda screwed there in terms of getting any help. But I don\u2019t see any errors in the switch logs and \u201cshow perf\u201d on the switches is showing I/O rates of 50-100 MB/sec on the in use ports, so I don\u2019t _think_ that\u2019s the issue.</div>
<div class=""><br class="">
</div>
<div class="">And this is the GPFS mailing list, after all \u2026 so let\u2019s talk about the NSD servers. Neither memory (64 GB) nor CPU (2 x quad-core Intel Xeon E5620\u2019s) appear to be an issue. But I have been looking at the output of \u201cmmfsadm saferdump nsd\u201d based
on what Aaron and then Steve said. Here\u2019s some fairly typical output from one of the SMALL queues (I\u2019ve checked several of my 8 NSD servers and they\u2019re all showing similar output):</div>
<div class=""><br class="">
</div>
<div class="">
<div class=""> Queue NSD type NsdQueueTraditional [244]: SMALL, threads started 12, active 3, highest 12, deferred 0, chgSize 0, draining 0, is_chg 0</div>
<div class=""> requests pending 0, highest pending 73, total processed 4859732</div>
<div class=""> mutex 0x7F3E449B8F10, reqCond 0x7F3E449B8F58, thCond 0x7F3E449B8F98, queue 0x7F3E449B8EF0, nFreeNsdRequests 29</div>
</div>
<div class=""><br class="">
</div>
<div class="">And for a LARGE queue:</div>
<div class=""><span style="color: rgb(255, 255, 255); font-family: Menlo; font-size: 16px;" class=""><br class="">
</span></div>
<div class=""> Queue NSD type NsdQueueTraditional [8]: LARGE, threads started 12, active 1, highest 12, deferred 0, chgSize 0, draining 0, is_chg 0</div>
<div class=""> requests pending 0, highest pending 71, total processed 2332966</div>
<div class=""> mutex 0x7F3E441F3890, reqCond 0x7F3E441F38D8, thCond 0x7F3E441F3918, queue 0x7F3E441F3870, nFreeNsdRequests 31</div>
<div class=""><br class="">
</div>
<div class="">So my large queues seem to be slightly less utilized than my small queues overall \u2026 i.e. I see more inactive large queues and they generally have a smaller \u201chighest pending\u201d value.</div>
<div class=""><br class="">
</div>
<div class="">Question: are those non-zero \u201chighest pending\u201d values something to be concerned about?</div>
<div class=""><br class="">
</div>
<div class="">I have the following thread-related parameters set:</div>
<div class=""><br class="">
</div>
<div class="">[common]</div>
<div class="">
<div class="">maxReceiverThreads 12</div>
<div class="">nsdMaxWorkerThreads 640</div>
<div class="">nsdThreadsPerQueue 4</div>
<div class="">nsdSmallThreadRatio 3</div>
<div class="">workerThreads 128</div>
<div class=""><br class="">
</div>
<div class="">[serverLicense]</div>
<div class="">nsdMaxWorkerThreads 1024</div>
<div class="">nsdThreadsPerQueue 12</div>
<div class="">nsdSmallThreadRatio 1</div>
<div class="">pitWorkerThreadsPerNode 3</div>
<div class="">workerThreads 1024</div>
</div>
<div class=""><br class="">
</div>
<div class="">Also, at the top of the \u201cmmfsadm saferdump nsd\u201d output I see:</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">Total server worker threads: running 1008, desired 147, forNSD 147, forGNR 0, nsdBigBufferSize 16777216</div>
<div class="">nsdMultiQueue: 256, nsdMultiQueueType: 1, nsdMinWorkerThreads: 16, nsdMaxWorkerThreads: 1024</div>
</div>
<div class=""><br class="">
</div>
<div class="">Question: is the fact that 1008 is pretty close to 1024 a concern?</div>
<div class=""><br class="">
</div>
<div class="">Anything jump out at anybody? I don\u2019t mind sharing full output, but it is rather lengthy. Is this worthy of a PMR?</div>
<div class=""><br class="">
</div>
<div class="">Thanks!</div>
<div class=""><br class="">
</div>
<div class="">--</div>
<div class="">
<div class="">
<div style="color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div class="">Kevin Buterbaugh - Senior System Administrator</div>
<div class="">Vanderbilt University - Advanced Computing Center for Research and Education</div>
<div class=""><a href="mailto:Kevin.Buterbaugh@vanderbilt.edu" class="">Kevin.Buterbaugh@vanderbilt.edu</a> - (615)875-9633</div>
</div>
</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Feb 17, 2019, at 1:01 PM, IBM Spectrum Scale <<a href="mailto:scale@us.ibm.com" class="">scale@us.ibm.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class=""><span style=" font-size:10pt;font-family:sans-serif" class="">Hi Kevin,</span><br class="">
<br class="">
<span style=" font-size:10pt;font-family:sans-serif" class="">The I/O hist shown by the command mmdiag --iohist actually depends on the node on which you are running this command from.</span><br class="">
<span style=" font-size:10pt;font-family:sans-serif" class="">If you are running this on a NSD server node then it will show the time taken to complete/serve the read or write I/O operation sent from the client node.
</span><br class="">
<span style=" font-size:10pt;font-family:sans-serif" class="">And if you are running this on a client (or non NSD server) node then it will show the complete time taken by the read or write I/O operation requested by the client node to complete.</span><br class="">
<span style=" font-size:10pt;font-family:sans-serif" class="">So in a nut shell for the NSD server case it is just the latency of the I/O done on disk by the server whereas for the NSD client case it also the latency of send and receive of I/O request to the
NSD server along with the latency of I/O done on disk by the NSD server.</span><br class="">
<span style=" font-size:10pt;font-family:sans-serif" class="">I hope this answers your query.</span><br class="">
<br class="">
<br class="">
<span style=" font-size:10pt;font-family:sans-serif" class="">Regards, The Spectrum Scale (GPFS) team<br class="">
<br class="">
------------------------------------------------------------------------------------------------------------------<br class="">
If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at
</span><a href="https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ibm.com%2Fdeveloperworks%2Fcommunity%2Fforums%2Fhtml%2Fforum%3Fid%3D11111111-0000-0000-0000-000000000479&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2bfb2e8e30e64fa06c0f08d6959b2d38%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636860891056267091&sdata=%2FWFsVfr73xZcfH25vIFYC4ts7LlWDFUIoh9fLheAEwE%3D&reserved=0" originalsrc="https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479" shash="uIDVumGFVRcQTXbM+9GirK4DaS9152BkfbSfKSXKaJ9oOhW4IBaT5orUgyPlYNp9GVyGnqPmLL+Izui6rF3cu8N10TOX4jIH2ty6rOrDp7YamcSII+K5+DxJb2ttYJQ/LldcNx1jIDqYcaKxxXrVRJEYkuBewaG/Fnfe8t7wom0=" class=""><span style=" font-size:10pt;font-family:sans-serif" class="">https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479</span></a><span style=" font-size:10pt;font-family:sans-serif" class="">.
<br class="">
<br class="">
If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries.
<br class="">
<br class="">
The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team.</span><br class="">
<br class="">
<br class="">
<br class="">
<span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif" class="">From: </span><span style=" font-size:9pt;font-family:sans-serif" class="">"Buterbaugh, Kevin L" <<a href="mailto:Kevin.Buterbaugh@Vanderbilt.Edu" class="">Kevin.Buterbaugh@Vanderbilt.Edu</a>></span><br class="">
<span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif" class="">To: </span><span style=" font-size:9pt;font-family:sans-serif" class="">gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" class="">gpfsug-discuss@spectrumscale.org</a>></span><br class="">
<span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif" class="">Date: </span><span style=" font-size:9pt;font-family:sans-serif" class="">02/16/2019 08:18 PM</span><br class="">
<span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif" class="">Subject: </span><span style=" font-size:9pt;font-family:sans-serif" class="">[gpfsug-discuss] Clarification of mmdiag --iohist output</span><br class="">
<span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif" class="">Sent by: </span><span style=" font-size:9pt;font-family:sans-serif" class=""><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" class="">gpfsug-discuss-bounces@spectrumscale.org</a></span><br class="">
<hr noshade="" class="">
<br class="">
<br class="">
<br class="">
<span style=" font-size:12pt" class="">Hi All, </span><br class="">
<br class="">
<span style=" font-size:12pt" class="">Been reading man pages, docs, and Googling, and haven\u2019t found a definitive answer to this question, so I knew exactly where to turn\u2026 ;-)</span><br class="">
<br class="">
<span style=" font-size:12pt" class="">I\u2019m dealing with some slow I/O\u2019s to certain storage arrays in our environments \u2026 like really, really slow I/O\u2019s \u2026 here\u2019s just one example from one of my NSD servers of a 10 second I/O:</span><br class="">
<br class="">
<span style=" font-size:12pt" class="">08:49:34.943186 W data 30:41615622144 2048 10115.192 srv dm-92 <client IP redacted></span><br class="">
<br class="">
<span style=" font-size:12pt" class="">So here\u2019s my question \u2026 when mmdiag \u2014iohist tells me that that I/O took slightly over 10 seconds, is that:</span><br class="">
<br class="">
<span style=" font-size:12pt" class="">1. The time from when the NSD server received the I/O request from the client until it shipped the data back onto the wire towards the client?</span><br class="">
<span style=" font-size:12pt" class="">2. The time from when the client issued the I/O request until it received the data back from the NSD server?</span><br class="">
<span style=" font-size:12pt" class="">3. Something else?</span><br class="">
<br class="">
<span style=" font-size:12pt" class="">I\u2019m thinking it\u2019s #1, but want to confirm. Which one it is has very obvious implications for our troubleshooting steps. Thanks in advance\u2026</span><br class="">
<br class="">
<span style=" font-size:12pt" class="">Kevin</span><br class="">
<span style=" font-size:12pt" class="">\u2014</span><br class="">
<span style=" font-size:12pt" class="">Kevin Buterbaugh - Senior System Administrator</span><br class="">
<span style=" font-size:12pt" class="">Vanderbilt University - Advanced Computing Center for Research and Education</span><br class="">
<a href="mailto:Kevin.Buterbaugh@vanderbilt.edu" class=""><span style=" font-size:12pt;color:blue" class=""><u class="">Kevin.Buterbaugh@vanderbilt.edu</u></span></a><span style=" font-size:12pt" class="">- (615)875-9633</span><br class="">
<tt class=""><span style=" font-size:10pt" class="">_______________________________________________<br class="">
gpfsug-discuss mailing list<br class="">
gpfsug-discuss at <a href="http://spectrumscale.org" class="">spectrumscale.org</a><br class="">
</span></tt><a href="https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2bfb2e8e30e64fa06c0f08d6959b2d38%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636860891056277100&sdata=PP%2Bs3UFJOHEIFNk7aOXJgo46GVeQr6P%2FLwgDUIGzAXQ%3D&reserved=0" originalsrc="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" shash="yyvafoPIbziq0tkdkRnqDMOYjlW2bK3ggKgbkv07YvpBi1BlUGlISV21XU39/4iQJQfV+0sv7BjE6cOqysttSSf0/FwmfeUy8pLuw/oZrUPLN6LSFQL5fGF2lhYKvObnIAOmY+PWYZnXj+Czz/VeWTGdQL2DqpYlQ9xMht/bdUQ=" class=""><tt class=""><span style=" font-size:10pt" class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</span></tt></a><tt class=""><span style=" font-size:10pt" class=""><br class="">
</span></tt><br class="">
<br class="">
<br class="">
_______________________________________________<br class="">
gpfsug-discuss mailing list<br class="">
gpfsug-discuss at <a href="http://spectrumscale.org" class="">spectrumscale.org</a><br class="">
<a href="https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&amp;data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2bfb2e8e30e64fa06c0f08d6959b2d38%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636860891056297114&amp;sdata=5pL67mhVyScJovkRHRqZog9bM5BZG8F2q972czIYAbA%3D&amp;reserved=0" class="">https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&amp;data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2bfb2e8e30e64fa06c0f08d6959b2d38%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636860891056297114&amp;sdata=5pL67mhVyScJovkRHRqZog9bM5BZG8F2q972czIYAbA%3D&amp;reserved=0</a><br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>