<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
Hi Stephen,
<div class=""><br class="">
</div>
<div class="">My apologies - Jordan\u2019s response had been snagged by the University's SPAM filter (I went and checked and found it after receiving your e-mail)\u2026</div>
<div class=""><br class="">
</div>
<div class="">Kevin</div>
<div class=""><br class="">
<div>
<blockquote type="cite" class="">
<div class="">On May 4, 2017, at 12:04 PM, Stephen Ulmer <<a href="mailto:ulmer@ulmer.org" class="">ulmer@ulmer.org</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
Look at Jordan\u2019s answer, he explains what significance 0 has. In short, GPFS will use one thread per core per server, so they could add to 31 quickly. ;)
<div class=""><br class="">
<div class="">
<div style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
-- <br class="">
Stephen<br class="">
<br class="">
<br class="">
</div>
</div>
<br class="">
<div class="">
<blockquote type="cite" class="">
<div class="">On May 4, 2017, at 12:56 PM, Buterbaugh, Kevin L <<a href="mailto:Kevin.Buterbaugh@vanderbilt.edu" class="">Kevin.Buterbaugh@Vanderbilt.Edu</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
Hi Kums,
<div class=""><br class="">
</div>
<div class="">Thanks for the info on the releases \u2026 can you clarify about pitWorkerThreadsPerNode? As I said in my original post, on all 8 NSD servers and the filesystem manager it is set to zero. No matter how many times I add zero to zero I don\u2019t get a
value > 31! ;-) So I take it that zero has some sort of unspecified significance? Thanks\u2026</div>
<div class=""><br class="">
</div>
<div class="">Kevin</div>
<div class=""><br class="">
<div class="">
<blockquote type="cite" class="">
<div class="">On May 4, 2017, at 11:49 AM, Kumaran Rajaram <<a href="mailto:kums@us.ibm.com" class="">kums@us.ibm.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class=""><font size="2" face="sans-serif" class="">Hi,</font><br class="">
<br class="">
<font size="3" color="red" class="">>>I\u2019m running 4.2.2.3 on my GPFS servers (some clients are on 4.2.1.1 or 4.2.0.3 and are gradually being upgraded). What version of GPFS fixes this? With what I\u2019m doing I need the ability to run mmrestripefs.</font><br class="">
<br class="">
<font size="2" face="sans-serif" class="">GPFS version 4.2.3.0 (and above) fixes this issue and supports "sum of pitWorkerThreadsPerNode of the participating nodes (-N parameter to mmrestripefs)" to exceed 31.</font><br class="">
<br class="">
<font size="2" face="sans-serif" class="">If you are using 4.2.2.3, then depending on "number of nodes participating in the mmrestripefs" then the GPFS config parameter "pitWorkerThreadsPerNode" need to be adjusted such that "sum of pitWorkerThreadsPerNode
of the participating nodes <= 31".</font><br class="">
<br class="">
<font size="2" face="sans-serif" class="">For example, if "number of nodes participating in the mmrestripefs" is 6 then adjust "mmchconfig pitWorkerThreadsPerNode=5 -N <participating_nodes>". GPFS would need to be restarted for this parameter to take effect
on the participating_nodes (verify with </font><font size="2" face="Courier New" class="">mmfsadm dump config | grep pitWorkerThreadsPerNode</font><font size="2" face="sans-serif" class="">)</font><br class="">
<br class="">
<font size="2" face="sans-serif" class="">Regards,</font><br class="">
<font size="2" face="sans-serif" class="">-Kums</font><br class="">
<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
<font size="1" color="#5f5f5f" face="sans-serif" class="">From: </font><font size="1" face="sans-serif" class="">"Buterbaugh, Kevin L" <<a href="mailto:Kevin.Buterbaugh@Vanderbilt.Edu" class="">Kevin.Buterbaugh@Vanderbilt.Edu</a>></font><br class="">
<font size="1" color="#5f5f5f" face="sans-serif" class="">To: </font><font size="1" face="sans-serif" class="">gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" class="">gpfsug-discuss@spectrumscale.org</a>></font><br class="">
<font size="1" color="#5f5f5f" face="sans-serif" class="">Date: </font><font size="1" face="sans-serif" class="">05/04/2017 12:08 PM</font><br class="">
<font size="1" color="#5f5f5f" face="sans-serif" class="">Subject: </font><font size="1" face="sans-serif" class="">Re: [gpfsug-discuss] Well, this is the pits...</font><br class="">
<font size="1" color="#5f5f5f" face="sans-serif" class="">Sent by: </font><font size="1" face="sans-serif" class=""><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" class="">gpfsug-discuss-bounces@spectrumscale.org</a></font><br class="">
<hr noshade="" class="">
<br class="">
<br class="">
<br class="">
<font size="3" class="">Hi Olaf, </font><br class="">
<br class="">
<font size="3" class="">I didn\u2019t touch pitWorkerThreadsPerNode \u2026 it was already zero.</font><br class="">
<br class="">
<font size="3" class="">I\u2019m running 4.2.2.3 on my GPFS servers (some clients are on 4.2.1.1 or 4.2.0.3 and are gradually being upgraded). What version of GPFS fixes this? With what I\u2019m doing I need the ability to run mmrestripefs.</font><br class="">
<br class="">
<font size="3" class="">It seems to me that mmrestripefs could check whether QOS is enabled \u2026 granted, it would have no way of knowing whether the values used actually are reasonable or not \u2026 but if QOS is enabled then \u201ctrust\u201d it to not overrun the system.</font><br class="">
<br class="">
<font size="3" class="">PMR time? Thanks..</font><br class="">
<br class="">
<font size="3" class="">Kevin</font><br class="">
<br class="">
<font size="3" class="">On May 4, 2017, at 10:54 AM, Olaf Weiser <</font><a href="mailto:olaf.weiser@de.ibm.com" class=""><font size="3" color="blue" class=""><u class="">olaf.weiser@de.ibm.com</u></font></a><font size="3" class="">> wrote:</font><br class="">
<br class="">
<font size="2" face="sans-serif" class="">HI Kevin, <br class="">
the number of NSDs is more or less nonsense .. it is just the number of nodes x PITWorker should not exceed to much the #mutex/FS block<br class="">
did you adjust/tune the PitWorker ? ... </font><font size="3" class=""><br class="">
</font><font size="2" face="sans-serif" class=""><br class="">
so far as I know.. that the code checks the number of NSDs is already considered as a defect and will be fixed / is already fixed ( I stepped into it here as well)
</font><font size="3" class=""><br class="">
</font><font size="2" face="sans-serif" class=""><br class="">
ps. QOS is the better approach to address this, but unfortunately.. not everyone is using it by default... that's why I suspect , the development decide to put in a check/limit here .. which in your case(with QOS) would'nt needed
</font><font size="3" class=""><br class="">
<br class="">
<br class="">
<br class="">
<br class="">
</font><font size="1" color="#5f5f5f" face="sans-serif" class=""><br class="">
From: </font><font size="1" face="sans-serif" class="">"Buterbaugh, Kevin L" <</font><a href="mailto:Kevin.Buterbaugh@Vanderbilt.Edu" class=""><font size="1" color="blue" face="sans-serif" class=""><u class="">Kevin.Buterbaugh@Vanderbilt.Edu</u></font></a><font size="1" face="sans-serif" class="">></font><font size="1" color="#5f5f5f" face="sans-serif" class=""><br class="">
To: </font><font size="1" face="sans-serif" class="">gpfsug main discussion list <</font><a href="mailto:gpfsug-discuss@spectrumscale.org" class=""><font size="1" color="blue" face="sans-serif" class=""><u class="">gpfsug-discuss@spectrumscale.org</u></font></a><font size="1" face="sans-serif" class="">></font><font size="1" color="#5f5f5f" face="sans-serif" class=""><br class="">
Date: </font><font size="1" face="sans-serif" class="">05/04/2017 05:44 PM</font><font size="1" color="#5f5f5f" face="sans-serif" class=""><br class="">
Subject: </font><font size="1" face="sans-serif" class="">Re: [gpfsug-discuss] Well, this is the pits...</font><font size="1" color="#5f5f5f" face="sans-serif" class=""><br class="">
Sent by: </font><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" class=""><font size="1" color="blue" face="sans-serif" class=""><u class="">gpfsug-discuss-bounces@spectrumscale.org</u></font></a><font size="3" class=""><br class="">
</font>
<hr noshade="" class="">
<font size="3" class=""><br class="">
<br class="">
<br class="">
Hi Olaf, <br class="">
<br class="">
Your explanation mostly makes sense, but...<br class="">
<br class="">
Failed with 4 nodes \u2026 failed with 2 nodes \u2026 not gonna try with 1 node. And this filesystem only has 32 disks, which I would imagine is not an especially large number compared to what some people reading this e-mail have in their filesystems.<br class="">
<br class="">
I thought that QOS (which I\u2019m using) was what would keep an mmrestripefs from overrunning the system \u2026 QOS has worked extremely well for us - it\u2019s one of my favorite additions to GPFS.<br class="">
<br class="">
Kevin<br class="">
<br class="">
On May 4, 2017, at 10:34 AM, Olaf Weiser <</font><a href="mailto:olaf.weiser@de.ibm.com" class=""><font size="3" color="blue" class=""><u class="">olaf.weiser@de.ibm.com</u></font></a><font size="3" class="">> wrote:<br class="">
</font><font size="2" face="sans-serif" class=""><br class="">
no.. it is just in the code, because we have to avoid to run out of mutexs / block<br class="">
<br class="">
reduce the number of nodes -N down to 4 (2nodes is even more safer) ... is the easiest way to solve it for now....<br class="">
<br class="">
I've been told the real root cause will be fixed in one of the next ptfs .. within this year ..
<br class="">
this warning messages itself should appear every time.. but unfortunately someone coded, that it depends on the number of disks (NSDs).. that's why I suspect you did'nt see it before<br class="">
but the fact , that we have to make sure, not to overrun the system by mmrestripe remains.. to please lower the -N number of nodes to 4 or better 2
<br class="">
<br class="">
(even though we know.. than the mmrestripe will take longer)</font><font size="3" class=""><br class="">
</font><font size="1" color="#5f5f5f" face="sans-serif" class=""><br class="">
<br class="">
From: </font><font size="1" face="sans-serif" class="">"Buterbaugh, Kevin L" <</font><a href="mailto:Kevin.Buterbaugh@Vanderbilt.Edu" class=""><font size="1" color="blue" face="sans-serif" class=""><u class="">Kevin.Buterbaugh@Vanderbilt.Edu</u></font></a><font size="1" face="sans-serif" class="">></font><font size="1" color="#5f5f5f" face="sans-serif" class=""><br class="">
To: </font><font size="1" face="sans-serif" class="">gpfsug main discussion list <</font><a href="mailto:gpfsug-discuss@spectrumscale.org" class=""><font size="1" color="blue" face="sans-serif" class=""><u class="">gpfsug-discuss@spectrumscale.org</u></font></a><font size="1" face="sans-serif" class="">></font><font size="1" color="#5f5f5f" face="sans-serif" class=""><br class="">
Date: </font><font size="1" face="sans-serif" class="">05/04/2017 05:26 PM</font><font size="1" color="#5f5f5f" face="sans-serif" class=""><br class="">
Subject: </font><font size="1" face="sans-serif" class="">[gpfsug-discuss] Well, this is the pits...</font><font size="1" color="#5f5f5f" face="sans-serif" class=""><br class="">
Sent by: </font><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" class=""><font size="1" color="blue" face="sans-serif" class=""><u class="">gpfsug-discuss-bounces@spectrumscale.org</u></font></a><font size="3" class=""><br class="">
</font>
<hr noshade="" class="">
<font size="3" class=""><br class="">
<br class="">
<br class="">
Hi All, <br class="">
<br class="">
Another one of those, \u201cI can open a PMR if I need to\u201d type questions\u2026<br class="">
<br class="">
We are in the process of combining two large GPFS filesystems into one new filesystem (for various reasons I won\u2019t get into here). Therefore, I\u2019m doing a lot of mmrestripe\u2019s, mmdeldisk\u2019s, and mmadddisk\u2019s.<br class="">
<br class="">
Yesterday I did an \u201cmmrestripefs <old fs> -r -N <my 8 NSD servers>\u201d (after suspending a disk, of course). Worked like it should.<br class="">
<br class="">
Today I did a \u201cmmrestripefs <new fs> -b -P capacity -N <those same 8 NSD servers>\u201d and got:<br class="">
<br class="">
mmrestripefs: The total number of PIT worker threads of all participating nodes has been exceeded to safely restripe the file system. The total number of PIT worker threads, which is the sum of pitWorkerThreadsPerNode of the participating nodes, cannot exceed
31. Reissue the command with a smaller set of participating nodes (-N option) and/or lower the pitWorkerThreadsPerNode configure setting. By default the file system manager node is counted as a participating node.<br class="">
mmrestripefs: Command failed. Examine previous error messages to determine cause.<br class="">
<br class="">
So there must be some difference in how the \u201c-r\u201d and \u201c-b\u201d options calculate the number of PIT worker threads. I did an \u201cmmfsadm dump all | grep pitWorkerThreadsPerNode\u201d on all 8 NSD servers and the filesystem manager node \u2026 they all say the same thing:<br class="">
<br class="">
pitWorkerThreadsPerNode 0<br class="">
<br class="">
Hmmm, so 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 > 31?!? I\u2019m confused...<br class="">
<br class="">
\u2014<br class="">
Kevin Buterbaugh - Senior System Administrator<br class="">
Vanderbilt University - Advanced Computing Center for Research and Education</font><font size="3" color="blue" class=""><u class=""><br class="">
</u></font><a href="mailto:Kevin.Buterbaugh@vanderbilt.edu" class=""><font size="3" color="blue" class=""><u class="">Kevin.Buterbaugh@vanderbilt.edu</u></font></a><font size="3" class="">- (615)875-9633<br class="">
</font><tt class=""><font size="2" class=""><br class="">
<br class="">
_______________________________________________<br class="">
gpfsug-discuss mailing list<br class="">
gpfsug-discuss at </font></tt><a href="http://spectrumscale.org/" class=""><tt class=""><font size="2" color="blue" class=""><u class="">spectrumscale.org</u></font></tt></a><font size="3" color="blue" class=""><u class=""><br class="">
</u></font><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" class=""><tt class=""><font size="2" color="blue" class=""><u class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></font></tt></a><font size="3" class=""><br class="">
<br class="">
<br class="">
<br class="">
_______________________________________________<br class="">
gpfsug-discuss mailing list<br class="">
gpfsug-discuss at </font><a href="http://spectrumscale.org/" class=""><font size="3" color="blue" class=""><u class="">spectrumscale.org</u></font></a><font size="3" color="blue" class=""><u class=""><br class="">
</u></font><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" class=""><font size="3" color="blue" class=""><u class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></font></a><tt class=""><font size="2" class=""><br class="">
_______________________________________________<br class="">
gpfsug-discuss mailing list<br class="">
gpfsug-discuss at </font></tt><a href="http://spectrumscale.org/" class=""><tt class=""><font size="2" color="blue" class=""><u class="">spectrumscale.org</u></font></tt></a><font size="3" color="blue" class=""><u class=""><br class="">
</u></font><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" class=""><tt class=""><font size="2" color="blue" class=""><u class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></font></tt></a><font size="3" class=""><br class="">
<br class="">
<br class="">
<br class="">
_______________________________________________<br class="">
gpfsug-discuss mailing list<br class="">
gpfsug-discuss at </font><a href="http://spectrumscale.org/" class=""><font size="3" color="blue" class=""><u class="">spectrumscale.org</u></font></a><font size="3" color="blue" class=""><u class=""><br class="">
</u></font><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" class=""><font size="3" color="blue" class=""><u class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></font></a><br class="">
<tt class=""><font size="2" class="">_______________________________________________<br class="">
gpfsug-discuss mailing list<br class="">
gpfsug-discuss at <a href="http://spectrumscale.org/" class="">spectrumscale.org</a><br class="">
</font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" class=""><tt class=""><font size="2" class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt class=""><font size="2" class=""><br class="">
</font></tt><br class="">
<br class="">
_______________________________________________<br class="">
gpfsug-discuss mailing list<br class="">
gpfsug-discuss at <a href="http://spectrumscale.org/" class="">spectrumscale.org</a><br class="">
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
_______________________________________________<br class="">
gpfsug-discuss mailing list<br class="">
gpfsug-discuss at <a href="http://spectrumscale.org/" class="">spectrumscale.org</a><br class="">
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>