<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
Hi Olaf,
<div class=""><br class="">
</div>
<div class="">Your explanation mostly makes sense, but...</div>
<div class=""><br class="">
</div>
<div class="">Failed with 4 nodes \u2026 failed with 2 nodes \u2026 not gonna try with 1 node. And this filesystem only has 32 disks, which I would imagine is not an especially large number compared to what some people reading this e-mail have in their filesystems.</div>
<div class=""><br class="">
</div>
<div class="">I thought that QOS (which I\u2019m using) was what would keep an mmrestripefs from overrunning the system \u2026 QOS has worked extremely well for us - it\u2019s one of my favorite additions to GPFS.</div>
<div class=""><br class="">
</div>
<div class="">Kevin</div>
<div class=""><br class="">
<div>
<blockquote type="cite" class="">
<div class="">On May 4, 2017, at 10:34 AM, Olaf Weiser <<a href="mailto:olaf.weiser@de.ibm.com" class="">olaf.weiser@de.ibm.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class=""><font size="2" face="sans-serif" class="">no.. it is just in the code, because we have to avoid to run out of mutexs / block</font><br class="">
<br class="">
<font size="2" face="sans-serif" class="">reduce the number of nodes -N down to 4 (2nodes is even more safer) ... is the easiest way to solve it for now....</font><br class="">
<br class="">
<font size="2" face="sans-serif" class="">I've been told the real root cause will be fixed in one of the next ptfs .. within this year ..
</font><br class="">
<div class=""><font size="2" face="sans-serif" class="">this warning messages itself should appear every time.. but unfortunately someone coded, that it depends on the number of disks (NSDs).. that's why I suspect you did'nt see it before</font><br class="">
<font size="2" face="sans-serif" class="">but the fact , that we have to make sure, not to overrun the system by mmrestripe remains.. to please lower the -N number of nodes to 4 or better 2
</font><br class="">
<br class="">
<font size="2" face="sans-serif" class="">(even though we know.. than the mmrestripe will take longer)</font><br class="">
<br class="">
<br class="">
<font size="1" color="#5f5f5f" face="sans-serif" class="">From: </font><font size="1" face="sans-serif" class="">"Buterbaugh, Kevin L" <<a href="mailto:Kevin.Buterbaugh@Vanderbilt.Edu" class="">Kevin.Buterbaugh@Vanderbilt.Edu</a>></font><br class="">
<font size="1" color="#5f5f5f" face="sans-serif" class="">To: </font><font size="1" face="sans-serif" class="">gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" class="">gpfsug-discuss@spectrumscale.org</a>></font><br class="">
<font size="1" color="#5f5f5f" face="sans-serif" class="">Date: </font><font size="1" face="sans-serif" class="">05/04/2017 05:26 PM</font><br class="">
<font size="1" color="#5f5f5f" face="sans-serif" class="">Subject: </font><font size="1" face="sans-serif" class="">[gpfsug-discuss] Well, this is the pits...</font><br class="">
<font size="1" color="#5f5f5f" face="sans-serif" class="">Sent by: </font><font size="1" face="sans-serif" class=""><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" class="">gpfsug-discuss-bounces@spectrumscale.org</a></font><br class="">
<hr noshade="" class="">
<br class="">
<br class="">
<br class="">
<font size="3" class="">Hi All, </font><br class="">
<br class="">
<font size="3" class="">Another one of those, \u201cI can open a PMR if I need to\u201d type questions\u2026</font><br class="">
<br class="">
<font size="3" class="">We are in the process of combining two large GPFS filesystems into one new filesystem (for various reasons I won\u2019t get into here). Therefore, I\u2019m doing a lot of mmrestripe\u2019s, mmdeldisk\u2019s, and mmadddisk\u2019s.</font><br class="">
<br class="">
<font size="3" class="">Yesterday I did an \u201cmmrestripefs <old fs> -r -N <my 8 NSD servers>\u201d (after suspending a disk, of course). Worked like it should.</font><br class="">
<br class="">
<font size="3" class="">Today I did a \u201cmmrestripefs <new fs> -b -P capacity -N <those same 8 NSD servers>\u201d and got:</font><br class="">
<br class="">
<font size="3" class="">mmrestripefs: The total number of PIT worker threads of all participating nodes has been exceeded to safely restripe the file system. The total number of PIT worker threads, which is the sum of pitWorkerThreadsPerNode of the participating
nodes, cannot exceed 31. Reissue the command with a smaller set of participating nodes (-N option) and/or lower the pitWorkerThreadsPerNode configure setting. By default the file system manager node is counted as a participating node.</font><br class="">
<font size="3" class="">mmrestripefs: Command failed. Examine previous error messages to determine cause.</font><br class="">
<br class="">
<font size="3" class="">So there must be some difference in how the \u201c-r\u201d and \u201c-b\u201d options calculate the number of PIT worker threads. I did an \u201cmmfsadm dump all | grep pitWorkerThreadsPerNode\u201d on all 8 NSD servers and the filesystem manager node \u2026 they all
say the same thing:</font><br class="">
<br class="">
<font size="3" class=""> pitWorkerThreadsPerNode 0</font><br class="">
<br class="">
<font size="3" class="">Hmmm, so 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 > 31?!? I\u2019m confused...</font><br class="">
<br class="">
<font size="3" class="">\u2014</font><br class="">
<font size="3" class="">Kevin Buterbaugh - Senior System Administrator</font><br class="">
<font size="3" class="">Vanderbilt University - Advanced Computing Center for Research and Education</font><br class="">
<a href="mailto:Kevin.Buterbaugh@vanderbilt.edu" class=""><font size="3" color="blue" class=""><u class="">Kevin.Buterbaugh@vanderbilt.edu</u></font></a><font size="3" class="">- (615)875-9633</font><br class="">
<br class="">
<br class="">
<tt class=""><font size="2" class="">_______________________________________________<br class="">
gpfsug-discuss mailing list<br class="">
gpfsug-discuss at <a href="http://spectrumscale.org" class="">spectrumscale.org</a><br class="">
</font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" class=""><tt class=""><font size="2" class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt class=""><font size="2" class=""><br class="">
</font></tt><br class="">
<br class="">
</div>
<br class="">
_______________________________________________<br class="">
gpfsug-discuss mailing list<br class="">
gpfsug-discuss at <a href="http://spectrumscale.org" class="">spectrumscale.org</a><br class="">
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>