<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Hi All,
<div class=""><br class="">
</div>
<div class="">So I was looking at the presentation referenced below and it states - on multiple slides - that there is one system storage pool per cluster. &nbsp;Really? &nbsp;Shouldn\u2019t that be one system storage pool per filesystem?!? &nbsp;If not, please explain how in
 my GPFS cluster with two (local) filesystems I see two different system pools with two different sets of NSDs, two different capacities, and two different percentages full???</div>
<div class=""><br class="">
</div>
<div class="">Thanks\u2026</div>
<div class=""><br class="">
</div>
<div class="">Kevin</div>
<div class=""><br class="">
<div class="">
<div style="color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div class="">\u2014</div>
<div class="">Kevin Buterbaugh - Senior System Administrator</div>
<div class="">Vanderbilt University - Advanced Computing Center for Research and Education</div>
<div class=""><a href="mailto:Kevin.Buterbaugh@vanderbilt.edu" class="">Kevin.Buterbaugh@vanderbilt.edu</a>&nbsp;- (615)875-9633</div>
</div>
</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Mar 26, 2019, at 11:27 AM, Dorigo Alvise (PSI) &lt;<a href="mailto:alvise.dorigo@psi.ch" class="">alvise.dorigo@psi.ch</a>&gt; wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; direction: ltr; font-family: Tahoma; font-size: 10pt;" class="">
<div class="">Hi Marc,<br class="">
</div>
<div class="">&quot;Indirect block size&quot; is well explained in this presentation:<span class="Apple-converted-space">&nbsp;</span><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><a href="https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Ffiles.gpfsug.org%2Fpresentations%2F2016%2Fsouth-bank%2FD2_P2_A_spectrum_scale_metadata_dark_V2a.pdf&amp;data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C5b28a9a0d39a47fd3f0608d6b208186a%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C1%7C636892145179816620&amp;sdata=SDAw4EXR1jNDYZRdIPtNVuhR1ZfBiUFcF7%2FCrprNwag%3D&amp;reserved=0" originalsrc="http://files.gpfsug.org/presentations/2016/south-bank/D2_P2_A_spectrum_scale_metadata_dark_V2a.pdf" shash="cTgclma2BYLWO8&#43;U2OSNl3mbJ5gHwSxskH4XL3Lqw8okCCRbJRsIrI5w53SGaJlxdkanrAi4R1qxnF9aBK4wdjpGYHlyNuxUVXQomC6uQFY0wjEbeX/WlaShhvNApZMyNeiCd9IVdFvOdm7PuYcQfnQNLksrS1XVpXwlAIAZl1U=" target="_blank" class="">http://files.gpfsug.org/presentations/2016/south-bank/D2_P2_A_spectrum_scale_metadata_dark_V2a.pdf</a></div>
<div class=""><br class="">
</div>
<div class="">pages 37-41<br class="">
</div>
<div class=""><br class="">
</div>
<div class="">Cheers,</div>
<div class=""><br class="">
</div>
<div class="">&nbsp;&nbsp; Alvise<br class="">
</div>
<div class=""><br class="">
</div>
<div style="font-family: &quot;Times New Roman&quot;; font-size: 16px;" class="">
<hr tabindex="-1" class="">
<div id="divRpF933280" style="direction: ltr;" class=""><font size="2" face="Tahoma" class=""><b class="">From:</b><span class="Apple-converted-space">&nbsp;</span><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" class="">gpfsug-discuss-bounces@spectrumscale.org</a><span class="Apple-converted-space">&nbsp;</span>[<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" class="">gpfsug-discuss-bounces@spectrumscale.org</a>]
 on behalf of Caubet Serrabou Marc (PSI) [<a href="mailto:marc.caubet@psi.ch" class="">marc.caubet@psi.ch</a>]<br class="">
<b class="">Sent:</b><span class="Apple-converted-space">&nbsp;</span>Tuesday, March 26, 2019 4:39 PM<br class="">
<b class="">To:</b><span class="Apple-converted-space">&nbsp;</span>gpfsug main discussion list<br class="">
<b class="">Subject:</b><span class="Apple-converted-space">&nbsp;</span>[gpfsug-discuss] GPFS v5: Blocksizes and subblocks<br class="">
</font><br class="">
</div>
<div class=""></div>
<div class="">
<div style="direction: ltr; font-family: Tahoma; font-size: 10pt;" class="">
<div class="">Hi all,</div>
<div class=""><br class="">
</div>
<div class="">according to several GPFS presentations as well as according to the man pages:</div>
<div class=""><br class="">
</div>
<div class=""><font face="Courier New" class="">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Table 1. Block sizes and subblock sizes<br class="">
<br class="">
&#43;-------------------------------&#43;-------------------------------&#43;<br class="">
| Block size&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; | Subblock size&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; |<br class="">
&#43;-------------------------------&#43;-------------------------------&#43;<br class="">
| 64 KiB&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; | 2 KiB&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; |<br class="">
&#43;-------------------------------&#43;-------------------------------&#43;<br class="">
| 128 KiB&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; | 4 KiB&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; |<br class="">
&#43;-------------------------------&#43;-------------------------------&#43;<br class="">
| 256 KiB, 512 KiB, 1 MiB, 2&nbsp;&nbsp;&nbsp; | 8 KiB&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; |<br class="">
| MiB, 4 MiB&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; |&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; |<br class="">
&#43;-------------------------------&#43;-------------------------------&#43;<br class="">
| 8 MiB, 16 MiB&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; | 16 KiB&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; |<br class="">
&#43;-------------------------------&#43;-------------------------------&#43;</font><br class="">
<br class="">
</div>
<div class="">A block size of 8MiB or 16MiB should contain subblocks of 16KiB.</div>
<div class=""><br class="">
</div>
<div class="">However, when creating a new filesystem with 16MiB blocksize, looks like is using 128KiB subblocks:</div>
<div class=""><br class="">
</div>
<div class=""><font face="Courier New" class="">[root@merlindssio01 ~]# mmlsfs merlin<br class="">
flag&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; value&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; description<br class="">
------------------- ------------------------ -----------------------------------<br class="">
&nbsp;-f&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 8192&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Minimum fragment (subblock) size in bytes (system pool)<br class="">
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 131072&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Minimum fragment (subblock) size in bytes (other pools)<br class="">
&nbsp;-i&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 4096&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Inode size in bytes<br class="">
&nbsp;-I&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 32768&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Indirect block size in bytes<br class="">
.</font></div>
<div class=""><font face="Courier New" class="">.</font></div>
<div class=""><font face="Courier New" class="">.<br class="">
</font></div>
<div class=""><font face="Courier New" class="">&nbsp;-n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 128&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Estimated number of nodes that will mount file system<br class="">
&nbsp;-B&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1048576&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Block size (system pool)<br class="">
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 16777216&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Block size (other pools)<br class="">
.</font></div>
<div class=""><font face="Courier New" class="">.</font></div>
<div class=""><font face="Courier New" class="">.</font><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">What am I missing? According to documentation, I expect this to be a fixed value, or it isn't at all?<br class="">
</div>
<div class=""><br class="">
</div>
<div class="">On the other hand, I don't really understand the concept 'Indirect block size in bytes', can somebody clarify or provide some details about this setting?</div>
<div class=""><br class="">
</div>
<div class="">Thanks a lot and best regards,</div>
<div class="">Marc&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="Apple-converted-space">&nbsp;</span><br class="">
<div style="font-family: Tahoma; font-size: 13px;" class="">
<div style="font-family: Tahoma; font-size: 13px;" class="">
<div class="BodyFragment"><font size="2" class=""><span style="font-size: 10pt;" class="">
<div class="PlainText">_________________________________________<br class="">
Paul Scherrer Institut<span class="Apple-converted-space">&nbsp;</span><br class="">
High Performance Computing<br class="">
Marc Caubet Serrabou<br class="">
Building/Room: WHGA/019A</div>
<div class="PlainText"><font size="2" class=""><span style="font-size: 10pt;" class="">Forschungsstrasse, 111</span></font></div>
<div class="PlainText">5232 Villigen PSI<br class="">
Switzerland<br class="">
<br class="">
Telephone: &#43;41 56 310 46 67<br class="">
E-Mail:<span class="Apple-converted-space">&nbsp;</span><a href="mailto:marc.caubet@psi.ch" class="">marc.caubet@psi.ch</a></div>
</span></font></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">_______________________________________________</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">gpfsug-discuss
 mailing list</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">gpfsug-discuss
 at<span class="Apple-converted-space">&nbsp;</span></span><a href="http://spectrumscale.org/" style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">spectrumscale.org</a><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<a href="https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&amp;amp;data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C5b28a9a0d39a47fd3f0608d6b208186a%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C1%7C636892145179836634&amp;amp;sdata=23F22sUiyCYEg0H3AdbkBAnhPpLVBVTh39zRr%2FLYCmc%3D&amp;amp;reserved=0" style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&amp;amp;data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C5b28a9a0d39a47fd3f0608d6b208186a%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C1%7C636892145179836634&amp;amp;sdata=23F22sUiyCYEg0H3AdbkBAnhPpLVBVTh39zRr%2FLYCmc%3D&amp;amp;reserved=0</a></div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>