<html><body><p>your metadata block size these days should be 1 MB and there are only very few workloads for which you should run with a filesystem blocksize below 1 MB. so if you don't know exactly what to pick, 1 MB is a good starting point. <br>the general rule still applies that your filesystem blocksize (metadata or data pool) should match your raid controller (or GNR vdisk) stripe size of the particular pool.<br><br>so if you use a 128k strip size(defaut in many midrange storage controllers) in a 8+2p raid array, your stripe or track size is 1 MB and therefore the blocksize of this pool should be 1 MB. i see many customers in the field using 1MB or even smaller blocksize on RAID stripes of 2 MB or above and your performance will be significant impacted by that. <br><br>Sven<br><br>------------------------------------------<br>Sven Oehme <br>Scalable Storage Research <br>email: oehmes@us.ibm.com <br>Phone: +1 (408) 824-8904 <br>IBM Almaden Research Lab <br>------------------------------------------<br><br><img width="16" height="16" src="cid:1__=07BB0AA4DFE625828f9e8a93df938690918c07B@" border="0" alt="Inactive hide details for Stephen Ulmer ---09/23/2016 12:16:34 PM---Not to be too pedantic, but I believe the the subblock size"><font color="#424282">Stephen Ulmer ---09/23/2016 12:16:34 PM---Not to be too pedantic, but I believe the the subblock size is 1/32 of the block size (which strengt</font><br><br><font size="2" color="#5F5F5F">From: </font><font size="2">Stephen Ulmer <ulmer@ulmer.org></font><br><font size="2" color="#5F5F5F">To: </font><font size="2">gpfsug main discussion list <gpfsug-discuss@spectrumscale.org></font><br><font size="2" color="#5F5F5F">Date: </font><font size="2">09/23/2016 12:16 PM</font><br><font size="2" color="#5F5F5F">Subject: </font><font size="2">Re: [gpfsug-discuss] Blocksize</font><br><font size="2" color="#5F5F5F">Sent by: </font><font size="2">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr width="100%" size="2" align="left" noshade style="color:#8091A5; "><br><br><br><font size="4">Not to be too pedantic, but I believe the the subblock size is 1/32 of the block size (which strengthens Luis\u2019s arguments below).</font><br><br><font size="4">I thought the the original question was NOT about inode size, but about metadata block size. You can specify that the system pool have a different block size from the rest of the filesystem, providing that it ONLY holds metadata (\u2014metadata-block-size option to mmcrfs).</font><br><br><font size="4">So with 4K inodes (which should be used for all new filesystems without some counter-indication), I would think that we\u2019d want to use a metadata block size of 4K*32=128K. This is independent of the regular block size, which you can calculate based on the workload if you\u2019re lucky.</font><br><br><font size="4">There could be a great reason NOT to use 128K metadata block size, but I don\u2019t know what it is. I\u2019d be happy to be corrected about this if it\u2019s out of whack.</font><br><br><font size="4">-- <br>Stephen<br><br></font><br>
<ul><ul><font size="4">On Sep 22, 2016, at 3:37 PM, Luis Bolinches <</font><a href="mailto:luis.bolinches@fi.ibm.com"><u><font size="4" color="#0000FF">luis.bolinches@fi.ibm.com</font></u></a><font size="4">> wrote:</font><br><br><font face="Arial">Hi</font><br><font face="Arial"> </font><br><font face="Arial">My 2 cents.</font><br><font face="Arial"> </font><br><font face="Arial">Leave at least 4K inodes, then you get massive improvement on small files (less 3.5K minus whatever you use on xattr)</font><br><font face="Arial"> </font><br><font face="Arial">About blocksize for data, unless you have actual data that suggest that you will actually benefit from smaller than 1MB block, leave there. GPFS uses sublocks where 1/16th of the BS can be allocated to different files, so the "waste" is much less than you think on 1MB and you get the throughput and less structures of much more data blocks.</font><br><font face="Arial"> </font><br><font face="Arial">No</font><b><font face="Arial"> warranty at all</font></b><font face="Arial"> but I try to do this when the BS talk comes in: (might need some clean up it could not be last note but you get the idea)</font><br><font face="Arial"> </font><br><font face="Arial">POSIX<br>find . -type f -name '*' -exec ls -l {} \; > find_ls_files.out</font><br><font face="Arial">GPFS<br>cd /usr/lpp/mmfs/samples/ilm<br>gcc mmfindUtil_processOutputFile.c -o mmfindUtil_processOutputFile<br>./mmfind /gpfs/shared -ls -type f > find_ls_files.out</font><br><font face="Arial"> CONVERT to CSV</font><br><font face="Arial"><br>POSIX<br>cat find_ls_files.out | awk '{print $5","}' > find_ls_files.out.csv<br>GPFS<br>cat find_ls_files.out | awk '{print $7","}' > find_ls_files.out.csv</font><br><font face="Arial"> LOAD in octave</font><br><font face="Arial"><br>FILESIZE = int32 (dlmread ("find_ls_files.out.csv", ","));</font><br><font face="Arial"> Clean the second column (OPTIONAL as the next clean up will do the same)</font><br><font face="Arial"><br>FILESIZE(:,[2]) = [];</font><br><font face="Arial"> If we are on 4K aligment we need to clean the files that go to inodes (WELL not exactly ... extended attributes! so maybe use a lower number!)</font><br><font face="Arial"><br>FILESIZE(FILESIZE<=3584) =[];</font><br><font face="Arial"> If we are not we need to clean the 0 size files</font><br><font face="Arial"><br>FILESIZE(FILESIZE==0) =[];</font><br><font face="Arial"> Median</font><br><font face="Arial"><br>FILESIZEMEDIAN = int32 (median (FILESIZE))</font><br><font face="Arial"> Mean</font><br><font face="Arial"><br>FILESIZEMEAN = int32 (mean (FILESIZE))</font><br><font face="Arial"> Variance</font><br><font face="Arial"><br>int32 (var (FILESIZE))</font><br><font face="Arial"> iqr interquartile range, i.e., the difference between the upper and lower quartile, of the input data.</font><br><font face="Arial"><br>int32 (iqr (FILESIZE))</font><br><font face="Arial"> Standard deviation</font><br><font face="Arial"> </font><br><font face="Arial"> </font><br><font face="Arial">For some FS with lots of files you might need a rather powerful machine to run the calculations on octave, I never hit anything could not manage on a 64GB RAM Power box. Most of the times it is enough with my laptop.</font><br><font face="Arial"> </font><br><font face="Arial"> </font><br><font face="Arial"><br>--<br>Ystävällisin terveisin / Kind regards / Saludos cordiales / Salutations<br><br>Luis Bolinches<br>Lab Services</font><u><font color="#0000FF" face="Arial"><br></font></u><a href="http://www-03.ibm.com/systems/services/labservices/"><u><font color="#0000FF" face="Arial">http://www-03.ibm.com/systems/services/labservices/</font></u></a><font face="Arial"><br><br>IBM Laajalahdentie 23 (main Entrance) Helsinki, 00330 Finland<br>Phone: +358 503112585<br><br>"If you continually give you will continually have." Anonymous</font><br><font face="Arial"> </font><br><font face="Arial"> </font><br><font face="Arial">----- Original message -----<br>From: Stef Coene <</font><a href="mailto:stef.coene@docum.org"><u><font color="#0000FF" face="Arial">stef.coene@docum.org</font></u></a><font face="Arial">><br>Sent by: </font><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org"><u><font color="#0000FF" face="Arial">gpfsug-discuss-bounces@spectrumscale.org</font></u></a><font face="Arial"><br>To: gpfsug main discussion list <</font><a href="mailto:gpfsug-discuss@spectrumscale.org"><u><font color="#0000FF" face="Arial">gpfsug-discuss@spectrumscale.org</font></u></a><font face="Arial">><br>Cc:<br>Subject: Re: [gpfsug-discuss] Blocksize<br>Date: Thu, Sep 22, 2016 10:30 PM<br> </font><br><tt>On 09/22/2016 09:07 PM, J. Eric Wonderley wrote:<br>> It defaults to 4k:<br>> mmlsfs testbs8M -i<br>> flag value description<br>> ------------------- ------------------------<br>> -----------------------------------<br>> -i 4096 Inode size in bytes<br>><br>> I think you can make as small as 512b. Gpfs will store very small<br>> files in the inode.<br>><br>> Typically you want your average file size to be your blocksize and your<br>> filesystem has one blocksize and one inodesize.<br><br>The files are not small, but around 20 MB on average.<br>So I calculated with IBM that a 1 MB or 2 MB block size is best.<br><br>But I'm not sure if it's better to use a smaller block size for the<br>metadata.<br><br>The file system is not that large (400 TB) and will hold backup data<br>from CommVault.<br><br><br>Stef<br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </tt><a href="http://spectrumscale.org/"><tt><u><font color="#0000FF">spectrumscale.org</font></u></tt></a><tt><u><font color="#0000FF"><br></font></u></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank"><tt><u><font color="#0000FF">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></u></tt></a><font face="Arial"><br> </font><br><font face="Arial"> </font><br><font size="4"><br>Ellei edellä ole toisin mainittu: / Unless stated otherwise above:<br>Oy IBM Finland Ab<br>PL 265, 00101 Helsinki, Finland<br>Business ID, Y-tunnus: 0195876-3 <br>Registered in Finland<br><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </font><a href="http://spectrumscale.org/"><u><font size="4" color="#0000FF">spectrumscale.org</font></u></a><u><font size="4" color="#0000FF"><br></font></u><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><u><font size="4" color="#0000FF">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></u></a></ul></ul><tt>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></tt><tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></tt><tt><br></tt><br><br><BR>
</body></html>