<html><body><p><tt>> to the question. If I were to hypothetically use a 256K metadata <br>> block size, does the \u201c1/32nd of a block\u201d come into play like it does<br>> for \u201cnot metadata\u201d? I.e. 256 / 32 = 8K, so am I reading / writing <br>> *2* inodes (assuming 4K inode size) minimum?</tt><br><br><tt>I think the point of confusion here is minimum allocation size vs minimum IO size -- those two are not one and the same. In fact in GPFS those are largely unrelated values. For low-level metadata files where multiple records are packed into the same block, it is possible to read/write either an individual record (such as an inode), or an entire block of records (which is what happens, for example, during inode copy-on-write). The minimum IO size in GPFS is 512 bytes. On a "4K-aligned" file system, GPFS vows to only do IOs in multiples of 4KiB. For data, GPFS tracks what portion of a given block is valid/dirty using an in-memory bitmap, and if 4K in the middle of a 16M block are modified, only 4K get written, not 16M (although this is more complicated for sparse file writes and appends, when some areas need to be zeroed out). For metadata writes, entire metadata objects are written, using the actual object size, rounded up to the nearest 512B or 4K boundary, as needed. </tt><br><br><tt>So a single modified inode results in a single inode write, regardless of the metadata block size. If you have snapshots, and the inode being modified needs to be copied to the previous snapshot, and happens to be the first inode in the block that needs a COW, an entire block of inodes is copied to the latest snapshot, as an optimization.</tt><br><br><tt>> And here\u2019s a really off the wall question \u2026 yesterday we were <br>> discussing the fact that there is now a single inode file. <br>> Historically, we have always used RAID 1 mirrors (first with <br>> spinning disk, as of last fall now on SSD) for metadata and then use<br>> GPFS replication on top of that. But given that there is a single <br>> inode file is that \u201cold way\u201d of doing things still the right way? <br>> In other words, could we potentially be better off by using a couple<br>> of 8+2P RAID 6 LUNs?</tt><br><br><tt>The old way is also the modern way in this case. Using RAID1 LUNs for GPFS metadata is still the right approach. You don't want to use RAID erasure codes that trigger read-modify-write for small IOs, which are typical for metadata (unless your RAID array has so much cache as to make RMW a moot point).</tt><br><br><tt>> One potential downside of that would be that we would then only have<br>> two NSD servers serving up metadata, so we discussed the idea of <br>> taking each RAID 6 LUN and splitting it up into multiple logical <br>> volumes (all that done on the storage array, of course) and then <br>> presenting those to GPFS as NSDs???</tt><br><br><tt>Like most performance questions, this one can ultimately only be answered definitively by running tests, but offhand I would suspect that the performance impact of RAID6, combined with extra contention for physical disks, is going to more than offset the benefits of using more NSD servers. Keep in mind that you aren't limited to 2 NSD servers per LUN. If you actually have the connectivity for more than 2 nodes on your RAID controller, GPFS allows up to 8 simultaneously active NSD servers per NSD.</tt><br><br><tt>yuri</tt><br><tt> <br>> On Sep 28, 2016, at 10:23 AM, Marc A Kaplan <makaplan@us.ibm.com> wrote:</tt><br><tt>> <br>> OKAY, I'll say it again. inodes are PACKED into a single inode <br>> file. So a 4KB inode takes 4KB, REGARDLESS of metadata blocksize. <br>> There is no wasted space.<br>> <br>> (Of course if you have metadata replication = 2, then yes, double <br>> that. And yes, there overhead for indirect blocks (indices), <br>> allocation maps, etc, etc.)<br>> <br>> And your choice is not just 512 or 4096. Maybe 1KB or 2KB is a good<br>> choice for your data distribution, to optimize packing of data and/<br>> or directories into inodes...<br>> <br>> Hmmm... I don't know why the doc leaves out 2048, perhaps a typo...<br>> <br>> mmcrfs x2K -i 2048<br>> <br>> [root@n2 charts]# mmlsfs x2K -i<br>> flag value description<br>> ------------------- ------------------------ <br>> -----------------------------------<br>> -i 2048 Inode size in bytes<br>> <br>> Works for me!<br>> _______________________________________________<br>> gpfsug-discuss mailing list<br>> gpfsug-discuss at spectrumscale.org<br>> <a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></tt><br><tt>> <br>> <br></tt><br><tt>> _______________________________________________<br>> gpfsug-discuss mailing list<br>> gpfsug-discuss at spectrumscale.org<br>> <a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br></tt><BR>
</body></html>