<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Sep 5, 2018, at 11:34 AM, Buterbaugh, Kevin L <<a href="mailto:Kevin.Buterbaugh@Vanderbilt.Edu" class="">Kevin.Buterbaugh@Vanderbilt.Edu</a>> wrote:</div><br class="Apple-interchange-newline"><div class="">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><br class=""></div></div></div></blockquote><div><br class=""></div>[\u2026]</div><div><br class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class="">
</div>
<div class="">Of course, if we increase the size of the inodes by a factor of 8 then we also need 8 times as much space to store those inodes. Given that Enterprise class SSDs are still very expensive and our budget is not unlimited, we\u2019re trying to get the
best bang for the buck.</div>
<div class=""><br class=""></div></div></div></blockquote><div><br class=""></div><div>Nobody has gone in this direction yet, so I\u2019ll play devil\u2019s advocate:</div><div><br class=""></div><div>Are you sure you need enterprise class SSDs? The only practical difference between enterprise class SSDs and "read intensive" SSDs is the "endurance" in DWPD[1]. Read-intensive SSDs usually have a DWPD of 1-ish. Enterprise SSDs can have a DWPD as high as 30.</div><div><br class=""></div><div>So, how many times do you think you\u2019ll actually write all of the data on the SSDs per day?</div><div><br class=""></div><div>I don\u2019t know how much (meta)data you\u2019ve got, but maybe consider buying the "cheap" SSDs (which will be *much* larger for your dollar) and just use fractions of them with GPFS replication[2] or maybe some vendor\u2019s {distributed, de-clustererd} RAID. Keep some spares.</div><div><br class=""></div><div>This is probably bad advice, but the thought exercise will let you find the edges of what you meant. :)</div><div><br class=""></div><div>[1] DWPD = Drive Writes Per Day \u2014 write all of the cells on the entire storage device every 24 hours.</div><div>[2] Okay, somebody already said to use GPFS replication. ;)</div><div><br class=""></div><div><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">-- <br class="">Stephen<br class=""><br class=""><br class=""></div></div></div><br class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class="">
</div>
<div class="">We have always - even back in the day when our metadata was on spinning disk and not SSD - used RAID 1 mirrors and metadata replication of two. However, we are wondering if it might be possible to switch to RAID 5? Specifically, what we are
considering doing is buying 8 new SSDs and creating two 3+1P RAID 5 LUNs (metadata replication would stay at two). That would give us 50% more usable space than if we configured those same 8 drives as four RAID 1 mirrors.</div>
<div class=""><br class="">
</div>
<div class="">Unfortunately, unless I\u2019m misunderstanding something, mean that the RAID stripe size and the GPFS block size could not match. Therefore, even though we don\u2019t need the space, would we be much better off to buy 10 SSDs and create two 4+1P RAID
5 LUNs?</div>
<div class=""><br class="">
</div>
<div class="">I\u2019ve searched the mailing list archives and scanned the DeveloperWorks wiki and even glanced at the GPFS documentation and haven\u2019t found anything that says \u201cbad idea, Kevin\u201d\u2026 ;-)</div>
<div class=""><br class="">
</div>
<div class="">Expanding on this further \u2026 if we just present those two RAID 5 LUNs to GPFS as NSDs then we can only have two NSD servers as primary for them. So another thing we\u2019re considering is to take those RAID 5 LUNs and further sub-divide them into a
total of 8 logical volumes, each of which could be a GPFS NSD and therefore would allow us to have each of our 8 NSD servers be primary for one of them. Even worse idea?!? Good idea?</div>
<div class=""><br class="">
</div>
<div class="">Anybody have any better ideas??? ;-)</div>
<div class=""><br class="">
</div>
<div class="">Oh, and currently we\u2019re on GPFS 4.2.3-10, but are also planning on moving to GPFS 5.0.1-x before creating the new filesystem.</div>
<div class=""><br class="">
</div>
<div class="">Thanks much\u2026</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">
<div class="">\u2014</div>
<div class="">Kevin Buterbaugh - Senior System Administrator</div>
<div class="">Vanderbilt University - Advanced Computing Center for Research and Education</div>
<div class=""><a href="mailto:Kevin.Buterbaugh@vanderbilt.edu" class="">Kevin.Buterbaugh@vanderbilt.edu</a> - (615)875-9633</div>
<div class=""><br class="">
</div>
<br class="Apple-interchange-newline">
</div>
<br class="">
</div>
</div>
_______________________________________________<br class="">gpfsug-discuss mailing list<br class="">gpfsug-discuss at <a href="http://spectrumscale.org" class="">spectrumscale.org</a><br class=""><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class=""></div></blockquote></div><br class=""></body></html>