<div dir="ltr"><div dir="ltr"><div dir="ltr"><div>I don't think it's going to figure this out automatically between the two rules.. I believe you will need to do something like the below (untested, and definitely not perfect!!) rebalancing:</div><div><br></div><div>define(<br> weight_junkedness,<br> (CASE<br> /* Create 3 classes of files */<br> /* ReallyOldStuff */<br> WHEN ((access_age > 365) AND (KB_ALLOCATED > 3584)) THEN 1000 <br> /* OldStuff */<br> WHEN ((access_age > 90) AND (KB_ALLOCATED > 3584)) THEN 100<br> /* everything else */<br> ELSE 0<br> END)<br>)<br><br>RULE 'defineTiers' GROUP POOL 'TIERS'<br> IS 'data' LIMIT(98)<br> THEN 'capacity' LIMIT(98)<br> THEN 'oc'<br><br>RULE 'Rebalance' MIGRATE FROM POOL 'TIERS' TO POOL 'TIERS' WEIGHT(weight_junkedness)<br></div><div><br></div><div>Based on /usr/lpp/mmfs/samples/ilm/mmpolicy-fileheat.sample.<br></div><div><br></div><div><br></div><div> -jf<br></div></div></div></div><br><div class="gmail_quote"><div dir="ltr">On Tue, Dec 18, 2018 at 9:10 AM Buterbaugh, Kevin L <<a href="mailto:Kevin.Buterbaugh@vanderbilt.edu">Kevin.Buterbaugh@vanderbilt.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div style="overflow-wrap: break-word;">
Hi All,
<div><br>
</div>
<div>As those of you who suffered thru my talk at SC18 already know, we\u2019re really short on space on one of our GPFS filesystems as the output of mmdf piped to grep pool shows:</div>
<div><br>
</div>
<div>
<div>Disks in storage pool: system (Maximum disk size allowed is 24 TB)</div>
<div>(pool total) 4.318T 1.078T ( 25%) 79.47G ( 2%)</div>
<div>Disks in storage pool: data (Maximum disk size allowed is 262 TB)</div>
<div>(pool total) 494.7T 38.15T ( 8%) 4.136T ( 1%)</div>
<div>Disks in storage pool: capacity (Maximum disk size allowed is 519 TB)</div>
<div>(pool total) 640.2T 14.56T ( 2%) 716.4G ( 0%)</div>
<div><br>
</div>
The system pool is metadata only. The data pool is the default pool. The capacity pool is where files with an atime (yes, atime) > 90 days get migrated. The capacity pool is comprised of NSDs that are 8+2P RAID 6 LUNs of 8 TB drives, so roughly 58.2 TB usable
space per NSD.</div>
<div><br>
</div>
<div>We have the new storage we purchased, but that\u2019s still being tested and held in reserve for after the first of the year when we create a new GPFS 5 formatted filesystem and start migrating everything to the new filesystem.</div>
<div><br>
</div>
<div>In the meantime, we have also purchased a 60-bay JBOD and 30 x 12 TB drives and will be hooking it up to one of our existing storage arrays on Wednesday. My plan is to create another 3 8+2P RAID 6 LUNs and present those to GPFS as NSDs. They
will be about 88 TB usable space each (because \u2026 beginning rant \u2026 a 12 TB drive is < 11 TB is size \u2026 and don\u2019t get me started on so-called \u201c4K\u201d TV\u2019s \u2026 end rant).</div>
<div><br>
</div>
<div>A very wise man who used to work at IBM but now hangs out with people in red polos (<grin>) once told me that it\u2019s OK to mix NSDs of slightly different sizes in the same pool, but you don\u2019t want to put NSDs of vastly different sizes in the same
pool because the smaller ones will fill first and then the larger ones will have to take all the I/O. I consider 58 TB and 88 TB to be pretty significantly different and am therefore planning on creating yet another pool called \u201coc\u201d (over capacity if a user
asks, old crap internally!) and migrating files with an atime greater than, say, 1 year to that pool. But since ALL of the files in the capacity pool haven\u2019t even been looked at in at least 90 days already, does it really matter? I.e. should I just add the
NSDs to the capacity pool and be done with it?</div>
<div><br>
</div>
<div>If it\u2019s a good idea to create another pool, then I have a question about mmapplypolicy and migrations. I believe I understand how things work, but after spending over an hour looking at the documentation I cannot find anything that explicitly
confirms my understanding \u2026 so if I have another pool called oc that\u2019s ~264 TB in size and I write a policy file that looks like:</div>
<div><br>
</div>
<div>
<div>define(access_age,(DAYS(CURRENT_TIMESTAMP) - DAYS(ACCESS_TIME)))</div>
<div><br>
</div>
<div>RULE 'ReallyOldStuff'</div>
<div> MIGRATE FROM POOL 'capacity'</div>
<div> TO POOL 'oc'</div>
<div> LIMIT(98)</div>
<div> SIZE(KB_ALLOCATED/NLINK)</div>
<div> WHERE ((access_age > 365) AND (KB_ALLOCATED > 3584))</div>
<div><br>
</div>
<div>RULE 'OldStuff'</div>
<div> MIGRATE FROM POOL 'data'</div>
<div> TO POOL 'capacity'</div>
<div> LIMIT(98)</div>
<div> SIZE(KB_ALLOCATED/NLINK)</div>
<div> WHERE ((access_age > 90) AND (KB_ALLOCATED > 3584))</div>
</div>
<div><br>
</div>
<div>Keeping in mind that my capacity pool is already 98% full, is mmapplypolicy smart enough to calculate how much space it\u2019s going to free up in the capacity pool by the \u201cReallyOldStuff\u201d rule and therefore be able to potentially also move a ton of
stuff from the data pool to the capacity pool via the 2nd rule with just one invocation of mmapplypolicy? That\u2019s what I expect that it will do. I\u2019m hoping I don\u2019t have to run the mmapplypolicy twice \u2026 the first to move stuff from capacity to oc and then
a second time for it to realize, oh, I\u2019ve got a much of space free in the capacity pool now.</div>
<div><br>
</div>
<div>Thanks in advance...</div>
<div><br>
</div>
<div>Kevin</div>
<div><br>
</div>
<div>P.S. In case you\u2019re scratching your head over the fact that we have files that people haven\u2019t even looked at for months and months (more than a year in some cases) sitting out there \u2026 we sell quota in 1 TB increments \u2026 once they\u2019ve bought the
quota, it\u2019s theirs. As long as they\u2019re paying us the monthly fee if they want to keep files relating to research they did during the George Bush Presidency out there \u2026 and I mean Bush 41, not Bush 43 \u2026.then that\u2019s their choice. We do not purge files.</div>
<div><br>
<div>
<div>\u2014</div>
<div>Kevin Buterbaugh - Senior System Administrator</div>
<div>Vanderbilt University - Advanced Computing Center for Research and Education</div>
<div><a href="mailto:Kevin.Buterbaugh@vanderbilt.edu" target="_blank">Kevin.Buterbaugh@vanderbilt.edu</a> - (615)875-9633</div>
<div><br>
</div>
<br class="gmail-m_-1284207655023731153Apple-interchange-newline">
</div>
<br>
</div>
</div>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
</blockquote></div>