<div dir="ltr"><div>Thanks for this example, very userful, but I'm still struggeling a bit at a customer..</div><div><br></div><div><br></div><div>We're doing heat daily based rebalancing, with fileheatlosspercent=20 and fileheatperiodminutes=720:</div><div><br></div><div>RULE "defineTiers" GROUP POOL 'Tiers'<br> IS 'ssdpool' LIMIT(70)<br> then 'saspool'</div><div><br></div><div>RULE 'Rebalance' MIGRATE FROM POOL 'Tiers' TO POOL 'Tiers' WEIGHT(FILE_HEAT) WHERE FILE_SIZE<10000000000</div><div><br></div><div>but are seeing too many files moved down to the saspool and too few are staying in the ssdpool. Right now we ran a test of this policy, and saw that it wanted to move 130k files / 300 GB down to the saspool, and a single small file up to the ssdpool -- even though the ssdpool is only 50% utilized. <br></div><div><br></div><div>Running your listing policy reveals lots of files with zero heat:</div><div><br></div><div><p class="MsoNormal"><7>
/gpfs/gpfs0/file1
RULE 'fh2' LIST 'fh' WEIGHT(0.000000) SHOW( _NULL_
_NULL_ _NULL_ +0.00000000000000E+000 _NULL_ 720 25 server.locale)<span></span></p>
<p class="MsoNormal"><7>
/gpfs/gpfs0/file2
RULE 'fh2' LIST 'fh' WEIGHT(0.000000) SHOW( _NULL_
_NULL_ _NULL_ +0.00000000000000E+000 _NULL_ 720 25 server.locale)<span></span></p>
<p class="MsoNormal"><7>
/gpfs/gpfs0/file3/HM_WVS_8P41017_1/HM_WVS_8P41017_1.S2206
RULE 'fh2' LIST 'fh' WEIGHT(0.000000) SHOW( _NULL_
_NULL_ _NULL_ +0.00000000000000E+000 _NULL_ 720 25 server.locale)<span></span></p>
<p class="MsoNormal"><span> </span></p><p class="MsoNormal"><span>and others with heat:</span></p><p class="MsoNormal"><span><br></span></p>
<p class="MsoNormal"><5>
/gpfs/gpfs0/file4 RULE 'fh2'
LIST 'fh' WEIGHT(0.004246) SHOW( 300401047 0 0
+4.24600492924153E-003 11E7C19700000000
720 25 server.locale)<span></span></p>
<p class="MsoNormal"><5>
/gpfs/gpfs0/file5 RULE 'fh2'
LIST 'fh' WEIGHT(0.001717) SHOW( 120971793 1 0
+1.71725239616613E-003 0735E21100010000
720 25 server.locale)</p></div><div><br></div><div>These are not new files -- so we're wondering if maybe the fileheat is reduced to zero/NULL after a while (how many times can it shrink by 25% before it's zero??).</div><div><br></div><div>Would it make sense to increase fileheatperiodeminutes and/or decrease fileheatlosspercentage? What would be good values? (BTW: we have relatime enabled)<br></div><div><br></div><div>Any other ideas for why it won't fill up our ssdpool to close to LIMIT(70) ?</div><div><br></div><div><br></div><div> -jf</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Aug 13, 2019 at 3:33 PM Marc A Kaplan <<a href="mailto:makaplan@us.ibm.com">makaplan@us.ibm.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><p><font size="2">Yes, you are correct. It should only be necessary to set fileHeatPeriodMinutes, since the loss percent does have a default value.</font><br><font size="2">But IIRC (I implemented part of this!) you must restart the daemon to get those fileheat parameter(s) "loaded"and initialized into the daemon processes.</font><br><br><font size="2">Not fully trusting my memory... I will now "prove" this works today as follows:</font><br><br><font size="2">To test, create and re-read a large file with dd...</font><br><br><font size="2">[root@/main/gpfs-git]$mmchconfig fileHeatPeriodMinutes=60</font><br><font size="2">mmchconfig: Command successfully completed</font><br><font size="2"> ...</font><br><font size="2">[root@/main/gpfs-git]$mmlsconfig | grep -i heat</font><br><font size="2">fileHeatPeriodMinutes 60</font><br><br><font size="2">[root@/main/gpfs-git]$mmshutdown</font><br><font size="2"> ...</font><br><font size="2">[root@/main/gpfs-git]$mmstartup</font><br><font size="2">...</font><br><font size="2">[root@/main/gpfs-git]$mmmount c23</font><br><font size="2">...</font><br><font size="2">[root@/main/gpfs-git]$ls -l /c23/10g</font><br><font size="2">-rw-r--r--. 1 root root 10737418240 May 16 15:09 /c23/10g</font><br><br><font size="2">[root@/main/gpfs-git]$mmlsattr -d -X /c23/10g</font><br><font size="2">file name: /c23/10g</font><br><font size="2">security.selinux<br></font><br><font size="2">(NO fileheat attribute yet...)</font><br><br><font size="2">[root@/main/gpfs-git]$dd if=/c23/10g bs=1M of=/dev/null</font><br><font size="2">...</font><br><font size="2">After the command finishes, you may need to wait a while for the metadata to flush to the inode on disk ... or you can force that with an unmount or a mmfsctl...</font><br><br><font size="2">Then the fileheat attribute will appear (I just waited by answering another email... No need to do any explicit operations on the file system..)</font><br><br><font size="2">[root@/main/gpfs-git]$mmlsattr -d -X /c23/10g</font><br><font size="2">file name: /c23/10g</font><br><font size="2">security.selinux</font><br><font size="2">gpfs.FileHeat</font><br><br><font size="2">To see its hex string value:</font><br><br><font size="2">[root@/main/gpfs-git]$mmlsattr -d -X -L /c23/10g</font><br><font size="2">file name: /c23/10g</font><br><font size="2"> ...</font><br><font size="2">security.selinux: 0x756E636F6E66696E65645F753A6F626A6563745F723A756E6C6162656C65645F743A733000</font><br><font size="2">gpfs.FileHeat: 0x000000EE42A40400</font><br><br><font size="2">Which will be interpreted by mmapplypolicy...</font><br><br><font size="2">YES, the interpretation is relative to last access time and current time, and done by a policy/sql function "computeFileHeat" <br>(You could find this using m4 directives in your policy file...)</font><br><br><font size="2">define([FILE_HEAT],[computeFileHeat(CURRENT_TIMESTAMP-ACCESS_TIME,xattr('gpfs.FileHeat'),KB_ALLOCATED)])</font><br><br><font size="2">Well gone that far, might as well try mmapplypolicy too....</font><br><br><font size="2">[root@/main/gpfs-git]$cat /gh/policies/fileheat.policy</font><br><font size="2">define(DISPLAY_NULL,[CASE WHEN ($1) IS NULL THEN '_NULL_' ELSE varchar($1) END])</font><br><br><font size="2">rule fh1 external list 'fh' exec ''</font><br><font size="2">rule fh2 list 'fh' weight(FILE_HEAT) show(DISPLAY_NULL(xattr_integer('gpfs.FileHeat',1,4,'B')) || ' ' ||</font><br><font size="2">DISPLAY_NULL(xattr_integer('gpfs.FileHeat',5,2,'B')) || ' ' ||</font><br><font size="2">DISPLAY_NULL(xattr_integer('gpfs.FileHeat',7,2,'B')) || ' ' ||</font><br><font size="2">DISPLAY_NULL(FILE_HEAT) || ' ' ||</font><br><font size="2">DISPLAY_NULL(hex(xattr('gpfs.FileHeat'))) || ' ' ||</font><br><font size="2">getmmconfig('fileHeatPeriodMinutes') || ' ' ||</font><br><font size="2">getmmconfig('fileHeatLossPercent') || ' ' ||</font><br><font size="2">getmmconfig('clusterName') )</font><br><br><br><font size="2">[root@/main/gpfs-git]$mmapplypolicy /c23 --maxdepth 1 -P /gh/policies/fileheat.policy -I test -L 3</font><br><font size="2"> ...</font><br><font size="2"><1> /c23/10g RULE 'fh2' LIST 'fh' WEIGHT(0.022363) SHOW( 238 17060 1024 +2.23632812500000E-002 000000EE42A40400 60 10 <a href="http://makaplan.sl.cloud9.ibm.com" target="_blank">makaplan.sl.cloud9.ibm.com</a>)</font><br><font size="2"> ...</font><br><font size="2">WEIGHT(0.022363) LIST 'fh' /c23/10g SHOW(238 17060 1024 +2.23632812500000E-002 000000EE42A40400 60 10 <a href="http://makaplan.sl.cloud9.ibm.com" target="_blank">makaplan.sl.cloud9.ibm.com</a>)</font><br><br><br><br><br><img src="cid:16cf722f78f308fca931" alt="Inactive hide details for Jan-Frode Myklebust ---08/13/2019 06:22:46 AM---What about filesystem atime updates. We recently chan" width="16" height="16" border="0"><font size="2" color="#424282">Jan-Frode Myklebust ---08/13/2019 06:22:46 AM---What about filesystem atime updates. We recently changed the default to Ğrelatimeğ. Could that maybe</font><br><br><font size="2" color="#5F5F5F">From: </font><font size="2">Jan-Frode Myklebust <<a href="mailto:janfrode@tanso.net" target="_blank">janfrode@tanso.net</a>></font><br><font size="2" color="#5F5F5F">To: </font><font size="2">gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.org</a>></font><br><font size="2" color="#5F5F5F">Date: </font><font size="2">08/13/2019 06:22 AM</font><br><font size="2" color="#5F5F5F">Subject: </font><font size="2">[EXTERNAL] Re: [gpfsug-discuss] Fileheat</font><br><font size="2" color="#5F5F5F">Sent by: </font><font size="2"><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">gpfsug-discuss-bounces@spectrumscale.org</a></font><br></p><hr style="color:rgb(128,145,165)" width="100%" size="2" noshade align="left"><br><br><br><br>What about filesystem atime updates. We recently changed the default to Ğrelatimeğ. Could that maybe influence heat tracking?<br><br><br><br> -jf<br><br><br>tir. 13. aug. 2019 kl. 11:29 skrev Ulrich Sibiller <<a href="mailto:u.sibiller@science-computing.de" target="_blank"><u><font color="#0000FF">u.sibiller@science-computing.de</font></u></a>>:
<ul>On 12.08.19 15:38, Marc A Kaplan wrote:<br>> My Admin guide says:<br>> <br>> The loss percentage and period are set via the configuration<br>> variables *fileHeatLossPercent *and *fileHeatPeriodMinutes*. By default, the file access temperature <br>> is not<br>> tracked. To use access temperature in policy, the tracking must first be enabled. To do this, set <br>> the two<br>> configuration variables as follows:*<br><br>Yes, I am aware of that.<br><br>> fileHeatLossPercent*<br>> The percentage (between 0 and 100) of file access temperature dissipated over the*<br>> fileHeatPeriodMinutes *time. The default value is 10.<br>> Chapter 25. Information lifecycle management for IBM Spectrum Scale *361**<br>> fileHeatPeriodMinutes*<br>> The number of minutes defined for the recalculation of file access temperature. To turn on<br>> tracking, *fileHeatPeriodMinutes *must be set to a nonzero value. The default value is 0<br>> <br>> <br>> SO Try setting both!<br><br>Well, I have not because the documentation explicitly mentions a default. What's the point of a <br>default if I have to explicitly configure it?<br><br>> ALSO to take effect you may have to mmshutdown and mmstartup, at least on the (client gpfs) nodes <br>> that are accessing the files of interest.<br><br>I have now configured both parameters and restarted GPFS. Ran a tar over a directory - still no <br>change. I will wait for 720minutes and retry (tomorrow).<br><br>Thanks<br><br>Uli<br><br>-- <br>Science + Computing AG<br>Vorstandsvorsitzender/Chairman of the board of management:<br>Dr. Martin Matzke<br>Vorstand/Board of Management:<br>Matthias Schempp, Sabine Hohenstein<br>Vorsitzender des Aufsichtsrats/<br>Chairman of the Supervisory Board:<br>Philippe Miltin<br>Aufsichtsrat/Supervisory Board:<br>Martin Wibbe, Ursula Morgenstern<br>Sitz/Registered Office: Tuebingen<br>Registergericht/Registration Court: Stuttgart<br>Registernummer/Commercial Register No.: HRB 382196<br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank"><u><font color="#0000FF">spectrumscale.org</font></u></a><u><font color="#0000FF"><br></font></u><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank"><u><font color="#0000FF">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></u></a><tt><font size="2">_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank">spectrumscale.org</a><br></font></tt><tt><font size="2"><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></font></tt><tt><font size="2"> <br></font></tt><br><br></ul><br>
<p></p></div>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
</blockquote></div>