<font size=2 face="sans-serif">1. What is the version of GPFS ? Eviction
should not be blocking the applications. Was partial file caching enabled
? Eviction cannot evict partially cached files in recent releases. Eviction
does not use space inside .afm directory, and its logs are stored under
/var/mmfs/tmp by default.</font><br><br><font size=2 face="sans-serif">2. I did not understand this requirement.</font><br><font size=2 face="sans-serif"> a. When IO to home fails with
quota exceeded or no space error , the messages are requeued at gateway
node and will be retried later (usually 15 minutes). Cache cannot
read home quotas today, and in most of the cases this is not valid. </font><br><font size=2 face="sans-serif"> b. When soft quota is exceeded
on AFM fileset, auto eviction clears data blocks on files based on
LRU policy to bring quota below soft limit. These evicted files are uncached
and there is no real migration of data to home during eviction. Eviction
should get triggered before fileset usage nearing hard quota and applications
getting errors.</font><br><br><font size=2 face="sans-serif">~Venkat (vpuvvada@in.ibm.com)</font><br><br><br><br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">Jake Carroll <jake.carroll@uq.edu.au></font><br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">"gpfsug-discuss@spectrumscale.org"
<gpfsug-discuss@spectrumscale.org></font><br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">03/05/2017 02:05 AM</font><br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">[gpfsug-discuss]
Quota issues, eviction, AFM won't stop throwing data to a full location
- probably a rookie AFM mistake?</font><br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr noshade><br><br><br><font size=3 face="Calibri">Hi all,</font><br><font size=3 face="Calibri"> </font><br><font size=3 face="Calibri">I think I need some help with GPFS quotas
and hard limits vs soft limits + eviction in AFM scenarios. We\u2019ve got
a couple of issues:</font><br><font size=3 face="Calibri"> </font><br><font size=3 face="Calibri"><b>One:</b></font><br><font size=3 face="Calibri">-------</font><br><font size=3 face="Calibri">We\u2019ve come across a scenario where if
a user hits the hard quota while ingesting into cache in an AFM \u201chome
to cache\u201d relationship whilst an eviction loop is being triggered, things
seem to go wrong \u2013 and the filesystem runs off into locking up territory.
The report I have on the last incident is that a file-set got stuck
at 100% (capacity utilisation), the eviction loop either failed or blocked
and the IO requests blocked and/or failed (this one I'm a little fuzzy
on). </font><br><font size=3 face="Calibri"> </font><br><font size=3 face="Calibri">Maybe it isn\u2019t a bug and our guess is
that someone on here will probably tell us that the likely \u201cfix\u201d is to
right-size our high and low water marks appropriately. We considered a
potential bug mechanism or race condition if the eviction loop uses space
in the file-set in the .afm directory \u2013 but I then thought better of it
and though \u201c<i>Nah, surely IBM would have thought of that!</i>\u201d.</font><br><font size=3 face="Calibri"> </font><br><font size=3 face="Calibri"><b>Two:</b></font><br><font size=3 face="Calibri">-------</font><br><font size=3 face="Calibri">We witness a scenario where AFM doesn't
back off if it gets a filesystem full error code when trying to make the
cache clean in migrating data to \u201chome\u201d. If this takes a couple of seconds
to raise the error each attempt, gpfs/mmfsd will deplete NFS daemons causing
a DoS against the NFS server that is powering the cache/home relationship
for the AFM transport.<br></font><br><font size=3 face="Calibri">We had a mental model that AFM cache wouldn\u2019t
or shouldn\u2019t overload hard and soft quota as the high and low watermarks
for cache eviction policies. I guess in our heads, we\u2019d like caches to
also enforce and respect <i>quotas</i> based on requests received from
home. There are probably lots of reasons this doesn\u2019t make sense programmatically,
or to the rest of scale \u2013 but it would (seem to us) that it would clean
up this problem or at least some of it.</font><br><font size=3 face="Calibri"> </font><br><font size=3 face="Calibri">Happy to chat through his further and explain
it more if anyone is interested. If there are any AFM users out there,
we\u2019d love to hear about how you deal with quotas, hwm/lwm and eviction
over-flow scenarios, if they exist in your environment.</font><br><font size=3 face="Calibri"> </font><br><font size=3 face="Calibri">Thank you as always, list.</font><br><font size=3 face="Calibri"> </font><br><font size=3 face="Calibri">-jc</font><br><font size=3 face="Calibri"> </font><br><font size=3 face="Calibri"> </font><tt><font size=2>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br></font></tt><br><br><BR>