<font size=2 face="sans-serif">yes (and no...) </font><br><font size=2 face="sans-serif">modern operating systems/ kernel-level(since
> 10 years) really occupy only buffers, which are "in use"
.. so no waste of space, only a ssh session(socket) needs less space </font><br><font size=2 face="sans-serif">but - in case of network issue/ outages..
network is preferred so all open sockets may fill up to the maximum.. so
I agree, one need to be a bit carful here...</font><br><font size=2 face="sans-serif">but as I said.. you can not cheat physics..
with RTT 180 MS you need to have according socket sizes.. (believe
it or not ;-) ... sure, it does not mean, that your problem will disappear
, because there are many potential possibilities for improvements... to
complex to discuss/solve by email here.. I would recommend to open a PMR
... </font><br><br><br><br><font size=2 face="sans-serif">in addition:</font><br><font size=2 face="sans-serif">nfsprefetchstrategy ... ranges
from 1...10 .. </font><br><font size=2 face="sans-serif">when a multithreaded NFSd gets IOs from
outside und directs them to GPFS .. the ordership of these IOs may not
be consistent.. making it a bit harder to detect sequential IO access pattern...
setting this value advises GPFS to consider all incoming IOs between <value>
block boundaries as sequential </font><br><br><font size=2 face="sans-serif">cheers</font><br><font size=2 face="sans-serif">laff</font><br><br><br><br><font size=2 face="sans-serif"> </font><br><div><font size=2 face="sans-serif">Mit freundlichen Gren / Kind regards</font><br><br><font size=2 face="sans-serif"> <br>Olaf Weiser<br> <br>EMEA Storage Competence Center Mainz, German / IBM Systems, Storage Platform,<br>-------------------------------------------------------------------------------------------------------------------------------------------<br>IBM Deutschland<br>IBM Allee 1<br>71139 Ehningen<br>Phone: +49-170-579-44-66<br>E-Mail: olaf.weiser@de.ibm.com<br>-------------------------------------------------------------------------------------------------------------------------------------------<br>IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter<br>Geschftsfhrung: Martina Koederitz (Vorsitzende), Susanne Peter, Norbert
Janzen, Dr. Christian Keller, Ivo Koerner, Markus Koerner<br>Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart,
HRB 14562 / WEEE-Reg.-Nr. DE 99369940 </font><br><br><br><br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">Loic Tortay <tortay@cc.in2p3.fr></font><br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">gpfsug main discussion
list <gpfsug-discuss@spectrumscale.org></font><br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">11/10/2016 07:39 AM</font><br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">Re: [gpfsug-discuss]
Tuning AFM for high throughput/high IO over _really_ long distances</font><br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr noshade><br><br><br><tt><font size=2>On 09/11/2016 21:53, Olaf Weiser wrote:<br>> let's say you have a RRT of 180 ms<br>> what you then need is your theoretical link speed - let's say
10 Gbit/s ...<br>> easily let's take 1 GB/s<br>><br>> this means, you socket must be capable to take your bandwidth (data
stream)<br>> during the "first" 180ms because it will take at least this
time to get back the<br>> first ACKs .. .<br>> so 1 GB / s x 0,180 s = 1024 MB/s x 0,180 s ==>> 185 MB
this means, you have<br>> to allow the operating system to accept socketsizes in that range...<br>><br>> set something like this - but increase these values to 185 MB<br>> sysctl -w net.ipv4.tcp_rmem="12194304 12194304 12194304"<br>> sysctl -w net.ipv4.tcp_wmem="12194304 12194304 12194304"<br>> sysctl -w net.ipv4.tcp_mem="12194304 12194304 12194304"<br>> sysctl -w net.core.rmem_max=12194304<br>> sysctl -w net.core.wmem_max=12194304<br>> sysctl -w net.core.rmem_default=12194304<br>> sysctl -w net.core.wmem_default=12194304<br>> sysctl -w net.core.optmem_max=12194304<br>><br>Hello,<br>In my opinion, some of these changes are, at best, misguided.<br>For instance, the unit for "tcp_mem" is not bytes but pages.
It's also <br>not a parameter for buffers but a parameter influencing global kernel <br>memory management for TCP (source: Linux kernel documentation/source).<br>Or setting the maximum TCP ancillary data buffer size ("optmem_max")
to <br>a very large value when, as far a I know/saw when testing AFM w/ NFS, <br>there is no ancillary data used.<br>Setting the min, default and max to the same value for the buffers is <br>also, in my opinion, highly debatable (do you really want, for instance,
<br>each and every SSH connection to have 185 MB TCP buffers? -- 185 MB <br>being the value suggested above).<br>I have seen the same suggestions in the AFM documentation, and in my <br>opinion, along with the unhelpful "nfsPrefetchStrategy" recommandation
<br>("it's critical: set it to at least 5 to 10", OK but how do I
chose <br>between 5 to 10 or should I use 42?, what's the unit?, what are the <br>criteria?), these do not contribute to give a good understanding of the
<br>configuration (let alone "optimization") required for AFM over
NFS.<br><br>I must add that, in my opinion, I have "enough" experience with
setting <br>these "sysctl" parameters of NFS "tuning" (so I'm not
overwhelmed by the <br>complexity or whatever), to think something is really not right in that
<br>part of the AFM documentation.<br><br><br>Loc.<br>-- <br>| Loc Tortay <tortay@cc.in2p3.fr> - IN2P3
Computing Centre |<br><br>[attachment "smime.p7s" deleted by Olaf Weiser/Germany/IBM] _______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br></font></tt><br><br></div><BR>