<font size=2 face="sans-serif">let's say you have a RRT of 180 ms </font><br><font size=2 face="sans-serif">what you then need is your theoretical
link speed - let's say 10 Gbit/s ... easily let's take 1 GB/s</font><br><br><font size=2 face="sans-serif">this means, you socket must be capable
to take your bandwidth (data stream) during the "first" 180ms
because it will take at least this time to get back the first ACKs .. .</font><br><font size=2 face="sans-serif">so 1 GB / s x 0,180 s = 1024 MB/s x
0,180 s ==>> 185 MB this means, you have to allow the operating
system to accept socketsizes in that range... </font><br><br><font size=2 face="sans-serif">set something like this - but increase
these values to 185 MB</font><br><font size=1 face="sans-serif">sysctl -w net.ipv4.tcp_rmem="12194304
12194304 12194304"
</font><br><font size=1 face="sans-serif">sysctl -w net.ipv4.tcp_wmem="12194304
12194304 12194304"</font><br><font size=1 face="sans-serif">sysctl -w net.ipv4.tcp_mem="12194304
12194304 12194304"</font><br><font size=1 face="sans-serif">sysctl -w net.core.rmem_max=12194304</font><br><font size=1 face="sans-serif">sysctl -w net.core.wmem_max=12194304</font><br><font size=1 face="sans-serif">sysctl -w net.core.rmem_default=12194304</font><br><font size=1 face="sans-serif">sysctl -w net.core.wmem_default=12194304</font><br><font size=1 face="sans-serif">sysctl -w net.core.optmem_max=12194304</font><br><br><font size=2 face="sans-serif">in addition set this :</font><br><font size=1 face="sans-serif">sysctl -w net.core.netdev_max_backlog=50000</font><br><font size=1 face="sans-serif">sysctl -w net.ipv4.tcp_no_metrics_save=1</font><br><font size=1 face="sans-serif">sysctl -w net.ipv4.tcp_timestamps=0</font><br><font size=1 face="sans-serif">sysctl -w net.ipv4.tcp_sack=1</font><br><font size=1 face="sans-serif">sysctl -w net.core.netdev_max_backlog=50000</font><br><font size=1 face="sans-serif">sysctl -w net.ipv4.tcp_max_syn_backlog=30000</font><br><br><br><font size=2 face="sans-serif">you need to "recycle" the
sockets.. means .. mmshutdown/stsartupo</font><br><br><font size=2 face="sans-serif">should fix you issue</font><br><br><br><br><div><font size=2 face="sans-serif">Mit freundlichen Gren / Kind regards</font><br><br><font size=2 face="sans-serif"> <br>Olaf Weiser<br> <br>EMEA Storage Competence Center Mainz, German / IBM Systems, Storage Platform,<br>-------------------------------------------------------------------------------------------------------------------------------------------<br>IBM Deutschland<br>IBM Allee 1<br>71139 Ehningen<br>Phone: +49-170-579-44-66<br>E-Mail: olaf.weiser@de.ibm.com<br>-------------------------------------------------------------------------------------------------------------------------------------------<br>IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter<br>Geschftsfhrung: Martina Koederitz (Vorsitzende), Susanne Peter, Norbert
Janzen, Dr. Christian Keller, Ivo Koerner, Markus Koerner<br>Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart,
HRB 14562 / WEEE-Reg.-Nr. DE 99369940 </font><br><br><br><br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">Jan-Frode Myklebust
<janfrode@tanso.net></font><br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">"gpfsug-discuss@spectrumscale.org"
<gpfsug-discuss@spectrumscale.org></font><br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">11/09/2016 07:05 PM</font><br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">Re: [gpfsug-discuss]
Tuning AFM for high throughput/high IO over _really_ long distances</font><br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr noshade><br><br><br><font size=3><br>Mostly curious, don't have experience in such environments, but ... Is
this AFM over NFS or NSD protocol? Might be interesting to try the other
option -- and also check how nsdperf performs over such distance/latency.<br><br><br><br>-jf</font><br><font size=3>ons. 9. nov. 2016 kl. 18.39 skrev Jake Carroll <</font><a href=mailto:jake.carroll@uq.edu.au><font size=3 color=blue><u>jake.carroll@uq.edu.au</u></font></a><font size=3>>:</font><br><font size=2>Hi.</font><p><font size=2> </font><p><font size=2>I’ve got an GPFS to GPFS AFM cache/home (IW) relationship
set up over a really long distance. About 180ms of latency between the
two clusters and around 13,000km of optical path. Fortunately for me, I’ve
actually got near theoretical maximum IO over the NIC’s between the clusters
and I’m iPerf’ing at around 8.90 to 9.2Gbit/sec over a 10GbE circuit.
All MTU9000 all the way through.</font><p><font size=2> </font><p><font size=2>Anyway – I’m finding my AFM traffic to be dragging its
feet and I don’t really understand why that might be. I’ve verified the
links and transports ability as I said above with iPerf, and CERN’s FDT
to near 10Gbit/sec. </font><p><font size=2> </font><p><font size=2>I also verified the clusters on both sides in terms of
disk IO and they both seem easily capable in IOZone and IOR tests of multiple
GB/sec of throughput.</font><p><font size=2> </font><p><font size=2>So – my questions:</font><p><font size=2> </font><p><font size=2>1. Are there very specific
tunings AFM needs for high latency/long distance IO? </font><p><font size=2>2. Are there very specific
NIC/TCP-stack tunings (beyond the type of thing we already have in place)
that benefits AFM over really long distances and high latency?</font><p><font size=2>3. We are seeing on
the “cache” side really lazy/sticky “ls –als” in the home mount. It
sometimes takes 20 to 30 seconds before the command line will report back
with a long listing of files. Any ideas why it’d take that long to get
a response from “home”.</font><p><font size=2> </font><p><font size=2>We’ve got our TCP stack setup fairly aggressively, on
all hosts that participate in these two clusters.</font><p><font size=2> </font><p><font size=2>ethtool -C enp2s0f0 adaptive-rx off</font><p><font size=2>ifconfig enp2s0f0 txqueuelen 10000</font><p><font size=2>sysctl -w net.core.rmem_max=536870912</font><p><font size=2>sysctl -w net.core.wmem_max=536870912</font><p><font size=2>sysctl -w net.ipv4.tcp_rmem="4096 87380 268435456"</font><p><font size=2>sysctl -w net.ipv4.tcp_wmem="4096 65536 268435456"</font><p><font size=2>sysctl -w net.core.netdev_max_backlog=250000</font><p><font size=2>sysctl -w net.ipv4.tcp_congestion_control=htcp</font><p><font size=2>sysctl -w net.ipv4.tcp_mtu_probing=1</font><p><font size=2> </font><p><font size=2>I modified a couple of small things on the AFM “cache”
side to see if it’d make a difference such as:</font><p><font size=2> </font><p><font size=2>mmchconfig afmNumWriteThreads=4</font><p><font size=2>mmchconfig afmNumReadThreads=4</font><p><font size=2> </font><p><font size=2>But no difference so far.</font><p><font size=2> </font><p><font size=2>Thoughts would be appreciated. I’ve done this before over
much shorter distances (30Km) and I’ve flattened a 10GbE wire without
really tuning…anything. Are my large in-flight-packets numbers/long-time-to-acknowledgement
semantics going to hurt here? I really thought AFM might be well designed
for exactly this kind of work at long distance *<b>and</b>* high throughput
– so I must be missing something!</font><p><font size=2> </font><p><font size=2>-jc</font><p><font size=2> </font><p><font size=2> </font><p><font size=2> </font><br><font size=3>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </font><a href=http://spectrumscale.org/ target=_blank><font size=3 color=blue><u>spectrumscale.org</u></font></a><font size=3 color=blue><u><br></u></font><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target=_blank><font size=3 color=blue><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></font></a><tt><font size=2>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br></font></tt><br><br></div><BR>