<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:Calibri;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:Calibri;
mso-fareast-language:EN-US;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:Calibri;
color:windowtext;}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:Calibri;
mso-fareast-language:EN-US;}
@page WordSection1
{size:595.0pt 842.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1878420819;
mso-list-type:hybrid;
mso-list-template-ids:-1274763810 134807567 134807577 134807579 134807567 134807577 134807579 134807567 134807577 134807579;}
@list l0:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style>
</head>
<body bgcolor="white" lang="EN-GB" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt">Hi.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt">I\u2019ve got an GPFS to GPFS AFM cache/home (IW) relationship set up over a really long distance. About 180ms of latency between the two clusters and around 13,000km of optical path. Fortunately for
me, I\u2019ve actually got near theoretical maximum IO over the NIC\u2019s between the clusters and I\u2019m iPerf\u2019ing at around 8.90 to 9.2Gbit/sec over a 10GbE circuit. All MTU9000 all the way through.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt">Anyway \u2013 I\u2019m finding my AFM traffic to be dragging its feet and I don\u2019t really understand why that might be. I\u2019ve verified the links and transports ability as I said above with iPerf, and CERN\u2019s
FDT to near 10Gbit/sec. <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt">I also verified the clusters on both sides in terms of disk IO and they both seem easily capable in IOZone and IOR tests of multiple GB/sec of throughput.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt">So \u2013 my questions:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-18.0pt;mso-list:l0 level1 lfo1"><![if !supportLists]><span lang="EN-AU" style="font-size:11.0pt"><span style="mso-list:Ignore">1.<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span lang="EN-AU" style="font-size:11.0pt">Are there very specific tunings AFM needs for high latency/long distance IO?
<o:p></o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-18.0pt;mso-list:l0 level1 lfo1"><![if !supportLists]><span lang="EN-AU" style="font-size:11.0pt"><span style="mso-list:Ignore">2.<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span lang="EN-AU" style="font-size:11.0pt">Are there very specific NIC/TCP-stack tunings (beyond the type of thing we already have in place) that benefits AFM over really long distances and high latency?<o:p></o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-18.0pt;mso-list:l0 level1 lfo1"><![if !supportLists]><span lang="EN-AU" style="font-size:11.0pt"><span style="mso-list:Ignore">3.<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span lang="EN-AU" style="font-size:11.0pt">We are seeing on the \u201ccache\u201d side really lazy/sticky \u201cls \u2013als\u201d in the home mount. It sometimes takes 20 to 30 seconds before the command line will report back with a long listing of
files. Any ideas why it\u2019d take that long to get a response from \u201chome\u201d.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt">We\u2019ve got our TCP stack setup fairly aggressively, on all hosts that participate in these two clusters.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt;font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas">ethtool -C enp2s0f0 adaptive-rx off<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas">ifconfig enp2s0f0 txqueuelen 10000<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas">sysctl -w net.core.rmem_max=536870912<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas">sysctl -w net.core.wmem_max=536870912<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas">sysctl -w net.ipv4.tcp_rmem="4096 87380 268435456"<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas">sysctl -w net.ipv4.tcp_wmem="4096 65536 268435456"<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas">sysctl -w net.core.netdev_max_backlog=250000<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas">sysctl -w net.ipv4.tcp_congestion_control=htcp<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas">sysctl -w net.ipv4.tcp_mtu_probing=1<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt">I modified a couple of small things on the AFM \u201ccache\u201d side to see if it\u2019d make a difference such as:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">mmchconfig afmNumWriteThreads=4<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">mmchconfig afmNumReadThreads=4<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt">But no difference so far.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt">Thoughts would be appreciated. I\u2019ve done this before over much shorter distances (30Km) and I\u2019ve flattened a 10GbE wire without really tuning\u2026anything. Are my large in-flight-packets numbers/long-time-to-acknowledgement
semantics going to hurt here? I really thought AFM might be well designed for exactly this kind of work at long distance *<b>and</b>* high throughput \u2013 so I must be missing something!<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt">-jc<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
</body>
</html>