<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Regarding MPI-IO, how do you mean \u201cbuilding the applications against GPFS\u201d?&nbsp;<div class="">We try to advise our users about things to avoid, but we have some poster-ready&nbsp;</div><div class="">\u201cchaos monkeys\u201d as well, who resist guidance. &nbsp;What apps do your users favor?</div><div class="">Molpro is one of our heaviest apps right now.</div><div class="">Thanks,</div><div class="">&nbsp;\u2014 ddj<br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Apr 26, 2018, at 11:25 AM, Nathan Harper &lt;<a href="mailto:nathan.harper@cfms.org.uk" class="">nathan.harper@cfms.org.uk</a>&gt; wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">Happy to share on the list in case anyone else finds it useful:<div class=""><br class=""></div><div class="">We use GPFS for home/scratch on our HPC clusters, supporting engineering applications, so 95+% of our jobs are multi-node MPI.&nbsp; &nbsp;We have had some questions/concerns about GPFS+Singularity+MPI-IO, as we've had issues with GPFS+MPI-IO in the past that was solved by building the applications against GPFS.&nbsp; &nbsp;If users start using Singularity containers, we then can't guarantee how the contained applications have been built.</div><div class=""><br class=""></div><div class="">I've got a small test system (2 nsd nodes, 6 compute nodes) to see if we can break it, before we deploy onto our production systems.&nbsp; &nbsp;Everything seems to be ok under synthetic benchmarks, but I've handed over to one of my chaos monkey users to let him do his worst.&nbsp;&nbsp;</div></div><div class="gmail_extra"><br class=""><div class="gmail_quote">On 26 April 2018 at 15:53, Yugendra Guvvala <span dir="ltr" class="">&lt;<a href="mailto:yguvvala@cambridgecomputer.com" target="_blank" class="">yguvvala@cambridgecomputer.com</a>&gt;</span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto" class="">I am interested to learn this too. So please add me sending a direct mail.&nbsp;<br class=""><br class=""><div id="m_-3404058168981204889AppleMailSignature" class="">Thanks,<div class="">Yugi</div></div><div class=""><div class="h5"><div class=""><br class="">On Apr 26, 2018, at 10:51 AM, Oesterlin, Robert &lt;<a href="mailto:Robert.Oesterlin@nuance.com" target="_blank" class="">Robert.Oesterlin@nuance.com</a>&gt; wrote:<br class=""><br class=""></div><blockquote type="cite" class=""><div class="">






<div class="m_-3404058168981204889WordSection1"><p class="MsoNormal"><span style="font-size:12.0pt" class="">Hi </span><span style="font-size:10.5pt;font-family:&quot;Arial&quot;,sans-serif" class="">Lohit, Nathan<u class=""></u><u class=""></u></span></p><p class="MsoNormal"><span style="font-size:12.0pt" class=""><u class=""></u>&nbsp;<u class=""></u></span></p><p class="MsoNormal"><span style="font-size:12.0pt" class="">Would you be willing to share some more details about your setup? We are just getting started here and I would like to hear about what your configuration looks like. Direct email to me is fine, thanks.<u class=""></u><u class=""></u></span></p>
<div class=""><p class="MsoNormal"><u class=""></u>&nbsp;<u class=""></u></p><p class="MsoNormal"><span style="font-size:12.0pt" class="">Bob Oesterlin<u class=""></u><u class=""></u></span></p><p class="MsoNormal"><span style="font-size:12.0pt" class="">Sr Principal Storage Engineer, Nuance<u class=""></u><u class=""></u></span></p>
</div><p class="MsoNormal"><span style="font-size:12.0pt" class=""><u class=""></u>&nbsp;<u class=""></u></span></p><p class="MsoNormal"><span style="font-size:12.0pt" class=""><u class=""></u>&nbsp;<u class=""></u></span></p>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in" class=""><p class="MsoNormal"><b class=""><span style="font-size: 12pt;" class="">From: </span></b><span style="font-size: 12pt;" class="">&lt;<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank" class="">gpfsug-discuss-bounces@<wbr class="">spectrumscale.org</a>&gt; on behalf of "<a href="mailto:valleru@cbio.mskcc.org" target="_blank" class="">valleru@cbio.mskcc.org</a>" &lt;<a href="mailto:valleru@cbio.mskcc.org" target="_blank" class="">valleru@cbio.mskcc.org</a>&gt;<br class="">
<b class="">Reply-To: </b>gpfsug main discussion list &lt;<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank" class="">gpfsug-discuss@spectrumscale.<wbr class="">org</a>&gt;<br class="">
<b class="">Date: </b>Thursday, April 26, 2018 at 9:45 AM<br class="">
<b class="">To: </b>gpfsug main discussion list &lt;<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank" class="">gpfsug-discuss@spectrumscale.<wbr class="">org</a>&gt;<br class="">
<b class="">Subject: </b>[EXTERNAL] Re: [gpfsug-discuss] Singularity + GPFS<u class=""></u><u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal"><u class=""></u>&nbsp;<u class=""></u></p>
</div>
<div name="messageBodySection" class=""><p class="MsoNormal"><span style="font-size:10.5pt;font-family:&quot;Arial&quot;,sans-serif" class="">We do run Singularity + GPFS, on our production HPC clusters.
<u class=""></u><u class=""></u></span></p>
<div class=""><p class="MsoNormal"><span style="font-size:10.5pt;font-family:&quot;Arial&quot;,sans-serif" class="">Most of the time things are fine without any issues.<u class=""></u><u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal"><span style="font-size:10.5pt;font-family:&quot;Arial&quot;,sans-serif" class=""><u class=""></u>&nbsp;<u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal"><span style="font-size:10.5pt;font-family:&quot;Arial&quot;,sans-serif" class="">However, i do see a significant performance loss when running some applications on singularity containers with GPFS.<u class=""></u><u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal"><span style="font-size:10.5pt;font-family:&quot;Arial&quot;,sans-serif" class=""><u class=""></u>&nbsp;<u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal"><span style="font-size:10.5pt;font-family:&quot;Arial&quot;,sans-serif" class="">As of now, the applications that have severe performance issues with singularity on GPFS - seem to be because of \u201cmmap io\u201d. (Deep learning applications)<u class=""></u><u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal"><span style="font-size:10.5pt;font-family:&quot;Arial&quot;,sans-serif" class="">When i run the same application on bare metal, they seem to have a huge difference in GPFS IO when compared to running on singularity containers.<u class=""></u><u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal"><span style="font-size:10.5pt;font-family:&quot;Arial&quot;,sans-serif" class="">I am yet to raise a PMR about this with IBM.<u class=""></u><u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal"><span style="font-size:10.5pt;font-family:&quot;Arial&quot;,sans-serif" class="">I have not seen performance degradation for any other kind of IO, but i am not sure.<u class=""></u><u class=""></u></span></p>
</div>
</div>
<div name="messageSignatureSection" class=""><p class="MsoNormal"><span style="font-size:10.5pt;font-family:&quot;Arial&quot;,sans-serif" class=""><br class="">
Regards,<br class="">
Lohit<u class=""></u><u class=""></u></span></p>
</div>
<div name="messageReplySection" class=""><p class="MsoNormal"><span style="font-size:10.5pt;font-family:&quot;Arial&quot;,sans-serif" class=""><br class="">
On Apr 26, 2018, 10:35 AM -0400, Nathan Harper &lt;<a href="mailto:nathan.harper@cfms.org.uk" target="_blank" class="">nathan.harper@cfms.org.uk</a>&gt;, wrote:<br class="">
<br class="">
<u class=""></u><u class=""></u></span></p>
<blockquote style="border:none;border-left:solid #1abc9c 1.0pt;padding:0in 0in 0in 8.0pt;margin-left:3.75pt;margin-top:3.75pt;margin-right:3.75pt;margin-bottom:3.75pt" class="">
<div class=""><p class="MsoNormal"><span style="font-size:10.5pt;font-family:&quot;Arial&quot;,sans-serif" class="">We are running on a test system at the moment, and haven't run into any issues yet, but so far it's only been 'hello world' and running FIO.
<u class=""></u><u class=""></u></span></p>
<div class=""><p class="MsoNormal"><span style="font-size:10.5pt;font-family:&quot;Arial&quot;,sans-serif" class=""><u class=""></u>&nbsp;<u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal"><span style="font-size:10.5pt;font-family:&quot;Arial&quot;,sans-serif" class="">I'm interested to hear about experience with MPI-IO within Singularity.<u class=""></u><u class=""></u></span></p>
</div>
</div>
<div class=""><p class="MsoNormal"><span style="font-size:10.5pt;font-family:&quot;Arial&quot;,sans-serif" class=""><u class=""></u>&nbsp;<u class=""></u></span></p>
<div class=""><p class="MsoNormal"><span style="font-size:10.5pt;font-family:&quot;Arial&quot;,sans-serif" class="">On 26 April 2018 at 15:20, Oesterlin, Robert &lt;<a href="mailto:Robert.Oesterlin@nuance.com" target="_blank" class="">Robert.Oesterlin@nuance.com</a>&gt; wrote:<u class=""></u><u class=""></u></span></p>
<blockquote style="border:none;border-left:solid #e67e22 1.0pt;padding:0in 0in 0in 8.0pt;margin-left:3.75pt;margin-top:3.75pt;margin-right:3.75pt;margin-bottom:3.75pt" class="">
<div class="">
<div class=""><p class="MsoNormal"><span style="font-size:10.5pt;font-family:&quot;Arial&quot;,sans-serif" class="">Anyone (including IBM) doing any work in this area? I would appreciate hearing from you.<u class=""></u><u class=""></u></span></p><p class="MsoNormal"><span style="font-family:&quot;Arial&quot;,sans-serif" class="">&nbsp;</span><span style="font-size:10.5pt;font-family:&quot;Arial&quot;,sans-serif" class=""><u class=""></u><u class=""></u></span></p><p class="MsoNormal"><span style="font-size:10.5pt;font-family:&quot;Arial&quot;,sans-serif" class="">Bob Oesterlin<u class=""></u><u class=""></u></span></p><p class="MsoNormal"><span style="font-size:10.5pt;font-family:&quot;Arial&quot;,sans-serif" class="">Sr Principal Storage Engineer, Nuance<u class=""></u><u class=""></u></span></p><p class="MsoNormal"><span style="font-size:10.5pt;font-family:&quot;Arial&quot;,sans-serif" class="">&nbsp;<u class=""></u><u class=""></u></span></p>
</div>
</div><p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:10.5pt;font-family:&quot;Arial&quot;,sans-serif" class=""><br class="">
______________________________<wbr class="">_________________<br class="">
gpfsug-discuss mailing list<br class="">
gpfsug-discuss at <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&amp;d=DwMFaQ&amp;c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&amp;r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&amp;m=_6XPyvEYCL-PnMjKsOcjeRy1H19CJ1ujTFPrAEwWnLo&amp;s=OTGc6UhyurUT-JrFZxFuLp_VKCTGGknEbzjMSKUa-QQ&amp;e=" target="_blank" class="">
spectrumscale.org</a><br class="">
<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&amp;d=DwMFaQ&amp;c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&amp;r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&amp;m=_6XPyvEYCL-PnMjKsOcjeRy1H19CJ1ujTFPrAEwWnLo&amp;s=hPDughTsSwQUdR8KHv4gxXMumoGw3jXLgGGXG1MQrJM&amp;e=" target="_blank" class="">http://gpfsug.org/mailman/<wbr class="">listinfo/gpfsug-discuss</a><u class=""></u><u class=""></u></span></p>
</blockquote>
</div><p class="MsoNormal"><span style="font-size:10.5pt;font-family:&quot;Arial&quot;,sans-serif" class=""><br class="">
<br clear="all" class="">
<u class=""></u><u class=""></u></span></p>
<div class=""><p class="MsoNormal"><span style="font-size:10.5pt;font-family:&quot;Arial&quot;,sans-serif" class=""><u class=""></u>&nbsp;<u class=""></u></span></p>
</div><p class="MsoNormal"><span style="font-size:10.5pt;font-family:&quot;Arial&quot;,sans-serif" class="">--<u class=""></u><u class=""></u></span></p>
<div class="">
<div class=""><p class="MsoNormal" style="background:white"><b class=""><span style="font-size:12.0pt;font-family:&quot;Arial&quot;,sans-serif;color:#222222" class="">Nathan</span></b><b class=""><span style="font-size:12.0pt;font-family:&quot;Arial&quot;,sans-serif;color:#515058" class="">&nbsp;</span></b><b class=""><span style="font-size:12.0pt;font-family:&quot;Arial&quot;,sans-serif;color:#222222" class="">Harper</span></b><span style="font-size:12.0pt;font-family:&quot;Arial&quot;,sans-serif;color:#00aeff" class="">&nbsp;//&nbsp;</span><span style="font-size:10.0pt;font-family:&quot;Arial&quot;,sans-serif;color:#222222" class="">IT
 Systems Lead</span><span style="font-size:10.5pt;font-family:&quot;Arial&quot;,sans-serif;color:#222222" class=""><u class=""></u><u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal" style="background:white"><span style="font-size:10.5pt;font-family:&quot;Arial&quot;,sans-serif;color:#222222" class=""><u class=""></u>&nbsp;<u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal" style="background:white"><span style="font-size:10.5pt;font-family:&quot;Arial&quot;,sans-serif;color:#222222" class=""><a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__cfms.org.uk_news-2Devents_events_2018_july_farnborough-2Dinternational-2Dairshow-2D2018_&amp;d=DwMFaQ&amp;c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&amp;r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&amp;m=_6XPyvEYCL-PnMjKsOcjeRy1H19CJ1ujTFPrAEwWnLo&amp;s=SAiH6rxM-6WYWwUBRecbBS9Yr7f4adg4GizgVJXIAqI&amp;e=" target="_blank" class=""><span style="color:#222222;text-decoration:none" class=""><span style="color:blue;border:solid windowtext 1.0pt;padding:0in" class=""><img border="0" width="32" height="32" style="width:.3333in;height:.3333in" id="m_-3404058168981204889_x0000_i1026" alt="Image removed by sender." class=""></span></span></a><u class=""></u><u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal" style="background:white"><span style="font-size:10.5pt;font-family:&quot;Arial&quot;,sans-serif;color:#222222" class=""><br class="">
</span><strong class=""><span style="font-size:7.5pt;font-family:&quot;Arial&quot;,sans-serif;color:#00aeff" class="">e:&nbsp;</span></strong><span style="font-size:7.5pt;font-family:&quot;Arial&quot;,sans-serif;color:#222222" class=""><a href="mailto:nathan.harper@cfms.org.uk" target="_blank" class=""><span style="color:#515058" class="">nathan.harper@cfms.org.uk</span></a>&nbsp;</span><span style="font-size:7.5pt;font-family:&quot;Arial&quot;,sans-serif;color:#00aeff" class="">&nbsp;</span><span style="font-size:7.5pt;font-family:&quot;Arial&quot;,sans-serif;color:#222222" class=""><wbr class="">&nbsp;</span><b class=""><span style="font-size:7.5pt;font-family:&quot;Arial&quot;,sans-serif;color:#00aeff" class="">t</span></b><span style="font-size:7.5pt;font-family:&quot;Arial&quot;,sans-serif;color:#00aeff" class="">:</span><span style="font-size:7.5pt;font-family:&quot;Arial&quot;,sans-serif;color:#222222" class="">&nbsp;0117
 906 1104&nbsp;&nbsp;</span><b class=""><span style="font-size:7.5pt;font-family:&quot;Arial&quot;,sans-serif;color:#00aeff" class="">m</span></b><span style="font-size:7.5pt;font-family:&quot;Arial&quot;,sans-serif;color:#00aeff" class="">:</span><span style="font-size:7.5pt;font-family:&quot;Arial&quot;,sans-serif;color:#222222" class="">&nbsp;
 0787 551 0891&nbsp;</span><span style="font-size:7.5pt;font-family:&quot;Arial&quot;,sans-serif;color:#00aeff" class="">&nbsp;<strong class=""><span style="font-family:&quot;Arial&quot;,sans-serif" class="">w:&nbsp;</span></strong></span><span style="font-size:7.5pt;font-family:&quot;Arial&quot;,sans-serif;color:#222222" class=""><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__www.cfms.org.uk_&amp;d=DwMFaQ&amp;c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&amp;r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&amp;m=_6XPyvEYCL-PnMjKsOcjeRy1H19CJ1ujTFPrAEwWnLo&amp;s=KAbz_Md-BsCIrN6LDOb6Yc7sriwSq5IeO4aaxRGQfjc&amp;e=" target="_blank" class=""><span style="color:#515058" class="">www.cfms.org.uk</span></a>&nbsp;</span><span style="font-size:7.5pt;font-family:&quot;Arial&quot;,sans-serif;color:#00aeff" class="">&nbsp;</span><span style="font-size:10.5pt;font-family:&quot;Arial&quot;,sans-serif;color:#222222" class=""><u class=""></u><u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal" style="background:white"><span style="font-size:7.5pt;font-family:&quot;Arial&quot;,sans-serif;color:#515058" class="">CFMS Services Ltd</span><span style="font-size:7.5pt;font-family:&quot;Arial&quot;,sans-serif;color:#00aeff" class="">&nbsp;//&nbsp;</span><span style="font-size:7.5pt;font-family:&quot;Arial&quot;,sans-serif;color:#515058" class="">Bristol
 &amp; Bath Science Park</span><span style="font-size:7.5pt;font-family:&quot;Arial&quot;,sans-serif;color:#222222" class="">&nbsp;</span><span style="font-size:7.5pt;font-family:&quot;Arial&quot;,sans-serif;color:#00aeff" class="">//</span><span style="font-size:7.5pt;font-family:&quot;Arial&quot;,sans-serif;color:#222222" class="">&nbsp;</span><span style="font-size:7.5pt;font-family:&quot;Arial&quot;,sans-serif;color:#515058" class="">Dirac
 Crescent</span><span style="font-size:7.5pt;font-family:&quot;Arial&quot;,sans-serif;color:#222222" class="">&nbsp;</span><span style="font-size:7.5pt;font-family:&quot;Arial&quot;,sans-serif;color:#00aeff" class="">//</span><span style="font-size:7.5pt;font-family:&quot;Arial&quot;,sans-serif;color:#222222" class="">&nbsp;</span><span style="font-size:7.5pt;font-family:&quot;Arial&quot;,sans-serif;color:#515058" class="">Emersons
 Green</span><span style="font-size:7.5pt;font-family:&quot;Arial&quot;,sans-serif;color:#222222" class="">&nbsp;</span><span style="font-size:7.5pt;font-family:&quot;Arial&quot;,sans-serif;color:#00aeff" class="">//</span><span style="font-size:7.5pt;font-family:&quot;Arial&quot;,sans-serif;color:#222222" class="">&nbsp;</span><span style="font-size:7.5pt;font-family:&quot;Arial&quot;,sans-serif;color:#515058" class="">Bristol</span><span style="font-size:7.5pt;font-family:&quot;Arial&quot;,sans-serif;color:#222222" class="">&nbsp;</span><span style="font-size:7.5pt;font-family:&quot;Arial&quot;,sans-serif;color:#00aeff" class="">//</span><span style="font-size:7.5pt;font-family:&quot;Arial&quot;,sans-serif;color:#222222" class="">&nbsp;</span><span style="font-size:7.5pt;font-family:&quot;Arial&quot;,sans-serif;color:#515058" class="">BS16
 7FR</span><span style="font-size:7.5pt;font-family:&quot;Arial&quot;,sans-serif;color:#222222" class="">&nbsp;</span><span style="font-size:10.5pt;font-family:&quot;Arial&quot;,sans-serif;color:#222222" class=""><u class=""></u><u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal" style="background:white"><span style="font-size:12.0pt;font-family:&quot;Verdana&quot;,sans-serif;color:#222222;border:solid windowtext 1.0pt;padding:0in" class=""><img border="0" width="32" height="32" style="width:.3333in;height:.3333in" id="m_-3404058168981204889_x0000_i1025" alt="Image removed by sender." class=""></span><span style="font-size:12.0pt;font-family:&quot;Verdana&quot;,sans-serif;color:#222222" class="">&nbsp;</span><span style="font-size:10.5pt;font-family:&quot;Arial&quot;,sans-serif;color:#222222" class=""><u class=""></u><u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal" style="background:white"><span style="font-size:7.5pt;font-family:&quot;Verdana&quot;,sans-serif;color:#515058" class="">CFMS Services Ltd is registered in England and Wales No 05742022 - a subsidiary of CFMS Ltd</span><span style="font-size:12.0pt;font-family:&quot;Verdana&quot;,sans-serif;color:#222222" class="">&nbsp;<br class="">
</span><span style="font-size:7.5pt;font-family:&quot;Verdana&quot;,sans-serif;color:#515058" class="">CFMS Services Ltd registered office //&nbsp;43 Queens Square // Bristol // BS1 4QP</span><span style="font-size:12.0pt;font-family:&quot;Arial&quot;,sans-serif;color:#222222" class=""><u class=""></u><u class=""></u></span></p>
</div>
</div>
</div><p class="MsoNormal"><span style="font-size:10.5pt;font-family:&quot;Arial&quot;,sans-serif" class="">______________________________<wbr class="">_________________<br class="">
gpfsug-discuss mailing list<br class="">
gpfsug-discuss at <a href="http://spectrumscale.org/" target="_blank" class="">spectrumscale.org</a><br class="">
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank" class="">http://gpfsug.org/mailman/<wbr class="">listinfo/gpfsug-discuss</a><u class=""></u><u class=""></u></span></p>
</blockquote>
</div>
</div>


</div></blockquote><blockquote type="cite" class=""><div class=""><span class="">______________________________<wbr class="">_________________</span><br class=""><span class="">gpfsug-discuss mailing list</span><br class=""><span class="">gpfsug-discuss at <a href="http://spectrumscale.org/" target="_blank" class="">spectrumscale.org</a></span><br class=""><span class=""><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank" class="">http://gpfsug.org/mailman/<wbr class="">listinfo/gpfsug-discuss</a></span><br class=""></div></blockquote></div></div></div><br class="">______________________________<wbr class="">_________________<br class="">
gpfsug-discuss mailing list<br class="">
gpfsug-discuss at <a href="http://spectrumscale.org/" rel="noreferrer" target="_blank" class="">spectrumscale.org</a><br class="">
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank" class="">http://gpfsug.org/mailman/<wbr class="">listinfo/gpfsug-discuss</a><br class="">
<br class=""></blockquote></div><br class=""><br clear="all" class=""><div class=""><br class=""></div>-- <br class=""><div class="gmail_signature" data-smartmail="gmail_signature"><div style="color:rgb(34,34,34);background-color:rgb(255,255,255)" class=""><b style="font-family:Arial;font-size:small" class="">Nathan<span style="color:rgb(81,80,88)" class="">&nbsp;</span>Harper</b><span style="font-family:Arial;font-size:small;color:rgb(0,174,255)" class="">&nbsp;//&nbsp;</span><font size="2" style="font-family:Arial" class="">IT Systems Lead</font></div><div style="color:rgb(34,34,34);background-color:rgb(255,255,255)" class=""><font size="2" style="font-family:Arial" class=""><br class=""></font></div><div style="color:rgb(34,34,34);background-color:rgb(255,255,255)" class=""><a href="https://cfms.org.uk/news-events/events/2018/july/farnborough-international-airshow-2018/" target="_blank" class=""><img style="width:500px;height:105px" src="https://cfms.org.uk/media/267028/fia-email-sig-blue-dark-3.jpg" class=""></a></div><div style="color:rgb(34,34,34);background-color:rgb(255,255,255)" class=""><br class=""><font face="Arial" size="1" class=""><strong class=""><span style="color:rgb(0,174,255)" class="">e:&nbsp;</span></strong><a href="mailto:nathan.harper@cfms.org.uk" style="color:rgb(81,80,88)" target="_blank" class="">nathan.harper@cfms.org.uk</a>&nbsp;<span style="color:rgb(0,174,255)" class="">&nbsp;</span><font color="#222222" class="">&nbsp;</font><font color="#00aeff" class=""><b class="">t</b>:</font><font color="#222222" class="">&nbsp;0117 906 1104</font>&nbsp;<font color="#222222" class="">&nbsp;</font><font color="#00aeff" class=""><b class="">m</b>:</font><font color="#222222" class="">&nbsp; 0787 551 0891&nbsp;</font><span style="color:rgb(0,174,255)" class="">&nbsp;<strong class="">w:&nbsp;</strong></span><a href="http://www.cfms.org.uk/" rel="noreferrer" style="color:rgb(81,80,88)" target="_blank" class="">www.cfms.org.uk</a>&nbsp;<span style="color:rgb(0,174,255)" class="">&nbsp;</span></font></div><div style="color:rgb(34,34,34);background-color:rgb(255,255,255)" class=""><font face="Arial" size="1" class=""><span style="color:rgb(81,80,88)" class="">CFMS Services Ltd</span><span style="color:rgb(0,174,255)" class="">&nbsp;//&nbsp;</span><span style="color:rgb(81,80,88)" class="">Bristol &amp; Bath Science Park</span>&nbsp;<span style="color:rgb(0,174,255)" class="">//</span>&nbsp;<span style="color:rgb(81,80,88)" class="">Dirac Crescent</span>&nbsp;<span style="color:rgb(0,174,255)" class="">//</span>&nbsp;<span style="color:rgb(81,80,88)" class="">Emersons Green</span>&nbsp;<span style="color:rgb(0,174,255)" class="">//</span>&nbsp;<span style="color:rgb(81,80,88)" class="">Bristol</span>&nbsp;<span style="color:rgb(0,174,255)" class="">//</span>&nbsp;<span style="color:rgb(81,80,88)" class="">BS16 7FR</span>&nbsp;</font></div><div style="color:rgb(34,34,34);background-color:rgb(255,255,255)" class=""><font face="Verdana" style="font-size:small" class=""><img src="http://cfms.org.uk/images/logo.png" class="">&nbsp;</font></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;background-color:rgb(255,255,255)" class=""><span style="font-family:Verdana" class=""><span style="color:rgb(81,80,88);font-size:xx-small" class="">CFMS Services Ltd is registered in England and Wales No 05742022 - a subsidiary of CFMS Ltd</span>&nbsp;<br class=""><span style="color:rgb(81,80,88);font-size:xx-small" class="">CFMS Services Ltd registered office //&nbsp;</span></span><font color="#515058" face="Verdana" class=""><span style="font-size:xx-small" class="">43 Queens Square // Bristol // BS1 4QP</span></font></div></div>
</div>
_______________________________________________<br class="">gpfsug-discuss mailing list<br class="">gpfsug-discuss at <a href="http://spectrumscale.org" class="">spectrumscale.org</a><br class=""><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class=""></div></blockquote></div><br class=""></div></body></html>