<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto">Hi Kevin,<div><br></div><div>I don\u2019t have any experience running GPFS over iSCSI (although does iSER count?). For what it\u2019s worth there are, I believe, 2 vendors in the FC space\u2014 Cisco and Brocade. You can get a fully licensed 16GB Cisco MDS edge switch for not a whole lot (<a href="https://m.cdw.com/product/cisco-mds-9148s-switch-48-ports-managed-rack-mountable/3520640">https://m.cdw.com/product/cisco-mds-9148s-switch-48-ports-managed-rack-mountable/3520640</a>). If you start with fewer ports licensed the cost goes down dramatically. I\u2019ve also found that, if it\u2019s an option, IB is stupidly cheap from a dollar per unit of bandwidth perspective and makes a great SAN backend. One other, other thought is that FCoE seems very attractive. Not sure if your arrays support that but I believe you get closer performance and behavior to FC with FCoE than with iSCSI and I don\u2019t think there\u2019s a huge cost difference. It\u2019s even more fun if you have multi-fabric FC switches that can do FC and FCoE because you can in theory bridge the two fabrics (e.g. use FCoE on your NSD servers to some 40G switches that support DCB and connect the 40G eth switches to an FC/FCoE switch and then address your 8Gb FC storage and FCoE storage using the same fabric). </div><div><br></div><div>-Aaron<br><br><div id="AppleMailSignature" dir="ltr">Sent from my iPhone</div><div dir="ltr"><br>On Dec 13, 2018, at 15:54, Buterbaugh, Kevin L <<a href="mailto:Kevin.Buterbaugh@Vanderbilt.Edu">Kevin.Buterbaugh@Vanderbilt.Edu</a>> wrote:<br><br></div><blockquote type="cite"><div dir="ltr">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
Hi All,
<div class=""><br class="">
</div>
<div class="">Googling \u201cGPFS and iSCSI\u201d doesn\u2019t produce a ton of hits! But we are interested to know if anyone is actually using GPFS over iSCSI?</div>
<div class=""><br class="">
</div>
<div class="">The reason why I\u2019m asking is that we currently use an 8 Gb FC SAN \u2026 QLogic SANbox 5800\u2019s, QLogic HBA\u2019s in our NSD servers \u2026 but we\u2019re seeing signs that, especially when we start using beefier storage arrays with more disks behind the controllers,
the 8 Gb FC could be a bottleneck.</div>
<div class=""><br class="">
</div>
<div class="">As many / most of you are already aware, I\u2019m sure, while 16 Gb FC exists, there\u2019s basically only one vendor in that game. And guess what happens to prices when there\u2019s only one vendor??? We bought our 8 Gb FC switches for approximately $5K apiece.
List price on a <vendor redacted mainly because I don\u2019t even want to honor them by mentioning their name> 16 Gb FC switch - $40K. Ouch.</div>
<div class=""><br class="">
</div>
<div class="">So the idea of being able to use commodity 10 or 40 Gb Ethernet switches and HBA\u2019s is very appealing \u2026 both from a cost and a performance perspective (last I checked 40 Gb was more than twice 16 Gb!). Anybody doing this already?</div>
<div class=""><br class="">
</div>
<div class="">As those of you who\u2019ve been on this list for a while and don\u2019t filter out e-mails from me (<grin>) already know, we have a much beefier Infortrend storage array we\u2019ve purchased that I\u2019m currently using to test various metadata configurations (and
I will report back results on that when done, I promise). That array also supports iSCSI, so I actually have our test cluster GPFS filesystem up and running over iSCSI. It was surprisingly easy to set up. But any tips, suggestions, warnings, etc. about
running GPFS over iSCSI are appreciated!</div>
<div class=""><br class="">
</div>
<div class="">Two things that I am already aware of are: 1) use jumbo frames, and 2) run iSCSI over it\u2019s own private network. Other things I should be aware of?!?</div>
<div class=""><br class="">
</div>
<div class="">Thanks all\u2026</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">
<div class="">\u2014</div>
<div class="">Kevin Buterbaugh - Senior System Administrator</div>
<div class="">Vanderbilt University - Advanced Computing Center for Research and Education</div>
<div class=""><a href="mailto:Kevin.Buterbaugh@vanderbilt.edu" class="">Kevin.Buterbaugh@vanderbilt.edu</a> - (615)875-9633</div>
<div class=""><br class="">
</div>
<br class="Apple-interchange-newline">
</div>
<br class="">
</div>
</div></blockquote><blockquote type="cite"><div dir="ltr"><span>_______________________________________________</span><br><span>gpfsug-discuss mailing list</span><br><span>gpfsug-discuss at <a href="http://spectrumscale.org">spectrumscale.org</a></span><br><span><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></span><br></div></blockquote></div></body></html>