<div>Just some datapoints, in hope that it helps..<br class="gmail_msg"><br class="gmail_msg">I've seen metadata performance improvements by turning down hyperthreading from 8/core to 4/core on Power8. Also it helped distributing the token managers over multiple nodes (6+) instead of fewer.<br class="gmail_msg"><br class="gmail_msg">I would expect this to flow over IP, not IB.</div><div><br class="gmail_msg"><br class="gmail_msg"><br class="gmail_msg"><br class="gmail_msg"> -jf</div><div><br></div><div><br class="gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg">tir. 24. jan. 2017 kl. 16.18 skrev Buterbaugh, Kevin L <<a href="mailto:Kevin.Buterbaugh@vanderbilt.edu" class="gmail_msg" target="_blank">Kevin.Buterbaugh@vanderbilt.edu</a>>:<br class="gmail_msg"></div><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Simon,<br class="gmail_msg">
<br class="gmail_msg">
FWIW, we have two servers dedicated to cluster and filesystem management functions (and 8 NSD servers). I guess you would describe our cluster as small to medium sized \u2026 ~700 nodes and a little over 1 PB of storage.<br class="gmail_msg">
<br class="gmail_msg">
Our two managers have 2 quad core (3 GHz) CPU\u2019s and 64 GB RAM. They\u2019ve got 10 GbE, but we don\u2019t use IB anywhere. We have an 8 Gb FC SAN and we do have them connected in to the SAN so that they don\u2019t have to ask the NSD servers to do any I/O for them.<br class="gmail_msg">
<br class="gmail_msg">
I do collect statistics on all the servers and plunk them into an RRDtool database. Looking at the last 30 days the load average on the two managers is in the 5-10 range. Memory utilization seems to be almost entirely dependent on how parameters like the pagepool are set on them.<br class="gmail_msg">
<br class="gmail_msg">
HTHAL\u2026<br class="gmail_msg">
<br class="gmail_msg">
Kevin<br class="gmail_msg">
<br class="gmail_msg">
> On Jan 24, 2017, at 4:00 AM, Simon Thompson (Research Computing - IT Services) <<a href="mailto:S.J.Thompson@bham.ac.uk" class="gmail_msg" target="_blank">S.J.Thompson@bham.ac.uk</a>> wrote:<br class="gmail_msg">
><br class="gmail_msg">
> We are looking at moving manager processes off our NSD nodes and on to<br class="gmail_msg">
> dedicated quorum/manager nodes.<br class="gmail_msg">
><br class="gmail_msg">
> Are there some broad recommended hardware specs for the function of these<br class="gmail_msg">
> nodes.<br class="gmail_msg">
><br class="gmail_msg">
> I assume they benefit from having high memory (for some value of high,<br class="gmail_msg">
> probably a function of number of clients, files, expected open files?, and<br class="gmail_msg">
> probably completely incalculable, so some empirical evidence may be useful<br class="gmail_msg">
> here?) (I'm going to ignore the docs that say you should have twice as<br class="gmail_msg">
> much swap as RAM!)<br class="gmail_msg">
><br class="gmail_msg">
> What about cores, do they benefit from high core counts or high clock<br class="gmail_msg">
> rates? For example would I benefit more form a high core count, low clock<br class="gmail_msg">
> speed, or going for higher clock speeds and reducing core count? Or is<br class="gmail_msg">
> memory bandwidth more important for manager nodes?<br class="gmail_msg">
><br class="gmail_msg">
> Connectivity, does token management run over IB or only over<br class="gmail_msg">
> Ethernet/admin network? I.e. Should I bother adding IB cards, or just have<br class="gmail_msg">
> fast Ethernet on them (my clients/NSDs all have IB).<br class="gmail_msg">
><br class="gmail_msg">
> I'm looking for some hints on what I would most benefit in investing in vs<br class="gmail_msg">
> keeping to budget.<br class="gmail_msg">
><br class="gmail_msg">
> Thanks<br class="gmail_msg">
><br class="gmail_msg">
> Simon<br class="gmail_msg">
><br class="gmail_msg">
> _______________________________________________<br class="gmail_msg">
> gpfsug-discuss mailing list<br class="gmail_msg">
> gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" class="gmail_msg" target="_blank">spectrumscale.org</a><br class="gmail_msg">
> <a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" class="gmail_msg" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class="gmail_msg">
<br class="gmail_msg">
_______________________________________________<br class="gmail_msg">
gpfsug-discuss mailing list<br class="gmail_msg">
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" class="gmail_msg" target="_blank">spectrumscale.org</a><br class="gmail_msg">
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" class="gmail_msg" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class="gmail_msg">
</blockquote></div></div>