<div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div dir="ltr" >On that RAID 6 what is the logical RAID block size? 128K, 256K, other?</div>
<div dir="ltr" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div dir="ltr" ><div style="font-size: 12pt; font-weight: bold; font-family: sans-serif; color: #7C7C5F;" ><span style="font-size:10pt;" ><font face="Arial" >--<br>Ystävällisin terveisin / Kind regards / Saludos cordiales / Salutations</font> / <font face="Arial" >Salutacions</font><br><font face="Arial" >Luis Bolinches</font></span></div>
<div style="font-size: 12pt; font-weight: bold; font-family: sans-serif; color: #7C7C5F;" ><div><span style="font-size:10pt;" ><font face="Arial" >Consultant IT Specialist</font></span></div>
<div><span style="font-size:10pt;" ><font face="Arial" >IBM Spectrum Scale development</font></span></div>
<div><span style="font-size:10pt;" ><font face="Arial" >ESS & client adoption teams</font></span></div>
<div><span style="font-size:10pt;" ><font face="Arial" >Mobile Phone: +358503112585</font></span></div>
<div> </div>
<div><span style="font-size:10pt;" ><a href="https://www.youracclaim.com/user/luis-bolinches" rel="noopener" target="_blank"><b><font face="Arial" >https://www.youracclaim.com/user/luis-bolinches</font></b></a></span></div>
<div> </div>
<div><div><span style="font-size:10pt;" ><font face="Arial" >Ab IBM Finland Oy</font></span></div>
<div><span style="font-size:10pt;" ><font face="Arial" >Laajalahdentie 23</font></span></div>
<div><span style="font-size:10pt;" ><font face="Arial" >00330 Helsinki</font></span></div>
<div><span style="font-size:10pt;" ><font face="Arial" >Uusimaa - Finland</font></span></div><br><span style="font-size:10pt;" ><b><font face="Arial" color="#7C7C5F" >"If you always give you will always have" -- Anonymous</font></b></span></div></div>
<div style="font-size: 8pt; font-family: sans-serif; margin-top: 10px;" ><div> </div></div></div></div></div></div></div></div></div></div></div></div></div>
<div dir="ltr" > </div>
<div dir="ltr" > </div>
<blockquote data-history-content-modified="1" dir="ltr" style="border-left:solid #aaaaaa 2px; margin-left:5px; padding-left:5px; direction:ltr; margin-right:0px" >----- Original message -----<br>From: Giovanni Bracco <giovanni.bracco@enea.it><br>Sent by: gpfsug-discuss-bounces@spectrumscale.org<br>To: Jan-Frode Myklebust <janfrode@tanso.net>, gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>Cc: Agostino Funel <agostino.funel@enea.it><br>Subject: [EXTERNAL] Re: [gpfsug-discuss] very low read performance in simple spectrum scale/gpfs cluster with a storage-server SAN<br>Date: Thu, Jun 11, 2020 10:53<br>
<div><font size="2" face="Default Monospace,Courier New,Courier,monospace" >Comments and updates in the text:<br><br>On 05/06/20 19:02, Jan-Frode Myklebust wrote:<br>> fre. 5. jun. 2020 kl. 15:53 skrev Giovanni Bracco<br>> <giovanni.bracco@enea.it <<a href="mailto:giovanni.bracco@enea.it" target="_blank">mailto:giovanni.bracco@enea.it</a>>>:<br>><br>> answer in the text<br>><br>> On 05/06/20 14:58, Jan-Frode Myklebust wrote:<br>> ><br>> > Could maybe be interesting to drop the NSD servers, and let all<br>> nodes<br>> > access the storage via srp ?<br>><br>> no we can not: the production clusters fabric is a mix of a QDR based<br>> cluster and a OPA based cluster and NSD nodes provide the service to<br>> both.<br>><br>><br>> You could potentially still do SRP from QDR nodes, and via NSD for your<br>> omnipath nodes. Going via NSD seems like a bit pointless indirection.<br><br>not really: both clusters, the 400 OPA nodes and the 300 QDR nodes share<br>the same data lake in Spectrum Scale/GPFS so the NSD servers support the<br>flexibility of the setup.<br><br>NSD servers make use of a IB SAN fabric (Mellanox FDR switch) where at<br>the moment 3 different generations of DDN storages are connected,<br>9900/QDR 7700/FDR and 7990/EDR. The idea was to be able to add some less<br>expensive storage, to be used when performance is not the first priority.<br><br>><br>><br>><br>> ><br>> > Maybe turn off readahead, since it can cause performance degradation<br>> > when GPFS reads 1 MB blocks scattered on the NSDs, so that<br>> read-ahead<br>> > always reads too much. This might be the cause of the slow read<br>> seen \u2014<br>> > maybe you\u2019ll also overflow it if reading from both NSD-servers at<br>> the<br>> > same time?<br>><br>> I have switched the readahead off and this produced a small (~10%)<br>> increase of performances when reading from a NSD server, but no change<br>> in the bad behaviour for the GPFS clients<br>><br>><br>> ><br>> ><br>> > Plus.. it\u2019s always nice to give a bit more pagepool to hhe<br>> clients than<br>> > the default.. I would prefer to start with 4 GB.<br>><br>> we'll do also that and we'll let you know!<br>><br>><br>> Could you show your mmlsconfig? Likely you should set maxMBpS to<br>> indicate what kind of throughput a client can do (affects GPFS<br>> readahead/writebehind). Would typically also increase workerThreads on<br>> your NSD servers.<br><br>At this moment this is the output of mmlsconfig<br><br># mmlsconfig<br>Configuration data for cluster GPFSEXP.portici.enea.it:<br>-------------------------------------------------------<br>clusterName GPFSEXP.portici.enea.it<br>clusterId 13274694257874519577<br>autoload no<br>dmapiFileHandleSize 32<br>minReleaseLevel 5.0.4.0<br>ccrEnabled yes<br>cipherList AUTHONLY<br>verbsRdma enable<br>verbsPorts qib0/1<br>[cresco-gpfq7,cresco-gpfq8]<br>verbsPorts qib0/2<br>[common]<br>pagepool 4G<br>adminMode central<br><br>File systems in cluster GPFSEXP.portici.enea.it:<br>------------------------------------------------<br>/dev/vsd_gexp2<br>/dev/vsd_gexp3<br><br><br>><br>><br>> 1 MB blocksize is a bit bad for your 9+p+q RAID with 256 KB strip size.<br>> When you write one GPFS block, less than a half RAID stripe is written,<br>> which means you need to read back some data to calculate new parities.<br>> I would prefer 4 MB block size, and maybe also change to 8+p+q so that<br>> one GPFS is a multiple of a full 2 MB stripe.<br>><br>><br>> -jf<br><br>we have now added another file system based on 2 NSD on RAID6 8+p+q,<br>keeping the 1MB block size just not to change too many things at the<br>same time, but no substantial change in very low readout performances,<br>that are still of the order of 50 MB/s while write performance are 1000MB/s<br><br>Any other suggestion is welcomed!<br><br>Giovanni<br><br><br><br>--<br>Giovanni Bracco<br>phone +39 351 8804788<br>E-mail giovanni.bracco@enea.it<br>WWW <a href="http://www.afs.enea.it/bracco" target="_blank">http://www.afs.enea.it/bracco</a> <br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a> </font><br> </div></blockquote>
<div dir="ltr" > </div></div><BR>
Ellei edellä ole toisin mainittu: / Unless stated otherwise above:<BR>
Oy IBM Finland Ab<BR>
PL 265, 00101 Helsinki, Finland<BR>
Business ID, Y-tunnus: 0195876-3 <BR>
Registered in Finland<BR>
<BR>