<div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div dir="ltr" >in addition, to Achim,</div>
<div dir="ltr" >where do you see those "smaller IO"...</div>
<div dir="ltr" >have you checked IO sizes with mmfsadm dump iohist on each NSDclient/Server ?... If ok on that level.. it's not GPFS</div>
<div dir="ltr" > </div>
<div dir="ltr" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div dir="ltr" ><div> </div>
<div> </div>
<div>Mit freundlichen Grüßen / Kind regards</div>
<div style="font-size: 8pt; font-family: sans-serif; margin-top: 10px;" ><div><div> <br>Olaf Weiser<br> </div></div></div></div></div></div></div>
<div dir="ltr" > </div>
<div dir="ltr" > </div>
<blockquote data-history-content-modified="1" dir="ltr" style="border-left:solid #aaaaaa 2px; margin-left:5px; padding-left:5px; direction:ltr; margin-right:0px" >----- Ursprüngliche Nachricht -----<br>Von: "Achim Rehor" <Achim.Rehor@de.ibm.com><br>Gesendet von: gpfsug-discuss-bounces@spectrumscale.org<br>An: "gpfsug main discussion list" <gpfsug-discuss@spectrumscale.org><br>CC:<br>Betreff: [EXTERNAL] Re: [gpfsug-discuss] IO sizes<br>Datum: Do, 24. Feb 2022 13:41<br>
<p><font size="2" face="sans-serif" >Hi Uwe, </font><br><br><font size="2" face="sans-serif" >first of all, glad to see you back in the GPFS space ;)<br><br>agreed, groups of subblocks being written will end up in IO sizes, being smaller than the 8MB filesystem blocksize,</font><br><font size="2" face="sans-serif" >also agreed, this cannot be metadata, since their size is MUCH smaller, like 4k or less, mostly.<br><br>But why would these grouped subblock reads/writes all end up on the same NSD server, while the others do full block writes ?<br><br>How is your NSD server setup per NSD ? did you 'round-robin' set the preferred NSD server per NSD ?<br>are the client nodes transferring the data in anyway doing specifics ?<br><br>Sorry for not having a solution for you, jsut sharing a few ideas ;) </font><br><br><br><font size="1" face="Arial" >Mit freundlichen Grüßen / Kind regards</font><br><br><font size="2" face="Arial" ><b>Achim Rehor</b></font><br><br><font size="1" face="Arial" color="#0055AA" >Technical Support Specialist Spectrum Scale and ESS (SME)</font><br><font size="1" face="Arial" color="#0055AA" >Advisory Product Services Professional</font><br><font size="1" face="Arial" color="#0055AA" >IBM Systems Storage Support - EMEA</font></p>
<table cellspacing="0" cellpadding="0" border="0" > <tbody> <tr valign="top" > <td colspan="4" width="800" valign="middle" ><img alt="" src="cid:1__=4EBB0D60DFD775728f9e8a93df938690@ibm.com" width="1" height="1" border="0" ></td> </tr> <tr valign="top" > <td colspan="4" width="800" valign="middle" ><img alt="" src="cid:1__=4EBB0D60DFD775728f9e8a93df938690@ibm.com" width="1" height="1" border="0" ></td> </tr> <tr valign="top" > <td width="73" valign="middle" ><img alt="" src="cid:1__=4EBB0D60DFD775728f9e8a93df938690@ibm.com" width="1" height="1" border="0" ></td> <td width="266" valign="middle" ><img alt="" src="cid:1__=4EBB0D60DFD775728f9e8a93df938690@ibm.com" width="1" height="1" border="0" ></td> <td width="345" valign="middle" ><img alt="" src="cid:1__=4EBB0D60DFD775728f9e8a93df938690@ibm.com" width="1" height="1" border="0" ></td> <td rowspan="3" width="116" valign="middle" ><img alt="" src="cid:1__=4EBB0D60DFD775728f9e8a93df938690@ibm.com" width="1" height="1" border="0" ></td> </tr> <tr valign="top" > <td width="73" valign="middle" ><img alt="" src="cid:1__=4EBB0D60DFD775728f9e8a93df938690@ibm.com" width="1" height="1" border="0" ></td> <td width="266" valign="middle" ><img alt="" src="cid:1__=4EBB0D60DFD775728f9e8a93df938690@ibm.com" width="1" height="1" border="0" ></td> <td width="345" valign="middle" ><img alt="" src="cid:1__=4EBB0D60DFD775728f9e8a93df938690@ibm.com" width="1" height="1" border="0" ></td> </tr> <tr valign="top" > <td width="73" valign="middle" ><img alt="" src="cid:1__=4EBB0D60DFD775728f9e8a93df938690@ibm.com" width="1" height="1" border="0" ></td> <td width="266" valign="middle" ><img alt="" src="cid:1__=4EBB0D60DFD775728f9e8a93df938690@ibm.com" width="1" height="1" border="0" ></td> <td width="345" valign="middle" ><img alt="" src="cid:1__=4EBB0D60DFD775728f9e8a93df938690@ibm.com" width="1" height="1" border="0" ></td> </tr> <tr valign="top" > <td colspan="4" width="800" valign="middle" ><img alt="" src="cid:1__=4EBB0D60DFD775728f9e8a93df938690@ibm.com" width="1" height="1" border="0" ></td> </tr> <tr valign="top" > <td colspan="4" width="800" valign="middle" ><img alt="" src="cid:1__=4EBB0D60DFD775728f9e8a93df938690@ibm.com" width="1" height="1" border="0" ></td> </tr> <tr valign="top" > <td colspan="4" width="800" valign="middle" ><img alt="" src="cid:1__=4EBB0D60DFD775728f9e8a93df938690@ibm.com" width="1" height="1" border="0" ></td> </tr> </tbody></table><br><tt><font size="3" face="" >gpfsug-discuss-bounces@spectrumscale.org wrote on 23/02/2022 22:20:11:<br><br>> From: "Andrew Beattie" <abeattie@au1.ibm.com></font></tt><br><tt><font size="3" face="" >> To: "gpfsug main discussion list" <gpfsug-discuss@spectrumscale.org></font></tt><br><tt><font size="3" face="" >> Date: 23/02/2022 22:20</font></tt><br><tt><font size="3" face="" >> Subject: [EXTERNAL] Re: [gpfsug-discuss] IO sizes</font></tt><br><tt><font size="3" face="" >> Sent by: gpfsug-discuss-bounces@spectrumscale.org</font></tt><br><tt><font size="3" face="" >><br>> Alex, Metadata will be 4Kib Depending on the filesystem version you<br>> will also have subblocks to consider V4 filesystems have 1/32<br>> subblocks, V5 filesystems have 1/1024 subblocks (assuming metadata<br>> and data block size is the same) \u200d\u200d\u200d\u200d\u200d\u200d\u200d\u200d\u200d\u200d\u200dZjQcmQRYFpfptBannerStart </font></tt><br><tt><font size="3" face="" >> This Message Is From an External Sender </font></tt><br><tt><font size="3" face="" >> This message came from outside your organization. </font></tt><br><tt><font size="3" face="" >> ZjQcmQRYFpfptBannerEnd<br>> Alex,</font></tt><br><tt><font size="3" face="" >><br>> Metadata will be 4Kib </font></tt><br><tt><font size="3" face="" >><br>> Depending on the filesystem version you will also have subblocks to<br>> consider V4 filesystems have 1/32 subblocks, V5 filesystems have 1/<br>> 1024 subblocks (assuming metadata and data block size is the same)</font></tt><br><tt><font size="3" face="" >><br>> My first question would be is \u201c Are you sure that Linux OS is<br>> configured the same on all 4 NSD servers?.</font></tt><br><tt><font size="3" face="" >><br>> My second question would be do you know what your average file size<br>> is if most of your files are smaller than your filesystem block<br>> size, then you are always going to be performing writes using groups<br>> of subblocks rather than a full block writes.</font></tt><br><tt><font size="3" face="" >><br>> Regards, </font></tt><br><tt><font size="3" face="" >><br>> Andrew</font></tt><br><tt><font size="3" face="" >><br>> On 24 Feb 2022, at 04:39, Alex Chekholko <alex@calicolabs.com> wrote:</font></tt><br><br><tt><font size="3" face="" >> \ufeff Hi, Metadata I/Os will always be smaller than the usual data block<br>> size, right? Which version of GPFS? Regards, Alex On Wed, Feb 23,<br>> 2022 at 10:26 AM Uwe Falke <uwe.falke@kit.edu> wrote: Dear all,<br>> sorry for asking a question which seems ZjQcmQRYFpfptBannerStart </font></tt><br><tt><font size="3" face="" >> This Message Is From an External Sender </font></tt><br><tt><font size="3" face="" >> This message came from outside your organization. </font></tt><br><tt><font size="3" face="" >> ZjQcmQRYFpfptBannerEnd</font></tt><br><tt><font size="3" face="" >> Hi,</font></tt><br><tt><font size="3" face="" >><br>> Metadata I/Os will always be smaller than the usual data block size, right?</font></tt><br><tt><font size="3" face="" >> Which version of GPFS?</font></tt><br><tt><font size="3" face="" >><br>> Regards,</font></tt><br><tt><font size="3" face="" >> Alex</font></tt><br><tt><font size="3" face="" >><br>> On Wed, Feb 23, 2022 at 10:26 AM Uwe Falke <uwe.falke@kit.edu> wrote:</font></tt><br><tt><font size="3" face="" >> Dear all,<br>><br>> sorry for asking a question which seems not directly GPFS related:<br>><br>> In a setup with 4 NSD servers (old-style, with storage controllers in<br>> the back end), 12 clients and 10 Seagate storage systems, I do see in<br>> benchmark tests that just one of the NSD servers does send smaller IO<br>> requests to the storage than the other 3 (that is, both reads and<br>> writes are smaller).<br>><br>> The NSD servers form 2 pairs, each pair is connected to 5 seagate boxes<br>> ( one server to the controllers A, the other one to controllers B of the<br>> Seagates, resp.).<br>><br>> All 4 NSD servers are set up similarly:<br>><br>> kernel: 3.10.0-1160.el7.x86_64 #1 SMP<br>><br>> HBA: Broadcom / LSI Fusion-MPT 12GSAS/PCIe Secure SAS38xx<br>><br>> driver : mpt3sas 31.100.01.00<br>><br>> max_sectors_kb=8192 (max_hw_sectors_kb=16383 , not 16384, as limited by<br>> mpt3sas) for all sd devices and all multipath (dm) devices built on top.<br>><br>> scheduler: deadline<br>><br>> multipath (actually we do have 3 paths to each volume, so there is some<br>> asymmetry, but that should not affect the IOs, shouldn't it?, and if it<br>> did we would see the same effect in both pairs of NSD servers, but we do<br>> not).<br>><br>> All 4 storage systems are also configured the same way (2 disk groups /<br>> pools / declustered arrays, one managed by ctrl A, one by ctrl B, and<br>> 8 volumes out of each; makes altogether 2 x 8 x 10 = 160 NSDs).<br>><br>><br>> GPFS BS is 8MiB , according to iohistory (mmdiag) we do see clean IO<br>> requests of 16384 disk blocks (i.e. 8192kiB) from GPFS.<br>><br>> The first question I have - but that is not my main one: I do see, both<br>> in iostat and on the storage systems, that the default IO requests are<br>> about 4MiB, not 8MiB as I'd expect from above settings (max_sectors_kb<br>> is really in terms of kiB, not sectors, cf.<br>> <a href="https://www.kernel.org/doc/Documentation/block/queue-sysfs.txt" target="_blank">https://www.kernel.org/doc/Documentation/block/queue-sysfs.txt</a>).<br>><br>> But what puzzles me even more: one of the server compiles IOs even<br>> smaller, varying between 3.2MiB and 3.6MiB mostly - both for reads and<br>> writes ... I just cannot see why.<br>><br>> I have to suspect that this will (in writing to the storage) cause<br>> incomplete stripe writes on our erasure-coded volumes (8+2p)(as long as<br>> the controller is not able to re-coalesce the data properly; and it<br>> seems it cannot do it completely at least)<br>><br>><br>> If someone of you has seen that already and/or knows a potential<br>> explanation I'd be glad to learn about.<br>><br>><br>> And if some of you wonder: yes, I (was) moved away from IBM and am now<br>> at KIT.<br>><br>> Many thanks in advance<br>><br>> Uwe<br>><br>><br>> --<br>> Karlsruhe Institute of Technology (KIT)<br>> Steinbuch Centre for Computing (SCC)<br>> Scientific Data Management (SDM)<br>><br>> Uwe Falke<br>><br>> Hermann-von-Helmholtz-Platz 1, Building 442, Room 187<br>> D-76344 Eggenstein-Leopoldshafen<br>><br>> Tel: +49 721 608 28024<br>> Email: uwe.falke@kit.edu<br>> www.scc.kit.edu<br>><br>> Registered office:<br>> Kaiserstraße 12, 76131 Karlsruhe, Germany<br>><br>> KIT \u2013 The Research University in the Helmholtz Association<br>><br>> _______________________________________________<br>> gpfsug-discuss mailing list<br>> gpfsug-discuss at spectrumscale.org<br>> <a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></font></tt><br><tt><font size="3" face="" >><br>> _______________________________________________<br>> gpfsug-discuss mailing list<br>> gpfsug-discuss at spectrumscale.org<br>> <a href="INVALID URI REMOVED" target="_blank">INVALID URI REMOVED</a><br>> u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-<br>> siA1ZOg&r=RGTETs2tk0Kz_VOpznDVDkqChhnfLapOTkxLvgmR2-M&m=-<br>> FdZvYBvHDPnBTu2FtPkLT09ahlYp2QsMutqNV2jWaY&s=S4C2D3_h4FJLAw0PUYLKhKE242vn_fwn-1_EJmHNpE8&e=</font></tt><br><br>
<div><font size="2" face="Default Monospace,Courier New,Courier,monospace" >_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a> </font></div></blockquote>
<div dir="ltr" > </div></div>
<BR>
<BR>