<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><br id="lineBreakAtBeginningOfSignature"><div dir="ltr"><p class="p1" style="margin: 0px; font-stretch: normal; line-height: normal; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; -webkit-text-size-adjust: auto;"><span class="s1"><a href="mailto:gpfsug-discuss@gpfsug.org">gpfsug-discuss@gpfsug.org</a></span></p><p class="p2" style="margin: 0px; font-stretch: normal; line-height: normal; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; min-height: 24px; -webkit-text-size-adjust: auto;"><span class="s1"></span><br></p><p class="p2" style="margin: 0px; font-stretch: normal; line-height: normal; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; min-height: 24px; -webkit-text-size-adjust: auto;"><span class="s1"></span><br></p><p class="p2" style="margin: 0px; font-stretch: normal; line-height: normal; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; min-height: 24px; -webkit-text-size-adjust: auto;"><span class="s1"></span><br></p><p class="p1" style="margin: 0px; font-stretch: normal; line-height: normal; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; -webkit-text-size-adjust: auto;"><span class="s1">Michal,</span></p><p class="p2" style="margin: 0px; font-stretch: normal; line-height: normal; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; min-height: 24px; -webkit-text-size-adjust: auto;"><span class="s1"></span><br></p><p class="p1" style="margin: 0px; font-stretch: normal; line-height: normal; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; -webkit-text-size-adjust: auto;"><span class="s1">I think you need to revise your testing method. Let me explain.</span></p><p class="p2" style="margin: 0px; font-stretch: normal; line-height: normal; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; min-height: 24px; -webkit-text-size-adjust: auto;"><span class="s1"></span><br></p><p class="p1" style="margin: 0px; font-stretch: normal; line-height: normal; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; -webkit-text-size-adjust: auto;"><span class="s1">Based on my understandings:</span></p><p class="p2" style="margin: 0px; font-stretch: normal; line-height: normal; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; min-height: 24px; -webkit-text-size-adjust: auto;"><span class="s1"></span><br></p><p class="p3" style="margin: 0px 0px 4px; font-stretch: normal; font-size: 25px; line-height: normal; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; -webkit-text-size-adjust: auto;"><span class="s2" style="font-weight: bold;">3 FE servers and one storage system</span></p><p class="p4" style="margin: 0px 0px 4px; font-stretch: normal; font-size: 25px; line-height: normal; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; min-height: 31.6px; -webkit-text-size-adjust: auto;"><span class="s2" style="font-weight: bold;"></span><br></p><p class="p4" style="margin: 0px 0px 4px; font-stretch: normal; font-size: 25px; line-height: normal; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; min-height: 31.6px; -webkit-text-size-adjust: auto;"><span class="s2" style="font-weight: bold;"></span><br></p><ul class="ul1" style="-webkit-text-size-adjust: auto;"><li class="li1" style="margin: 0px; font-stretch: normal; line-height: normal; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span class="s3" style="font-family: Menlo-Regular; font-size: 14px;"></span><span class="s1">~4500 MiB/s from 8 RAID groups using XFS (one XFS per one RAID group) and parallel fio test.</span></li><li class="li1" style="margin: 0px; font-stretch: normal; line-height: normal; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span class="s3" style="font-family: Menlo-Regular; font-size: 14px;"></span><span class="s1">one FS across all 8 RAID groups and we observed performance drop down to ~3300</span></li></ul><p class="p2" style="margin: 0px; font-stretch: normal; line-height: normal; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; min-height: 24px; -webkit-text-size-adjust: auto;"><span class="s1"></span><br></p><p class="p4" style="margin: 0px 0px 4px; font-stretch: normal; font-size: 25px; line-height: normal; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; min-height: 31.6px; -webkit-text-size-adjust: auto;"><span class="s2" style="font-weight: bold;"></span><br></p><p class="p3" style="margin: 0px 0px 4px; font-stretch: normal; font-size: 25px; line-height: normal; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; -webkit-text-size-adjust: auto;"><span class="s2" style="font-weight: bold;">The test you are running is a non-clustered fs versus a clustered fs.</span></p><p class="p4" style="margin: 0px 0px 4px; font-stretch: normal; font-size: 25px; line-height: normal; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; min-height: 31.6px; -webkit-text-size-adjust: auto;"><span class="s2" style="font-weight: bold;"></span><br></p><p class="p3" style="margin: 0px 0px 4px; font-stretch: normal; font-size: 25px; line-height: normal; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; -webkit-text-size-adjust: auto;"><span class="s2" style="font-weight: bold;">XFS,</span></p><p class="p2" style="margin: 0px; font-stretch: normal; line-height: normal; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; min-height: 24px; -webkit-text-size-adjust: auto;"><span class="s1"></span><br></p><ul class="ul1" style="-webkit-text-size-adjust: auto;"><li class="li1" style="margin: 0px; font-stretch: normal; line-height: normal; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span class="s3" style="font-family: Menlo-Regular; font-size: 14px;"></span><span class="s1">8 XFS filesystems.</span></li><li class="li1" style="margin: 0px; font-stretch: normal; line-height: normal; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span class="s3" style="font-family: Menlo-Regular; font-size: 14px;"></span><span class="s1">Each FS has it own Array and independent Meta, not shared between nodes</span></li><li class="li1" style="margin: 0px; font-stretch: normal; line-height: normal; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span class="s3" style="font-family: Menlo-Regular; font-size: 14px;"></span><span class="s1">Array will see sequential IO for each array and will be able to aggregate IO\u2019s and prefetch on read.</span></li><li class="li1" style="margin: 0px; font-stretch: normal; line-height: normal; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span class="s3" style="font-family: Menlo-Regular; font-size: 14px;"></span><span class="s1">No lock traffic between nodes</span></li><li class="li1" style="margin: 0px; font-stretch: normal; line-height: normal; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span class="s3" style="font-family: Menlo-Regular; font-size: 14px;"></span><span class="s1">Didn\u2019t mention for the FIO runs is this one node or the three nodes with fs\u2019s spread across?</span></li></ul><p class="p2" style="margin: 0px; font-stretch: normal; line-height: normal; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; min-height: 24px; -webkit-text-size-adjust: auto;"><span class="s1"></span><br></p><p class="p1" style="margin: 0px; font-stretch: normal; line-height: normal; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; -webkit-text-size-adjust: auto;"><span class="s4" style="font-family: UICTFontTextStyleEmphasizedBody; font-weight: bold;">Clustered,</span></p><p class="p2" style="margin: 0px; font-stretch: normal; line-height: normal; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; min-height: 24px; -webkit-text-size-adjust: auto;"><span class="s1"></span><br></p><ul class="ul1" style="-webkit-text-size-adjust: auto;"><li class="li1" style="margin: 0px; font-stretch: normal; line-height: normal; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span class="s3" style="font-family: Menlo-Regular; font-size: 14px;"></span><span class="s1">1 Filesystem (fs0) In this case</span></li><li class="li1" style="margin: 0px; font-stretch: normal; line-height: normal; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span class="s3" style="font-family: Menlo-Regular; font-size: 14px;"></span><span class="s1">Parallel Filesystem with shared Meta and access</span></li><li class="li1" style="margin: 0px; font-stretch: normal; line-height: normal; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span class="s3" style="font-family: Menlo-Regular; font-size: 14px;"></span><span class="s1">Lock and Meta traffic across nodes</span></li><li class="li1" style="margin: 0px; font-stretch: normal; line-height: normal; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span class="s3" style="font-family: Menlo-Regular; font-size: 14px;"></span><span class="s1">GPFS Stripes across NSD, 8 in this case.&nbsp;&nbsp;</span></li><li class="li1" style="margin: 0px; font-stretch: normal; line-height: normal; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span class="s3" style="font-family: Menlo-Regular; font-size: 14px;"></span><span class="s1">Each FIO stream will in a less sequential stream at the array level</span></li><li class="li1" style="margin: 0px; font-stretch: normal; line-height: normal; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span class="s3" style="font-family: Menlo-Regular; font-size: 14px;"></span><span class="s1">The LBA will be spread causing the array to work harder due to striped up.</span></li><li class="li1" style="margin: 0px; font-stretch: normal; line-height: normal; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span class="s3" style="font-family: Menlo-Regular; font-size: 14px;"></span><span class="s1">Array logic will not see this a sequential and delivery a much lower performance from a sequential point of view as each stream is intermixed.</span></li></ul><p class="p2" style="margin: 0px; font-stretch: normal; line-height: normal; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; min-height: 24px; -webkit-text-size-adjust: auto;"><span class="s1"></span><br></p><p class="p4" style="margin: 0px 0px 4px; font-stretch: normal; font-size: 25px; line-height: normal; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; min-height: 31.6px; -webkit-text-size-adjust: auto;"><span class="s2" style="font-weight: bold;">&nbsp;</span></p><p class="p3" style="margin: 0px 0px 4px; font-stretch: normal; font-size: 25px; line-height: normal; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; -webkit-text-size-adjust: auto;"><span class="s2" style="font-weight: bold;">What to do,</span></p><p class="p4" style="margin: 0px 0px 4px; font-stretch: normal; font-size: 25px; line-height: normal; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; min-height: 31.6px; -webkit-text-size-adjust: auto;"><span class="s2" style="font-weight: bold;"></span><br></p><p class="p3" style="margin: 0px 0px 4px; font-stretch: normal; font-size: 25px; line-height: normal; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; -webkit-text-size-adjust: auto;"><span class="s2" style="font-weight: bold;">Try&nbsp;</span></p><p class="p2" style="margin: 0px; font-stretch: normal; line-height: normal; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; min-height: 24px; -webkit-text-size-adjust: auto;"><span class="s1"></span><br></p><ol class="ol1" style="-webkit-text-size-adjust: auto;"><li class="li1" style="margin: 0px; font-stretch: normal; line-height: normal; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span class="s1">8 FS with your FIO test like XFS test</span></li><li class="li1" style="margin: 0px; font-stretch: normal; line-height: normal; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span class="s1">1 FS 1 Array and matching 1 FIO ( then x8 result)</span></li></ol><p class="p2" style="margin: 0px; font-stretch: normal; line-height: normal; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; min-height: 24px; -webkit-text-size-adjust: auto;"><span class="s1"></span><br></p><p class="p4" style="margin: 0px 0px 4px; font-stretch: normal; font-size: 25px; line-height: normal; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; min-height: 31.6px; -webkit-text-size-adjust: auto;"><span class="s2" style="font-weight: bold;"></span><br></p><p class="p3" style="margin: 0px 0px 4px; font-stretch: normal; font-size: 25px; line-height: normal; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; -webkit-text-size-adjust: auto;"><span class="s2" style="font-weight: bold;">PS: You haven\u2019t mention the type of array used? Sometimes the following is important.</span></p><p class="p4" style="margin: 0px 0px 4px; font-stretch: normal; font-size: 25px; line-height: normal; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; min-height: 31.6px; -webkit-text-size-adjust: auto;"><span class="s2" style="font-weight: bold;"></span><br></p><p class="p2" style="margin: 0px; font-stretch: normal; line-height: normal; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; min-height: 24px; -webkit-text-size-adjust: auto;"><span class="s1"></span><br></p><ul class="ul1" style="-webkit-text-size-adjust: auto;"><li class="li1" style="margin: 0px; font-stretch: normal; line-height: normal; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span class="s3" style="font-family: Menlo-Regular; font-size: 14px;"></span><span class="s1">Disable prefetch at the array.&nbsp;<span class="Apple-converted-space">&nbsp;</span>This causes the array to sometimes over work the backend due to incorrectly fetching data that is never used causing extra io and cache displacement.&nbsp;<span class="Apple-converted-space">&nbsp;</span>Ie GPFS aggressively prefetches which triggers the array to do further prefetch and both are not used.</span></li></ul><p class="p2" style="margin: 0px; font-stretch: normal; line-height: normal; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; min-height: 24px; -webkit-text-size-adjust: auto;"><span class="s1"></span><br></p><p class="p4" style="margin: 0px 0px 4px; font-stretch: normal; font-size: 25px; line-height: normal; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; min-height: 31.6px; -webkit-text-size-adjust: auto;"><span class="s2" style="font-weight: bold;"></span><br></p><p class="p1" style="margin: 0px; font-stretch: normal; line-height: normal; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; -webkit-text-size-adjust: auto;"><span class="s1">Dale</span></p></div><div dir="ltr"><br><blockquote type="cite">On 9 Feb 2024, at 13:06, KG &lt;spectrumscale@kiranghag.com&gt; wrote:<br><br></blockquote></div><blockquote type="cite"><div dir="ltr">\ufeff<div dir="ltr"><div dir="ltr"><div class="gmail_default" style="font-family:verdana,sans-serif">Can you check the value of&nbsp;<span style="color:rgb(22,22,22);font-family:&quot;IBM Plex Sans&quot;,&quot;Helvetica Neue&quot;,Arial,sans-serif;font-size:16px;font-weight:700;letter-spacing:0.16px">workerThreads</span>? It seems to be default and that is 48 without protocols.</div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Feb 9, 2024 at 3:04\u202fAM dale mac &lt;<a href="mailto:macthev@gmail.com">macthev@gmail.com</a>&gt; wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>Michal,<div><br></div><div>I think you need to revise your testing method. Let me explain.</div><div><br></div><div>Based on my understandings:</div><div><br></div><div><p style="margin:0px;font-style:normal;font-variant-caps:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:&quot;Helvetica Neue&quot;;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal">3 FE servers and one storage system</p><p style="margin:0px;font-style:normal;font-variant-caps:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:&quot;Helvetica Neue&quot;;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal"><br></p><p style="margin:0px;font-style:normal;font-variant-caps:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:&quot;Helvetica Neue&quot;;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal"></p><ul><li>~4500 MiB/s from 8 RAID groups using XFS (one XFS per one RAID group) and parallel fio test.</li><li>&nbsp;one FS accross all 8 RAID groups and we observed performance drop down to ~3300</li></ul><p></p><p style="margin:0px;font-style:normal;font-variant-caps:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:&quot;Helvetica Neue&quot;;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal"><br></p><p style="margin:0px;font-style:normal;font-variant-caps:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:&quot;Helvetica Neue&quot;;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal">The test you are running is a non-clustered fs versus a clustered fs.</p><p style="margin:0px;font-style:normal;font-variant-caps:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:&quot;Helvetica Neue&quot;;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal"><br></p><p style="margin:0px;font-style:normal;font-variant-caps:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:&quot;Helvetica Neue&quot;;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal">XFS,</p><p style="margin:0px;font-style:normal;font-variant-caps:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:&quot;Helvetica Neue&quot;;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal"><br></p><p style="margin:0px;font-stretch:normal;line-height:normal;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal"></p><ul><li><font face="Helvetica Neue" size="2">8 XFS filesystems.</font></li><li><font face="Helvetica Neue" size="2">Each FS has it own Array and independent Meta, not shared between nodes</font></li><li><font face="Helvetica Neue" size="2">Array will see sequential IO for each array and will be able to aggregate IO\u2019s and&nbsp;prefetch on write.</font></li><li><font face="Helvetica Neue" size="2">No lock traffic between nodes</font></li><li><font face="Helvetica Neue" size="2">Didn\u2019t mention for the FIO runs is this one node or the three nodes with fs\u2019s spread&nbsp;across?</font></li></ul><div><font face="Helvetica Neue" size="2"><br></font></div><div><font face="Helvetica Neue" size="2">Clusterd,</font></div><div><font face="Helvetica Neue" size="2"><br></font></div><div><ul><li><font face="Helvetica Neue" size="2">1 Filesystem (fs0) In this case</font></li><li><font face="Helvetica Neue" size="2">Parallel Filsystem with shared Meta and access</font></li><li><font face="Helvetica Neue" size="2">Lock and Meta traffic across nodes</font></li><li><font face="Helvetica Neue" size="2">GPFS Stripes across NSD, 8 in this case. &nbsp;</font></li><li><font face="Helvetica Neue" size="2">Each FIO stream will in a less sequential stream at the array level</font></li><li><font face="Helvetica Neue" size="2">The LBA will be spread causing the array to work harder</font></li><li><font face="Helvetica Neue" size="2">Array logics will not see this a sequential and delivery a much lower performance from a sequential point of view.</font></li></ul></div><p></p><p style="margin:0px;font-style:normal;font-variant-caps:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:&quot;Helvetica Neue&quot;;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal">&nbsp;</p><p style="margin:0px;font-style:normal;font-variant-caps:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:&quot;Helvetica Neue&quot;;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal"><br></p><p style="margin:0px;font-style:normal;font-variant-caps:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:&quot;Helvetica Neue&quot;;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal"><br></p><p style="margin:0px;font-style:normal;font-variant-caps:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:&quot;Helvetica Neue&quot;;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal">What to do,</p><p style="margin:0px;font-style:normal;font-variant-caps:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:&quot;Helvetica Neue&quot;;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal"><br></p><p style="margin:0px;font-style:normal;font-variant-caps:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:&quot;Helvetica Neue&quot;;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal">Try&nbsp;</p><p style="margin:0px;font-stretch:normal;line-height:normal;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal"></p><ol><li><font face="Helvetica Neue" size="2">8 FS with your FIO test like XFS test</font></li><li><font face="Helvetica Neue" size="2">1 FS 1 Array and matching 1 FIO ( then x8 result)</font></li></ol><div><font face="Helvetica Neue" size="2"><br></font></div><div><font face="Helvetica Neue" size="2"><br></font></div><p></p><p style="margin:0px;font-style:normal;font-variant-caps:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:&quot;Helvetica Neue&quot;;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal"><br></p><p style="margin:0px;font-style:normal;font-variant-caps:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:&quot;Helvetica Neue&quot;;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal">PS: You haven\u2019t mention the type of array used? Sometimes the following is important.</p><p style="margin:0px;font-style:normal;font-variant-caps:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:&quot;Helvetica Neue&quot;;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal"><br></p><p style="margin:0px;font-stretch:normal;line-height:normal;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal"></p><ul><li><font face="Helvetica Neue" size="2">Disable&nbsp;prefetch at the array.&nbsp; This causes the array to sometimes over work it backend due to incorrectly fetching data that is never used causing xtra io and cache displacement.&nbsp; Ie GPFS aggressively&nbsp;prefetches which triggers the array to do further prefetch and both are not used.</font></li></ul><p></p><p style="margin:0px;font-style:normal;font-variant-caps:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:&quot;Helvetica Neue&quot;;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal"><br></p></div><div>Dale<br><div><br><blockquote type="cite"><div>On 9 Feb 2024, at 6:30 am, Alec &lt;<a href="mailto:anacreo@gmail.com" target="_blank">anacreo@gmail.com</a>&gt; wrote:</div><br><div><div dir="auto">This won't affect your current issue but if you're doing a lot of large sequential IO you may want to consider setting prefetchPct to 40 to 60 percent instead of default 20%.&nbsp; In our environment that has measurable impact, but we have a lot less ram than you do in the pagepool (8g versus 64g).<div dir="auto"><br></div><div dir="auto">Also do you have a dedicated meta pool?&nbsp; If not that could be a source of contention.&nbsp; Highly recommend a small pinnable LUN or two as a dedicated meta pool.</div><div dir="auto"><br></div><div dir="auto">Alec</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Feb 8, 2024, 7:01\u202fAM Michal Hru\u0161ka &lt;<a href="mailto:Michal.Hruska@mcomputers.cz" target="_blank">Michal.Hruska@mcomputers.cz</a>&gt; wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">





<div lang="CS">
<div><p class="MsoNormal">@Aaron<u></u><u></u></p><p class="MsoNormal">Yes, I can confirm that 2MB blocks are transfered over.<br>
<br>
<u></u><u></u></p><p class="MsoNormal">@ Jan-Frode<u></u><u></u></p><p class="MsoNormal">We tried to change multiple parameters, but if you know the best combination for sequential IO, please let me know.<u></u><u></u></p><p class="MsoNormal"><u></u>&nbsp;<u></u></p><p class="MsoNormal">#mmlsconfig<u></u><u></u></p><p class="MsoNormal">autoload no<u></u><u></u></p><p class="MsoNormal">dmapiFileHandleSize 32<u></u><u></u></p><p class="MsoNormal">minReleaseLevel 5.1.9.0<u></u><u></u></p><p class="MsoNormal">tscCmdAllowRemoteConnections no<u></u><u></u></p><p class="MsoNormal">ccrEnabled yes<u></u><u></u></p><p class="MsoNormal">cipherList AUTHONLY<u></u><u></u></p><p class="MsoNormal">sdrNotifyAuthEnabled yes<u></u><u></u></p><p class="MsoNormal">pagepool 64G<u></u><u></u></p><p class="MsoNormal">maxblocksize 16384K<u></u><u></u></p><p class="MsoNormal">maxMBpS 40000<u></u><u></u></p><p class="MsoNormal">maxReceiverThreads 32<u></u><u></u></p><p class="MsoNormal">nsdMaxWorkerThreads 512<u></u><u></u></p><p class="MsoNormal">nsdMinWorkerThreads 8<u></u><u></u></p><p class="MsoNormal">nsdMultiQueue 256<u></u><u></u></p><p class="MsoNormal">nsdSmallThreadRatio 0<u></u><u></u></p><p class="MsoNormal">nsdThreadsPerQueue 3<u></u><u></u></p><p class="MsoNormal">prefetchAggressiveness 2<u></u><u></u></p><p class="MsoNormal">adminMode central<u></u><u></u></p><p class="MsoNormal"><u></u>&nbsp;<u></u></p><p class="MsoNormal">/dev/fs0<br>
<br>
<u></u><u></u></p><p class="MsoNormal">@Uwe<u></u><u></u></p><p class="MsoNormal">Using iohist we found out that gpfs is overloading one dm-device (it took about 500ms to finish IOs). We replaced the \u201eproblematic\u201c dm-device (as we have enough drives to play with) for new one but the overloading issue just jumped to another
 dm-device.<br>
We believe that this behaviour is caused by the gpfs but we are unable to locate the root cause of it.<u></u><u></u></p><p class="MsoNormal"><u></u>&nbsp;<u></u></p>
<div>
<div><p class="MsoNormal"><span style="font-size:9pt;font-family:Verdana,sans-serif;color:rgb(89,89,89)">Best,<br>
Michal<u></u><u></u></span></p>
</div>
</div><p class="MsoNormal"><u></u>&nbsp;<u></u></p>
</div>
</div>

_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://gpfsug.org/" rel="noreferrer noreferrer" target="_blank">gpfsug.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org" rel="noreferrer noreferrer" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org</a><br>
</blockquote></div>
_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at <a href="http://gpfsug.org" target="_blank">gpfsug.org</a><br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org</a><br></div></blockquote></div><br></div></div>_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://gpfsug.org" rel="noreferrer" target="_blank">gpfsug.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org</a><br>
</blockquote></div>
<span>_______________________________________________</span><br><span>gpfsug-discuss mailing list</span><br><span>gpfsug-discuss at gpfsug.org</span><br><span>http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org</span><br></div></blockquote></body></html>