<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Verdana;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:"Segoe UI Emoji";
panose-1:2 11 5 2 4 2 4 2 2 3;}
@font-face
{font-family:Aptos;}
@font-face
{font-family:"IBM Plex Sans";}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Aptos",sans-serif;
mso-ligatures:standardcontextual;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#467886;
text-decoration:underline;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#467886" vlink="#96607D" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif">I\u2019m not going to wade into the non-technical aspects of this, but from a technical point-of-view I can share what some of have done about things like this\u2026<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif">I would consider important cluster management roles like the \u201cmanager\u201d node role which dictates token server layout and where the filesystem manager roles are assigned to be things you want
much more control over than an appliance will provide. Using a storage appliance like the ESS or DSS makes sense to me for bandwidth and parallel I/O, but I don\u2019t think it\u2019s a good choice for these other functions. We\u2019ve kept those roles on systems where
we manage things with our standard tools, including logging/alerting/etc for years, even before we went the commodity route which you mentioned. This approach has proven to be a long-term good choice in terms of maintaining visibility and management of our
environment. <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif">I do understand that the licensing space with Scale has been a moving target over the years. But with the advent of capacity-based licensing, at least one only needs to purchase the server
hardware and not also Scale PVUs in order to run your own managers, so I do think that the story has improved in terms of not discouraging customers from opting into better overall architectures using a software-defined storage approach.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif">Thx<br>
Paul<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-family:"Calibri",sans-serif;mso-ligatures:none">From:</span></b><span style="font-family:"Calibri",sans-serif;mso-ligatures:none"> gpfsug-discuss <gpfsug-discuss-bounces@gpfsug.org>
<b>On Behalf Of </b>Buterbaugh, Kevin Lynn<br>
<b>Sent:</b> Wednesday, January 24, 2024 12:09<br>
<b>To:</b> gpfsug-discuss@spectrumscale.org<br>
<b>Subject:</b> [gpfsug-discuss] Wouldn't you like to know if you had filesystem corruption?<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p><span style="font-size:8.5pt;font-family:"Verdana",sans-serif;color:#CC0000">This message was sent by an external party.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;mso-ligatures:none"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal">Hi All,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Wouldn\u2019t you like to know if your IBM ESS had filesystem corruption? If you answered \u201cno\u201d my guess is that you\u2019ve never experienced undetected filesystem corruption!
<span style="font-family:"Segoe UI Emoji",sans-serif">😉</span><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Did you know that if you\u2019ve got an IBM ESS set up in its\u2019 default configuration, which also matches the recommended configuration in every last piece of IBM documentation that I\u2019ve ever come across, you WILL NOT be notified of filesystem
corruption?!?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Do you think IBM should fix this ASAP? If so, please up vote
<a href="https://ideas.ibm.com/ideas/ESS-I-61">https://ideas.ibm.com/ideas/ESS-I-61</a>.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">If you, like me, consider this a bug in the existing product and not a \u201cfeature enhancement\u201d to maybe be included in some future release if we\u2019re lucky, then please keep reading.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Here\u2019s the gory details to the best of my understanding\u2026<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Your IBM ESS can and will detect filesystem corruption (FS_STRUCT errors). But it currently will NOT, and cannot, let you know that it\u2019s happened. The reason is that FS_STRUCT errors are detected only on the filesystem manager node, which
makes sense. But if you\u2019re running in the default and recommended configuration your filesystem manager node is one of the I/O nodes, not the EMS node. The I/O nodes have no way to communicate anything out to you unless IBM decides to configure them to do
so \u2013 like they ALREADY DO with other things like hardware events \u2013 by routing the error thru the EMS node which can send it on to you.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">You could fix this problem yourself by writing a custom callback script to send you an e-mail (or a text) whenever an FS_STRUCT error is detected by the filesystem manager node \u2026 EXCEPT that you\u2019d need mailx / postfix or something like
that and IBM doesn\u2019t provide you with a way to install them on the I/O nodes. As an aside, if you\u2019re NOT on an ESS (i.e. running GPFS on some sort of commodity hardware) you can and should do this!<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">There is a workaround for this issue, which is to run your filesystem manager(s) on the EMS node. However, 1) this goes against IBM\u2019s recommendations (and defaults), and 2) is not possible for larger ESS systems as the EMS node doesn\u2019t
have enough RAM to handle the filesystem manager function.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Personally, I think it\u2019s absolutely crazy that an I/O node can tell you that you\u2019ve got a pdisk failure but can\u2019t tell you that you\u2019ve got filesystem corruption! If you agree, then please up vote the RFE above.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><rant><o:p></o:p></p>
<p class="MsoNormal">Even if you don\u2019t agree, let me ask you to consider up voting the RFE anyway. Why? To send a message to IBM that you consider it unacceptable for them to allow a customer (me, obviously) to open up a support ticket for this very issue
(again, I consider this a very serious bug, not a feature enhancement) in July of 2023, work with the customer for 6 months, and then blow the customer off by telling them, and I quote:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p style="margin:0in;line-height:18.0pt;vertical-align:baseline">\u201c<span style="font-size:9.0pt;font-family:"IBM Plex Sans",sans-serif;color:#161616">As per the dev team, this feature has been in this way since really old versions and has not changed which means
that is not going to change soon. You can request an RFE with your idea for the development team to take it into account. Below I share the link where you can share your idea (RFE):\u201d<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">\u201cNot going to change soon.\u201d Thanks for nothing, IBM \u2026 well, I do appreciate your honesty. I\u2019ve got one other RFE out there - submitted in August of 2022 - and its\u2019 status is still \u201cFuture Consideration.\u201d I guess I\u2019ll just keep my fingers
crossed that I never have filesystem corruption on an ESS. But if I do, let me highly recommend to you that you not assign me one of your support personnel who does not understand that 1 plus 4 does not equal 6 \u2026 or that October comes before November on the
calendar (both of which I have actually had happen to me in the last 6 months; no, sadly, I am not joking or exaggerating in the least).<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">To all the IBMers reading this I want you to know that I personally consider the ESS and GPFS to be the best storage solution out there from a technical perspective \u2026 I truly do. But that is rapidly becoming irrelevant when you are also
doing things like the above, especially when you are overly proud (I think you know what I mean) of your support even if it was good, which it used to be but sadly no longer is.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">IBMers, I\u2019m sure you don\u2019t like this bit of public shaming. Guess what? I don\u2019t like doing it. But I have complained directly to IBM about these things for quite some time now (ask my sales rep if you don\u2019t believe me) and it\u2019s done
no good whatsoever. Not only did I count to 100 before composing this e-mail, I slept on it. I don\u2019t know what else to do when things aren\u2019t changing. But I promise you this, if you\u2019ll stop doing stuff like this I will absolutely be more than glad to never
have to send another e-mail like this one again. Deal?<o:p></o:p></p>
<p class="MsoNormal"></rant><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thank you, all\u2026<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Kevin B.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>