<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<div>Fred is most probably correct here. the two errors are not necessarily the same.
</div>
<div><br>
</div>
<div>i would guess that looking at </div>
<div><br>
</div>
<div># mmlsreoverygroupevents <span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">dssg2 </span></div>
<div><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">or </span></div>
<div># mmvdisk recoverygroup list --recovery-group dssg2 --events</div>
<div><br>
</div>
<div>you would see e1d2s25 multiple times, changing its state from ok to diagnosing, and back to ok<br>
<br>
If you feel this is recurring too often (and i tend to agree, given the number of IOErrors,</div>
<div>you can always '--simulate-failing' this pdisk , and replace it </div>
<div># mmvdisk pdisk change --recovery-group dssg2 --pdisk e1d2s25 --simulate-failing</div>
<div><br>
</div>
<div><br>
</div>
<div><span>
<pre>-- <br></pre>
<div data-evo-paragraph="" class="" style="width: 71ch;" data-evo-signature-plain-text-mode="">
Mit freundlichen Grüßen / Kind regards</div>
<div data-evo-paragraph="" class="" style="width: 71ch;"><br>
</div>
<div data-evo-paragraph="" class="" style="width: 71ch;">Achim Rehor</div>
<div data-evo-paragraph="" class="" style="width: 71ch;"><br>
</div>
<div data-evo-paragraph="" class="" style="width: 71ch;">Technical Support Specialist S\u200bpectrum Scale and ESS (SME)</div>
<div data-evo-paragraph="" class="" style="width: 71ch;">Advisory Product Services Professional</div>
<div data-evo-paragraph="" class="" style="width: 71ch;">IBM Systems Storage Support - EMEA</div>
<div data-evo-paragraph="" class="" style="width: 71ch;"><br>
</div>
<div data-evo-paragraph="" class="" style="width: 71ch;">Achim.Rehor@de.ibm.com +49-170-4521194
</div>
<div data-evo-paragraph="" class="" style="width: 71ch;">IBM Deutschland GmbH </div>
<div data-evo-paragraph="" class="" style="width: 71ch;">Vorsitzender des Aufsichtsrats: Sebastian Krause</div>
<div data-evo-paragraph="" class="" style="width: 71ch;">Geschäftsführung: Gregor Pillen (Vorsitzender), Nicole Reimer, </div>
<div data-evo-paragraph="" class="" style="width: 71ch;">Gabriele Schwarenthorer, Christine Rupp, Frank Theisen</div>
<div data-evo-paragraph="" class="" style="width: 71ch;">Sitz der Gesellschaft: Ehningen / Registergericht: AmtsgerichtStuttgart, HRB 14562 / WEEE-Reg.-Nr. DE 99369940</div>
<div data-evo-paragraph="" class="" style="width: 71ch;"><br>
</div>
</span></div>
<div><br>
</div>
<div>-----Original Message-----</div>
<div>From: Fred Stock <stockf@us.ibm.com></div>
<div>Reply-To: gpfsug main discussion list <gpfsug-discuss@gpfsug.org></div>
<div>To: gpfsug main discussion list <gpfsug-discuss@gpfsug.org></div>
<div>Subject: [EXTERNAL] Re: [gpfsug-discuss] Bad disk but not failed in DSS-G</div>
<div>Date: Thu, 20 Jun 2024 21:02:43 +0000</div>
<div><br>
</div>
<!-- text/html --><!-- BaNnErBlUrFlE-BoDy-start --><!-- Preheader Text : BEGIN -->
<div style="visibility: hidden; font-size: 1px; color: rgb(255, 255, 255); line-height: 1px; height: 0px; max-height: 0px; opacity: 0; overflow: hidden; display: none !important;">
I think you are seeing two different errors. The backup is failing due to a stale file handle error which usually means the file system was unmounted while the file handle was open. The write error on the physical disk, may have contributed</div>
<!-- Preheader Text : END --><!-- Email Banner : BEGIN -->
<div style="visibility: hidden; font-size: 1px; color: rgb(255, 255, 255); line-height: 1px; height: 0px; max-height: 0px; opacity: 0; overflow: hidden; display: none !important;">
</div>
<!-- Email Banner : END --><!-- BaNnErBlUrFlE-BoDy-end --><!-- BaNnErBlUrFlE-HeAdEr-start -->#pfptBannerqlqwaj3 { all: revert !important; display: block !important; visibility: visible !important; opacity: 1 !important; background-color: #D0D8DC !important;
max-width: none !important; max-height: none !important } .pfptPrimaryButtonqlqwaj3:hover, .pfptPrimaryButtonqlqwaj3:focus { background-color: #b4c1c7 !important; } .pfptPrimaryButtonqlqwaj3:active { background-color: #90a4ae !important; }<!-- BaNnErBlUrFlE-HeAdEr-end --><!--
/* Font Definitions */ @font-face {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4;} @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} @font-face {font-family:Verdana; panose-1:2 11 6 4 3 5 4 4 2 4;} @font-face {font-family:Aptos; panose-1:2
11 0 4 2 2 2 2 2 4;} @font-face {font-family:"Times New Roman \(Body CS\)"; panose-1:2 11 6 4 2 2 2 2 2 4;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0in; font-size:12.0pt; font-family:"Aptos",sans-serif;} a:link, span.MsoHyperlink
{mso-style-priority:99; color:blue; text-decoration:underline;} span.EmailStyle19 {mso-style-type:personal-reply; font-family:"Arial",sans-serif; color:windowtext; font-weight:normal; font-style:normal;} .MsoChpDefault {mso-style-type:export-only; font-size:10.0pt;
mso-ligatures:none;} @page WordSection1 {size:8.5in 11.0in; margin:1.0in 1.0in 1.0in 1.0in;} div.WordSection1 {page:WordSection1;} -->
<div class="WordSection1">
<div>I think you are seeing two different errors. The backup is failing due to a stale file handle error which usually means the file system was unmounted while the file handle was open. The write error on the physical disk, may have contributed to the stale
file handle but I doubt that is the case. As I understand a single IO error on a physical disk in an ESS (DSS) system will not cause the disk to be considered bad. This is likely why the system considers the disk to be ok. I suggest you track down the source
of the stale file handle and correct that issue to see if your backups will then again be successful.</div>
<div> </div>
<div>
<div>
<div>Fred</div>
<div> </div>
<div>Fred Stock, Spectrum Scale Development Advocacy</div>
<div>stockf@us.ibm.com | 720-430-8821</div>
<div> </div>
</div>
</div>
<div> </div>
<div> </div>
<div id="mail-editor-reference-message-container">
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<div>From:gpfsug-discuss <gpfsug-discuss-bounces@gpfsug.org> on behalf of Jonathan Buzzard <jonathan.buzzard@strath.ac.uk><br>
Date: Thursday, June 20, 2024 at 4:16\u202fPM<br>
To: gpfsug-discuss@gpfsug.org <gpfsug-discuss@gpfsug.org><br>
Subject: [EXTERNAL] [gpfsug-discuss] Bad disk but not failed in DSS-G</div>
</div>
<div>
<div><br>
So came to light because I was checking the mmbackup logs and found that <br>
we had not been getting any successful backups for several days and <br>
seeing lots of errors like this<br>
<br>
Wed Jun 19 21:45:28 2024 mmbackup:Error encountered in policy scan: [E] <br>
Error on gpfs_iopen([/gpfs/users/xxxyyyyy/.swr],68050746): Stale file handle<br>
Wed Jun 19 21:45:28 2024 mmbackup:Error encountered in policy scan: [E] <br>
Summary of errors:: _dirscan failures:3, _serious unclassified errors:3.<br>
<br>
After some digging around wondering what was going on I came across <br>
these being logged on one of the DSS-G nodes<br>
<br>
[Wed Jun 12 22:22:05 2024] blk_update_request: I/O error, dev sdbv, <br>
sector 9144672512 op 0x1:(WRITE) flags 0x700 phys_seg 17 prio class 0<br>
<br>
Yikes looks like I have a failed disk/ However if I do<br>
<br>
[root@gpfs2 ~]# mmvdisk pdisk list --recovery-group all --not-ok<br>
mmvdisk: All pdisks are ok.<br>
<br>
Clearly that's a load of rubbish.<br>
<br>
After a lot more prodding<br>
<br>
[root@gpfs2 ~]# mmvdisk pdisk list --recovery-group dssg2 --pdisk e1d2s25 -L<br>
pdisk:<br>
replacementPriority = 1000<br>
name = "e1d2s25"<br>
device = <br>
"//gpfs1/dev/sdft(notEnabled),//gpfs1/dev/sdfu(notEnabled),//gpfs2/dev/sdfb,//gpfs2/dev/sdbv"<br>
recoveryGroup = "dssg2"<br>
declusteredArray = "DA1"<br>
state = "ok"<br>
IOErrors = 444<br>
IOTimeouts = 8958<br>
mediaErrors = 15<br>
<br>
<br>
What on earth gives? Why has the disk not been failed? It's not great <br>
that a clearly bad disk is allowed to stick around in the file system <br>
and cause problems IMHO.<br>
<br>
When I try and prepare the disk for removal I get<br>
<br>
[root@gpfs2 ~]# mmvdisk pdisk replace --prepare --rg dssg2 --pdisk e1d2s25<br>
mmvdisk: Pdisk e1d2s25 of recovery group dssg2 is not currently <br>
scheduled for replacement.<br>
mmvdisk:<br>
mmvdisk:<br>
mmvdisk: Command failed. Examine previous error messages to determine cause.<br>
<br>
Do I have to use the --force option? I would like to get this disk out <br>
the file system ASAP.<br>
<br>
<br>
<br>
JAB.<br>
<br>
</div>
<div>_______________________________________________<br>
</div>
<div>gpfsug-discuss mailing list<br>
</div>
<div>gpfsug-discuss at gpfsug.org<br>
</div>
<div>http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org<br>
</div>
</div>
</div>
</div>
</div>
</body>
</html>