<div class="socmaildefaultfont" dir="ltr" style="font-family:"Helvetica Neue", Helvetica, Arial, sans-serif;font-size:10.5pt" ><div dir="ltr" ><span style="font-size:0.857em;" >"Kenneth Waegeman" <kenneth.waegeman@ugent.be> wrote:</span></div>
<div dir="ltr" > </div>
<div dir="ltr" ><div><span style="font-size:0.857em;" ><font face="Default Monospace,Courier New,Courier,monospace" ><font color="#000000" >></font></font><font color="#000000" ><font face="Default Monospace,Courier New,Courier,monospace" > </font></font>Currently our file system is down due to down/unrecovered disks. We</span></div>
<div><span style="font-size:0.857em;" >> try to start the disks again with mmchdisk, but when we do this, we</span></div>
<div><span style="font-size:0.857em;" >> see this error in our mmfs.log: </span></div>
<div><span style="font-size:0.857em;" >> ...</span></div>
<div><span style="font-size:0.857em;" >> This is a 3-way replicated vdisk, and not one of the recovering disks,</span><span style="font-size:0.857em;" > but</span></div>
<div><span style="font-size:0.857em;" >> this disk is in 'up' state..</span></div>
<div> </div>
<div><span style="font-size:0.857em;" >First, please open a PMR through your normal support organization, and make it clear in the PMR that the problem is GPFS and GNR (a.k.a. ESS). Like that, it will be assigned to the correct support group. Support will request that you upload a snap.</span></div>
<div> </div>
<div><span style="font-size:0.857em;" >There seems to be a combination of two problems here: </span></div>
<div> </div>
<div><span style="font-size:0.857em;" >One, a NSD (which is also a GNR vdisk) is down, which is usually caused by an IO error on the vdisk, or by both servers for the recovery group that contains the vdisk being down simultaneously. Usually, that is easily fixed by running mmchdisk with a start option, but you tried that and it didn't work. This problem is at the NSD layer (meaning in the GPFS client that accesses the GNR vdisk), not in the GNR layer.</span></div>
<div> </div>
<div><span style="font-size:0.857em;" >Second, another vdisk has an internal error, caused by read error from the physical disks (which is what "uncorrectable read error" means). Now, give that you say that this vdisk is 3-way replicated, that probably means that there are multiple problems. This error is purely in the GNR layer, and the error message you quote "smallRead VIO..." comes from the GNR layer. Now, an error from one vdisk can't prevent mmchdisk on a different vdisk from working, so these two problems seem unrelated.</span></div>
<div> </div>
<div><span style="font-size:0.857em;" >Furthermore, I'm going to bet that the two problems (which at first seem unrelated) must in reality have a common root cause; it would be too weird a coincidence to get two problems that are unrelated at the same time. To debug this requires looking at way more information than a single line from the mmfs.log file, which is why the support organization needs a complete PMR opened, and then have the usual snap (with logs, dumps, ...) uploaded, so it can see what the cause of the problem is.</span></div>
<div> </div>
<div><span style="font-size:0.857em;" >Good luck!</span></div>
<div> </div><span style="font-size:0.857em;" >Ralph Becker-Szendy<br>IBM Almaden Research Center - Computer Science -Storage Systems<br>ralphbsz@us.ibm.com<br>408-927-2752<br>650 Harry Road, K56-B3, San Jose, CA 95120</span></div>
<div dir="ltr" > </div></div><BR>