<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Hi All,
<div class=""><br class="">
</div>
<div class="">We are in a partial cluster downtime today to do firmware upgrades on our storage arrays. It is a partial downtime because we have two GPFS filesystems:</div>
<div class=""><br class="">
</div>
<div class="">1. gpfs23 - 900+ TB and which corresponds to /scratch and /data, and which I\u2019ve unmounted across the cluster because it has data replication set to 1.</div>
<div class=""><br class="">
</div>
<div class="">2. gpfs22 - 42 TB and which corresponds to /home. It has data replication set to two, so what we\u2019re doing is \u201cmmchdisk gpfs22 suspend -d <the gpfs22 NSD>\u201d, then doing the firmware upgrade, and once the array is back we\u2019re doing a \u201cmmchdisk gpfs22
resume -d <NSD>\u201d, followed by \u201cmmchdisk gpfs22 start -d <NSD>\u201d.</div>
<div class=""><br class="">
</div>
<div class="">On the 1st storage array this went very smoothly \u2026 the mmchdisk took about 5 minutes, which is what I would expect.</div>
<div class=""><br class="">
</div>
<div class="">But on the 2nd storage array the mmchdisk appears to either be hung or proceeding at a glacial pace. For more than an hour it\u2019s been stuck at:</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">mmchdisk: Processing continues ...</div>
<div class="">Scanning file system metadata, phase 1 \u2026</div>
<div class=""><br class="">
</div>
<div class="">There are no waiters of any significance and \u201cmmdiag \u2014iohist\u201d doesn\u2019t show any issues either.</div>
<div class=""><br class="">
</div>
<div class="">Any ideas, anyone? Unless I can figure this out I\u2019m hosed for this downtime, as I\u2019ve got 7 more arrays to do after this one!</div>
<div class=""><br class="">
</div>
<div class="">Thanks!</div>
<br class="">
<div class="">
<div class="">\u2014</div>
<div class="">Kevin Buterbaugh - Senior System Administrator</div>
<div class="">Vanderbilt University - Advanced Computing Center for Research and Education</div>
<div class=""><a href="mailto:Kevin.Buterbaugh@vanderbilt.edu" class="">Kevin.Buterbaugh@vanderbilt.edu</a> - (615)875-9633</div>
<div class=""><br class="">
</div>
<br class="Apple-interchange-newline">
</div>
<br class="">
</div>
</body>
</html>