<div dir="ltr">Hmm, this sounds suspicious. We have 10 NSDs in a pool called system. These were previously set to data+metaData with a policy that placed our home directory filesets on this pool. A few weeks ago the NSDs in this pool all filled up. To remedy that I<div><br></div><div>1. removed old snapshots</div><div>2. deleted some old homedir filesets</div><div>3. set the NSDs in this pool to metadataOnly</div><div>4. changed the policy to point homedir filesets to another pool.</div><div>5. ran a migrate policy to migrate all homedir filesets to this other pool</div><div><br></div><div>After all that I now have ~30% free space on the metadata pool. </div><div><br></div><div>Our three pools are system (metadataOnly), sas0 (data), sata0 (data)</div><div><br></div><div>mmrestripefs gsfs0 -r fails immdieately</div><div>mmrestripefs gsfs0 -r -P system fails immediately</div><div>mmrestripefs gsfs0 -r -P sas0 fails immediately</div><div>mmrestripefs gsfs0 -r -P sata0 is running (currently about 3% done)</div><div><br></div><div>Is the change from data+metadata to metadataOnly the same as removing a disk (for the purposes of this problem) or is it possible my policy is confusing things?</div><div><br></div><div><div>[root@scg-gs0 ~]# mmlspolicy gsfs0</div><div>Policy for file system '/dev/gsfs0':</div><div> Installed by root@scg-gs0 on Wed Nov 1 09:30:40 2017.</div><div> First line of policy 'policy_placement.txt' is:</div><div>RULE 'homedirs' SET POOL 'sas0' WHERE FILESET_NAME LIKE 'home.%'</div></div><div><br></div><div>The policy I used to migrate these filesets is:</div><div><br></div><div>RULE 'homedirs' MIGRATE TO POOL 'sas0' WHERE FILESET_NAME LIKE 'home.%'<br></div><div><br></div><div>jbh</div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Nov 2, 2017 at 8:44 AM, Scott Fadden <span dir="ltr"><<a href="mailto:sfadden@us.ibm.com" target="_blank">sfadden@us.ibm.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="m_697047665309176941socmaildefaultfont" dir="ltr" style="font-family:Arial,Helvetica,sans-serif;font-size:10.5pt"><div dir="ltr">I opened a defect on this the other day, in my case it was an incorrect error message. What it meant to say was,"The pool is not empty." Are you trying to remove the last disk in a pool? If so did you empty the pool with a MIGRATE policy first? </div>
<div dir="ltr"> </div>
<div dir="ltr"> </div>
<div dir="ltr">Scott Fadden<br>Spectrum Scale - Technical Marketing<br>Phone: <a href="tel:(503)%20880-5833" value="+15038805833" target="_blank">(503) 880-5833</a><br><a href="mailto:sfadden@us.ibm.com" target="_blank">sfadden@us.ibm.com</a><br><a href="http://www.ibm.com/systems/storage/spectrum/scale" target="_blank">http://www.ibm.com/systems/<wbr>storage/spectrum/scale</a></div>
<div dir="ltr"> </div>
<div dir="ltr"> </div>
<blockquote dir="ltr" style="border-left:solid #aaaaaa 2px;margin-left:5px;padding-left:5px;direction:ltr;margin-right:0px"><span class="">----- Original message -----<br>From: John Hanks <<a href="mailto:griznog@gmail.com" target="_blank">griznog@gmail.com</a>><br>Sent by: <a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">gpfsug-discuss-bounces@<wbr>spectrumscale.org</a><br>To: gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.<wbr>org</a>><br></span><span class="">Cc:<br>Subject: Re: [gpfsug-discuss] mmrestripefs "No space left on device"<br>Date: Thu, Nov 2, 2017 8:34 AM<br>
</span><div dir="ltr"><span class="">We have no snapshots ( they were the first to go when we initially hit the full metadata NSDs).
<div> </div>
<div>I've increased quotas so that no filesets have hit a space quota. </div>
<div> </div>
<div>Verified that there are no inode quotas anywhere.</div>
<div> </div>
<div>mmdf shows the least amount of free space on any nsd to be 9% free.</div>
<div> </div>
<div>Still getting this error:</div>
<div> </div>
</span><div><span class=""><div>[root@scg-gs0 ~]# mmrestripefs gsfs0 -r -N scg-gs0,scg-gs1,scg-gs2,scg-<wbr>gs3</div>
<div>Scanning file system metadata, phase 1 ... </div>
<div>Scan completed successfully.</div>
<div>Scanning file system metadata, phase 2 ... </div>
<div>Scanning file system metadata for sas0 storage pool</div>
<div>Scanning file system metadata for sata0 storage pool</div>
<div>Scan completed successfully.</div>
<div>Scanning file system metadata, phase 3 ... </div>
<div>Scan completed successfully.</div>
<div>Scanning file system metadata, phase 4 ... </div>
<div>Scan completed successfully.</div>
</span><span class=""><div>Scanning user file metadata ...</div>
</span><span class=""><div>Error processing user file metadata.</div>
</span><span class=""><div>No space left on device</div>
</span><div>Check file '/var/mmfs/tmp/gsfs0.pit.<wbr>interestingInodes.12888779711' on scg-gs0 for inodes with broken disk addresses or failures.</div><span class="">
<div>mmrestripefs: Command failed. Examine previous error messages to determine cause.</div></span></div>
<div> </div><span class="">
<div>I should note too that this fails almost immediately, far to quickly to fill up any location it could be trying to write to.</div>
<div> </div>
</span><div>jbh</div></div>
<div>
<div><span class="">On Thu, Nov 2, 2017 at 7:57 AM, David Johnson <span dir="ltr"><<a href="mailto:david_johnson@brown.edu" target="_blank">david_johnson@brown.edu</a>></span> wrote:
</span><div><div class="h5"><blockquote style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">One thing that may be relevant is if you have snapshots, depending on your release level,
<div>inodes in the snapshot may considered immutable, and will not be migrated. Once the snapshots</div>
<div>have been deleted, the inodes are freed up and you won\u2019t see the (somewhat misleading) message</div>
<div>about no space.</div>
<div> </div>
<div> \u2014 ddj</div>
<div>Dave Johnson</div>
<div>Brown University</div>
<div>
<div><blockquote type="cite"><div><div><div>On Nov 2, 2017, at 10:43 AM, John Hanks <<a href="mailto:griznog@gmail.com" target="_blank">griznog@gmail.com</a>> wrote:</div></div></div>
<div><div><div><div dir="ltr">Thanks all for the suggestions.
<div> </div>
<div>Having our metadata NSDs fill up was what prompted this exercise, but space was previously feed up on those by switching them from metadata+data to metadataOnly and using a policy to migrate files out of that pool. So these now have about 30% free space (more if you include fragmented space). The restripe attempt is just to make a final move of any remaining data off those devices. All the NSDs now have free space on them.</div>
<div> </div>
<div>df -i shows inode usage at about 84%, so plenty of free inodes for the filesystem as a whole.</div>
<div> </div>
<div>We did have old .quota files laying around but removing them didn't have any impact. </div>
<div> </div>
<div>mmlsfileset fs -L -i is taking a while to complete, I'll let it simmer while getting to work.</div>
<div> </div>
<div>mmrepquota does show about a half-dozen filesets that have hit their quota for space (we don't set quotas on inodes). Once I'm settled in this morning I'll try giving them a little extra space and see what happens.</div>
<div> </div>
<div>jbh</div>
<div> </div></div>
<div>
<div>On Thu, Nov 2, 2017 at 4:19 AM, Oesterlin, Robert <span dir="ltr"><<a href="mailto:Robert.Oesterlin@nuance.com" target="_blank">Robert.Oesterlin@nuance.com</a>></span> wrote:
<blockquote style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div lang="EN-US" link="#0563C1" vlink="#954F72"><div><p>One thing that I\u2019ve run into before is that on older file systems you had the \u201c*.quota\u201d files in the file system root. If you upgraded the file system to a newer version (so these files aren\u2019t used) - There was a bug at one time where these didn\u2019t get properly migrated during a restripe. Solution was to just remove them</p>
<p> </p>
<div><p> </p>
<p><span style="font-size:12.0pt">Bob Oesterlin</span></p>
<p><span style="font-size:12.0pt">Sr Principal Storage Engineer, Nuance</span></p></div>
<p> </p>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in"><p><b><span style="font-size:12pt">From: </span></b><span style="font-size:12pt"><<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">gpfsug-discuss-bounces@spectr<wbr>umscale.org</a>> on behalf of John Hanks <<a href="mailto:griznog@gmail.com" target="_blank">griznog@gmail.com</a>><br><b>Reply-To: </b>gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.<wbr>org</a>><br><b>Date: </b>Wednesday, November 1, 2017 at 5:55 PM<br><b>To: </b>gpfsug <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.<wbr>org</a>><br><b>Subject: </b>[EXTERNAL] [gpfsug-discuss] mmrestripefs "No space left on device"</span></p></div>
<div><p><span> </span></p></div>
<div><div><p><span><a name="m_697047665309176941_m_7792501056088615300_m_6737186114830475911__MailOriginalBody">Hi all,</a></span></p></div>
<div><p><span><span> </span></span></p></div>
<div><p><span><span>I'm trying to do a restripe after setting some nsds to metadataOnly and I keep running into this error:</span></span></p></div>
<div><p><span><span> </span></span></p></div>
<div><p><span><span>Scanning user file metadata ...</span></span></p></div>
<div><p><span><span> 0.01 % complete on Wed Nov 1 15:36:01 2017 ( 40960 inodes with total 531689 MB data processed)</span></span></p></div>
<div><p><span><span>Error processing user file metadata. </span></span></p></div>
<div><p><span><span>Check file '/var/mmfs/tmp/gsfs0.pit.inter<wbr>estingInodes.12888779708' on scg-gs0 for inodes with broken disk addresses or failures.</span></span></p></div>
<div><p><span><span>mmrestripefs: Command failed. Examine previous error messages to determine cause.</span></span></p></div>
<div><p><span><span> </span></span></p></div>
<div><p><span><span>The file it points to says:</span></span></p></div>
<div><p><span><span> </span></span></p></div>
<div><div><p><span><span>This inode list was generated in the Parallel Inode Traverse on Wed Nov 1 15:36:06 2017</span></span></p></div>
<div><p><span><span>INODE_NUMBER DUMMY_INFO SNAPSHOT_ID ISGLOBAL_SNAPSHOT INDEPENDENT_FSETID MEMO(INODE_FLAGS FILE_TYPE [ERROR])</span></span></p></div>
<div><p><span><span> 53504 0:0 0 1 0 illreplicated REGULAR_FILE RESERVED Error: 28 No space left on device</span></span></p></div></div>
<div><p><span><span> </span></span></p></div>
<div><p><span><span> </span></span></p></div>
<div><p><span><span>/var on the node I am running this on has > 128 GB free, all the NSDs have plenty of free space, the filesystem being restriped has plenty of free space and if I watch the node while running this no filesystem on it even starts to get full. Could someone tell me where mmrestripefs is attempting to write and/or how to point it at a different location?</span></span></p></div>
<div><p><span><span> </span></span></p></div>
<div><p><span><span>Thanks,</span></span></p></div>
<div><p><span><span> </span></span></p></div>
<div><p><span><span>jbh</span></span></p></div></div></div></div></blockquote></div></div></div></div><span>______________________________<wbr>_________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=WDtkF9zLTGGYqFnVnJ3rywZM6KHROA4FpMYi6cUkkKY&m=hKtOnoUDijNQoFnSlxQfek9m6h2qKbqjcCswbjHg2-E&s=n5P1NWESV2GUb3EXICXGj62_QDAPfSAWVPz_i59CNKk&e=" target="_blank">spectrumscale.org</a></span><br><a href="http://gpfsug.org/mailman" target="_blank">http://gpfsug.org/mailman</a>/list<wbr>info/gpfsug-discuss</div></blockquote></div></div></div><br>______________________________<wbr>_________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=WDtkF9zLTGGYqFnVnJ3rywZM6KHROA4FpMYi6cUkkKY&m=hKtOnoUDijNQoFnSlxQfek9m6h2qKbqjcCswbjHg2-E&s=n5P1NWESV2GUb3EXICXGj62_QDAPfSAWVPz_i59CNKk&e=" rel="noreferrer" target="_blank">spectrumscale.org</a><br><a href="http://gpfsug.org/mailman" target="_blank">http://gpfsug.org/mailman</a>/list<wbr>info/gpfsug-discuss<br> </blockquote></div></div></div></div>
<div><font face="Default Monospace,Courier New,Courier,monospace" size="2"><div><div class="h5">______________________________<wbr>_________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank">spectrumscale.org</a><br></div></div><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=WDtkF9zLTGGYqFnVnJ3rywZM6KHROA4FpMYi6cUkkKY&m=hKtOnoUDijNQoFnSlxQfek9m6h2qKbqjcCswbjHg2-E&s=j7eYU1VnwYXrTnflbJki13EfnMjqAro0RdCiLkVrgzE&e=" target="_blank">https://urldefense.proofpoint.<wbr>com/v2/url?u=http-3A__gpfsug.<wbr>org_mailman_listinfo_gpfsug-<wbr>2Ddiscuss&d=DwICAg&c=jf_<wbr>iaSHvJObTbx-siA1ZOg&r=<wbr>WDtkF9zLTGGYqFnVnJ3rywZM6KHROA<wbr>4FpMYi6cUkkKY&m=<wbr>hKtOnoUDijNQoFnSlxQfek9m6h2qKb<wbr>qjcCswbjHg2-E&s=<wbr>j7eYU1VnwYXrTnflbJki13EfnMjqAr<wbr>o0RdCiLkVrgzE&e=</a></font></div></blockquote>
<div dir="ltr"> </div></div><br>
<br>______________________________<wbr>_________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/<wbr>listinfo/gpfsug-discuss</a><br>
<br></blockquote></div><br></div>