<font size=2 face="sans-serif">so as you already figured out.. as long
your 10GbE is saturated.. it will take time to transfer all the data over
the wire.. </font><br><font size=2 face="sans-serif">"all the data" .. is the key
here.. in older releases /file system versions..we only flagged "data
update miss" or "meta data update miss" in the MD of a file,
when something was written to the file during some NSDs 're missing/down...
</font><br><font size=2 face="sans-serif">means.. when all the disks are back
" active " -.we scan the file systmes meta data to find
all files with one of these flags .. and resync the data..
</font><br><font size=2 face="sans-serif">means, even if you only change some
Bytes of a 10 TB file .. the whole 10TB file needs to be resynced </font><br><br><font size=2 face="sans-serif">with 4.2 we introduced "rapid repair"
.. (note, you need to be on that demon level and you need to update you
file system version) </font><br><font size=2 face="sans-serif">So with Rapid Repair.. we not only flag
MD or D update miss, we write an extra bit on every disk address, which
has changed.. so that now... when after a side failure (or disk outage)
we are able to find all files , which has changed (like before in
the good old days) but now - we know, which disk address has changed and
we just need to sync that changed data </font><br><font size=2 face="sans-serif">in other words.. if you 've changed
1MB of a 10 TB file .. only that 1 MB is re-synced</font><br><br><font size=2 face="sans-serif">you can check, if rapid repair is in
place by mmlsfs command (enable disable by mmchfs ) </font><br><br><font size=2 face="sans-serif">of course.. if everything( every disk
address) has changed during our NSD've been down... rapid repair
won't help ..but depending on the amount of your changed data rate .. it
will definitively shorten your sync times in the future .. </font><br><br><font size=2 face="sans-serif">cheers</font><br><br><div><font size=2 face="sans-serif">Mit freundlichen Gren / Kind regards</font><br><br><font size=2 face="sans-serif"> <br>Olaf Weiser<br> <br>EMEA Storage Competence Center Mainz, German / IBM Systems, Storage Platform,<br>-------------------------------------------------------------------------------------------------------------------------------------------<br>IBM Deutschland<br>IBM Allee 1<br>71139 Ehningen<br>Phone: +49-170-579-44-66<br>E-Mail: olaf.weiser@de.ibm.com<br>-------------------------------------------------------------------------------------------------------------------------------------------<br>IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter<br>Geschftsfhrung: Martina Koederitz (Vorsitzende), Susanne Peter, Norbert
Janzen, Dr. Christian Keller, Ivo Koerner, Markus Koerner<br>Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart,
HRB 14562 / WEEE-Reg.-Nr. DE 99369940 </font><br><br><br><br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">Valdis Kletnieks <Valdis.Kletnieks@vt.edu></font><br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">gpfsug-discuss@spectrumscale.org</font><br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">11/18/2016 08:06 PM</font><br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">[gpfsug-discuss]
mmchdisk performance/behavior in a stretch cluster
config?</font><br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr noshade><br><br><br><tt><font size=2>So as a basis for our archive solution, we're using
a GPFS cluster<br>in a stretch configuration, with 2 sites separated by about 20ms worth<br>of 10G link. Each end has 2 protocol servers doing NFS and 3 NSD
servers.<br>Identical disk arrays and LTFS/EE at both ends, and all metadata and<br>userdata are replicated to both sites.<br><br>We had a fiber issue for about 8 hours yesterday, and as expected (since
there<br>are only 5 quorum nodes, 3 local and 2 at the far end) the far end fell
off the<br>cluster and down'ed all the NSDs on the remote arrays.<br><br>There's about 123T of data at each end, 6 million files in there so far.<br><br>So after the fiber came back up after a several-hour downtime, I<br>did the 'mmchdisk archive start -a'. That was at 17:45 yesterday.<br>I'm now 20 hours in, at:<br><br> 62.15 % complete on Fri Nov 18 13:52:59 2016 ( 4768429
inodes with total 173675926 MB data processed)<br> 62.17 % complete on Fri Nov 18 13:53:20 2016 ( 4769416
inodes with total 173710731 MB data processed)<br> 62.18 % complete on Fri Nov 18 13:53:40 2016 ( 4772481
inodes with total 173762456 MB data processed)<br><br>network statistics indicate that the 3 local NSDs are all tossing out<br>packets at about 400Mbytes/second, which means the 10G pipe is pretty damned<br>close to totally packed full, and the 3 remotes are sending back ACKs<br>of all the data.<br><br>Rough back-of-envelop calculations indicate that (a) if I'm at 62% after<br>20 hours, it will take 30 hours to finish and (b) a 10G link takes about<br>29 hours at full blast to move 123T of data. So it certainly *looks*<br>like it's resending everything.<br><br>And that's even though at least 100T of that 123T is test data that was<br>written by one of our users back on Nov 12/13, and thus theoretically *should*<br>already have been at the remote site.<br><br>Any ideas what's going on here?<br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br><br></font></tt><br><br></div><BR>