<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
        {font-family:"Cambria Math";
        panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
        {font-family:Calibri;
        panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
        {margin:0in;
        margin-bottom:.0001pt;
        font-size:12.0pt;
        font-family:"Times New Roman";}
a:link, span.MsoHyperlink
        {mso-style-priority:99;
        color:blue;
        text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
        {mso-style-priority:99;
        color:purple;
        text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
        {mso-style-priority:34;
        margin-top:0in;
        margin-right:0in;
        margin-bottom:0in;
        margin-left:.5in;
        margin-bottom:.0001pt;
        font-size:12.0pt;
        font-family:"Times New Roman";}
span.hoenzb
        {mso-style-name:hoenzb;}
span.EmailStyle18
        {mso-style-type:personal-reply;
        font-family:Calibri;
        color:windowtext;}
span.msoIns
        {mso-style-type:export-only;
        mso-style-name:"";
        text-decoration:underline;
        color:teal;}
.MsoChpDefault
        {mso-style-type:export-only;
        font-size:10.0pt;}
@page WordSection1
        {size:8.5in 11.0in;
        margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
        {page:WordSection1;}
/* List Definitions */
@list l0
        {mso-list-id:53893230;
        mso-list-type:hybrid;
        mso-list-template-ids:735846728 67698705 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
        {mso-level-text:"%1\)";
        mso-level-tab-stop:none;
        mso-level-number-position:left;
        text-indent:-.25in;}
@list l0:level2
        {mso-level-number-format:alpha-lower;
        mso-level-tab-stop:none;
        mso-level-number-position:left;
        text-indent:-.25in;}
@list l0:level3
        {mso-level-number-format:roman-lower;
        mso-level-tab-stop:none;
        mso-level-number-position:right;
        text-indent:-9.0pt;}
@list l0:level4
        {mso-level-tab-stop:none;
        mso-level-number-position:left;
        text-indent:-.25in;}
@list l0:level5
        {mso-level-number-format:alpha-lower;
        mso-level-tab-stop:none;
        mso-level-number-position:left;
        text-indent:-.25in;}
@list l0:level6
        {mso-level-number-format:roman-lower;
        mso-level-tab-stop:none;
        mso-level-number-position:right;
        text-indent:-9.0pt;}
@list l0:level7
        {mso-level-tab-stop:none;
        mso-level-number-position:left;
        text-indent:-.25in;}
@list l0:level8
        {mso-level-number-format:alpha-lower;
        mso-level-tab-stop:none;
        mso-level-number-position:left;
        text-indent:-.25in;}
@list l0:level9
        {mso-level-number-format:roman-lower;
        mso-level-tab-stop:none;
        mso-level-number-position:right;
        text-indent:-9.0pt;}
ol
        {margin-bottom:0in;}
ul
        {margin-bottom:0in;}
--></style>
</head>
<body bgcolor="white" lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Some interesting discussion here. Perhaps I should have been a bit clearer on what I\u2019m looking at here:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p>&nbsp;</o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">I have 12 servers with 70*4TB drives each \u2013 so the hardware is free. What\u2019s the best strategy for using these as GPFS NSD servers, given that I don\u2019t want to relay on any \u201cbleeding edge\u201d
 technologies.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p>&nbsp;</o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">1) My first choice would be GNR on commodity hardware \u2013 if IBM would give that to us. :-)
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">2) Use standard RAID groups with no replication \u2013 downside is data availability of you lose an NSD and RAID group rebuild time with large disks<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">3) RAID groups with replication \u2013 but I lose a LOT of space (20% for RAID &#43; 50% of what\u2019s left for replication)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">4) No raid groups, single NSD per disk, single failure group per servers, replication. Downside here is I need to restripe every time a disk fails to get the filesystem back to a good state.
 Might be OK using QoS to get the IO impact down<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">5) FPO doesn\u2019t seem to by me anything, as these are straight NSD servers and no computation is going on these servers, and I still must live with the re-stripe.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p>&nbsp;</o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Option (4) seems the best of the \u201cno great options\u201d I have in front of me.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p>&nbsp;</o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-family:Calibri">Bob Oesterlin<br>
Sr Principal Storage Engineer, Nuance<br>
<br>
<o:p></o:p></span></p>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p>&nbsp;</o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p>&nbsp;</o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-family:Calibri;color:black">From: </span>
</b><span style="font-family:Calibri;color:black">&lt;gpfsug-discuss-bounces@spectrumscale.org&gt; on behalf of Zachary Giles &lt;zgiles@gmail.com&gt;<br>
<b>Reply-To: </b>gpfsug main discussion list &lt;gpfsug-discuss@spectrumscale.org&gt;<br>
<b>Date: </b>Wednesday, November 30, 2016 at 10:27 PM<br>
<b>To: </b>gpfsug main discussion list &lt;gpfsug-discuss@spectrumscale.org&gt;<br>
<b>Subject: </b>[EXTERNAL] Re: [gpfsug-discuss] Strategies - servers with local SAS disks<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
</div>
<div>
<p class="MsoNormal">Aaron, Thanks for jumping onboard. It's nice to see others confirming this. Sometimes I feel alone on this topic.&nbsp;
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
<div>
<p class="MsoNormal">It's should also be possible to use ZFS with ZVOLs presented as block devices for a backing store for NSDs. I'm not claiming it's stable, nor a good idea, nor performant.. but should be possible. :) There are various reports about it. Might
 be at least worth looking in to compared to Linux &quot;md raid&quot; if one truly needs an all-software solution that already exists.&nbsp; Something to think about and test over.<o:p></o:p></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
<div>
<p class="MsoNormal">On Wed, Nov 30, 2016 at 11:15 PM, Aaron Knister &lt;<a href="mailto:aaron.s.knister@nasa.gov" target="_blank">aaron.s.knister@nasa.gov</a>&gt; wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal">Thanks Zach, I was about to echo similar sentiments and you saved me a ton of typing :)<br>
<br>
Bob, I know this doesn't help you today since I'm pretty sure its not yet available, but if one scours the interwebs they can find mention of something called Mestor.<br>
<br>
There's very very limited information here:<br>
<br>
- <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__indico.cern.ch_event_531810_contributions_2306222_attachments_1357265_2053960_Spectrum-5FScale-2DHEPIX-5FV1a.pdf&amp;d=CwMFaQ&amp;c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&amp;r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&amp;m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&amp;s=okpUZQo5UPYWvaBHBnDMMc_HKPdMppZvehm2-Wf24ms&amp;e=" target="_blank">
https://indico.cern.ch/event/531810/contributions/2306222/attachments/1357265/2053960/Spectrum_Scale-HEPIX_V1a.pdf</a><br>
- <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__www.yumpu.com_en_document_view_5544551_ibm-2Dsystem-2Dx-2Dgpfs-2Dstorage-2Dserver-2Dstfc&amp;d=CwMFaQ&amp;c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&amp;r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&amp;m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&amp;s=UXlFxGKhufPyXf4aGSI3vVn_FOZ50Kibcgps8lgpzF4&amp;e=" target="_blank">
https://www.yumpu.com/en/document/view/5544551/ibm-system-x-gpfs-storage-server-stfc</a> (slide 20)<br>
<br>
Sounds like if it were available it would fit this use case very well.<br>
<br>
I also had preliminary success with using sheepdog (<a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__sheepdog.github.io_sheepdog_&amp;d=CwMFaQ&amp;c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&amp;r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&amp;m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&amp;s=2TOkgjDy0toVMOUXditrShHtONR_T_twsOMfWVN4lFU&amp;e=" target="_blank">https://sheepdog.github.io/sheepdog/</a>)
 as a backing store for GPFS in a similar situation. It's perhaps at a very high conceptually level similar to Mestor. You erasure code your data across the nodes w/ the SAS disks and then present those block devices to your NSD servers. I proved it could work
 but never tried to to much with it because the requirements changed.<br>
<br>
My money would be on your first option-- creating local RAIDs and then replicating to give you availability in the event a node goes offline.<br>
<br>
-Aaron<br>
<br>
<br>
On 11/30/16 10:59 PM, Zachary Giles wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal">Just remember that replication protects against data availability, not<br>
integrity. GPFS still requires the underlying block device to return<br>
good data.<br>
<br>
If you're using it on plain disks (SAS or SSD), and the drive returns<br>
corrupt data, GPFS won't know any better and just deliver it to the<br>
client. Further, if you do a partial read followed by a write, both<br>
replicas could be destroyed. There's also no efficient way to force use<br>
of a second replica if you realize the first is bad, short of taking the<br>
first entirely offline. In that case while migrating data, there's no<br>
good way to prevent read-rewrite of other corrupt data on your drive<br>
that has the &quot;good copy&quot; while restriping off a faulty drive.<br>
<br>
Ideally RAID would have a goal of only returning data that passed the<br>
RAID algorithm, so shouldn't be corrupt, or made good by recreating from<br>
parity. However, as we all know RAID controllers are definitely prone to<br>
failures as well for many reasons, but at least a drive can go bad in<br>
various ways (bad sectors, slow, just dead, poor SSD cell wear, etc)<br>
without (hopefully) silent corruption..<br>
<br>
Just something to think about while considering replication ..<br>
<br>
<br>
<br>
On Wed, Nov 30, 2016 at 11:28 AM, Uwe Falke &lt;<a href="mailto:UWEFALKE@de.ibm.com" target="_blank">UWEFALKE@de.ibm.com</a><o:p></o:p></p>
<div>
<div>
<p class="MsoNormal">&lt;mailto:<a href="mailto:UWEFALKE@de.ibm.com" target="_blank">UWEFALKE@de.ibm.com</a>&gt;&gt; wrote:<br>
<br>
&nbsp; &nbsp; I have once set up a small system with just a few SSDs in two NSD<br>
&nbsp; &nbsp; servers,<br>
&nbsp; &nbsp; providin a scratch file system in a computing cluster.<br>
&nbsp; &nbsp; No RAID, two replica.<br>
&nbsp; &nbsp; works, as long the admins do not do silly things (like rebooting servers<br>
&nbsp; &nbsp; in sequence without checking for disks being up in between).<br>
&nbsp; &nbsp; Going for RAIDs without GPFS replication protects you against single<br>
&nbsp; &nbsp; disk<br>
&nbsp; &nbsp; failures, but you're lost if just one of your NSD servers goes off.<br>
<br>
&nbsp; &nbsp; FPO makes sense only sense IMHO if your NSD servers are also processing<br>
&nbsp; &nbsp; the data (and then you need to control that somehow).<br>
<br>
&nbsp; &nbsp; Other ideas? what else can you do with GPFS and local disks than<br>
&nbsp; &nbsp; what you<br>
&nbsp; &nbsp; considered? I suppose nothing reasonable ...<br>
<br>
<br>
&nbsp; &nbsp; Mit freundlichen Grüßen / Kind regards<br>
<br>
<br>
&nbsp; &nbsp; Dr. Uwe Falke<br>
<br>
&nbsp; &nbsp; IT Specialist<br>
&nbsp; &nbsp; High Performance Computing Services / Integrated Technology Services /<br>
&nbsp; &nbsp; Data Center Services<br>
&nbsp; &nbsp; -------------------------------------------------------------------------------------------------------------------------------------------<br>
&nbsp; &nbsp; IBM Deutschland<br>
&nbsp; &nbsp; Rathausstr. 7<br>
&nbsp; &nbsp; 09111 Chemnitz<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">&nbsp; &nbsp; Phone: <a href="tel:%2B49%20371%206978%202165" target="_blank">
&#43;49 371 6978 2165</a> &lt;tel:%2B49%20371%206978%202165&gt;<br>
&nbsp; &nbsp; Mobile: <a href="tel:%2B49%20175%20575%202877" target="_blank">&#43;49 175 575 2877</a> &lt;tel:%2B49%20175%20575%202877&gt;<br>
&nbsp; &nbsp; E-Mail: <a href="mailto:uwefalke@de.ibm.com" target="_blank">uwefalke@de.ibm.com</a> &lt;mailto:<a href="mailto:uwefalke@de.ibm.com" target="_blank">uwefalke@de.ibm.com</a>&gt;<br>
&nbsp; &nbsp; -------------------------------------------------------------------------------------------------------------------------------------------<br>
&nbsp; &nbsp; IBM Deutschland Business &amp; Technology Services GmbH / Geschäftsführung:<br>
&nbsp; &nbsp; Frank Hammer, Thorsten Moehring<br>
&nbsp; &nbsp; Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht<br>
&nbsp; &nbsp; Stuttgart,<br>
&nbsp; &nbsp; HRB 17122<br>
<br>
<br>
<br>
<br>
&nbsp; &nbsp; From:&nbsp; &nbsp;&quot;Oesterlin, Robert&quot; &lt;<a href="mailto:Robert.Oesterlin@nuance.com" target="_blank">Robert.Oesterlin@nuance.com</a><br>
&nbsp; &nbsp; &lt;mailto:<a href="mailto:Robert.Oesterlin@nuance.com" target="_blank">Robert.Oesterlin@nuance.com</a>&gt;&gt;<br>
&nbsp; &nbsp; To:&nbsp; &nbsp; &nbsp;gpfsug main discussion list<br>
&nbsp; &nbsp; &lt;<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.org</a><br>
&nbsp; &nbsp; &lt;mailto:<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.org</a>&gt;&gt;<br>
&nbsp; &nbsp; Date:&nbsp; &nbsp;11/30/2016 03:34 PM<br>
&nbsp; &nbsp; Subject:&nbsp; &nbsp; &nbsp; &nbsp; [gpfsug-discuss] Strategies - servers with local SAS<br>
&nbsp; &nbsp; disks<br>
&nbsp; &nbsp; Sent by:&nbsp; &nbsp; &nbsp; &nbsp; <a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">
gpfsug-discuss-bounces@spectrumscale.org</a><br>
&nbsp; &nbsp; &lt;mailto:<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">gpfsug-discuss-bounces@spectrumscale.org</a>&gt;<br>
<br>
<br>
<br>
&nbsp; &nbsp; Looking for feedback/strategies in setting up several GPFS servers with<br>
&nbsp; &nbsp; local SAS. They would all be part of the same file system. The<br>
&nbsp; &nbsp; systems are<br>
&nbsp; &nbsp; all similar in configuration - 70 4TB drives.<br>
<br>
&nbsp; &nbsp; Options I?m considering:<br>
<br>
&nbsp; &nbsp; - Create RAID arrays of the disks on each server (worried about the RAID<br>
&nbsp; &nbsp; rebuild time when a drive fails with 4, 6, 8TB drives)<br>
&nbsp; &nbsp; - No RAID with 2 replicas, single drive per NSD. When a drive fails,<br>
&nbsp; &nbsp; recreate the NSD ? but then I need to fix up the data replication via<br>
&nbsp; &nbsp; restripe<br>
&nbsp; &nbsp; - FPO ? with multiple failure groups -&nbsp; letting the system manage<br>
&nbsp; &nbsp; replica<br>
&nbsp; &nbsp; placement and then have GPFS due the restripe on disk failure<br>
&nbsp; &nbsp; automatically<br>
<br>
&nbsp; &nbsp; Comments or other ideas welcome.<br>
<br>
&nbsp; &nbsp; Bob Oesterlin<br>
&nbsp; &nbsp; Sr Principal Storage Engineer, Nuance<br>
&nbsp; &nbsp; <a href="tel:507-269-0413" target="_blank">507-269-0413</a> &lt;tel:<a href="tel:507-269-0413" target="_blank">507-269-0413</a>&gt;<br>
<br>
&nbsp; &nbsp; &nbsp;_______________________________________________<br>
&nbsp; &nbsp; gpfsug-discuss mailing list<br>
&nbsp; &nbsp; gpfsug-discuss at <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&amp;d=CwMFaQ&amp;c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&amp;r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&amp;m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&amp;s=5YbzvsgvARAfEma_-9ivfs2B950q-viphuylZrZc6Xo&amp;e=" target="_blank">
spectrumscale.org</a> &lt;<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&amp;d=CwMFaQ&amp;c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&amp;r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&amp;m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&amp;s=5YbzvsgvARAfEma_-9ivfs2B950q-viphuylZrZc6Xo&amp;e=" target="_blank">http://spectrumscale.org</a>&gt;<br>
&nbsp; &nbsp; <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&amp;d=CwMFaQ&amp;c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&amp;r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&amp;m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&amp;s=Mk9i5Ca4nkt7TzIPrrNMWMTcAQQDHGQ2uxhEtTsUKG0&amp;e=" target="_blank">
http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
&nbsp; &nbsp; &lt;<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&amp;d=CwMFaQ&amp;c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&amp;r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&amp;m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&amp;s=Mk9i5Ca4nkt7TzIPrrNMWMTcAQQDHGQ2uxhEtTsUKG0&amp;e=" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a>&gt;<br>
<br>
<br>
<br>
<br>
&nbsp; &nbsp; _______________________________________________<br>
&nbsp; &nbsp; gpfsug-discuss mailing list<br>
&nbsp; &nbsp; gpfsug-discuss at <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&amp;d=CwMFaQ&amp;c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&amp;r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&amp;m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&amp;s=5YbzvsgvARAfEma_-9ivfs2B950q-viphuylZrZc6Xo&amp;e=" target="_blank">
spectrumscale.org</a> &lt;<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&amp;d=CwMFaQ&amp;c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&amp;r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&amp;m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&amp;s=5YbzvsgvARAfEma_-9ivfs2B950q-viphuylZrZc6Xo&amp;e=" target="_blank">http://spectrumscale.org</a>&gt;<br>
&nbsp; &nbsp; <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&amp;d=CwMFaQ&amp;c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&amp;r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&amp;m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&amp;s=Mk9i5Ca4nkt7TzIPrrNMWMTcAQQDHGQ2uxhEtTsUKG0&amp;e=" target="_blank">
http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
&nbsp; &nbsp; &lt;<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&amp;d=CwMFaQ&amp;c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&amp;r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&amp;m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&amp;s=Mk9i5Ca4nkt7TzIPrrNMWMTcAQQDHGQ2uxhEtTsUKG0&amp;e=" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a>&gt;<br>
<br>
<br>
<br>
<br>
--<br>
Zach Giles<br>
<a href="mailto:zgiles@gmail.com" target="_blank">zgiles@gmail.com</a> &lt;mailto:<a href="mailto:zgiles@gmail.com" target="_blank">zgiles@gmail.com</a>&gt;<br>
<br>
<br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&amp;d=CwMFaQ&amp;c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&amp;r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&amp;m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&amp;s=5YbzvsgvARAfEma_-9ivfs2B950q-viphuylZrZc6Xo&amp;e=" target="_blank">
spectrumscale.org</a><br>
<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&amp;d=CwMFaQ&amp;c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&amp;r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&amp;m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&amp;s=Mk9i5Ca4nkt7TzIPrrNMWMTcAQQDHGQ2uxhEtTsUKG0&amp;e=" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><span style="color:#888888"><br>
<span class="hoenzb">-- </span><br>
<span class="hoenzb">Aaron Knister</span><br>
<span class="hoenzb">NASA Center for Climate Simulation (Code 606.2)</span><br>
<span class="hoenzb">Goddard Space Flight Center</span><br>
<span class="hoenzb"><a href="tel:%28301%29%20286-2776" target="_blank">(301) 286-2776</a></span></span>
<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"><br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&amp;d=CwMFaQ&amp;c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&amp;r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&amp;m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&amp;s=5YbzvsgvARAfEma_-9ivfs2B950q-viphuylZrZc6Xo&amp;e=" target="_blank">
spectrumscale.org</a><br>
<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&amp;d=CwMFaQ&amp;c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&amp;r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&amp;m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&amp;s=Mk9i5Ca4nkt7TzIPrrNMWMTcAQQDHGQ2uxhEtTsUKG0&amp;e=" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><o:p></o:p></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><br>
<br clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
</div>
<p class="MsoNormal">-- <o:p></o:p></p>
<div>
<p class="MsoNormal">Zach Giles<br>
<a href="mailto:zgiles@gmail.com" target="_blank">zgiles@gmail.com</a><o:p></o:p></p>
</div>
</div>
</div>
</body>
</html>