<div dir="ltr"><div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">Thanks for your insights.</div><div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">Alex, we did as you mentioned but after using mmimportfs there are a lot of errors on all commands having to do with filesystems:</div><div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">GPFS: 6027-419 Failed to read a file system descriptor.<br>There is an input or output error</div><div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">that occurs for:</div><div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">mmlsfs <br></div><div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">mmlsdisk</div><div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">mmdf <br></div><div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">obviously, fs won't mount.</div><div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small"><br></div></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><img src="https://docs.google.com/uc?export=download&id=1qBobw7lgHUL5kvclOrp0EeujITqmX4ZE&revid=0B2nrOyWM3zSNeVA3M1VXN1U2bzJncmxQSVV0cDFiOGUyYXVRPQ"><br></div></div></div></div></div>
<br><div class="gmail_quote">2018-01-27 0:02 GMT-05:00 Alex Levin <span dir="ltr"><<a href="mailto:alevin@gmail.com" target="_blank">alevin@gmail.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><span style="font-family:sans-serif;font-size:13.3333px">Steve,</span><div><span style="font-family:sans-serif;font-size:13.3333px"> I've read that </span><span style="font-family:sans-serif;font-size:13.3333px">"mmfsctl suspend or suspend-write"</span><span style="font-family:sans-serif;font-size:13.3333px"> should be executed, but in real life it is impossible for DR scenario.</span></div><div><span style="font-family:sans-serif;font-size:13.3333px"><br></span></div><div><span style="font-family:sans-serif;font-size:13.3333px">We tested both cases,</span></div><div><span style="font-family:sans-serif;font-size:13.3333px"> the graceful one when the failover to another site is planned, applications stopped and i/o suspended </span></div><div><span style="font-family:sans-serif;font-size:13.3333px">and the case when there was no advanced notice about the disaster in the primary site.</span></div><div><span style="font-family:sans-serif;font-size:13.3333px"><br></span></div><div><span style="font-family:sans-serif;font-size:13.3333px">Both worked and for the second case various loads were simulated including heavy writes and reads/writes.</span></div><div><font face="sans-serif"><span style="font-size:13.3333px">In disaster model as expected some data were lost (due to incomplete writes, replication latency ... ), </span></font></div><div><font face="sans-serif"><span style="font-size:13.3333px"> but mmfsck was always able to repair and the applications databases located on the affected filesystem were in an acceptable state.</span></font></div><div><font face="sans-serif"><span style="font-size:13.3333px"><br></span></font></div><div><span style="font-family:sans-serif;font-size:13.3333px"><br></span></div><div><span style="font-family:sans-serif;font-size:13.3333px">it is possible that we were just lucky and the test case didn't cover all possible scenarios.</span></div><div><span style="font-family:sans-serif;font-size:13.3333px"><br></span></div><div><span style="font-family:sans-serif;font-size:13.3333px">Harold, </span></div><div><span style="font-family:sans-serif;font-size:13.3333px"> In our case, it is Linux, not AIX, but it shouldn't matter</span></div><div><span style="font-size:13.3333px;font-family:sans-serif"> And our </span>DR <span style="font-size:13.3333px;font-family:sans-serif"> cluster is fully configured ( different IPs, hostnames and cluster name ) and running without filesystems at all or with a differently named filesystem.</span><br></div><div><span style="font-size:13.3333px;font-family:sans-serif"><br></span></div><div><font face="sans-serif"><span style="font-size:13.3333px"> Before running mmimport , make sure that all expected LUNs are visible and writable.</span></font></div><div><font face="sans-serif"><span style="font-size:13.3333px"> You can verify if the device is correct by reading first blocks of the device ( for example dd if=/dev/NSD_LUN_device bs=1M count=16 | strings ) </span></font></div><div><font face="sans-serif"><span style="font-size:13.3333px"><br></span></font></div><div><font face="sans-serif"><span style="font-size:13.3333px">not sure where you are "</span></font><span style="font-family:tahoma,sans-serif"> </span><span style="font-family:tahoma,sans-serif">moved the mmsdrfs" </span></div><div><span style="font-family:tahoma,sans-serif">you don't need to move/modify </span><span style="font-family:tahoma,sans-serif">mmsdrfs file on the target ( disaster recovery ) cluster </span></div><div><span style="font-family:tahoma,sans-serif"><br></span></div><div><span style="font-family:tahoma,sans-serif">Just copy the one from primary to /tmp or /var/tmp and try to run<span style="background-color:rgb(255,255,255)"><font color="#0000ff"> </font></span></span><span style="background-color:rgb(255,255,255)"><font color="#0000ff"><span style="font-size:12.8px">mmimportfs fs_name -i copy_of_mmsdrfs /tmp/</span><span style="font-family:tahoma,sans-serif">mmsdrfs </span></font></span></div><div><br></div><div><br></div><div><span style="font-family:sans-serif;font-size:13.3333px"><br></span></div><div><span style="font-family:sans-serif;font-size:13.3333px">Glen, as Harold has no IP connectivity between the clusters "</span><span style="font-family:sans-serif">mmfsctl syncFSconfig" is not an option ... </span></div><div><span style="font-family:sans-serif"><br></span></div><div><span style="font-family:sans-serif">Thanks</span></div><div><font face="sans-serif">--Alex</font></div><div><font face="sans-serif"><br></font></div><div><span style="font-family:sans-serif"><br></span></div><div><span style="font-family:sans-serif;font-size:13.3333px"><br></span></div></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="h5">On Fri, Jan 26, 2018 at 4:04 PM, Steve Xiao <span dir="ltr"><<a href="mailto:sxiao@us.ibm.com" target="_blank">sxiao@us.ibm.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5"><span style="font-size:10pt;font-family:sans-serif">When using this method
of replication, you need to either issue "mmfsctl suspend or suspend-write"
command before replication or setup a single consistency group for all
LUNs. This is needed to ensure replica contain a consistent
copy of GPFS data. </span><br><span style="font-size:10pt;font-family:sans-serif"><br>Steve Y. Xiao<br></span><br><tt><span style="font-size:10pt"><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">gpfsug-discuss-bounces@spectru<wbr>mscale.org</a>
wrote on 01/26/2018 03:21:23 PM:<br><br>> From: <a href="mailto:gpfsug-discuss-request@spectrumscale.org" target="_blank">gpfsug-discuss-request@spectru<wbr>mscale.org</a></span></tt><br><tt><span style="font-size:10pt">> To: <a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.o<wbr>rg</a></span></tt><br><tt><span style="font-size:10pt">> Date: 01/26/2018 03:21 PM</span></tt><br><tt><span style="font-size:10pt">> Subject: gpfsug-discuss Digest,
Vol 72, Issue 69</span></tt><br><tt><span style="font-size:10pt">> Sent by: <a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">gpfsug-discuss-bounces@spectru<wbr>mscale.org</a></span></tt><br><tt><span style="font-size:10pt">> <br>> Send gpfsug-discuss mailing list submissions to<br>> <a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.<wbr>org</a><br>> <br>> To subscribe or unsubscribe via the World Wide Web, visit<br>> </span></tt><a href="https://urldefense.proofpoint.com/v2/url?" target="_blank"><tt><span style="font-size:10pt">https://urldefense.proofpoint<wbr>.com/v2/url?</span></tt></a><tt><span style="font-size:10pt"><br>> u=http-3A__gpfsug.org_mailman_<wbr>listinfo_gpfsug-2Ddiscuss&d=Dw<wbr>ICAg&c=jf_iaSHvJObTbx-<br>> siA1ZOg&r=ck4PYlaRFvCcNKlfHMPh<wbr>oA&m=ddkAIwRPQmQKBLLh6nBzhdt-<br>> OKoJwOucQ8oaQet8mkE&s=Emkr3VzC<wbr>YTecA6E61hAk1AeB6ka34dGAYip6fG<wbr>aKuwU&e=<br>> or, via email, send a message with subject or body 'help' to<br>> <a href="mailto:gpfsug-discuss-request@spectrumscale.org" target="_blank">gpfsug-discuss-request@spectr<wbr>umscale.org</a><br>> <br>> You can reach the person managing the list at<br>> <a href="mailto:gpfsug-discuss-owner@spectrumscale.org" target="_blank">gpfsug-discuss-owner@spectrum<wbr>scale.org</a><br>> <br>> When replying, please edit your Subject line so it is more specific<br>> than "Re: Contents of gpfsug-discuss digest..."<br>> <br>> <br>> Today's Topics:<br>> <br>> 1. Re: storage-based replication for Spectrum Scale (Harold
Morales)<br>> 2. Re: storage-based replication for Spectrum Scale (Glen
Corneau)<br>> <br>> <br>> ------------------------------<wbr>------------------------------<wbr>----------<br>> <br>> Message: 1<br>> Date: Fri, 26 Jan 2018 13:29:09 -0500<span><br>> From: Harold Morales <<a href="mailto:hmorales@optimizeit.co" target="_blank">hmorales@optimizeit.co</a>><br>> To: gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.<wbr>org</a>><br></span><span>> Subject: Re: [gpfsug-discuss] storage-based replication for Spectrum<br>> Scale<br></span>> Message-ID:<br>> <<a href="mailto:CAGQnKWaAJAuFxYT2PzjMW47vuOsMg4evUAT8%2BSV49rrVkkQLBg@mail.gmail.com" target="_blank">CAGQnKWaAJAuFxYT2PzjMW47vuOs<wbr>Mg4evUAT8+SV49rrVkkQLBg@mail.g<wbr>mail.com</a>><br>> Content-Type: text/plain; charset="utf-8"<div><div class="m_-7928888765953349550h5"><br>> <br>> Hi Alex, This set up seems close to what I am trying to achieve.<br>> <br>> With regards to this kind of replication: any prereqs need to be met
in the<br>> target environment for this to work? for example, should disk devices<br>> naming on AIX be the same as in the source environment? when importing
the<br>> mmsdrfs file, how is Scale going to know which disks should it assign
to<br>> the cluster? by their hdisk name alone?<br>> <br>> Thanks again,<br>> <br>> <br>> <br>> 2018-01-24 2:30 GMT-05:00 Alex Levin <<a href="mailto:alevin@gmail.com" target="_blank">alevin@gmail.com</a>>:<br>> <br>> > Hi,<br>> ><br>> > We are using a similar type of replication.<br>> > I assume the site B is the cold site prepared for DR<br>> ><br>> > The storage layer is EMC VMAX and the LUNs are replicated with
SRDF.<br>> > All LUNs ( NSDs ) of the gpfs filesystem are in the same VMAX
replication<br>> > group to ensure consistency.<br>> ><br>> > The cluster name, IP addresses , hostnames of the cluster
nodes are<br>> > different on another site - it can be a pre-configured cluster
without<br>> > gpfs filesystems or with another filesystem.<br>> > Same names and addresses shouldn't be a problem.<br>> ><br>> > Additionally to the replicated LUNs/NSDs you need to deliver
copy<br>> > of /var/mmfs/gen/mmsdrfs file from A to B site.<br>> > There is no need to replicate it in real-time, only after the
change of<br>> > the cluster configuration.<br>> ><br>> > To activate site B - present replicated LUNs to the nodes
in the DR<br>> > cluster and run mmimportfs as "mmimportfs fs_name
-i copy_of_mmsdrfs"<br>> ><br>> > Tested with multiples LUNs and filesystems on various workloads
- seems<br>> > to be working<br>> ><br>> > --Alex<br>> ><br>> ><br>> > On Wed, Jan 24, 2018 at 1:33 AM, Harold Morales <<a href="mailto:hmorales@optimizeit.co" target="_blank">hmorales@optimizeit.co</a>><br>> > wrote:<br>> ><br>> >> Thanks for answering.<br>> >><br>> >> Essentially, the idea being explored is to replicate LUNs
between<br>> >> identical storage hardware (HP 3PAR volumesrein) on both
sites. There is an<br>> >> IP connection between storage boxes but not between servers
on both sites,<br>> >> there is a dark fiber connecting both sites. Here they dont
want to explore<br>> >> the idea of a scaled-based.<br>> >><br>> >><br>> >> ______________________________<wbr>_________________<br>> >> gpfsug-discuss mailing list<br>> >> gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank">spectrumscale.org</a><br></div></div>> >> </span></tt><a href="https://urldefense.proofpoint.com/v2/url?" target="_blank"><tt><span style="font-size:10pt">https://urldefense.proofpoint.<wbr>com/v2/url?</span></tt></a><tt><span style="font-size:10pt"><br>> u=http-3A__gpfsug.org_mailman_<wbr>listinfo_gpfsug-2Ddiscuss&d=Dw<wbr>ICAg&c=jf_iaSHvJObTbx-<br>> siA1ZOg&r=ck4PYlaRFvCcNKlfHMPh<wbr>oA&m=ddkAIwRPQmQKBLLh6nBzhdt-<br>> OKoJwOucQ8oaQet8mkE&s=Emkr3VzC<wbr>YTecA6E61hAk1AeB6ka34dGAYip6fG<wbr>aKuwU&e=<span><br>> >><br>> >><br>> ><br>> > ______________________________<wbr>_________________<br>> > gpfsug-discuss mailing list<br>> > gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank">spectrumscale.org</a><br>> > </span></span></tt><a href="https://urldefense.proofpoint.com/v2/url?" target="_blank"><tt><span style="font-size:10pt">https://urldefense.proofpoint.<wbr>com/v2/url?</span></tt></a><tt><span style="font-size:10pt"><br>> u=http-3A__gpfsug.org_mailman_<wbr>listinfo_gpfsug-2Ddiscuss&d=Dw<wbr>ICAg&c=jf_iaSHvJObTbx-<br>> siA1ZOg&r=ck4PYlaRFvCcNKlfHMPh<wbr>oA&m=ddkAIwRPQmQKBLLh6nBzhdt-<br>> OKoJwOucQ8oaQet8mkE&s=Emkr3VzC<wbr>YTecA6E61hAk1AeB6ka34dGAYip6fG<wbr>aKuwU&e=<br>> ><br>> ><br>> -------------- next part --------------<br>> An HTML attachment was scrubbed...<br>> URL: <</span></tt><a href="https://urldefense.proofpoint.com/v2/url?" target="_blank"><tt><span style="font-size:10pt">https://urldefense.proofpoint<wbr>.com/v2/url?</span></tt></a><tt><span style="font-size:10pt"><br>> u=http-3A__gpfsug.org_pipermai<wbr>l_gpfsug-2Ddiscuss_attachments<wbr>_20180126_1503995b_attachment-<wbr>2D0001.html&d=DwICAg&c=jf_<wbr>iaSHvJObTbx-<br>> siA1ZOg&r=ck4PYlaRFvCcNKlfHMPh<wbr>oA&m=ddkAIwRPQmQKBLLh6nBzhdt-<br>> OKoJwOucQ8oaQet8mkE&s=SKdMmQae<wbr>8uzHNWZq3vuRTp5UVwYFeeusLAxtba<wbr>posX0&e=><br>> <br>> ------------------------------<br>> <br>> Message: 2<br>> Date: Fri, 26 Jan 2018 14:21:15 -0600<br>> From: "Glen Corneau" <<a href="mailto:gcorneau@us.ibm.com" target="_blank">gcorneau@us.ibm.com</a>><span><br>> To: gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.<wbr>org</a>><br></span><span>> Subject: Re: [gpfsug-discuss] storage-based replication for Spectrum<br>> Scale<br></span>> Message-ID:<br>> <OFEAC48C60.2890CAEF-ON862582<wbr>21.006F3F43-86258221.<br>> <a href="mailto:006FCEEE@notes.na.collabserv.com" target="_blank">006FCEEE@notes.na.collabserv.c<wbr>om</a>><br>> <br>> Content-Type: text/plain; charset="us-ascii"<span><br>> <br>> Scale will walk across all discovered disks upon start time and attempt
to <br>> read the NSD identifiers from the disks. Once it finds them,
it makes a <br>> local map file that correlates the NSD id and the hdiskX identifier.
The <br>> names do not have to be the same as either the source cluster or even
from <br>> node-to-node.<br>> <br>> The main thing to keep in mind is to keep the file system definitions
in <br>> sync between the source and destination clusters. The "syncFSconfig"
user <br>> exit is the best way to do it because it's automatic. You generally
<br>> shouldn't be shuffling the mmsdrfs file between sites, that's what
the <br>> "mmfsctl syncFSconfig" does for you, on a per-file system
basis.<br>> <br>> GPFS+AIX customers have been using this kind of storage replication
for <br>> over 10 years, it's business as usual.<br>> <br>> ------------------<br>> Glen Corneau<br>> Power Systems<br>> Washington Systems Center<br>> <a href="mailto:gcorneau@us.ibm.com" target="_blank">gcorneau@us.ibm.com</a><br>> <br>> <br>> <br>> <br>> <br></span><span>> From: Harold Morales <<a href="mailto:hmorales@optimizeit.co" target="_blank">hmorales@optimizeit.co</a>><br>> To: gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.<wbr>org</a>><br>> Date: 01/26/2018 12:30 PM<br>> Subject: Re: [gpfsug-discuss] storage-based
replication for <br>> Spectrum Scale<br>> Sent by: <a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">gpfsug-discuss-bounces@spectr<wbr>umscale.org</a><br>> <br>> <br>> <br></span><div><div class="m_-7928888765953349550h5">> Hi Alex, This set up seems close to what I am trying to achieve.<br>> <br>> With regards to this kind of replication: any prereqs need to be met
in <br>> the target environment for this to work? for example, should disk
devices <br>> naming on AIX be the same as in the source environment? when importing
the <br>> mmsdrfs file, how is Scale going to know which disks should it assign
to <br>> the cluster? by their hdisk name alone?<br>> <br>> Thanks again,<br>> <br>> <br>> <br>> 2018-01-24 2:30 GMT-05:00 Alex Levin <<a href="mailto:alevin@gmail.com" target="_blank">alevin@gmail.com</a>>:<br>> Hi, <br>> <br>> We are using a similar type of replication.<br>> I assume the site B is the cold site prepared for DR<br>> <br>> The storage layer is EMC VMAX and the LUNs are replicated with SRDF.<br>> All LUNs ( NSDs ) of the gpfs filesystem are in the same VMAX replication
<br>> group to ensure consistency.<br>> <br>> The cluster name, IP addresses , hostnames of the cluster nodes
are <br>> different on another site - it can be a pre-configured cluster without
<br>> gpfs filesystems or with another filesystem.<br>> Same names and addresses shouldn't be a problem.<br>> <br>> Additionally to the replicated LUNs/NSDs you need to deliver copy
<br>> of /var/mmfs/gen/mmsdrfs file from A to B site.<br>> There is no need to replicate it in real-time, only after the change
of <br>> the cluster configuration.<br>> <br>> To activate site B - present replicated LUNs to the nodes in
the DR <br>> cluster and run mmimportfs as "mmimportfs fs_name
-i copy_of_mmsdrfs"<br>> <br>> Tested with multiples LUNs and filesystems on various workloads
- seems <br>> to be working <br>> <br>> --Alex<br>> <br>> <br>> <br>> On Wed, Jan 24, 2018 at 1:33 AM, Harold Morales <<a href="mailto:hmorales@optimizeit.co" target="_blank">hmorales@optimizeit.co</a>>
<br>> wrote:<br>> Thanks for answering.<br>> <br>> Essentially, the idea being explored is to replicate LUNs between
<br>> identical storage hardware (HP 3PAR volumesrein) on both sites. There
is <br>> an IP connection between storage boxes but not between servers on
both <br>> sites, there is a dark fiber connecting both sites. Here they dont
want to <br>> explore the idea of a scaled-based.<br>> <br>> <br>> ______________________________<wbr>_________________<br>> gpfsug-discuss mailing list<br>> gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank">spectrumscale.org</a><br></div></div>> </span></tt><a href="https://urldefense.proofpoint.com/v2/url?" target="_blank"><tt><span style="font-size:10pt">https://urldefense.proofpoint.<wbr>com/v2/url?</span></tt></a><tt><span style="font-size:10pt"><br>> u=http-3A__gpfsug.org_mailman_<wbr>listinfo_gpfsug-2Ddiscuss&d=Dw<wbr>ICAg&c=jf_iaSHvJObTbx-<br>> siA1ZOg&r=ck4PYlaRFvCcNKlfHMPh<wbr>oA&m=ddkAIwRPQmQKBLLh6nBzhdt-<br>> OKoJwOucQ8oaQet8mkE&s=Emkr3VzC<wbr>YTecA6E61hAk1AeB6ka34dGAYip6fG<wbr>aKuwU&e=<span><br>> <br>> <br>> <br>> ______________________________<wbr>_________________<br>> gpfsug-discuss mailing list<br>> gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank">spectrumscale.org</a><br>> </span></span></tt><a href="https://urldefense.proofpoint.com/v2/url?" target="_blank"><tt><span style="font-size:10pt">https://urldefense.proofpoint.<wbr>com/v2/url?</span></tt></a><tt><span style="font-size:10pt"><br>> u=http-3A__gpfsug.org_mailman_<wbr>listinfo_gpfsug-2Ddiscuss&d=Dw<wbr>ICAg&c=jf_iaSHvJObTbx-<br>> siA1ZOg&r=ck4PYlaRFvCcNKlfHMPh<wbr>oA&m=ddkAIwRPQmQKBLLh6nBzhdt-<br>> OKoJwOucQ8oaQet8mkE&s=Emkr3VzC<wbr>YTecA6E61hAk1AeB6ka34dGAYip6fG<wbr>aKuwU&e=<span><br>> <br>> ______________________________<wbr>_________________<br>> gpfsug-discuss mailing list<br>> gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank">spectrumscale.org</a><br>> </span></span></tt><a href="https://urldefense.proofpoint.com/v2/url?" target="_blank"><tt><span style="font-size:10pt">https://urldefense.proofpoint.<wbr>com/v2/url?</span></tt></a><tt><span style="font-size:10pt"><br>> u=http-3A__gpfsug.org_mailman_<wbr>listinfo_gpfsug-2Ddiscuss&d=Dw<wbr>ICAg&c=jf_iaSHvJObTbx-<br>> siA1ZOg&r=d-<br>> vphLEe_UlGazP6RdYAyyAA3Qv5S9IR<wbr>VNuO1i9vjJc&m=VbfWaftYSVjx8fMb<wbr>2vHGBi6XUhDJOsKf_dKOX3J8s1A&s=<wbr>p2mGvPrlPLO1oyEh-<br>> GeVJiVS49opBwwCFs-FKQrQ7rc&e=<br>> <br>> <br>> <br>> <br>> <br>> -------------- next part --------------<br>> An HTML attachment was scrubbed...<br>> URL: <</span></tt><a href="https://urldefense.proofpoint.com/v2/url?" target="_blank"><tt><span style="font-size:10pt">https://urldefense.proofpoint<wbr>.com/v2/url?</span></tt></a><tt><span style="font-size:10pt"><br>> u=http-3A__gpfsug.org_pipermai<wbr>l_gpfsug-2Ddiscuss_attachments<wbr>_20180126_e291af63_attachment.<wbr>html&d=DwICAg&c=jf_iaSHvJObTbx<wbr>-<br>> siA1ZOg&r=ck4PYlaRFvCcNKlfHMPh<wbr>oA&m=ddkAIwRPQmQKBLLh6nBzhdt-<br>> OKoJwOucQ8oaQet8mkE&s=bYnf-7v0<wbr>CxYUkGth-QaVeUQdIlG8f1Gro-hwOx<wbr>ok7Qw&e=><br>> -------------- next part --------------<br>> A non-text attachment was scrubbed...<br>> Name: not available<br>> Type: image/jpeg<br>> Size: 26117 bytes<br>> Desc: not available<br>> URL: <</span></tt><a href="https://urldefense.proofpoint.com/v2/url?" target="_blank"><tt><span style="font-size:10pt">https://urldefense.proofpoint<wbr>.com/v2/url?</span></tt></a><tt><span style="font-size:10pt"><br>> u=http-3A__gpfsug.org_pipermai<wbr>l_gpfsug-2Ddiscuss_attachments<wbr>_20180126_e291af63_attachment.<wbr>jpe&d=DwICAg&c=jf_iaSHvJObTbx-<br>> siA1ZOg&r=ck4PYlaRFvCcNKlfHMPh<wbr>oA&m=ddkAIwRPQmQKBLLh6nBzhdt-<br>> OKoJwOucQ8oaQet8mkE&s=jYdnqhQB<wbr>lnpf58oxunzBcTs9XdcbeOtLDQdgnA<wbr>SidDA&e=><br>> <br>> ------------------------------<span><br>> <br>> ______________________________<wbr>_________________<br>> gpfsug-discuss mailing list<br>> gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank">spectrumscale.org</a><br>> </span></span></tt><a href="https://urldefense.proofpoint.com/v2/url?" target="_blank"><tt><span style="font-size:10pt">https://urldefense.proofpoint.<wbr>com/v2/url?</span></tt></a><tt><span style="font-size:10pt"><br>> u=http-3A__gpfsug.org_mailman_<wbr>listinfo_gpfsug-2Ddiscuss&d=Dw<wbr>ICAg&c=jf_iaSHvJObTbx-<br>> siA1ZOg&r=ck4PYlaRFvCcNKlfHMPh<wbr>oA&m=ddkAIwRPQmQKBLLh6nBzhdt-<br>> OKoJwOucQ8oaQet8mkE&s=Emkr3VzC<wbr>YTecA6E61hAk1AeB6ka34dGAYip6fG<wbr>aKuwU&e=<br>> <br>> <br>> End of gpfsug-discuss Digest, Vol 72, Issue 69<br>> ******************************<wbr>****************<br>> <br></span></tt><br><br>______________________________<wbr>_________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
</div></div><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/list<wbr>info/gpfsug-discuss</a><br>
<br></blockquote></div><br></div>
<br>______________________________<wbr>_________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/<wbr>listinfo/gpfsug-discuss</a><br>
<br></blockquote></div><br></div>