Can I ask what the fascination with zvols is? Using a copy-on-write file system to underpin another block based file system seems counterintuitive. Perhaps I've missed something vital, in which case I'd be delighted to have my eyes opened!<br><br><div class="gmail_quote"><div dir="ltr">On Tue, 14 Mar 2017, 00:06 Aaron Knister, <<a href="mailto:aaron.s.knister@nasa.gov">aaron.s.knister@nasa.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I was doing this in testing (with fantastic performance too) until I<br class="gmail_msg">
realized the issue with ZFS's behavior with direct io on zvols (e.g. not<br class="gmail_msg">
flushing a write to stable storage after acknowledging it to GPFS).<br class="gmail_msg">
After setting the sync=always parameter to not lose data in the event of<br class="gmail_msg">
a crash or power outage the write performance became unbearably slow<br class="gmail_msg">
(under 100MB/s of writes for an 8+2 RAIDZ2 if I recall correctly). I<br class="gmail_msg">
even tried adding a battery-backed PCIe write cache<br class="gmail_msg">
(<a href="http://www.netlist.com/products/vault-memory-storage/expressvault-pcIe-ev3/default.aspx" rel="noreferrer" class="gmail_msg" target="_blank">http://www.netlist.com/products/vault-memory-storage/expressvault-pcIe-ev3/default.aspx</a>)<br class="gmail_msg">
as a log device to the zpool but the performance was still really slow.<br class="gmail_msg">
I posted to the ZFS mailing list asking about how to optimize for a<br class="gmail_msg">
large block streaming workload but I didn't many bites<br class="gmail_msg">
(<a href="http://list.zfsonlinux.org/pipermail/zfs-discuss/2016-February/024851.html" rel="noreferrer" class="gmail_msg" target="_blank">http://list.zfsonlinux.org/pipermail/zfs-discuss/2016-February/024851.html</a>).<br class="gmail_msg">
<br class="gmail_msg">
I've got an RFE open with IBM<br class="gmail_msg">
(<a href="https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=84994" rel="noreferrer" class="gmail_msg" target="_blank">https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=84994</a>)<br class="gmail_msg">
to see if the behavior of GPFS could be changed such that it would issue<br class="gmail_msg">
explicit cache flushes that would allow it to work with ZFS (it might<br class="gmail_msg">
even be beneficial in FPO environments too).<br class="gmail_msg">
<br class="gmail_msg">
-Aaron<br class="gmail_msg">
<br class="gmail_msg">
On 3/13/17 4:44 PM, Buterbaugh, Kevin L wrote:<br class="gmail_msg">
> Hi All,<br class="gmail_msg">
><br class="gmail_msg">
> Two things:<br class="gmail_msg">
><br class="gmail_msg">
> 1) Paul\u2019s suggestion to look at the nsddevices script was the answer I<br class="gmail_msg">
> needed to fix my mmcrfs issue. Thanks.<br class="gmail_msg">
><br class="gmail_msg">
> 2) I am also interested in hearing if anyone is using ZFS to create the<br class="gmail_msg">
> equivalent of RAID 8+2P hardware RAID 6 LUNs and presenting that to GPFS<br class="gmail_msg">
> to use as disks?<br class="gmail_msg">
><br class="gmail_msg">
> Thanks\u2026<br class="gmail_msg">
><br class="gmail_msg">
> Kevin<br class="gmail_msg">
><br class="gmail_msg">
>> On Mar 11, 2017, at 4:37 AM, Daniel Kidger <<a href="mailto:daniel.kidger@uk.ibm.com" class="gmail_msg" target="_blank">daniel.kidger@uk.ibm.com</a><br class="gmail_msg">
>> <mailto:<a href="mailto:daniel.kidger@uk.ibm.com" class="gmail_msg" target="_blank">daniel.kidger@uk.ibm.com</a>>> wrote:<br class="gmail_msg">
>><br class="gmail_msg">
>> On the subject of using zvols for software Raid/ replication, can ask<br class="gmail_msg">
>> as a quick poll, how many people are doing this?<br class="gmail_msg">
>><br class="gmail_msg">
>> And any feedback on stability, tuning and performance?<br class="gmail_msg">
>><br class="gmail_msg">
>> Daniel<br class="gmail_msg">
>> IBM Technical Presales<br class="gmail_msg">
>><br class="gmail_msg">
>> > On 10 Mar 2017, at 22:44, Aaron Knister <<a href="mailto:aaron.s.knister@nasa.gov" class="gmail_msg" target="_blank">aaron.s.knister@nasa.gov</a> <mailto:<a href="mailto:aaron.s.knister@nasa.gov" class="gmail_msg" target="_blank">aaron.s.knister@nasa.gov</a>>> wrote:<br class="gmail_msg">
>> ><br class="gmail_msg">
>> > Those look like zvol's. Out of curiosity have you set sync=always on the<br class="gmail_msg">
>> > filesystem root or zvols themselves? It's my understanding that without<br class="gmail_msg">
>> > that you risk data loss since GPFS won't ever cause a sync to be issued<br class="gmail_msg">
>> > to the zvol for zfs to flush acknowledged but uncommitted writes.<br class="gmail_msg">
>> ><br class="gmail_msg">
>> > -Aaron<br class="gmail_msg">
>> ><br class="gmail_msg">
>> >> On 3/10/17 4:36 PM, Sanchez, Paul wrote:<br class="gmail_msg">
>> >> See:<br class="gmail_msg">
>> >> <a href="https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adm.doc/bl1adm_nsddevices.htm" rel="noreferrer" class="gmail_msg" target="_blank">https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adm.doc/bl1adm_nsddevices.htm</a><br class="gmail_msg">
>> >><br class="gmail_msg">
>> >><br class="gmail_msg">
>> >><br class="gmail_msg">
>> >> GPFS has a limited set of device search specs it uses to find connected<br class="gmail_msg">
>> >> NSDs. When using exotic devices, you need to whitelist the devices<br class="gmail_msg">
>> >> yourself using the user exit script at /var/mmfs/etc/nsddevices.<br class="gmail_msg">
>> >><br class="gmail_msg">
>> >><br class="gmail_msg">
>> >><br class="gmail_msg">
>> >> *From:*<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" class="gmail_msg" target="_blank">gpfsug-discuss-bounces@spectrumscale.org</a><br class="gmail_msg">
>> <mailto:<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" class="gmail_msg" target="_blank">gpfsug-discuss-bounces@spectrumscale.org</a>><br class="gmail_msg">
>> >> [mailto:<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" class="gmail_msg" target="_blank">gpfsug-discuss-bounces@spectrumscale.org</a>] *On Behalf Of<br class="gmail_msg">
>> >> *Buterbaugh, Kevin L<br class="gmail_msg">
>> >> *Sent:* Friday, March 10, 2017 3:44 PM<br class="gmail_msg">
>> >> *To:* gpfsug main discussion list<br class="gmail_msg">
>> >> *Subject:* [gpfsug-discuss] mmcrfs issue<br class="gmail_msg">
>> >><br class="gmail_msg">
>> >><br class="gmail_msg">
>> >><br class="gmail_msg">
>> >> Hi All,<br class="gmail_msg">
>> >><br class="gmail_msg">
>> >><br class="gmail_msg">
>> >><br class="gmail_msg">
>> >> We are testing out some flash storage. I created a couple of NSDs<br class="gmail_msg">
>> >> successfully (?):<br class="gmail_msg">
>> >><br class="gmail_msg">
>> >><br class="gmail_msg">
>> >><br class="gmail_msg">
>> >> root@nec:~/gpfs# mmlsnsd -F<br class="gmail_msg">
>> >><br class="gmail_msg">
>> >><br class="gmail_msg">
>> >><br class="gmail_msg">
>> >> File system Disk name NSD servers<br class="gmail_msg">
>> >><br class="gmail_msg">
>> >> ---------------------------------------------------------------------------<br class="gmail_msg">
>> >><br class="gmail_msg">
>> >> (free disk) nsd1 nec<br class="gmail_msg">
>> >><br class="gmail_msg">
>> >> (free disk) nsd2 nec<br class="gmail_msg">
>> >><br class="gmail_msg">
>> >><br class="gmail_msg">
>> >><br class="gmail_msg">
>> >> root@nec:~/gpfs#<br class="gmail_msg">
>> >><br class="gmail_msg">
>> >><br class="gmail_msg">
>> >><br class="gmail_msg">
>> >> So I tried to create a filesystem:<br class="gmail_msg">
>> >><br class="gmail_msg">
>> >><br class="gmail_msg">
>> >><br class="gmail_msg">
>> >> root@nec:~/gpfs# mmcrfs gpfs0 -F ~/gpfs/flash.stanza -A yes -B 1M -j<br class="gmail_msg">
>> >> scatter -k all -m 1 -M 3 -Q no -r 1 -R 3 -T /gpfs0<br class="gmail_msg">
>> >><br class="gmail_msg">
>> >> GPFS: 6027-441 Unable to open disk 'nsd2' on node nec.<br class="gmail_msg">
>> >><br class="gmail_msg">
>> >> GPFS: 6027-441 Unable to open disk 'nsd1' on node nec.<br class="gmail_msg">
>> >><br class="gmail_msg">
>> >> No such device<br class="gmail_msg">
>> >><br class="gmail_msg">
>> >> No such device<br class="gmail_msg">
>> >><br class="gmail_msg">
>> >> GPFS: 6027-538 Error accessing disks.<br class="gmail_msg">
>> >><br class="gmail_msg">
>> >> mmcrfs: 6027-1200 tscrfs failed. Cannot create gpfs0<br class="gmail_msg">
>> >><br class="gmail_msg">
>> >> mmcrfs: 6027-1639 Command failed. Examine previous error messages to<br class="gmail_msg">
>> >> determine cause.<br class="gmail_msg">
>> >><br class="gmail_msg">
>> >> root@nec:~/gpfs#<br class="gmail_msg">
>> >><br class="gmail_msg">
>> >><br class="gmail_msg">
>> >><br class="gmail_msg">
>> >> Does this output from readdescraw look normal?<br class="gmail_msg">
>> >><br class="gmail_msg">
>> >><br class="gmail_msg">
>> >><br class="gmail_msg">
>> >> root@nec:~/gpfs# mmfsadm test readdescraw /dev/zd16<br class="gmail_msg">
>> >><br class="gmail_msg">
>> >> NSD descriptor in sector 64 of /dev/zd16<br class="gmail_msg">
>> >><br class="gmail_msg">
>> >> NSDid: 0A0023D258C1C02C format version: 1403 Label:<br class="gmail_msg">
>> >><br class="gmail_msg">
>> >> Paxos sector: -1931478434 number of sectors: 8192 isPdisk: 0<br class="gmail_msg">
>> >><br class="gmail_msg">
>> >> Comment: NSD descriptor for <unknown> Thu Mar 9 14:50:52 2017<br class="gmail_msg">
>> >><br class="gmail_msg">
>> >> No Disk descriptor in sector 96 of /dev/zd16<br class="gmail_msg">
>> >><br class="gmail_msg">
>> >> No FS descriptor in sector 2048 of /dev/zd16<br class="gmail_msg">
>> >><br class="gmail_msg">
>> >> root@nec:~/gpfs# mmfsadm test readdescraw /dev/zd32<br class="gmail_msg">
>> >><br class="gmail_msg">
>> >> NSD descriptor in sector 64 of /dev/zd32<br class="gmail_msg">
>> >><br class="gmail_msg">
>> >> NSDid: 0A0023D258C1C02B format version: 1403 Label:<br class="gmail_msg">
>> >><br class="gmail_msg">
>> >> Paxos sector: -1880562609 number of sectors: 8192 isPdisk: 0<br class="gmail_msg">
>> >><br class="gmail_msg">
>> >> Comment: NSD descriptor for <unknown> Thu Mar 9 14:50:51 2017<br class="gmail_msg">
>> >><br class="gmail_msg">
>> >> No Disk descriptor in sector 96 of /dev/zd32<br class="gmail_msg">
>> >><br class="gmail_msg">
>> >> No FS descriptor in sector 2048 of /dev/zd32<br class="gmail_msg">
>> >><br class="gmail_msg">
>> >> root@nec:~/gpfs#<br class="gmail_msg">
>> >><br class="gmail_msg">
>> >><br class="gmail_msg">
>> >><br class="gmail_msg">
>> >> Thanks in advance, all\u2026<br class="gmail_msg">
>> >><br class="gmail_msg">
>> >><br class="gmail_msg">
>> >><br class="gmail_msg">
>> >> Kevin<br class="gmail_msg">
>> >><br class="gmail_msg">
>> >> \u2014<br class="gmail_msg">
>> >><br class="gmail_msg">
>> >> Kevin Buterbaugh - Senior System Administrator<br class="gmail_msg">
>> >><br class="gmail_msg">
>> >> Vanderbilt University - Advanced Computing Center for Research and Education<br class="gmail_msg">
>> >><br class="gmail_msg">
>> >> <a href="mailto:Kevin.Buterbaugh@vanderbilt.edu" class="gmail_msg" target="_blank">Kevin.Buterbaugh@vanderbilt.edu</a> <mailto:<a href="mailto:Kevin.Buterbaugh@vanderbilt.edu" class="gmail_msg" target="_blank">Kevin.Buterbaugh@vanderbilt.edu</a>><br class="gmail_msg">
>> >> <mailto:<a href="mailto:Kevin.Buterbaugh@vanderbilt.edu" class="gmail_msg" target="_blank">Kevin.Buterbaugh@vanderbilt.edu</a>> - (615)875-9633<br class="gmail_msg">
>> >><br class="gmail_msg">
>> >><br class="gmail_msg">
>> >><br class="gmail_msg">
>> >><br class="gmail_msg">
>> >><br class="gmail_msg">
>> >><br class="gmail_msg">
>> >><br class="gmail_msg">
>> >><br class="gmail_msg">
>> >><br class="gmail_msg">
>> >> _______________________________________________<br class="gmail_msg">
>> >> gpfsug-discuss mailing list<br class="gmail_msg">
>> >> gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" class="gmail_msg" target="_blank">spectrumscale.org</a> <<a href="http://spectrumscale.org" rel="noreferrer" class="gmail_msg" target="_blank">http://spectrumscale.org</a>><br class="gmail_msg">
>> >> <a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" class="gmail_msg" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class="gmail_msg">
>> >><br class="gmail_msg">
>> ><br class="gmail_msg">
>> > --<br class="gmail_msg">
>> > Aaron Knister<br class="gmail_msg">
>> > NASA Center for Climate Simulation (Code 606.2)<br class="gmail_msg">
>> > Goddard Space Flight Center<br class="gmail_msg">
>> > (301) 286-2776<br class="gmail_msg">
>> > _______________________________________________<br class="gmail_msg">
>> > gpfsug-discuss mailing list<br class="gmail_msg">
>> > gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" class="gmail_msg" target="_blank">spectrumscale.org</a> <<a href="http://spectrumscale.org" rel="noreferrer" class="gmail_msg" target="_blank">http://spectrumscale.org</a>><br class="gmail_msg">
>> > <a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" class="gmail_msg" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class="gmail_msg">
>> ><br class="gmail_msg">
>> Unless stated otherwise above:<br class="gmail_msg">
>> IBM United Kingdom Limited - Registered in England and Wales with<br class="gmail_msg">
>> number 741598.<br class="gmail_msg">
>> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU<br class="gmail_msg">
>><br class="gmail_msg">
>> _______________________________________________<br class="gmail_msg">
>> gpfsug-discuss mailing list<br class="gmail_msg">
>> gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" class="gmail_msg" target="_blank">spectrumscale.org</a> <<a href="http://spectrumscale.org" rel="noreferrer" class="gmail_msg" target="_blank">http://spectrumscale.org</a>><br class="gmail_msg">
>> <a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" class="gmail_msg" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class="gmail_msg">
><br class="gmail_msg">
><br class="gmail_msg">
><br class="gmail_msg">
> _______________________________________________<br class="gmail_msg">
> gpfsug-discuss mailing list<br class="gmail_msg">
> gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" class="gmail_msg" target="_blank">spectrumscale.org</a><br class="gmail_msg">
> <a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" class="gmail_msg" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class="gmail_msg">
><br class="gmail_msg">
<br class="gmail_msg">
--<br class="gmail_msg">
Aaron Knister<br class="gmail_msg">
NASA Center for Climate Simulation (Code 606.2)<br class="gmail_msg">
Goddard Space Flight Center<br class="gmail_msg">
(301) 286-2776<br class="gmail_msg">
_______________________________________________<br class="gmail_msg">
gpfsug-discuss mailing list<br class="gmail_msg">
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" class="gmail_msg" target="_blank">spectrumscale.org</a><br class="gmail_msg">
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" class="gmail_msg" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class="gmail_msg">
</blockquote></div>