<div dir="ltr">Hi,<div><br></div><div>the code assumption is that the underlying device has no volatile write cache, i was absolute sure we have that somewhere in the FAQ, but i couldn't find it, so i will talk to somebody to correct this. </div><div>if i understand <a href="https://www.kernel.org/doc/Documentation/block/writeback_cache_control.txt">https://www.kernel.org/doc/Documentation/block/writeback_cache_control.txt</a> correct one could enforce this by setting <span style="white-space:pre-wrap">REQ_FUA, but thats not explicitly set today, at least i can't see it. i will discuss this with one of our devs who owns this code and come back. </span><br></div><div><span style="white-space:pre-wrap"><br></span></div><div><span style="white-space:pre-wrap">sven</span></div><div><span style="white-space:pre-wrap"><br></span></div></div><br><div class="gmail_quote"><div dir="ltr">On Thu, Sep 7, 2017 at 8:05 PM Aaron Knister <<a href="mailto:aaron.s.knister@nasa.gov">aaron.s.knister@nasa.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Thanks Sven. I didn't think GPFS itself was caching anything on that<br>
layer, but it's my understanding that O_DIRECT isn't sufficient to force<br>
I/O to be flushed (e.g. the device itself might have a volatile caching<br>
layer). Take someone using ZFS zvol's as NSDs. I can write() all day log<br>
to that zvol (even with O_DIRECT) but there is absolutely no guarantee<br>
those writes have been committed to stable storage and aren't just<br>
sitting in RAM until an fsync() occurs (or some other bio function that<br>
causes a flush). I also don't believe writing to a SATA drive with<br>
O_DIRECT will force cache flushes of the drive's writeback cache..<br>
although I just tested that one and it seems to actually trigger a scsi<br>
cache sync. Interesting.<br>
<br>
-Aaron<br>
<br>
On 9/7/17 10:55 PM, Sven Oehme wrote:<br>
> I am not sure what exactly you are looking for but all blockdevices are<br>
> opened with O_DIRECT , we never cache anything on this layer .<br>
><br>
><br>
> On Thu, Sep 7, 2017, 7:11 PM Aaron Knister <<a href="mailto:aaron.s.knister@nasa.gov" target="_blank">aaron.s.knister@nasa.gov</a><br>
> <mailto:<a href="mailto:aaron.s.knister@nasa.gov" target="_blank">aaron.s.knister@nasa.gov</a>>> wrote:<br>
><br>
> Hi Everyone,<br>
><br>
> This is something that's come up in the past and has recently resurfaced<br>
> with a project I've been working on, and that is-- it seems to me as<br>
> though mmfsd never attempts to flush the cache of the block devices its<br>
> writing to (looking at blktrace output seems to confirm this). Is this<br>
> actually the case? I've looked at the gpl headers for linux and I don't<br>
> see any sign of blkdev_fsync, blkdev_issue_flush, WRITE_FLUSH, or<br>
> REQ_FLUSH. I'm sure there's other ways to trigger this behavior that<br>
> GPFS may very well be using that I've missed. That's why I'm asking :)<br>
><br>
> I figure with FPO being pushed as an HDFS replacement using commodity<br>
> drives this feature has *got* to be in the code somewhere.<br>
><br>
> -Aaron<br>
><br>
> --<br>
> Aaron Knister<br>
> NASA Center for Climate Simulation (Code 606.2)<br>
> Goddard Space Flight Center<br>
> <a href="tel:(301)%20286-2776" value="+13012862776" target="_blank">(301) 286-2776</a><br>
> _______________________________________________<br>
> gpfsug-discuss mailing list<br>
> gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a> <<a href="http://spectrumscale.org" rel="noreferrer" target="_blank">http://spectrumscale.org</a>><br>
> <a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
><br>
><br>
><br>
> _______________________________________________<br>
> gpfsug-discuss mailing list<br>
> gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
> <a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
><br>
<br>
--<br>
Aaron Knister<br>
NASA Center for Climate Simulation (Code 606.2)<br>
Goddard Space Flight Center<br>
<a href="tel:(301)%20286-2776" value="+13012862776" target="_blank">(301) 286-2776</a><br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
</blockquote></div>