<div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div dir="ltr" >Does your internally developed application do only reads during in its monthly run? If so, can you change it to use PROT_READ flag during the mmap call? That way you will not get the 10-block decompression IO overhead and your files will remain compressed. The decompression happens upon pagein's only if the mmap call includes the PROT_WRITE flag (or upon actual writes for non-mmap IOs).</div>
<div dir="ltr" > </div>
<div dir="ltr" >Leo</div>
<div dir="ltr" > </div>
<blockquote data-history-content-modified="1" dir="ltr" style="border-left:solid #aaaaaa 2px; margin-left:5px; padding-left:5px; direction:ltr; margin-right:0px" >----- Original message -----<br>From: Zachary Giles <zgiles@gmail.com><br>Sent by: gpfsug-discuss-bounces@spectrumscale.org<br>To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>Cc:<br>Subject: Re: [gpfsug-discuss] Questions about mmap GPFS and compression<br>Date: Tue, Feb 14, 2017 8:10 AM<br>
<div><font face="Default Monospace,Courier New,Courier,monospace" size="2" >Hi Leo,<br><br>I agree with your view on compression and what it should be used for,<br>in general. The read bandwidth amplification is definitely something<br>we're seeing.<br><br>Just a little more background on the files:<br>The files themselves are not "cold" (archive), however, they are very<br>lightly used. The data set is thousands of files that are each<br>100-200GB, totaling about a PB. the read pattern is a few GB from<br>about 20% of the files once a month. So the total read is only several<br>TB out of a PB every month. ( approximate ). We can get a compression<br>of about 5:1 using GPFS with these files, so we can gain back 800TB<br>with compression. The total run time of the app (reading all those all<br>chunks, when uncompressed) is maybe an hour total.<br><br>Although leaving the files uncompressed would let the app work,<br>there's a huge gain to be had if we can make compression work by<br>saving ~800TB As it's such a small amount of data read each time, and<br>also not too predictable (it's semi-random historical), and as the<br>length of the job is short enough, it's hard to justify decompressing<br>large chunks of the system to run 1 job. I would have to decompress<br>200TB to read 10TB, recompress them, and decompress a different<br>(overlapping) 200TB next month. The compression / decompression of<br>sizable portions of the data takes days.<br><br>I think there maybe more of an issue that just performance though..<br>The decompression thread is running, internal file metadata is read<br>fine, most of the file is read fine. Just at times it gets stuck.. the<br>decompression thread is running in GPFS, the app is polling, it just<br>never comes back with the block. I feel like there's a race condition<br>here where a block is read, available for the app, but thrown away<br>before the app can read it, only to be decompressed again.<br>It's strange how some block positions are slow (expected) and others<br>just never come back (it will poll for days on a certain address).<br>However, reading the file in-order is fine.<br><br>Is this a block caching issue? Can we tune up the amount of blocks kept?<br>I think with mmap the blocks are not kept in page pool, correct?<br><br>-Zach<br><br>On Sat, Feb 11, 2017 at 5:23 PM, Leo Luan <leoluan@us.ibm.com> wrote:<br>> Hi Zachary,<br>><br>> When a compressed file is mmapped, each 4K read in your tests causes the<br>> accessed part of the file to be decompressed (in the granularity of 10 GPFS<br>> blocks). For usual file sizes, the parts being accessed will be<br>> decompressed and IOs speed will be normal except for the first 4K IO in each<br>> 10-GPFS-block group. For very large files, a large percentage of small<br>> random IOs may keep getting amplified to 10-block decompression IO for a<br>> long time. This is probably what happened in your mmap application run.<br>><br>> The suggestion is to not compress files until they have become cold (not<br>> likely to be accessed any time soon) and avoid compressing very large files<br>> that may be accessed through mmap later. The product already has a built-in<br>> protection preventing compression of files that are mmapped at compression<br>> time. You can add an exclude rule in the compression policy run for files<br>> that are identified to have mmap performance issues (in case they get<br>> mmapped after being compressed in a periodical policy run).<br>><br>> Leo Luan<br>><br>> From: Zachary Giles <zgiles@gmail.com><br>> To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>> Date: 02/10/2017 01:57 PM<br>> Subject: [gpfsug-discuss] Questions about mmap GPFS and compression<br>> Sent by: gpfsug-discuss-bounces@spectrumscale.org<br>> ________________________________<br>><br>><br>><br>> Hello All,<br>><br>> I've been seeing some less than desirable behavior with mmap and<br>> compression in GPFS. Curious if others see similar or have any ideas<br>> if this is accurate..<br>> The guys here want me to open an IBM ticket, but I figured I'd see if<br>> anyone has had this experience before.<br>><br>> We have an internally developed app that runs on our cluster<br>> referencing data sitting in GPFS. It is using mmap to access the files<br>> due to a library we're using that requires it.<br>><br>> If we run the app against some data on GPFS, it performs well..<br>> finishing in a few minutes time -- Great. However, if we compress the<br>> file (in GPFS), the app is still running after 2 days time.<br>> stracing the app shows that is polling on a file descriptor, forever..<br>> as if a data block is still pending.<br>><br>> I know mmap is supported with compression according to the manual<br>> (with some stipulations), and that performance is expected to be much<br>> less since it's more large-block oriented due to decompressed in<br>> groups.. no problem. But it seems like some data should get returned.<br>><br>> I'm surprised to find that a very small amount of data is sitting in<br>> the buffers (mmfsadm dump buffers) in reference to the inodes. The<br>> decompression thread is running continuously, while the app is still<br>> polling for data from memory and sleeping, retrying, sleeping, repeat.<br>><br>> What I believe is happening is that the 4k pages are being pulled out<br>> of large decompression groups from an mmap read request, put in the<br>> buffer, then the compression group data is thrown away since it has<br>> the result it wants, only to need another piece of data that would<br>> have been in that group slightly later, which is recalled, put in the<br>> buffer.. etc. Thus an infinite slowdown. Perhaps also the data is<br>> expiring out of the buffer before the app has a chance to read it. I<br>> can't tell. In any case, the app makes zero progress.<br>><br>> I tried without our app, using fio.. mmap on an uncompressed file with<br>> 1 thread 1 iodepth, random read, 4k blocks, yields ~76MB/s (not<br>> impressive). However, on a compressed file it is only 20KB/s max. (<br>> far less impressive ). Reading a file using aio etc is over 3GB/s on a<br>> single thread without even trying.<br>><br>> What do you think?<br>> Anyone see anything like this? Perhaps there are some tunings to waste<br>> a bit more memory on cached blocks rather than make decompression<br>> recycle?<br>><br>> I've searched back the archives a bit. There's a May 2013 thread about<br>> slowness as well. I think we're seeing much much less than that. Our<br>> page pools are of decent size. Its not just slowness, it's as if the<br>> app never gets a block back at all. ( We could handle slowness .. )<br>><br>> Thanks. Open to ideas..<br>><br>> -Zach Giles<br>><br>><br>> _______________________________________________<br>> gpfsug-discuss mailing list<br>> gpfsug-discuss at spectrumscale.org<br>> <a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank" >http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>><br><br><br><br>--<br>Zach Giles<br>zgiles@gmail.com<br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank" >http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></font><br> </div></blockquote>
<div dir="ltr" > </div></div><BR>