<font size=2 face="Arial">Hi Alexander,</font><br><br><font size=2 face="Arial">1. >>When writing to GPFS directly
I'm able to write ~1800 files / second in a test setup. <br>>>This is roughly the same on the protocol nodes (NSD client), as
well as on the ESS IO nodes (NSD server). <br></font><br><font size=2 face="Arial">2. >> When writing to the NFS export
on the protocol node itself (to avoid any network effects) I'm only able
to write ~230 files / second.<br></font><br><font size=2 face="Arial">IMHO #2, writing to the NFS export on the
protocol node should be same as #1. Protocol node is also a NSD client
and when you write from a protocol node, it will use the NSD protocol to
write to the ESS IO nodes. In #1, you cite seeing ~1800 files from protocol
node and in #2 you cite seeing ~230 file/sec which seem to contradict each
other. </font><br><br><font size=2 face="Arial">>>Writing to the NFS export from another
node (now including network latency) gives me ~220 files / second.</font><br><br><font size=2 face="Arial">IMHO, this workload "single client,
single thread, small files, single directory - tar xf" is synchronous
is nature and will result in single outstanding file to be sent from the
NFS client to the CES node. Hence, the performance will be limited by network
latency/capability between the NFS client and CES node for small IO size
(~5KB file size). </font><br><br><font size=2 face="Arial">Also, what is the network interconnect/interface
between the NFS client and CES node? Is the network 10GigE since
@220 file/s for 5KiB file-size will saturate 1 x 10GigE link. </font><br><br><font size=2 face="Arial">220 files/sec * 5KiB file size ==> ~1.126
GB/s. </font><br><br><font size=2 face="Arial">>> I'm aware that 'the real thing'
would be to work with larger files in a multithreaded manner from multiple
nodes - and that this scenario will scale quite well.</font><br><br><font size=2 face="Arial">Yes, larger file-size + multiple threads
+ multiple NFS client nodes will help to scale performance further by having
more NFS I/O requests scheduled/pipelined over the network and processed
on the CES nodes. </font><br><br><font size=2 face="Arial">>> I just want to ensure that I'm not
missing something obvious over reiterating that massage to customers.<br></font><br><font size=2 face="Arial">Adding NFS experts/team, for advise. </font><br><br><font size=2 face="Arial">My two cents.</font><br><br><font size=2 face="Arial">Best Regards,</font><br><font size=2 face="Arial">-Kums</font><br><br><br><br><br><br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">"Alexander Saupp"
<Alexander.Saupp@de.ibm.com></font><br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">gpfsug-discuss@spectrumscale.org</font><br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">10/15/2018 02:20 PM</font><br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">[gpfsug-discuss]
Tuning: single client, single thread, small
files - native Scale vs NFS</font><br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr noshade><br><br><br><font size=2>Dear Spectrum Scale mailing list,</font><font size=3><br></font><font size=2><br>I'm part of IBM Lab Services - currently i'm having multiple customers
asking me for optimization of a similar workloads.</font><font size=3><br></font><font size=2><br>The task is to tune a Spectrum Scale system (comprising ESS and CES protocol
nodes) for the following workload: <br>A single Linux NFS client mounts an NFS export, extracts a flat tar archive
with lots of ~5KB files. <br>I'm measuring the speed at which those 5KB files are written (`time tar
xf archive.tar`). </font><font size=3><br></font><font size=2><br>I do understand that Spectrum Scale is not designed for such workload (single
client, single thread, small files, single directory), and that such benchmark
in not appropriate to benmark the system. <br>Yet I find myself explaining the performance for such scenario (git clone..)
quite frequently, as customers insist that optimization of that scenario
would impact individual users as it shows task duration.<br>I want to make sure that I have optimized the system as much as possible
for the given workload, and that I have not overlooked something obvious.</font><font size=3><br><br></font><font size=2><br>When writing to GPFS directly I'm able to write ~1800 files / second in
a test setup. <br>This is roughly the same on the protocol nodes (NSD client), as well as
on the ESS IO nodes (NSD server). <br>When writing to the NFS export on the protocol node itself (to avoid any
network effects) I'm only able to write ~230 files / second.<br>Writing to the NFS export from another node (now including network latency)
gives me ~220 files / second.</font><font size=3><br><br></font><font size=2><br>There seems to be a huge performance degradation by adding NFS-Ganesha
to the software stack alone. I wonder what can be done to minimize the
impact.</font><font size=3><br><br></font><font size=2><br>- Ganesha doesn't seem to support 'async' or 'no_wdelay' options... anything
equivalent available?<br>- Is there and expected advantage of using the network-latency tuned profile,
as opposed to the ESS default throughput-performance profile?<br>- Are there other relevant Kernel params?<br>- Is there an expected advantage of raising the number of threads (NSD
server (nsd*WorkerThreads) / NSD client (workerThreads) / Ganesha (NB_WORKER))
for the given workload (single client, single thread, small files)?<br>- Are there other relevant GPFS params?<br>- Impact of Sync replication, disk latency, etc is understood. <br>- I'm aware that 'the real thing' would be to work with larger files in
a multithreaded manner from multiple nodes - and that this scenario will
scale quite well.<br>I just want to ensure that I'm not missing something obvious over reiterating
that massage to customers.</font><font size=3><br></font><font size=2><br>Any help was greatly appreciated - thanks much in advance!<br>Alexander Saupp<br>IBM Germany</font><font size=3><br><br></font><font size=2><br>Mit freundlichen Gren / Kind regards</font><font size=3><br><b><br>Alexander Saupp</b><br></font><font size=3 color=#0060a0><br>IBM Systems, Storage Platform, EMEA Storage Competence Center</font><table width=800 style="border-collapse:collapse;"><tr height=8><td width=800 colspan=4 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><hr><tr height=8><td width=800 colspan=4 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><img align=bottom src=cid:_1_0831ED280831EABC007C099285258327 width=1 height=1 style="border:0px solid;"><tr height=8><td width=88 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0>Phone:</font><td width=322 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0>+49
7034-643-1512</font><td width=250 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0>IBM
Deutschland GmbH</font><td width=140 rowspan=4 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><div align=right><img align=bottom src=cid:_1_08322E1C08322A1C007C099285258327 style="border:0px solid;"></div><tr height=8><td width=88 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0>Mobile:</font><td width=322 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0>+49-172
7251072</font><td width=250 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0>Am
Weiher 24</font><tr height=8><td width=88 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0>Email:</font><td width=322 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0>alexander.saupp@de.ibm.com</font><td width=250 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0>65451
Kelsterbach</font><tr height=8><td width=88 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><img align=bottom src=cid:_1_08321DE408321B78007C099585258327 width=1 height=1 style="border:0px solid;"><td width=322 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><img align=bottom src=cid:_1_083223780832210C007C099585258327 width=1 height=1 style="border:0px solid;"><td width=250 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0>Germany</font><tr height=8><td width=800 colspan=4 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><hr><tr height=8><td width=800 colspan=4 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#a2a2a2>IBM
Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter<br>Geschftsfhrung: Matthias Hartmann (Vorsitzender), Norbert Janzen, Stefan
Lutz, Nicole Reimer, Dr. Klaus Seifert, Wolfgang Wendt<br>Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart,
HRB 14562 / WEEE-Reg.-Nr. DE 99369940 </font></table><br><font size=3><br></font><tt><font size=2>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br></font></tt><br><BR>