<tt><font size=2>Jonathon,</font></tt><br><br><tt><font size=2>its important to separate the two issues "high
CPU consumption" and "CPU Jitter".</font></tt><br><br><tt><font size=2>As mentioned, we are aware of the CPU jitter issue
and already put several improvements in place. (more to come with the next
release)</font></tt><br><tt><font size=2>Did you try with a lower polling frequency and/or
enabling clock alignment as Mike suggested ? </font></tt><br><br><tt><font size=2>Non-MPI workloads are usually not impacted by CPU
jitter, but might be impacted by high CPU consumption. </font></tt><br><tt><font size=2>But we don't see such such high CPU consumption in
the lab and therefore ask affected customers to get in contact with IBM
support to find the root cause. </font></tt><br><br><br><tt><font size=2>Kind regards<br> <br>Mathias Dietz<br> <br>IBM Spectrum Scale - Release Lead Architect and RAS Architect</font></tt><font size=2 face="sans-serif"><br> <br></font><br><br><tt><font size=2>gpfsug-discuss-bounces@spectrumscale.org wrote on
07/19/2017 07:52:14 PM:<br><br>> From: Jonathon A Anderson <jonathon.anderson@colorado.edu></font></tt><br><tt><font size=2>> To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org></font></tt><br><tt><font size=2>> Date: 07/19/2017 07:52 PM</font></tt><br><tt><font size=2>> Subject: Re: [gpfsug-discuss] mmsysmon.py revisited</font></tt><br><tt><font size=2>> Sent by: gpfsug-discuss-bounces@spectrumscale.org</font></tt><br><tt><font size=2>> <br>> > It might be a problem specific to your system environment or
a <br>> wrong configuration therefore please get in contact with IBM support<br>> to analyze the root cause of the high usage.<br>> <br>> I suspect it’s actually a result of frequent IO interrupts causing
<br>> jitter in conflict with MPI on the shared Intel Omni-Path network,
<br>> in our case.<br>> <br>> We’ve already tried pursuing support on this through our vendor,
<br>> DDN, and got no-where. Eventually we were the ones who tried killing<br>> mmsysmon, and that fixed our problem.<br>> <br>> The official company line of “we don't see significant CPU <br>> consumption by mmsysmon on our test systems” isn’t helping. Do you
<br>> have a test system with OPA?<br>> <br>> ~jonathon<br>> <br>> <br>> On 7/19/17, 7:05 AM, "gpfsug-discuss-bounces@spectrumscale.org
on <br>> behalf of Mathias Dietz" <gpfsug-discuss-bounces@spectrumscale.org
<br>> on behalf of MDIETZ@de.ibm.com> wrote:<br>> <br>> thanks for the feedback. <br>> <br>> Let me clarify what mmsysmon is doing.<br>> Since IBM Spectrum Scale 4.2.1 the mmsysmon process
is used for <br>> the overall health monitoring and CES failover handling.<br>> Even without CES it is an essential part of the system
because <br>> it monitors the individual components and provides health state <br>> information and error events.<br>> <br>> This information is needed by other Spectrum Scale components
<br>> (mmhealth command, the IBM Spectrum Scale GUI, Support tools, <br>> Install Toolkit,..) and therefore disabling mmsysmon will impact them.<br>> <br>> <br>> > It’s a huge problem. I don’t understand why it
hasn’t been given<br>> <br>> > much credit by dev or support.<br>> <br>> Over the last couple of month, the development team
has put a <br>> strong focus on this topic.<br>> <br>> In order to monitor the health of the individual components,
<br>> mmsysmon listens for notifications/callback but also has to do some
polling.<br>> We are trying to reduce the polling overhead constantly
and <br>> replace polling with notifications when possible.<br>> <br>> <br>> Several improvements have been added to 4.2.3, including
the <br>> ability to configure the polling frequency to reduce the overhead.
<br>> (mmhealth config interval)<br>> <br>> See </font></tt><a href=https://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.3/><tt><font size=2>https://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.3/</font></tt></a><tt><font size=2><br>> com.ibm.spectrum.scale.v4r23.doc/bl1adm_mmhealth.htm<br>> In addition a new option has been introduced to clock
align the <br>> monitoring threads in order to reduce CPU jitter.<br>> <br>> <br>> Nevertheless, we don't see significant CPU consumption
by <br>> mmsysmon on our test systems.<br>> <br>> It might be a problem specific to your system environment
or a <br>> wrong configuration therefore please get in contact with IBM support<br>> to analyze the root cause of the high usage.<br>> <br>> Kind regards<br>> <br>> Mathias Dietz<br>> <br>> IBM Spectrum Scale - Release Lead Architect and RAS
Architect<br>> <br>> <br>> <br>> gpfsug-discuss-bounces@spectrumscale.org wrote on 07/18/2017
07:51:21 PM:<br>> <br>> > From: Jonathon A Anderson <jonathon.anderson@colorado.edu><br>> > To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>> > Date: 07/18/2017 07:51 PM<br>> > Subject: Re: [gpfsug-discuss] mmsysmon.py revisited<br>> > Sent by: gpfsug-discuss-bounces@spectrumscale.org<br>> > <br>> > There’s no official way to cleanly disable it
so far as I know yet; <br>> > but you can defacto disable it by deleting /var/mmfs/mmsysmon/<br>> > mmsysmonitor.conf.<br>> > <br>> > It’s a huge problem. I don’t understand why it
hasn’t been given <br>> > much credit by dev or support.<br>> > <br>> > ~jonathon<br>> > <br>> > <br>> > On 7/18/17, 11:21 AM, "gpfsug-discuss-bounces@spectrumscale.org
on <br>> > behalf of David Johnson" <gpfsug-discuss-bounces@spectrumscale.org
<br>> > on behalf of david_johnson@brown.edu> wrote:<br>> > <br>> > <br>> > <br>> > <br>> > We also noticed a fair amount of
CPU time accumulated by <br>> mmsysmon.py on<br>> > our diskless compute nodes. I read
the earlier query, where it <br>> > was answered:<br>> > <br>> > <br>> > <br>> > <br>> > ces == Cluster Export Services, mmsysmon.py
comes from <br>> > mmcesmon. It is used for managing export services
of GPFS. If it is <br>> > killed, your nfs/smb etc will be out of work.<br>> > Their overhead is small and they
are very important. Don't <br>> > attempt to kill them.<br>> > <br>> > <br>> > <br>> > <br>> > <br>> > <br>> > Our question is this — we don’t
run the latest “protocols", our <br>> > NFS is CNFS, and our CIFS is clustered CIFS.<br>> > I can understand it might be needed
with Ganesha, but on <br>> every node? <br>> > <br>> > <br>> > Why in the world would I be getting
this daemon running on all <br>> > client nodes, when I didn’t install the “protocols"
version <br>> > of the distribution? We have
release 4.2.2 at the moment. How<br>> > can we disable this?<br>> > <br>> > <br>> > Thanks,<br>> > — ddj<br>> > <br>> > <br>> > _______________________________________________<br>> > gpfsug-discuss mailing list<br>> > gpfsug-discuss at spectrumscale.org<br>> > </font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br>> <br>> <br>> <br>> _______________________________________________<br>> gpfsug-discuss mailing list<br>> gpfsug-discuss at spectrumscale.org<br>> </font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br></font></tt><BR>