<div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:12pt" ><div dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div dir="ltr" >I filed a defect here, let us see what Redhat says. Yes, it doesn't work for any kernel threads. It doesn't work for user level threads/processes.</div>
<div dir="ltr" > </div>
<div dir="ltr" ><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1803640">https://bugzilla.redhat.com/show_bug.cgi?id=1803640</a></div>
<div dir="ltr" > </div>
<div dir="ltr" >Regards, Malahal.</div>
<div dir="ltr" > </div>
<blockquote data-history-content-modified="1" dir="ltr" style="border-left:solid #aaaaaa 2px; margin-left:5px; padding-left:5px; direction:ltr; margin-right:0px" >----- Original message -----<br>From: Bryan Hill <bhill@physics.ucsd.edu><br>Sent by: gpfsug-discuss-bounces@spectrumscale.org<br>To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>Cc:<br>Subject: [EXTERNAL] Re: [gpfsug-discuss] CNFS issue after upgrading from 4.2.3.11 to 5.0.4.2<br>Date: Mon, Feb 17, 2020 8:26 AM<br>
<div dir="ltr" ><div style="font-family:tahoma,sans-serif" >Ah wait, I see what you might mean. pidof works but not specifically for processes like nfsd. That is odd.</div>
<div><div data-smartmail="gmail_signature" dir="ltr" ><br>Thanks,<br>Bryan<br><br> </div></div></div>
<div><div dir="ltr" >On Sun, Feb 16, 2020 at 10:19 AM Bryan Hill <<a href="mailto:bhill@physics.ucsd.edu" target="_blank">bhill@physics.ucsd.edu</a>> wrote:</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" ><div dir="ltr" ><div style="font-family:tahoma,sans-serif" >Hi Malahal:</div>
<div style="font-family:tahoma,sans-serif" > </div>
<div style="font-family:tahoma,sans-serif" >Just to clarify, are you saying that on your VM pidof is missing? Or that it is there and not working as it did prior to RHEL/CentOS 8? pidof is returning pid numbers on my system. I've been looking at the mmnfsmonitor script and trying to see where the check for nfsd might be failing, but I've not been able to figure it out yet.</div>
<div style="font-family:tahoma,sans-serif" > </div>
<div style="font-family:tahoma,sans-serif" > </div>
<div><div dir="ltr" ><br>Thanks,<br>Bryan<br><br>---<br>Bryan Hill<br>Lead System Administrator<br>UCSD Physics Computing Facility<br><br>9500 Gilman Dr. # 0319<br>La Jolla, CA 92093<br>+1-858-534-5538<br><a href="mailto:bhill@ucsd.edu" target="_blank">bhill@ucsd.edu</a></div></div></div>
<div><div dir="ltr" >On Sat, Feb 15, 2020 at 2:03 AM Malahal R Naineni <<a href="mailto:mnaineni@in.ibm.com" target="_blank">mnaineni@in.ibm.com</a>> wrote:</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" ><div dir="ltr" style="font-family:Arial,Helvetica,sans-serif;font-size:10pt" ><div dir="ltr" >I am not familiar with CNFS but looking at git source seems to indicate that it uses 'pidof' to check if a program is running or not. "pidof nfsd" works on RHEL7.x but it fails on my centos8.1 I just created. So either we need to make sure pidof works on kernel threads or fix CNFS scripts.</div>
<div dir="ltr" > </div>
<div dir="ltr" >Regards, Malahal.</div>
<div dir="ltr" > </div>
<blockquote dir="ltr" style="border-left:2px solid rgb(170,170,170);margin-left:5px;padding-left:5px;direction:ltr;margin-right:0px" >----- Original message -----<br>From: Bryan Hill <<a href="mailto:bhill@physics.ucsd.edu" target="_blank">bhill@physics.ucsd.edu</a>><br>Sent by: <a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">gpfsug-discuss-bounces@spectrumscale.org</a><br>To: <a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.org</a><br>Cc:<br>Subject: [EXTERNAL] [gpfsug-discuss] CNFS issue after upgrading from 4.2.3.11 to 5.0.4.2<br>Date: Fri, Feb 14, 2020 11:40 PM<br>
<div dir="ltr" ><div style="font-family:tahoma,sans-serif" ><span style="font-family:Arial,Helvetica,sans-serif" >Hi All:</span></div>
<div><div dir="ltr" ><br>I'm performing a rolling upgrade of one of our GPFS clusters. This particular cluster has 2 CNFS servers for some of our NFS clients. I wiped one of the nodes and installed RHEL 8.1 and GPFS 5.0.4.2. The filesystem mounts fine on the node when I disable CNFS on the node, but with it enabled it's a no go. It appears mmnfsmonitor doesn't recognize that nfsd has started, so it assumes the worst and shuts down the file system (I currently have reboot on failure disabled to debug this). The thing is, it actually does start nfsd processes when running mmstartup on the node. Doing a "ps" shows 32 nfsd threads are running.<br><br>Below is <span style="font-family:tahoma,sans-serif" >the </span>CNFS-specific output from an attempt to start the node:<br><br>CNFS[27243]: Restarting lockd to start grace<br>CNFS[27588]: Enabling 172.16.69.76<br>CNFS[27694]: Restarting lockd to start grace<br>CNFS[27699]: Starting NFS services<br>CNFS[27764]: NFS clients of node 172.16.69.122 notified to reclaim NLM locks<br>CNFS[27910]: Monitor has started pid=27787<br>CNFS[28702]: Monitor detected nfsd was not running, will attempt to start it<br>CNFS[28705]: Starting NFS services<br>CNFS[28730]: NFS clients of node 172.16.69.122 notified to reclaim NLM locks<br>CNFS[28755]: Monitor detected nfsd was not running, will attempt to start it<br>CNFS[28758]: Starting NFS services<br>CNFS[28789]: NFS clients of node 172.16.69.122 notified to reclaim NLM locks<br>CNFS[28813]: Monitor detected nfsd was not running, will attempt to start it<br>CNFS[28816]: Starting NFS services<br>CNFS[28844]: NFS clients of node 172.16.69.122 notified to reclaim NLM locks<br>CNFS[28867]: Monitor detected nfsd was not running, will attempt to start it<br>CNFS[28874]: Monitoring detected NFSD is inactive. mmnfsmonitor: NFS server is not running or responding. Node failure initiated as configured.<br>CNFS[28924]: Unexporting all GPFS filesystems<br><br>Any thoughts? My other CNFS node is handling everything for the time being, thankfully!<br> </div>
<div dir="ltr" >Thanks,<br>Bryan<br><br>---<br>Bryan Hill<br>Lead System Administrator<br>UCSD Physics Computing Facility<br><br>9500 Gilman Dr. # 0319<br>La Jolla, CA 92093<br>+1-858-534-5538<br><a href="mailto:bhill@ucsd.edu" target="_blank">bhill@ucsd.edu</a></div></div></div>
<div><font face="Default Monospace,Courier New,Courier,monospace" size="2" >_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank">spectrumscale.org</a><br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a> </font></div></blockquote>
<div dir="ltr" > </div></div><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></blockquote></div></blockquote></div>
<div><font face="Default Monospace,Courier New,Courier,monospace" size="2" >_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a> </font></div></blockquote>
<div dir="ltr" > </div></div></div><BR>