<font size=2 face="sans-serif">Hi Chris, </font><br><font size=2 face="sans-serif">I've seen it several times too - can
u please address this by a regular PMR </font><br><font size=2 face="sans-serif">thx in advance </font><br><font size=2 face="sans-serif">cheers</font><br><font size=2 face="sans-serif">olaf</font><br><br><br><br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">"Fey, Christian"
<Christian.Fey@sva.de></font><br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">gpfsug main discussion
list <gpfsug-discuss@spectrumscale.org></font><br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">04/10/2017 06:04 PM</font><br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">Re: [gpfsug-discuss]
CES doesn't assign addresses to nodes</font><br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr noshade><br><br><br><tt><font size=2>Hi,<br><br>I'm just dealing with a maybe similar issue that also seems to be related
to the output of "tsctl shownodes up" (before CES i actually
never had to do with this command).<br><br>In my case the output of a "mmlscluster" for example shows the
nodes like "node1.acme.local" but in " tsctl shownodes up"
they are displayed as "node1.acme.local.acme.local" for example.<br><br>This maybe causes a fresh CES implementation in a existing GPFS cluster
to also not spread ip-adresses. It instead loops in the same way as it
did in your case @jonathon. I think it tries to search for "node1.acme.local"
but doesn't find it since tsctl shows it with doubled suffix.<br><br>Can anyone explain, from where the "tsctl shownodes up" reads
the data? Additionally does anyone have an idea why the dns suffix is doubled?<br><br><br>Kind regards<br>Christian<br><br>-----Ursprngliche Nachricht-----<br>Von: gpfsug-discuss-bounces@spectrumscale.org [</font></tt><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org"><tt><font size=2>mailto:gpfsug-discuss-bounces@spectrumscale.org</font></tt></a><tt><font size=2>]
Im Auftrag von Jonathon A Anderson<br>Gesendet: Donnerstag, 23. Mrz 2017 16:02<br>An: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>Betreff: Re: [gpfsug-discuss] CES doesn't assign addresses to nodes<br><br>Achtung! Die Absender-Adresse ist mglicherweise geflscht. Bitte berprfen
Sie die Plausibilitt der Email und lassen bei enthaltenen Anhngen und
Links besondere Vorsicht walten.<br>Wenden Sie sich im Zweifelsfall an das CIT unter cit@sva.de oder 06122
536 350.<br>(Stichwort: DKIM Test Fehlgeschlagen)<br><br>----------------------------------------------------------------------------------------------------------------<br><br>Thanks! I’m looking forward to upgrading our CES nodes and resuming work
on the project.<br><br>~jonathon<br><br><br>On 3/23/17, 8:24 AM, "gpfsug-discuss-bounces@spectrumscale.org on
behalf of Olaf Weiser" <gpfsug-discuss-bounces@spectrumscale.org
on behalf of olaf.weiser@de.ibm.com> wrote:<br><br> the issue is fixed, <br> an APAR will be released soon - IV93100<br> <br> <br> <br> From: Olaf Weiser/Germany/IBM@IBMDE<br> To: "gpfsug main discussion
list" <gpfsug-discuss@spectrumscale.org><br> Cc: "gpfsug main discussion
list" <gpfsug-discuss@spectrumscale.org><br> Date: 01/31/2017 11:47 PM<br> Subject: Re: [gpfsug-discuss]
CES doesn't assign addresses to nodes<br> Sent by: gpfsug-discuss-bounces@spectrumscale.org<br> ________________________________________<br> <br> <br> <br> Yeah... depending on the #nodes you 're affected or not.
.....<br> So if your remote ces cluster is small enough in terms
of the #nodes ... you'll neuer hit into this issue <br> <br> Gesendet von IBM Verse<br> <br> Simon Thompson (Research Computing - IT Services) --- Re:
[gpfsug-discuss] CES doesn't assign addresses to nodes ---<br> <br> Von:"Simon Thompson (Research Computing - IT Services)"
<S.J.Thompson@bham.ac.uk>An:"gpfsug main discussion list"
<gpfsug-discuss@spectrumscale.org>Datum:Di. 31.01.2017 21:07Betreff:Re:
[gpfsug-discuss] CES doesn't assign addresses to nodes<br> ________________________________________<br> <br> We use multicluster for our environment, storage systems
in a separate cluster to hpc nodes on a separate cluster from protocol
nodes.<br> <br> According to the docs, this isn't supported, but we haven't
seen any issues. Note unsupported as opposed to broken.<br> <br> Simon<br> ________________________________________<br> From: gpfsug-discuss-bounces@spectrumscale.org [gpfsug-discuss-bounces@spectrumscale.org]
on behalf of Jonathon A Anderson [jonathon.anderson@colorado.edu]<br> Sent: 31 January 2017 17:47<br> To: gpfsug main discussion list<br> Subject: Re: [gpfsug-discuss] CES doesn't assign addresses
to nodes<br> <br> Yeah, I searched around for places where ` tsctl shownodes
up` appears in the GPFS code I have access to (i.e., the ksh and python
stuff); but it’s only in CES. I suspect there just haven’t been that
many people exporting CES out of an HPC cluster environment.<br> <br> ~jonathon<br> <br> <br> From: <gpfsug-discuss-bounces@spectrumscale.org> on
behalf of Olaf Weiser <olaf.weiser@de.ibm.com><br> Reply-To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br> Date: Tuesday, January 31, 2017 at 10:45 AM<br> To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br> Subject: Re: [gpfsug-discuss] CES doesn't assign addresses
to nodes<br> <br> I ll open a pmr here for my env ... the issue may hurt you
in a ces env. only... but needs to be fixed in core gpfs.base i thi
k<br> <br> Gesendet von IBM Verse<br> Jonathon A Anderson --- Re: [gpfsug-discuss] CES doesn't
assign addresses to nodes ---<br> <br> Von:<br> <br> "Jonathon A Anderson" <jonathon.anderson@colorado.edu><br> <br> An:<br> <br> "gpfsug main discussion list" <gpfsug-discuss@spectrumscale.org><br> <br> Datum:<br> <br> Di. 31.01.2017 17:32<br> <br> Betreff:<br> <br> Re: [gpfsug-discuss] CES doesn't assign addresses to nodes<br> <br> ________________________________<br> <br> No, I’m having trouble getting this through DDN support
because, while we have a GPFS server license and GRIDScaler support, apparently
we don’t have “protocol node” support, so they’ve pushed back on supporting
this as an overall CES-rooted effort.<br> <br> I do have a DDN case open, though: 78804. If you are (as
I suspect) a GPFS developer, do you mind if I cite your info from here
in my DDN case to get them to open a PMR?<br> <br> Thanks.<br> <br> ~jonathon<br> <br> <br> From: <gpfsug-discuss-bounces@spectrumscale.org> on
behalf of Olaf Weiser <olaf.weiser@de.ibm.com><br> Reply-To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br> Date: Tuesday, January 31, 2017 at 8:42 AM<br> To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br> Subject: Re: [gpfsug-discuss] CES doesn't assign addresses
to nodes<br> <br> ok.. so obviously ... it seems , that we have several issues..<br> the 3983 characters is obviously a defect<br> have you already raised a PMR , if so , can you send me the
number ?<br> <br> <br> <br> <br> From: Jonathon A Anderson <jonathon.anderson@colorado.edu><br> To: gpfsug main discussion list
<gpfsug-discuss@spectrumscale.org><br> Date: 01/31/2017 04:14 PM<br> Subject: Re: [gpfsug-discuss]
CES doesn't assign addresses to nodes<br> Sent by: gpfsug-discuss-bounces@spectrumscale.org<br> ________________________________<br> <br> <br> <br> The tail isn’t the issue; that’ my addition, so that I
didn’t have to paste the hundred or so line nodelist into the thread.<br> <br> The actual command is<br> <br> tsctl shownodes up | $tr ',' '\n' | $sort -o $upnodefile<br> <br> But you can see in my tailed output that the last hostname
listed is cut-off halfway through the hostname. Less obvious in the example,
but true, is the fact that it’s only showing the first 120 hosts, when
we have 403 nodes in our gpfs cluster.<br> <br> [root@sgate2 ~]# tsctl shownodes up | tr ',' '\n' | wc -l<br> 120<br> <br> [root@sgate2 ~]# mmlscluster | grep '\-opa' | wc -l<br> 403<br> <br> Perhaps more explicitly, it looks like `tsctl shownodes up`
can only transmit 3983 characters.<br> <br> [root@sgate2 ~]# tsctl shownodes up | wc -c<br> 3983<br> <br> Again, I’m convinced this is a bug not only because the
command doesn’t actually produce a list of all of the up nodes in our
cluster; but because the last name listed is incomplete.<br> <br> [root@sgate2 ~]# tsctl shownodes up | tr ',' '\n' | tail
-n 1<br> shas0260-opa.rc.int.col[root@sgate2 ~]#<br> <br> I’d continue my investigation within tsctl itself but, alas,
it’s a binary with no source code available to me. :)<br> <br> I’m trying to get this opened as a bug / PMR; but I’m still
working through the DDN support infrastructure. Thanks for reporting it,
though.<br> <br> For the record:<br> <br> [root@sgate2 ~]# rpm -qa | grep -i gpfs<br> gpfs.base-4.2.1-2.x86_64<br> gpfs.msg.en_US-4.2.1-2.noarch<br> gpfs.gplbin-3.10.0-327.el7.x86_64-4.2.1-0.x86_64<br> gpfs.gskit-8.0.50-57.x86_64<br> gpfs.gpl-4.2.1-2.noarch<br> nfs-ganesha-gpfs-2.3.2-0.ibm24.el7.x86_64<br> gpfs.ext-4.2.1-2.x86_64<br> gpfs.gplbin-3.10.0-327.36.3.el7.x86_64-4.2.1-2.x86_64<br> gpfs.docs-4.2.1-2.noarch<br> <br> ~jonathon<br> <br> <br> From: <gpfsug-discuss-bounces@spectrumscale.org> on
behalf of Olaf Weiser <olaf.weiser@de.ibm.com><br> Reply-To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br> Date: Tuesday, January 31, 2017 at 1:30 AM<br> To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br> Subject: Re: [gpfsug-discuss] CES doesn't assign addresses
to nodes<br> <br> Hi ...same thing here.. everything after 10 nodes will be
truncated..<br> though I don't have an issue with it ... I 'll open a PMR
.. and I recommend you to do the same thing.. ;-)<br> <br> the reason seems simple.. it is the "| tail" .at
the end of the command.. .. which truncates the output to the last 10 items...<br> <br> should be easy to fix..<br> cheers<br> olaf<br> <br> <br> <br> <br> <br> From: Jonathon A Anderson <jonathon.anderson@colorado.edu><br> To: "gpfsug-discuss@spectrumscale.org"
<gpfsug-discuss@spectrumscale.org><br> Date: 01/30/2017 11:11 PM<br> Subject: Re: [gpfsug-discuss]
CES doesn't assign addresses to nodes<br> Sent by: gpfsug-discuss-bounces@spectrumscale.org<br> ________________________________<br> <br> <br> <br> <br> In trying to figure this out on my own, I’m relatively certain
I’ve found a bug in GPFS related to the truncation of output from `tsctl
shownodes up`. Any chance someone in development can confirm?<br> <br> <br> Here are the details of my investigation:<br> <br> <br> ## GPFS is up on sgate2<br> <br> [root@sgate2 ~]# mmgetstate<br> <br> Node number Node name GPFS
state<br> ------------------------------------------<br> 414 sgate2-opa
active<br> <br> <br> ## but if I tell ces to explicitly put one of our ces addresses
on that node, it says that GPFS is down<br> <br> [root@sgate2 ~]# mmces address move --ces-ip 10.225.71.102
--ces-node sgate2-opa<br> mmces address move: GPFS is down on this node.<br> mmces address move: Command failed. Examine previous error
messages to determine cause.<br> <br> <br> ## the “GPFS is down on this node” message is defined as
code 109 in mmglobfuncs<br> <br> [root@sgate2 ~]# grep --before-context=1 "GPFS is down
on this node." /usr/lpp/mmfs/bin/mmglobfuncs<br> 109 ) msgTxt=\<br> "%s: GPFS is down on this node."<br> <br> <br> ## and is generated by printErrorMsg in mmcesnetmvaddress
when it detects that the current node is identified as “down” by getDownCesNodeList<br> <br> [root@sgate2 ~]# grep --before-context=5 'printErrorMsg 109'
/usr/lpp/mmfs/bin/mmcesnetmvaddress<br> downNodeList=$(getDownCesNodeList)<br> for downNode in $downNodeList<br> do<br> if [[ $toNodeName == $downNode ]]<br> then<br> printErrorMsg 109 "$mmcmd"<br> <br> <br> ## getDownCesNodeList is the intersection of all ces nodes
with GPFS cluster nodes listed in `tsctl shownodes up`<br> <br> [root@sgate2 ~]# grep --after-context=16 '^function getDownCesNodeList'
/usr/lpp/mmfs/bin/mmcesfuncs<br> function getDownCesNodeList<br> {<br> typeset sourceFile="mmcesfuncs.sh"<br> [[ -n $DEBUG || -n $DEBUGgetDownCesNodeList ]] &&set
-x<br> $mmTRACE_ENTER "$*"<br> <br> typeset upnodefile=${cmdTmpDir}upnodefile<br> typeset downNodeList<br> <br> # get all CES nodes<br> $sort -o $nodefile $mmfsCesNodes.dae<br> <br> $tsctl shownodes up | $tr ',' '\n' | $sort -o $upnodefile<br> <br> downNodeList=$($comm -23 $nodefile $upnodefile)<br> print -- $downNodeList<br> } #----- end of function getDownCesNodeList --------------------<br> <br> <br> ## but not only are the sgate nodes not listed by `tsctl
shownodes up`; its output is obviously and erroneously truncated<br> <br> [root@sgate2 ~]# tsctl shownodes up | tr ',' '\n' | tail<br> shas0251-opa.rc.int.colorado.edu<br> shas0252-opa.rc.int.colorado.edu<br> shas0253-opa.rc.int.colorado.edu<br> shas0254-opa.rc.int.colorado.edu<br> shas0255-opa.rc.int.colorado.edu<br> shas0256-opa.rc.int.colorado.edu<br> shas0257-opa.rc.int.colorado.edu<br> shas0258-opa.rc.int.colorado.edu<br> shas0259-opa.rc.int.colorado.edu<br> shas0260-opa.rc.int.col[root@sgate2 ~]#<br> <br> <br> ## I expect that this is a bug in GPFS, likely related to
a maximum output buffer for `tsctl shownodes up`.<br> <br> <br> <br> On 1/24/17, 12:48 PM, "Jonathon A Anderson" <jonathon.anderson@colorado.edu>
wrote:<br> <br> I think I'm having the same issue described here:<br> <br> </font></tt><a href="http://www.spectrumscale.org/pipermail/gpfsug-discuss/2016-October/002288.html"><tt><font size=2>http://www.spectrumscale.org/pipermail/gpfsug-discuss/2016-October/002288.html</font></tt></a><tt><font size=2><br> <br> Any advice or further troubleshooting steps would be much
appreciated. Full disclosure: I also have a DDN case open. (78804)<br> <br> We've got a four-node (snsd{1..4}) DDN gridscaler system.
I'm trying to add two CES protocol nodes (sgate{1,2}) to serve NFS.<br> <br> Here's the steps I took:<br> <br> ---<br> mmcrnodeclass protocol -N sgate1-opa,sgate2-opa<br> mmcrnodeclass nfs -N sgate1-opa,sgate2-opa<br> mmchconfig cesSharedRoot=/gpfs/summit/ces<br> mmchcluster --ccr-enable<br> mmchnode --ces-enable -N protocol<br> mmces service enable NFS<br> mmces service start NFS -N nfs<br> mmces address add --ces-ip 10.225.71.104,10.225.71.105<br> mmces address policy even-coverage<br> mmces address move --rebalance<br> ---<br> <br> This worked the very first time I ran it, but the CES addresses
weren't re-distributed after restarting GPFS or a node reboot.<br> <br> Things I've tried:<br> <br> * disabling ces on the sgate nodes and re-running the above
procedure<br> * moving the cluster and filesystem managers to different
snsd nodes<br> * deleting and re-creating the cesSharedRoot directory<br> <br> Meanwhile, the following log entry appears in mmfs.log.latest
every ~30s:<br> <br> ---<br> Mon Jan 23 20:31:20 MST 2017: mmcesnetworkmonitor: Found
unassigned address 10.225.71.104<br> Mon Jan 23 20:31:20 MST 2017: mmcesnetworkmonitor: Found
unassigned address 10.225.71.105<br> Mon Jan 23 20:31:20 MST 2017: mmcesnetworkmonitor: handleNetworkProblem
with lock held: assignIP 10.225.71.104_0-_+,10.225.71.105_0-_+ 1<br> Mon Jan 23 20:31:20 MST 2017: mmcesnetworkmonitor: Assigning
addresses: 10.225.71.104_0-_+,10.225.71.105_0-_+<br> Mon Jan 23 20:31:20 MST 2017: mmcesnetworkmonitor: moveCesIPs:
10.225.71.104_0-_+,10.225.71.105_0-_+<br> ---<br> <br> Also notable, whenever I add or remove addresses now, I
see this in mmsysmonitor.log (among a lot of other entries):<br> <br> ---<br> 2017-01-23T20:40:56.363 sgate1 D ET_cesnetwork Entity state
without requireUnique: ces_network_ips_down WARNING No CES relevant NICs
detected - Service.calculateAndUpdateState:275<br> 2017-01-23T20:40:11.364 sgate1 D ET_cesnetwork Update multiple
entities at once {'p2p2': 1, 'bond0': 1, 'p2p1': 1} - Service.setLocalState:333<br> ---<br> <br> For the record, here's the interface I expect to get the
address on sgate1:<br> <br> ---<br> 11: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP>
mtu 9000 qdisc noqueue state UP<br> link/ether 3c:fd:fe:08:a7:c0 brd ff:ff:ff:ff:ff:ff<br> inet 10.225.71.107/20 brd 10.225.79.255 scope global bond0<br> valid_lft forever preferred_lft forever<br> inet6 fe80::3efd:feff:fe08:a7c0/64 scope link<br> valid_lft forever preferred_lft forever<br> ---<br> <br> which is a bond of p2p1 and p2p2.<br> <br> ---<br> 6: p2p1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu
9000 qdisc mq master bond0 state UP qlen 1000<br> link/ether 3c:fd:fe:08:a7:c0 brd ff:ff:ff:ff:ff:ff<br> 7: p2p2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu
9000 qdisc mq master bond0 state UP qlen 1000<br> link/ether 3c:fd:fe:08:a7:c0 brd ff:ff:ff:ff:ff:ff<br> ---<br> <br> A similar bond0 exists on sgate2.<br> <br> I crawled around in /usr/lpp/mmfs/lib/mmsysmon/CESNetworkService.py
for a while trying to figure it out, but have been unsuccessful so far.<br> <br> <br> <br> _______________________________________________<br> gpfsug-discuss mailing list<br> gpfsug-discuss at spectrumscale.org<br> </font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br> <br> <br> <br> _______________________________________________<br> gpfsug-discuss mailing list<br> gpfsug-discuss at spectrumscale.org<br> </font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br> <br> <br> <br> <br> <br> <br> _______________________________________________<br> gpfsug-discuss mailing list<br> gpfsug-discuss at spectrumscale.org<br> </font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br> <br> _______________________________________________<br> gpfsug-discuss mailing list<br> gpfsug-discuss at spectrumscale.org<br> </font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br> <br> <br> <br> <br> <br><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br>[attachment "smime.p7s" deleted by Olaf Weiser/Germany/IBM] _______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br></font></tt><br><br><BR>