<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css">BODY {
FONT-FAMILY: arial
}
</style>
</head>
<body dir="auto">
<p>Hallo Simon,</p>
<div>Welcome to the Club. These behavior are a Bug in tsctl to change the DNS names . We had this already 4 weeks ago. The fix was Update to 5.0.2.1. </div>
<div>Regards Renar</div>
<div><br>
<br>
<div id="AppleMailSignature" dir="ltr">Von meinem iPhone gesendet</div>
<div dir="ltr"><br>
</div>
</div>
<p></p>
<p><font style="font-family:Arial;">Renar</font> <font style="font-family:Arial;">Grunenberg</font><br>
<font style="font-family:Arial;">Abteilung Informatik - Betrieb</font><br>
<br>
<font style="font-family:Arial;">HUK-COBURG</font><br>
<font style="font-family:Arial;">Bahnhofsplatz</font><br>
<font style="font-family:Arial;">96444</font> <font style="font-family:Arial;">Coburg</font><br>
<table class="1f5f1f51-d3bc-41eb-9831-6d8eac88aba5Table" style="font-family:Arial;">
<tbody>
<tr>
<td><font style="font-family:Arial;"><font style="font-family:Arial;"><font style="font-family:Arial;">Telefon:</font></font></font></td>
<td><font style="font-family:Arial;"><font style="font-family:Arial;"><font style="font-family:Arial;">09561 96-44110</font></font></font></td>
</tr>
<tr>
<td><font style="font-family:Arial;"><font style="font-family:Arial;"><font style="font-family:Arial;">Telefax:</font></font></font></td>
<td><font style="font-family:Arial;"><font style="font-family:Arial;"><font style="font-family:Arial;">09561 96-44104</font></font></font></td>
</tr>
<tr>
<td><font style="font-family:Arial;"><font style="font-family:Arial;"><font style="font-family:Arial;">E-Mail:</font></font></font></td>
<td><font style="font-family:Arial;"><font style="font-family:Arial;">Renar.Grunenberg@huk-coburg.de</font></font></td>
</tr>
<tr>
<td><font style="font-family:Arial;"><font style="font-family:Arial;"><font style="font-family:Arial;">Internet:</font></font></font></td>
<td><font style="font-family:Arial;"><font style="font-family:Arial;"><font style="font-family:Arial;">www.huk.de</font></font></font></td>
</tr>
</tbody>
</table>
</p>
<hr id="HR1">
<font face="Arial"><font size="1">HUK-COBURG Haftpflicht-Unterstützungs-Kasse kraftfahrender Beamter Deutschlands a. G. in Coburg<br>
Reg.-Gericht Coburg HRB 100; St.-Nr. 9212/101/00021<br>
Sitz der Gesellschaft: Bahnhofsplatz, 96444 Coburg<br>
Vorsitzender des Aufsichtsrats: Prof. Dr. Heinrich R. Schradin.<br>
Vorstand: Klaus-Jürgen Heitmann (Sprecher), Stefan Gronbach, Dr. Hans Olav Herøy, Dr. Jörg Rheinländer (stv.), Sarah Rössler, Daniel Thomas.<br>
</font>
<hr id="HR1">
</font><font size="1" face="Arial">Diese Nachricht enthält vertrauliche und/oder rechtlich geschützte Informationen.<br>
Wenn Sie nicht der richtige Adressat sind oder diese Nachricht irrtümlich erhalten haben,<br>
informieren Sie bitte sofort den Absender und vernichten Sie diese Nachricht.<br>
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Nachricht ist nicht gestattet.<br>
<br>
This information may contain confidential and/or privileged information.<br>
If you are not the intended recipient (or have received this information in error) please notify the<br>
sender immediately and destroy this information.<br>
Any unauthorized copying, disclosure or distribution of the material in this information is strictly forbidden.<br>
</font>
<hr id="HR1">
<p></p>
<p></p>
<p></p>
<div>
<div dir="ltr">Am 11.01.2019 um 15:19 schrieb Simon Thompson <<a href="mailto:S.J.Thompson@bham.ac.uk">S.J.Thompson@bham.ac.uk</a>>:<br>
<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style>
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I\u2019ll start by saying this is our experience, maybe we did something stupid along the way, but just in case others see similar issues \u2026<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">We have a cluster which contains protocol nodes, these were all happily running GPFS 5.0.1-2 code. But the cluster was a only 4 nodes + 1 quorum node \u2013 manager and quorum functions were handled by the 4 protocol
nodes.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Then one day we needed to reboot a protocol node. We did so and its disk controller appeared to have failed. Oh well, we thought we\u2019ll fix that another day, we still have three other quorum nodes.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">As they are all getting a little long in the tooth and were starting to struggle, we thought, well we have DME, lets add some new nodes for quorum and token functions. Being shiny and new they were all installed
with GPFS 5.0.2-1 code.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">All was well.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">The some-time later, we needed to restart another of the CES nodes, when we started GPFS on the node, it was causing havock in our cluster \u2013 CES IPs were constantly being assigned, then removed from the remaining
nodes in the cluster. Crap we thought and disabled the node in the cluster. This made things stabilise and as we\u2019d been having other GPFS issues, we didn\u2019t want service to be interrupted whilst we dug into this. Besides, it was nearly Christmas and we had
conferences and other work to content with.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">More time passes and we\u2019re about to cut over all our backend storage to some shiny new DSS-G kit, so we plan a whole system maintenance window. We finish all our data sync\u2019s and then try to start our protocol
nodes to test them. No dice \u2026 we can\u2019t get any of the nodes to bring up IPs, the logs look like they start the assignment process, but then gave up.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">A lot of digging in the mm korn shell scripts, and some studious use of DEBUG=1 when testing, we find that mmcesnetmvaddress is calling \u201ctsctl shownodes up\u201d. On our protocol nodes, we find output of the form:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">bear-er-dtn01.bb2.cluster.cluster,rds-aw-ctdb01-data.bb2.cluster.cluster,rds-er-ctdb01-data.bb2.cluster.cluster,bber-irods-ires01-data.bb2.cluster.cluster,bber-irods-icat01-data.bb2.cluster.cluster,bbaw-irods-icat01-data.bb2.cluster.cluster,proto-pg-mgr01.bear.cluster.cluster,proto-pg-pf01.bear.cluster.cluster,proto-pg-dtn01.bear.cluster.cluster,proto-er-mgr01.bear.cluster.cluster,proto-er-pf01.bear.cluster.cluster,proto-aw-mgr01.bear.cluster.cluster,proto-aw-pf01.bear.cluster.cluster<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Now our DNS name for these nodes is bb2.cluster \u2026 something is repeating the DNS name.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">So we dig around, resolv.conf, /etc/hosts etc all look good and name resolution seems fine.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">We look around on the manager/quorum nodes and they don\u2019t do this cluster.cluster thing. We can\u2019t find anything else Linux config wise that looks bad. In fact the only difference is that our CES nodes are
running 5.0.1-2 and the manager nodes 5.0.2-1. Given we\u2019re changing the whole storage hardware, we didn\u2019t want to change the GPFS/NFS/SMB code on the CES nodes, (we\u2019ve been bitten before with SMB packages not working properly in our environment), but we go
ahead and do GPFS and NFS packages.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Suddenly, magically all is working again. CES starts fine and IPs get assigned OK. And tsctl gives the correct output.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">So, my supposition is that there is some incompatibility between 5.0.1-2 and 5.0.2-1 when running CES and the cluster manager is running on 5.0.2-1. As I said before, I don\u2019t have hard evidence we did something
stupid, but it certainly is fishy. We\u2019re guessing this same \u201cfeature\u201d was the cause of the CES issues we saw when we rebooted a CES node and the IPs kept deassigning\u2026 It looks like all was well as we added the manager nodes after CES was started, but when
a CES node restarted, things broke.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">We got everything working again in house so didn\u2019t raise a PMR, but if you find yourself in this upgrade path, beware!<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Simon<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
</div>
</blockquote>
<blockquote type="cite">
<div dir="ltr"><span>_______________________________________________</span><br>
<span>gpfsug-discuss mailing list</span><br>
<span>gpfsug-discuss at <a href="http://spectrumscale.org">spectrumscale.org</a></span><br>
<span><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></span><br>
</div>
</blockquote>
</div>
<p></p>
<p></p>
<p></p>
<p></p>
<p></p>
<p></p>
<p></p>
<p></p>
</body>
</html>