<html>
  <head>
    <meta content="text/html; charset=ISO-8859-1"
      http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    This is an interesting point!<br>
    <br>
    We use ethernet ( 10g links on the clients) but we dont have a
    separate network for the admin network. <br>
    <br>
    Could you explain this a bit further, because the clients and the
    servers we have are on different subnet so the packet are routed.. I
    don't see a practical way to separate them. The clients are blades
    in a chassis so even if i create 2 interfaces, they will physically
    use the came "cable" to go to the first switch. even the clients (
    600 clients) have different subsets.<br>
    <br>
    I will forward this consideration to our network admin , so see if
    we can work on a dedicated network.<br>
    <br>
    thanks for your tip.<br>
    <br>
    Regards,<br>
    Salvatore<br>
    <br>
    <br>
    <br>
    <br>
    <div class="moz-cite-prefix">On 21/08/14 14:03, Vic Cornell wrote:<br>
    </div>
    <blockquote
      cite="mid:9B247872-CD75-4F86-A10E-33AAB6BD414A@gmail.com"
      type="cite">
      <meta http-equiv="Content-Type" content="text/html;
        charset=ISO-8859-1">
      Hi Salvatore,
      <div><br>
      </div>
      <div>Are you using ethernet or infiniband as the GPFS interconnect
        to your clients?</div>
      <div><br>
      </div>
      <div>If 10/40GbE - do you have a separate admin network?</div>
      <div><br>
      </div>
      <div>I have seen behaviour similar to this where the storage
        traffic causes congestion and the "admin" traffic gets lost or
        delayed causing expels.</div>
      <div><br>
      </div>
      <div>Vic</div>
      <div><br>
      </div>
      <div><br>
      </div>
      <div><br>
        <div>
          <div>On 21 Aug 2014, at 10:04, Salvatore Di Nardo &lt;<a
              moz-do-not-send="true" href="mailto:sdinardo@ebi.ac.uk">sdinardo@ebi.ac.uk</a>&gt;
            wrote:</div>
          <br class="Apple-interchange-newline">
          <blockquote type="cite">
            <meta content="text/html; charset=ISO-8859-1"
              http-equiv="Content-Type">
            <div bgcolor="#FFFFFF" text="#000000"> Thanks for the
              feedback, but we managed to find a scenario that excludes
              network problems.<br>
              <br>
              we have a file called <b><i>input_file</i></b> of nearly
              100GB:<br>
              <br>
              if from <b>client A</b> we do:<br>
              <br>
              cat input_file &gt;&gt; output_file<br>
              <br>
              it start copying.. and we see waiter goeg a bit up,secs
              but then they flushes back to 0, so we xcan say that the
              copy proceed well...<br>
              <br>
              <br>
              if now we do the same from another client ( or just
              another shell on the same client) <b>client B</b> :<br>
              <br>
              cat input_file &gt;&gt; output_file<br>
              <br>
              <br>
              &nbsp;( in other words we are trying to write to the same
              destination) all the waiters gets up until one node get
              expelled.<br>
              <br>
              <br>
              Now, while its understandable that the destination file is
              locked for one of the "cat", so have to wait ( and since
              the file is BIG , have to wait for a while), its not
              understandable why it stop the renewal lease. <br>
              Why its doen't return just a timeout error on the copy&nbsp;
              instead to expel the node? We can reproduce this every
              time, and since our users to operations like this on files
              over 100GB each you can imagine the result.<br>
              <br>
              <br>
              <br>
              As you can imagine even if its a bit silly to write at the
              same time to the same destination, its also quite common
              if we want to dump to a log file logs and for some reason
              one of the writers, write for a lot of time keeping the
              file locked.<br>
              Our expels are not due to network congestion, but because
              a write attempts have to wait another one. What i really
              dont understand is why to take a so expreme mesure to
              expell jest because a process is waiteing "to too much
              time".<br>
              <br>
              <br>
              I have ticket opened to IBM for this and the issue is
              under investigation, but no luck so far..<br>
              <br>
              Regards,<br>
              Salvatore<br>
              <br>
              <br>
              <br>
              <div class="moz-cite-prefix">On 21/08/14 09:20, Jez Tucker
                (Chair) wrote:<br>
              </div>
              <blockquote cite="mid:53F5ABD7.80107@gpfsug.org"
                type="cite">
                <meta content="text/html; charset=ISO-8859-1"
                  http-equiv="Content-Type">
                Hi there,<br>
                <br>
                &nbsp; I've seen the on several 'stock'?&nbsp; 'core'? GPFS system
                (we need a better term now GSS is out) and seen ping
                'working', but alongside ejections from the cluster.<br>
                The GPFS internode 'ping' is somewhat more circumspect
                than unix ping - and rightly so.<br>
                <br>
                In my experience this has _always_ been a network issue
                of one sort of another.&nbsp; If the network is experiencing
                issues, nodes will be ejected.<br>
                Of course it could be unresponsive mmfsd or high
                loadavg, but I've seen that only twice in 10 years over
                many versions of GPFS.<br>
                <br>
                You need to follow the logs through from each machine in
                time order to determine who could not see who and in
                what order.<br>
                Your best way forward is to log a SEV2 case with IBM
                support, directly or via your OEM and collect and supply
                a snap and traces as required by support.<br>
                <br>
                Without knowing your full setup, it's hard to help
                further.<br>
                <br>
                Jez<br>
                <br>
                <div class="moz-cite-prefix">On 20/08/14 08:57,
                  Salvatore Di Nardo wrote:<br>
                </div>
                <blockquote cite="mid:53F454E3.40803@ebi.ac.uk"
                  type="cite">
                  <meta content="text/html; charset=ISO-8859-1"
                    http-equiv="Content-Type">
                  Still problems. Here some more detailed examples:<br>
                  <br>
                  <b>EXAMPLE 1:</b><br>
                  <blockquote>
                    <blockquote>
                      <blockquote><b><tt>EBI5-220</tt></b><b><tt> (
                            CLIENT)</tt></b><b><br>
                        </b><tt>Tue Aug 19 11:03:04.980 2014: <b>Timed
                            out waiting for a reply from node &lt;GSS02B
                            IP&gt; gss02b</b></tt><br>
                        <tt>Tue Aug 19 11:03:04.981 2014: Request sent
                          to &lt;GSS02A IP&gt; (gss02a in <a
                            moz-do-not-send="true"
                            href="http://GSS.ebi.ac.uk">GSS.ebi.ac.uk</a>)
                          to expel &lt;GSS02B IP&gt; (gss02b in <a
                            moz-do-not-send="true"
                            href="http://GSS.ebi.ac.uk">GSS.ebi.ac.uk</a>)
                          from cluster <a moz-do-not-send="true"
                            href="http://GSS.ebi.ac.uk">GSS.ebi.ac.uk</a></tt><br>
                        <tt>Tue Aug 19 11:03:04.982 2014: This node will
                          be expelled from cluster <a
                            moz-do-not-send="true"
                            href="http://GSS.ebi.ac.uk">GSS.ebi.ac.uk</a>
                          due to expel msg from &lt;EBI5-220 IP&gt;
                          (ebi5-220)</tt><br>
                        <tt>Tue Aug 19 11:03:09.319 2014: Cluster
                          Manager connection broke. Probing cluster <a
                            moz-do-not-send="true"
                            href="http://GSS.ebi.ac.uk">GSS.ebi.ac.uk</a></tt><br>
                        <tt>Tue Aug 19 11:03:10.321 2014: Unable to
                          contact any quorum nodes during cluster probe.</tt><br>
                        <tt>Tue Aug 19 11:03:10.322 2014: Lost
                          membership in cluster <a
                            moz-do-not-send="true"
                            href="http://GSS.ebi.ac.uk">GSS.ebi.ac.uk</a>.
                          Unmounting file systems.</tt><br>
                        <tt>Tue Aug 19 11:03:10 BST 2014: mmcommon
                          preunmount invoked.&nbsp; File system: gpfs1&nbsp;
                          Reason: SGPanic</tt><br>
                        <tt>Tue Aug 19 11:03:12.066 2014: Connecting to
                          &lt;GSS02A IP&gt; gss02a &lt;c1p687&gt;</tt><br>
                        <tt>Tue Aug 19 11:03:12.070 2014: Connected to
                          &lt;GSS02A IP&gt; gss02a &lt;c1p687&gt;</tt><br>
                        <tt>Tue Aug 19 11:03:17.071 2014: Connecting to
                          &lt;GSS02B IP&gt; gss02b &lt;c1p686&gt;</tt><br>
                        <tt>Tue Aug 19 11:03:17.072 2014: Connecting to
                          &lt;GSS03B IP&gt; gss03b &lt;c1p685&gt;</tt><br>
                        <tt>Tue Aug 19 11:03:17.079 2014: Connecting to
                          &lt;GSS03A IP&gt; gss03a &lt;c1p684&gt;</tt><br>
                        <tt>Tue Aug 19 11:03:17.080 2014: Connecting to
                          &lt;GSS01B IP&gt; gss01b &lt;c1p683&gt;</tt><br>
                        <tt>Tue Aug 19 11:03:17.079 2014: Connecting to
                          &lt;GSS01A IP&gt; gss01a &lt;c1p1&gt;</tt><br>
                        <tt>Tue Aug 19 11:04:23.105 2014: Connected to
                          &lt;GSS02B IP&gt; gss02b &lt;c1p686&gt;</tt><br>
                        <tt>Tue Aug 19 11:04:23.107 2014: Connected to
                          &lt;GSS03B IP&gt; gss03b &lt;c1p685&gt;</tt><br>
                        <tt>Tue Aug 19 11:04:23.112 2014: Connected to
                          &lt;GSS03A IP&gt; gss03a &lt;c1p684&gt;</tt><br>
                        <tt>Tue Aug 19 11:04:23.115 2014: Connected to
                          &lt;GSS01B IP&gt; gss01b &lt;c1p683&gt;</tt><br>
                        <tt>Tue Aug 19 11:04:23.121 2014: Connected to
                          &lt;GSS01A IP&gt; gss01a &lt;c1p1&gt;</tt><br>
                        <tt>Tue Aug 19 11:12:28.992 2014: Node
                          &lt;GSS02A IP&gt; (gss02a in <a
                            moz-do-not-send="true"
                            href="http://GSS.ebi.ac.uk">GSS.ebi.ac.uk</a>)
                          is now the Group Leader.</tt><br>
                        <br>
                        <b><tt>GSS02B ( NSD SERVER)</tt></b><br>
                        <tt>...<br>
                          Tue Aug 19 11:03:17.070 2014: Killing
                          connection from <b>&lt;EBI5-220 IP&gt;</b>
                          because the group is not ready for it to
                          rejoin, err 46<br>
                          Tue Aug 19 11:03:25.016 2014: Killing
                          connection from &lt;EBI5-102 IP&gt; because
                          the group is not ready for it to rejoin, err
                          46<br>
                          Tue Aug 19 11:03:28.080 2014: Killing
                          connection from </tt><tt><tt><b>&lt;EBI5-220
                              IP&gt;</b></tt> because the group is not
                          ready for it to rejoin, err 46<br>
                          Tue Aug 19 11:03:36.019 2014: Killing
                          connection from &lt;EBI5-102 IP&gt; because
                          the group is not ready for it to rejoin, err
                          46<br>
                          Tue Aug 19 11:03:39.083 2014: Killing
                          connection from </tt><tt><tt><b>&lt;EBI5-220
                              IP&gt;</b></tt> because the group is not
                          ready for it to rejoin, err 46<br>
                          Tue Aug 19 11:03:47.023 2014: Killing
                          connection from &lt;EBI5-102 IP&gt; because
                          the group is not ready for it to rejoin, err
                          46<br>
                          Tue Aug 19 11:03:50.088 2014: Killing
                          connection from </tt><tt><tt><b>&lt;EBI5-220
                              IP&gt;</b></tt> because the group is not
                          ready for it to rejoin, err 46<br>
                          Tue Aug 19 11:03:52.218 2014: Killing
                          connection from </tt><tt><tt>&lt;EBI5-043
                            IP&gt;</tt> because the group is not ready
                          for it to rejoin, err 46<br>
                          Tue Aug 19 11:03:58.030 2014: Killing
                          connection from &lt;EBI5-102 IP&gt; because
                          the group is not ready for it to rejoin, err
                          46<br>
                          Tue Aug 19 11:04:01.092 2014: Killing
                          connection from </tt><tt><tt><b>&lt;EBI5-220
                              IP&gt;</b></tt> because the group is not
                          ready for it to rejoin, err 46<br>
                          Tue Aug 19 11:04:03.220 2014: Killing
                          connection from </tt><tt><tt><tt>&lt;EBI5-043
                              IP&gt;</tt></tt> because the group is not
                          ready for it to rejoin, err 46<br>
                          Tue Aug 19 11:04:09.034 2014: Killing
                          connection from &lt;EBI5-102 IP&gt; because
                          the group is not ready for it to rejoin, err
                          46<br>
                          Tue Aug 19 11:04:12.096 2014: Killing
                          connection from </tt><tt><tt><b>&lt;EBI5-220
                              IP&gt;</b></tt> because the group is not
                          ready for it to rejoin, err 46<br>
                          Tue Aug 19 11:04:14.224 2014: Killing
                          connection from </tt><tt><tt><tt>&lt;EBI5-043
                              IP&gt;</tt></tt> because the group is not
                          ready for it to rejoin, err 46<br>
                          Tue Aug 19 11:04:20.037 2014: Killing
                          connection from &lt;EBI5-102 IP&gt; because
                          the group is not ready for it to rejoin, err
                          46<br>
                          Tue Aug 19 11:04:23.103 2014: Accepted and
                          connected to </tt><tt><tt><b>&lt;EBI5-220
                              IP&gt;</b></tt> ebi5-220 &lt;c0n618&gt;<br>
                          ...</tt><br>
                        <br>
                        <b><tt>GSS02a ( NSD SERVER)</tt></b><br>
                        <tt>Tue Aug 19 11:03:04.980 2014: Expel
                          &lt;GSS02B IP&gt; (gss02b) request from
                          &lt;EBI5-220 IP&gt; (ebi5-220 in <a
                            moz-do-not-send="true"
                            href="http://ebi-cluster.ebi.ac.uk">ebi-cluster.ebi.ac.uk</a>).
                          Expelling: &lt;EBI5-220 IP&gt; (ebi5-220 in <a
                            moz-do-not-send="true"
                            href="http://ebi-cluster.ebi.ac.uk">ebi-cluster.ebi.ac.uk</a>)</tt><br>
                        <tt>Tue Aug 19 11:03:12.069 2014: Accepted and
                          connected to &lt;EBI5-220 IP&gt; ebi5-220
                          &lt;c0n618&gt;</tt><br>
                        <br>
                        <br>
                      </blockquote>
                    </blockquote>
                  </blockquote>
                  ===============================================<br>
                  <b>EXAMPLE 2</b>:<br>
                  <br>
                  <blockquote>
                    <blockquote>
                      <blockquote><b><tt>EBI5-038</tt></b><br>
                        <tt>Tue Aug 19 11:32:34.227 2014: <b>Disk lease
                            period expired in cluster <a
                              moz-do-not-send="true"
                              href="http://GSS.ebi.ac.uk">GSS.ebi.ac.uk</a>.
                            Attempting to reacquire lease.</b></tt><br>
                        <tt>Tue Aug 19 11:33:34.258 2014: <b>Lease is
                            overdue. Probing cluster <a
                              moz-do-not-send="true"
                              href="http://GSS.ebi.ac.uk">GSS.ebi.ac.uk</a></b></tt><br>
                        <tt>Tue Aug 19 11:35:24.265 2014: Close
                          connection to &lt;GSS02A IP&gt; gss02a
                          &lt;c1n2&gt; (Connection reset by peer).
                          Attempting reconnect.</tt><br>
                        <tt>Tue Aug 19 11:35:24.865 2014: Close
                          connection to &lt;EBI5-014 IP&gt; ebi5-014
                          &lt;c1n457&gt; (Connection reset by peer).
                          Attempting reconnect.</tt><br>
                        <tt>...</tt><br>
                        <tt>LOT MORE RESETS BY PEER</tt><br>
                        <tt>...</tt><br>
                        <tt>Tue Aug 19 11:35:25.096 2014: Close
                          connection to &lt;EBI5-167 IP&gt; ebi5-167
                          &lt;c1n155&gt; (Connection reset by peer).
                          Attempting reconnect.</tt><br>
                        <tt>Tue Aug 19 11:35:25.267 2014: Connecting to
                          &lt;GSS02A IP&gt; gss02a &lt;c1n2&gt;</tt><br>
                        <tt>Tue Aug 19 11:35:25.268 2014: Close
                          connection to &lt;GSS02A IP&gt; gss02a
                          &lt;c1n2&gt; (Connection failed because
                          destination is still processing previous node
                          failure)</tt><br>
                        <tt>Tue Aug 19 11:35:26.267 2014: Retry
                          connection to &lt;GSS02A IP&gt; gss02a
                          &lt;c1n2&gt;</tt><br>
                        <tt>Tue Aug 19 11:35:26.268 2014: Close
                          connection to &lt;GSS02A IP&gt; gss02a
                          &lt;c1n2&gt; (Connection failed because
                          destination is still processing previous node
                          failure)</tt><br>
                        <tt>Tue Aug 19 11:36:24.276 2014: Unable to
                          contact any quorum nodes during cluster probe.</tt><br>
                        <tt>Tue Aug 19 11:36:24.277 2014: <b>Lost
                            membership in cluster <a
                              moz-do-not-send="true"
                              href="http://GSS.ebi.ac.uk">GSS.ebi.ac.uk</a>.
                            Unmounting file systems.</b></tt><br>
                        <br>
                        <b><tt>GSS02a</tt></b><br>
                        <tt>Tue Aug 19 11:35:24.263 2014: Node
                          &lt;EBI5-038 IP&gt; (ebi5-038 in <a
                            moz-do-not-send="true"
                            href="http://ebi-cluster.ebi.ac.uk">ebi-cluster.ebi.ac.uk</a>)
                          <b>is being expelled because of an expired
                            lease.</b> Pings sent: 60. Replies received:
                          60.</tt><br>
                      </blockquote>
                    </blockquote>
                  </blockquote>
                  <br>
                  <br>
                  <br>
                  In example 1 seems that an NSD was not repliyng to the
                  client, but the servers seems working fine.. how can i
                  trace better ( to solve) the problem? <br>
                  <br>
                  In example 2 it seems to me that for some reason the
                  manager are not renewing the lease in time. when this
                  happens , its not a single client. <br>
                  Loads of them fail to get the lease renewed. Why this
                  is happening? how can i trace to the source of the
                  problem?<br>
                  <br>
                  <br>
                  <br>
                  Thanks in advance for any tips.<br>
                  <br>
                  Regards,<br>
                  Salvatore<br>
                  <br>
                  <br>
                  <br>
                  <br>
                  <br>
                  <br>
                  <br>
                  <fieldset class="mimeAttachmentHeader"></fieldset>
                  <br>
                  <pre wrap="">_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at <a moz-do-not-send="true" href="http://gpfsug.org">gpfsug.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a>
</pre>
                </blockquote>
                <br>
                <br>
                <fieldset class="mimeAttachmentHeader"></fieldset>
                <br>
                <pre wrap="">_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at <a moz-do-not-send="true" href="http://gpfsug.org">gpfsug.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a>
</pre>
              </blockquote>
              <br>
            </div>
            _______________________________________________<br>
            gpfsug-discuss mailing list<br>
            gpfsug-discuss at <a moz-do-not-send="true"
              href="http://gpfsug.org">gpfsug.org</a><br>
            <a moz-do-not-send="true"
              href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
          </blockquote>
        </div>
        <br>
      </div>
      <br>
      <fieldset class="mimeAttachmentHeader"></fieldset>
      <br>
      <pre wrap="">_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
<a class="moz-txt-link-freetext" href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a>
</pre>
    </blockquote>
    <br>
  </body>
</html>