<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Exchange Server">
<!-- converted from rtf -->
<style><!-- .EmailQuote { margin-left: 1pt; padding-left: 4pt; border-left: #800000 2px solid; } --></style>
</head>
<body>
<font face="Calibri" size="2"><span style="font-size:11pt;">
<div>After some puzzling debugging on our new Broadwell servers, all of which slowly became brick-like upon after getting stuck starting GPFS, we discovered that this was already a known issue in the FAQ. Adding “nosmap” to the kernel command line in grub
prevents SMAP from seeing the kernel-userspace memory interactions of GPFS as a reason to slowly grind all cores to a standstill, apparently spinning on stuck locks(?). (Big thanks go to RedHat for turning us on to the answer when we opened a case.)</div>
<div> </div>
<div>From <a href="https://www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.html"><font color="blue"><u>https://www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.html</u></font></a>, section 3.2:</div>
<div> </div>
<div style="padding-left:18pt;"><i>Note:</i><i> </i><i>In order for IBM Spectrum Scale on RHEL 7 to run on the Haswell processor </i></div>
<ul style="margin:0;padding-left:54pt;">
<li><i>Disable the Supervisor Mode Access Prevention (smap) kernel parameter</i></li><li><i>Reboot the RHEL 7 node before using GPFS</i></li></ul>
<div> </div>
<div> </div>
<div>Some observations worth noting:</div>
<div> </div>
<ol style="margin:0;padding-left:36pt;">
<li>We’ve been running for a year with Haswell processors and have hundreds of Haswell RHEL7 nodes which do not exhibit this problem. So maybe this only really affects Broadwell CPUs?<br>
</li><li>It would be very nice for SpectrumScale to take a peek at /proc/cpuinfo and /proc/cmdline before starting up, and refuse to break the host when it has affected processors and kernel without “nosmap”. Instead, an error message describing the fix would have
made my day.<br>
</li><li>I’m going to have to start using a script to diff the FAQ for these gotchas, unless anyone knows of a better way to subscribe just to updates to this doc. </li></ol>
<div> </div>
<div>Thanks,</div>
<div>Paul Sanchez</div>
<div> </div>
</span></font>
</body>
</html>