<div dir="ltr">Hi,<div><br></div><div>the problem is the cleanup of the tokens and/or the openfile objects. i suggest you open a defect for this. </div><div><br></div><div>sven</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr">On Thu, Jul 12, 2018 at 8:22 AM Billich Heinrich Rainer (PSI) <<a href="mailto:heiner.billich@psi.ch">heiner.billich@psi.ch</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="m_4518286962567634524WordSection1">
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal">Hello Sven,<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">The machine has<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">maxFilesToCache 204800 (2M)<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">it will become a CES node, hence the higher than default value. It\u2019s just a 3 node cluster with remote cluster mount and no activity (yet). But all three nodes are listed as token server by \u2018mmdiag \u2013tokenmgr\u2019.<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Top showed 100% idle on core 55. This matches the kernel messages about rmmod being stuck on core 55.<u></u><u></u></p>
<p class="MsoNormal">I didn\u2019t see a dominating thread/process, but many kernel threads showed 30-40% CPU, in sum that used about 50% of all cpu available.<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">This time mmshutdown did return and left the module loaded, next mmstartup tried to remove the \u2018old\u2019 module and got stuck :-(<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">I append two links to screenshots <u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Thank you,<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Heiner<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><a href="https://pasteboard.co/Hu86DKf.png" target="_blank">https://pasteboard.co/Hu86DKf.png</a><u></u><u></u></p>
<p class="MsoNormal"><a href="https://pasteboard.co/Hu86rg4.png" target="_blank">https://pasteboard.co/Hu86rg4.png</a><u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">If the links don\u2019t work I can post the images to the list.<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Kernel messages:<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">[ 857.791050] CPU: 55 PID: 16429 Comm: rmmod Tainted: G W OEL ------------ 3.10.0-693.17.1.el7.x86_64 #1<u></u><u></u></p>
<p class="MsoNormal">[ 857.842265] Hardware name: HP ProLiant DL380 Gen9/ProLiant DL380 Gen9, BIOS P89 01/22/2018<u></u><u></u></p>
<p class="MsoNormal">[ 857.884938] task: ffff883ffafe8fd0 ti: ffff88342af30000 task.ti: ffff88342af30000<u></u><u></u></p>
<p class="MsoNormal">[ 857.924120] RIP: 0010:[<ffffffff8119202e>] [<ffffffff8119202e>] compound_unlock_irqrestore+0xe/0x20<u></u><u></u></p>
<p class="MsoNormal">[ 857.970708] RSP: 0018:ffff88342af33d38 EFLAGS: 00000246<u></u><u></u></p>
<p class="MsoNormal">[ 857.999742] RAX: 0000000000000000 RBX: ffff88207ffda068 RCX: 00000000000000e5<u></u><u></u></p>
<p class="MsoNormal">[ 858.037165] RDX: 0000000000000246 RSI: 0000000000000246 RDI: 0000000000000246<u></u><u></u></p>
<p class="MsoNormal">[ 858.074416] RBP: ffff88342af33d38 R08: 0000000000000000 R09: 0000000000000000<u></u><u></u></p>
<p class="MsoNormal">[ 858.111519] R10: ffff88207ffcfac0 R11: ffffea00fff40280 R12: 0000000000000200<u></u><u></u></p>
<p class="MsoNormal">[ 858.148421] R13: 00000001fff40280 R14: ffffffff8118cd84 R15: ffff88342af33ce8<u></u><u></u></p>
<p class="MsoNormal">[ 858.185845] FS: 00007fc797d1e740(0000) GS:ffff883fff0c0000(0000) knlGS:0000000000000000<u></u><u></u></p>
<p class="MsoNormal">[ 858.227062] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033<u></u><u></u></p>
<p class="MsoNormal">[ 858.257819] CR2: 00000000004116d0 CR3: 0000003fc2ec0000 CR4: 00000000001607e0<u></u><u></u></p>
<p class="MsoNormal">[ 858.295143] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000<u></u><u></u></p>
<p class="MsoNormal">[ 858.332145] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400<u></u><u></u></p>
<p class="MsoNormal">[ 858.369097] Call Trace:<u></u><u></u></p>
<p class="MsoNormal">[ 858.384829] [<ffffffff816a29f6>] put_compound_page+0x149/0x174<u></u><u></u></p>
<p class="MsoNormal">[ 858.416176] [<ffffffff81192275>] put_page+0x45/0x50<u></u><u></u></p>
<p class="MsoNormal">[ 858.443185] [<ffffffffc09be4ba>] cxiReleaseAndForgetPages+0xda/0x220 [mmfslinux]<u></u><u></u></p>
<p class="MsoNormal">[ 858.481751] [<ffffffffc09beaed>] ? cxiDeallocPageList+0xbd/0x110 [mmfslinux]<u></u><u></u></p>
<p class="MsoNormal">[ 858.518206] [<ffffffffc09bea75>] cxiDeallocPageList+0x45/0x110 [mmfslinux]<u></u><u></u></p>
<p class="MsoNormal">[ 858.554438] [<ffffffff816adfe0>] ? _raw_spin_lock+0x10/0x30<u></u><u></u></p>
<p class="MsoNormal">[ 858.585522] [<ffffffffc09bec6a>] cxiFreeSharedMemory+0x12a/0x130 [mmfslinux]<u></u><u></u></p>
<p class="MsoNormal">[ 858.622670] [<ffffffffc0b69982>] kxFreeAllSharedMemory+0xe2/0x160 [mmfs26]<u></u><u></u></p>
<p class="MsoNormal">[ 858.659246] [<ffffffffc0b54d15>] mmfs+0xc85/0xca0 [mmfs26]<u></u><u></u></p>
<p class="MsoNormal">[ 858.689379] [<ffffffffc09a3d26>] gpfs_clean+0x26/0x30 [mmfslinux]<u></u><u></u></p>
<p class="MsoNormal">[ 858.722330] [<ffffffffc0c9c945>] cleanup_module+0x25/0x30 [mmfs26]<u></u><u></u></p>
<p class="MsoNormal">[ 858.755431] [<ffffffff8110044b>] SyS_delete_module+0x19b/0x300<u></u><u></u></p>
<p class="MsoNormal">[ 858.786882] [<ffffffff816b89fd>] system_call_fastpath+0x16/0x1b<u></u><u></u></p>
<p class="MsoNormal">[ 858.818776] Code: 89 ca 44 89 c1 4c 8d 43 10 e8 6f 2b ff ff 89 c2 48 89 13 5b 5d c3 0f 1f 80 00 00 00 00 55 48 89 e5 f0 80 67 03 fe 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 90 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44<u></u><u></u></p>
<p class="MsoNormal">[ 859.068528] hrtimer: interrupt took 2877171 ns<u></u><u></u></p>
<p class="MsoNormal">[ 870.517924] INFO: rcu_sched self-detected stall on CPU { 55} (t=240003 jiffies g=18437 c=18436 q=194992)<u></u><u></u></p>
<p class="MsoNormal">[ 870.577882] Task dump for CPU 55:<u></u><u></u></p>
<p class="MsoNormal">[ 870.602837] rmmod R running task 0 16429 16374 0x00000008<u></u><u></u></p>
<p class="MsoNormal">[ 870.645206] Call Trace:<u></u><u></u></p>
<p class="MsoNormal">[ 870.666388] <IRQ> [<ffffffff810c58a8>] sched_show_task+0xa8/0x110<u></u><u></u></p>
<p class="MsoNormal">[ 870.704271] [<ffffffff810c9309>] dump_cpu_task+0x39/0x70<u></u><u></u></p>
<p class="MsoNormal">[ 870.738421] [<ffffffff811399f0>] rcu_dump_cpu_stacks+0x90/0xd0<u></u><u></u></p>
<p class="MsoNormal">[ 870.775339] [<ffffffff8113d012>] rcu_check_callbacks+0x442/0x730<u></u><u></u></p>
<p class="MsoNormal">[ 870.812353] [<ffffffff810f4ee0>] ? tick_sched_do_timer+0x50/0x50<u></u><u></u></p>
<p class="MsoNormal">[ 870.848875] [<ffffffff8109c076>] update_process_times+0x46/0x80<u></u><u></u></p>
<p class="MsoNormal">[ 870.884847] [<ffffffff810f4ce0>] tick_sched_handle+0x30/0x70<u></u><u></u></p>
<p class="MsoNormal">[ 870.919740] [<ffffffff810f4f19>] tick_sched_timer+0x39/0x80<u></u><u></u></p>
<p class="MsoNormal">[ 870.953660] [<ffffffff810b6864>] __hrtimer_run_queues+0xd4/0x260<u></u><u></u></p>
<p class="MsoNormal">[ 870.989276] [<ffffffff810b6dff>] hrtimer_interrupt+0xaf/0x1d0<u></u><u></u></p>
<p class="MsoNormal">[ 871.023481] [<ffffffff81053a05>] local_apic_timer_interrupt+0x35/0x60<u></u><u></u></p>
<p class="MsoNormal">[ 871.061233] [<ffffffff816bea4d>] smp_apic_timer_interrupt+0x3d/0x50<u></u><u></u></p>
<p class="MsoNormal">[ 871.097838] [<ffffffff816b9d32>] apic_timer_interrupt+0x232/0x240<u></u><u></u></p>
<p class="MsoNormal">[ 871.133232] <EOI> [<ffffffff816a287e>] ? put_page_testzero+0x8/0x15<u></u><u></u></p>
<p class="MsoNormal">[ 871.170089] [<ffffffff816a29fe>] put_compound_page+0x151/0x174<u></u><u></u></p>
<p class="MsoNormal">[ 871.204221] [<ffffffff81192275>] put_page+0x45/0x50<u></u><u></u></p>
<p class="MsoNormal">[ 871.234554] [<ffffffffc09be4ba>] cxiReleaseAndForgetPages+0xda/0x220 [mmfslinux]<u></u><u></u></p>
<p class="MsoNormal">[ 871.275763] [<ffffffffc09beaed>] ? cxiDeallocPageList+0xbd/0x110 [mmfslinux]<u></u><u></u></p>
<p class="MsoNormal">[ 871.316987] [<ffffffffc09bea75>] cxiDeallocPageList+0x45/0x110 [mmfslinux]<u></u><u></u></p>
<p class="MsoNormal">[ 871.356886] [<ffffffff816adfe0>] ? _raw_spin_lock+0x10/0x30<u></u><u></u></p>
<p class="MsoNormal">[ 871.389455] [<ffffffffc09bec6a>] cxiFreeSharedMemory+0x12a/0x130 [mmfslinux]<u></u><u></u></p>
<p class="MsoNormal">[ 871.429784] [<ffffffffc0b69982>] kxFreeAllSharedMemory+0xe2/0x160 [mmfs26]<u></u><u></u></p>
<p class="MsoNormal">[ 871.468753] [<ffffffffc0b54d15>] mmfs+0xc85/0xca0 [mmfs26]<u></u><u></u></p>
<p class="MsoNormal">[ 871.501196] [<ffffffffc09a3d26>] gpfs_clean+0x26/0x30 [mmfslinux]<u></u><u></u></p>
<p class="MsoNormal">[ 871.536562] [<ffffffffc0c9c945>] cleanup_module+0x25/0x30 [mmfs26]<u></u><u></u></p>
<p class="MsoNormal">[ 871.572110] [<ffffffff8110044b>] SyS_delete_module+0x19b/0x300<u></u><u></u></p>
<p class="MsoNormal">[ 871.606048] [<ffffffff816b89fd>] system_call_fastpath+0x16/0x1b<u></u><u></u></p></div></div><div lang="EN-US" link="#0563C1" vlink="#954F72"><div class="m_4518286962567634524WordSection1">
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal">--<u></u><u></u></p>
<p class="MsoNormal">Paul Scherrer Institut<u></u><u></u></p>
<p class="MsoNormal">Science IT<u></u><u></u></p>
<p class="MsoNormal">Heiner Billich<u></u><u></u></p>
<p class="MsoNormal">WHGA 106<u></u><u></u></p>
<p class="MsoNormal">CH 5232 Villigen PSI<u></u><u></u></p>
<p class="MsoNormal">056 310 36 02<u></u><u></u></p>
<p class="MsoNormal"><a href="https://www.psi.ch" target="_blank">https://www.psi.ch</a><u></u><u></u></p>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
</div></div><div lang="EN-US" link="#0563C1" vlink="#954F72"><div class="m_4518286962567634524WordSection1"><div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black"><<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">gpfsug-discuss-bounces@spectrumscale.org</a>> on behalf of Sven Oehme <<a href="mailto:oehmes@gmail.com" target="_blank">oehmes@gmail.com</a>></span></p></div></div></div><div lang="EN-US" link="#0563C1" vlink="#954F72"><div class="m_4518286962567634524WordSection1"><div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0cm 0cm 0cm"><p class="MsoNormal"><span style="font-size:12.0pt;color:black"><br>
<b>Reply-To: </b>gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.org</a>><br>
</span></p></div></div></div><div lang="EN-US" link="#0563C1" vlink="#954F72"><div class="m_4518286962567634524WordSection1"><div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0cm 0cm 0cm"><p class="MsoNormal"><span style="font-size:12.0pt;color:black"><b>Date: </b>Thursday 12 July 2018 at 15:42</span></p></div></div></div><div lang="EN-US" link="#0563C1" vlink="#954F72"><div class="m_4518286962567634524WordSection1"><div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0cm 0cm 0cm"><p class="MsoNormal"><span style="font-size:12.0pt;color:black"><br>
<b>To: </b>gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.org</a>><br>
<b>Subject: </b>Re: [gpfsug-discuss] /sbin/rmmod mmfs26 hangs on mmshutdown</span></p></div></div></div><div lang="EN-US" link="#0563C1" vlink="#954F72"><div class="m_4518286962567634524WordSection1"><div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0cm 0cm 0cm"><p class="MsoNormal"><span style="font-size:12.0pt;color:black"></span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<p class="MsoNormal"><span style="font-size:13.5pt;font-family:"-webkit-standard",serif;color:black">if that happens it would be interesting what top reports</span><u></u><u></u></p></div></div><div lang="EN-US" link="#0563C1" vlink="#954F72"><div class="m_4518286962567634524WordSection1">
<div>
<p class="MsoNormal"><span style="font-family:"-webkit-standard",serif;color:black"> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"-webkit-standard",serif;color:black">start top in a large resolution window (like 330x80) , press shift-H , this will break it down per Thread, also press 1 to have a list of each cpu individually and see if you
can either spot one core on the top list with 0% idle or on the thread list on the bottom if any of the threads run at 100% core speed.</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"-webkit-standard",serif;color:black">attached is a screenshot which columns to look at , this system is idle, so nothing to see, just to show you where to look </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"-webkit-standard",serif;color:black"> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"-webkit-standard",serif;color:black">does this machine by any chance has either large maxfilestochache or is a token server ? </span><u></u><u></u></p>
</div>
<p class="MsoNormal"><br>
<br>
<br>
<u></u><u></u></p>
</div></div>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
</blockquote></div>