From leonardo.sala at psi.ch Mon Jun 1 13:23:13 2026 From: leonardo.sala at psi.ch (Leonardo Sala) Date: Mon, 1 Jun 2026 14:23:13 +0200 Subject: [gpfsug-discuss] SMB performance issues with CES Message-ID: <32cdc2ee-8fcb-4767-807a-a14369c4cd28@psi.ch> Hallo everyone I have just notices some possible performance issues on SMB when using CES, at least compared to NFS. Using NFSv4 I can easily get 1 GB/s on a single client (linux), while when using a CIFS mount I get tops 500 MB/s on a single client. Using two separate servers as clients gets a 1 GB/s total throughput on the CES server, so I would say there not a strict limitation on the CES server itself. Any hints on how to improve this / debugging hints? I spent some time on IBM documentation and some user groups talks, but got no improvement... Thanks! cheers leo -- Paul Scherrer Institut Dr. Leonardo Sala Group Leader Data Analysis and Research Infrastructure Group Leader Data Curation a.i. Deputy Department Head Science IT Infrastructure and Services department Science IT Infrastructure and Services department (AWI) OBBA/230 Forschungstrasse 111 5232 Villigen PSI Switzerland Phone: +41 56 310 3369 leonardo.sala at psi.ch www.psi.ch -------------- next part -------------- An HTML attachment was scrubbed... URL: From janfrode at tanso.net Mon Jun 1 14:24:41 2026 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Mon, 1 Jun 2026 15:24:41 +0200 Subject: [gpfsug-discuss] SMB performance issues with CES In-Reply-To: <32cdc2ee-8fcb-4767-807a-a14369c4cd28@psi.ch> References: <32cdc2ee-8fcb-4767-807a-a14369c4cd28@psi.ch> Message-ID: Not too long ago there were some reports about SMB signing being very costly. Signing would give approximately 500 MB/s, while encryption would give closer to 1200 MB/s. Signing is controlled by Security Settings -> Local Policies -> Security options "Microsoft network client: Digitally Sign Communications (always)". -jf On Mon, Jun 1, 2026 at 2:24 PM Leonardo Sala via gpfsug-discuss wrote: > > Hallo everyone > > I have just notices some possible performance issues on SMB when using CES, at least compared to NFS. Using NFSv4 I can easily get 1 GB/s on a single client (linux), while when using a CIFS mount I get tops 500 MB/s on a single client. Using two separate servers as clients gets a 1 GB/s total throughput on the CES server, so I would say there not a strict limitation on the CES server itself. Any hints on how to improve this / debugging hints? I spent some time on IBM documentation and some user groups talks, but got no improvement... > > Thanks! > > cheers > > leo > > -- > Paul Scherrer Institut > Dr. Leonardo Sala > Group Leader Data Analysis and Research Infrastructure > Group Leader Data Curation a.i. > Deputy Department Head Science IT Infrastructure and Services department > Science IT Infrastructure and Services department (AWI) > OBBA/230 > Forschungstrasse 111 > 5232 Villigen PSI > Switzerland > > Phone: +41 56 310 3369 > leonardo.sala at psi.ch > www.psi.ch > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org From janfrode at tanso.net Mon Jun 1 14:24:41 2026 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Mon, 1 Jun 2026 15:24:41 +0200 Subject: [gpfsug-discuss] SMB performance issues with CES In-Reply-To: <32cdc2ee-8fcb-4767-807a-a14369c4cd28@psi.ch> References: <32cdc2ee-8fcb-4767-807a-a14369c4cd28@psi.ch> Message-ID: Not too long ago there were some reports about SMB signing being very costly. Signing would give approximately 500 MB/s, while encryption would give closer to 1200 MB/s. Signing is controlled by Security Settings -> Local Policies -> Security options "Microsoft network client: Digitally Sign Communications (always)". -jf On Mon, Jun 1, 2026 at 2:24 PM Leonardo Sala via gpfsug-discuss wrote: > > Hallo everyone > > I have just notices some possible performance issues on SMB when using CES, at least compared to NFS. Using NFSv4 I can easily get 1 GB/s on a single client (linux), while when using a CIFS mount I get tops 500 MB/s on a single client. Using two separate servers as clients gets a 1 GB/s total throughput on the CES server, so I would say there not a strict limitation on the CES server itself. Any hints on how to improve this / debugging hints? I spent some time on IBM documentation and some user groups talks, but got no improvement... > > Thanks! > > cheers > > leo > > -- > Paul Scherrer Institut > Dr. Leonardo Sala > Group Leader Data Analysis and Research Infrastructure > Group Leader Data Curation a.i. > Deputy Department Head Science IT Infrastructure and Services department > Science IT Infrastructure and Services department (AWI) > OBBA/230 > Forschungstrasse 111 > 5232 Villigen PSI > Switzerland > > Phone: +41 56 310 3369 > leonardo.sala at psi.ch > www.psi.ch > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org From leonardo.sala at psi.ch Mon Jun 1 16:38:51 2026 From: leonardo.sala at psi.ch (Leonardo Sala) Date: Mon, 1 Jun 2026 17:38:51 +0200 Subject: [gpfsug-discuss] SMB performance issues with CES In-Reply-To: References: <32cdc2ee-8fcb-4767-807a-a14369c4cd28@psi.ch> Message-ID: Hi Jan-Frode interesting! Is this something I could apply to the linux clients, too, or is this only for windows clients? cheers leo Paul Scherrer Institut Dr. Leonardo Sala Group Leader Data Analysis and Research Infrastructure Group Leader Data Curation a.i. Deputy Department Head Science IT Infrastructure and Services department Science IT Infrastructure and Services department (AWI) OBBA/230 Forschungstrasse 111 5232 Villigen PSI Switzerland Phone: +41 56 310 3369 leonardo.sala at psi.ch www.psi.ch On 6/1/26 15:24, Jan-Frode Myklebust wrote: > Not too long ago there were some reports about SMB signing being very > costly. Signing would give approximately 500 MB/s, while encryption > would give closer to 1200 MB/s. Signing is controlled by Security > Settings -> Local Policies -> Security options "Microsoft network > client: Digitally Sign Communications (always)". > > > -jf > > On Mon, Jun 1, 2026 at 2:24 PM Leonardo Sala via gpfsug-discuss > wrote: >> Hallo everyone >> >> I have just notices some possible performance issues on SMB when using CES, at least compared to NFS. Using NFSv4 I can easily get 1 GB/s on a single client (linux), while when using a CIFS mount I get tops 500 MB/s on a single client. Using two separate servers as clients gets a 1 GB/s total throughput on the CES server, so I would say there not a strict limitation on the CES server itself. Any hints on how to improve this / debugging hints? I spent some time on IBM documentation and some user groups talks, but got no improvement... >> >> Thanks! >> >> cheers >> >> leo >> >> -- >> Paul Scherrer Institut >> Dr. Leonardo Sala >> Group Leader Data Analysis and Research Infrastructure >> Group Leader Data Curation a.i. >> Deputy Department Head Science IT Infrastructure and Services department >> Science IT Infrastructure and Services department (AWI) >> OBBA/230 >> Forschungstrasse 111 >> 5232 Villigen PSI >> Switzerland >> >> Phone: +41 56 310 3369 >> leonardo.sala at psi.ch >> www.psi.ch >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at gpfsug.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From leonardo.sala at psi.ch Mon Jun 1 16:38:51 2026 From: leonardo.sala at psi.ch (Leonardo Sala) Date: Mon, 1 Jun 2026 17:38:51 +0200 Subject: [gpfsug-discuss] SMB performance issues with CES In-Reply-To: References: <32cdc2ee-8fcb-4767-807a-a14369c4cd28@psi.ch> Message-ID: Hi Jan-Frode interesting! Is this something I could apply to the linux clients, too, or is this only for windows clients? cheers leo Paul Scherrer Institut Dr. Leonardo Sala Group Leader Data Analysis and Research Infrastructure Group Leader Data Curation a.i. Deputy Department Head Science IT Infrastructure and Services department Science IT Infrastructure and Services department (AWI) OBBA/230 Forschungstrasse 111 5232 Villigen PSI Switzerland Phone: +41 56 310 3369 leonardo.sala at psi.ch www.psi.ch On 6/1/26 15:24, Jan-Frode Myklebust wrote: > Not too long ago there were some reports about SMB signing being very > costly. Signing would give approximately 500 MB/s, while encryption > would give closer to 1200 MB/s. Signing is controlled by Security > Settings -> Local Policies -> Security options "Microsoft network > client: Digitally Sign Communications (always)". > > > -jf > > On Mon, Jun 1, 2026 at 2:24 PM Leonardo Sala via gpfsug-discuss > wrote: >> Hallo everyone >> >> I have just notices some possible performance issues on SMB when using CES, at least compared to NFS. Using NFSv4 I can easily get 1 GB/s on a single client (linux), while when using a CIFS mount I get tops 500 MB/s on a single client. Using two separate servers as clients gets a 1 GB/s total throughput on the CES server, so I would say there not a strict limitation on the CES server itself. Any hints on how to improve this / debugging hints? I spent some time on IBM documentation and some user groups talks, but got no improvement... >> >> Thanks! >> >> cheers >> >> leo >> >> -- >> Paul Scherrer Institut >> Dr. Leonardo Sala >> Group Leader Data Analysis and Research Infrastructure >> Group Leader Data Curation a.i. >> Deputy Department Head Science IT Infrastructure and Services department >> Science IT Infrastructure and Services department (AWI) >> OBBA/230 >> Forschungstrasse 111 >> 5232 Villigen PSI >> Switzerland >> >> Phone: +41 56 310 3369 >> leonardo.sala at psi.ch >> www.psi.ch >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at gpfsug.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From janfrode at tanso.net Tue Jun 2 06:28:06 2026 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Tue, 2 Jun 2026 07:28:06 +0200 Subject: [gpfsug-discuss] SMB performance issues with CES In-Reply-To: References: <32cdc2ee-8fcb-4767-807a-a14369c4cd28@psi.ch>

Message-ID: Sorry, I don't know. I tried checking, and it looks like signing is disabled by default in the linux cifs module: cifsfs.c:MODULE_PARM_DESC(enable_negotiate_signing, "Enable negotiating packet signing algorithm with server. Default: n/N/0"); Elsewhere I find that mount option sec=ntlmssp also might influence this to only do signing if server requires it.. -jf On Mon, Jun 1, 2026 at 5:39 PM Leonardo Sala wrote: > > Hi Jan-Frode > > interesting! Is this something I could apply to the linux clients, too, or is this only for windows clients? > > cheers > > leo > > Paul Scherrer Institut > Dr. Leonardo Sala > Group Leader Data Analysis and Research Infrastructure > Group Leader Data Curation a.i. > Deputy Department Head Science IT Infrastructure and Services department > Science IT Infrastructure and Services department (AWI) > OBBA/230 > Forschungstrasse 111 > 5232 Villigen PSI > Switzerland > > Phone: +41 56 310 3369 > leonardo.sala at psi.ch > www.psi.ch > > On 6/1/26 15:24, Jan-Frode Myklebust wrote: > > Not too long ago there were some reports about SMB signing being very > costly. Signing would give approximately 500 MB/s, while encryption > would give closer to 1200 MB/s. Signing is controlled by Security > Settings -> Local Policies -> Security options "Microsoft network > client: Digitally Sign Communications (always)". > > > -jf > > On Mon, Jun 1, 2026 at 2:24 PM Leonardo Sala via gpfsug-discuss > wrote: > > Hallo everyone > > I have just notices some possible performance issues on SMB when using CES, at least compared to NFS. Using NFSv4 I can easily get 1 GB/s on a single client (linux), while when using a CIFS mount I get tops 500 MB/s on a single client. Using two separate servers as clients gets a 1 GB/s total throughput on the CES server, so I would say there not a strict limitation on the CES server itself. Any hints on how to improve this / debugging hints? I spent some time on IBM documentation and some user groups talks, but got no improvement... > > Thanks! > > cheers > > leo > > -- > Paul Scherrer Institut > Dr. Leonardo Sala > Group Leader Data Analysis and Research Infrastructure > Group Leader Data Curation a.i. > Deputy Department Head Science IT Infrastructure and Services department > Science IT Infrastructure and Services department (AWI) > OBBA/230 > Forschungstrasse 111 > 5232 Villigen PSI > Switzerland > > Phone: +41 56 310 3369 > leonardo.sala at psi.ch > www.psi.ch > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org From jonathan.buzzard at strath.ac.uk Fri Jun 19 17:48:47 2026 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Fri, 19 Jun 2026 17:48:47 +0100 Subject: [gpfsug-discuss] DSS-G sharing IPMI network Message-ID: <6f66e42e-fa87-4468-84f0-fea3fd035539@strath.ac.uk> So it says this in the documentation In DSS-G, the recommendation is to share the IPMI and administrative LAN interfaces on the same Ethernet port of the DSS server, rather than using the dedicated network connector of the XCC for IPMI. The question is what is the reasoning behind this? I want that management interface strictly separated from my "normal" network. What is the value of having it "shared" rather than separate? Noting that we have never used the OCP/LOM card on our DSS-G servers for anything other than provisioning and then immediately unconfigured then and relied on the "HPC" network for everything. Unhelpfully the servers have come configured from Lenovo in dedicated mode and I cannot see anything in the XCC to change it. Does it require booting into BIOS setup to change? JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From ncalimet at lenovo.com Fri Jun 19 18:23:06 2026 From: ncalimet at lenovo.com (Nicolas CALIMET) Date: Fri, 19 Jun 2026 17:23:06 +0000 Subject: [gpfsug-discuss] [External] DSS-G sharing IPMI network In-Reply-To: <6f66e42e-fa87-4468-84f0-fea3fd035539@strath.ac.uk> References: <6f66e42e-fa87-4468-84f0-fea3fd035539@strath.ac.uk> Message-ID: Hi, The recommendation is likely originating from environments where switch ports and cabling may be too constrained if using the dedicated LOM/OCP port. There is also a correlation with setting up the Confluent environment. So to answer your question - I'm assuming you're dealing with a V3 server (SR655 V3), which indeed may have the dedicated port enabled by default: $ nodeconfig dss45 -c IMM.SharedNicMode dss45: IMM.SharedNicMode: Dedicated (Default: Dedicated) $ nodeconfig dss45 IMM.SharedNicMode=jnk dss45: jnk is not a valid value for IMM.SharedNicMode (Shared:nic0,Shared,Dedicated,Shared:nic3,Shared:nic4,Shared:nic5,Shared:nic6,Shared:nic7,Shared:nic8) So if you want to stick to the recommendation, "nodeconfig IMM.SharedNicMode=Shared" should be what you're looking for if you pick up the correct physical OCP port as documented. -- Nicolas Calimet, PhD | HPC System Architect | Lenovo ISG | https://www.lenovo.com/dssg -----Original Message----- From: gpfsug-discuss On Behalf Of Jonathan Buzzard via gpfsug-discuss Sent: Friday, June 19, 2026 18:49 To: gpfsug-discuss at spectrumscale.org Cc: Jonathan Buzzard Subject: [External] [gpfsug-discuss] DSS-G sharing IPMI network So it says this in the documentation In DSS-G, the recommendation is to share the IPMI and administrative LAN interfaces on the same Ethernet port of the DSS server, rather than using the dedicated network connector of the XCC for IPMI. The question is what is the reasoning behind this? I want that management interface strictly separated from my "normal" network. What is the value of having it "shared" rather than separate? Noting that we have never used the OCP/LOM card on our DSS-G servers for anything other than provisioning and then immediately unconfigured then and relied on the "HPC" network for everything. Unhelpfully the servers have come configured from Lenovo in dedicated mode and I cannot see anything in the XCC to change it. Does it require booting into BIOS setup to change? JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org From ncalimet at lenovo.com Fri Jun 19 18:23:06 2026 From: ncalimet at lenovo.com (Nicolas CALIMET) Date: Fri, 19 Jun 2026 17:23:06 +0000 Subject: [gpfsug-discuss] [External] DSS-G sharing IPMI network In-Reply-To: <6f66e42e-fa87-4468-84f0-fea3fd035539@strath.ac.uk> References: <6f66e42e-fa87-4468-84f0-fea3fd035539@strath.ac.uk> Message-ID: Hi, The recommendation is likely originating from environments where switch ports and cabling may be too constrained if using the dedicated LOM/OCP port. There is also a correlation with setting up the Confluent environment. So to answer your question - I'm assuming you're dealing with a V3 server (SR655 V3), which indeed may have the dedicated port enabled by default: $ nodeconfig dss45 -c IMM.SharedNicMode dss45: IMM.SharedNicMode: Dedicated (Default: Dedicated) $ nodeconfig dss45 IMM.SharedNicMode=jnk dss45: jnk is not a valid value for IMM.SharedNicMode (Shared:nic0,Shared,Dedicated,Shared:nic3,Shared:nic4,Shared:nic5,Shared:nic6,Shared:nic7,Shared:nic8) So if you want to stick to the recommendation, "nodeconfig IMM.SharedNicMode=Shared" should be what you're looking for if you pick up the correct physical OCP port as documented. -- Nicolas Calimet, PhD | HPC System Architect | Lenovo ISG | https://www.lenovo.com/dssg -----Original Message----- From: gpfsug-discuss On Behalf Of Jonathan Buzzard via gpfsug-discuss Sent: Friday, June 19, 2026 18:49 To: gpfsug-discuss at spectrumscale.org Cc: Jonathan Buzzard Subject: [External] [gpfsug-discuss] DSS-G sharing IPMI network So it says this in the documentation In DSS-G, the recommendation is to share the IPMI and administrative LAN interfaces on the same Ethernet port of the DSS server, rather than using the dedicated network connector of the XCC for IPMI. The question is what is the reasoning behind this? I want that management interface strictly separated from my "normal" network. What is the value of having it "shared" rather than separate? Noting that we have never used the OCP/LOM card on our DSS-G servers for anything other than provisioning and then immediately unconfigured then and relied on the "HPC" network for everything. Unhelpfully the servers have come configured from Lenovo in dedicated mode and I cannot see anything in the XCC to change it. Does it require booting into BIOS setup to change? JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org From jonathan.buzzard at strath.ac.uk Wed Jun 24 11:25:50 2026 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Wed, 24 Jun 2026 11:25:50 +0100 Subject: [gpfsug-discuss] [External] DSS-G sharing IPMI network In-Reply-To: References: <6f66e42e-fa87-4468-84f0-fea3fd035539@strath.ac.uk> Message-ID: <1bb694c5-3965-4282-b2a6-050aec8e4256@strath.ac.uk> On 19/06/2026 18:23, Nicolas CALIMET wrote: > > Hi, > > The recommendation is likely originating from environments where > switch ports and cabling may be too constrained if using the > dedicated LOM/OCP port. There is also a correlation with setting up > the Confluent environment. > I just don't get what that correlation might be. Noting I having setup the servers with dedicated connections for the XCC using confluent without issue. I guess that is my fault for presuming that because they came configured in dedicated mode that is what I should be using. Are we an outlier having a dedicated management network? I would have thought that was the norm/best practice? > So to answer your question - I'm assuming you're dealing with a V3 > server (SR655 V3), which indeed may have the dedicated port enabled > by default: > > $ nodeconfig dss45 -c IMM.SharedNicMode dss45: IMM.SharedNicMode: > Dedicated (Default: Dedicated) > > $ nodeconfig dss45 IMM.SharedNicMode=jnk dss45: jnk is not a valid > value for IMM.SharedNicMode > (Shared:nic0,Shared,Dedicated,Shared:nic3,Shared:nic4,Shared:nic5,Shared:nic6,Shared:nic7,Shared:nic8) > > > So if you want to stick to the recommendation, "nodeconfig > IMM.SharedNicMode=Shared" should be what you're looking for if you > pick up the correct physical OCP port as documented. > Is there a support issue with just staying with the dedicated NIC for the XCC connection? I was about to start the migration off our old DSS-G but if I need to rejig this then now is the time. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From novosirj at rutgers.edu Wed Jun 24 17:37:11 2026 From: novosirj at rutgers.edu (Ryan Novosielski) Date: Wed, 24 Jun 2026 16:37:11 +0000 Subject: [gpfsug-discuss] [External] DSS-G sharing IPMI network In-Reply-To: <1bb694c5-3965-4282-b2a6-050aec8e4256@strath.ac.uk> References: <6f66e42e-fa87-4468-84f0-fea3fd035539@strath.ac.uk> <1bb694c5-3965-4282-b2a6-050aec8e4256@strath.ac.uk> Message-ID: On Jun 24, 2026, at 06:25, Jonathan Buzzard via gpfsug-discuss wrote: On 19/06/2026 18:23, Nicolas CALIMET wrote: Hi, The recommendation is likely originating from environments where switch ports and cabling may be too constrained if using the dedicated LOM/OCP port. There is also a correlation with setting up the Confluent environment. I just don't get what that correlation might be. Noting I having setup the servers with dedicated connections for the XCC using confluent without issue. I guess that is my fault for presuming that because they came configured in dedicated mode that is what I should be using. Are we an outlier having a dedicated management network? I would have thought that was the norm/best practice? We use shared networking for everything. Less of a hassle overall, less money spent in switches, etc. I know the support was at one time a bit shoddy for VLANs on the BMCs, but these days I think it’s pretty good. We do use the dedicated ports for our PHI/secure system, but I think if we were speccing it today, we’d have used VLANs instead. Dedicated doesn’t necessarily mean physical. This has also been a hiccup in our setup, just because the delivery default is now different. Lenovo Confluent actually has a script called configbmc that you can use with genesis, and so we set up the machines and boot them, genesis fixes the setting, and we continue on. So to answer your question - I'm assuming you're dealing with a V3 server (SR655 V3), which indeed may have the dedicated port enabled by default: $ nodeconfig dss45 -c IMM.SharedNicMode dss45: IMM.SharedNicMode: Dedicated (Default: Dedicated) $ nodeconfig dss45 IMM.SharedNicMode=jnk dss45: jnk is not a valid value for IMM.SharedNicMode (Shared:nic0,Shared,Dedicated,Shared:nic3,Shared:nic4,Shared:nic5,Shared:nic6,Shared:nic7,Shared:nic8) So if you want to stick to the recommendation, "nodeconfig IMM.SharedNicMode=Shared" should be what you're looking for if you pick up the correct physical OCP port as documented. Is there a support issue with just staying with the dedicated NIC for the XCC connection? I was about to start the migration off our old DSS-G but if I need to rejig this then now is the time. I believe the most you’re likely to run into is the scripts expecting a “shared” connection. I can’t recall as to whether they will try to enforce it these days. I think no? -- #BlackLivesMatter ____ || \\UTGERS, |---------------------------*O*--------------------------- ||_// the State | Ryan Novosielski (he/him) - novosirj at rutgers.edu || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus || \\ of NJ | Office of Advanced Research Computing - Newark `' -------------- next part -------------- An HTML attachment was scrubbed... URL: From novosirj at rutgers.edu Wed Jun 24 17:37:11 2026 From: novosirj at rutgers.edu (Ryan Novosielski) Date: Wed, 24 Jun 2026 16:37:11 +0000 Subject: [gpfsug-discuss] [External] DSS-G sharing IPMI network In-Reply-To: <1bb694c5-3965-4282-b2a6-050aec8e4256@strath.ac.uk> References: <6f66e42e-fa87-4468-84f0-fea3fd035539@strath.ac.uk> <1bb694c5-3965-4282-b2a6-050aec8e4256@strath.ac.uk> Message-ID: On Jun 24, 2026, at 06:25, Jonathan Buzzard via gpfsug-discuss wrote: On 19/06/2026 18:23, Nicolas CALIMET wrote: Hi, The recommendation is likely originating from environments where switch ports and cabling may be too constrained if using the dedicated LOM/OCP port. There is also a correlation with setting up the Confluent environment. I just don't get what that correlation might be. Noting I having setup the servers with dedicated connections for the XCC using confluent without issue. I guess that is my fault for presuming that because they came configured in dedicated mode that is what I should be using. Are we an outlier having a dedicated management network? I would have thought that was the norm/best practice? We use shared networking for everything. Less of a hassle overall, less money spent in switches, etc. I know the support was at one time a bit shoddy for VLANs on the BMCs, but these days I think it’s pretty good. We do use the dedicated ports for our PHI/secure system, but I think if we were speccing it today, we’d have used VLANs instead. Dedicated doesn’t necessarily mean physical. This has also been a hiccup in our setup, just because the delivery default is now different. Lenovo Confluent actually has a script called configbmc that you can use with genesis, and so we set up the machines and boot them, genesis fixes the setting, and we continue on. So to answer your question - I'm assuming you're dealing with a V3 server (SR655 V3), which indeed may have the dedicated port enabled by default: $ nodeconfig dss45 -c IMM.SharedNicMode dss45: IMM.SharedNicMode: Dedicated (Default: Dedicated) $ nodeconfig dss45 IMM.SharedNicMode=jnk dss45: jnk is not a valid value for IMM.SharedNicMode (Shared:nic0,Shared,Dedicated,Shared:nic3,Shared:nic4,Shared:nic5,Shared:nic6,Shared:nic7,Shared:nic8) So if you want to stick to the recommendation, "nodeconfig IMM.SharedNicMode=Shared" should be what you're looking for if you pick up the correct physical OCP port as documented. Is there a support issue with just staying with the dedicated NIC for the XCC connection? I was about to start the migration off our old DSS-G but if I need to rejig this then now is the time. I believe the most you’re likely to run into is the scripts expecting a “shared” connection. I can’t recall as to whether they will try to enforce it these days. I think no? -- #BlackLivesMatter ____ || \\UTGERS, |---------------------------*O*--------------------------- ||_// the State | Ryan Novosielski (he/him) - novosirj at rutgers.edu || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus || \\ of NJ | Office of Advanced Research Computing - Newark `' -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncalimet at lenovo.com Thu Jun 25 08:18:36 2026 From: ncalimet at lenovo.com (Nicolas CALIMET) Date: Thu, 25 Jun 2026 07:18:36 +0000 Subject: [gpfsug-discuss] [External] DSS-G sharing IPMI network In-Reply-To: <1bb694c5-3965-4282-b2a6-050aec8e4256@strath.ac.uk> References: <6f66e42e-fa87-4468-84f0-fea3fd035539@strath.ac.uk> <1bb694c5-3965-4282-b2a6-050aec8e4256@strath.ac.uk> Message-ID: Confluent has the hardwaremanagement.port node attribute: https://hpc.lenovo.com/users/documentation/confluentosdeploy.html DSS-G is agnostic toward shared/dedicated here. -- Nicolas Calimet, PhD | HPC System Architect | Lenovo ISG | https://www.lenovo.com/dssg -----Original Message----- From: Jonathan Buzzard Sent: Wednesday, June 24, 2026 12:26 To: Nicolas CALIMET ; gpfsug-discuss at spectrumscale.org Subject: Re: [External] [gpfsug-discuss] DSS-G sharing IPMI network On 19/06/2026 18:23, Nicolas CALIMET wrote: > > Hi, > > The recommendation is likely originating from environments where > switch ports and cabling may be too constrained if using the > dedicated LOM/OCP port. There is also a correlation with setting up > the Confluent environment. > I just don't get what that correlation might be. Noting I having setup the servers with dedicated connections for the XCC using confluent without issue. I guess that is my fault for presuming that because they came configured in dedicated mode that is what I should be using. Are we an outlier having a dedicated management network? I would have thought that was the norm/best practice? > So to answer your question - I'm assuming you're dealing with a V3 > server (SR655 V3), which indeed may have the dedicated port enabled > by default: > > $ nodeconfig dss45 -c IMM.SharedNicMode dss45: IMM.SharedNicMode: > Dedicated (Default: Dedicated) > > $ nodeconfig dss45 IMM.SharedNicMode=jnk dss45: jnk is not a valid > value for IMM.SharedNicMode > (Shared:nic0,Shared,Dedicated,Shared:nic3,Shared:nic4,Shared:nic5,Shared:nic6,Shared:nic7,Shared:nic8) > > > So if you want to stick to the recommendation, "nodeconfig > IMM.SharedNicMode=Shared" should be what you're looking for if you > pick up the correct physical OCP port as documented. > Is there a support issue with just staying with the dedicated NIC for the XCC connection? I was about to start the migration off our old DSS-G but if I need to rejig this then now is the time. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From jonathan.buzzard at strath.ac.uk Thu Jun 25 11:26:22 2026 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Thu, 25 Jun 2026 11:26:22 +0100 Subject: [gpfsug-discuss] [External] DSS-G sharing IPMI network In-Reply-To: References: <6f66e42e-fa87-4468-84f0-fea3fd035539@strath.ac.uk> <1bb694c5-3965-4282-b2a6-050aec8e4256@strath.ac.uk> Message-ID: On 24/06/2026 17:37, Ryan Novosielski wrote: > We use shared networking for everything. Less of a hassle overall, less > money spent in switches, etc. I know the support was at one time a bit > shoddy for VLANs on the BMCs, but these days I think it’s pretty good. > We do use the dedicated ports for our PHI/secure system, but I think if > we were speccing it today, we’d have used VLANs instead. Dedicated > doesn’t necessarily mean physical. > For us that would require every server to come with a shared port that does at least 10Gbps on an SFP+ which even in 2026 is not the case. Even then you cannot put the shared port in a bond if it is being shared so that is a number of our servers that would be ruled out of using a shared port for the IPMI. Then there is all the PDU's and other "stuff" that is never going to come with something suitable for hooking up to the primary network. Hence a dedicated management network. It's not like 1Gbps Ethernet switches are expensive. My primary network switches are five figures a pop, so three figure management network switches that last a decade are an inconsequential expense :-) JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG