From sadaniel at us.ibm.com Thu Jul 10 17:16:23 2025 From: sadaniel at us.ibm.com (Steve Daniels) Date: Thu, 10 Jul 2025 16:16:23 +0000 Subject: [gpfsug-discuss] mmdsh rest api command In-Reply-To: <01000197f5091206-156eef6c-fcd1-44f7-92fe-6d8f692226e3-000000@email.amazonses.com> References: <01000197f5091206-156eef6c-fcd1-44f7-92fe-6d8f692226e3-000000@email.amazonses.com> Message-ID: All, Please review the idea below and vote if it is important to you. The mmdsh command is currently based on ssh and will not have a replacement unless development is convinced it will impact your ability to manage your clusters effectively. Also, if you have other ideas that you want the community to consider, please submit them for IBM’s consideration here - https://ideas.ibm.com/ Thanks, Steve Thank you for submitting your idea: ESS-I-114 mmdsh rest api command With the transition from ssh-based commanding to the Rest API, the functionality of mmdsh needs to be maintained. Currently, this command's functionality will no longer be [IBM System Storage Ideas Portal] Thank you for submitting your idea: ESS-I-114 mmdsh rest api command With the transition from ssh-based commanding to the Rest API, the functionality of mmdsh needs to be maintained. Currently, this command's functionality will no longer be supported once a cluster is migrated off of ssh-based communications. Many administrators leverage the command for day-to-day management of the cluster. It is important to have a tightly integrated command to be able to pull information from some or all of the nodes. View idea You're receiving notifications because you created this idea. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.buzzard at strath.ac.uk Mon Jul 21 16:21:13 2025 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Mon, 21 Jul 2025 16:21:13 +0100 Subject: [gpfsug-discuss] mmdsh rest api command In-Reply-To: References: <01000197f5091206-156eef6c-fcd1-44f7-92fe-6d8f692226e3-000000@email.amazonses.com> Message-ID: On 10/07/2025 17:16, Steve Daniels wrote: > All, > > > Please review the idea below and vote if it is important to you. The > mmdsh command is currently based on ssh and will not have a replacement > unless development is convinced it will impact your ability to manage > your clusters effectively. > I have been thinking about this and for what it's worth here's my tuppence on the subject :-) Having only ever used GPFS in the context of an HPC system I have never ever used mmdsh in all the many years (nearly two decades now) I have been using GPFS. My guess is in 2025 people are using the likes of Ansible, SaltStack, Puppet etc. to manage their GPFS cluster. If not what are you doing, rocking it like it's still the naught's is unprofessional IMHO. For the ad hoc stuff they are likely going to be using the likes of xdsh (anything deployed via xCAT/Confluent will have that out the box) or pdsh etc. for running arbitrary commands on groups of nodes. As such mmdsh is reinventing the wheel which is a *bad* thing. There are better tools for what mmdsh does so if someone actually needs that functionality then they can install xdsh/pdsh etc. if they have not already and get the functionality regardless of the existence of mmdsh. I would note at this point that not every node I manage runs GPFS but every node I manage *can* respond to the multi node SSH setup I am using (xdsh because Confluent, because DSS-G). This probably explains why I have never used mmdsh, because it has never covered all the bases, and as it will never cover all the basis it's just another tool to learn and well I haven't got time or the brain space for that. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From sadaniel at us.ibm.com Mon Jul 21 17:45:37 2025 From: sadaniel at us.ibm.com (Steve Daniels) Date: Mon, 21 Jul 2025 16:45:37 +0000 Subject: [gpfsug-discuss] mmdsh rest api command In-Reply-To: References: <01000197f5091206-156eef6c-fcd1-44f7-92fe-6d8f692226e3-000000@email.amazonses.com> Message-ID: Jonathan, et al, The https://ibm.com/ideas website is where you can submit requests for enhancements. If you are having any issues with voting, please let us know. It is simply a thumbs up for a vote. I had submitted the idea on behalf of my clients. I will certainly look into SaltStack. I am familiar with the other commands mentioned, below. The main use I have for mmdsh is the collection/examination of logs and checking on the health of the network (finding down links, bad mtu, misconfigured bonds, etc) . I also have used it for connectivity checks prior to the advent of the mmnetverify command which is a better way. Always looking for a better way, so any ideas are welcome. Aren't xcat, pdsh, etc, based on passwordless root ssh as well? If so, they don't solve my clients issues. I don't see them as better than mmdsh just different authors of the same type of tool. Thanks for the feedback, Steve Steven A. Daniels Fax and Voice: 303-810-1229 -----Original Message----- From: gpfsug-discuss On Behalf Of Jonathan Buzzard Sent: Monday, July 21, 2025 9:21 AM To: gpfsug-discuss at gpfsug.org Subject: [EXTERNAL] Re: [gpfsug-discuss] mmdsh rest api command On 10/07/2025 17:16, Steve Daniels wrote: > All, > > > Please review the idea below and vote if it is important to you. The > mmdsh command is currently based on ssh and will not have a > replacement unless development is convinced it will impact your > ability to manage your clusters effectively. > I have been thinking about this and for what it's worth here's my tuppence on the subject :-) Having only ever used GPFS in the context of an HPC system I have never ever used mmdsh in all the many years (nearly two decades now) I have been using GPFS. My guess is in 2025 people are using the likes of Ansible, SaltStack, Puppet etc. to manage their GPFS cluster. If not what are you doing, rocking it like it's still the naught's is unprofessional IMHO. For the ad hoc stuff they are likely going to be using the likes of xdsh (anything deployed via xCAT/Confluent will have that out the box) or pdsh etc. for running arbitrary commands on groups of nodes. As such mmdsh is reinventing the wheel which is a *bad* thing. There are better tools for what mmdsh does so if someone actually needs that functionality then they can install xdsh/pdsh etc. if they have not already and get the functionality regardless of the existence of mmdsh. I would note at this point that not every node I manage runs GPFS but every node I manage *can* respond to the multi node SSH setup I am using (xdsh because Confluent, because DSS-G). This probably explains why I have never used mmdsh, because it has never covered all the bases, and as it will never cover all the basis it's just another tool to learn and well I haven't got time or the brain space for that. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org From anacreo at gmail.com Mon Jul 21 18:04:18 2025 From: anacreo at gmail.com (Alec) Date: Mon, 21 Jul 2025 10:04:18 -0700 Subject: [gpfsug-discuss] mmdsh rest api command In-Reply-To: References: <01000197f5091206-156eef6c-fcd1-44f7-92fe-6d8f692226e3-000000@email.amazonses.com> Message-ID: I do agree with JAB on this one. But my first thought was if they aren't already using a solution... Why not just SuperPutty. In our environment we modify the two wrapper programs and we do our GPFS control via a non root system ID, say "gpfs" user with SSH enabled on that ID. So modify the remote shell command and remote file copy commands to a modified perl script, leveraging that user's ssh is and access... And we do our environments equivalent of sudo to launch that user's needed commands as privileged. In our space we needed mmremote, mmsdrrestore, scp, and echo to run as privileged commands. And everything worked fine from there once those were granted. Trial and error, we put a debug log into that script so we could figure out which commands it was actually calling. Using copilot was able to convert my Perl script to Python.. and have AI explain it back to me.. we probably haven't look at this code in 7+ years, it's been pretty reliable. Hope that helps! On Thu, Jul 10, 2025, 9:19 AM Steve Daniels wrote: > > > All, > > > Please review the idea below and vote if it is important to you. The mmdsh > command is currently based on ssh and will not have a replacement unless > development is convinced it will impact your ability to manage your > clusters effectively. > > > > Also, if you have other ideas that you want the community to consider, > please submit them for IBM’s consideration here - https://ideas.ibm.com/ > > > > Thanks, > > Steve > > Thank you for submitting your idea: ESS-I-114 mmdsh rest api command With > the transition from ssh-based commanding to the Rest API, the functionality > of mmdsh needs to be maintained. Currently, this command's functionality > will no longer be > > > > [image: IBM System Storage Ideas Portal] > Thank you for submitting your idea: ESS-I-114 mmdsh rest api command > > With the transition from ssh-based commanding to the Rest API, the > functionality of mmdsh needs to be maintained. Currently, this command's > functionality will no longer be supported once a cluster is migrated off of > ssh-based communications. Many administrators leverage the command for > day-to-day management of the cluster. It is important to have a tightly > integrated command to be able to pull information from some or all of the > nodes. > > *View idea * > > > > > > You're receiving notifications because you created this idea. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.buzzard at strath.ac.uk Mon Jul 21 19:08:10 2025 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Mon, 21 Jul 2025 19:08:10 +0100 Subject: [gpfsug-discuss] mmdsh rest api command In-Reply-To: References: <01000197f5091206-156eef6c-fcd1-44f7-92fe-6d8f692226e3-000000@email.amazonses.com> Message-ID: <73777e2e-e14d-4753-987b-11c95359bc77@strath.ac.uk> [SNIP] > > Aren't xcat, pdsh, etc, based on passwordless root ssh as well? If > so, they don't solve my clients issues. I don't see them as better > than mmdsh just different authors of the same type of tool. > Currently GPFS requires all nodes to be able to SSH onto all other nodes as root without a password. Noting at the moment the native RestAPI is an experimental feature. This root level access across the entire system in a many to many fashion has always been an security issue. This is especially true in an HPC environment were end users get to log onto nodes that are part of a GPFS cluster. If anyone gets root on any node on the system then its game over. The likes of xdsh and pdsh allow *designated* nodes to be able to SSH onto other nodes without a password in a one to many fashion. That is fundamentally different to mmdsh. Further you can configure them to need an SSH key which is secured with a passphrase for additional security. Basically in this sort of scenario with xdsh/pdsh etc. only running on highly protected nodes with limited access you have substantially enhanced your security over mmdsh and why mmdsh's continued existence is not only not required but not desirable IMHO. There is also no need for the host running xdsh/pdsh etc. to be part of the GPFS cluster. That does mean some people relying on mmdsh will have to change how they work. However continuing with bad practice when other more secure options exist is IMHO unprofessional at best and give the current cyber security environment frankly down right negligent. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From sadaniel at us.ibm.com Mon Jul 21 19:30:00 2025 From: sadaniel at us.ibm.com (Steve Daniels) Date: Mon, 21 Jul 2025 18:30:00 +0000 Subject: [gpfsug-discuss] mmdsh rest api command In-Reply-To: <73777e2e-e14d-4753-987b-11c95359bc77@strath.ac.uk> References: <01000197f5091206-156eef6c-fcd1-44f7-92fe-6d8f692226e3-000000@email.amazonses.com> <73777e2e-e14d-4753-987b-11c95359bc77@strath.ac.uk> Message-ID: Unless, there is something, i am unaware of mmdsh can also be constraind to a set of designated nodes based on ssh keys. So not sure why it is more or less than xdsh or pdsh. It seems the same. Steven A. Daniels Fax and Voice: 303-810-1229 ________________________________ From: gpfsug-discuss on behalf of Jonathan Buzzard Sent: Monday, July 21, 2025 12:08 PM To: gpfsug-discuss at gpfsug.org Subject: [EXTERNAL] Re: [gpfsug-discuss] mmdsh rest api command [SNIP] > > Aren't xcat, pdsh, etc, based on passwordless root ssh as well? If > so, they don't solve my clients issues. I don't see them as better > than mmdsh just different authors of the same type of tool. > Currently GPFS requires all nodes to be able to SSH onto all other nodes as root without a password. Noting at the moment the native RestAPI is an experimental feature. This root level access across the entire system in a many to many fashion has always been an security issue. This is especially true in an HPC environment were end users get to log onto nodes that are part of a GPFS cluster. If anyone gets root on any node on the system then its game over. The likes of xdsh and pdsh allow *designated* nodes to be able to SSH onto other nodes without a password in a one to many fashion. That is fundamentally different to mmdsh. Further you can configure them to need an SSH key which is secured with a passphrase for additional security. Basically in this sort of scenario with xdsh/pdsh etc. only running on highly protected nodes with limited access you have substantially enhanced your security over mmdsh and why mmdsh's continued existence is not only not required but not desirable IMHO. There is also no need for the host running xdsh/pdsh etc. to be part of the GPFS cluster. That does mean some people relying on mmdsh will have to change how they work. However continuing with bad practice when other more secure options exist is IMHO unprofessional at best and give the current cyber security environment frankly down right negligent. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From novosirj at rutgers.edu Mon Jul 21 19:46:51 2025 From: novosirj at rutgers.edu (Ryan Novosielski) Date: Mon, 21 Jul 2025 18:46:51 +0000 Subject: [gpfsug-discuss] mmdsh rest api command In-Reply-To: <73777e2e-e14d-4753-987b-11c95359bc77@strath.ac.uk> References: <01000197f5091206-156eef6c-fcd1-44f7-92fe-6d8f692226e3-000000@email.amazonses.com> <73777e2e-e14d-4753-987b-11c95359bc77@strath.ac.uk> Message-ID: <020A2073-E8B2-44D6-8254-AA26F40978BA@rutgers.edu> To my knowledge, this hasn’t been true for a while, and as a matter of fact, that is not the way we have our environment configured. There are nodes that do require access to all other nodes, but the same is not true in the other direction, and I believe there is some limited connectivity SSH that the nodes have between each other that is required for GPFS, controlled by what the keys are allowed to do. It does somewhat negatively interact with mmnetverify, but so far this is the only downside I’ve seen. There’s a section on it in the manual. We implemented it probably a couple of years ago now, but it has been there since sometime early in 5.x, IIRC. I guess we’ve gotten a bit off topic here though. Is there a reason to switch away from SSH itself that I’m not aware of? I certainly don’t mind more configuration options, even if I wouldn’t likely use them. Sent from my iPhone > On Jul 21, 2025, at 14:11, Jonathan Buzzard wrote: >  > [SNIP] > >> Aren't xcat, pdsh, etc, based on passwordless root ssh as well? If >> so, they don't solve my clients issues. I don't see them as better >> than mmdsh just different authors of the same type of tool. >> > Currently GPFS requires all nodes to be able to SSH onto all other nodes as root without a password. Noting at the moment the native RestAPI is an experimental feature. > > This root level access across the entire system in a many to many fashion has always been an security issue. This is especially true in an HPC environment were end users get to log onto nodes that are part of a GPFS cluster. If anyone gets root on any node on the system then its game over. > > JAB. From sadaniel at us.ibm.com Mon Jul 21 22:51:49 2025 From: sadaniel at us.ibm.com (Steve Daniels) Date: Mon, 21 Jul 2025 21:51:49 +0000 Subject: [gpfsug-discuss] mmdsh rest api command In-Reply-To: <020A2073-E8B2-44D6-8254-AA26F40978BA@rutgers.edu> References: <01000197f5091206-156eef6c-fcd1-44f7-92fe-6d8f692226e3-000000@email.amazonses.com> <73777e2e-e14d-4753-987b-11c95359bc77@strath.ac.uk> <020A2073-E8B2-44D6-8254-AA26F40978BA@rutgers.edu> Message-ID: Agree. There are three different methods (two really) of allowing internode communications for the ssh commanding. Centralized management where select nodes have one way root passwordless ssh access to all of the rest of the nodes and n-to-n where all nodes have access to all other nodes via passwordless ssh. I believe to JAB's point that the centralized is more common in 2025 and mmdsh adheres to either situation. Then we have ssh sudo wrappers which leverage sudo to provide an effective Scale manager user but underlying this is still passwordless ssh (just not the root user). Steven A. Daniels Fax and Voice: 303-810-1229 ________________________________ From: gpfsug-discuss on behalf of Ryan Novosielski Sent: Monday, July 21, 2025 12:46 PM To: gpfsug main discussion list Cc: gpfsug-discuss at gpfsug.org Subject: [EXTERNAL] Re: [gpfsug-discuss] mmdsh rest api command To my knowledge, this hasn’t been true for a while, and as a matter of fact, that is not the way we have our environment configured. There are nodes that do require access to all other nodes, but the same is not true in the other direction, and I believe there is some limited connectivity SSH that the nodes have between each other that is required for GPFS, controlled by what the keys are allowed to do. It does somewhat negatively interact with mmnetverify, but so far this is the only downside I’ve seen. There’s a section on it in the manual. We implemented it probably a couple of years ago now, but it has been there since sometime early in 5.x, IIRC. I guess we’ve gotten a bit off topic here though. Is there a reason to switch away from SSH itself that I’m not aware of? I certainly don’t mind more configuration options, even if I wouldn’t likely use them. Sent from my iPhone > On Jul 21, 2025, at 14:11, Jonathan Buzzard wrote: >  > [SNIP] > >> Aren't xcat, pdsh, etc, based on passwordless root ssh as well? If >> so, they don't solve my clients issues. I don't see them as better >> than mmdsh just different authors of the same type of tool. >> > Currently GPFS requires all nodes to be able to SSH onto all other nodes as root without a password. Noting at the moment the native RestAPI is an experimental feature. > > This root level access across the entire system in a many to many fashion has always been an security issue. This is especially true in an HPC environment were end users get to log onto nodes that are part of a GPFS cluster. If anyone gets root on any node on the system then its game over. > > JAB. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From anacreo at gmail.com Mon Jul 21 23:20:10 2025 From: anacreo at gmail.com (Alec) Date: Mon, 21 Jul 2025 15:20:10 -0700 Subject: [gpfsug-discuss] mmdsh rest api command In-Reply-To: References: <01000197f5091206-156eef6c-fcd1-44f7-92fe-6d8f692226e3-000000@email.amazonses.com> <73777e2e-e14d-4753-987b-11c95359bc77@strath.ac.uk> <020A2073-E8B2-44D6-8254-AA26F40978BA@rutgers.edu> Message-ID: Yes, it's up to you which way you create your ssh privileges but at least 1 node must be able to push to other nodes via SSH (or lesser secure protocol, o.O) to get GPFS working as far as I know from 4.x experience, maybe things have changed with 5.x. Once we got away from root ssh we were able to pass muster with security... Least of our problems from compliance perspective and that's saying a lot in our environment. Web interface via REST is more modern, but would actually give us more issues with currency and known issues, certificate management, etc. Less is more. Only wish with GPFS is that management understood how much money you save, and performance/efficiency you get, by right sizing the IO to CPU, and seems to me all these years later GPFS is the only real solution to get disk I/O to match the Computer throughout. Oh and maybe IBM would give up and change the name back to GPFS. Thanks to all the work in the community, and IBM for this amazing product. On Mon, Jul 21, 2025, 2:54 PM Steve Daniels wrote: > Agree. > > There are three different methods (two really) of allowing internode > communications for the ssh commanding. > > Centralized management where select nodes have one way root passwordless > ssh access to all of the rest of the nodes and n-to-n where all nodes have > access to all other nodes via passwordless ssh. > > I believe to JAB's point that the centralized is more common in 2025 and > mmdsh adheres to either situation. > > Then we have ssh sudo wrappers which leverage sudo to provide an effective > Scale manager user but underlying this is still passwordless ssh (just not > the root user). > > Steven A. Daniels > > Fax and Voice: 303-810-1229 > > > ------------------------------ > *From:* gpfsug-discuss on behalf of > Ryan Novosielski > *Sent:* Monday, July 21, 2025 12:46 PM > *To:* gpfsug main discussion list > *Cc:* gpfsug-discuss at gpfsug.org > *Subject:* [EXTERNAL] Re: [gpfsug-discuss] mmdsh rest api command > > To my knowledge, this hasn’t been true for a while, and as a matter of > fact, that is not the way we have our environment configured. > > There are nodes that do require access to all other nodes, but the same is > not true in the other direction, and I believe there is some limited > connectivity SSH that the nodes have between each other that is required > for GPFS, controlled by what the keys are allowed to do. > > It does somewhat negatively interact with mmnetverify, but so far this is > the only downside I’ve seen. > > There’s a section on it in the manual. We implemented it probably a couple > of years ago now, but it has been there since sometime early in 5.x, IIRC. > > I guess we’ve gotten a bit off topic here though. Is there a reason to > switch away from SSH itself that I’m not aware of? I certainly don’t mind > more configuration options, even if I wouldn’t likely use them. > > Sent from my iPhone > > > On Jul 21, 2025, at 14:11, Jonathan Buzzard < > jonathan.buzzard at strath.ac.uk> wrote: > >  > > [SNIP] > > > >> Aren't xcat, pdsh, etc, based on passwordless root ssh as well? If > >> so, they don't solve my clients issues. I don't see them as better > >> than mmdsh just different authors of the same type of tool. > >> > > Currently GPFS requires all nodes to be able to SSH onto all other nodes > as root without a password. Noting at the moment the native RestAPI is an > experimental feature. > > > > This root level access across the entire system in a many to many > fashion has always been an security issue. This is especially true in an > HPC environment were end users get to log onto nodes that are part of a > GPFS cluster. If anyone gets root on any node on the system then its game > over. > > > > JAB. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss-5Fgpfsug.org&d=DwIGaQ&c=BSDicqBQBDjDI9RkVyTcHQ&r=poV0PwVYTQCODtr5Roh1IeohBrObo4EP_Tx9IkCIbHo&m=qb84pFD2OGyNw2_770L1Ddg0HkNFST8YS0o-H3kVc_O8OJW_cMlSuVhfoC1iDNUp&s=XNqx3vVFU6sb7lud9KgKja-VTd6BQuapYlV8R-MJ6Zw&e= > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.buzzard at strath.ac.uk Wed Jul 23 18:18:07 2025 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Wed, 23 Jul 2025 18:18:07 +0100 Subject: [gpfsug-discuss] mmdsh rest api command In-Reply-To: References: <01000197f5091206-156eef6c-fcd1-44f7-92fe-6d8f692226e3-000000@email.amazonses.com> <73777e2e-e14d-4753-987b-11c95359bc77@strath.ac.uk> <020A2073-E8B2-44D6-8254-AA26F40978BA@rutgers.edu> Message-ID: <28812160-0bdb-4099-83cb-b2688aa1e39b@strath.ac.uk> On 21/07/2025 22:51, Steve Daniels wrote: > > There are three different methods (two really) of allowing internode > communications for the ssh commanding. > > Centralized management where select nodes have one way root passwordless > ssh access to all of the rest of the nodes and n-to-n where all nodes > have access to all other nodes via passwordless ssh. > When the central administration mode was first introduced you still needed n-to-n ssh access or it all still fell apart despite being only able to issue "administration" commands from the central nodes. From recollection a slew of what I would call "user" commands (such as changing an ACL on your *own* files for example) all stopped working unless the n-to-n was maintained. I am not precluding that this had changed in the meantime, but once bitten twice shy as they say. I still maintain reinventing the wheel, which will be a whole bunch of infrequently tested code paths is a really bad idea in the modern security threat environment. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From jonathan.buzzard at strath.ac.uk Wed Jul 23 18:19:00 2025 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Wed, 23 Jul 2025 18:19:00 +0100 Subject: [gpfsug-discuss] RHEL 10 Message-ID: <391ebb18-b67a-4c96-b678-a0ba990c59de@strath.ac.uk> Obviously with all the usual caveats etc. is there a timeline for RHEL10 to be supported? JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From jonathan.buzzard at strath.ac.uk Wed Jul 30 16:41:48 2025 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Wed, 30 Jul 2025 16:41:48 +0100 Subject: [gpfsug-discuss] DSS-G100 and networking Message-ID: <61e86d78-7200-4aab-ab3e-a0a286efd144@strath.ac.uk> So these servers (at least in their SR630 V2 variant) come with three single port ConnectX-6 Dx cards capable of 200Gbps. The question is why do I need three? My thinking is that a LACP bond into a couple of switches doing MLAG (or insert whatever your vendor calls it) is ample connectivity? Noting that these are going into a pair of SN3700M switches at the full 200Gbps. So I don't get the purpose of the third card? What am I missing? Extra connectivity if you want to do Infiniband as well? Direct link between two of them? JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From ott.oopkaup at ut.ee Wed Jul 30 16:54:07 2025 From: ott.oopkaup at ut.ee (Ott Oopkaup) Date: Wed, 30 Jul 2025 18:54:07 +0300 Subject: [gpfsug-discuss] DSS-G100 and networking In-Reply-To: <61e86d78-7200-4aab-ab3e-a0a286efd144@strath.ac.uk> References: <61e86d78-7200-4aab-ab3e-a0a286efd144@strath.ac.uk> Message-ID: Isn't this configurable during time of purchase? Our DSS-G251 and 260 run 2x 2slot VPI cards configured to 2x ethernet in LACP bond and 2x IB. Basically configured for redundancy (we can lose a whole card), although we have maxed throughput over single port speeds. Other thing that comes to mind is that single slot cards can effectively use 100% of the PCI-E link while 2 port cards might need to compromise. This is now again a configuration question and dependent on the ratio of ethernet and IB traffic. Best, Ott Oopkaup University of Tartu, High Performance Computing Centre Systems Administrator On 7/30/25 6:41 PM, Jonathan Buzzard wrote: > > So these servers (at least in their SR630 V2 variant) come with three > single port ConnectX-6 Dx cards capable of 200Gbps. > > The question is why do I need three? My thinking is that a LACP bond > into a couple of switches doing MLAG (or insert whatever your vendor > calls it) is ample connectivity? Noting that these are going into a > pair of SN3700M switches at the full 200Gbps. > > So I don't get the purpose of the third card? What am I missing? Extra > connectivity if you want to do Infiniband as well? Direct link between > two of them? > > > JAB. > From novosirj at rutgers.edu Wed Jul 30 22:23:20 2025 From: novosirj at rutgers.edu (Ryan Novosielski) Date: Wed, 30 Jul 2025 21:23:20 +0000 Subject: [gpfsug-discuss] DSS-G100 and networking In-Reply-To: <61e86d78-7200-4aab-ab3e-a0a286efd144@strath.ac.uk> References: <61e86d78-7200-4aab-ab3e-a0a286efd144@strath.ac.uk> Message-ID: <8A6F031A-4784-418F-8216-7E9698F8EF25@rutgers.edu> We have never bought any smaller than a DSS-G220, and they’ve always had the 2U servers (so SR650, in our gen2 systems), so I imagine it’s a little different, but we have 2x 2 port cards, all in use for Infiniband, and then often another Connect-X card for high-speed ethernet (which we would bond for bandwidth/redundancy), so it would stand to reason that you’d have some form of that as an option on the G100. It’s configurable, within reason (eg. I’d imagine you can’t take out the minimum required, etc.). -- #BlackLivesMatter ____ || \\UTGERS, |---------------------------*O*--------------------------- ||_// the State | Ryan Novosielski - novosirj at rutgers.edu || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus || \\ of NJ | Office of Advanced Research Computing - MSB A555B, Newark `' On Jul 30, 2025, at 11:41, Jonathan Buzzard wrote: So these servers (at least in their SR630 V2 variant) come with three single port ConnectX-6 Dx cards capable of 200Gbps. The question is why do I need three? My thinking is that a LACP bond into a couple of switches doing MLAG (or insert whatever your vendor calls it) is ample connectivity? Noting that these are going into a pair of SN3700M switches at the full 200Gbps. So I don't get the purpose of the third card? What am I missing? Extra connectivity if you want to do Infiniband as well? Direct link between two of them? JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From taylorm at us.ibm.com Thu Jul 31 00:14:15 2025 From: taylorm at us.ibm.com (Michael Taylor) Date: Wed, 30 Jul 2025 23:14:15 +0000 Subject: [gpfsug-discuss] gpfsug-discuss RHEL 10 In-Reply-To: References: Message-ID: Storage Scale is evaluating RHEL10 and aiming to support the new major RHEL10 OS in the early 4Q release of 2025. Please note that this is a tentative support timeline and subject to change at our discretion, and can be adjusted as needed but it's actively being worked. ________________________________ From: gpfsug-discuss on behalf of gpfsug-discuss-request at gpfsug.org Sent: Thursday, July 24, 2025 4:00 AM To: gpfsug-discuss at gpfsug.org Subject: [EXTERNAL] gpfsug-discuss Digest, Vol 156, Issue 6 Send gpfsug-discuss mailing list submissions to gpfsug-discuss at gpfsug.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at gpfsug.org You can reach the person managing the list at gpfsug-discuss-owner at gpfsug.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Re: mmdsh rest api command (Jonathan Buzzard) 2. RHEL 10 (Jonathan Buzzard) ---------------------------------------------------------------------- Message: 1 Date: Wed, 23 Jul 2025 18:18:07 +0100 From: Jonathan Buzzard To: gpfsug-discuss at gpfsug.org Subject: Re: [gpfsug-discuss] mmdsh rest api command Message-ID: <28812160-0bdb-4099-83cb-b2688aa1e39b at strath.ac.uk> Content-Type: text/plain; charset=UTF-8; format=flowed On 21/07/2025 22:51, Steve Daniels wrote: > > There are three different methods (two really) of allowing internode > communications for the ssh commanding. > > Centralized management where select nodes have one way root passwordless > ssh access to all of the rest of the nodes and n-to-n where all nodes > have access to all other nodes via passwordless ssh. > When the central administration mode was first introduced you still needed n-to-n ssh access or it all still fell apart despite being only able to issue "administration" commands from the central nodes. From recollection a slew of what I would call "user" commands (such as changing an ACL on your *own* files for example) all stopped working unless the n-to-n was maintained. I am not precluding that this had changed in the meantime, but once bitten twice shy as they say. I still maintain reinventing the wheel, which will be a whole bunch of infrequently tested code paths is a really bad idea in the modern security threat environment. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG ------------------------------ Message: 2 Date: Wed, 23 Jul 2025 18:19:00 +0100 From: Jonathan Buzzard To: gpfsug main discussion list Subject: [gpfsug-discuss] RHEL 10 Message-ID: <391ebb18-b67a-4c96-b678-a0ba990c59de at strath.ac.uk> Content-Type: text/plain; charset=UTF-8; format=flowed Obviously with all the usual caveats etc. is there a timeline for RHEL10 to be supported? JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG ------------------------------ Subject: Digest Footer _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org ------------------------------ End of gpfsug-discuss Digest, Vol 156, Issue 6 ********************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: