From p.childs at qmul.ac.uk Thu Jan 15 18:21:18 2026 From: p.childs at qmul.ac.uk (Peter Childs) Date: Thu, 15 Jan 2026 18:21:18 +0000 Subject: [gpfsug-discuss] mmbackup valiating the backups Message-ID: Hi All, We use mmbackup to backup our Scale Storage to two IBM Protect Instances. We would like to validate our backups to ensure that the files we have really are backed up and we don't have any problems. So far I've worked out I can query the backups using "query backup --detail" in Protect and get times and dates, From this I can compare with `stat` and `ls` (or using mmapplypolicy (which should be faster)) to check the three indexes match, and check the contents of each directory. (Using --subdir=yes looks great on paper but the output takes days to appear and checking it file by file can be done incrementally and I can use mmapplypolicy to run the report. Given the stats from Protect the two backup servers are reporting different occupancy figures which suggests to me we may have some inconsistencies. (We're talking a 1/2 TB difference between the two servers according to Protect Query Occupancy) but I'm aware that Protect's figures are estimates and not always accurate. Verify the backup is always a good plan even if your 101% sure its correct anyway (and I'm not maybe). (Your backups are only as good as the last time you recalled them and all that) We could use the shadow database. As this looks to be what Storage Archive does to say yes this file is backed up before it is archived. Does anyone know the format of the shadow database, which fields are which, as I think knowing the format might allow us to at least know what the differences; and increase our confidence in the backup. Thanks in advance. Peter Childs From timm.stamer at uni-oldenburg.de Fri Jan 16 06:53:11 2026 From: timm.stamer at uni-oldenburg.de (Timm Stamer) Date: Fri, 16 Jan 2026 06:53:11 +0000 Subject: [gpfsug-discuss] mmbackup valiating the backups In-Reply-To: References: Message-ID: Hi , we're running mmbackup with -q option once a quarter to be sure everything is backed up. I think this is much simpler than your approach but maybe does not cover all your needs. -q Performs a query operation before issuing mmbackup. The IBM Storage Protect server might have data stored already that is not recognized as having been backed up by mmbackup and its own shadow database. To properly compute the set of files that currently need to be backed up, mmbackup can perform an IBM Storage Protect query and process the results to update its shadow database. Use the -q switch to perform this query and then immediately commence the requested backup operation. https://www.ibm.com/docs/en/storage-scale/5.2.3?topic=reference-mmbackup-command [...] we=$(LC_TIME=C date +%A) dm=$(date +%d) dmonth=$(date +%m) checkMonths=("01" "04" "07" "10") [[ ${checkMonths[@]} =~ $dmonth ]] && [ "$we" = "Friday" ] && [ "$dm" -le 7 ] && QUERYBACKUP="-q" mmbackup ... ${QUERYBACKUP} ... [...] Kind regards Timm Stamer Am Donnerstag, dem 15.01.2026 um 18:21 +0000 schrieb Peter Childs: > > Hi All, > > We use mmbackup to backup our Scale Storage to two IBM Protect > Instances. > > We would like to validate our backups to ensure that the files we > have really are backed up and we don't have any problems. > > So far I've worked out I can query the backups using "query backup -- > detail" in Protect and get times and dates, From this I can compare > with `stat` and `ls` (or using mmapplypolicy (which should be > faster)) to check the three indexes match, and check the contents of > each directory. (Using --subdir=yes looks great on paper but the > output takes days to appear and checking it file by file can be done > incrementally and I can use mmapplypolicy to run the report. > > Given the stats from Protect the two backup servers are reporting > different occupancy figures which suggests to me we may have some > inconsistencies. (We're talking a 1/2 TB difference between the two > servers according to Protect Query Occupancy) but I'm aware that > Protect's figures are estimates and not always accurate. > > Verify the backup is always a good plan even if your 101% sure its > correct anyway (and I'm not maybe). (Your backups are only as good as > the last time you recalled them and all that) > > We could use the shadow database. As this looks to be what Storage > Archive does to say yes this file is backed up before it is archived. > > Does anyone know the format of the shadow database, which fields are > which, as I think knowing the format might allow us to at least know > what the differences; and increase our confidence in the backup. > > Thanks in advance. > > Peter Childs > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org From st.graf at fz-juelich.de Fri Jan 16 09:29:40 2026 From: st.graf at fz-juelich.de (Stephan Graf) Date: Fri, 16 Jan 2026 10:29:40 +0100 Subject: [gpfsug-discuss] mmbackup valiating the backups In-Reply-To: References: Message-ID: <71b47392-3544-464a-80a4-6623de559f13@fz-juelich.de> Hi, I also recommend to use the '-q' option in a regularly way to be sure not to get some kind of 'split brain' situation: The TSM DB content and the shadow database. If You run mmbackup on a large file system where a lot of changes are ongoing, we always see in the final report of mmbackup that some files failed. As far as I remember the situation was, that the file was deleted by the user before the dsmc was started. But it is hard to check it for all files so we ignore those failures. But it could be that something different is the root cause of not backing up this file. In the past we saw it when users used strange characters in their file name. For this it would be nice to get some kind of better report from mmbackup. So for example a filelist of all files which failed. May be with the reason - file not found - dsmc backup failed - ... This is a question to the IBM Devs: What do You think? Is it worth of opening an IBM IDEA? Stephan On 1/16/26 07:53, Timm Stamer wrote: > Hi , > > we're running mmbackup with -q option once a quarter to be sure > everything is backed up. > I think this is much simpler than your approach but maybe does not > cover all your needs. > > > -q > Performs a query operation before issuing mmbackup. The IBM Storage > Protect server might have data stored already that is not recognized as > having been backed up by mmbackup and its own shadow database. To > properly compute the set of files that currently need to be backed up, > mmbackup can perform an IBM Storage Protect query and process the > results to update its shadow database. Use the -q switch to perform > this query and then immediately commence the requested backup > operation. > > https://www.ibm.com/docs/en/storage-scale/5.2.3?topic=reference-mmbackup-command > > > [...] > we=$(LC_TIME=C date +%A) > dm=$(date +%d) > dmonth=$(date +%m) > checkMonths=("01" "04" "07" "10") > [[ ${checkMonths[@]} =~ $dmonth ]] && [ "$we" = "Friday" ] && [ "$dm" -le 7 ] && QUERYBACKUP="-q" > > mmbackup ... ${QUERYBACKUP} ... > [...] > > > Kind regards > > Timm Stamer > > > > > Am Donnerstag, dem 15.01.2026 um 18:21 +0000 schrieb Peter Childs: >> >> Hi All, >> >> We use mmbackup to backup our Scale Storage to two IBM Protect >> Instances. >> >> We would like to validate our backups to ensure that the files we >> have really are backed up and we don't have any problems. >> >> So far I've worked out I can query the backups using "query backup -- >> detail" in Protect and get times and dates, From this I can compare >> with `stat` and `ls` (or using mmapplypolicy (which should be >> faster)) to check the three indexes match, and check the contents of >> each directory. (Using --subdir=yes looks great on paper but the >> output takes days to appear and checking it file by file can be done >> incrementally and I can use mmapplypolicy to run the report. >> >> Given the stats from Protect the two backup servers are reporting >> different occupancy figures which suggests to me we may have some >> inconsistencies. (We're talking a 1/2 TB difference between the two >> servers according to Protect Query Occupancy) but I'm aware that >> Protect's figures are estimates and not always accurate. >> >> Verify the backup is always a good plan even if your 101% sure its >> correct anyway (and I'm not maybe). (Your backups are only as good as >> the last time you recalled them and all that) >> >> We could use the shadow database. As this looks to be what Storage >> Archive does to say yes this file is backed up before it is archived. >> >> Does anyone know the format of the shadow database, which fields are >> which, as I think knowing the format might allow us to at least know >> what the differences; and increase our confidence in the backup. >> >> Thanks in advance. >> >> Peter Childs >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at gpfsug.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org -- Stephan Graf #GernePerDu HPC, Cloud and Data Systems and Services Jülich Supercomputing Centre Phone: +49-2461-61-6578 Fax: +49-2461-61-6656 E-mail: st.graf at fz-juelich.de WWW: http://www.fz-juelich.de/jsc/ --------------------------------------------------------------------------------------------- --------------------------------------------------------------------------------------------- Forschungszentrum Jülich GmbH 52425 Jülich Sitz der Gesellschaft: Jülich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Stefan Müller Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), Dr. Stephanie Bauer (stellv. Vorsitzende) Prof. Dr. Ir. Pieter Jansens, Prof. Dr. Laurens Kuipers --------------------------------------------------------------------------------------------- --------------------------------------------------------------------------------------------- -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 6199 bytes Desc: S/MIME Cryptographic Signature URL: From ewahl at osc.edu Fri Jan 16 14:31:11 2026 From: ewahl at osc.edu (Wahl, Edward) Date: Fri, 16 Jan 2026 14:31:11 +0000 Subject: [gpfsug-discuss] mmbackup valiating the backups In-Reply-To: <71b47392-3544-464a-80a4-6623de559f13@fz-juelich.de> References: <71b47392-3544-464a-80a4-6623de559f13@fz-juelich.de> Message-ID: I've been hesitant to reply to this thread as I am not a TSM/Storage Protect expert even after dealing with it for more than a decade and a half, but we've been backing up some moderately sized GPFS file systems (8-16P) with it for many years now so I do have some experience with it. We've had all the problems you can have from a corrupted DB2, to issues with a zillion tiny files taking more than 24 hours, to the once a week or so issues with users uploading/creating file with the dreaded "newline" or control characters in the names. >If You run mmbackup on a large file system where a lot of changes are ongoing, we always see in the final report of mmbackup that some files failed. As this thread started out asking about "validating" the backups let's start here and address a couple different issues from different folks. Are you not using snapshots to do the mmbackup? Other than 'bad characters' which GPFS allows and SP does not(GPFS is MUCH more permissive than TSM about that even with things like WILDCARDSARELITERAL/QUOTESARELITERAL), there shouldn't really be any issues. I HIGHLY recommend using a snapshot at the minimum for backups. This will bypass any 'file not found' issues at least. Though this DOES introduce the whole 'issues with the snaps' you might see if compute nodes are too overloaded and will not pause for the snaps to be created/deleted. Snapshots will also allow the validation testing to have a perfect copy of the backed up data to validate with. Backups and snapshots are both 'point in time' on a live/production file system, so validation is difficult without being at that point in time. Just keep the snapshot around as long as you can stand, for the tests. (I realize not everyone can keep snapshots for a long time due to inode/block issues) Maybe a scripted 'xxx#sum' for everything backed up? Up to you how you do this. For the comment about sending the data to multiple initial SP servers, that sounds like a major shadow file problem waiting to happen. Which server was it built with last? Which is the source of truth?, etc. Either send to a single SP server and have it replicate (disaster recovery is your friend), and/or divide up the file system by filesets and send them to different servers. Using snapshots with multiple SP servers and gpfs filesets allows for much better parallelization of the backup itself. As we already use independent filesets by 'project code' for quotas, this is quite helpful for us. Apologies if I used GPFS or TSM in here and folks were confused. Just replace those with Storage Scale and Storage Protect. Ed Wahl Ohio Supercomputer Center ________________________________ From: gpfsug-discuss on behalf of Stephan Graf Sent: Friday, January 16, 2026 4:29 AM To: gpfsug-discuss at gpfsug.org Subject: Re: [gpfsug-discuss] mmbackup valiating the backups Hi, I also recommend to use the '-q' option in a regularly way to be sure not to get some kind of 'split brain' situation: The TSM DB content and the shadow database. If You run mmbackup on a large file system where a lot of changes are ongoing, we always see in the final report of mmbackup that some files failed. As far as I remember the situation was, that the file was deleted by the user before the dsmc was started. But it is hard to check it for all files so we ignore those failures. But it could be that something different is the root cause of not backing up this file. In the past we saw it when users used strange characters in their file name. For this it would be nice to get some kind of better report from mmbackup. So for example a filelist of all files which failed. May be with the reason - file not found - dsmc backup failed - ... This is a question to the IBM Devs: What do You think? Is it worth of opening an IBM IDEA? Stephan On 1/16/26 07:53, Timm Stamer wrote: > Hi , > > we're running mmbackup with -q option once a quarter to be sure > everything is backed up. > I think this is much simpler than your approach but maybe does not > cover all your needs. > > > -q > Performs a query operation before issuing mmbackup. The IBM Storage > Protect server might have data stored already that is not recognized as > having been backed up by mmbackup and its own shadow database. To > properly compute the set of files that currently need to be backed up, > mmbackup can perform an IBM Storage Protect query and process the > results to update its shadow database. Use the -q switch to perform > this query and then immediately commence the requested backup > operation. > > https://urldefense.com/v3/__https://www.ibm.com/docs/en/storage-scale/5.2.3?topic=reference-mmbackup-command__;!!KGKeukY!2Gc-96JLXSQApClIceidsjnn3c3yGp2w_RSrcXcylKv3vSYzH4zqmFX0o222MdyEb07sy4QfsZ9bKkKyvQNr$ > > > [...] > we=$(LC_TIME=C date +%A) > dm=$(date +%d) > dmonth=$(date +%m) > checkMonths=("01" "04" "07" "10") > [[ ${checkMonths[@]} =~ $dmonth ]] && [ "$we" = "Friday" ] && [ "$dm" -le 7 ] && QUERYBACKUP="-q" > > mmbackup ... ${QUERYBACKUP} ... > [...] > > > Kind regards > > Timm Stamer > > > > > Am Donnerstag, dem 15.01.2026 um 18:21 +0000 schrieb Peter Childs: >> >> Hi All, >> >> We use mmbackup to backup our Scale Storage to two IBM Protect >> Instances. >> >> We would like to validate our backups to ensure that the files we >> have really are backed up and we don't have any problems. >> >> So far I've worked out I can query the backups using "query backup -- >> detail" in Protect and get times and dates, From this I can compare >> with `stat` and `ls` (or using mmapplypolicy (which should be >> faster)) to check the three indexes match, and check the contents of >> each directory. (Using --subdir=yes looks great on paper but the >> output takes days to appear and checking it file by file can be done >> incrementally and I can use mmapplypolicy to run the report. >> >> Given the stats from Protect the two backup servers are reporting >> different occupancy figures which suggests to me we may have some >> inconsistencies. (We're talking a 1/2 TB difference between the two >> servers according to Protect Query Occupancy) but I'm aware that >> Protect's figures are estimates and not always accurate. >> >> Verify the backup is always a good plan even if your 101% sure its >> correct anyway (and I'm not maybe). (Your backups are only as good as >> the last time you recalled them and all that) >> >> We could use the shadow database. As this looks to be what Storage >> Archive does to say yes this file is backed up before it is archived. >> >> Does anyone know the format of the shadow database, which fields are >> which, as I think knowing the format might allow us to at least know >> what the differences; and increase our confidence in the backup. >> >> Thanks in advance. >> >> Peter Childs >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at gpfsug.org >> https://urldefense.com/v3/__http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org__;!!KGKeukY!2Gc-96JLXSQApClIceidsjnn3c3yGp2w_RSrcXcylKv3vSYzH4zqmFX0o222MdyEb07sy4QfsZ9bKh7-Yo7f$ > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > https://urldefense.com/v3/__http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org__;!!KGKeukY!2Gc-96JLXSQApClIceidsjnn3c3yGp2w_RSrcXcylKv3vSYzH4zqmFX0o222MdyEb07sy4QfsZ9bKh7-Yo7f$ -- Stephan Graf #GernePerDu HPC, Cloud and Data Systems and Services Jülich Supercomputing Centre Phone: +49-2461-61-6578 Fax: +49-2461-61-6656 E-mail: st.graf at fz-juelich.de WWW: https://urldefense.com/v3/__http://www.fz-juelich.de/jsc/__;!!KGKeukY!2Gc-96JLXSQApClIceidsjnn3c3yGp2w_RSrcXcylKv3vSYzH4zqmFX0o222MdyEb07sy4QfsZ9bKkMvcWKi$ --------------------------------------------------------------------------------------------- --------------------------------------------------------------------------------------------- Forschungszentrum Jülich GmbH 52425 Jülich Sitz der Gesellschaft: Jülich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Stefan Müller Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), Dr. Stephanie Bauer (stellv. Vorsitzende) Prof. Dr. Ir. Pieter Jansens, Prof. Dr. Laurens Kuipers --------------------------------------------------------------------------------------------- --------------------------------------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From kraemerf at de.ibm.com Fri Jan 16 20:55:27 2026 From: kraemerf at de.ibm.com (Frank Kramer) Date: Fri, 16 Jan 2026 20:55:27 +0000 Subject: [gpfsug-discuss] IBM German Storage Scale Days 2026: Registration Now Open Message-ID: IBM German Storage Scale Days 2026: Registration Now Open We are pleased to announce that IBM Storage Scale Days 2026 (SSD26) will take place on March 10–11, 2026, in Ehningen, Germany. Registration is now open, and we invite all Storage Scale professionals and enthusiasts to secure their spot. SSD26 is part of the broader IBM Storage Strategy Days 2026 conference, themed “The Future of Data Storage.” This integrated event brings together experts, customers, and IBM specialists to explore the latest innovations, strategies, and real-world use cases in modern data storage. Attendance is free of charge. IBM Storage Scale Days | March 10–11, 2026 https://www.ibm.com/events/reg/flow/ibm/8961lwmb/landing/page/landing In addition, we are excited to offer a dedicated Scale New User Day on March 9, 2026. This pre-conference event is designed specifically for newcomers, providing a practical introduction to IBM Storage Scale and helping new users get up to speed quickly. Scale New User Day | March 9, 2026 https://www.ibm.com/events/reg/flow/ibm/qguohomb/landing/page/landing We look forward to welcoming you in Ehningen for an engaging and insightful week focused on the future of data storage. If you have any questions, feel free to reach out. Best regards, IBM Storage Scale Days 2026 Team Frank Kraemer IBM Senior Technical Specialist Tower One, Bruesseler Street 1-3, 60327 Frankfurt mailto:kraemerf at de.ibm.com Mobile +491713043699 IBM Germany https://www.ibm.com/privacy/us/en/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From skylar2 at uw.edu Wed Jan 21 22:12:33 2026 From: skylar2 at uw.edu (Skylar Thompson) Date: Wed, 21 Jan 2026 14:12:33 -0800 Subject: [gpfsug-discuss] mmbackup valiating the backups In-Reply-To: References: <71b47392-3544-464a-80a4-6623de559f13@fz-juelich.de> Message-ID: I'm also not sure that I'm a TSM or a GPFS expert, though have been using both for a couple decades now. My experiences mirror Ed's, though: we have ~10PB (~900 million files) in GPFS that we backup with mmbackup. We have three large storage systems that we backup as single TSM nodes, and then a "condo" system that people can buy storage on, allocated as independent filesets. We backup each fileset as a separate node with mmbackup, and currently have 90 of them, each with a daily backup schedule. We create snapshots for each backup, and just have to stagger the schedule start times so we don't get lock contention in the admin layer. The only errors we get are from tape library/drive problems, or very occasionally the storage is just too busy to make the snapshot. We only run "mmbackup -q" if there was a storage outage that kept the shadow database from being updated, or if we've made include/exclude rule changes that require it. On Fri, Jan 16, 2026 at 02:31:11PM +0000, Wahl, Edward wrote: > I've been hesitant to reply to this thread as I am not a TSM/Storage Protect expert even after dealing with it for more than a decade and a half, but we've been backing up some moderately sized GPFS file systems (8-16P) with it for many years now so I do have some experience with it. We've had all the problems you can have from a corrupted DB2, to issues with a zillion tiny files taking more than 24 hours, to the once a week or so issues with users uploading/creating file with the dreaded "newline" or control characters in the names. > > > >If You run mmbackup on a large file system where a lot of changes are ongoing, we always see in the final report of mmbackup that some files failed. > > As this thread started out asking about "validating" the backups let's start here and address a couple different issues from different folks. > Are you not using snapshots to do the mmbackup? Other than 'bad characters' which GPFS allows and SP does not(GPFS is MUCH more permissive than TSM about that even with things like WILDCARDSARELITERAL/QUOTESARELITERAL), there shouldn't really be any issues. I HIGHLY recommend using a snapshot at the minimum for backups. This will bypass any 'file not found' issues at least. Though this DOES introduce the whole 'issues with the snaps' you might see if compute nodes are too overloaded and will not pause for the snaps to be created/deleted. > > Snapshots will also allow the validation testing to have a perfect copy of the backed up data to validate with. Backups and snapshots are both 'point in time' on a live/production file system, so validation is difficult without being at that point in time. Just keep the snapshot around as long as you can stand, for the tests. (I realize not everyone can keep snapshots for a long time due to inode/block issues) Maybe a scripted 'xxx#sum' for everything backed up? Up to you how you do this. > > > For the comment about sending the data to multiple initial SP servers, that sounds like a major shadow file problem waiting to happen. Which server was it built with last? Which is the source of truth?, etc. Either send to a single SP server and have it replicate (disaster recovery is your friend), and/or divide up the file system by filesets and send them to different servers. Using snapshots with multiple SP servers and gpfs filesets allows for much better parallelization of the backup itself. As we already use independent filesets by 'project code' for quotas, this is quite helpful for us. > > > Apologies if I used GPFS or TSM in here and folks were confused. Just replace those with Storage Scale and Storage Protect. > > Ed Wahl > Ohio Supercomputer Center > ________________________________ > From: gpfsug-discuss on behalf of Stephan Graf > Sent: Friday, January 16, 2026 4:29 AM > To: gpfsug-discuss at gpfsug.org > Subject: Re: [gpfsug-discuss] mmbackup valiating the backups > > Hi, > > I also recommend to use the '-q' option in a regularly way to be sure > not to get some kind of 'split brain' situation: The TSM DB content and > the shadow database. > > If You run mmbackup on a large file system where a lot of changes are > ongoing, we always see in the final report of mmbackup that some files > failed. > As far as I remember the situation was, that the file was deleted by the > user before the dsmc was started. > But it is hard to check it for all files so we ignore those failures. > But it could be that something different is the root cause of not > backing up this file. In the past we saw it when users used strange > characters in their file name. > > For this it would be nice to get some kind of better report from mmbackup. > So for example a filelist of all files which failed. May be with the reason > - file not found > - dsmc backup failed > - ... > > This is a question to the IBM Devs: What do You think? Is it worth of > opening an IBM IDEA? > > Stephan > > > On 1/16/26 07:53, Timm Stamer wrote: > > Hi , > > > > we're running mmbackup with -q option once a quarter to be sure > > everything is backed up. > > I think this is much simpler than your approach but maybe does not > > cover all your needs. > > > > > > -q > > Performs a query operation before issuing mmbackup. The IBM Storage > > Protect server might have data stored already that is not recognized as > > having been backed up by mmbackup and its own shadow database. To > > properly compute the set of files that currently need to be backed up, > > mmbackup can perform an IBM Storage Protect query and process the > > results to update its shadow database. Use the -q switch to perform > > this query and then immediately commence the requested backup > > operation. > > > > https://urldefense.com/v3/__https://www.ibm.com/docs/en/storage-scale/5.2.3?topic=reference-mmbackup-command__;!!KGKeukY!2Gc-96JLXSQApClIceidsjnn3c3yGp2w_RSrcXcylKv3vSYzH4zqmFX0o222MdyEb07sy4QfsZ9bKkKyvQNr$ > > > > > > [...] > > we=$(LC_TIME=C date +%A) > > dm=$(date +%d) > > dmonth=$(date +%m) > > checkMonths=("01" "04" "07" "10") > > [[ ${checkMonths[@]} =~ $dmonth ]] && [ "$we" = "Friday" ] && [ "$dm" -le 7 ] && QUERYBACKUP="-q" > > > > mmbackup ... ${QUERYBACKUP} ... > > [...] > > > > > > Kind regards > > > > Timm Stamer > > > > > > > > > > Am Donnerstag, dem 15.01.2026 um 18:21 +0000 schrieb Peter Childs: > >> > >> Hi All, > >> > >> We use mmbackup to backup our Scale Storage to two IBM Protect > >> Instances. > >> > >> We would like to validate our backups to ensure that the files we > >> have really are backed up and we don't have any problems. > >> > >> So far I've worked out I can query the backups using "query backup -- > >> detail" in Protect and get times and dates, From this I can compare > >> with `stat` and `ls` (or using mmapplypolicy (which should be > >> faster)) to check the three indexes match, and check the contents of > >> each directory. (Using --subdir=yes looks great on paper but the > >> output takes days to appear and checking it file by file can be done > >> incrementally and I can use mmapplypolicy to run the report. > >> > >> Given the stats from Protect the two backup servers are reporting > >> different occupancy figures which suggests to me we may have some > >> inconsistencies. (We're talking a 1/2 TB difference between the two > >> servers according to Protect Query Occupancy) but I'm aware that > >> Protect's figures are estimates and not always accurate. > >> > >> Verify the backup is always a good plan even if your 101% sure its > >> correct anyway (and I'm not maybe). (Your backups are only as good as > >> the last time you recalled them and all that) > >> > >> We could use the shadow database. As this looks to be what Storage > >> Archive does to say yes this file is backed up before it is archived. > >> > >> Does anyone know the format of the shadow database, which fields are > >> which, as I think knowing the format might allow us to at least know > >> what the differences; and increase our confidence in the backup. > >> > >> Thanks in advance. > >> > >> Peter Childs > >> _______________________________________________ > >> gpfsug-discuss mailing list > >> gpfsug-discuss at gpfsug.org > >> https://urldefense.com/v3/__http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org__;!!KGKeukY!2Gc-96JLXSQApClIceidsjnn3c3yGp2w_RSrcXcylKv3vSYzH4zqmFX0o222MdyEb07sy4QfsZ9bKh7-Yo7f$ > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at gpfsug.org > > https://urldefense.com/v3/__http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org__;!!KGKeukY!2Gc-96JLXSQApClIceidsjnn3c3yGp2w_RSrcXcylKv3vSYzH4zqmFX0o222MdyEb07sy4QfsZ9bKh7-Yo7f$ > > -- > Stephan Graf > #GernePerDu > HPC, Cloud and Data Systems and Services > J?lich Supercomputing Centre > > Phone: +49-2461-61-6578 > Fax: +49-2461-61-6656 > E-mail: st.graf at fz-juelich.de > WWW: https://urldefense.com/v3/__http://www.fz-juelich.de/jsc/__;!!KGKeukY!2Gc-96JLXSQApClIceidsjnn3c3yGp2w_RSrcXcylKv3vSYzH4zqmFX0o222MdyEb07sy4QfsZ9bKkMvcWKi$ > --------------------------------------------------------------------------------------------- > --------------------------------------------------------------------------------------------- > Forschungszentrum J?lich GmbH > 52425 J?lich > Sitz der Gesellschaft: J?lich > Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 > Vorsitzender des Aufsichtsrats: MinDir Stefan M?ller > Gesch?ftsf?hrung: Prof. Dr. Astrid Lambrecht (Vorsitzende), > Dr. Stephanie Bauer (stellv. Vorsitzende) > Prof. Dr. Ir. Pieter Jansens, Prof. Dr. Laurens Kuipers > --------------------------------------------------------------------------------------------- > --------------------------------------------------------------------------------------------- > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > https://urldefense.com/v3/__http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org__;!!K-Hz7m0Vt54!gDeNVxZMfaV8Ner2Lx3DkTWfq16K8E3E8631B98UhVVjCtacSctrwYYonoYF-aOmKXWnjX-2QkZm$ -- -- Skylar Thompson (skylar2 at u.washington.edu) -- Genome Sciences Department (UW Medicine), System Administrator -- Foege Building S046, (206)-685-7354 -- Pronouns: They/Them From l.r.sudbery at bham.ac.uk Fri Jan 23 16:38:43 2026 From: l.r.sudbery at bham.ac.uk (Luke Sudbery) Date: Fri, 23 Jan 2026 16:38:43 +0000 Subject: [gpfsug-discuss] Anyone using RoCE? Message-ID: Is anyone using RoCE with good results? We are planning on it, but initial tests are not great - we get much better performance using plain Ethernet over the exact same links. It's up and working, I can see RDMA connections and counters, no errors, but performance is unstable. And worse than Ethernet, which was just meant to be a sanity check! Things I've looked at based on Lenovo and IBM guides, which I think are all configured correctly: * RoCE interfaces all on the same subnet * They all have IPv6 enabled with addresses using eui64 addr-gen-mode * DSCP trust mode on NICs * PFC flow control on NICs * Global Pause disabled on NICs * ToS configured for RDMA_CM * Source based routing for multiple interfaces on the same subnet. * Switches (nvidia cumulus) all enabled for RoCE QOS Iperf and GPFS over plain Ethernet get nearly 3GB/s, which is near the line speed of the NIC in question - 25Gbps. Testing basic RDMA connections with ib_send_bw gets about the same. But GPFS over RoCE gets from 0.7GB/s to 1.9GB/s. The servers have 4x 200G Mellanox cards. The client has 1x 25G card. What's frustrating and confusing is that we get better performance when we just enable 1 card at the server end, and also get better performance if we have 1 fabric ID per NIC on the server (with all 4 fabric ID on the same NIC at the client end). I can go into more details if anyone has experience! Does this sound familiar to anyone? I am planning to open a call with Lenovo and/or IBM as I'm not quite sure where to look next. Cheers, Luke -- Luke Sudbery Principal Engineer (HPC and Storage). Architecture, Infrastructure and Systems Advanced Research Computing, IT Services Room 132, Computer Centre G5, Elms Road Please note I don't work on Monday. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Luis.Bolinches at worldquant.com Fri Jan 23 19:23:06 2026 From: Luis.Bolinches at worldquant.com (Bolinches, Luis (WorldQuant)) Date: Fri, 23 Jan 2026 19:23:06 +0000 Subject: [gpfsug-discuss] [EXTERNAL] Anyone using RoCE? In-Reply-To: References: Message-ID: Hi We are Somethings is odd We had pretty good results. Latency halved which had a great improvement on throughput We use two separated fabrics. Configuration was not straight forward as this case is not covered by essgennetworks but worth it In your case you are down to one fabric as clients have one 25G port ________________________________ From: gpfsug-discuss on behalf of Luke Sudbery Sent: Friday, January 23, 2026 5:38:43 PM To: gpfsug main discussion list Subject: [EXTERNAL] [gpfsug-discuss] Anyone using RoCE? Is anyone using RoCE with good results? We are planning on it, but initial tests are not great – we get much better performance using plain Ethernet over the exact same links. It’s up and working, I can see RDMA connections and counters, no errors, but performance is unstable. And worse than Ethernet, which was just meant to be a sanity check! Things I’ve looked at based on Lenovo and IBM guides, which I think are all configured correctly: * RoCE interfaces all on the same subnet * They all have IPv6 enabled with addresses using eui64 addr-gen-mode * DSCP trust mode on NICs * PFC flow control on NICs * Global Pause disabled on NICs * ToS configured for RDMA_CM * Source based routing for multiple interfaces on the same subnet. * Switches (nvidia cumulus) all enabled for RoCE QOS Iperf and GPFS over plain Ethernet get nearly 3GB/s, which is near the line speed of the NIC in question – 25Gbps. Testing basic RDMA connections with ib_send_bw gets about the same. But GPFS over RoCE gets from 0.7GB/s to 1.9GB/s. The servers have 4x 200G Mellanox cards. The client has 1x 25G card. What’s frustrating and confusing is that we get better performance when we just enable 1 card at the server end, and also get better performance if we have 1 fabric ID per NIC on the server (with all 4 fabric ID on the same NIC at the client end). I can go into more details if anyone has experience! Does this sound familiar to anyone? I am planning to open a call with Lenovo and/or IBM as I’m not quite sure where to look next. Cheers, Luke -- Luke Sudbery Principal Engineer (HPC and Storage). Architecture, Infrastructure and Systems Advanced Research Computing, IT Services Room 132, Computer Centre G5, Elms Road Please note I don’t work on Monday. ################################################################################### The information contained in this communication is confidential, may be subject to legal privilege, and is intended only for the individual named. If you are not the named addressee, please notify the sender immediately and delete this email from your system. The views expressed in this email are the views of the sender only. Outgoing and incoming electronic communications to this address are electronically archived and subject to review and/or disclosure to someone other than the recipient. ################################################################################### -------------- next part -------------- An HTML attachment was scrubbed... URL: