Hi What permissions you have ? Do u have only Posix , or also SMB attributes ? If only posix attributes you can do the following: - rsync (which will work on different filesets/directories in parallel. - AFM (but in case you need rollback - it will be problematic) Regards Yaron Daniel 94 Em Ha'Moshavot Rd Storage Architect ? IL Lab Services (Storage) Petach Tiqva, 49527 IBM Global Markets, Systems HW Sales Israel Phone: +972-3-916-5672 Fax: +972-3-916-5672 Mobile: +972-52-8395593 e-mail: yard at il.ibm.com IBM Israel From: "Oesterlin, Robert" <Robert.Oesterlin at nuance.com> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org> Date: 03/05/2019 11:57 PM Subject: [gpfsug-discuss] Migrating billions of files? Sent by: gpfsug-discuss-bounces at spectrumscale.org I?m looking at migration 3-4 Billion files, maybe 3PB of data between GPFS clusters. Most of the files are small - 60% 8K or less. Ideally I?d like to copy at least 15-20M files per day - ideally 50M. Any thoughts on how achievable this is? Or what to use? Either with AFM, mpifileutils, rsync.. other? Many of these files would be in 4k inodes. Destination is ESS. Bob Oesterlin Sr Principal Storage Engineer, Nuance _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Bn1XE9uK2a9CZQ8qKnJE3Q&m=uXadyLeBnskK8mq-S8OjwY-ESxuNxXme9Akj9QaQBiE&s=UdKoJNySkr8itrQaRD9XMkVjBGnVaU8XnyxuKCldX-8&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190306/9f1cf8cd/attachment.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 1851 bytes Desc: not available URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190306/9f1cf8cd/attachment.gif> -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 4376 bytes Desc: not available URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190306/9f1cf8cd/attachment-0001.gif> -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 5093 bytes Desc: not available URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190306/9f1cf8cd/attachment-0002.gif> -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 4746 bytes Desc: not available URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190306/9f1cf8cd/attachment-0003.gif> -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 4557 bytes Desc: not available URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190306/9f1cf8cd/attachment-0004.gif> -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 5093 bytes Desc: not available URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190306/9f1cf8cd/attachment-0005.gif> -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 4786 bytes Desc: not available URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190306/9f1cf8cd/attachment-0006.gif> -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 5054 bytes Desc: not available URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190306/9f1cf8cd/attachment-0007.gif>
