<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Dear All,<br>
Happy new year to everyone!</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">My goal here at the Paul Scherrer Institut is to merge two different GPFS building
blocks. In partucular, these are not the same technology or not even the same brand:</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">- an IBM ESS-3500 (a NVMe/Performance storage system) consisting of a Power9 confluent
node and two AMD canisters, and 12 NVMe drives</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">- a Lenovo G242 "hybrid" consisting of 4 HDD enclosures, 2 SSD enclosures, 1 Intel support
node and 2 Intel storage nodes.</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">The final configuration I would expect is a single building block with 4 IO nodes, 3
declustered array: 1 for HDDs, 1 for SSDs, 1 for NVMe (the last one to be used as a cache pool).</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">First of all, I would like to know if anyone has already tried this solution successfully.</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Then, below is the description of what I have done.</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">I will preface by saying that I was able to configure the two storage clusters separately
without any problem; therefore, I would exclude any inherent problem in each building block (which was installed from scratch). But when I try to have a single cluster, with different node classes, I have problems.</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">The steps I followed (based on documentation I found in IBM pages, https://www.ibm.com/docs/en/ess-p8/5.3.1?topic=command-outline-mmvdisk-use-case)
are as follows:</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">1 access one of the 2 building blocks (that already has a storage cluster configured, with no recoverygroups
defined)</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">2 run "mmaddnode -N <the_two_IO_nodes_to_add_from_the_other_BB>"</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">3 mmchlicense...</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">3 mmvdisk nodeclass create ... to isolate the two "new" IO nodes in a dedicated nodeclass
for the purpose of differentiating configuration parameters, connected drive topology, and then recovery groups</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">4 perform topology discovery with: mmvdisk server list --node-class ess --disk-topology</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">In the following the cluster and node classes:</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div class="elementToProof"><span style="font-family: monospace; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div class="elementToProof"><span style="font-family: monospace; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div class="elementToProof"><span style="font-family: monospace; font-size: 12pt; color: rgb(0, 0, 0);"><br>
Node Daemon node name IP address Admin node name Designation <br>
---------------------------------------------------------------------- <br>
1 sfdssio1.psi.ch 129.129.241.67 sfdssio1.psi.ch quorum-manager <br>
2 sfdssio2.psi.ch 129.129.241.68 sfdssio2.psi.ch quorum-manager <br>
3 sfessio1.psi.ch 129.129.241.27 sfessio1.psi.ch quorum-manager <br>
4 sfessio2.psi.ch 129.129.241.28 sfessio2.psi.ch quorum-manager <br>
<br>
Node Class Name Members <br>
--------------------- -----------------------------------------------------------
<br>
ess sfessio1.psi.ch,sfessio2.psi.ch <br>
dss sfdssio1.psi.ch,sfdssio2.psi.ch<br>
<br>
</span><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">The "mmnodeadd" operation was performed while logged into sfdssio1 (which belongs to
the Lenovo G242).<br>
Then:</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div class="elementToProof"><span style="font-family: monospace; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">[root@sfdssio1 ~]# mmvdisk server list --node-class ess --disk-topology
</span></div>
<div class="elementToProof"><span style="font-family: monospace; font-size: 12pt; color: rgb(0, 0, 0);"><br>
node needs matching <br>
number server attention metric disk topology <br>
------ -------------------------------- --------- -------- ------------- <br>
3 sfessio1.psi.ch yes - unmatched server topology
<br>
4 sfessio2.psi.ch yes - unmatched server topology
<br>
<br>
mmvdisk: To see what needs attention, use the command: <br>
mmvdisk: mmvdisk server list -N sfessio1.psi.ch --disk-topology -L <br>
mmvdisk: mmvdisk server list -N sfessio2.psi.ch --disk-topology -L</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div class="elementToProof"><span style="font-family: monospace; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">[root@sfdssio1 ~]# mmvdisk server list -N sfessio1.psi.ch --disk-topology -L
</span></div>
<div class="elementToProof"><span style="font-family: monospace; font-size: 12pt; color: rgb(0, 0, 0);">Unable to find a matching topology specification for topology file '/var/mmfs/tmp/cmdTmpDir.mmvdisk.1468913/pdisk-topology.sfessio1.psi.ch'.
<br>
<br>
Topology component identification is using these CST stanza files: <br>
<br>
/usr/lpp/mmfs/data/compSpec-1304.stanza <br>
/usr/lpp/mmfs/data/compSpec-1400.stanza <br>
/usr/lpp/mmfs/data/cst/compSpec-Lenovo.stanza <br>
/usr/lpp/mmfs/data/cst/compSpec-topology.stanza <br>
<br>
Server component: serverType 'ESS3500-5141-FN2' serverArch 'x86_64' serverName 'sfessio1.psi.ch'
<br>
<br>
Enclosure components: 1 found connected to HBAs <br>
Enclosure component: serialNumber '78E4395' enclosureClass 'unknown' <br>
<br>
HBA components: none found connected to enclosures <br>
<br>
Cabling: enclosure '78E4395' controller '' cabled to HBA slot 'UNKNOWN' port 'unknown'
<br>
<br>
Disks: 12 SSDs 0 HDDs <br>
NVRAM: 0 devices/partitions <br>
<br>
Unable to match these components to a serverTopology specification. <br>
mmvdisk: Command failed. Examine previous error messages to determine cause.</span></div>
<div class="elementToProof"><span style="font-family: monospace; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div class="elementToProof"><span style="font-family: monospace; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div class="elementToProof"><span style="font-family: monospace; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">If I try to do a symmetric operation (I access an IO node of the IBM ESS3500 and try
to add Lenovo nodes, trying to discover their drive topology) I get the same error; but, of course, the topology involved this time is that of the Lenovo hardware.</span></div>
<div class="elementToProof"><span style="font-family: monospace; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Now, I suspect there is a (hidden?) step I would be supposed to know, but unfortunately
I don't (this is my first experience with different and etherogenous building blocks merge). So I'd like to receive from you any suggestions, including\u2002a better documentation page (if any) covering this particular use case I have.<br>
<br>
Hope the description of the context is clear enough, in case it is not I apologize and please just ask for any further details required to understand my environment.</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Thank you very much,</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> Alvise Dorigo</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
</body>
</html>