== conf-sanity test 69: replace an OST with the same index ========================================================== 16:04:07 (1710101047) start mds service on oleg327-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg327-server: oleg327-server.virtnet: executing set_default_debug -1 all pdsh@oleg327-client: oleg327-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg327-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg327-server: oleg327-server.virtnet: executing set_default_debug -1 all pdsh@oleg327-client: oleg327-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg327-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg327-server: oleg327-server.virtnet: executing set_default_debug -1 all pdsh@oleg327-client: oleg327-server: ssh exited with exit code 1 Started lustre-OST0000 oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg327-client.virtnet: -o user_xattr,flock oleg327-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11840 inodes available. Want 99936. rc=0 - open/close 1238 (time 1710101091.01 total 10.00 last 123.79) - open/close 2259 (time 1710101101.01 total 20.01 last 102.04) - open/close 3240 (time 1710101111.01 total 30.01 last 98.06) - open/close 4219 (time 1710101121.03 total 40.02 last 97.77) - open/close 5102 (time 1710101131.04 total 50.04 last 88.18) - open/close 5930 (time 1710101141.06 total 60.06 last 82.64) - open/close 6691 (time 1710101151.06 total 70.06 last 76.10) - open/close 7520 (time 1710101161.07 total 80.07 last 82.81) - open/close 8216 (time 1710101171.08 total 90.08 last 69.55) - open/close 8972 (time 1710101181.09 total 100.09 last 75.49) - open/close 9685 (time 1710101191.10 total 110.10 last 71.24) total: 10000 open/close in 114.16 seconds: 87.60 ops/second - unlinked 0 (time 1710101197 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 1193 (time 1710101282.51 total 10.00 last 119.30) - open/close 2306 (time 1710101292.51 total 20.00 last 111.29) - open/close 3350 (time 1710101302.51 total 30.00 last 104.37) - open/close 4384 (time 1710101312.51 total 40.00 last 103.40) - open/close 5340 (time 1710101322.52 total 50.01 last 95.53) - open/close 6203 (time 1710101332.52 total 60.02 last 86.26) - open/close 7005 (time 1710101342.54 total 70.03 last 80.07) - open/close 7811 (time 1710101352.54 total 80.03 last 80.58) - open/close 8541 (time 1710101373.46 total 100.95 last 34.90) - open/close 9294 (time 1710101383.47 total 110.97 last 75.18) total: 10000 open/close in 120.42 seconds: 83.04 ops/second - unlinked 0 (time 1710101394 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 1154 (time 1710101480.80 total 10.00 last 115.35) - open/close 2276 (time 1710101490.80 total 20.01 last 112.18) - open/close 3324 (time 1710101500.81 total 30.01 last 104.79) - open/close 4291 (time 1710101510.81 total 40.02 last 96.61) - open/close 5211 (time 1710101520.82 total 50.02 last 91.97) - open/close 6076 (time 1710101530.82 total 60.02 last 86.50) - open/close 6871 (time 1710101540.83 total 70.03 last 79.42) - open/close 7592 (time 1710101550.83 total 80.03 last 72.08) - open/close 8414 (time 1710101560.84 total 90.04 last 82.14) - open/close 9152 (time 1710101570.84 total 100.04 last 73.80) - open/close 9913 (time 1710101580.84 total 110.04 last 76.09) total: 10000 open/close in 111.30 seconds: 89.85 ops/second - unlinked 0 (time 1710101583 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 1221 (time 1710101672.48 total 10.00 last 122.05) - open/close 2401 (time 1710101682.48 total 20.00 last 117.99) - open/close 3470 (time 1710101692.49 total 30.01 last 106.81) - open/close 4414 (time 1710101702.50 total 40.02 last 94.32) - open/close 5344 (time 1710101712.51 total 50.04 last 92.87) - open/close 6266 (time 1710101722.51 total 60.04 last 92.19) - open/close 7154 (time 1710101732.52 total 70.04 last 88.76) - open/close 7995 (time 1710101742.52 total 80.05 last 84.06) - open/close 8861 (time 1710101752.53 total 90.05 last 86.56) - open/close 9689 (time 1710101762.53 total 100.06 last 82.74) total: 10000 open/close in 103.78 seconds: 96.36 ops/second - unlinked 0 (time 1710101767 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 1328 (time 1710101845.74 total 10.01 last 132.72) - open/close 2524 (time 1710101855.74 total 20.01 last 119.58) - open/close 3654 (time 1710101865.74 total 30.01 last 112.99) - open/close 4653 (time 1710101875.75 total 40.02 last 99.82) - open/close 5573 (time 1710101885.76 total 50.03 last 91.85) - open/close 6495 (time 1710101895.77 total 60.04 last 92.15) - open/close 7238 (time 1710101905.77 total 70.04 last 74.27) - open/close 8005 (time 1710101915.78 total 80.04 last 76.67) - open/close 8725 (time 1710101925.78 total 90.05 last 71.98) - open/close 9444 (time 1710101935.80 total 100.07 last 71.73) total: 10000 open/close in 107.52 seconds: 93.01 ops/second - unlinked 0 (time 1710101944 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 1303 (time 1710102023.04 total 10.01 last 130.23) - open/close 2516 (time 1710102033.04 total 20.01 last 121.26) - open/close 3626 (time 1710102043.05 total 30.01 last 110.94) - open/close 4706 (time 1710102053.06 total 40.02 last 107.89) - open/close 5690 (time 1710102063.06 total 50.03 last 98.38) - open/close 6604 (time 1710102073.07 total 60.04 last 91.29) - open/close 7509 (time 1710102083.08 total 70.05 last 90.41) - open/close 8351 (time 1710102093.10 total 80.06 last 84.10) - open/close 9186 (time 1710102103.10 total 90.07 last 83.43) - open/close 9975 (time 1710102113.12 total 100.08 last 78.81) total: 10000 open/close in 100.41 seconds: 99.59 ops/second - unlinked 0 (time 1710102115 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 1497 (time 1710102189.81 total 10.00 last 149.65) - open/close 2898 (time 1710102199.81 total 20.01 last 140.07) - open/close 4152 (time 1710102209.81 total 30.01 last 125.36) - open/close 5346 (time 1710102219.82 total 40.02 last 119.31) - open/close 6418 (time 1710102229.83 total 50.02 last 107.17) - open/close 7458 (time 1710102239.83 total 60.02 last 103.94) - open/close 8413 (time 1710102249.84 total 70.03 last 95.42) - open/close 9323 (time 1710102259.84 total 80.04 last 90.97) total: 10000 open/close in 87.67 seconds: 114.06 ops/second - unlinked 0 (time 1710102268 ; total 0 ; last 0) total: 10000 unlinks in 59 seconds: 169.491531 unlinks/second - open/close 1615 (time 1710102338.57 total 10.00 last 161.44) - open/close 3059 (time 1710102348.57 total 20.01 last 144.33) - open/close 4341 (time 1710102358.58 total 30.01 last 128.15) - open/close 5573 (time 1710102368.58 total 40.02 last 123.11) - open/close 6737 (time 1710102378.59 total 50.02 last 116.35) - open/close 7847 (time 1710102388.59 total 60.02 last 110.99) - open/close 8884 (time 1710102398.60 total 70.03 last 103.60) - open/close 9843 (time 1710102408.60 total 80.04 last 95.86) total: 10000 open/close in 81.70 seconds: 122.40 ops/second - unlinked 0 (time 1710102411 ; total 0 ; last 0) total: 10000 unlinks in 57 seconds: 175.438599 unlinks/second - open/close 1719 (time 1710102479.72 total 10.00 last 171.87) - open/close 3302 (time 1710102489.72 total 20.00 last 158.25) - open/close 4816 (time 1710102499.72 total 30.01 last 151.33) - open/close 6243 (time 1710102509.73 total 40.02 last 142.60) - open/close 7563 (time 1710102519.74 total 50.02 last 131.89) - open/close 8785 (time 1710102529.74 total 60.03 last 122.18) - open/close 9944 (time 1710102539.75 total 70.03 last 115.82) total: 10000 open/close in 70.51 seconds: 141.83 ops/second - unlinked 0 (time 1710102541 ; total 0 ; last 0) total: 10000 unlinks in 41 seconds: 243.902435 unlinks/second - open/close 3203 (time 1710102593.22 total 10.00 last 320.30) - open/close 5793 (time 1710102603.22 total 20.00 last 258.99) - open/close 7898 (time 1710102613.23 total 30.01 last 210.39) - open/close 9747 (time 1710102623.24 total 40.01 last 184.79) total: 10000 open/close in 41.61 seconds: 240.33 ops/second - unlinked 0 (time 1710102625 ; total 0 ; last 0) total: 10000 unlinks in 27 seconds: 370.370361 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg327-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg327-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg327-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.127@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.127@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg327-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg327-server: oleg327-server.virtnet: executing set_default_debug -1 all pdsh@oleg327-client: oleg327-server: ssh exited with exit code 1 Started lustre-OST0000 oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg327-server: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg327-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 12 sec oleg327-server: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg327-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg327-client.virtnet: -o user_xattr,flock oleg327-server@tcp:/lustre /mnt/lustre On OST0, 20428 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg327-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg327-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg327-server stop mds service on oleg327-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg327-server stop mds service on oleg327-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg327-server unloading modules on: 'oleg327-server' oleg327-server: oleg327-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg327-client: oleg327-client: ssh exited with exit code 2 pdsh@oleg327-client: oleg327-server: ssh exited with exit code 2 pdsh@oleg327-client: oleg327-client: ssh exited with exit code 2 pdsh@oleg327-client: oleg327-server: ssh exited with exit code 2