== conf-sanity test 69: replace an OST with the same index ========================================================== 15:57:34 (1710100654) start mds service on oleg221-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg221-server: oleg221-server.virtnet: executing set_default_debug -1 all pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg221-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg221-server: oleg221-server.virtnet: executing set_default_debug -1 all pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1 Started lustre-OST0000 oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg221-client.virtnet: -o user_xattr,flock oleg221-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99936. rc=0 - open/close 2859 (time 1710100682.08 total 10.00 last 285.80) - open/close 5422 (time 1710100692.08 total 20.00 last 256.27) - open/close 7396 (time 1710100702.09 total 30.01 last 197.24) - open/close 8742 (time 1710100712.09 total 40.02 last 134.55) - open/close 9826 (time 1710100722.10 total 50.02 last 108.33) total: 10000 open/close in 51.88 seconds: 192.77 ops/second - unlinked 0 (time 1710100725 ; total 0 ; last 0) total: 10000 unlinks in 43 seconds: 232.558136 unlinks/second - open/close 2503 (time 1710100779.40 total 10.00 last 250.23) - open/close 3959 (time 1710100789.40 total 20.00 last 145.58) - open/close 5204 (time 1710100799.41 total 30.01 last 124.41) - open/close 6250 (time 1710100809.42 total 40.02 last 104.48) - open/close 7209 (time 1710100819.43 total 50.03 last 95.87) - open/close 8125 (time 1710100829.43 total 60.03 last 91.54) - open/close 9041 (time 1710100839.44 total 70.04 last 91.51) total: 10000 open/close in 79.68 seconds: 125.50 ops/second - unlinked 0 (time 1710100850 ; total 0 ; last 0) total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second - open/close 1285 (time 1710100920.18 total 10.01 last 128.40) - open/close 2550 (time 1710100930.19 total 20.01 last 126.43) - open/close 3566 (time 1710100940.20 total 30.02 last 101.49) - open/close 4501 (time 1710100950.20 total 40.02 last 93.50) - open/close 5342 (time 1710100960.20 total 50.03 last 84.07) - open/close 6247 (time 1710100970.21 total 60.03 last 90.42) - open/close 7139 (time 1710100980.21 total 70.04 last 89.19) - open/close 7964 (time 1710100990.22 total 80.04 last 82.44) - open/close 8837 (time 1710101000.22 total 90.05 last 87.26) - open/close 9667 (time 1710101010.24 total 100.06 last 82.86) total: 10000 open/close in 104.71 seconds: 95.50 ops/second - unlinked 0 (time 1710101016 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 1139 (time 1710101101.81 total 10.01 last 113.81) - open/close 2155 (time 1710101111.81 total 20.01 last 101.58) - open/close 3107 (time 1710101121.82 total 30.01 last 95.17) - open/close 3938 (time 1710101131.83 total 40.02 last 83.02) - open/close 4836 (time 1710101141.83 total 50.03 last 89.72) - open/close 5647 (time 1710101151.84 total 60.03 last 81.09) - open/close 6352 (time 1710101161.84 total 70.04 last 70.47) - open/close 7113 (time 1710101171.91 total 80.10 last 75.59) - open/close 7807 (time 1710101181.92 total 90.11 last 69.33) - open/close 8460 (time 1710101191.93 total 100.13 last 65.22) - open/close 9101 (time 1710101201.95 total 110.14 last 63.99) - open/close 9709 (time 1710101211.95 total 120.15 last 60.79) total: 10000 open/close in 124.55 seconds: 80.29 ops/second - unlinked 0 (time 1710101218 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 1032 (time 1710101314.96 total 10.00 last 103.17) - open/close 2042 (time 1710101324.97 total 20.01 last 100.93) - open/close 2950 (time 1710101334.97 total 30.01 last 90.76) - open/close 3764 (time 1710101344.97 total 40.01 last 81.39) - open/close 4499 (time 1710101354.98 total 50.02 last 73.47) - open/close 5244 (time 1710101364.99 total 60.03 last 74.44) - open/close 6012 (time 1710101374.99 total 70.03 last 76.79) - open/close 6765 (time 1710101385.01 total 80.05 last 75.16) - open/close 7465 (time 1710101395.01 total 90.05 last 69.98) - open/close 8107 (time 1710101405.02 total 100.06 last 64.15) - open/close 8723 (time 1710101415.02 total 110.06 last 61.58) - open/close 9329 (time 1710101425.04 total 120.08 last 60.49) - open/close 9737 (time 1710101443.12 total 138.16 last 22.57) total: 10000 open/close in 142.67 seconds: 70.09 ops/second - unlinked 0 (time 1710101449 ; total 0 ; last 0) total: 10000 unlinks in 102 seconds: 98.039215 unlinks/second - open/close 1060 (time 1710101563.50 total 10.01 last 105.90) - open/close 2095 (time 1710101573.50 total 20.01 last 103.49) - open/close 2934 (time 1710101583.52 total 30.03 last 83.71) - open/close 3789 (time 1710101593.52 total 40.03 last 85.49) - open/close 4570 (time 1710101603.53 total 50.04 last 78.04) - open/close 5399 (time 1710101613.53 total 60.05 last 82.87) - open/close 6149 (time 1710101623.54 total 70.06 last 74.93) - open/close 6876 (time 1710101633.55 total 80.06 last 72.66) - open/close 7555 (time 1710101643.56 total 90.07 last 67.85) - open/close 8181 (time 1710101653.56 total 100.07 last 62.58) - open/close 8831 (time 1710101663.57 total 110.08 last 64.96) - open/close 9439 (time 1710101673.57 total 120.08 last 60.79) total: 10000 open/close in 129.36 seconds: 77.30 ops/second - unlinked 0 (time 1710101684 ; total 0 ; last 0) total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second - open/close 1087 (time 1710101793.49 total 10.01 last 108.59) - open/close 1983 (time 1710101803.50 total 20.02 last 89.53) - open/close 2888 (time 1710101813.51 total 30.02 last 90.46) - open/close 3746 (time 1710101823.51 total 40.02 last 85.79) - open/close 4565 (time 1710101833.51 total 50.03 last 81.85) - open/close 5371 (time 1710101843.51 total 60.03 last 80.59) - open/close 6116 (time 1710101853.52 total 70.04 last 74.42) - open/close 6808 (time 1710101863.53 total 80.04 last 69.19) - open/close 7466 (time 1710101873.53 total 90.05 last 65.75) - open/close 8121 (time 1710101883.53 total 100.05 last 65.50) - open/close 8737 (time 1710101904.51 total 121.03 last 29.36) - open/close 9339 (time 1710101914.51 total 131.03 last 60.20) - open/close 9958 (time 1710101924.52 total 141.04 last 61.87) total: 10000 open/close in 141.73 seconds: 70.56 ops/second - unlinked 0 (time 1710101927 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 1026 (time 1710102028.13 total 10.01 last 102.50) - open/close 2047 (time 1710102038.14 total 20.02 last 102.01) - open/close 2997 (time 1710102048.14 total 30.02 last 94.95) - open/close 3827 (time 1710102058.15 total 40.02 last 82.98) - open/close 4644 (time 1710102068.15 total 50.03 last 81.63) - open/close 5461 (time 1710102078.16 total 60.04 last 81.62) - open/close 6235 (time 1710102088.17 total 70.05 last 77.31) - open/close 6465 (time 1710102102.23 total 84.11 last 16.36) - open/close 7134 (time 1710102112.24 total 94.12 last 66.89) - open/close 7801 (time 1710102122.24 total 104.12 last 66.64) - open/close 8483 (time 1710102132.26 total 114.14 last 68.13) - open/close 9097 (time 1710102142.27 total 124.15 last 61.30) - open/close 9701 (time 1710102152.29 total 134.17 last 60.29) total: 10000 open/close in 139.54 seconds: 71.66 ops/second - unlinked 0 (time 1710102159 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 1220 (time 1710102258.30 total 10.00 last 121.98) - open/close 2226 (time 1710102268.31 total 20.01 last 100.56) - open/close 3074 (time 1710102278.31 total 30.01 last 84.75) - open/close 3859 (time 1710102288.32 total 40.02 last 78.44) - open/close 4696 (time 1710102298.33 total 50.02 last 83.67) - open/close 5467 (time 1710102308.33 total 60.03 last 77.09) - open/close 6243 (time 1710102318.35 total 70.04 last 77.46) - open/close 6982 (time 1710102328.36 total 80.06 last 73.81) - open/close 7675 (time 1710102338.36 total 90.06 last 69.26) - open/close 8048 (time 1710102354.38 total 106.08 last 23.29) - open/close 8688 (time 1710102364.39 total 116.09 last 63.91) - open/close 9343 (time 1710102374.40 total 126.10 last 65.45) - open/close 9989 (time 1710102384.40 total 136.10 last 64.59) total: 10000 open/close in 136.29 seconds: 73.37 ops/second - unlinked 0 (time 1710102386 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 1099 (time 1710102482.16 total 10.01 last 109.83) - open/close 2157 (time 1710102492.16 total 20.01 last 105.72) - open/close 3120 (time 1710102502.17 total 30.02 last 96.20) - open/close 4021 (time 1710102512.18 total 40.03 last 90.07) - open/close 4858 (time 1710102522.18 total 50.03 last 83.67) - open/close 5535 (time 1710102541.55 total 69.40 last 34.96) - open/close 6364 (time 1710102551.55 total 79.40 last 82.88) - open/close 7110 (time 1710102561.56 total 89.41 last 74.54) - open/close 7829 (time 1710102571.57 total 99.42 last 71.79) - open/close 8098 (time 1710102586.55 total 114.41 last 17.95) - open/close 8747 (time 1710102596.57 total 124.42 last 64.83) - open/close 9442 (time 1710102606.58 total 134.43 last 69.42) total: 10000 open/close in 142.79 seconds: 70.03 ops/second - unlinked 0 (time 1710102616 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg221-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg221-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg221-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.121@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.121@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg221-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg221-server: oleg221-server.virtnet: executing set_default_debug -1 all pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1 Started lustre-OST0000 oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg221-server: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg221-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec mount lustre on /mnt/lustre..... Starting client: oleg221-client.virtnet: -o user_xattr,flock oleg221-server@tcp:/lustre /mnt/lustre On OST0, 9781 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg221-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg221-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg221-server stop mds service on oleg221-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg221-server unloading modules on: 'oleg221-server' oleg221-server: oleg221-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg221-client: oleg221-client: ssh exited with exit code 2 pdsh@oleg221-client: oleg221-server: ssh exited with exit code 2 pdsh@oleg221-client: oleg221-client: ssh exited with exit code 2 pdsh@oleg221-client: oleg221-server: ssh exited with exit code 2