== conf-sanity test 69: replace an OST with the same index ========================================================== 16:02:08 (1710100928) start mds service on oleg160-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg160-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 Started lustre-OST0000 oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg160-client.virtnet: -o user_xattr,flock oleg160-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99936. rc=0 - open/close 1224 (time 1710100958.80 total 10.01 last 122.30) - open/close 2218 (time 1710100968.81 total 20.01 last 99.34) - open/close 3239 (time 1710100978.81 total 30.02 last 102.08) - open/close 4061 (time 1710100988.81 total 40.02 last 82.18) - open/close 4882 (time 1710100998.81 total 50.02 last 82.10) - open/close 5711 (time 1710101008.82 total 60.03 last 82.83) - open/close 6578 (time 1710101018.83 total 70.03 last 86.64) - open/close 7284 (time 1710101028.85 total 80.05 last 70.46) - open/close 8051 (time 1710101038.86 total 90.06 last 76.63) - open/close 8751 (time 1710101048.87 total 100.07 last 69.93) - open/close 9462 (time 1710101058.88 total 110.08 last 71.02) total: 10000 open/close in 117.21 seconds: 85.31 ops/second - unlinked 0 (time 1710101067 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 1038 (time 1710101162.08 total 10.01 last 103.74) - open/close 1996 (time 1710101172.08 total 20.01 last 95.80) - open/close 2767 (time 1710101182.11 total 30.03 last 76.90) - open/close 3617 (time 1710101192.11 total 40.03 last 84.99) - open/close 4377 (time 1710101202.12 total 50.05 last 75.89) - open/close 5101 (time 1710101212.13 total 60.05 last 72.37) - open/close 5750 (time 1710101222.13 total 70.06 last 64.85) - open/close 6372 (time 1710101232.14 total 80.06 last 62.16) - open/close 6977 (time 1710101242.15 total 90.07 last 60.43) - open/close 7580 (time 1710101252.16 total 100.09 last 60.22) - open/close 8155 (time 1710101262.18 total 110.10 last 57.41) - open/close 8655 (time 1710101272.20 total 120.13 last 49.88) - open/close 9177 (time 1710101282.22 total 130.14 last 52.13) - open/close 9787 (time 1710101292.22 total 140.15 last 60.95) total: 10000 open/close in 144.13 seconds: 69.38 ops/second - unlinked 0 (time 1710101298 ; total 0 ; last 0) total: 10000 unlinks in 109 seconds: 91.743118 unlinks/second - open/close 916 (time 1710101420.05 total 10.00 last 91.60) - open/close 1805 (time 1710101430.05 total 20.01 last 88.84) - open/close 2660 (time 1710101440.06 total 30.01 last 85.46) - open/close 3451 (time 1710101450.07 total 40.02 last 79.02) - open/close 4126 (time 1710101460.07 total 50.03 last 67.48) - open/close 4756 (time 1710101470.11 total 60.06 last 62.77) - open/close 5402 (time 1710101480.11 total 70.07 last 64.57) - open/close 6012 (time 1710101490.11 total 80.07 last 60.99) - open/close 6576 (time 1710101500.11 total 90.07 last 56.39) - open/close 7145 (time 1710101510.13 total 100.08 last 56.82) - open/close 7731 (time 1710101520.13 total 110.09 last 58.57) - open/close 8322 (time 1710101530.14 total 120.09 last 59.09) - open/close 8790 (time 1710101540.14 total 130.09 last 46.80) - open/close 9378 (time 1710101550.18 total 140.14 last 58.54) - open/close 9877 (time 1710101560.19 total 150.15 last 49.85) total: 10000 open/close in 152.39 seconds: 65.62 ops/second - unlinked 0 (time 1710101564 ; total 0 ; last 0) total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second - open/close 932 (time 1710101676.92 total 10.00 last 93.17) - open/close 1839 (time 1710101686.94 total 20.01 last 90.59) - open/close 2665 (time 1710101696.94 total 30.02 last 82.59) - open/close 3452 (time 1710101706.95 total 40.03 last 78.62) - open/close 4103 (time 1710101716.96 total 50.03 last 65.04) - open/close 4576 (time 1710101726.97 total 60.05 last 47.25) - open/close 5199 (time 1710101736.98 total 70.06 last 62.19) - open/close 5733 (time 1710101746.99 total 80.07 last 53.35) - open/close 6347 (time 1710101757.01 total 90.09 last 61.31) - open/close 6883 (time 1710101767.02 total 100.10 last 53.56) - open/close 7409 (time 1710101777.07 total 110.15 last 52.33) - open/close 7966 (time 1710101787.08 total 120.16 last 55.65) - open/close 8011 (time 1710101798.92 total 132.00 last 3.80) - open/close 8471 (time 1710101808.93 total 142.01 last 45.96) - open/close 8913 (time 1710101818.93 total 152.01 last 44.20) - open/close 9365 (time 1710101828.94 total 162.02 last 45.15) - open/close 9871 (time 1710101838.95 total 172.03 last 50.52) total: 10000 open/close in 174.74 seconds: 57.23 ops/second - unlinked 0 (time 1710101844 ; total 0 ; last 0) total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second - open/close 955 (time 1710101953.21 total 10.01 last 95.40) - open/close 1827 (time 1710101963.22 total 20.02 last 87.14) - open/close 2618 (time 1710101973.22 total 30.02 last 79.07) - open/close 3379 (time 1710101983.23 total 40.03 last 76.02) - open/close 4128 (time 1710101993.25 total 50.05 last 74.78) - open/close 4726 (time 1710102003.31 total 60.11 last 59.43) - open/close 5390 (time 1710102013.32 total 70.12 last 66.35) - open/close 5993 (time 1710102023.32 total 80.12 last 60.29) - open/close 6614 (time 1710102033.33 total 90.13 last 62.02) - open/close 7248 (time 1710102043.34 total 100.14 last 63.36) - open/close 7825 (time 1710102053.34 total 110.14 last 57.69) - open/close 8355 (time 1710102063.35 total 120.15 last 52.97) - open/close 8879 (time 1710102073.35 total 130.15 last 52.38) - open/close 9401 (time 1710102083.36 total 140.16 last 52.15) - open/close 9915 (time 1710102093.37 total 150.16 last 51.37) total: 10000 open/close in 151.85 seconds: 65.85 ops/second - unlinked 0 (time 1710102097 ; total 0 ; last 0) total: 10000 unlinks in 105 seconds: 95.238098 unlinks/second - open/close 994 (time 1710102215.57 total 10.01 last 99.27) - open/close 1902 (time 1710102225.58 total 20.02 last 90.72) - open/close 2728 (time 1710102235.59 total 30.03 last 82.51) - open/close 3490 (time 1710102245.61 total 40.05 last 76.09) - open/close 4148 (time 1710102255.62 total 50.06 last 65.73) - open/close 4825 (time 1710102265.62 total 60.06 last 67.67) - open/close 5484 (time 1710102275.62 total 70.06 last 65.89) - open/close 6126 (time 1710102285.63 total 80.07 last 64.17) - open/close 6652 (time 1710102295.63 total 90.07 last 52.57) - open/close 7146 (time 1710102305.64 total 100.08 last 49.35) - open/close 7744 (time 1710102315.65 total 110.09 last 59.78) - open/close 8256 (time 1710102325.65 total 120.09 last 51.18) - open/close 8811 (time 1710102335.65 total 130.09 last 55.49) - open/close 9297 (time 1710102345.67 total 140.11 last 48.54) - open/close 9849 (time 1710102355.68 total 150.12 last 55.14) - open/close 9852 (time 1710102366.89 total 161.33 last 0.27) total: 10000 open/close in 164.10 seconds: 60.94 ops/second - unlinked 0 (time 1710102371 ; total 0 ; last 0) total: 10000 unlinks in 102 seconds: 98.039215 unlinks/second - open/close 1006 (time 1710102486.10 total 10.01 last 100.54) - open/close 1861 (time 1710102496.11 total 20.01 last 85.43) - open/close 2610 (time 1710102506.11 total 30.01 last 74.90) - open/close 3396 (time 1710102516.12 total 40.03 last 78.51) - open/close 4144 (time 1710102526.13 total 50.03 last 74.75) - open/close 4860 (time 1710102536.13 total 60.04 last 71.57) - open/close 5510 (time 1710102546.14 total 70.05 last 64.94) - open/close 6094 (time 1710102556.14 total 80.05 last 58.39) - open/close 6589 (time 1710102566.15 total 90.05 last 49.48) - open/close 7052 (time 1710102576.16 total 100.06 last 46.25) - open/close 7672 (time 1710102586.16 total 110.07 last 61.98) - open/close 8209 (time 1710102596.16 total 120.07 last 53.69) - open/close 8736 (time 1710102606.17 total 130.08 last 52.65) - open/close 9292 (time 1710102616.18 total 140.09 last 55.54) - open/close 9771 (time 1710102626.20 total 150.10 last 47.83) total: 10000 open/close in 154.56 seconds: 64.70 ops/second - unlinked 0 (time 1710102632 ; total 0 ; last 0) total: 10000 unlinks in 103 seconds: 97.087379 unlinks/second - open/close 964 (time 1710102748.05 total 10.01 last 96.32) - open/close 1541 (time 1710102765.62 total 27.58 last 32.83) - open/close 2354 (time 1710102775.64 total 37.60 last 81.18) - open/close 3142 (time 1710102785.65 total 47.61 last 78.71) - open/close 3905 (time 1710102795.66 total 57.62 last 76.18) - open/close 4638 (time 1710102805.70 total 67.66 last 73.01) - open/close 5162 (time 1710102815.71 total 77.67 last 52.38) - open/close 5799 (time 1710102825.71 total 87.67 last 63.68) - open/close 6397 (time 1710102835.72 total 97.68 last 59.76) - open/close 7015 (time 1710102845.72 total 107.68 last 61.79) - open/close 7547 (time 1710102855.72 total 117.68 last 53.19) - open/close 8135 (time 1710102865.74 total 127.70 last 58.68) - open/close 8676 (time 1710102875.76 total 137.72 last 53.99) - open/close 9199 (time 1710102885.78 total 147.74 last 52.22) - open/close 9640 (time 1710102895.79 total 157.75 last 44.04) total: 10000 open/close in 166.42 seconds: 60.09 ops/second - unlinked 0 (time 1710102907 ; total 0 ; last 0) total: 10000 unlinks in 114 seconds: 87.719299 unlinks/second - open/close 808 (time 1710103033.88 total 10.00 last 80.78) - open/close 1668 (time 1710103043.88 total 20.01 last 85.94) - open/close 2519 (time 1710103053.89 total 30.01 last 85.06) - open/close 3248 (time 1710103074.02 total 50.15 last 36.21) - open/close 3709 (time 1710103091.96 total 68.09 last 25.69) - open/close 4481 (time 1710103101.97 total 78.10 last 77.12) - open/close 5208 (time 1710103111.98 total 88.10 last 72.67) - open/close 5887 (time 1710103121.98 total 98.10 last 67.90) - open/close 6485 (time 1710103131.99 total 108.12 last 59.70) - open/close 7092 (time 1710103142.00 total 118.12 last 60.68) - open/close 7701 (time 1710103152.00 total 128.12 last 60.89) - open/close 8321 (time 1710103162.00 total 138.13 last 61.96) - open/close 8838 (time 1710103172.02 total 148.15 last 51.60) - open/close 9379 (time 1710103182.02 total 158.15 last 54.10) - open/close 9923 (time 1710103192.03 total 168.15 last 54.38) total: 10000 open/close in 169.60 seconds: 58.96 ops/second - unlinked 0 (time 1710103195 ; total 0 ; last 0) total: 10000 unlinks in 101 seconds: 99.009903 unlinks/second - open/close 1018 (time 1710103308.91 total 10.03 last 101.46) - open/close 2083 (time 1710103318.92 total 20.04 last 106.44) - open/close 2974 (time 1710103328.92 total 30.04 last 89.07) - open/close 3893 (time 1710103338.94 total 40.06 last 91.77) - open/close 4774 (time 1710103348.94 total 50.06 last 88.06) - open/close 5557 (time 1710103359.01 total 60.12 last 77.81) - open/close 6326 (time 1710103369.02 total 70.14 last 76.80) - open/close 7106 (time 1710103379.02 total 80.14 last 77.97) - open/close 7864 (time 1710103389.02 total 90.14 last 75.78) - open/close 8654 (time 1710103399.03 total 100.15 last 78.96) - open/close 9326 (time 1710103409.04 total 110.16 last 67.12) total: 10000 open/close in 119.27 seconds: 83.84 ops/second - unlinked 0 (time 1710103419 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg160-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg160-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.160@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.160@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg160-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 Started lustre-OST0000 oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg160-server: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg160-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec mount lustre on /mnt/lustre..... Starting client: oleg160-client.virtnet: -o user_xattr,flock oleg160-server@tcp:/lustre /mnt/lustre On OST0, 20325 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg160-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg160-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server stop mds service on oleg160-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server unloading modules on: 'oleg160-server' oleg160-server: oleg160-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg160-client: oleg160-client: ssh exited with exit code 2 pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2 pdsh@oleg160-client: oleg160-client: ssh exited with exit code 2 pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2