== conf-sanity test 69: replace an OST with the same index ========================================================== 16:00:49 (1710100849) start mds service on oleg232-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg232-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-OST0000 oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg232-client.virtnet: -o user_xattr,flock oleg232-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99936. rc=0 - open/close 1458 (time 1710100880.01 total 10.00 last 145.78) - open/close 2896 (time 1710100890.01 total 20.00 last 143.76) - open/close 4117 (time 1710100900.01 total 30.01 last 122.09) - open/close 5292 (time 1710100910.02 total 40.01 last 117.40) - open/close 6118 (time 1710100920.03 total 50.02 last 82.54) - open/close 7022 (time 1710100930.03 total 60.02 last 90.39) - open/close 7790 (time 1710100940.05 total 70.04 last 76.69) - open/close 8442 (time 1710100950.06 total 80.05 last 65.08) - open/close 9062 (time 1710100960.07 total 90.06 last 61.95) - open/close 9769 (time 1710100970.08 total 100.07 last 70.62) total: 10000 open/close in 103.21 seconds: 96.89 ops/second - unlinked 0 (time 1710100974 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 1170 (time 1710101057.65 total 10.01 last 116.92) - open/close 2144 (time 1710101067.66 total 20.02 last 97.29) - open/close 2805 (time 1710101086.41 total 38.76 last 35.27) - open/close 3649 (time 1710101096.41 total 48.76 last 84.39) - open/close 4474 (time 1710101106.41 total 58.77 last 82.47) - open/close 5202 (time 1710101116.41 total 68.77 last 72.79) - open/close 5870 (time 1710101126.41 total 78.77 last 66.78) - open/close 6562 (time 1710101136.42 total 88.78 last 69.13) - open/close 7273 (time 1710101146.43 total 98.79 last 71.04) - open/close 7914 (time 1710101156.45 total 108.80 last 64.02) - open/close 8514 (time 1710101166.47 total 118.83 last 59.83) - open/close 9126 (time 1710101176.50 total 128.85 last 61.06) - open/close 9683 (time 1710101186.52 total 138.88 last 55.57) total: 10000 open/close in 145.04 seconds: 68.95 ops/second - unlinked 0 (time 1710101194 ; total 0 ; last 0) total: 10000 unlinks in 104 seconds: 96.153847 unlinks/second - open/close 914 (time 1710101310.40 total 10.00 last 91.39) - open/close 1774 (time 1710101320.41 total 20.01 last 85.95) - open/close 2638 (time 1710101330.42 total 30.02 last 86.32) - open/close 3377 (time 1710101340.43 total 40.03 last 73.79) - open/close 4036 (time 1710101350.43 total 50.03 last 65.88) - open/close 4725 (time 1710101360.45 total 60.05 last 68.80) - open/close 5406 (time 1710101370.46 total 70.06 last 68.02) - open/close 6112 (time 1710101380.47 total 80.07 last 70.55) - open/close 6784 (time 1710101390.47 total 90.07 last 67.20) - open/close 7406 (time 1710101400.47 total 100.07 last 62.17) - open/close 8018 (time 1710101410.48 total 110.08 last 61.14) - open/close 8591 (time 1710101420.49 total 120.09 last 57.26) - open/close 9169 (time 1710101430.51 total 130.11 last 57.69) - open/close 9787 (time 1710101440.51 total 140.11 last 61.80) total: 10000 open/close in 144.12 seconds: 69.39 ops/second - unlinked 0 (time 1710101446 ; total 0 ; last 0) total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second - open/close 1020 (time 1710101554.56 total 10.00 last 102.00) - open/close 1924 (time 1710101564.56 total 20.00 last 90.39) - open/close 2798 (time 1710101574.57 total 30.01 last 87.37) - open/close 3572 (time 1710101584.57 total 40.01 last 77.34) - open/close 4343 (time 1710101594.58 total 50.02 last 77.06) - open/close 5029 (time 1710101604.60 total 60.04 last 68.47) - open/close 5764 (time 1710101614.60 total 70.04 last 73.47) - open/close 6452 (time 1710101624.61 total 80.05 last 68.77) - open/close 7152 (time 1710101634.62 total 90.06 last 69.89) - open/close 7769 (time 1710101644.62 total 100.06 last 61.69) - open/close 8370 (time 1710101654.63 total 110.07 last 60.06) - open/close 9031 (time 1710101664.64 total 120.07 last 66.06) - open/close 9615 (time 1710101674.65 total 130.08 last 58.34) total: 10000 open/close in 137.02 seconds: 72.98 ops/second - unlinked 0 (time 1710101683 ; total 0 ; last 0) total: 10000 unlinks in 102 seconds: 98.039215 unlinks/second - open/close 1012 (time 1710101797.47 total 10.01 last 101.10) - open/close 1886 (time 1710101807.47 total 20.02 last 87.35) - open/close 2663 (time 1710101817.48 total 30.02 last 77.64) - open/close 3483 (time 1710101827.48 total 40.03 last 81.98) - open/close 4267 (time 1710101837.49 total 50.03 last 78.37) - open/close 4991 (time 1710101847.50 total 60.04 last 72.35) - open/close 5618 (time 1710101857.50 total 70.05 last 62.65) - open/close 6225 (time 1710101867.51 total 80.05 last 60.64) - open/close 6823 (time 1710101877.53 total 90.07 last 59.69) - open/close 7444 (time 1710101887.54 total 100.08 last 62.04) - open/close 8014 (time 1710101897.55 total 110.09 last 56.94) - open/close 8627 (time 1710101907.57 total 120.11 last 61.19) - open/close 9170 (time 1710101917.58 total 130.12 last 54.24) - open/close 9746 (time 1710101927.59 total 140.13 last 57.58) total: 10000 open/close in 145.38 seconds: 68.78 ops/second - unlinked 0 (time 1710101934 ; total 0 ; last 0) total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second - open/close 1075 (time 1710102043.47 total 10.00 last 107.46) - open/close 2015 (time 1710102053.48 total 20.01 last 93.93) - open/close 2831 (time 1710102063.49 total 30.02 last 81.52) - open/close 3647 (time 1710102073.49 total 40.03 last 81.53) - open/close 4454 (time 1710102083.51 total 50.04 last 80.62) - open/close 5156 (time 1710102093.51 total 60.05 last 70.15) - open/close 5873 (time 1710102103.52 total 70.05 last 71.67) - open/close 6487 (time 1710102113.52 total 80.05 last 61.40) - open/close 7094 (time 1710102123.52 total 90.05 last 60.69) - open/close 7722 (time 1710102133.53 total 100.06 last 62.73) - open/close 8287 (time 1710102143.55 total 110.09 last 56.38) - open/close 8860 (time 1710102153.55 total 120.09 last 57.28) - open/close 9384 (time 1710102163.58 total 130.11 last 52.29) - open/close 9899 (time 1710102173.58 total 140.11 last 51.48) total: 10000 open/close in 142.01 seconds: 70.42 ops/second - unlinked 0 (time 1710102177 ; total 0 ; last 0) total: 10000 unlinks in 105 seconds: 95.238098 unlinks/second - open/close 1015 (time 1710102294.36 total 10.00 last 101.45) - open/close 1932 (time 1710102304.37 total 20.01 last 91.61) - open/close 2793 (time 1710102314.38 total 30.02 last 86.07) - open/close 3624 (time 1710102324.38 total 40.02 last 83.06) - open/close 4317 (time 1710102334.39 total 50.03 last 69.25) - open/close 5073 (time 1710102344.39 total 60.03 last 75.58) - open/close 5815 (time 1710102354.40 total 70.04 last 74.14) - open/close 6455 (time 1710102364.40 total 80.04 last 63.98) - open/close 7093 (time 1710102374.41 total 90.05 last 63.78) - open/close 7747 (time 1710102384.41 total 100.05 last 65.37) - open/close 8300 (time 1710102394.42 total 110.06 last 55.24) - open/close 8879 (time 1710102404.43 total 120.07 last 57.85) - open/close 9473 (time 1710102414.44 total 130.08 last 59.36) total: 10000 open/close in 139.99 seconds: 71.44 ops/second - unlinked 0 (time 1710102426 ; total 0 ; last 0) total: 10000 unlinks in 102 seconds: 98.039215 unlinks/second - open/close 1072 (time 1710102540.23 total 10.01 last 107.12) - open/close 2079 (time 1710102550.24 total 20.01 last 100.65) - open/close 2968 (time 1710102560.25 total 30.02 last 88.84) - open/close 3806 (time 1710102570.26 total 40.03 last 83.70) - open/close 4621 (time 1710102580.27 total 50.04 last 81.40) - open/close 5384 (time 1710102590.29 total 60.06 last 76.19) - open/close 6093 (time 1710102600.29 total 70.06 last 70.86) - open/close 6821 (time 1710102610.31 total 80.08 last 72.66) - open/close 7488 (time 1710102620.33 total 90.10 last 66.59) - open/close 8127 (time 1710102630.33 total 100.10 last 63.90) - open/close 8747 (time 1710102640.33 total 110.11 last 61.97) - open/close 9325 (time 1710102650.34 total 120.11 last 57.77) - open/close 9855 (time 1710102660.35 total 130.12 last 52.94) total: 10000 open/close in 132.86 seconds: 75.27 ops/second - unlinked 0 (time 1710102665 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 1079 (time 1710102764.69 total 10.00 last 107.89) - open/close 2182 (time 1710102774.70 total 20.01 last 110.19) - open/close 3165 (time 1710102784.70 total 30.01 last 98.30) - open/close 4066 (time 1710102794.70 total 40.02 last 90.07) - open/close 4930 (time 1710102804.71 total 50.02 last 86.33) - open/close 5738 (time 1710102814.71 total 60.02 last 80.79) - open/close 6560 (time 1710102824.73 total 70.04 last 82.08) - open/close 7298 (time 1710102834.74 total 80.05 last 73.70) - open/close 8052 (time 1710102844.76 total 90.07 last 75.25) - open/close 8734 (time 1710102854.78 total 100.09 last 68.08) - open/close 9434 (time 1710102864.78 total 110.09 last 69.97) total: 10000 open/close in 117.45 seconds: 85.14 ops/second - unlinked 0 (time 1710102873 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 1574 (time 1710102951.98 total 10.00 last 157.35) - open/close 3060 (time 1710102961.99 total 20.01 last 148.54) - open/close 4506 (time 1710102971.99 total 30.01 last 144.52) - open/close 5947 (time 1710102981.99 total 40.01 last 144.09) - open/close 7464 (time 1710102992.00 total 50.01 last 151.68) - open/close 8842 (time 1710103002.00 total 60.02 last 137.73) total: 10000 open/close in 69.15 seconds: 144.61 ops/second - unlinked 0 (time 1710103012 ; total 0 ; last 0) total: 10000 unlinks in 42 seconds: 238.095245 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg232-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg232-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg232-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.132@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.132@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg232-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-OST0000 oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg232-server: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg232-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec mount lustre on /mnt/lustre..... Starting client: oleg232-client.virtnet: -o user_xattr,flock oleg232-server@tcp:/lustre /mnt/lustre On OST0, 17717 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg232-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg232-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg232-server stop mds service on oleg232-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg232-server unloading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg232-client: oleg232-client: ssh exited with exit code 2 pdsh@oleg232-client: oleg232-server: ssh exited with exit code 2 pdsh@oleg232-client: oleg232-client: ssh exited with exit code 2 pdsh@oleg232-client: oleg232-server: ssh exited with exit code 2