== conf-sanity test 69: replace an OST with the same index ========================================================== 15:58:53 (1710100733) start mds service on oleg424-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg424-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-OST0000 oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg424-client.virtnet: -o user_xattr,flock oleg424-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99936. rc=0 - open/close 1847 (time 1710100764.74 total 10.00 last 184.69) - open/close 3346 (time 1710100774.77 total 20.03 last 149.53) - open/close 4211 (time 1710100784.77 total 30.03 last 86.48) - open/close 5221 (time 1710100794.77 total 40.03 last 100.94) - open/close 6096 (time 1710100804.78 total 50.04 last 87.44) - open/close 6949 (time 1710100814.78 total 60.04 last 85.26) - open/close 7880 (time 1710100824.79 total 70.05 last 93.08) - open/close 8807 (time 1710100834.80 total 80.06 last 92.61) - open/close 9700 (time 1710100844.80 total 90.06 last 89.29) total: 10000 open/close in 93.20 seconds: 107.29 ops/second - unlinked 0 (time 1710100849 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 1400 (time 1710100924.94 total 10.01 last 139.86) - open/close 2390 (time 1710100934.95 total 20.02 last 98.93) - open/close 3395 (time 1710100944.95 total 30.02 last 100.47) - open/close 4596 (time 1710100954.96 total 40.02 last 120.04) - open/close 5661 (time 1710100964.96 total 50.03 last 106.44) - open/close 6458 (time 1710100974.97 total 60.04 last 79.64) - open/close 7299 (time 1710100984.97 total 70.04 last 84.09) - open/close 8093 (time 1710100994.97 total 80.04 last 79.38) - open/close 8851 (time 1710101004.97 total 90.04 last 75.80) - open/close 9628 (time 1710101014.98 total 100.05 last 77.63) total: 10000 open/close in 106.28 seconds: 94.09 ops/second - unlinked 0 (time 1710101023 ; total 0 ; last 0) total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second - open/close 1066 (time 1710101129.92 total 10.01 last 106.53) - open/close 1918 (time 1710101139.92 total 20.02 last 85.13) - open/close 2845 (time 1710101149.93 total 30.02 last 92.68) - open/close 3606 (time 1710101159.94 total 40.03 last 76.02) - open/close 4347 (time 1710101169.95 total 50.04 last 74.00) - open/close 4989 (time 1710101179.97 total 60.06 last 64.10) - open/close 5598 (time 1710101189.98 total 70.07 last 60.81) - open/close 6244 (time 1710101199.99 total 80.08 last 64.55) - open/close 6915 (time 1710101209.99 total 90.08 last 67.09) - open/close 7486 (time 1710101220.00 total 100.09 last 57.07) - open/close 7977 (time 1710101230.01 total 110.10 last 49.05) - open/close 8518 (time 1710101240.02 total 120.11 last 54.04) - open/close 8976 (time 1710101250.04 total 130.13 last 45.73) - open/close 9434 (time 1710101260.07 total 140.16 last 45.64) - open/close 9885 (time 1710101270.09 total 150.18 last 45.02) total: 10000 open/close in 152.53 seconds: 65.56 ops/second - unlinked 0 (time 1710101274 ; total 0 ; last 0) total: 10000 unlinks in 101 seconds: 99.009903 unlinks/second - open/close 988 (time 1710101387.50 total 10.01 last 98.73) - open/close 1902 (time 1710101397.51 total 20.02 last 91.29) - open/close 2697 (time 1710101407.52 total 30.02 last 79.47) - open/close 3402 (time 1710101417.53 total 40.03 last 70.43) - open/close 4186 (time 1710101427.54 total 50.05 last 78.28) - open/close 4891 (time 1710101437.56 total 60.06 last 70.38) - open/close 5604 (time 1710101447.57 total 70.07 last 71.25) - open/close 6237 (time 1710101457.58 total 80.08 last 63.22) - open/close 6813 (time 1710101467.59 total 90.09 last 57.54) - open/close 7386 (time 1710101477.60 total 100.10 last 57.24) - open/close 7952 (time 1710101487.61 total 110.12 last 56.52) - open/close 8472 (time 1710101497.62 total 120.13 last 51.94) - open/close 8930 (time 1710101507.63 total 130.14 last 45.77) - open/close 9440 (time 1710101517.68 total 140.18 last 50.77) - open/close 9908 (time 1710101527.69 total 150.20 last 46.72) total: 10000 open/close in 152.02 seconds: 65.78 ops/second - unlinked 0 (time 1710101531 ; total 0 ; last 0) total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second - open/close 1012 (time 1710101643.87 total 10.01 last 101.12) - open/close 1865 (time 1710101653.89 total 20.03 last 85.14) - open/close 2696 (time 1710101663.90 total 30.03 last 83.03) - open/close 3500 (time 1710101673.91 total 40.04 last 80.33) - open/close 4205 (time 1710101683.92 total 50.05 last 70.44) - open/close 4976 (time 1710101693.92 total 60.05 last 77.09) - open/close 5624 (time 1710101703.92 total 70.06 last 64.79) - open/close 6306 (time 1710101713.92 total 80.06 last 68.18) - open/close 6942 (time 1710101723.92 total 90.06 last 63.59) - open/close 7434 (time 1710101733.93 total 100.06 last 49.20) - open/close 8046 (time 1710101743.94 total 110.07 last 61.12) - open/close 8262 (time 1710101758.76 total 124.90 last 14.57) - open/close 8791 (time 1710101768.78 total 134.91 last 52.83) - open/close 9360 (time 1710101778.78 total 144.92 last 56.87) - open/close 9884 (time 1710101788.79 total 154.93 last 52.34) total: 10000 open/close in 157.52 seconds: 63.48 ops/second - unlinked 0 (time 1710101793 ; total 0 ; last 0) total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second - open/close 983 (time 1710101899.87 total 10.03 last 98.01) - open/close 1821 (time 1710101909.87 total 20.03 last 83.78) - open/close 2586 (time 1710101919.87 total 30.03 last 76.49) - open/close 3335 (time 1710101929.88 total 40.04 last 74.83) - open/close 4062 (time 1710101939.88 total 50.05 last 72.67) - open/close 4715 (time 1710101949.89 total 60.05 last 65.27) - open/close 5412 (time 1710101959.90 total 70.06 last 69.63) - open/close 6114 (time 1710101969.91 total 80.07 last 70.13) - open/close 6748 (time 1710101979.93 total 90.09 last 63.28) - open/close 7373 (time 1710101989.93 total 100.09 last 62.48) - open/close 7963 (time 1710101999.93 total 110.09 last 58.99) - open/close 8547 (time 1710102009.95 total 120.11 last 58.30) - open/close 9058 (time 1710102019.96 total 130.12 last 51.08) - open/close 9607 (time 1710102029.97 total 140.13 last 54.84) total: 10000 open/close in 147.50 seconds: 67.80 ops/second - unlinked 0 (time 1710102039 ; total 0 ; last 0) total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second - open/close 1020 (time 1710102145.89 total 10.01 last 101.92) - open/close 1995 (time 1710102155.90 total 20.02 last 97.42) - open/close 2866 (time 1710102165.92 total 30.03 last 86.94) - open/close 3798 (time 1710102175.92 total 40.03 last 93.20) - open/close 4512 (time 1710102185.92 total 50.03 last 71.40) - open/close 5193 (time 1710102195.92 total 60.03 last 68.10) - open/close 5834 (time 1710102205.92 total 70.04 last 64.09) - open/close 6510 (time 1710102215.92 total 80.04 last 67.60) - open/close 7133 (time 1710102225.93 total 90.05 last 62.22) - open/close 7724 (time 1710102235.95 total 100.06 last 59.03) - open/close 8289 (time 1710102245.95 total 110.07 last 56.48) - open/close 8882 (time 1710102255.95 total 120.07 last 59.27) - open/close 9422 (time 1710102265.96 total 130.07 last 53.98) - open/close 9944 (time 1710102275.97 total 140.08 last 52.14) total: 10000 open/close in 141.16 seconds: 70.84 ops/second - unlinked 0 (time 1710102279 ; total 0 ; last 0) total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second - open/close 1072 (time 1710102388.36 total 10.01 last 107.14) - open/close 1950 (time 1710102398.37 total 20.02 last 87.70) - open/close 2745 (time 1710102408.37 total 30.02 last 79.48) - open/close 3488 (time 1710102418.38 total 40.03 last 74.19) - open/close 4231 (time 1710102428.40 total 50.05 last 74.21) - open/close 4970 (time 1710102438.41 total 60.06 last 73.82) - open/close 5650 (time 1710102448.41 total 70.06 last 67.95) - open/close 6345 (time 1710102458.42 total 80.08 last 69.42) - open/close 7004 (time 1710102468.43 total 90.08 last 65.84) - open/close 7678 (time 1710102478.44 total 100.09 last 67.33) - open/close 8305 (time 1710102488.45 total 110.10 last 62.68) - open/close 8856 (time 1710102498.46 total 120.11 last 55.04) - open/close 9382 (time 1710102508.47 total 130.12 last 52.52) - open/close 9929 (time 1710102518.49 total 140.14 last 54.60) total: 10000 open/close in 141.35 seconds: 70.75 ops/second - unlinked 0 (time 1710102521 ; total 0 ; last 0) total: 10000 unlinks in 117 seconds: 85.470085 unlinks/second - open/close 1079 (time 1710102649.96 total 10.00 last 107.85) - open/close 2060 (time 1710102659.97 total 20.02 last 97.98) - open/close 2942 (time 1710102669.98 total 30.02 last 88.13) - open/close 3911 (time 1710102679.98 total 40.03 last 96.88) - open/close 4753 (time 1710102689.98 total 50.03 last 84.17) - open/close 5550 (time 1710102699.99 total 60.03 last 79.67) - open/close 6298 (time 1710102709.99 total 70.04 last 74.79) - open/close 6969 (time 1710102720.00 total 80.04 last 67.04) - open/close 7658 (time 1710102730.03 total 90.08 last 68.69) - open/close 8326 (time 1710102740.09 total 100.14 last 66.40) - open/close 8964 (time 1710102750.13 total 110.18 last 63.51) - open/close 9572 (time 1710102760.14 total 120.19 last 60.77) total: 10000 open/close in 127.39 seconds: 78.50 ops/second - unlinked 0 (time 1710102769 ; total 1 ; last 1) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 1203 (time 1710102855.92 total 10.03 last 119.99) - open/close 2305 (time 1710102865.94 total 20.04 last 110.07) - open/close 3402 (time 1710102875.95 total 30.05 last 109.60) - open/close 4449 (time 1710102885.95 total 40.05 last 104.61) - open/close 5491 (time 1710102895.97 total 50.07 last 104.08) - open/close 6585 (time 1710102905.97 total 60.07 last 109.34) - open/close 7594 (time 1710102915.98 total 70.08 last 100.86) - open/close 8539 (time 1710102925.98 total 80.08 last 94.46) - open/close 9446 (time 1710102935.99 total 90.09 last 90.65) total: 10000 open/close in 96.25 seconds: 103.90 ops/second - unlinked 0 (time 1710102943 ; total 0 ; last 0) total: 10000 unlinks in 50 seconds: 200.000000 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg424-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg424-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg424-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.124@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.124@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg424-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-OST0000 oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg424-server: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg424-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec mount lustre on /mnt/lustre..... Starting client: oleg424-client.virtnet: -o user_xattr,flock oleg424-server@tcp:/lustre /mnt/lustre On OST0, 17709 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg424-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg424-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg424-server stop mds service on oleg424-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg424-server unloading modules on: 'oleg424-server' oleg424-server: oleg424-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg424-client: oleg424-client: ssh exited with exit code 2 pdsh@oleg424-client: oleg424-server: ssh exited with exit code 2 pdsh@oleg424-client: oleg424-server: ssh exited with exit code 2 pdsh@oleg424-client: oleg424-client: ssh exited with exit code 2