== conf-sanity test 69: replace an OST with the same index ========================================================== 16:00:06 (1710100806) start mds service on oleg210-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg210-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0000 oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg210-client.virtnet: -o user_xattr,flock oleg210-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99936. rc=0 - open/close 1355 (time 1710100833.74 total 10.01 last 135.35) - open/close 2783 (time 1710100843.74 total 20.01 last 142.78) - open/close 4033 (time 1710100853.75 total 30.02 last 124.96) - open/close 5094 (time 1710100863.75 total 40.02 last 106.05) - open/close 6149 (time 1710100873.76 total 50.03 last 105.44) - open/close 7148 (time 1710100883.77 total 60.04 last 99.79) - open/close 8103 (time 1710100893.77 total 70.04 last 95.50) - open/close 9006 (time 1710100903.78 total 80.04 last 90.23) - open/close 9831 (time 1710100913.79 total 90.06 last 82.40) total: 10000 open/close in 93.03 seconds: 107.49 ops/second - unlinked 0 (time 1710100918 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 1437 (time 1710101001.79 total 10.00 last 143.67) - open/close 2602 (time 1710101011.80 total 20.01 last 116.40) - open/close 3573 (time 1710101021.81 total 30.02 last 97.01) - open/close 4664 (time 1710101031.81 total 40.02 last 109.09) - open/close 5702 (time 1710101041.82 total 50.03 last 103.71) - open/close 6570 (time 1710101051.83 total 60.04 last 86.73) - open/close 7393 (time 1710101061.84 total 70.05 last 82.22) - open/close 8106 (time 1710101071.84 total 80.05 last 71.26) - open/close 8746 (time 1710101081.85 total 90.05 last 63.99) - open/close 9400 (time 1710101091.88 total 100.08 last 65.20) total: 10000 open/close in 109.31 seconds: 91.48 ops/second - unlinked 0 (time 1710101103 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 959 (time 1710101200.89 total 10.00 last 95.89) - open/close 1899 (time 1710101210.90 total 20.01 last 93.93) - open/close 2769 (time 1710101220.91 total 30.02 last 86.90) - open/close 3523 (time 1710101230.91 total 40.02 last 75.40) - open/close 4270 (time 1710101240.91 total 50.02 last 74.69) - open/close 5025 (time 1710101250.92 total 60.03 last 75.44) - open/close 5732 (time 1710101260.93 total 70.04 last 70.63) - open/close 6374 (time 1710101270.96 total 80.08 last 63.97) - open/close 7025 (time 1710101280.98 total 90.09 last 65.01) - open/close 7634 (time 1710101291.06 total 100.18 last 60.38) - open/close 8240 (time 1710101301.07 total 110.19 last 60.54) - open/close 8828 (time 1710101311.08 total 120.20 last 58.74) - open/close 9393 (time 1710101321.11 total 130.22 last 56.34) total: 10000 open/close in 140.09 seconds: 71.38 ops/second - unlinked 0 (time 1710101333 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 1043 (time 1710101432.37 total 10.00 last 104.27) - open/close 2023 (time 1710101442.38 total 20.01 last 97.91) - open/close 2903 (time 1710101452.38 total 30.02 last 87.96) - open/close 3683 (time 1710101462.39 total 40.02 last 77.94) - open/close 4502 (time 1710101472.40 total 50.03 last 81.83) - open/close 5221 (time 1710101482.42 total 60.05 last 71.76) - open/close 5910 (time 1710101492.42 total 70.06 last 68.86) - open/close 6554 (time 1710101502.44 total 80.07 last 64.32) - open/close 7195 (time 1710101512.44 total 90.08 last 64.06) - open/close 7800 (time 1710101522.48 total 100.12 last 60.27) - open/close 8415 (time 1710101532.48 total 110.12 last 61.48) - open/close 8989 (time 1710101542.49 total 120.12 last 57.38) - open/close 9586 (time 1710101552.50 total 130.13 last 59.63) total: 10000 open/close in 137.51 seconds: 72.72 ops/second - unlinked 0 (time 1710101562 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 1117 (time 1710101660.50 total 10.04 last 111.21) - open/close 2096 (time 1710101670.50 total 20.05 last 97.86) - open/close 2996 (time 1710101680.51 total 30.05 last 89.95) - open/close 3845 (time 1710101690.53 total 40.08 last 84.71) - open/close 4725 (time 1710101700.53 total 50.08 last 87.97) - open/close 5512 (time 1710101710.53 total 60.08 last 78.70) - open/close 6292 (time 1710101720.54 total 70.09 last 77.95) - open/close 7016 (time 1710101730.55 total 80.10 last 72.32) - open/close 7705 (time 1710101740.57 total 90.11 last 68.79) - open/close 8300 (time 1710101750.58 total 100.12 last 59.45) - open/close 8940 (time 1710101760.59 total 110.13 last 63.94) - open/close 9543 (time 1710101770.59 total 120.14 last 60.26) total: 10000 open/close in 127.34 seconds: 78.53 ops/second - unlinked 0 (time 1710101779 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 1017 (time 1710101874.48 total 10.01 last 101.57) - open/close 2005 (time 1710101884.50 total 20.03 last 98.63) - open/close 2914 (time 1710101894.51 total 30.04 last 90.81) - open/close 3835 (time 1710101904.52 total 40.05 last 92.03) - open/close 4630 (time 1710101914.52 total 50.05 last 79.49) - open/close 5403 (time 1710101924.52 total 60.05 last 77.28) - open/close 6100 (time 1710101934.54 total 70.07 last 69.58) - open/close 6722 (time 1710101944.54 total 80.07 last 62.18) - open/close 7339 (time 1710101954.54 total 90.07 last 61.68) - open/close 7949 (time 1710101964.55 total 100.08 last 60.95) - open/close 8598 (time 1710101974.56 total 110.09 last 64.87) - open/close 9212 (time 1710101984.57 total 120.10 last 61.33) - open/close 9861 (time 1710101994.58 total 130.11 last 64.80) total: 10000 open/close in 132.48 seconds: 75.48 ops/second - unlinked 0 (time 1710101998 ; total 0 ; last 0) total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second - open/close 1083 (time 1710102106.12 total 10.01 last 108.19) - open/close 2040 (time 1710102116.13 total 20.02 last 95.60) - open/close 2928 (time 1710102126.13 total 30.02 last 88.79) - open/close 3834 (time 1710102136.14 total 40.03 last 90.56) - open/close 4627 (time 1710102146.15 total 50.03 last 79.23) - open/close 5449 (time 1710102156.15 total 60.04 last 82.14) - open/close 6137 (time 1710102166.15 total 70.04 last 68.79) - open/close 6874 (time 1710102176.17 total 80.06 last 73.60) - open/close 7560 (time 1710102186.17 total 90.06 last 68.60) - open/close 8190 (time 1710102196.18 total 100.06 last 62.96) - open/close 8834 (time 1710102206.19 total 110.08 last 64.31) - open/close 9422 (time 1710102216.20 total 120.09 last 58.75) total: 10000 open/close in 129.89 seconds: 76.99 ops/second - unlinked 0 (time 1710102227 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 1155 (time 1710102322.55 total 10.00 last 115.48) - open/close 2114 (time 1710102332.55 total 20.01 last 95.86) - open/close 3065 (time 1710102342.56 total 30.01 last 95.07) - open/close 4016 (time 1710102352.56 total 40.01 last 95.06) - open/close 4842 (time 1710102362.57 total 50.02 last 82.51) - open/close 5663 (time 1710102372.58 total 60.03 last 82.03) - open/close 6478 (time 1710102382.59 total 70.04 last 81.43) - open/close 7165 (time 1710102392.59 total 80.04 last 68.69) - open/close 7811 (time 1710102402.60 total 90.05 last 64.55) - open/close 8496 (time 1710102412.62 total 100.08 last 68.32) - open/close 9118 (time 1710102422.63 total 110.08 last 62.20) - open/close 9784 (time 1710102432.63 total 120.08 last 66.57) total: 10000 open/close in 123.62 seconds: 80.90 ops/second - unlinked 0 (time 1710102437 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 1058 (time 1710102528.22 total 10.00 last 105.76) - open/close 2052 (time 1710102538.23 total 20.01 last 99.34) - open/close 3061 (time 1710102548.23 total 30.01 last 100.88) - open/close 3965 (time 1710102558.24 total 40.02 last 90.32) - open/close 4773 (time 1710102568.24 total 50.03 last 80.75) - open/close 5560 (time 1710102578.25 total 60.04 last 78.61) - open/close 6264 (time 1710102588.27 total 70.06 last 70.27) - open/close 6934 (time 1710102598.28 total 80.06 last 66.94) - open/close 7657 (time 1710102608.29 total 90.07 last 72.26) - open/close 8323 (time 1710102618.34 total 100.12 last 66.26) - open/close 8973 (time 1710102628.34 total 110.12 last 65.00) - open/close 9570 (time 1710102638.35 total 120.13 last 59.63) total: 10000 open/close in 126.61 seconds: 78.98 ops/second - unlinked 0 (time 1710102646 ; total 0 ; last 0) total: 10000 unlinks in 102 seconds: 98.039215 unlinks/second - open/close 1216 (time 1710102760.61 total 10.01 last 121.53) - open/close 2367 (time 1710102770.61 total 20.01 last 115.08) - open/close 3466 (time 1710102780.61 total 30.01 last 109.90) - open/close 4468 (time 1710102790.62 total 40.02 last 100.12) - open/close 5355 (time 1710102800.63 total 50.02 last 88.63) - open/close 6280 (time 1710102810.63 total 60.03 last 92.47) - open/close 7201 (time 1710102820.64 total 70.03 last 92.03) - open/close 8074 (time 1710102830.64 total 80.04 last 87.25) - open/close 8914 (time 1710102840.65 total 90.05 last 83.94) - open/close 9521 (time 1710102858.73 total 108.12 last 33.58) total: 10000 open/close in 113.97 seconds: 87.74 ops/second - unlinked 0 (time 1710102866 ; total 0 ; last 0) total: 10000 unlinks in 59 seconds: 169.491531 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg210-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg210-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg210-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.110@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.110@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg210-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0000 oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg210-server: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg210-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 14 sec mount lustre on /mnt/lustre..... Starting client: oleg210-client.virtnet: -o user_xattr,flock oleg210-server@tcp:/lustre /mnt/lustre On OST0, 14877 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg210-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg210-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg210-server stop mds service on oleg210-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg210-server unloading modules on: 'oleg210-server' oleg210-server: oleg210-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg210-client: oleg210-client: ssh exited with exit code 2 pdsh@oleg210-client: oleg210-server: ssh exited with exit code 2 pdsh@oleg210-client: oleg210-client: ssh exited with exit code 2 pdsh@oleg210-client: oleg210-server: ssh exited with exit code 2