== conf-sanity test 69: replace an OST with the same index ========================================================== 16:04:01 (1710101041) start mds service on oleg403-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg403-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg403-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Started lustre-OST0000 oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg403-client.virtnet: -o user_xattr,flock oleg403-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49440 inodes available. Want 99872. rc=0 - open/close 1076 (time 1710101089.15 total 10.00 last 107.55) - open/close 2022 (time 1710101099.15 total 20.01 last 94.56) - open/close 3016 (time 1710101109.15 total 30.01 last 99.37) - open/close 3902 (time 1710101119.16 total 40.01 last 88.57) - open/close 4830 (time 1710101129.16 total 50.02 last 92.78) - open/close 5565 (time 1710101139.17 total 60.02 last 73.45) - open/close 6371 (time 1710101149.17 total 70.03 last 80.55) - open/close 7026 (time 1710101159.18 total 80.04 last 65.44) - open/close 7733 (time 1710101169.20 total 90.05 last 70.61) - open/close 8338 (time 1710101179.21 total 100.07 last 60.42) - open/close 8920 (time 1710101189.21 total 110.07 last 58.19) - open/close 9543 (time 1710101199.21 total 120.07 last 62.30) - open/close 9676 (time 1710101212.32 total 133.18 last 10.14) total: 10000 open/close in 138.72 seconds: 72.09 ops/second - unlinked 0 (time 1710101220 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 1092 (time 1710101318.59 total 10.01 last 109.12) - open/close 2081 (time 1710101328.59 total 20.01 last 98.86) - open/close 3033 (time 1710101338.60 total 30.02 last 95.13) - open/close 3930 (time 1710101348.61 total 40.02 last 89.65) - open/close 4771 (time 1710101358.61 total 50.03 last 84.04) - open/close 5562 (time 1710101368.62 total 60.04 last 79.06) - open/close 6264 (time 1710101378.63 total 70.04 last 70.15) - open/close 7016 (time 1710101388.63 total 80.05 last 75.13) - open/close 7738 (time 1710101398.64 total 90.06 last 72.13) - open/close 8342 (time 1710101408.66 total 100.08 last 60.29) - open/close 8936 (time 1710101418.67 total 110.09 last 59.34) - open/close 9582 (time 1710101428.69 total 120.11 last 64.47) total: 10000 open/close in 127.03 seconds: 78.72 ops/second - unlinked 0 (time 1710101437 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 1092 (time 1710101531.95 total 10.00 last 109.17) - open/close 2105 (time 1710101541.96 total 20.01 last 101.24) - open/close 2939 (time 1710101551.96 total 30.01 last 83.38) - open/close 3845 (time 1710101561.97 total 40.02 last 90.51) - open/close 4673 (time 1710101572.02 total 50.07 last 82.42) - open/close 5498 (time 1710101582.03 total 60.08 last 82.41) - open/close 6305 (time 1710101592.04 total 70.09 last 80.62) - open/close 7043 (time 1710101602.05 total 80.11 last 73.67) - open/close 7723 (time 1710101612.06 total 90.12 last 67.94) - open/close 8362 (time 1710101632.00 total 110.05 last 32.06) - open/close 8996 (time 1710101642.00 total 120.05 last 63.40) - open/close 9570 (time 1710101652.00 total 130.05 last 57.37) total: 10000 open/close in 137.62 seconds: 72.67 ops/second - unlinked 0 (time 1710101661 ; total 0 ; last 0) total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second - open/close 1125 (time 1710101768.08 total 10.00 last 112.46) - open/close 2145 (time 1710101778.08 total 20.01 last 101.96) - open/close 3069 (time 1710101788.08 total 30.01 last 92.39) - open/close 3885 (time 1710101798.10 total 40.03 last 81.40) - open/close 4647 (time 1710101808.11 total 50.04 last 76.17) - open/close 5205 (time 1710101825.99 total 67.92 last 31.20) - open/close 5878 (time 1710101836.00 total 77.93 last 67.25) - open/close 6659 (time 1710101846.00 total 87.93 last 78.09) - open/close 7381 (time 1710101856.00 total 97.93 last 72.17) - open/close 8078 (time 1710101866.03 total 107.95 last 69.54) - open/close 8823 (time 1710101876.05 total 117.98 last 74.32) - open/close 9422 (time 1710101886.05 total 127.98 last 59.90) - open/close 9994 (time 1710101896.05 total 137.98 last 57.18) total: 10000 open/close in 138.04 seconds: 72.44 ops/second - unlinked 0 (time 1710101898 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 1068 (time 1710101993.55 total 10.03 last 106.50) - open/close 2129 (time 1710102003.56 total 20.03 last 106.03) - open/close 3096 (time 1710102013.56 total 30.04 last 96.68) - open/close 4011 (time 1710102023.56 total 40.04 last 91.48) - open/close 4855 (time 1710102033.57 total 50.05 last 84.34) - open/close 5626 (time 1710102043.58 total 60.06 last 76.97) - open/close 6350 (time 1710102053.60 total 70.07 last 72.32) - open/close 7040 (time 1710102063.60 total 80.08 last 68.96) - open/close 7728 (time 1710102073.62 total 90.10 last 68.70) - open/close 8373 (time 1710102083.63 total 100.10 last 64.44) - open/close 9034 (time 1710102093.63 total 110.11 last 66.05) - open/close 9660 (time 1710102103.64 total 120.12 last 62.53) total: 10000 open/close in 125.87 seconds: 79.45 ops/second - unlinked 0 (time 1710102111 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 1107 (time 1710102201.44 total 10.00 last 110.68) - open/close 2166 (time 1710102211.45 total 20.01 last 105.83) - open/close 3118 (time 1710102221.45 total 30.01 last 95.15) - open/close 3997 (time 1710102231.47 total 40.03 last 87.77) - open/close 4787 (time 1710102241.47 total 50.03 last 78.99) - open/close 5594 (time 1710102251.48 total 60.04 last 80.64) - open/close 6411 (time 1710102261.48 total 70.05 last 81.63) - open/close 7138 (time 1710102271.49 total 80.05 last 72.68) - open/close 7249 (time 1710102284.52 total 93.09 last 8.51) - open/close 7898 (time 1710102294.53 total 103.09 last 64.85) - open/close 8581 (time 1710102304.54 total 113.11 last 68.21) - open/close 9187 (time 1710102314.56 total 123.12 last 60.52) - open/close 9815 (time 1710102324.57 total 133.14 last 62.70) total: 10000 open/close in 135.94 seconds: 73.56 ops/second - unlinked 0 (time 1710102329 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 1154 (time 1710102422.00 total 10.00 last 115.37) - open/close 2193 (time 1710102432.01 total 20.01 last 103.82) - open/close 3127 (time 1710102442.01 total 30.01 last 93.38) - open/close 4011 (time 1710102452.01 total 40.01 last 88.38) - open/close 4901 (time 1710102462.02 total 50.02 last 88.94) - open/close 5750 (time 1710102472.02 total 60.02 last 84.89) - open/close 6610 (time 1710102482.03 total 70.03 last 85.94) - open/close 7382 (time 1710102492.04 total 80.04 last 77.14) - open/close 7961 (time 1710102502.05 total 90.05 last 57.84) - open/close 8690 (time 1710102512.06 total 100.06 last 72.84) - open/close 9339 (time 1710102522.08 total 110.08 last 64.75) - open/close 9934 (time 1710102532.09 total 120.09 last 59.44) total: 10000 open/close in 121.18 seconds: 82.52 ops/second - unlinked 0 (time 1710102535 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 1165 (time 1710102627.87 total 10.00 last 116.50) - open/close 2206 (time 1710102637.88 total 20.01 last 104.04) - open/close 3214 (time 1710102647.88 total 30.01 last 100.77) - open/close 4181 (time 1710102657.88 total 40.01 last 96.69) - open/close 5061 (time 1710102667.88 total 50.01 last 87.99) - open/close 5182 (time 1710102680.28 total 62.41 last 9.76) - open/close 6021 (time 1710102690.30 total 72.42 last 83.75) - open/close 6865 (time 1710102700.30 total 82.43 last 84.39) - open/close 7653 (time 1710102710.31 total 92.44 last 78.71) - open/close 8383 (time 1710102720.31 total 102.44 last 72.99) - open/close 9145 (time 1710102730.32 total 112.45 last 76.10) - open/close 9847 (time 1710102740.34 total 122.46 last 70.11) total: 10000 open/close in 124.70 seconds: 80.19 ops/second - unlinked 0 (time 1710102744 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 1339 (time 1710102825.88 total 10.01 last 133.78) - open/close 2589 (time 1710102835.88 total 20.01 last 124.96) - open/close 3651 (time 1710102845.89 total 30.02 last 106.14) - open/close 4638 (time 1710102855.89 total 40.02 last 98.68) - open/close 5585 (time 1710102865.89 total 50.02 last 94.67) - open/close 5960 (time 1710102881.09 total 65.22 last 24.68) - open/close 6910 (time 1710102891.10 total 75.23 last 94.90) - open/close 7962 (time 1710102901.10 total 85.23 last 105.16) - open/close 8991 (time 1710102911.11 total 95.24 last 102.85) - open/close 9933 (time 1710102921.11 total 105.24 last 94.17) total: 10000 open/close in 105.98 seconds: 94.36 ops/second - unlinked 0 (time 1710102923 ; total 0 ; last 0) total: 10000 unlinks in 51 seconds: 196.078430 unlinks/second - open/close 1906 (time 1710102985.22 total 10.00 last 190.55) - open/close 3575 (time 1710102995.23 total 20.00 last 166.87) - open/close 5141 (time 1710103005.23 total 30.00 last 156.60) - open/close 6509 (time 1710103015.23 total 40.00 last 136.80) - open/close 7861 (time 1710103025.23 total 50.01 last 135.12) - open/close 9172 (time 1710103035.24 total 60.02 last 131.01) total: 10000 open/close in 66.59 seconds: 150.16 ops/second - unlinked 0 (time 1710103042 ; total 0 ; last 0) total: 10000 unlinks in 36 seconds: 277.777771 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg403-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg403-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg403-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.103@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg403-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Started lustre-OST0000 oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg403-server: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg403-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 16 sec oleg403-server: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg403-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg403-client.virtnet: -o user_xattr,flock oleg403-server@tcp:/lustre /mnt/lustre On OST0, 17717 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg403-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg403-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg403-server stop mds service on oleg403-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg403-server stop mds service on oleg403-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg403-server unloading modules on: 'oleg403-server' oleg403-server: oleg403-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg403-client: oleg403-client: ssh exited with exit code 2 pdsh@oleg403-client: oleg403-server: ssh exited with exit code 2 pdsh@oleg403-client: oleg403-client: ssh exited with exit code 2 pdsh@oleg403-client: oleg403-server: ssh exited with exit code 2