== conf-sanity test 69: replace an OST with the same index ========================================================== 16:00:32 (1710100832) start mds service on oleg150-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg150-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-OST0000 oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg150-client.virtnet: -o user_xattr,flock oleg150-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49602 inodes available. Want 99872. rc=0 - open/close 1309 (time 1710100861.80 total 10.00 last 130.89) - open/close 2272 (time 1710100871.81 total 20.01 last 96.24) - open/close 3119 (time 1710100881.83 total 30.03 last 84.55) - open/close 4046 (time 1710100891.84 total 40.04 last 92.61) - open/close 4859 (time 1710100901.85 total 50.05 last 81.17) - open/close 5591 (time 1710100911.86 total 60.06 last 73.14) - open/close 6329 (time 1710100921.86 total 70.06 last 73.79) - open/close 7061 (time 1710100931.87 total 80.07 last 73.15) - open/close 7797 (time 1710100941.88 total 90.08 last 73.49) - open/close 8513 (time 1710100951.89 total 100.09 last 71.54) - open/close 9193 (time 1710100961.91 total 110.11 last 67.86) - open/close 9805 (time 1710100971.93 total 120.13 last 61.07) total: 10000 open/close in 123.18 seconds: 81.18 ops/second - unlinked 0 (time 1710100976 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 1176 (time 1710101073.49 total 10.01 last 117.53) - open/close 2118 (time 1710101083.49 total 20.01 last 94.19) - open/close 3068 (time 1710101093.49 total 30.01 last 94.97) - open/close 3984 (time 1710101103.50 total 40.02 last 91.55) - open/close 4802 (time 1710101113.50 total 50.02 last 81.78) - open/close 5535 (time 1710101123.51 total 60.03 last 73.25) - open/close 6105 (time 1710101133.51 total 70.03 last 56.98) - open/close 6682 (time 1710101143.52 total 80.04 last 57.66) - open/close 7292 (time 1710101153.53 total 90.05 last 60.93) - open/close 7889 (time 1710101163.54 total 100.06 last 59.63) - open/close 8448 (time 1710101173.56 total 110.08 last 55.78) - open/close 8992 (time 1710101183.59 total 120.11 last 54.26) - open/close 9544 (time 1710101193.60 total 130.11 last 55.16) total: 10000 open/close in 139.48 seconds: 71.69 ops/second - unlinked 0 (time 1710101205 ; total 0 ; last 0) total: 10000 unlinks in 108 seconds: 92.592590 unlinks/second - open/close 828 (time 1710101325.68 total 10.05 last 82.39) - open/close 1665 (time 1710101335.69 total 20.06 last 83.65) - open/close 2421 (time 1710101345.69 total 30.06 last 75.56) - open/close 3096 (time 1710101355.71 total 40.08 last 67.38) - open/close 3776 (time 1710101365.72 total 50.09 last 67.95) - open/close 4407 (time 1710101375.72 total 60.09 last 63.08) - open/close 5044 (time 1710101385.72 total 70.09 last 63.67) - open/close 5658 (time 1710101395.72 total 80.09 last 61.40) - open/close 6222 (time 1710101405.73 total 90.10 last 56.38) - open/close 6800 (time 1710101415.73 total 100.10 last 57.80) - open/close 7321 (time 1710101425.74 total 110.11 last 52.04) - open/close 7868 (time 1710101435.76 total 120.13 last 54.61) - open/close 8385 (time 1710101445.77 total 130.14 last 51.61) - open/close 8872 (time 1710101455.78 total 140.15 last 48.66) - open/close 9307 (time 1710101465.79 total 150.15 last 43.48) - open/close 9720 (time 1710101475.81 total 160.18 last 41.21) total: 10000 open/close in 167.06 seconds: 59.86 ops/second - unlinked 0 (time 1710101485 ; total 0 ; last 0) total: 10000 unlinks in 115 seconds: 86.956520 unlinks/second - open/close 848 (time 1710101613.28 total 10.00 last 84.80) - open/close 1679 (time 1710101623.29 total 20.01 last 83.04) - open/close 2436 (time 1710101633.30 total 30.02 last 75.64) - open/close 3145 (time 1710101643.30 total 40.02 last 70.85) - open/close 3768 (time 1710101653.31 total 50.03 last 62.25) - open/close 4379 (time 1710101663.35 total 60.07 last 60.87) - open/close 4948 (time 1710101673.35 total 70.07 last 56.89) - open/close 5138 (time 1710101687.23 total 83.95 last 13.69) - open/close 5791 (time 1710101697.23 total 93.95 last 65.26) - open/close 6407 (time 1710101707.25 total 103.97 last 61.51) - open/close 6932 (time 1710101717.26 total 113.98 last 52.46) - open/close 7425 (time 1710101727.26 total 123.98 last 49.28) - open/close 7937 (time 1710101737.28 total 134.00 last 51.08) - open/close 8469 (time 1710101747.29 total 144.01 last 53.15) - open/close 8957 (time 1710101757.30 total 154.02 last 48.77) - open/close 9460 (time 1710101767.30 total 164.02 last 50.27) - open/close 9929 (time 1710101777.34 total 174.06 last 46.73) total: 10000 open/close in 175.39 seconds: 57.01 ops/second - unlinked 0 (time 1710101780 ; total 0 ; last 0) total: 10000 unlinks in 118 seconds: 84.745766 unlinks/second - open/close 852 (time 1710101921.39 total 20.72 last 41.12) - open/close 1668 (time 1710101931.39 total 30.72 last 81.57) - open/close 2410 (time 1710101941.41 total 40.74 last 74.09) - open/close 3170 (time 1710101951.42 total 50.75 last 75.93) - open/close 3670 (time 1710101969.22 total 68.55 last 28.08) - open/close 4347 (time 1710101979.23 total 78.56 last 67.66) - open/close 5019 (time 1710101989.24 total 88.57 last 67.15) - open/close 5687 (time 1710101999.24 total 98.57 last 66.76) - open/close 6276 (time 1710102009.25 total 108.59 last 58.83) - open/close 6821 (time 1710102019.26 total 118.59 last 54.47) - open/close 7311 (time 1710102029.26 total 128.59 last 48.99) - open/close 7856 (time 1710102039.27 total 138.60 last 54.47) - open/close 8477 (time 1710102049.27 total 148.60 last 62.09) - open/close 8983 (time 1710102059.27 total 158.61 last 50.57) - open/close 9535 (time 1710102069.29 total 168.62 last 55.13) total: 10000 open/close in 178.45 seconds: 56.04 ops/second - unlinked 0 (time 1710102081 ; total 0 ; last 0) total: 10000 unlinks in 128 seconds: 78.125000 unlinks/second - open/close 331 (time 1710102227.80 total 15.25 last 21.71) - open/close 1210 (time 1710102237.80 total 25.25 last 87.90) - open/close 1969 (time 1710102247.81 total 35.25 last 75.85) - open/close 2700 (time 1710102257.81 total 45.26 last 73.05) - open/close 3414 (time 1710102267.81 total 55.26 last 71.40) - open/close 3654 (time 1710102282.36 total 69.80 last 16.50) - open/close 4417 (time 1710102292.36 total 79.81 last 76.23) - open/close 5091 (time 1710102302.37 total 89.82 last 67.38) - open/close 5745 (time 1710102312.37 total 99.82 last 65.40) - open/close 6389 (time 1710102322.39 total 109.84 last 64.27) - open/close 6944 (time 1710102332.39 total 119.84 last 55.46) - open/close 7475 (time 1710102342.40 total 129.85 last 53.05) - open/close 8010 (time 1710102352.42 total 139.86 last 53.43) - open/close 8552 (time 1710102362.43 total 149.87 last 54.15) - open/close 9121 (time 1710102372.45 total 159.90 last 56.75) - open/close 9658 (time 1710102382.46 total 169.91 last 53.64) total: 10000 open/close in 177.84 seconds: 56.23 ops/second - unlinked 0 (time 1710102393 ; total 0 ; last 0) total: 10000 unlinks in 102 seconds: 98.039215 unlinks/second - open/close 935 (time 1710102507.94 total 10.01 last 93.40) - open/close 1754 (time 1710102517.96 total 20.02 last 81.79) - open/close 2597 (time 1710102527.97 total 30.03 last 84.23) - open/close 3352 (time 1710102537.97 total 40.04 last 75.44) - open/close 4002 (time 1710102547.98 total 50.04 last 65.00) - open/close 4649 (time 1710102557.98 total 60.05 last 64.65) - open/close 5268 (time 1710102567.98 total 70.05 last 61.89) - open/close 5882 (time 1710102577.99 total 80.06 last 61.34) - open/close 6473 (time 1710102588.01 total 90.07 last 59.03) - open/close 7026 (time 1710102598.01 total 100.08 last 55.28) - open/close 7581 (time 1710102608.02 total 110.08 last 55.47) - open/close 8098 (time 1710102618.02 total 120.08 last 51.70) - open/close 8614 (time 1710102628.03 total 130.09 last 51.55) - open/close 9142 (time 1710102638.04 total 140.10 last 52.74) - open/close 9639 (time 1710102648.06 total 150.12 last 49.61) total: 10000 open/close in 158.29 seconds: 63.17 ops/second - unlinked 0 (time 1710102659 ; total 0 ; last 0) total: 10000 unlinks in 106 seconds: 94.339622 unlinks/second - open/close 557 (time 1710102785.61 total 17.42 last 31.98) - open/close 1436 (time 1710102795.62 total 27.42 last 87.85) - open/close 2211 (time 1710102805.63 total 37.44 last 77.38) - open/close 2587 (time 1710102821.56 total 53.37 last 23.60) - open/close 3315 (time 1710102831.58 total 63.39 last 72.68) - open/close 3956 (time 1710102841.59 total 73.40 last 64.04) - open/close 4573 (time 1710102851.60 total 83.40 last 61.67) - open/close 5166 (time 1710102861.60 total 93.40 last 59.28) - open/close 5758 (time 1710102871.61 total 103.41 last 59.15) - open/close 6303 (time 1710102881.61 total 113.42 last 54.47) - open/close 6840 (time 1710102891.62 total 123.42 last 53.68) - open/close 7352 (time 1710102901.62 total 133.43 last 51.17) - open/close 7852 (time 1710102911.64 total 143.45 last 49.91) - open/close 8399 (time 1710102921.65 total 153.45 last 54.67) - open/close 8894 (time 1710102931.65 total 163.45 last 49.48) - open/close 9392 (time 1710102941.65 total 173.46 last 49.80) - open/close 9867 (time 1710102951.67 total 183.47 last 47.43) total: 10000 open/close in 186.45 seconds: 53.63 ops/second - unlinked 0 (time 1710102957 ; total 0 ; last 0) total: 10000 unlinks in 104 seconds: 96.153847 unlinks/second - open/close 977 (time 1710103074.70 total 10.01 last 97.63) - open/close 1778 (time 1710103084.70 total 20.01 last 80.10) - open/close 2625 (time 1710103094.71 total 30.02 last 84.63) - open/close 3398 (time 1710103104.71 total 40.02 last 77.29) - open/close 4135 (time 1710103114.74 total 50.04 last 73.51) - open/close 4791 (time 1710103124.74 total 60.05 last 65.59) - open/close 5424 (time 1710103134.74 total 70.05 last 63.27) - open/close 5984 (time 1710103144.76 total 80.07 last 55.91) - open/close 6564 (time 1710103154.76 total 90.07 last 57.98) - open/close 7138 (time 1710103164.76 total 100.07 last 57.39) - open/close 7645 (time 1710103174.77 total 110.08 last 50.68) - open/close 8277 (time 1710103184.78 total 120.09 last 63.11) - open/close 8845 (time 1710103194.78 total 130.09 last 56.80) - open/close 9033 (time 1710103209.58 total 144.88 last 12.71) - open/close 9125 (time 1710103222.23 total 157.54 last 7.27) - open/close 9595 (time 1710103232.24 total 167.55 last 46.98) total: 10000 open/close in 175.98 seconds: 56.82 ops/second - unlinked 0 (time 1710103242 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 1104 (time 1710103342.56 total 10.00 last 110.39) - open/close 2136 (time 1710103352.57 total 20.01 last 103.12) - open/close 3034 (time 1710103362.58 total 30.01 last 89.77) - open/close 3946 (time 1710103372.58 total 40.01 last 91.19) - open/close 4783 (time 1710103382.58 total 50.02 last 83.67) - open/close 5592 (time 1710103392.58 total 60.02 last 80.87) - open/close 6379 (time 1710103402.59 total 70.03 last 78.62) - open/close 7058 (time 1710103412.60 total 80.04 last 67.84) - open/close 7822 (time 1710103422.61 total 90.04 last 76.38) - open/close 8514 (time 1710103432.61 total 100.04 last 69.18) - open/close 9238 (time 1710103442.61 total 110.05 last 72.35) - open/close 9921 (time 1710103452.63 total 120.06 last 68.22) total: 10000 open/close in 121.27 seconds: 82.46 ops/second - unlinked 0 (time 1710103455 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg150-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg150-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg150-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.150@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg150-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-OST0000 oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg150-server: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg150-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 13 sec mount lustre on /mnt/lustre..... Starting client: oleg150-client.virtnet: -o user_xattr,flock oleg150-server@tcp:/lustre /mnt/lustre On OST0, 20235 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg150-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg150-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg150-server stop mds service on oleg150-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg150-server unloading modules on: 'oleg150-server' oleg150-server: oleg150-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg150-client: oleg150-client: ssh exited with exit code 2 pdsh@oleg150-client: oleg150-server: ssh exited with exit code 2 pdsh@oleg150-client: oleg150-client: ssh exited with exit code 2 pdsh@oleg150-client: oleg150-server: ssh exited with exit code 2