== conf-sanity test 69: replace an OST with the same index ========================================================== 16:04:09 (1710101049) start mds service on oleg215-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg215-server: oleg215-server.virtnet: executing set_default_debug -1 all pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg215-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg215-server: oleg215-server.virtnet: executing set_default_debug -1 all pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg215-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg215-server: oleg215-server.virtnet: executing set_default_debug -1 all pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 Started lustre-OST0000 oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg215-client.virtnet: -o user_xattr,flock oleg215-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49440 inodes available. Want 99872. rc=0 - open/close 1023 (time 1710101088.62 total 10.00 last 102.26) - open/close 1995 (time 1710101098.63 total 20.01 last 97.17) - open/close 2922 (time 1710101108.63 total 30.02 last 92.63) - open/close 3735 (time 1710101118.65 total 40.03 last 81.20) - open/close 4483 (time 1710101128.65 total 50.04 last 74.74) - open/close 5237 (time 1710101138.66 total 60.04 last 75.34) - open/close 6015 (time 1710101148.67 total 70.06 last 77.70) - open/close 6695 (time 1710101158.68 total 80.06 last 67.96) - open/close 7361 (time 1710101168.69 total 90.07 last 66.54) - open/close 8011 (time 1710101178.71 total 100.09 last 64.88) - open/close 8607 (time 1710101188.71 total 110.09 last 59.58) - open/close 9181 (time 1710101198.73 total 120.11 last 57.31) - open/close 9733 (time 1710101208.75 total 130.13 last 55.10) total: 10000 open/close in 134.91 seconds: 74.12 ops/second - unlinked 0 (time 1710101215 ; total 0 ; last 0) total: 10000 unlinks in 106 seconds: 94.339622 unlinks/second - open/close 1025 (time 1710101333.26 total 10.00 last 102.47) - open/close 1902 (time 1710101343.26 total 20.01 last 87.68) - open/close 2685 (time 1710101353.27 total 30.01 last 78.23) - open/close 3416 (time 1710101363.28 total 40.02 last 73.03) - open/close 4195 (time 1710101373.29 total 50.03 last 77.86) - open/close 4960 (time 1710101383.31 total 60.05 last 76.36) - open/close 5673 (time 1710101393.31 total 70.05 last 71.28) - open/close 6327 (time 1710101403.32 total 80.06 last 65.35) - open/close 6971 (time 1710101413.33 total 90.07 last 64.35) - open/close 7604 (time 1710101423.34 total 100.08 last 63.24) - open/close 8192 (time 1710101433.35 total 110.09 last 58.71) - open/close 8774 (time 1710101443.35 total 120.09 last 58.19) - open/close 8882 (time 1710101456.19 total 132.93 last 8.42) - open/close 9406 (time 1710101466.19 total 142.93 last 52.38) - open/close 9903 (time 1710101476.22 total 152.96 last 49.55) total: 10000 open/close in 154.92 seconds: 64.55 ops/second - unlinked 0 (time 1710101480 ; total 0 ; last 0) total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second - open/close 970 (time 1710101587.08 total 10.00 last 96.96) - open/close 1884 (time 1710101597.08 total 20.01 last 91.36) - open/close 2717 (time 1710101607.09 total 30.02 last 83.21) - open/close 3611 (time 1710101617.10 total 40.03 last 89.35) - open/close 4303 (time 1710101627.10 total 50.03 last 69.17) - open/close 5079 (time 1710101637.11 total 60.04 last 77.50) - open/close 5727 (time 1710101647.12 total 70.04 last 64.79) - open/close 6401 (time 1710101657.12 total 80.05 last 67.37) - open/close 7097 (time 1710101667.14 total 90.06 last 69.50) - open/close 7746 (time 1710101677.14 total 100.06 last 64.90) - open/close 8368 (time 1710101687.14 total 110.06 last 62.20) - open/close 8974 (time 1710101697.14 total 120.07 last 60.57) - open/close 9520 (time 1710101707.18 total 130.11 last 54.40) total: 10000 open/close in 139.03 seconds: 71.93 ops/second - unlinked 0 (time 1710101718 ; total 0 ; last 0) total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second - open/close 1090 (time 1710101825.24 total 10.01 last 108.89) - open/close 2017 (time 1710101835.25 total 20.02 last 92.61) - open/close 2915 (time 1710101845.26 total 30.03 last 89.68) - open/close 3717 (time 1710101855.27 total 40.04 last 80.15) - open/close 4420 (time 1710101865.28 total 50.06 last 70.17) - open/close 5120 (time 1710101875.29 total 60.07 last 69.93) - open/close 5796 (time 1710101885.30 total 70.07 last 67.55) - open/close 6452 (time 1710101895.31 total 80.08 last 65.54) - open/close 7122 (time 1710101905.31 total 90.09 last 66.98) - open/close 7724 (time 1710101915.33 total 100.10 last 60.12) - open/close 8345 (time 1710101925.37 total 110.14 last 61.85) - open/close 8876 (time 1710101935.38 total 120.15 last 53.05) - open/close 9372 (time 1710101945.45 total 130.22 last 49.24) - open/close 9875 (time 1710101955.46 total 140.23 last 50.25) total: 10000 open/close in 142.65 seconds: 70.10 ops/second - unlinked 0 (time 1710101960 ; total 0 ; last 0) total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second - open/close 1017 (time 1710102065.66 total 10.01 last 101.60) - open/close 1944 (time 1710102075.66 total 20.02 last 92.65) - open/close 2862 (time 1710102085.67 total 30.02 last 91.78) - open/close 3656 (time 1710102095.68 total 40.03 last 79.29) - open/close 4417 (time 1710102105.68 total 50.03 last 76.09) - open/close 5140 (time 1710102115.69 total 60.04 last 72.26) - open/close 5831 (time 1710102125.69 total 70.04 last 69.09) - open/close 6058 (time 1710102140.22 total 84.57 last 15.62) - open/close 6684 (time 1710102150.23 total 94.58 last 62.55) - open/close 7318 (time 1710102160.23 total 104.58 last 63.39) - open/close 7915 (time 1710102170.24 total 114.59 last 59.65) - open/close 8526 (time 1710102180.25 total 124.60 last 61.03) - open/close 9063 (time 1710102190.27 total 134.62 last 53.63) - open/close 9588 (time 1710102200.28 total 144.63 last 52.44) total: 10000 open/close in 152.29 seconds: 65.66 ops/second - unlinked 0 (time 1710102210 ; total 0 ; last 0) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second - open/close 1046 (time 1710102313.44 total 10.01 last 104.53) - open/close 1997 (time 1710102323.45 total 20.01 last 95.07) - open/close 2854 (time 1710102333.46 total 30.02 last 85.59) - open/close 3653 (time 1710102343.46 total 40.03 last 79.87) - open/close 4486 (time 1710102353.47 total 50.03 last 83.25) - open/close 5222 (time 1710102363.48 total 60.04 last 73.52) - open/close 5923 (time 1710102373.48 total 70.05 last 70.07) - open/close 6622 (time 1710102383.49 total 80.05 last 69.86) - open/close 7269 (time 1710102393.51 total 90.07 last 64.58) - open/close 7872 (time 1710102403.52 total 100.08 last 60.25) - open/close 8506 (time 1710102413.53 total 110.09 last 63.32) - open/close 9081 (time 1710102423.53 total 120.09 last 57.50) - open/close 9685 (time 1710102433.54 total 130.10 last 60.33) total: 10000 open/close in 135.85 seconds: 73.61 ops/second - unlinked 0 (time 1710102441 ; total 0 ; last 0) total: 10000 unlinks in 98 seconds: 102.040817 unlinks/second - open/close 1115 (time 1710102551.38 total 10.00 last 111.49) - open/close 2138 (time 1710102561.38 total 20.00 last 102.30) - open/close 3126 (time 1710102571.39 total 30.01 last 98.76) - open/close 3996 (time 1710102581.39 total 40.01 last 86.99) - open/close 4790 (time 1710102591.41 total 50.03 last 79.23) - open/close 5569 (time 1710102601.42 total 60.04 last 77.80) - open/close 6359 (time 1710102611.42 total 70.04 last 79.00) - open/close 7049 (time 1710102621.44 total 80.06 last 68.87) - open/close 7723 (time 1710102631.45 total 90.06 last 67.38) - open/close 8354 (time 1710102641.47 total 100.09 last 62.97) - open/close 8958 (time 1710102651.48 total 110.09 last 60.34) - open/close 9519 (time 1710102661.48 total 120.09 last 56.10) total: 10000 open/close in 129.48 seconds: 77.23 ops/second - unlinked 0 (time 1710102672 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 1170 (time 1710102771.28 total 10.00 last 116.99) - open/close 2215 (time 1710102781.29 total 20.01 last 104.43) - open/close 3232 (time 1710102791.30 total 30.02 last 101.59) - open/close 4127 (time 1710102801.30 total 40.02 last 89.49) - open/close 5025 (time 1710102811.30 total 50.02 last 89.76) - open/close 5866 (time 1710102821.31 total 60.03 last 84.06) - open/close 6674 (time 1710102831.31 total 70.03 last 80.76) - open/close 7468 (time 1710102841.32 total 80.04 last 79.37) - open/close 8252 (time 1710102851.32 total 90.04 last 78.36) - open/close 8977 (time 1710102861.34 total 100.06 last 72.41) - open/close 9763 (time 1710102871.35 total 110.07 last 78.51) total: 10000 open/close in 113.16 seconds: 88.37 ops/second - unlinked 0 (time 1710102875 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 1514 (time 1710102948.83 total 10.00 last 151.37) - open/close 2997 (time 1710102958.83 total 20.00 last 148.27) - open/close 4424 (time 1710102968.84 total 30.01 last 142.62) - open/close 5813 (time 1710102978.84 total 40.01 last 138.90) - open/close 7392 (time 1710102988.84 total 50.01 last 157.90) - open/close 8992 (time 1710102998.84 total 60.01 last 159.95) total: 10000 open/close in 67.30 seconds: 148.60 ops/second - unlinked 0 (time 1710103007 ; total 0 ; last 0) total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second - open/close 3022 (time 1710103056.81 total 10.00 last 302.19) - open/close 5497 (time 1710103066.81 total 20.00 last 247.47) - open/close 7539 (time 1710103076.81 total 30.00 last 204.13) - open/close 9360 (time 1710103086.81 total 40.01 last 182.04) total: 10000 open/close in 43.74 seconds: 228.61 ops/second - unlinked 0 (time 1710103091 ; total 0 ; last 0) total: 10000 unlinks in 24 seconds: 416.666656 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg215-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg215-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg215-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.115@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg215-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg215-server: oleg215-server.virtnet: executing set_default_debug -1 all pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 Started lustre-OST0000 oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg215-server: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg215-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 16 sec oleg215-server: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg215-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg215-client.virtnet: -o user_xattr,flock oleg215-server@tcp:/lustre /mnt/lustre On OST0, 20429 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg215-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg215-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg215-server stop mds service on oleg215-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg215-server stop mds service on oleg215-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg215-server unloading modules on: 'oleg215-server' oleg215-server: oleg215-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg215-client: oleg215-client: ssh exited with exit code 2 pdsh@oleg215-client: oleg215-server: ssh exited with exit code 2 pdsh@oleg215-client: oleg215-client: ssh exited with exit code 2 pdsh@oleg215-client: oleg215-server: ssh exited with exit code 2