== conf-sanity test 69: replace an OST with the same index ========================================================== 15:58:06 (1710100686) start mds service on oleg427-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg427-server: oleg427-server.virtnet: executing set_default_debug -1 all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg427-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg427-server: oleg427-server.virtnet: executing set_default_debug -1 all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-OST0000 oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg427-client.virtnet: -o user_xattr,flock oleg427-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49602 inodes available. Want 99872. rc=0 - open/close 2484 (time 1710100714.69 total 10.00 last 248.39) - open/close 4593 (time 1710100724.69 total 20.00 last 210.83) - open/close 5885 (time 1710100734.70 total 30.01 last 129.18) - open/close 6497 (time 1710100752.64 total 47.95 last 34.11) - open/close 7689 (time 1710100762.64 total 57.95 last 119.17) - open/close 8789 (time 1710100772.64 total 67.95 last 110.00) - open/close 9508 (time 1710100782.64 total 77.95 last 71.87) total: 10000 open/close in 84.14 seconds: 118.85 ops/second - unlinked 0 (time 1710100790 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 1756 (time 1710100863.43 total 10.00 last 175.59) - open/close 3144 (time 1710100873.44 total 20.01 last 138.72) - open/close 4309 (time 1710100883.44 total 30.01 last 116.43) - open/close 5317 (time 1710100893.45 total 40.02 last 100.74) - open/close 6153 (time 1710100903.45 total 50.02 last 83.59) - open/close 7018 (time 1710100913.46 total 60.03 last 86.44) - open/close 7930 (time 1710100923.47 total 70.04 last 91.06) - open/close 8656 (time 1710100933.48 total 80.05 last 72.54) - open/close 9392 (time 1710100943.50 total 90.07 last 73.45) total: 10000 open/close in 98.06 seconds: 101.98 ops/second - unlinked 0 (time 1710100952 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 1028 (time 1710101032.87 total 10.00 last 102.75) - open/close 2027 (time 1710101042.87 total 20.01 last 99.87) - open/close 2815 (time 1710101052.87 total 30.01 last 78.80) - open/close 3601 (time 1710101062.89 total 40.02 last 78.47) - open/close 4402 (time 1710101072.89 total 50.03 last 80.06) - open/close 5176 (time 1710101082.91 total 60.04 last 77.31) - open/close 5853 (time 1710101092.91 total 70.04 last 67.69) - open/close 6510 (time 1710101102.91 total 80.04 last 65.70) - open/close 7178 (time 1710101112.92 total 90.06 last 66.71) - open/close 7786 (time 1710101122.93 total 100.07 last 60.73) - open/close 8439 (time 1710101132.94 total 110.07 last 65.25) - open/close 8979 (time 1710101142.96 total 120.09 last 53.91) - open/close 9590 (time 1710101152.97 total 130.11 last 60.99) total: 10000 open/close in 138.31 seconds: 72.30 ops/second - unlinked 0 (time 1710101163 ; total 0 ; last 0) total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second - open/close 966 (time 1710101269.06 total 10.02 last 96.41) - open/close 1861 (time 1710101279.07 total 20.03 last 89.41) - open/close 2648 (time 1710101289.08 total 30.03 last 78.66) - open/close 3413 (time 1710101299.08 total 40.04 last 76.45) - open/close 4230 (time 1710101309.10 total 50.05 last 81.59) - open/close 4945 (time 1710101319.10 total 60.06 last 71.49) - open/close 5630 (time 1710101329.11 total 70.07 last 68.42) - open/close 6336 (time 1710101339.12 total 80.07 last 70.55) - open/close 6976 (time 1710101349.13 total 90.09 last 63.91) - open/close 7592 (time 1710101359.13 total 100.09 last 61.59) - open/close 8171 (time 1710101369.14 total 110.09 last 57.88) - open/close 8719 (time 1710101379.15 total 120.11 last 54.71) - open/close 9228 (time 1710101389.16 total 130.12 last 50.84) - open/close 9754 (time 1710101399.18 total 140.14 last 52.50) total: 10000 open/close in 144.86 seconds: 69.03 ops/second - unlinked 0 (time 1710101406 ; total 0 ; last 0) total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second - open/close 935 (time 1710101512.36 total 10.00 last 93.46) - open/close 1811 (time 1710101522.37 total 20.01 last 87.53) - open/close 2433 (time 1710101540.84 total 38.48 last 33.69) - open/close 3203 (time 1710101550.84 total 48.49 last 76.94) - open/close 3998 (time 1710101560.84 total 58.49 last 79.49) - open/close 4750 (time 1710101570.86 total 68.50 last 75.06) - open/close 5481 (time 1710101580.87 total 78.51 last 73.07) - open/close 6146 (time 1710101590.87 total 88.52 last 66.45) - open/close 6810 (time 1710101600.89 total 98.53 last 66.31) - open/close 7400 (time 1710101610.90 total 108.54 last 58.93) - open/close 8032 (time 1710101620.90 total 118.54 last 63.19) - open/close 8568 (time 1710101630.91 total 128.55 last 53.54) - open/close 9086 (time 1710101640.93 total 138.57 last 51.70) - open/close 9609 (time 1710101650.93 total 148.57 last 52.28) total: 10000 open/close in 156.58 seconds: 63.87 ops/second - unlinked 0 (time 1710101661 ; total 0 ; last 0) total: 10000 unlinks in 107 seconds: 93.457947 unlinks/second - open/close 1086 (time 1710101780.38 total 10.01 last 108.53) - open/close 2023 (time 1710101790.39 total 20.02 last 93.58) - open/close 2866 (time 1710101800.40 total 30.02 last 84.26) - open/close 3620 (time 1710101810.40 total 40.03 last 75.39) - open/close 4391 (time 1710101820.41 total 50.03 last 77.06) - open/close 4955 (time 1710101838.93 total 68.56 last 30.44) - open/close 5023 (time 1710101851.11 total 80.74 last 5.58) - open/close 5678 (time 1710101861.12 total 90.75 last 65.44) - open/close 6451 (time 1710101871.13 total 100.75 last 77.24) - open/close 7154 (time 1710101881.13 total 110.76 last 70.25) - open/close 7748 (time 1710101891.14 total 120.77 last 59.37) - open/close 8331 (time 1710101901.15 total 130.78 last 58.21) - open/close 8863 (time 1710101911.17 total 140.79 last 53.15) - open/close 9351 (time 1710101921.18 total 150.81 last 48.71) - open/close 9883 (time 1710101931.19 total 160.82 last 53.15) total: 10000 open/close in 163.03 seconds: 61.34 ops/second - unlinked 0 (time 1710101935 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second - open/close 1057 (time 1710102039.54 total 10.00 last 105.67) - open/close 1932 (time 1710102049.55 total 20.01 last 87.42) - open/close 2683 (time 1710102059.56 total 30.02 last 75.06) - open/close 3516 (time 1710102069.56 total 40.02 last 83.28) - open/close 4271 (time 1710102079.56 total 50.02 last 75.49) - open/close 5052 (time 1710102089.56 total 60.03 last 78.06) - open/close 5808 (time 1710102099.57 total 70.03 last 75.55) - open/close 6496 (time 1710102109.58 total 80.04 last 68.73) - open/close 7177 (time 1710102119.59 total 90.05 last 68.08) - open/close 7289 (time 1710102132.14 total 102.60 last 8.92) - open/close 7929 (time 1710102142.14 total 112.60 last 63.97) - open/close 8027 (time 1710102154.69 total 125.15 last 7.81) - open/close 8600 (time 1710102164.70 total 135.17 last 57.23) - open/close 9244 (time 1710102174.71 total 145.17 last 64.37) - open/close 9799 (time 1710102184.71 total 155.18 last 55.47) total: 10000 open/close in 159.16 seconds: 62.83 ops/second - unlinked 0 (time 1710102190 ; total 0 ; last 0) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second - open/close 1004 (time 1710102293.27 total 10.00 last 100.36) - open/close 1976 (time 1710102303.27 total 20.01 last 97.19) - open/close 2849 (time 1710102313.28 total 30.01 last 87.26) - open/close 3659 (time 1710102323.28 total 40.01 last 80.99) - open/close 4465 (time 1710102333.30 total 50.03 last 80.47) - open/close 5204 (time 1710102343.30 total 60.03 last 73.87) - open/close 5918 (time 1710102353.30 total 70.04 last 71.36) - open/close 6610 (time 1710102363.31 total 80.04 last 69.18) - open/close 7293 (time 1710102373.32 total 90.05 last 68.23) - open/close 7965 (time 1710102383.33 total 100.06 last 67.13) - open/close 8566 (time 1710102393.34 total 110.07 last 60.03) - open/close 9155 (time 1710102403.35 total 120.08 last 58.86) - open/close 9702 (time 1710102413.35 total 130.08 last 54.66) total: 10000 open/close in 135.49 seconds: 73.81 ops/second - unlinked 0 (time 1710102420 ; total 0 ; last 0) total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second - open/close 1037 (time 1710102532.10 total 10.01 last 103.60) - open/close 1964 (time 1710102542.10 total 20.01 last 92.66) - open/close 2878 (time 1710102552.10 total 30.02 last 91.39) - open/close 3757 (time 1710102562.11 total 40.02 last 87.85) - open/close 4599 (time 1710102572.11 total 50.02 last 84.18) - open/close 5422 (time 1710102582.12 total 60.03 last 82.27) - open/close 6144 (time 1710102592.12 total 70.03 last 72.20) - open/close 6810 (time 1710102602.12 total 80.03 last 66.57) - open/close 7490 (time 1710102612.12 total 90.04 last 67.98) - open/close 8102 (time 1710102622.13 total 100.04 last 61.17) - open/close 8732 (time 1710102632.14 total 110.05 last 62.96) - open/close 9359 (time 1710102642.14 total 120.05 last 62.65) - open/close 9974 (time 1710102652.15 total 130.06 last 61.47) total: 10000 open/close in 130.51 seconds: 76.62 ops/second - unlinked 0 (time 1710102654 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 1173 (time 1710102749.05 total 10.01 last 117.16) - open/close 2199 (time 1710102759.06 total 20.02 last 102.52) - open/close 3154 (time 1710102769.06 total 30.02 last 95.49) - open/close 4123 (time 1710102779.07 total 40.02 last 96.86) - open/close 5035 (time 1710102789.07 total 50.03 last 91.15) - open/close 5935 (time 1710102799.07 total 60.03 last 89.97) - open/close 6790 (time 1710102809.08 total 70.04 last 85.45) - open/close 7620 (time 1710102819.09 total 80.05 last 82.93) - open/close 8415 (time 1710102829.10 total 90.06 last 79.38) - open/close 9184 (time 1710102839.11 total 100.07 last 76.87) - open/close 9873 (time 1710102849.11 total 110.07 last 68.86) total: 10000 open/close in 111.99 seconds: 89.30 ops/second - unlinked 0 (time 1710102852 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg427-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg427-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg427-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.127@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg427-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg427-server: oleg427-server.virtnet: executing set_default_debug -1 all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-OST0000 oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg427-server: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg427-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 6 sec mount lustre on /mnt/lustre..... Starting client: oleg427-client.virtnet: -o user_xattr,flock oleg427-server@tcp:/lustre /mnt/lustre On OST0, 7672 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg427-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg427-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg427-server stop mds service on oleg427-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg427-server unloading modules on: 'oleg427-server' oleg427-server: oleg427-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg427-client: oleg427-client: ssh exited with exit code 2 pdsh@oleg427-client: oleg427-server: ssh exited with exit code 2 pdsh@oleg427-client: oleg427-client: ssh exited with exit code 2 pdsh@oleg427-client: oleg427-server: ssh exited with exit code 2