== conf-sanity test 69: replace an OST with the same index ========================================================== 16:02:35 (1710100955) start mds service on oleg407-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg407-server: oleg407-server.virtnet: executing set_default_debug -1 all pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg407-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg407-server: oleg407-server.virtnet: executing set_default_debug -1 all pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg407-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg407-server: oleg407-server.virtnet: executing set_default_debug -1 all pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Started lustre-OST0000 oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg407-client.virtnet: -o user_xattr,flock oleg407-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49440 inodes available. Want 99872. rc=0 - open/close 1222 (time 1710100988.78 total 10.01 last 122.13) - open/close 2378 (time 1710100998.79 total 20.02 last 115.46) - open/close 3448 (time 1710101008.80 total 30.02 last 106.92) - open/close 4404 (time 1710101018.81 total 40.03 last 95.54) - open/close 5141 (time 1710101028.81 total 50.04 last 73.65) - open/close 5876 (time 1710101038.86 total 60.09 last 73.13) - open/close 6503 (time 1710101048.87 total 70.09 last 62.68) - open/close 7097 (time 1710101058.87 total 80.09 last 59.39) - open/close 7755 (time 1710101068.87 total 90.10 last 65.78) - open/close 8381 (time 1710101078.87 total 100.10 last 62.59) - open/close 8988 (time 1710101088.89 total 110.12 last 60.58) - open/close 9502 (time 1710101098.89 total 120.12 last 51.39) total: 10000 open/close in 129.19 seconds: 77.40 ops/second - unlinked 0 (time 1710101110 ; total 0 ; last 0) total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second - open/close 1062 (time 1710101215.40 total 10.00 last 106.18) - open/close 1960 (time 1710101225.40 total 20.01 last 89.76) - open/close 2830 (time 1710101235.40 total 30.01 last 86.99) - open/close 3569 (time 1710101245.40 total 40.01 last 73.88) - open/close 4278 (time 1710101255.41 total 50.02 last 70.86) - open/close 4944 (time 1710101265.44 total 60.05 last 66.39) - open/close 5591 (time 1710101275.46 total 70.06 last 64.62) - open/close 6212 (time 1710101285.46 total 80.06 last 62.09) - open/close 6834 (time 1710101295.47 total 90.08 last 62.11) - open/close 7427 (time 1710101305.48 total 100.08 last 59.26) - open/close 7999 (time 1710101315.51 total 110.12 last 57.02) - open/close 8539 (time 1710101325.52 total 120.13 last 53.95) - open/close 9067 (time 1710101335.54 total 130.15 last 52.69) - open/close 9565 (time 1710101345.55 total 140.16 last 49.76) total: 10000 open/close in 148.66 seconds: 67.27 ops/second - unlinked 0 (time 1710101356 ; total 0 ; last 0) total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second - open/close 1037 (time 1710101462.12 total 10.00 last 103.68) - open/close 1981 (time 1710101472.12 total 20.00 last 94.40) - open/close 2850 (time 1710101482.12 total 30.00 last 86.89) - open/close 3628 (time 1710101492.12 total 40.00 last 77.80) - open/close 3956 (time 1710101507.59 total 55.48 last 21.20) - open/close 4683 (time 1710101517.60 total 65.48 last 72.66) - open/close 5340 (time 1710101527.60 total 75.48 last 65.69) - open/close 6019 (time 1710101537.60 total 85.49 last 67.86) - open/close 6643 (time 1710101547.61 total 95.50 last 62.34) - open/close 7245 (time 1710101557.62 total 105.51 last 60.15) - open/close 7818 (time 1710101567.64 total 115.52 last 57.22) - open/close 8423 (time 1710101577.65 total 125.54 last 60.40) - open/close 8948 (time 1710101587.67 total 135.56 last 52.41) - open/close 9487 (time 1710101597.69 total 145.58 last 53.78) - open/close 9970 (time 1710101607.71 total 155.59 last 48.24) total: 10000 open/close in 156.19 seconds: 64.02 ops/second - unlinked 0 (time 1710101610 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 1127 (time 1710101713.17 total 10.00 last 112.65) - open/close 2102 (time 1710101723.18 total 20.01 last 97.43) - open/close 2890 (time 1710101733.18 total 30.01 last 78.80) - open/close 3765 (time 1710101743.18 total 40.01 last 87.48) - open/close 4509 (time 1710101753.19 total 50.03 last 74.28) - open/close 5223 (time 1710101763.21 total 60.04 last 71.32) - open/close 5906 (time 1710101773.21 total 70.05 last 68.27) - open/close 6669 (time 1710101783.21 total 80.05 last 76.28) - open/close 7303 (time 1710101793.22 total 90.05 last 63.39) - open/close 7849 (time 1710101803.23 total 100.07 last 54.51) - open/close 8418 (time 1710101813.25 total 110.08 last 56.82) - open/close 8999 (time 1710101823.26 total 120.10 last 58.02) - open/close 9510 (time 1710101833.26 total 130.10 last 51.09) total: 10000 open/close in 139.21 seconds: 71.83 ops/second - unlinked 0 (time 1710101844 ; total 0 ; last 0) total: 10000 unlinks in 104 seconds: 96.153847 unlinks/second - open/close 486 (time 1710101966.89 total 16.09 last 30.21) - open/close 1495 (time 1710101976.90 total 26.09 last 100.85) - open/close 2357 (time 1710101986.90 total 36.09 last 86.17) - open/close 2707 (time 1710102002.04 total 51.23 last 23.12) - open/close 3544 (time 1710102012.05 total 61.25 last 83.61) - open/close 4328 (time 1710102022.08 total 71.28 last 78.15) - open/close 5103 (time 1710102032.09 total 81.28 last 77.46) - open/close 5781 (time 1710102042.09 total 91.29 last 67.78) - open/close 6393 (time 1710102052.10 total 101.29 last 61.18) - open/close 7012 (time 1710102062.11 total 111.30 last 61.84) - open/close 7602 (time 1710102072.11 total 121.31 last 58.96) - open/close 8171 (time 1710102082.12 total 131.31 last 56.87) - open/close 8753 (time 1710102092.13 total 141.32 last 58.14) - open/close 9301 (time 1710102102.14 total 151.33 last 54.75) - open/close 9827 (time 1710102112.15 total 161.34 last 52.56) total: 10000 open/close in 164.56 seconds: 60.77 ops/second - unlinked 0 (time 1710102117 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second - open/close 1029 (time 1710102221.44 total 10.00 last 102.85) - open/close 1931 (time 1710102231.45 total 20.01 last 90.16) - open/close 2804 (time 1710102241.45 total 30.02 last 87.24) - open/close 3650 (time 1710102251.46 total 40.03 last 84.51) - open/close 4454 (time 1710102261.50 total 50.06 last 80.10) - open/close 5128 (time 1710102271.52 total 60.08 last 67.30) - open/close 5799 (time 1710102281.52 total 70.08 last 67.06) - open/close 6418 (time 1710102291.53 total 80.09 last 61.88) - open/close 7027 (time 1710102301.54 total 90.10 last 60.82) - open/close 7625 (time 1710102311.54 total 100.10 last 59.78) - open/close 8230 (time 1710102321.55 total 110.11 last 60.46) - open/close 8827 (time 1710102331.60 total 120.16 last 59.39) - open/close 9397 (time 1710102341.63 total 130.19 last 56.85) - open/close 9937 (time 1710102351.64 total 140.20 last 53.93) total: 10000 open/close in 141.62 seconds: 70.61 ops/second - unlinked 0 (time 1710102355 ; total 0 ; last 0) total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second - open/close 482 (time 1710102465.94 total 15.64 last 30.82) - open/close 1515 (time 1710102475.94 total 25.64 last 103.30) - open/close 2459 (time 1710102485.94 total 35.64 last 94.38) - open/close 3300 (time 1710102495.94 total 45.64 last 84.10) - open/close 4057 (time 1710102505.95 total 55.65 last 75.66) - open/close 4863 (time 1710102515.96 total 65.66 last 80.51) - open/close 5612 (time 1710102525.96 total 75.66 last 74.87) - open/close 6274 (time 1710102535.97 total 85.67 last 66.15) - open/close 6925 (time 1710102545.99 total 95.69 last 64.96) - open/close 7520 (time 1710102556.00 total 105.70 last 59.43) - open/close 8145 (time 1710102566.01 total 115.71 last 62.49) - open/close 8770 (time 1710102576.02 total 125.72 last 62.42) - open/close 9358 (time 1710102586.02 total 135.72 last 58.79) - open/close 9882 (time 1710102596.03 total 145.73 last 52.35) total: 10000 open/close in 147.90 seconds: 67.61 ops/second - unlinked 0 (time 1710102600 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 1158 (time 1710102697.92 total 10.00 last 115.77) - open/close 2177 (time 1710102707.92 total 20.01 last 101.87) - open/close 3058 (time 1710102717.93 total 30.01 last 88.05) - open/close 3920 (time 1710102727.93 total 40.02 last 86.13) - open/close 4758 (time 1710102737.94 total 50.03 last 83.73) - open/close 5551 (time 1710102747.95 total 60.04 last 79.23) - open/close 6301 (time 1710102757.96 total 70.04 last 74.93) - open/close 7032 (time 1710102767.97 total 80.06 last 73.00) - open/close 7774 (time 1710102777.98 total 90.07 last 74.15) - open/close 8487 (time 1710102787.98 total 100.07 last 71.26) - open/close 9190 (time 1710102797.99 total 110.07 last 70.27) - open/close 9852 (time 1710102808.00 total 120.09 last 66.12) total: 10000 open/close in 122.27 seconds: 81.79 ops/second - unlinked 0 (time 1710102811 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 1398 (time 1710102898.30 total 10.01 last 139.73) - open/close 2771 (time 1710102908.31 total 20.01 last 137.21) - open/close 3968 (time 1710102918.31 total 30.02 last 119.65) - open/close 5101 (time 1710102928.32 total 40.02 last 113.29) - open/close 6175 (time 1710102938.32 total 50.02 last 107.40) - open/close 7251 (time 1710102948.32 total 60.02 last 107.58) - open/close 8315 (time 1710102958.32 total 70.02 last 106.40) - open/close 9421 (time 1710102968.32 total 80.02 last 110.60) total: 10000 open/close in 85.39 seconds: 117.11 ops/second - unlinked 0 (time 1710102974 ; total 0 ; last 0) total: 10000 unlinks in 47 seconds: 212.765961 unlinks/second - open/close 2284 (time 1710103032.33 total 10.00 last 228.35) - open/close 4179 (time 1710103042.33 total 20.00 last 189.47) - open/close 5929 (time 1710103052.34 total 30.01 last 174.90) - open/close 7544 (time 1710103062.34 total 40.01 last 161.44) - open/close 8969 (time 1710103072.35 total 50.02 last 142.44) total: 10000 open/close in 57.31 seconds: 174.48 ops/second - unlinked 0 (time 1710103080 ; total 0 ; last 0) total: 10000 unlinks in 34 seconds: 294.117645 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg407-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg407-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg407-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.107@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg407-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg407-server: oleg407-server.virtnet: executing set_default_debug -1 all pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Started lustre-OST0000 oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg407-server: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg407-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 11 sec oleg407-server: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg407-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg407-client.virtnet: -o user_xattr,flock oleg407-server@tcp:/lustre /mnt/lustre On OST0, 12525 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg407-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg407-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg407-server stop mds service on oleg407-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg407-server stop mds service on oleg407-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg407-server unloading modules on: 'oleg407-server' oleg407-server: oleg407-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg407-client: oleg407-client: ssh exited with exit code 2 pdsh@oleg407-client: oleg407-server: ssh exited with exit code 2 pdsh@oleg407-client: oleg407-client: ssh exited with exit code 2 pdsh@oleg407-client: oleg407-server: ssh exited with exit code 2