== conf-sanity test 69: replace an OST with the same index ========================================================== 16:03:49 (1710101029) start mds service on oleg419-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg419-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg419-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-OST0000 oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg419-client.virtnet: -o user_xattr,flock oleg419-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49440 inodes available. Want 99872. rc=0 - open/close 996 (time 1710101078.24 total 10.00 last 99.58) - open/close 1875 (time 1710101088.25 total 20.01 last 87.82) - open/close 2683 (time 1710101098.25 total 30.02 last 80.76) - open/close 3505 (time 1710101108.26 total 40.02 last 82.13) - open/close 4293 (time 1710101118.26 total 50.03 last 78.78) - open/close 5074 (time 1710101128.27 total 60.03 last 78.05) - open/close 5748 (time 1710101138.29 total 70.06 last 67.25) - open/close 6451 (time 1710101148.30 total 80.07 last 70.22) - open/close 7096 (time 1710101158.31 total 90.08 last 64.43) - open/close 7709 (time 1710101168.32 total 100.09 last 61.24) - open/close 8258 (time 1710101178.33 total 110.09 last 54.88) - open/close 8805 (time 1710101188.34 total 120.10 last 54.64) - open/close 9353 (time 1710101198.34 total 130.11 last 54.79) - open/close 9925 (time 1710101208.35 total 140.11 last 57.15) total: 10000 open/close in 141.58 seconds: 70.63 ops/second - unlinked 0 (time 1710101211 ; total 0 ; last 0) total: 10000 unlinks in 98 seconds: 102.040817 unlinks/second - open/close 976 (time 1710101321.79 total 10.00 last 97.58) - open/close 1902 (time 1710101331.80 total 20.01 last 92.52) - open/close 2752 (time 1710101341.81 total 30.02 last 84.92) - open/close 2931 (time 1710101355.09 total 43.30 last 13.47) - open/close 3730 (time 1710101365.10 total 53.31 last 79.82) - open/close 4508 (time 1710101375.11 total 63.32 last 77.78) - open/close 5180 (time 1710101385.11 total 73.33 last 67.14) - open/close 5871 (time 1710101395.12 total 83.33 last 69.10) - open/close 6510 (time 1710101405.13 total 93.34 last 63.83) - open/close 7083 (time 1710101415.13 total 103.34 last 57.29) - open/close 7708 (time 1710101425.14 total 113.35 last 62.42) - open/close 8301 (time 1710101435.15 total 123.36 last 59.26) - open/close 8928 (time 1710101445.15 total 133.36 last 62.68) - open/close 9472 (time 1710101455.15 total 143.36 last 54.39) total: 10000 open/close in 153.36 seconds: 65.21 ops/second - unlinked 0 (time 1710101467 ; total 0 ; last 0) total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second - open/close 1020 (time 1710101574.33 total 10.00 last 101.96) - open/close 1952 (time 1710101584.33 total 20.01 last 93.15) - open/close 2842 (time 1710101594.34 total 30.01 last 88.95) - open/close 3617 (time 1710101604.34 total 40.02 last 77.47) - open/close 4358 (time 1710101614.34 total 50.02 last 74.09) - open/close 5128 (time 1710101624.35 total 60.02 last 76.98) - open/close 5808 (time 1710101634.35 total 70.02 last 67.99) - open/close 6479 (time 1710101644.35 total 80.02 last 67.09) - open/close 7084 (time 1710101654.35 total 90.03 last 60.47) - open/close 7680 (time 1710101664.36 total 100.04 last 59.56) - open/close 8264 (time 1710101674.37 total 110.05 last 58.33) - open/close 8829 (time 1710101684.38 total 120.05 last 56.49) - open/close 9384 (time 1710101694.39 total 130.07 last 55.41) - open/close 9858 (time 1710101704.40 total 140.08 last 47.36) total: 10000 open/close in 142.84 seconds: 70.01 ops/second - unlinked 0 (time 1710101709 ; total 0 ; last 0) total: 10000 unlinks in 107 seconds: 93.457947 unlinks/second - open/close 1053 (time 1710101828.86 total 10.01 last 105.24) - open/close 2039 (time 1710101838.87 total 20.01 last 98.56) - open/close 2922 (time 1710101848.87 total 30.02 last 88.24) - open/close 3743 (time 1710101858.88 total 40.03 last 82.02) - open/close 4572 (time 1710101868.89 total 50.03 last 82.88) - open/close 5359 (time 1710101878.89 total 60.03 last 78.66) - open/close 6058 (time 1710101888.90 total 70.04 last 69.82) - open/close 6711 (time 1710101898.91 total 80.05 last 65.26) - open/close 7279 (time 1710101908.92 total 90.06 last 56.72) - open/close 7819 (time 1710101918.94 total 100.09 last 53.89) - open/close 8403 (time 1710101928.96 total 110.10 last 58.32) - open/close 8934 (time 1710101938.96 total 120.11 last 53.07) - open/close 9450 (time 1710101948.97 total 130.11 last 51.59) - open/close 9948 (time 1710101958.99 total 140.13 last 49.69) total: 10000 open/close in 141.13 seconds: 70.86 ops/second - unlinked 0 (time 1710101962 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second - open/close 1036 (time 1710102066.92 total 10.01 last 103.48) - open/close 1930 (time 1710102076.92 total 20.02 last 89.36) - open/close 2823 (time 1710102086.93 total 30.02 last 89.23) - open/close 3626 (time 1710102096.94 total 40.04 last 80.19) - open/close 4361 (time 1710102106.95 total 50.04 last 73.48) - open/close 5087 (time 1710102116.97 total 60.06 last 72.44) - open/close 5310 (time 1710102130.97 total 74.06 last 15.93) - open/close 6012 (time 1710102140.99 total 84.08 last 70.08) - open/close 6701 (time 1710102151.00 total 94.10 last 68.80) - open/close 7339 (time 1710102161.00 total 104.10 last 63.80) - open/close 7988 (time 1710102171.01 total 114.10 last 64.87) - open/close 8592 (time 1710102181.01 total 124.11 last 60.37) - open/close 8948 (time 1710102199.00 total 142.09 last 19.79) - open/close 9484 (time 1710102209.01 total 152.10 last 53.55) total: 10000 open/close in 161.35 seconds: 61.98 ops/second - unlinked 0 (time 1710102220 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second - open/close 1011 (time 1710102324.36 total 10.00 last 101.05) - open/close 2020 (time 1710102334.36 total 20.01 last 100.89) - open/close 2957 (time 1710102344.37 total 30.01 last 93.67) - open/close 3779 (time 1710102354.37 total 40.02 last 82.14) - open/close 4590 (time 1710102364.38 total 50.03 last 81.03) - open/close 5353 (time 1710102374.39 total 60.03 last 76.23) - open/close 6096 (time 1710102384.39 total 70.04 last 74.29) - open/close 6726 (time 1710102394.40 total 80.05 last 62.93) - open/close 7367 (time 1710102404.41 total 90.05 last 64.06) - open/close 7929 (time 1710102414.42 total 100.07 last 56.11) - open/close 8510 (time 1710102424.43 total 110.08 last 58.04) - open/close 9094 (time 1710102434.44 total 120.08 last 58.39) - open/close 9638 (time 1710102444.44 total 130.08 last 54.39) total: 10000 open/close in 135.86 seconds: 73.60 ops/second - unlinked 0 (time 1710102452 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 1066 (time 1710102546.67 total 10.01 last 106.50) - open/close 2044 (time 1710102556.67 total 20.01 last 97.77) - open/close 3043 (time 1710102566.68 total 30.02 last 99.82) - open/close 3967 (time 1710102576.69 total 40.03 last 92.31) - open/close 4843 (time 1710102586.69 total 50.03 last 87.59) - open/close 5636 (time 1710102596.70 total 60.04 last 79.24) - open/close 6399 (time 1710102606.70 total 70.04 last 76.28) - open/close 7092 (time 1710102616.71 total 80.05 last 69.28) - open/close 7834 (time 1710102626.71 total 90.05 last 74.15) - open/close 8522 (time 1710102636.72 total 100.06 last 68.72) - open/close 9183 (time 1710102646.73 total 110.07 last 66.09) - open/close 9795 (time 1710102656.73 total 120.07 last 61.17) total: 10000 open/close in 123.08 seconds: 81.25 ops/second - unlinked 0 (time 1710102661 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 1099 (time 1710102753.25 total 10.01 last 109.78) - open/close 2132 (time 1710102763.26 total 20.02 last 103.22) - open/close 3138 (time 1710102773.26 total 30.02 last 100.56) - open/close 4096 (time 1710102783.27 total 40.03 last 95.77) - open/close 5041 (time 1710102793.27 total 50.03 last 94.43) - open/close 5931 (time 1710102803.27 total 60.04 last 88.98) - open/close 6776 (time 1710102813.28 total 70.04 last 84.44) - open/close 7610 (time 1710102823.29 total 80.05 last 83.36) - open/close 8392 (time 1710102833.30 total 90.06 last 78.09) - open/close 9116 (time 1710102843.31 total 100.07 last 72.36) - open/close 9782 (time 1710102853.31 total 110.07 last 66.55) total: 10000 open/close in 113.38 seconds: 88.20 ops/second - unlinked 0 (time 1710102858 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 1499 (time 1710102935.30 total 10.00 last 149.83) - open/close 2883 (time 1710102945.31 total 20.01 last 138.32) - open/close 4236 (time 1710102955.31 total 30.01 last 135.28) - open/close 5642 (time 1710102965.31 total 40.01 last 140.57) - open/close 6994 (time 1710102975.32 total 50.02 last 135.13) - open/close 8199 (time 1710102985.32 total 60.03 last 120.42) - open/close 9336 (time 1710102995.33 total 70.03 last 113.63) total: 10000 open/close in 75.73 seconds: 132.05 ops/second - unlinked 0 (time 1710103002 ; total 0 ; last 0) total: 10000 unlinks in 43 seconds: 232.558136 unlinks/second - open/close 2696 (time 1710103056.05 total 10.00 last 269.58) - open/close 4721 (time 1710103066.05 total 20.01 last 202.40) - open/close 6429 (time 1710103076.06 total 30.01 last 170.71) - open/close 8060 (time 1710103086.06 total 40.02 last 163.01) - open/close 9550 (time 1710103096.07 total 50.02 last 148.90) total: 10000 open/close in 53.02 seconds: 188.61 ops/second - unlinked 0 (time 1710103099 ; total 0 ; last 0) total: 10000 unlinks in 31 seconds: 322.580658 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg419-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg419-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg419-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.119@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg419-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-OST0000 oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg419-server: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg419-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec oleg419-server: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg419-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg419-client.virtnet: -o user_xattr,flock oleg419-server@tcp:/lustre /mnt/lustre On OST0, 20269 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg419-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg419-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg419-server stop mds service on oleg419-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg419-server stop mds service on oleg419-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg419-server unloading modules on: 'oleg419-server' oleg419-server: oleg419-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg419-client: oleg419-client: ssh exited with exit code 2 pdsh@oleg419-client: oleg419-server: ssh exited with exit code 2 pdsh@oleg419-client: oleg419-client: ssh exited with exit code 2 pdsh@oleg419-client: oleg419-server: ssh exited with exit code 2