== conf-sanity test 69: replace an OST with the same index ========================================================== 16:05:44 (1710101144) start mds service on oleg117-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg117-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg117-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg117-client.virtnet: -o user_xattr,flock oleg117-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49440 inodes available. Want 99872. rc=0 - open/close 963 (time 1710101190.74 total 10.01 last 96.16) - open/close 1876 (time 1710101200.74 total 20.02 last 91.27) - open/close 2609 (time 1710101210.74 total 30.02 last 73.26) - open/close 3349 (time 1710101220.75 total 40.02 last 73.99) - open/close 4033 (time 1710101230.75 total 50.03 last 68.38) - open/close 4613 (time 1710101240.76 total 60.04 last 57.95) - open/close 5190 (time 1710101250.77 total 70.05 last 57.64) - open/close 5760 (time 1710101260.77 total 80.05 last 56.98) - open/close 6342 (time 1710101270.79 total 90.06 last 58.13) - open/close 6629 (time 1710101287.45 total 106.73 last 17.22) - open/close 7210 (time 1710101297.46 total 116.73 last 58.05) - open/close 7705 (time 1710101307.48 total 126.76 last 49.38) - open/close 8166 (time 1710101317.50 total 136.78 last 46.01) - open/close 8611 (time 1710101327.51 total 146.79 last 44.44) - open/close 9092 (time 1710101337.51 total 156.79 last 48.10) - open/close 9527 (time 1710101347.52 total 166.80 last 43.46) - open/close 9952 (time 1710101357.52 total 176.80 last 42.49) total: 10000 open/close in 177.90 seconds: 56.21 ops/second - unlinked 0 (time 1710101360 ; total 0 ; last 0) total: 10000 unlinks in 119 seconds: 84.033615 unlinks/second - open/close 405 (time 1710101497.50 total 15.08 last 26.86) - open/close 1191 (time 1710101507.51 total 25.09 last 78.53) - open/close 2005 (time 1710101517.52 total 35.10 last 81.32) - open/close 2767 (time 1710101527.53 total 45.10 last 76.15) - open/close 3252 (time 1710101546.02 total 63.60 last 26.22) - open/close 3883 (time 1710101556.02 total 73.60 last 63.09) - open/close 4503 (time 1710101566.03 total 83.60 last 61.98) - open/close 5125 (time 1710101576.03 total 93.61 last 62.15) - open/close 5645 (time 1710101586.05 total 103.63 last 51.90) - open/close 6167 (time 1710101596.07 total 113.65 last 52.12) - open/close 6683 (time 1710101606.09 total 123.67 last 51.50) - open/close 7222 (time 1710101616.10 total 133.68 last 53.83) - open/close 7697 (time 1710101626.10 total 143.68 last 47.49) - open/close 8207 (time 1710101636.11 total 153.69 last 50.94) - open/close 8657 (time 1710101646.12 total 163.70 last 44.97) - open/close 9119 (time 1710101656.14 total 173.72 last 46.10) - open/close 9598 (time 1710101666.15 total 183.72 last 47.88) total: 10000 open/close in 192.79 seconds: 51.87 ops/second - unlinked 0 (time 1710101677 ; total 0 ; last 0) total: 10000 unlinks in 133 seconds: 75.187973 unlinks/second - open/close 792 (time 1710101823.21 total 10.02 last 79.06) - open/close 1211 (time 1710101839.73 total 26.54 last 25.36) - open/close 2054 (time 1710101849.73 total 36.55 last 84.27) - open/close 2809 (time 1710101859.76 total 46.57 last 75.32) - open/close 3472 (time 1710101869.76 total 56.57 last 66.27) - open/close 3904 (time 1710101887.51 total 74.32 last 24.35) - open/close 4611 (time 1710101897.52 total 84.33 last 70.62) - open/close 5225 (time 1710101907.53 total 94.34 last 61.33) - open/close 5763 (time 1710101917.53 total 104.34 last 53.78) - open/close 6310 (time 1710101927.55 total 114.36 last 54.61) - open/close 6554 (time 1710101944.02 total 130.83 last 14.81) - open/close 7118 (time 1710101954.03 total 140.84 last 56.37) - open/close 7663 (time 1710101964.04 total 150.85 last 54.44) - open/close 8131 (time 1710101974.04 total 160.85 last 46.79) - open/close 8613 (time 1710101984.08 total 170.89 last 48.01) - open/close 9126 (time 1710101994.08 total 180.89 last 51.28) - open/close 9575 (time 1710102004.10 total 190.91 last 44.85) total: 10000 open/close in 200.55 seconds: 49.86 ops/second - unlinked 0 (time 1710102015 ; total 0 ; last 0) total: 10000 unlinks in 134 seconds: 74.626869 unlinks/second - open/close 918 (time 1710102161.78 total 10.02 last 91.65) - open/close 1707 (time 1710102171.79 total 20.02 last 78.87) - open/close 2427 (time 1710102181.80 total 30.03 last 71.92) - open/close 3135 (time 1710102191.80 total 40.04 last 70.77) - open/close 3822 (time 1710102201.81 total 50.04 last 68.64) - open/close 4427 (time 1710102211.83 total 60.06 last 60.42) - open/close 4970 (time 1710102221.84 total 70.07 last 54.25) - open/close 5574 (time 1710102231.85 total 80.08 last 60.31) - open/close 6065 (time 1710102241.86 total 90.09 last 49.04) - open/close 6542 (time 1710102251.87 total 100.10 last 47.66) - open/close 7036 (time 1710102261.88 total 110.11 last 49.37) - open/close 7543 (time 1710102271.89 total 120.12 last 50.65) - open/close 8098 (time 1710102281.90 total 130.14 last 55.40) - open/close 8592 (time 1710102291.91 total 140.14 last 49.39) - open/close 9029 (time 1710102301.91 total 150.15 last 43.67) - open/close 9538 (time 1710102311.93 total 160.16 last 50.84) - open/close 9980 (time 1710102321.93 total 170.16 last 44.17) total: 10000 open/close in 170.64 seconds: 58.60 ops/second - unlinked 0 (time 1710102325 ; total 0 ; last 0) total: 10000 unlinks in 121 seconds: 82.644630 unlinks/second - open/close 821 (time 1710102458.67 total 10.01 last 81.98) - open/close 1104 (time 1710102472.96 total 24.31 last 19.80) - open/close 1809 (time 1710102482.97 total 34.31 last 70.47) - open/close 2613 (time 1710102492.99 total 44.33 last 80.23) - open/close 3266 (time 1710102502.99 total 54.34 last 65.25) - open/close 3883 (time 1710102513.00 total 64.34 last 61.68) - open/close 4514 (time 1710102523.03 total 74.37 last 62.92) - open/close 5103 (time 1710102543.47 total 94.81 last 28.82) - open/close 5627 (time 1710102553.47 total 104.82 last 52.36) - open/close 6165 (time 1710102563.52 total 114.87 last 53.55) - open/close 6489 (time 1710102580.76 total 132.10 last 18.80) - open/close 6998 (time 1710102590.78 total 142.12 last 50.82) - open/close 7481 (time 1710102600.79 total 152.13 last 48.24) - open/close 8001 (time 1710102610.79 total 162.13 last 51.99) - open/close 8457 (time 1710102620.84 total 172.19 last 45.36) - open/close 8534 (time 1710102633.48 total 184.83 last 6.09) - open/close 9030 (time 1710102643.51 total 194.85 last 49.49) - open/close 9457 (time 1710102653.51 total 204.85 last 42.69) - open/close 9874 (time 1710102663.51 total 214.85 last 41.69) total: 10000 open/close in 217.89 seconds: 45.89 ops/second - unlinked 0 (time 1710102669 ; total 0 ; last 0) total: 10000 unlinks in 111 seconds: 90.090088 unlinks/second - open/close 868 (time 1710102793.07 total 10.01 last 86.75) - open/close 1750 (time 1710102803.07 total 20.01 last 88.17) - open/close 2481 (time 1710102813.07 total 30.01 last 73.09) - open/close 3202 (time 1710102823.10 total 40.03 last 71.94) - open/close 3816 (time 1710102833.11 total 50.04 last 61.33) - open/close 4435 (time 1710102843.11 total 60.05 last 61.86) - open/close 4996 (time 1710102853.18 total 70.12 last 55.72) - open/close 5556 (time 1710102863.20 total 80.14 last 55.89) - open/close 6110 (time 1710102873.22 total 90.15 last 55.31) - open/close 6602 (time 1710102883.24 total 100.18 last 49.10) - open/close 7122 (time 1710102893.24 total 110.18 last 51.98) - open/close 7584 (time 1710102903.26 total 120.19 last 46.14) - open/close 8077 (time 1710102913.27 total 130.21 last 49.23) - open/close 8562 (time 1710102923.32 total 140.25 last 48.29) - open/close 9041 (time 1710102933.40 total 150.34 last 47.51) - open/close 9476 (time 1710102943.40 total 160.34 last 43.49) - open/close 9938 (time 1710102953.40 total 170.34 last 46.20) total: 10000 open/close in 171.63 seconds: 58.27 ops/second - unlinked 0 (time 1710102957 ; total 0 ; last 0) total: 10000 unlinks in 120 seconds: 83.333336 unlinks/second - open/close 929 (time 1710103090.40 total 10.07 last 92.30) - open/close 1740 (time 1710103100.41 total 20.07 last 81.04) - open/close 2227 (time 1710103117.30 total 36.96 last 28.83) - open/close 2970 (time 1710103127.31 total 46.97 last 74.24) - open/close 3624 (time 1710103137.32 total 56.99 last 65.31) - open/close 3856 (time 1710103152.62 total 72.28 last 15.17) - open/close 4572 (time 1710103162.63 total 82.29 last 71.54) - open/close 5163 (time 1710103172.64 total 92.30 last 59.02) - open/close 5847 (time 1710103182.69 total 102.35 last 68.09) - open/close 6452 (time 1710103192.70 total 112.36 last 60.43) - open/close 7049 (time 1710103202.71 total 122.38 last 59.60) - open/close 7574 (time 1710103212.72 total 132.39 last 52.45) - open/close 8144 (time 1710103222.73 total 142.39 last 57.00) - open/close 8619 (time 1710103232.77 total 152.43 last 47.27) - open/close 9067 (time 1710103242.80 total 162.46 last 44.71) - open/close 9590 (time 1710103252.81 total 172.47 last 52.23) total: 10000 open/close in 181.40 seconds: 55.13 ops/second - unlinked 0 (time 1710103263 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 1093 (time 1710103365.34 total 10.00 last 109.29) - open/close 2106 (time 1710103375.35 total 20.01 last 101.17) - open/close 3022 (time 1710103385.36 total 30.02 last 91.54) - open/close 3988 (time 1710103395.37 total 40.03 last 96.52) - open/close 4834 (time 1710103405.37 total 50.03 last 84.58) - open/close 5620 (time 1710103415.38 total 60.04 last 78.53) - open/close 6429 (time 1710103425.39 total 70.05 last 80.83) - open/close 7195 (time 1710103435.40 total 80.06 last 76.53) - open/close 7894 (time 1710103445.40 total 90.06 last 69.88) - open/close 8595 (time 1710103455.41 total 100.07 last 70.06) - open/close 9269 (time 1710103465.41 total 110.07 last 67.35) - open/close 9925 (time 1710103475.42 total 120.08 last 65.57) total: 10000 open/close in 121.29 seconds: 82.45 ops/second - unlinked 0 (time 1710103478 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 1439 (time 1710103556.22 total 10.00 last 143.86) - open/close 2813 (time 1710103566.22 total 20.00 last 137.39) - open/close 4156 (time 1710103576.22 total 30.01 last 134.21) - open/close 5447 (time 1710103586.23 total 40.02 last 129.02) - open/close 6681 (time 1710103596.23 total 50.02 last 123.35) - open/close 7798 (time 1710103606.24 total 60.03 last 111.59) - open/close 8948 (time 1710103616.25 total 70.04 last 114.91) total: 10000 open/close in 79.97 seconds: 125.04 ops/second - unlinked 0 (time 1710103627 ; total 0 ; last 0) total: 10000 unlinks in 40 seconds: 250.000000 unlinks/second - open/close 2934 (time 1710103678.75 total 10.00 last 293.30) - open/close 5418 (time 1710103688.76 total 20.01 last 248.34) - open/close 7502 (time 1710103698.76 total 30.01 last 208.37) - open/close 9296 (time 1710103708.76 total 40.01 last 179.31) total: 10000 open/close in 43.97 seconds: 227.43 ops/second - unlinked 0 (time 1710103713 ; total 0 ; last 0) total: 10000 unlinks in 25 seconds: 400.000000 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg117-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg117-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg117-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.117@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg117-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg117-server: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg117-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec oleg117-server: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg117-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg117-client.virtnet: -o user_xattr,flock oleg117-server@tcp:/lustre /mnt/lustre On OST0, 15229 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg117-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg117-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg117-server stop mds service on oleg117-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg117-server stop mds service on oleg117-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg117-server unloading modules on: 'oleg117-server' oleg117-server: oleg117-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg117-client: oleg117-client: ssh exited with exit code 2 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 2 pdsh@oleg117-client: oleg117-client: ssh exited with exit code 2 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 2