== conf-sanity test 69: replace an OST with the same index ========================================================== 16:03:00 (1710100980) start mds service on oleg105-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg105-server: oleg105-server.virtnet: executing set_default_debug -1 all pdsh@oleg105-client: oleg105-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg105-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg105-server: oleg105-server.virtnet: executing set_default_debug -1 all pdsh@oleg105-client: oleg105-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg105-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg105-server: oleg105-server.virtnet: executing set_default_debug -1 all pdsh@oleg105-client: oleg105-server: ssh exited with exit code 1 Started lustre-OST0000 oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg105-client.virtnet: -o user_xattr,flock oleg105-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49440 inodes available. Want 99872. rc=0 - open/close 1237 (time 1710101023.28 total 10.00 last 123.67) - open/close 2165 (time 1710101033.28 total 20.00 last 92.78) - open/close 2197 (time 1710101044.73 total 31.45 last 2.80) - open/close 3060 (time 1710101054.74 total 41.46 last 86.25) - open/close 4027 (time 1710101064.74 total 51.46 last 96.65) - open/close 4971 (time 1710101074.74 total 61.46 last 94.37) - open/close 5703 (time 1710101084.75 total 71.47 last 73.17) - open/close 6484 (time 1710101094.76 total 81.48 last 78.03) - open/close 7252 (time 1710101104.77 total 91.49 last 76.73) - open/close 7947 (time 1710101114.79 total 101.51 last 69.37) - open/close 8596 (time 1710101124.80 total 111.52 last 64.82) - open/close 9135 (time 1710101134.82 total 121.54 last 53.77) - open/close 9673 (time 1710101144.83 total 131.55 last 53.77) total: 10000 open/close in 137.35 seconds: 72.81 ops/second - unlinked 0 (time 1710101152 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 1028 (time 1710101250.03 total 10.00 last 102.80) - open/close 2028 (time 1710101260.04 total 20.00 last 99.97) - open/close 2849 (time 1710101270.04 total 30.00 last 82.10) - open/close 3644 (time 1710101280.04 total 40.00 last 79.49) - open/close 4470 (time 1710101290.04 total 50.01 last 82.58) - open/close 5198 (time 1710101300.05 total 60.02 last 72.72) - open/close 5819 (time 1710101310.06 total 70.03 last 62.04) - open/close 6456 (time 1710101320.06 total 80.03 last 63.69) - open/close 7072 (time 1710101330.08 total 90.04 last 61.50) - open/close 7744 (time 1710101340.08 total 100.05 last 67.19) - open/close 8289 (time 1710101350.09 total 110.06 last 54.44) - open/close 8845 (time 1710101360.09 total 120.06 last 55.59) - open/close 9390 (time 1710101370.10 total 130.06 last 54.47) - open/close 9926 (time 1710101380.10 total 140.07 last 53.59) total: 10000 open/close in 141.21 seconds: 70.81 ops/second - unlinked 0 (time 1710101383 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 919 (time 1710101483.45 total 10.00 last 91.90) - open/close 1826 (time 1710101493.45 total 20.00 last 90.68) - open/close 2583 (time 1710101513.91 total 40.46 last 37.00) - open/close 3496 (time 1710101534.94 total 61.49 last 43.41) - open/close 4241 (time 1710101544.94 total 71.50 last 74.48) - open/close 4992 (time 1710101554.95 total 81.50 last 75.08) - open/close 5681 (time 1710101564.95 total 91.50 last 68.89) - open/close 5823 (time 1710101577.69 total 104.25 last 11.14) - open/close 6445 (time 1710101587.70 total 114.25 last 62.15) - open/close 7089 (time 1710101597.70 total 124.26 last 64.38) - open/close 7654 (time 1710101607.71 total 134.26 last 56.46) - open/close 8237 (time 1710101617.74 total 144.29 last 58.15) - open/close 8808 (time 1710101627.75 total 154.30 last 57.05) - open/close 9386 (time 1710101637.75 total 164.30 last 57.79) - open/close 9910 (time 1710101647.77 total 174.32 last 52.29) total: 10000 open/close in 176.00 seconds: 56.82 ops/second - unlinked 0 (time 1710101652 ; total 0 ; last 0) total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second - open/close 1089 (time 1710101761.69 total 10.00 last 108.85) - open/close 2122 (time 1710101771.69 total 20.01 last 103.26) - open/close 3041 (time 1710101781.70 total 30.02 last 91.82) - open/close 3795 (time 1710101791.70 total 40.02 last 75.39) - open/close 3914 (time 1710101804.31 total 52.63 last 9.44) - open/close 4660 (time 1710101814.34 total 62.66 last 74.36) - open/close 5318 (time 1710101824.35 total 72.67 last 65.76) - open/close 5937 (time 1710101834.36 total 82.68 last 61.83) - open/close 6619 (time 1710101844.38 total 92.70 last 68.08) - open/close 7321 (time 1710101854.40 total 102.72 last 70.02) - open/close 7969 (time 1710101864.41 total 112.73 last 64.72) - open/close 8606 (time 1710101874.42 total 122.74 last 63.64) - open/close 9176 (time 1710101884.44 total 132.76 last 56.91) - open/close 9777 (time 1710101894.45 total 142.77 last 60.06) total: 10000 open/close in 146.78 seconds: 68.13 ops/second - unlinked 0 (time 1710101900 ; total 0 ; last 0) total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second - open/close 674 (time 1710102017.90 total 17.58 last 38.34) - open/close 1658 (time 1710102027.90 total 27.58 last 98.38) - open/close 2543 (time 1710102037.90 total 37.58 last 88.47) - open/close 3483 (time 1710102047.91 total 47.59 last 93.96) - open/close 4226 (time 1710102068.59 total 68.27 last 35.92) - open/close 4944 (time 1710102078.60 total 78.28 last 71.73) - open/close 5607 (time 1710102088.61 total 88.30 last 66.22) - open/close 6160 (time 1710102098.62 total 98.30 last 55.26) - open/close 6778 (time 1710102108.63 total 108.32 last 61.73) - open/close 7354 (time 1710102118.64 total 118.33 last 57.54) - open/close 7915 (time 1710102128.65 total 128.34 last 56.05) - open/close 8499 (time 1710102138.66 total 138.34 last 58.36) - open/close 9123 (time 1710102148.66 total 148.35 last 62.39) - open/close 9712 (time 1710102158.69 total 158.37 last 58.75) total: 10000 open/close in 163.68 seconds: 61.09 ops/second - unlinked 0 (time 1710102166 ; total 0 ; last 0) total: 10000 unlinks in 103 seconds: 97.087379 unlinks/second - open/close 1030 (time 1710102293.51 total 20.97 last 49.13) - open/close 2064 (time 1710102303.52 total 30.97 last 103.35) - open/close 2536 (time 1710102319.31 total 46.77 last 29.88) - open/close 3366 (time 1710102329.32 total 56.77 last 82.96) - open/close 4135 (time 1710102339.32 total 66.78 last 76.87) - open/close 4947 (time 1710102349.33 total 76.78 last 81.18) - open/close 5684 (time 1710102359.37 total 86.82 last 73.38) - open/close 6432 (time 1710102369.38 total 96.84 last 74.72) - open/close 7132 (time 1710102379.38 total 106.84 last 69.99) - open/close 7741 (time 1710102389.38 total 116.84 last 60.90) - open/close 8286 (time 1710102399.41 total 126.86 last 54.36) - open/close 8828 (time 1710102409.41 total 136.87 last 54.17) - open/close 9354 (time 1710102419.41 total 146.87 last 52.60) - open/close 9956 (time 1710102429.42 total 156.87 last 60.18) total: 10000 open/close in 157.85 seconds: 63.35 ops/second - unlinked 0 (time 1710102432 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 1108 (time 1710102531.42 total 10.01 last 110.69) - open/close 2069 (time 1710102541.43 total 20.02 last 96.04) - open/close 2988 (time 1710102551.46 total 30.05 last 91.59) - open/close 3837 (time 1710102561.47 total 40.06 last 84.85) - open/close 4577 (time 1710102571.47 total 50.06 last 73.97) - open/close 5391 (time 1710102581.48 total 60.07 last 81.31) - open/close 6086 (time 1710102591.49 total 70.07 last 69.49) - open/close 6417 (time 1710102607.62 total 86.20 last 20.52) - open/close 6787 (time 1710102624.45 total 103.04 last 21.98) - open/close 7445 (time 1710102634.46 total 113.04 last 65.76) - open/close 8077 (time 1710102644.49 total 123.07 last 63.00) - open/close 8172 (time 1710102656.93 total 135.52 last 7.63) - open/close 8806 (time 1710102666.96 total 145.54 last 63.24) - open/close 9381 (time 1710102677.02 total 155.60 last 57.14) - open/close 9908 (time 1710102687.04 total 165.62 last 52.59) total: 10000 open/close in 167.51 seconds: 59.70 ops/second - unlinked 0 (time 1710102691 ; total 0 ; last 0) total: 10000 unlinks in 105 seconds: 95.238098 unlinks/second - open/close 1074 (time 1710102807.95 total 10.00 last 107.36) - open/close 2033 (time 1710102817.96 total 20.01 last 95.87) - open/close 2894 (time 1710102827.96 total 30.01 last 86.04) - open/close 3695 (time 1710102837.96 total 40.01 last 80.10) - open/close 4462 (time 1710102847.97 total 50.02 last 76.65) - open/close 5144 (time 1710102857.97 total 60.02 last 68.18) - open/close 5902 (time 1710102867.99 total 70.04 last 75.69) - open/close 6471 (time 1710102878.00 total 80.05 last 56.82) - open/close 7059 (time 1710102888.00 total 90.05 last 58.78) - open/close 7692 (time 1710102898.03 total 100.08 last 63.16) - open/close 8251 (time 1710102908.04 total 110.09 last 55.84) - open/close 8874 (time 1710102918.04 total 120.09 last 62.26) - open/close 9424 (time 1710102928.05 total 130.10 last 54.94) - open/close 9975 (time 1710102938.07 total 140.12 last 55.03) total: 10000 open/close in 140.62 seconds: 71.11 ops/second - unlinked 0 (time 1710102941 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 1077 (time 1710103042.53 total 10.00 last 107.65) - open/close 1078 (time 1710103053.60 total 21.07 last 0.09) - open/close 2037 (time 1710103063.61 total 31.08 last 95.80) - open/close 2997 (time 1710103073.62 total 41.09 last 95.94) - open/close 3832 (time 1710103083.63 total 51.10 last 83.40) - open/close 4686 (time 1710103093.64 total 61.11 last 85.36) - open/close 5425 (time 1710103103.64 total 71.11 last 73.87) - open/close 5822 (time 1710103120.23 total 87.70 last 23.92) - open/close 6487 (time 1710103130.26 total 97.73 last 66.35) - open/close 7101 (time 1710103150.98 total 118.45 last 29.63) - open/close 7779 (time 1710103160.98 total 128.45 last 67.79) - open/close 8376 (time 1710103171.07 total 138.54 last 59.18) - open/close 9048 (time 1710103181.07 total 148.54 last 67.18) - open/close 9734 (time 1710103191.08 total 158.55 last 68.55) total: 10000 open/close in 162.91 seconds: 61.38 ops/second - unlinked 0 (time 1710103197 ; total 0 ; last 0) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second - open/close 1267 (time 1710103300.51 total 10.01 last 126.60) - open/close 2104 (time 1710103319.13 total 28.63 last 44.94) - open/close 3198 (time 1710103329.14 total 38.64 last 109.35) - open/close 4256 (time 1710103339.15 total 48.65 last 105.70) - open/close 5253 (time 1710103349.15 total 58.65 last 99.65) - open/close 6134 (time 1710103359.16 total 68.66 last 88.06) - open/close 7067 (time 1710103369.16 total 78.66 last 93.25) - open/close 7412 (time 1710103384.15 total 93.65 last 23.02) - open/close 7932 (time 1710103401.58 total 111.07 last 29.84) - open/close 8729 (time 1710103411.58 total 121.08 last 79.67) - open/close 9563 (time 1710103421.59 total 131.09 last 83.34) total: 10000 open/close in 136.94 seconds: 73.02 ops/second - unlinked 0 (time 1710103428 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg105-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg105-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg105-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.105@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg105-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg105-server: oleg105-server.virtnet: executing set_default_debug -1 all pdsh@oleg105-client: oleg105-server: ssh exited with exit code 1 Started lustre-OST0000 oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg105-server: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg105-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 14 sec oleg105-server: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg105-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg105-client.virtnet: -o user_xattr,flock oleg105-server@tcp:/lustre /mnt/lustre On OST0, 19613 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg105-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg105-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg105-server stop mds service on oleg105-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg105-server stop mds service on oleg105-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg105-server unloading modules on: 'oleg105-server' oleg105-server: oleg105-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg105-client: oleg105-client: ssh exited with exit code 2 pdsh@oleg105-client: oleg105-server: ssh exited with exit code 2 pdsh@oleg105-client: oleg105-client: ssh exited with exit code 2 pdsh@oleg105-client: oleg105-server: ssh exited with exit code 2