== conf-sanity test 69: replace an OST with the same index ========================================================== 16:03:10 (1710100990) start mds service on oleg451-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg451-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg451-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-OST0000 oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg451-client.virtnet: -o user_xattr,flock oleg451-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49440 inodes available. Want 99872. rc=0 - open/close 1016 (time 1710101031.01 total 10.00 last 101.56) - open/close 1976 (time 1710101041.01 total 20.00 last 96.00) - open/close 2792 (time 1710101051.02 total 30.01 last 81.54) - open/close 3593 (time 1710101061.04 total 40.03 last 79.99) - open/close 4388 (time 1710101071.05 total 50.04 last 79.35) - open/close 5075 (time 1710101081.06 total 60.05 last 68.64) - open/close 5777 (time 1710101091.08 total 70.07 last 70.10) - open/close 6441 (time 1710101101.10 total 80.09 last 66.25) - open/close 7074 (time 1710101111.10 total 90.09 last 63.28) - open/close 7687 (time 1710101121.11 total 100.10 last 61.24) - open/close 8327 (time 1710101131.13 total 110.12 last 63.90) - open/close 8879 (time 1710101141.14 total 120.13 last 55.17) - open/close 9469 (time 1710101151.14 total 130.13 last 58.99) - open/close 9966 (time 1710101161.14 total 140.13 last 49.68) total: 10000 open/close in 140.76 seconds: 71.04 ops/second - unlinked 0 (time 1710101163 ; total 0 ; last 0) total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second - open/close 856 (time 1710101270.35 total 10.01 last 85.53) - open/close 1735 (time 1710101280.35 total 20.01 last 87.87) - open/close 2492 (time 1710101290.36 total 30.02 last 75.62) - open/close 3267 (time 1710101300.37 total 40.03 last 77.47) - open/close 4056 (time 1710101310.37 total 50.03 last 78.85) - open/close 4746 (time 1710101320.39 total 60.04 last 68.93) - open/close 4962 (time 1710101334.61 total 74.26 last 15.19) - open/close 5689 (time 1710101344.61 total 84.27 last 72.69) - open/close 6358 (time 1710101354.62 total 94.28 last 66.82) - open/close 7002 (time 1710101364.62 total 104.28 last 64.37) - open/close 7599 (time 1710101374.63 total 114.29 last 59.64) - open/close 7704 (time 1710101387.59 total 127.24 last 8.11) - open/close 8333 (time 1710101397.60 total 137.26 last 62.82) - open/close 8881 (time 1710101407.69 total 147.34 last 54.32) - open/close 9455 (time 1710101417.69 total 157.35 last 57.38) total: 10000 open/close in 166.78 seconds: 59.96 ops/second - unlinked 0 (time 1710101429 ; total 0 ; last 0) total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second - open/close 985 (time 1710101536.35 total 10.01 last 98.43) - open/close 1894 (time 1710101546.36 total 20.01 last 90.83) - open/close 2687 (time 1710101556.36 total 30.02 last 79.29) - open/close 3489 (time 1710101566.37 total 40.02 last 80.15) - open/close 4289 (time 1710101576.37 total 50.03 last 79.97) - open/close 5017 (time 1710101586.38 total 60.03 last 72.76) - open/close 5695 (time 1710101596.39 total 70.04 last 67.73) - open/close 6318 (time 1710101606.40 total 80.06 last 62.20) - open/close 6395 (time 1710101618.69 total 92.34 last 6.27) - open/close 7014 (time 1710101628.69 total 102.35 last 61.88) - open/close 7649 (time 1710101638.70 total 112.36 last 63.44) - open/close 8219 (time 1710101648.71 total 122.37 last 56.93) - open/close 8786 (time 1710101658.72 total 132.38 last 56.66) - open/close 9343 (time 1710101668.74 total 142.40 last 55.60) - open/close 9906 (time 1710101678.75 total 152.40 last 56.26) total: 10000 open/close in 154.21 seconds: 64.85 ops/second - unlinked 0 (time 1710101682 ; total 0 ; last 0) total: 10000 unlinks in 102 seconds: 98.039215 unlinks/second - open/close 1035 (time 1710101796.40 total 10.00 last 103.46) - open/close 1892 (time 1710101806.40 total 20.00 last 85.69) - open/close 2742 (time 1710101816.41 total 30.01 last 84.97) - open/close 2870 (time 1710101829.17 total 42.77 last 10.03) - open/close 3640 (time 1710101839.17 total 52.77 last 77.00) - open/close 4429 (time 1710101849.18 total 62.78 last 78.86) - open/close 5136 (time 1710101859.21 total 72.81 last 70.46) - open/close 5874 (time 1710101869.22 total 82.82 last 73.74) - open/close 6602 (time 1710101879.23 total 92.83 last 72.74) - open/close 7224 (time 1710101889.23 total 102.83 last 62.19) - open/close 7812 (time 1710101899.27 total 112.87 last 58.55) - open/close 8330 (time 1710101909.30 total 122.90 last 51.67) - open/close 8842 (time 1710101919.31 total 132.91 last 51.15) - open/close 9398 (time 1710101929.31 total 142.92 last 55.57) - open/close 9877 (time 1710101949.68 total 163.28 last 23.52) total: 10000 open/close in 165.58 seconds: 60.39 ops/second - unlinked 0 (time 1710101954 ; total 0 ; last 0) total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second - open/close 926 (time 1710102055.72 total 10.01 last 92.50) - open/close 1844 (time 1710102065.73 total 20.02 last 91.75) - open/close 2668 (time 1710102075.74 total 30.02 last 82.35) - open/close 3469 (time 1710102085.74 total 40.03 last 80.04) - open/close 4270 (time 1710102095.75 total 50.03 last 80.07) - open/close 5021 (time 1710102105.76 total 60.05 last 75.00) - open/close 5757 (time 1710102115.77 total 70.06 last 73.53) - open/close 6494 (time 1710102125.77 total 80.06 last 73.67) - open/close 7197 (time 1710102135.78 total 90.07 last 70.24) - open/close 7287 (time 1710102148.27 total 102.55 last 7.21) - open/close 7961 (time 1710102158.27 total 112.56 last 67.35) - open/close 8565 (time 1710102168.29 total 122.57 last 60.33) - open/close 9206 (time 1710102178.31 total 132.60 last 63.94) - open/close 9711 (time 1710102188.32 total 142.60 last 50.46) total: 10000 open/close in 148.46 seconds: 67.36 ops/second - unlinked 0 (time 1710102196 ; total 0 ; last 0) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second - open/close 954 (time 1710102299.45 total 10.02 last 95.25) - open/close 1894 (time 1710102309.45 total 20.02 last 93.99) - open/close 2742 (time 1710102319.47 total 30.03 last 84.65) - open/close 3525 (time 1710102329.48 total 40.05 last 78.21) - open/close 4320 (time 1710102339.49 total 50.05 last 79.44) - open/close 5030 (time 1710102349.50 total 60.06 last 70.94) - open/close 5720 (time 1710102359.50 total 70.06 last 68.98) - open/close 6384 (time 1710102369.51 total 80.07 last 66.35) - open/close 6853 (time 1710102388.28 total 98.85 last 24.98) - open/close 7488 (time 1710102398.29 total 108.85 last 63.46) - open/close 8051 (time 1710102408.29 total 118.86 last 56.26) - open/close 8604 (time 1710102418.30 total 128.87 last 55.26) - open/close 8979 (time 1710102436.34 total 146.91 last 20.79) - open/close 9473 (time 1710102456.02 total 166.58 last 25.11) total: 10000 open/close in 176.07 seconds: 56.80 ops/second - unlinked 0 (time 1710102467 ; total 0 ; last 0) total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second - open/close 1051 (time 1710102568.44 total 10.00 last 105.09) - open/close 2064 (time 1710102578.45 total 20.01 last 101.22) - open/close 3012 (time 1710102588.45 total 30.01 last 94.75) - open/close 3817 (time 1710102598.45 total 40.01 last 80.49) - open/close 4626 (time 1710102608.49 total 50.05 last 80.59) - open/close 5380 (time 1710102618.49 total 60.06 last 75.39) - open/close 6113 (time 1710102628.50 total 70.06 last 73.29) - open/close 6822 (time 1710102638.51 total 80.07 last 70.83) - open/close 7537 (time 1710102648.52 total 90.08 last 71.41) - open/close 8195 (time 1710102658.52 total 100.08 last 65.80) - open/close 8807 (time 1710102668.53 total 110.09 last 61.15) - open/close 9479 (time 1710102678.53 total 120.10 last 67.15) total: 10000 open/close in 128.16 seconds: 78.02 ops/second - unlinked 0 (time 1710102688 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 1163 (time 1710102781.92 total 10.01 last 116.23) - open/close 2296 (time 1710102791.92 total 20.01 last 113.25) - open/close 3335 (time 1710102801.93 total 30.01 last 103.88) - open/close 4317 (time 1710102811.93 total 40.01 last 98.20) - open/close 5265 (time 1710102821.93 total 50.02 last 94.75) - open/close 6153 (time 1710102831.93 total 60.02 last 88.78) - open/close 6990 (time 1710102841.95 total 70.03 last 83.61) - open/close 7743 (time 1710102852.01 total 80.09 last 74.85) - open/close 8492 (time 1710102862.01 total 90.09 last 74.89) - open/close 9176 (time 1710102872.01 total 100.10 last 68.36) - open/close 9919 (time 1710102882.02 total 110.11 last 74.25) total: 10000 open/close in 111.24 seconds: 89.90 ops/second - unlinked 0 (time 1710102884 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second - open/close 1618 (time 1710102955.48 total 10.00 last 161.74) - open/close 3261 (time 1710102965.48 total 20.00 last 164.30) - open/close 4845 (time 1710102975.48 total 30.01 last 158.31) - open/close 6222 (time 1710102985.48 total 40.01 last 137.70) - open/close 7519 (time 1710102995.49 total 50.02 last 129.62) - open/close 8808 (time 1710103005.49 total 60.02 last 128.82) - open/close 9951 (time 1710103015.50 total 70.02 last 114.30) total: 10000 open/close in 70.46 seconds: 141.92 ops/second - unlinked 0 (time 1710103017 ; total 0 ; last 0) total: 10000 unlinks in 40 seconds: 250.000000 unlinks/second - open/close 2455 (time 1710103068.15 total 10.00 last 245.45) - open/close 4432 (time 1710103078.15 total 20.01 last 197.61) - open/close 6320 (time 1710103088.15 total 30.01 last 188.80) - open/close 8061 (time 1710103098.15 total 40.01 last 174.04) - open/close 9610 (time 1710103108.16 total 50.01 last 154.88) total: 10000 open/close in 52.80 seconds: 189.40 ops/second - unlinked 0 (time 1710103111 ; total 0 ; last 0) total: 10000 unlinks in 30 seconds: 333.333344 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg451-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg451-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg451-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.151@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg451-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-OST0000 oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg451-server: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg451-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 16 sec oleg451-server: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg451-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg451-client.virtnet: -o user_xattr,flock oleg451-server@tcp:/lustre /mnt/lustre On OST0, 17573 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg451-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg451-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg451-server stop mds service on oleg451-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg451-server stop mds service on oleg451-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg451-server unloading modules on: 'oleg451-server' oleg451-server: oleg451-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg451-client: oleg451-server: ssh exited with exit code 2 pdsh@oleg451-client: oleg451-client: ssh exited with exit code 2 pdsh@oleg451-client: oleg451-client: ssh exited with exit code 2 pdsh@oleg451-client: oleg451-server: ssh exited with exit code 2