== conf-sanity test 69: replace an OST with the same index ========================================================== 16:02:08 (1710100928) start mds service on oleg456-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg456-server: oleg456-server.virtnet: executing set_default_debug -1 all pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg456-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg456-server: oleg456-server.virtnet: executing set_default_debug -1 all pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg456-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg456-server: oleg456-server.virtnet: executing set_default_debug -1 all pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 Started lustre-OST0000 oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg456-client.virtnet: -o user_xattr,flock oleg456-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49440 inodes available. Want 99872. rc=0 - open/close 1632 (time 1710100963.74 total 10.01 last 163.10) - open/close 2810 (time 1710100973.74 total 20.01 last 117.78) - open/close 3914 (time 1710100983.75 total 30.01 last 110.32) - open/close 5029 (time 1710100993.75 total 40.01 last 111.50) - open/close 6081 (time 1710101003.76 total 50.02 last 105.11) - open/close 7090 (time 1710101013.76 total 60.03 last 100.87) - open/close 7893 (time 1710101023.76 total 70.03 last 80.29) - open/close 8554 (time 1710101033.78 total 80.04 last 66.01) - open/close 9246 (time 1710101043.78 total 90.05 last 69.15) - open/close 9867 (time 1710101053.79 total 100.05 last 62.07) total: 10000 open/close in 102.42 seconds: 97.64 ops/second - unlinked 0 (time 1710101058 ; total 0 ; last 0) total: 10000 unlinks in 114 seconds: 87.719299 unlinks/second - open/close 506 (time 1710101190.92 total 15.93 last 31.76) - open/close 1671 (time 1710101200.93 total 25.94 last 116.38) - open/close 2687 (time 1710101210.93 total 35.95 last 101.58) - open/close 3555 (time 1710101220.94 total 45.95 last 86.75) - open/close 4341 (time 1710101230.94 total 55.96 last 78.54) - open/close 4473 (time 1710101243.39 total 68.41 last 10.61) - open/close 4688 (time 1710101257.57 total 82.59 last 15.16) - open/close 5357 (time 1710101267.58 total 92.59 last 66.87) - open/close 6089 (time 1710101277.58 total 102.60 last 73.17) - open/close 6735 (time 1710101287.59 total 112.61 last 64.54) - open/close 7402 (time 1710101297.60 total 122.62 last 66.62) - open/close 8070 (time 1710101307.61 total 132.63 last 66.74) - open/close 8691 (time 1710101317.62 total 142.64 last 62.04) - open/close 9267 (time 1710101327.62 total 152.64 last 57.58) - open/close 9885 (time 1710101337.63 total 162.65 last 61.78) total: 10000 open/close in 164.48 seconds: 60.80 ops/second - unlinked 0 (time 1710101341 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 1163 (time 1710101437.37 total 10.01 last 116.22) - open/close 2258 (time 1710101447.37 total 20.01 last 109.49) - open/close 3215 (time 1710101457.38 total 30.02 last 95.61) - open/close 4077 (time 1710101467.38 total 40.02 last 86.19) - open/close 4883 (time 1710101477.38 total 50.02 last 80.57) - open/close 5626 (time 1710101487.40 total 60.04 last 74.15) - open/close 6337 (time 1710101497.41 total 70.05 last 71.01) - open/close 6987 (time 1710101507.43 total 80.07 last 64.93) - open/close 7401 (time 1710101524.23 total 96.87 last 24.63) - open/close 8063 (time 1710101534.24 total 106.88 last 66.14) - open/close 8687 (time 1710101544.24 total 116.88 last 62.38) - open/close 9284 (time 1710101554.25 total 126.88 last 59.70) - open/close 9910 (time 1710101564.25 total 136.89 last 62.58) total: 10000 open/close in 138.33 seconds: 72.29 ops/second - unlinked 0 (time 1710101568 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 1072 (time 1710101663.58 total 10.00 last 107.19) - open/close 2058 (time 1710101673.58 total 20.01 last 98.55) - open/close 2952 (time 1710101683.58 total 30.01 last 89.38) - open/close 3101 (time 1710101696.15 total 42.58 last 11.85) - open/close 3919 (time 1710101706.16 total 52.58 last 81.76) - open/close 4770 (time 1710101716.16 total 62.59 last 85.07) - open/close 5502 (time 1710101726.17 total 72.59 last 73.18) - open/close 6208 (time 1710101736.18 total 82.60 last 70.50) - open/close 6973 (time 1710101746.21 total 92.63 last 76.28) - open/close 7621 (time 1710101756.22 total 102.64 last 64.73) - open/close 8274 (time 1710101766.22 total 112.65 last 65.27) - open/close 8910 (time 1710101776.24 total 122.66 last 63.52) - open/close 9562 (time 1710101786.25 total 132.68 last 65.09) total: 10000 open/close in 140.25 seconds: 71.30 ops/second - unlinked 0 (time 1710101796 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 1096 (time 1710101888.82 total 10.00 last 109.58) - open/close 2163 (time 1710101898.82 total 20.00 last 106.69) - open/close 2867 (time 1710101908.83 total 30.01 last 70.36) - open/close 3642 (time 1710101918.83 total 40.01 last 77.48) - open/close 4500 (time 1710101928.84 total 50.02 last 85.76) - open/close 5273 (time 1710101938.84 total 60.02 last 77.26) - open/close 6003 (time 1710101948.84 total 70.02 last 72.98) - open/close 6694 (time 1710101958.85 total 80.03 last 69.08) - open/close 7445 (time 1710101968.85 total 90.03 last 75.10) - open/close 8099 (time 1710101978.88 total 100.06 last 65.18) - open/close 8735 (time 1710101988.89 total 110.06 last 63.58) - open/close 9291 (time 1710101998.89 total 120.07 last 55.56) - open/close 9386 (time 1710102011.37 total 132.55 last 7.61) - open/close 9989 (time 1710102021.38 total 142.56 last 60.27) total: 10000 open/close in 142.67 seconds: 70.09 ops/second - unlinked 0 (time 1710102023 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 1105 (time 1710102117.69 total 10.00 last 110.48) - open/close 2184 (time 1710102127.70 total 20.01 last 107.81) - open/close 3190 (time 1710102137.71 total 30.02 last 100.52) - open/close 4085 (time 1710102147.71 total 40.02 last 89.49) - open/close 4489 (time 1710102164.01 total 56.32 last 24.78) - open/close 5409 (time 1710102174.01 total 66.32 last 91.98) - open/close 6179 (time 1710102184.02 total 76.33 last 76.95) - open/close 6806 (time 1710102194.02 total 86.33 last 62.69) - open/close 7523 (time 1710102204.03 total 96.34 last 71.64) - open/close 8227 (time 1710102214.05 total 106.36 last 70.26) - open/close 8942 (time 1710102224.06 total 116.37 last 71.39) - open/close 9581 (time 1710102234.07 total 126.38 last 63.88) total: 10000 open/close in 132.96 seconds: 75.21 ops/second - unlinked 0 (time 1710102242 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second - open/close 1060 (time 1710102346.09 total 10.00 last 105.95) - open/close 2028 (time 1710102356.09 total 20.01 last 96.77) - open/close 2944 (time 1710102366.12 total 30.04 last 91.33) - open/close 3793 (time 1710102376.13 total 40.05 last 84.80) - open/close 4681 (time 1710102386.17 total 50.09 last 88.45) - open/close 5436 (time 1710102396.18 total 60.10 last 75.44) - open/close 6203 (time 1710102406.18 total 70.10 last 76.70) - open/close 6863 (time 1710102416.18 total 80.10 last 65.99) - open/close 7528 (time 1710102436.87 total 100.78 last 32.15) - open/close 8235 (time 1710102446.87 total 110.79 last 70.64) - open/close 8910 (time 1710102456.89 total 120.80 last 67.42) - open/close 9479 (time 1710102476.96 total 140.87 last 28.35) total: 10000 open/close in 148.48 seconds: 67.35 ops/second - unlinked 0 (time 1710102486 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 1153 (time 1710102588.26 total 10.00 last 115.29) - open/close 2124 (time 1710102598.27 total 20.01 last 97.02) - open/close 3102 (time 1710102608.28 total 30.02 last 97.70) - open/close 3983 (time 1710102618.28 total 40.02 last 88.10) - open/close 4833 (time 1710102628.29 total 50.03 last 84.90) - open/close 5652 (time 1710102638.31 total 60.04 last 81.81) - open/close 6485 (time 1710102648.32 total 70.05 last 83.21) - open/close 7275 (time 1710102658.32 total 80.06 last 78.96) - open/close 8012 (time 1710102668.32 total 90.06 last 73.68) - open/close 8841 (time 1710102678.33 total 100.07 last 82.84) - open/close 9612 (time 1710102688.34 total 110.08 last 77.04) - open/close 9835 (time 1710102702.62 total 124.36 last 15.61) - open/close 9975 (time 1710102715.61 total 137.35 last 10.78) total: 10000 open/close in 137.76 seconds: 72.59 ops/second - unlinked 0 (time 1710102717 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 1335 (time 1710102800.74 total 10.00 last 133.50) - open/close 2582 (time 1710102810.75 total 20.01 last 124.60) - open/close 3778 (time 1710102820.76 total 30.01 last 119.56) - open/close 4841 (time 1710102830.76 total 40.01 last 106.28) - open/close 5864 (time 1710102840.76 total 50.02 last 102.28) - open/close 6758 (time 1710102850.77 total 60.03 last 89.30) - open/close 7649 (time 1710102860.77 total 70.03 last 89.08) - open/close 8478 (time 1710102870.78 total 80.03 last 82.89) - open/close 9366 (time 1710102880.78 total 90.04 last 88.75) total: 10000 open/close in 98.00 seconds: 102.04 ops/second - unlinked 0 (time 1710102890 ; total 0 ; last 0) total: 10000 unlinks in 53 seconds: 188.679245 unlinks/second - open/close 1708 (time 1710102954.80 total 10.00 last 170.74) - open/close 3339 (time 1710102964.80 total 20.00 last 163.10) - open/close 4915 (time 1710102974.80 total 30.00 last 157.60) - open/close 6328 (time 1710102984.80 total 40.01 last 141.25) - open/close 7621 (time 1710102994.81 total 50.02 last 129.20) - open/close 8862 (time 1710103004.82 total 60.02 last 124.03) - open/close 9995 (time 1710103014.82 total 70.03 last 113.22) total: 10000 open/close in 70.07 seconds: 142.72 ops/second - unlinked 0 (time 1710103015 ; total 0 ; last 0) total: 10000 unlinks in 41 seconds: 243.902435 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg456-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg456-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg456-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.156@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg456-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg456-server: oleg456-server.virtnet: executing set_default_debug -1 all pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 Started lustre-OST0000 oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg456-server: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg456-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 16 sec oleg456-server: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg456-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg456-client.virtnet: -o user_xattr,flock oleg456-server@tcp:/lustre /mnt/lustre On OST0, 20429 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg456-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg456-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg456-server stop mds service on oleg456-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg456-server stop mds service on oleg456-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg456-server unloading modules on: 'oleg456-server' oleg456-server: oleg456-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg456-client: oleg456-client: ssh exited with exit code 2 pdsh@oleg456-client: oleg456-server: ssh exited with exit code 2 pdsh@oleg456-client: oleg456-client: ssh exited with exit code 2 pdsh@oleg456-client: oleg456-server: ssh exited with exit code 2