== conf-sanity test 69: replace an OST with the same index ========================================================== 16:06:03 (1710101163) start mds service on oleg251-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg251-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg251-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-OST0000 oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg251-client.virtnet: -o user_xattr,flock oleg251-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11840 inodes available. Want 99936. rc=0 - open/close 951 (time 1710101210.69 total 10.01 last 95.05) - open/close 1901 (time 1710101220.69 total 20.01 last 95.00) - open/close 2691 (time 1710101230.69 total 30.01 last 78.98) - open/close 3466 (time 1710101240.70 total 40.02 last 77.44) - open/close 4237 (time 1710101250.72 total 50.04 last 76.93) - open/close 4995 (time 1710101260.72 total 60.04 last 75.77) - open/close 5713 (time 1710101270.73 total 70.05 last 71.71) - open/close 6402 (time 1710101280.75 total 80.07 last 68.79) - open/close 7066 (time 1710101290.76 total 90.08 last 66.31) - open/close 7637 (time 1710101300.77 total 100.09 last 57.04) - open/close 8143 (time 1710101310.79 total 110.11 last 50.53) - open/close 8683 (time 1710101320.81 total 120.13 last 53.90) - open/close 9247 (time 1710101330.82 total 130.13 last 56.36) - open/close 9751 (time 1710101340.82 total 140.14 last 50.36) total: 10000 open/close in 145.76 seconds: 68.61 ops/second - unlinked 0 (time 1710101348 ; total 0 ; last 0) total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second - open/close 1059 (time 1710101455.16 total 10.01 last 105.82) - open/close 1980 (time 1710101465.16 total 20.01 last 92.09) - open/close 2850 (time 1710101475.16 total 30.01 last 87.00) - open/close 3658 (time 1710101485.17 total 40.02 last 80.71) - open/close 4398 (time 1710101495.18 total 50.03 last 73.95) - open/close 5137 (time 1710101505.18 total 60.03 last 73.87) - open/close 5763 (time 1710101515.19 total 70.04 last 62.55) - open/close 6398 (time 1710101525.20 total 80.05 last 63.45) - open/close 7080 (time 1710101535.21 total 90.06 last 68.12) - open/close 7744 (time 1710101545.22 total 100.07 last 66.35) - open/close 8383 (time 1710101555.22 total 110.07 last 63.88) - open/close 8976 (time 1710101565.23 total 120.08 last 59.23) - open/close 9609 (time 1710101575.26 total 130.11 last 63.15) - open/close 9777 (time 1710101589.59 total 144.44 last 11.72) total: 10000 open/close in 148.38 seconds: 67.39 ops/second - unlinked 0 (time 1710101595 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 1076 (time 1710101697.55 total 10.01 last 107.51) - open/close 2034 (time 1710101707.55 total 20.01 last 95.80) - open/close 2039 (time 1710101718.63 total 31.09 last 0.45) - open/close 2877 (time 1710101728.63 total 41.09 last 83.78) - open/close 3748 (time 1710101738.64 total 51.09 last 87.06) - open/close 4304 (time 1710101757.34 total 69.80 last 29.72) - open/close 5031 (time 1710101767.36 total 79.81 last 72.61) - open/close 5811 (time 1710101777.36 total 89.82 last 77.99) - open/close 6519 (time 1710101787.38 total 99.83 last 70.67) - open/close 7167 (time 1710101797.38 total 109.84 last 64.78) - open/close 7786 (time 1710101807.38 total 119.84 last 61.88) - open/close 8390 (time 1710101817.39 total 129.85 last 60.35) - open/close 8957 (time 1710101827.40 total 139.86 last 56.65) - open/close 9557 (time 1710101837.40 total 149.86 last 60.00) total: 10000 open/close in 157.31 seconds: 63.57 ops/second - unlinked 0 (time 1710101847 ; total 0 ; last 0) total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second - open/close 965 (time 1710101956.34 total 10.00 last 96.46) - open/close 1786 (time 1710101966.34 total 20.01 last 82.07) - open/close 2658 (time 1710101976.35 total 30.02 last 87.11) - open/close 3394 (time 1710101986.36 total 40.02 last 73.56) - open/close 4204 (time 1710101996.37 total 50.03 last 80.93) - open/close 4918 (time 1710102006.37 total 60.03 last 71.38) - open/close 5639 (time 1710102016.38 total 70.05 last 72.02) - open/close 6286 (time 1710102026.39 total 80.05 last 64.66) - open/close 6903 (time 1710102036.39 total 90.05 last 61.68) - open/close 7557 (time 1710102046.40 total 100.07 last 65.32) - open/close 8172 (time 1710102056.42 total 110.08 last 61.40) - open/close 8753 (time 1710102066.42 total 120.09 last 58.09) - open/close 9368 (time 1710102076.44 total 130.10 last 61.41) - open/close 9975 (time 1710102086.45 total 140.12 last 60.61) total: 10000 open/close in 140.58 seconds: 71.14 ops/second - unlinked 0 (time 1710102088 ; total 0 ; last 0) total: 10000 unlinks in 102 seconds: 98.039215 unlinks/second - open/close 883 (time 1710102212.05 total 19.80 last 44.59) - open/close 1820 (time 1710102222.05 total 29.81 last 93.66) - open/close 2807 (time 1710102232.07 total 39.82 last 98.59) - open/close 3582 (time 1710102242.08 total 49.83 last 77.40) - open/close 4457 (time 1710102252.09 total 59.84 last 87.41) - open/close 5239 (time 1710102262.10 total 69.85 last 78.14) - open/close 5930 (time 1710102272.10 total 79.85 last 69.08) - open/close 6601 (time 1710102282.11 total 89.86 last 67.03) - open/close 7239 (time 1710102292.12 total 99.87 last 63.74) - open/close 7883 (time 1710102302.13 total 109.88 last 64.33) - open/close 8546 (time 1710102312.14 total 119.89 last 66.26) - open/close 9161 (time 1710102322.14 total 129.89 last 61.49) - open/close 9741 (time 1710102332.16 total 139.91 last 57.89) total: 10000 open/close in 144.37 seconds: 69.27 ops/second - unlinked 0 (time 1710102338 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 922 (time 1710102442.14 total 19.54 last 47.19) - open/close 1942 (time 1710102452.15 total 29.55 last 101.86) - open/close 2883 (time 1710102462.16 total 39.56 last 94.03) - open/close 3732 (time 1710102472.17 total 49.57 last 84.82) - open/close 4575 (time 1710102482.17 total 59.57 last 84.28) - open/close 5421 (time 1710102492.17 total 69.57 last 84.59) - open/close 6223 (time 1710102502.17 total 79.57 last 80.17) - open/close 6948 (time 1710102512.18 total 89.58 last 72.45) - open/close 7686 (time 1710102522.19 total 99.59 last 73.75) - open/close 8378 (time 1710102532.19 total 109.59 last 69.17) - open/close 9059 (time 1710102542.20 total 119.60 last 68.06) - open/close 9700 (time 1710102552.25 total 129.65 last 63.77) total: 10000 open/close in 134.33 seconds: 74.45 ops/second - unlinked 0 (time 1710102558 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 1036 (time 1710102651.76 total 10.01 last 103.52) - open/close 2016 (time 1710102661.76 total 20.01 last 97.97) - open/close 2936 (time 1710102671.77 total 30.02 last 91.92) - open/close 3869 (time 1710102681.77 total 40.02 last 93.26) - open/close 4712 (time 1710102691.78 total 50.03 last 84.24) - open/close 5638 (time 1710102701.78 total 60.04 last 92.56) - open/close 6400 (time 1710102711.79 total 70.05 last 76.12) - open/close 7151 (time 1710102721.80 total 80.05 last 75.08) - open/close 7847 (time 1710102731.80 total 90.06 last 69.54) - open/close 8575 (time 1710102741.81 total 100.06 last 72.75) - open/close 9200 (time 1710102751.83 total 110.08 last 62.38) - open/close 9865 (time 1710102761.85 total 120.10 last 66.40) total: 10000 open/close in 122.13 seconds: 81.88 ops/second - unlinked 0 (time 1710102765 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 1299 (time 1710102850.57 total 10.00 last 129.84) - open/close 2445 (time 1710102860.58 total 20.01 last 114.55) - open/close 3632 (time 1710102870.59 total 30.02 last 118.56) - open/close 4764 (time 1710102880.59 total 40.02 last 113.16) - open/close 5792 (time 1710102890.59 total 50.03 last 102.78) - open/close 6784 (time 1710102900.60 total 60.03 last 99.17) - open/close 7738 (time 1710102910.60 total 70.03 last 95.38) - open/close 8718 (time 1710102920.61 total 80.04 last 97.94) - open/close 9659 (time 1710102930.61 total 90.04 last 94.08) total: 10000 open/close in 93.84 seconds: 106.56 ops/second - unlinked 0 (time 1710102935 ; total 0 ; last 0) total: 10000 unlinks in 50 seconds: 200.000000 unlinks/second - open/close 2630 (time 1710102996.02 total 10.00 last 262.99) - open/close 4558 (time 1710103006.03 total 20.00 last 192.71) - open/close 6275 (time 1710103016.03 total 30.01 last 171.66) - open/close 7690 (time 1710103026.03 total 40.01 last 141.48) - open/close 9024 (time 1710103036.03 total 50.01 last 133.39) total: 10000 open/close in 57.31 seconds: 174.50 ops/second - unlinked 0 (time 1710103044 ; total 0 ; last 0) total: 10000 unlinks in 34 seconds: 294.117645 unlinks/second - open/close 3206 (time 1710103089.20 total 10.00 last 320.50) - open/close 5842 (time 1710103099.20 total 20.01 last 263.54) - open/close 8168 (time 1710103109.20 total 30.01 last 232.53) total: 10000 open/close in 37.75 seconds: 264.87 ops/second - unlinked 0 (time 1710103117 ; total 0 ; last 0) total: 10000 unlinks in 26 seconds: 384.615387 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg251-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg251-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg251-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.151@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.151@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg251-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-OST0000 oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg251-server: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg251-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec oleg251-server: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg251-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg251-client.virtnet: -o user_xattr,flock oleg251-server@tcp:/lustre /mnt/lustre On OST0, 20413 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg251-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg251-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg251-server stop mds service on oleg251-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg251-server stop mds service on oleg251-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg251-server unloading modules on: 'oleg251-server' oleg251-server: oleg251-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg251-client: oleg251-client: ssh exited with exit code 2 pdsh@oleg251-client: oleg251-server: ssh exited with exit code 2 pdsh@oleg251-client: oleg251-client: ssh exited with exit code 2 pdsh@oleg251-client: oleg251-server: ssh exited with exit code 2