== conf-sanity test 69: replace an OST with the same index ========================================================== 16:00:30 (1710100830) start mds service on oleg116-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg116-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Started lustre-OST0000 oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg116-client.virtnet: -o user_xattr,flock oleg116-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99936. rc=0 - open/close 227 (time 1710100861.18 total 13.07 last 17.37) - open/close 1277 (time 1710100871.19 total 23.08 last 104.90) - open/close 2195 (time 1710100881.19 total 33.08 last 91.79) - open/close 3139 (time 1710100891.20 total 43.09 last 94.26) - open/close 3991 (time 1710100901.20 total 53.09 last 85.20) - open/close 4698 (time 1710100911.21 total 63.10 last 70.67) - open/close 5421 (time 1710100921.21 total 73.10 last 72.30) - open/close 6158 (time 1710100931.21 total 83.11 last 73.65) - open/close 6878 (time 1710100941.23 total 93.12 last 71.91) - open/close 7600 (time 1710100951.24 total 103.13 last 72.10) - open/close 8293 (time 1710100961.25 total 113.14 last 69.26) - open/close 8893 (time 1710100971.26 total 123.15 last 59.91) - open/close 9487 (time 1710100981.31 total 133.20 last 59.10) total: 10000 open/close in 143.04 seconds: 69.91 ops/second - unlinked 0 (time 1710100993 ; total 0 ; last 0) total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second - open/close 1054 (time 1710101094.25 total 10.00 last 105.36) - open/close 2028 (time 1710101104.25 total 20.00 last 97.39) - open/close 2878 (time 1710101114.26 total 30.01 last 84.92) - open/close 3598 (time 1710101124.26 total 40.02 last 71.97) - open/close 4205 (time 1710101134.27 total 50.03 last 60.63) - open/close 4821 (time 1710101144.28 total 60.03 last 61.58) - open/close 5487 (time 1710101154.28 total 70.03 last 66.58) - open/close 6113 (time 1710101164.29 total 80.04 last 62.55) - open/close 6675 (time 1710101174.30 total 90.05 last 56.14) - open/close 7220 (time 1710101184.31 total 100.06 last 54.44) - open/close 7817 (time 1710101194.31 total 110.07 last 59.70) - open/close 8312 (time 1710101204.33 total 120.09 last 49.39) - open/close 8817 (time 1710101214.34 total 130.09 last 50.47) - open/close 9288 (time 1710101224.34 total 140.10 last 47.08) - open/close 9706 (time 1710101234.35 total 150.11 last 41.76) total: 10000 open/close in 157.07 seconds: 63.66 ops/second - unlinked 0 (time 1710101243 ; total 0 ; last 0) total: 10000 unlinks in 116 seconds: 86.206894 unlinks/second - open/close 809 (time 1710101372.08 total 10.00 last 80.89) - open/close 1588 (time 1710101382.10 total 20.01 last 77.80) - open/close 2336 (time 1710101392.11 total 30.02 last 74.72) - open/close 3040 (time 1710101402.11 total 40.03 last 70.38) - open/close 3679 (time 1710101412.12 total 50.04 last 63.84) - open/close 4266 (time 1710101422.14 total 60.06 last 58.59) - open/close 4885 (time 1710101432.15 total 70.07 last 61.83) - open/close 5450 (time 1710101442.15 total 80.07 last 56.48) - open/close 6033 (time 1710101452.16 total 90.08 last 58.23) - open/close 6546 (time 1710101462.21 total 100.12 last 51.09) - open/close 7036 (time 1710101472.21 total 110.13 last 48.96) - open/close 7546 (time 1710101482.24 total 120.16 last 50.87) - open/close 8066 (time 1710101492.26 total 130.18 last 51.87) - open/close 8533 (time 1710101502.27 total 140.19 last 46.66) - open/close 8986 (time 1710101512.29 total 150.21 last 45.20) - open/close 9340 (time 1710101530.56 total 168.48 last 19.38) - open/close 9728 (time 1710101540.57 total 178.49 last 38.76) total: 10000 open/close in 184.80 seconds: 54.11 ops/second - unlinked 0 (time 1710101549 ; total 0 ; last 0) total: 10000 unlinks in 115 seconds: 86.956520 unlinks/second - open/close 830 (time 1710101677.10 total 10.00 last 83.00) - open/close 1701 (time 1710101687.10 total 20.00 last 87.06) - open/close 2455 (time 1710101697.12 total 30.02 last 75.30) - open/close 3175 (time 1710101707.17 total 40.07 last 71.60) - open/close 3842 (time 1710101717.21 total 50.11 last 66.47) - open/close 4423 (time 1710101727.21 total 60.11 last 58.09) - open/close 5029 (time 1710101737.21 total 70.12 last 60.55) - open/close 5595 (time 1710101747.22 total 80.12 last 56.59) - open/close 6148 (time 1710101757.23 total 90.14 last 55.20) - open/close 6685 (time 1710101767.27 total 100.17 last 53.53) - open/close 7219 (time 1710101777.28 total 110.18 last 53.35) - open/close 7765 (time 1710101787.29 total 120.19 last 54.54) - open/close 8265 (time 1710101797.30 total 130.21 last 49.92) - open/close 8748 (time 1710101807.41 total 140.31 last 47.80) - open/close 9205 (time 1710101817.43 total 150.33 last 45.61) - open/close 9632 (time 1710101827.43 total 160.34 last 42.68) total: 10000 open/close in 169.17 seconds: 59.11 ops/second - unlinked 0 (time 1710101838 ; total 0 ; last 0) total: 10000 unlinks in 111 seconds: 90.090088 unlinks/second - open/close 876 (time 1710101961.99 total 10.00 last 87.59) - open/close 1659 (time 1710101971.99 total 20.00 last 78.29) - open/close 2372 (time 1710101981.99 total 30.00 last 71.29) - open/close 3108 (time 1710101992.01 total 40.02 last 73.48) - open/close 3781 (time 1710102002.01 total 50.03 last 67.26) - open/close 4403 (time 1710102012.03 total 60.04 last 62.12) - open/close 5006 (time 1710102022.04 total 70.05 last 60.22) - open/close 5482 (time 1710102041.88 total 89.90 last 23.99) - open/close 6076 (time 1710102051.89 total 99.90 last 59.37) - open/close 6576 (time 1710102061.89 total 109.90 last 49.99) - open/close 7152 (time 1710102071.90 total 119.91 last 57.54) - open/close 7648 (time 1710102081.92 total 129.93 last 49.50) - open/close 8157 (time 1710102091.92 total 139.94 last 50.89) - open/close 8625 (time 1710102101.96 total 149.97 last 46.65) - open/close 9031 (time 1710102111.97 total 159.98 last 40.56) - open/close 9479 (time 1710102121.97 total 169.99 last 44.77) - open/close 9954 (time 1710102131.98 total 180.00 last 47.45) total: 10000 open/close in 180.79 seconds: 55.31 ops/second - unlinked 0 (time 1710102135 ; total 0 ; last 0) total: 10000 unlinks in 139 seconds: 71.942444 unlinks/second - open/close 869 (time 1710102287.42 total 10.01 last 86.85) - open/close 1621 (time 1710102297.43 total 20.02 last 75.10) - open/close 2367 (time 1710102307.44 total 30.03 last 74.54) - open/close 3102 (time 1710102317.45 total 40.03 last 73.45) - open/close 3699 (time 1710102327.46 total 50.04 last 59.65) - open/close 4320 (time 1710102337.47 total 60.05 last 62.02) - open/close 4936 (time 1710102347.47 total 70.06 last 61.59) - open/close 5534 (time 1710102357.49 total 80.07 last 59.72) - open/close 6133 (time 1710102367.50 total 90.08 last 59.83) - open/close 6726 (time 1710102377.50 total 100.09 last 59.26) - open/close 7189 (time 1710102387.51 total 110.09 last 46.28) - open/close 7625 (time 1710102397.52 total 120.10 last 43.56) - open/close 8113 (time 1710102407.53 total 130.11 last 48.76) - open/close 8595 (time 1710102417.53 total 140.11 last 48.18) - open/close 9084 (time 1710102427.54 total 150.13 last 48.83) - open/close 9611 (time 1710102437.56 total 160.14 last 52.63) total: 10000 open/close in 168.98 seconds: 59.18 ops/second - unlinked 0 (time 1710102449 ; total 0 ; last 0) total: 10000 unlinks in 124 seconds: 80.645164 unlinks/second - open/close 873 (time 1710102585.87 total 10.00 last 87.29) - open/close 1649 (time 1710102595.87 total 20.01 last 77.57) - open/close 2389 (time 1710102605.87 total 30.01 last 73.97) - open/close 3042 (time 1710102615.88 total 40.02 last 65.26) - open/close 3650 (time 1710102625.89 total 50.03 last 60.74) - open/close 4283 (time 1710102635.90 total 60.04 last 63.23) - open/close 4886 (time 1710102645.90 total 70.04 last 60.28) - open/close 5456 (time 1710102655.93 total 80.07 last 56.86) - open/close 6017 (time 1710102665.94 total 90.07 last 56.05) - open/close 6378 (time 1710102684.46 total 108.60 last 19.49) - open/close 6896 (time 1710102694.50 total 118.64 last 51.59) - open/close 7363 (time 1710102704.52 total 128.65 last 46.62) - open/close 7820 (time 1710102714.53 total 138.66 last 45.66) - open/close 8263 (time 1710102724.54 total 148.67 last 44.25) - open/close 8697 (time 1710102734.56 total 158.69 last 43.31) - open/close 9187 (time 1710102744.57 total 168.71 last 48.92) - open/close 9623 (time 1710102754.58 total 178.71 last 43.59) total: 10000 open/close in 186.31 seconds: 53.67 ops/second - unlinked 0 (time 1710102765 ; total 0 ; last 0) total: 10000 unlinks in 124 seconds: 80.645164 unlinks/second - open/close 791 (time 1710102901.52 total 10.01 last 79.03) - open/close 1581 (time 1710102911.52 total 20.01 last 78.98) - open/close 2324 (time 1710102921.55 total 30.04 last 74.07) - open/close 3043 (time 1710102931.56 total 40.06 last 71.80) - open/close 3668 (time 1710102941.57 total 50.06 last 62.48) - open/close 4262 (time 1710102951.57 total 60.06 last 59.38) - open/close 4850 (time 1710102961.58 total 70.07 last 58.74) - open/close 5437 (time 1710102971.59 total 80.08 last 58.64) - open/close 5980 (time 1710102981.59 total 90.09 last 54.29) - open/close 6534 (time 1710102991.60 total 100.10 last 55.35) - open/close 6689 (time 1710103005.53 total 114.02 last 11.13) - open/close 7175 (time 1710103015.53 total 124.02 last 48.60) - open/close 7678 (time 1710103025.53 total 134.03 last 50.27) - open/close 7828 (time 1710103039.72 total 148.21 last 10.57) - open/close 8298 (time 1710103049.74 total 158.23 last 46.90) - open/close 8791 (time 1710103059.74 total 168.23 last 49.30) - open/close 9277 (time 1710103069.78 total 178.27 last 48.41) - open/close 9693 (time 1710103079.82 total 188.31 last 41.44) total: 10000 open/close in 195.19 seconds: 51.23 ops/second - unlinked 0 (time 1710103089 ; total 0 ; last 0) total: 10000 unlinks in 105 seconds: 95.238098 unlinks/second - open/close 915 (time 1710103206.41 total 10.01 last 91.43) - open/close 1742 (time 1710103216.42 total 20.01 last 82.66) - open/close 2561 (time 1710103226.42 total 30.02 last 81.88) - open/close 3256 (time 1710103236.42 total 40.02 last 69.49) - open/close 3981 (time 1710103246.43 total 50.02 last 72.44) - open/close 4649 (time 1710103256.44 total 60.03 last 66.74) - open/close 5344 (time 1710103266.44 total 70.03 last 69.49) - open/close 6051 (time 1710103276.45 total 80.04 last 70.66) - open/close 6690 (time 1710103286.46 total 90.05 last 63.81) - open/close 7296 (time 1710103296.47 total 100.06 last 60.56) - open/close 7921 (time 1710103306.48 total 110.07 last 62.45) - open/close 8541 (time 1710103316.48 total 120.08 last 61.95) - open/close 9141 (time 1710103326.49 total 130.08 last 59.98) - open/close 9737 (time 1710103336.49 total 140.09 last 59.57) total: 10000 open/close in 144.97 seconds: 68.98 ops/second - unlinked 0 (time 1710103343 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 1050 (time 1710103441.49 total 10.01 last 104.91) - open/close 2041 (time 1710103451.50 total 20.02 last 99.02) - open/close 2987 (time 1710103461.50 total 30.02 last 94.57) - open/close 3915 (time 1710103471.50 total 40.03 last 92.75) - open/close 4759 (time 1710103481.51 total 50.03 last 84.34) - open/close 5629 (time 1710103491.52 total 60.04 last 86.94) - open/close 6554 (time 1710103501.52 total 70.04 last 92.45) - open/close 7428 (time 1710103511.53 total 80.05 last 87.38) - open/close 8333 (time 1710103521.53 total 90.05 last 90.45) - open/close 9215 (time 1710103531.53 total 100.05 last 88.20) total: 10000 open/close in 109.46 seconds: 91.35 ops/second - unlinked 0 (time 1710103542 ; total 0 ; last 0) total: 10000 unlinks in 53 seconds: 188.679245 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg116-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg116-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg116-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.116@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.116@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg116-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Started lustre-OST0000 oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg116-server: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg116-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec mount lustre on /mnt/lustre..... Starting client: oleg116-client.virtnet: -o user_xattr,flock oleg116-server@tcp:/lustre /mnt/lustre On OST0, 17669 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg116-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg116-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg116-server stop mds service on oleg116-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg116-server unloading modules on: 'oleg116-server' oleg116-server: oleg116-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg116-client: oleg116-client: ssh exited with exit code 2 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 2 pdsh@oleg116-client: oleg116-client: ssh exited with exit code 2 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 2