== conf-sanity test 69: replace an OST with the same index ========================================================== 16:07:31 (1710101251) start mds service on oleg133-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg133-server: oleg133-server.virtnet: executing set_default_debug -1 all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg133-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg133-server: oleg133-server.virtnet: executing set_default_debug -1 all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg133-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg133-server: oleg133-server.virtnet: executing set_default_debug -1 all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Started lustre-OST0000 oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg133-client.virtnet: -o user_xattr,flock oleg133-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11840 inodes available. Want 99936. rc=0 - open/close 845 (time 1710101303.43 total 10.01 last 84.46) - open/close 1612 (time 1710101313.44 total 20.02 last 76.62) - open/close 2407 (time 1710101323.44 total 30.02 last 79.49) - open/close 3060 (time 1710101333.44 total 40.02 last 65.26) - open/close 3806 (time 1710101343.44 total 50.02 last 74.59) - open/close 4427 (time 1710101353.45 total 60.03 last 62.05) - open/close 5053 (time 1710101363.45 total 70.03 last 62.58) - open/close 5668 (time 1710101373.46 total 80.04 last 61.46) - open/close 5968 (time 1710101389.51 total 96.09 last 18.69) - open/close 6140 (time 1710101403.78 total 110.36 last 12.05) - open/close 6408 (time 1710101421.35 total 127.93 last 15.26) - open/close 6940 (time 1710101431.36 total 137.94 last 53.14) - open/close 7461 (time 1710101441.36 total 147.94 last 52.08) - open/close 8042 (time 1710101451.37 total 157.95 last 58.08) - open/close 8577 (time 1710101461.37 total 167.95 last 53.47) - open/close 9076 (time 1710101471.38 total 177.96 last 49.87) - open/close 9535 (time 1710101481.38 total 187.96 last 45.88) total: 10000 open/close in 197.50 seconds: 50.63 ops/second - unlinked 0 (time 1710101493 ; total 0 ; last 0) total: 10000 unlinks in 103 seconds: 97.087379 unlinks/second - open/close 891 (time 1710101608.75 total 10.01 last 89.05) - open/close 1684 (time 1710101618.75 total 20.01 last 79.30) - open/close 2423 (time 1710101628.76 total 30.02 last 73.79) - open/close 3171 (time 1710101638.78 total 40.03 last 74.72) - open/close 3847 (time 1710101648.79 total 50.05 last 67.49) - open/close 4406 (time 1710101658.80 total 60.06 last 55.83) - open/close 5029 (time 1710101668.81 total 70.06 last 62.28) - open/close 5611 (time 1710101678.81 total 80.07 last 58.19) - open/close 6248 (time 1710101688.81 total 90.07 last 63.67) - open/close 6859 (time 1710101698.83 total 100.08 last 61.02) - open/close 7456 (time 1710101708.83 total 110.09 last 59.67) - open/close 7986 (time 1710101718.83 total 120.09 last 53.00) - open/close 8481 (time 1710101728.85 total 130.10 last 49.42) - open/close 8989 (time 1710101738.87 total 140.13 last 50.69) - open/close 9461 (time 1710101748.88 total 150.13 last 47.18) - open/close 9529 (time 1710101761.38 total 162.63 last 5.44) total: 10000 open/close in 172.52 seconds: 57.97 ops/second - unlinked 0 (time 1710101773 ; total 0 ; last 0) total: 10000 unlinks in 114 seconds: 87.719299 unlinks/second - open/close 865 (time 1710101899.35 total 10.02 last 86.30) - open/close 1753 (time 1710101909.36 total 20.03 last 88.75) - open/close 2525 (time 1710101919.36 total 30.03 last 77.19) - open/close 3200 (time 1710101929.38 total 40.04 last 67.40) - open/close 3903 (time 1710101939.38 total 50.05 last 70.26) - open/close 4535 (time 1710101949.39 total 60.06 last 63.15) - open/close 4634 (time 1710101961.99 total 72.66 last 7.85) - open/close 5291 (time 1710101972.00 total 82.67 last 65.64) - open/close 5907 (time 1710101982.01 total 92.68 last 61.55) - open/close 6508 (time 1710101992.06 total 102.73 last 59.78) - open/close 7058 (time 1710102002.06 total 112.73 last 54.99) - open/close 7535 (time 1710102012.07 total 122.74 last 47.69) - open/close 8066 (time 1710102022.08 total 132.75 last 53.05) - open/close 8575 (time 1710102032.12 total 142.79 last 50.69) - open/close 9096 (time 1710102042.13 total 152.80 last 52.05) - open/close 9624 (time 1710102052.13 total 162.80 last 52.77) total: 10000 open/close in 171.50 seconds: 58.31 ops/second - unlinked 0 (time 1710102063 ; total 0 ; last 0) total: 10000 unlinks in 122 seconds: 81.967216 unlinks/second - open/close 976 (time 1710102198.18 total 10.00 last 97.56) - open/close 1780 (time 1710102208.26 total 20.09 last 79.74) - open/close 2515 (time 1710102218.27 total 30.10 last 73.41) - open/close 3169 (time 1710102228.27 total 40.10 last 65.40) - open/close 3779 (time 1710102238.27 total 50.10 last 60.99) - open/close 4352 (time 1710102248.30 total 60.12 last 57.17) - open/close 4884 (time 1710102258.30 total 70.13 last 53.17) - open/close 5413 (time 1710102268.31 total 80.14 last 52.85) - open/close 5931 (time 1710102278.35 total 90.17 last 51.63) - open/close 6546 (time 1710102288.36 total 100.19 last 61.41) - open/close 7102 (time 1710102298.37 total 110.20 last 55.53) - open/close 7371 (time 1710102314.31 total 126.14 last 16.88) - open/close 7876 (time 1710102324.31 total 136.14 last 50.49) - open/close 8354 (time 1710102334.32 total 146.15 last 47.75) - open/close 8847 (time 1710102344.33 total 156.16 last 49.25) - open/close 9347 (time 1710102354.34 total 166.17 last 49.96) - open/close 9879 (time 1710102364.35 total 176.17 last 53.16) total: 10000 open/close in 178.42 seconds: 56.05 ops/second - unlinked 0 (time 1710102368 ; total 0 ; last 0) total: 10000 unlinks in 129 seconds: 77.519379 unlinks/second - open/close 951 (time 1710102510.03 total 10.01 last 95.04) - open/close 1745 (time 1710102520.03 total 20.01 last 79.40) - open/close 2590 (time 1710102530.06 total 30.04 last 84.27) - open/close 3360 (time 1710102540.08 total 40.06 last 76.85) - open/close 4025 (time 1710102550.08 total 50.06 last 66.48) - open/close 4653 (time 1710102560.09 total 60.07 last 62.76) - open/close 5312 (time 1710102570.10 total 70.08 last 65.84) - open/close 5714 (time 1710102587.02 total 86.99 last 23.76) - open/close 6314 (time 1710102597.02 total 97.00 last 59.98) - open/close 6710 (time 1710102615.45 total 115.42 last 21.49) - open/close 7235 (time 1710102625.45 total 125.43 last 52.47) - open/close 7823 (time 1710102635.48 total 135.46 last 58.63) - open/close 8388 (time 1710102645.49 total 145.47 last 56.44) - open/close 8924 (time 1710102655.50 total 155.48 last 53.53) - open/close 9445 (time 1710102665.52 total 165.50 last 52.00) - open/close 9865 (time 1710102675.53 total 175.51 last 41.95) total: 10000 open/close in 179.33 seconds: 55.76 ops/second - unlinked 0 (time 1710102682 ; total 0 ; last 0) total: 10000 unlinks in 109 seconds: 91.743118 unlinks/second - open/close 933 (time 1710102803.95 total 10.00 last 93.29) - open/close 1673 (time 1710102813.95 total 20.00 last 73.99) - open/close 2465 (time 1710102823.96 total 30.01 last 79.14) - open/close 3226 (time 1710102833.97 total 40.02 last 76.06) - open/close 3904 (time 1710102843.98 total 50.03 last 67.73) - open/close 4413 (time 1710102853.99 total 60.04 last 50.83) - open/close 4990 (time 1710102863.99 total 70.04 last 57.70) - open/close 5612 (time 1710102874.01 total 80.06 last 62.07) - open/close 6198 (time 1710102884.03 total 90.07 last 58.53) - open/close 6775 (time 1710102894.03 total 100.08 last 57.66) - open/close 7340 (time 1710102904.04 total 110.09 last 56.47) - open/close 7912 (time 1710102914.04 total 120.09 last 57.20) - open/close 8442 (time 1710102934.60 total 140.65 last 25.78) - open/close 8960 (time 1710102944.62 total 150.67 last 51.69) - open/close 9435 (time 1710102954.63 total 160.68 last 47.43) - open/close 9934 (time 1710102964.63 total 170.68 last 49.89) total: 10000 open/close in 171.93 seconds: 58.16 ops/second - unlinked 0 (time 1710102968 ; total 0 ; last 0) total: 10000 unlinks in 115 seconds: 86.956520 unlinks/second - open/close 893 (time 1710103095.73 total 10.00 last 89.27) - open/close 1736 (time 1710103105.73 total 20.01 last 84.24) - open/close 2643 (time 1710103115.74 total 30.02 last 90.64) - open/close 3407 (time 1710103125.75 total 40.03 last 76.34) - open/close 4122 (time 1710103135.75 total 50.03 last 71.49) - open/close 4768 (time 1710103145.77 total 60.05 last 64.49) - open/close 5050 (time 1710103160.83 total 75.11 last 18.72) - open/close 5588 (time 1710103170.83 total 85.11 last 53.78) - open/close 5790 (time 1710103186.27 total 100.55 last 13.08) - open/close 6414 (time 1710103196.28 total 110.55 last 62.40) - open/close 7071 (time 1710103206.28 total 120.56 last 65.65) - open/close 7682 (time 1710103216.29 total 130.56 last 61.09) - open/close 8304 (time 1710103226.29 total 140.57 last 62.18) - open/close 8840 (time 1710103236.29 total 150.57 last 53.58) - open/close 9075 (time 1710103251.87 total 166.15 last 15.08) - open/close 9628 (time 1710103261.88 total 176.16 last 55.26) total: 10000 open/close in 182.19 seconds: 54.89 ops/second - unlinked 0 (time 1710103270 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 1092 (time 1710103366.31 total 10.00 last 109.17) - open/close 2108 (time 1710103376.31 total 20.00 last 101.59) - open/close 3099 (time 1710103386.31 total 30.01 last 99.05) - open/close 4152 (time 1710103396.32 total 40.01 last 105.27) - open/close 5033 (time 1710103406.32 total 50.02 last 88.05) - open/close 5414 (time 1710103422.26 total 65.95 last 23.91) - open/close 6271 (time 1710103432.27 total 75.96 last 85.63) - open/close 7077 (time 1710103442.28 total 85.97 last 80.52) - open/close 7839 (time 1710103452.28 total 95.97 last 76.19) - open/close 8592 (time 1710103462.29 total 105.98 last 75.23) - open/close 9319 (time 1710103472.30 total 115.99 last 72.65) total: 10000 open/close in 125.37 seconds: 79.77 ops/second - unlinked 0 (time 1710103483 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 1564 (time 1710103555.49 total 10.00 last 156.33) - open/close 3018 (time 1710103565.50 total 20.01 last 145.30) - open/close 4469 (time 1710103575.51 total 30.02 last 144.99) - open/close 5864 (time 1710103585.51 total 40.02 last 139.50) - open/close 7191 (time 1710103595.51 total 50.02 last 132.68) - open/close 8440 (time 1710103605.51 total 60.02 last 124.87) - open/close 9673 (time 1710103615.51 total 70.02 last 123.29) total: 10000 open/close in 72.69 seconds: 137.57 ops/second - unlinked 0 (time 1710103619 ; total 0 ; last 0) total: 10000 unlinks in 43 seconds: 232.558136 unlinks/second - open/close 2897 (time 1710103673.24 total 10.00 last 289.67) - open/close 5387 (time 1710103683.24 total 20.01 last 248.90) - open/close 7611 (time 1710103693.24 total 30.01 last 222.34) - open/close 9559 (time 1710103703.24 total 40.01 last 194.77) total: 10000 open/close in 42.51 seconds: 235.23 ops/second - unlinked 0 (time 1710103706 ; total 0 ; last 0) total: 10000 unlinks in 25 seconds: 400.000000 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg133-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg133-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg133-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.133@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.133@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg133-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg133-server: oleg133-server.virtnet: executing set_default_debug -1 all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Started lustre-OST0000 oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg133-server: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg133-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec oleg133-server: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg133-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg133-client.virtnet: -o user_xattr,flock oleg133-server@tcp:/lustre /mnt/lustre On OST0, 20438 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg133-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg133-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg133-server stop mds service on oleg133-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg133-server stop mds service on oleg133-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg133-server unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg133-client: oleg133-client: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-client: ssh exited with exit code 2