== conf-sanity test 69: replace an OST with the same index ========================================================== 16:06:26 (1710101186) start mds service on oleg136-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg136-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg136-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1 Started lustre-OST0000 oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg136-client.virtnet: -o user_xattr,flock oleg136-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11840 inodes available. Want 99936. rc=0 - open/close 843 (time 1710101242.22 total 10.00 last 84.26) - open/close 1669 (time 1710101252.23 total 20.01 last 82.55) - open/close 2464 (time 1710101262.23 total 30.01 last 79.46) - open/close 3108 (time 1710101272.26 total 40.04 last 64.23) - open/close 3796 (time 1710101282.29 total 50.07 last 68.59) - open/close 4503 (time 1710101292.32 total 60.10 last 70.47) - open/close 5040 (time 1710101302.32 total 70.11 last 53.69) - open/close 5667 (time 1710101312.33 total 80.11 last 62.68) - open/close 5905 (time 1710101327.18 total 94.96 last 16.03) - open/close 6518 (time 1710101337.20 total 104.98 last 61.17) - open/close 7071 (time 1710101347.22 total 115.00 last 55.21) - open/close 7547 (time 1710101367.69 total 135.48 last 23.24) - open/close 7982 (time 1710101386.49 total 154.28 last 23.14) - open/close 8431 (time 1710101396.50 total 164.29 last 44.87) - open/close 8886 (time 1710101406.52 total 174.30 last 45.42) - open/close 9423 (time 1710101416.52 total 184.31 last 53.68) - open/close 9887 (time 1710101426.55 total 194.33 last 46.29) total: 10000 open/close in 196.51 seconds: 50.89 ops/second - unlinked 0 (time 1710101430 ; total 0 ; last 0) total: 10000 unlinks in 108 seconds: 92.592590 unlinks/second - open/close 995 (time 1710101551.49 total 10.01 last 99.43) - open/close 1805 (time 1710101571.98 total 30.49 last 39.54) - open/close 2600 (time 1710101582.01 total 40.53 last 79.23) - open/close 3347 (time 1710101592.03 total 50.55 last 74.57) - open/close 4050 (time 1710101602.04 total 60.56 last 70.23) - open/close 4636 (time 1710101612.06 total 70.57 last 58.49) - open/close 5132 (time 1710101622.07 total 80.59 last 49.53) - open/close 5633 (time 1710101632.08 total 90.60 last 50.05) - open/close 6236 (time 1710101642.10 total 100.61 last 60.23) - open/close 6811 (time 1710101652.11 total 110.63 last 57.41) - open/close 7380 (time 1710101662.13 total 120.64 last 56.80) - open/close 7928 (time 1710101672.14 total 130.65 last 54.74) - open/close 8411 (time 1710101682.15 total 140.66 last 48.25) - open/close 8843 (time 1710101692.17 total 150.68 last 43.12) - open/close 9279 (time 1710101702.17 total 160.68 last 43.59) - open/close 9759 (time 1710101712.19 total 170.70 last 47.91) total: 10000 open/close in 176.43 seconds: 56.68 ops/second - unlinked 0 (time 1710101721 ; total 0 ; last 0) total: 10000 unlinks in 106 seconds: 94.339622 unlinks/second - open/close 886 (time 1710101839.87 total 10.02 last 88.45) - open/close 1832 (time 1710101849.89 total 20.04 last 94.40) - open/close 2594 (time 1710101859.90 total 30.05 last 76.14) - open/close 3228 (time 1710101869.91 total 40.05 last 63.35) - open/close 3968 (time 1710101879.92 total 50.06 last 73.94) - open/close 4690 (time 1710101889.92 total 60.06 last 72.20) - open/close 5287 (time 1710101909.38 total 79.52 last 30.68) - open/close 5876 (time 1710101919.38 total 89.53 last 58.88) - open/close 6512 (time 1710101929.39 total 99.53 last 63.55) - open/close 7070 (time 1710101939.39 total 109.54 last 55.78) - open/close 7609 (time 1710101949.40 total 119.55 last 53.84) - open/close 8196 (time 1710101959.40 total 129.55 last 58.69) - open/close 8692 (time 1710101969.42 total 139.56 last 49.53) - open/close 9218 (time 1710101979.44 total 149.59 last 52.46) - open/close 9775 (time 1710101989.45 total 159.60 last 55.67) total: 10000 open/close in 164.10 seconds: 60.94 ops/second - unlinked 0 (time 1710101996 ; total 0 ; last 0) total: 10000 unlinks in 102 seconds: 98.039215 unlinks/second - open/close 900 (time 1710102110.56 total 10.01 last 89.91) - open/close 1811 (time 1710102120.56 total 20.01 last 91.06) - open/close 2679 (time 1710102130.57 total 30.02 last 86.78) - open/close 3472 (time 1710102140.67 total 40.12 last 78.52) - open/close 4224 (time 1710102150.67 total 50.12 last 75.16) - open/close 4923 (time 1710102160.68 total 60.13 last 69.82) - open/close 5534 (time 1710102170.70 total 70.15 last 60.99) - open/close 6146 (time 1710102180.72 total 80.17 last 61.10) - open/close 6746 (time 1710102190.73 total 90.18 last 59.91) - open/close 7344 (time 1710102200.74 total 100.19 last 59.74) - open/close 7926 (time 1710102210.77 total 110.23 last 58.01) - open/close 8445 (time 1710102220.79 total 120.24 last 51.83) - open/close 9020 (time 1710102230.79 total 130.24 last 57.50) - open/close 9481 (time 1710102240.80 total 140.25 last 46.04) - open/close 9959 (time 1710102250.81 total 150.26 last 47.77) total: 10000 open/close in 151.04 seconds: 66.21 ops/second - unlinked 0 (time 1710102254 ; total 0 ; last 0) total: 10000 unlinks in 112 seconds: 89.285713 unlinks/second - open/close 1004 (time 1710102379.24 total 10.00 last 100.35) - open/close 1781 (time 1710102389.25 total 20.01 last 77.66) - open/close 2510 (time 1710102399.25 total 30.01 last 72.87) - open/close 3202 (time 1710102409.25 total 40.02 last 69.17) - open/close 3951 (time 1710102419.25 total 50.02 last 74.90) - open/close 4660 (time 1710102429.26 total 60.02 last 70.85) - open/close 5384 (time 1710102439.27 total 70.04 last 72.30) - open/close 6036 (time 1710102449.28 total 80.05 last 65.15) - open/close 6655 (time 1710102459.28 total 90.05 last 61.89) - open/close 7269 (time 1710102469.29 total 100.05 last 61.35) - open/close 7817 (time 1710102479.30 total 110.06 last 54.75) - open/close 8437 (time 1710102489.32 total 120.09 last 61.84) - open/close 8959 (time 1710102499.37 total 130.13 last 51.98) - open/close 9496 (time 1710102509.37 total 140.14 last 53.66) - open/close 9921 (time 1710102519.38 total 150.15 last 42.45) total: 10000 open/close in 152.37 seconds: 65.63 ops/second - unlinked 0 (time 1710102524 ; total 0 ; last 0) total: 10000 unlinks in 107 seconds: 93.457947 unlinks/second - open/close 960 (time 1710102644.01 total 10.01 last 95.94) - open/close 1807 (time 1710102654.01 total 20.01 last 84.65) - open/close 2582 (time 1710102664.02 total 30.01 last 77.48) - open/close 3278 (time 1710102674.04 total 40.04 last 69.46) - open/close 3994 (time 1710102684.05 total 50.05 last 71.52) - open/close 4627 (time 1710102694.05 total 60.05 last 63.28) - open/close 5242 (time 1710102704.05 total 70.05 last 61.50) - open/close 5802 (time 1710102714.07 total 80.07 last 55.91) - open/close 6449 (time 1710102724.08 total 90.08 last 64.61) - open/close 6965 (time 1710102734.09 total 100.09 last 51.55) - open/close 7414 (time 1710102744.10 total 110.10 last 44.87) - open/close 7997 (time 1710102754.11 total 120.11 last 58.22) - open/close 8608 (time 1710102764.11 total 130.11 last 61.10) - open/close 9148 (time 1710102774.13 total 140.13 last 53.91) - open/close 9237 (time 1710102786.75 total 152.75 last 7.05) - open/close 9739 (time 1710102796.78 total 162.77 last 50.06) total: 10000 open/close in 167.92 seconds: 59.55 ops/second - unlinked 0 (time 1710102804 ; total 0 ; last 0) total: 10000 unlinks in 113 seconds: 88.495575 unlinks/second - open/close 920 (time 1710102929.85 total 10.01 last 91.93) - open/close 1245 (time 1710102944.77 total 24.93 last 21.78) - open/close 1871 (time 1710102954.77 total 34.93 last 62.59) - open/close 2657 (time 1710102964.78 total 44.94 last 78.55) - open/close 3408 (time 1710102974.79 total 54.95 last 75.01) - open/close 4019 (time 1710102984.80 total 64.96 last 61.06) - open/close 4666 (time 1710102994.86 total 75.02 last 64.34) - open/close 5295 (time 1710103004.86 total 85.02 last 62.85) - open/close 5877 (time 1710103014.87 total 95.03 last 58.14) - open/close 6477 (time 1710103024.88 total 105.04 last 59.93) - open/close 7098 (time 1710103034.90 total 115.06 last 62.02) - open/close 7605 (time 1710103044.91 total 125.07 last 50.64) - open/close 8183 (time 1710103054.92 total 135.08 last 57.75) - open/close 8748 (time 1710103064.93 total 145.09 last 56.47) - open/close 9270 (time 1710103085.84 total 166.00 last 24.96) - open/close 9829 (time 1710103095.84 total 176.00 last 55.90) total: 10000 open/close in 178.95 seconds: 55.88 ops/second - unlinked 0 (time 1710103101 ; total 0 ; last 0) total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second - open/close 972 (time 1710103210.10 total 10.00 last 97.19) - open/close 1925 (time 1710103220.11 total 20.01 last 95.22) - open/close 2635 (time 1710103230.12 total 30.01 last 70.96) - open/close 3316 (time 1710103240.13 total 40.03 last 67.98) - open/close 4169 (time 1710103250.16 total 50.06 last 85.09) - open/close 4912 (time 1710103260.16 total 60.06 last 74.28) - open/close 5639 (time 1710103270.17 total 70.07 last 72.65) - open/close 6373 (time 1710103280.17 total 80.07 last 73.37) - open/close 6974 (time 1710103290.19 total 90.08 last 60.02) - open/close 7542 (time 1710103300.21 total 100.10 last 56.69) - open/close 8173 (time 1710103310.21 total 110.11 last 63.07) - open/close 8865 (time 1710103320.22 total 120.12 last 69.12) - open/close 9480 (time 1710103330.22 total 130.12 last 61.49) total: 10000 open/close in 138.66 seconds: 72.12 ops/second - unlinked 0 (time 1710103340 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 1148 (time 1710103430.43 total 10.00 last 114.80) - open/close 2199 (time 1710103440.43 total 20.00 last 105.05) - open/close 3176 (time 1710103450.44 total 30.01 last 97.62) - open/close 4123 (time 1710103460.45 total 40.02 last 94.65) - open/close 5068 (time 1710103470.45 total 50.02 last 94.45) - open/close 5866 (time 1710103490.70 total 70.27 last 39.41) - open/close 6857 (time 1710103500.70 total 80.27 last 99.09) - open/close 7767 (time 1710103510.71 total 90.28 last 90.96) - open/close 8702 (time 1710103520.71 total 100.28 last 93.43) - open/close 9619 (time 1710103530.72 total 110.29 last 91.65) total: 10000 open/close in 114.64 seconds: 87.23 ops/second - unlinked 0 (time 1710103536 ; total 0 ; last 0) total: 10000 unlinks in 53 seconds: 188.679245 unlinks/second - open/close 1953 (time 1710103600.94 total 10.00 last 195.22) - open/close 3673 (time 1710103610.94 total 20.01 last 171.95) - open/close 5297 (time 1710103620.94 total 30.01 last 162.37) - open/close 6728 (time 1710103630.95 total 40.01 last 143.03) - open/close 8065 (time 1710103640.95 total 50.02 last 133.68) - open/close 9366 (time 1710103650.95 total 60.02 last 130.08) total: 10000 open/close in 65.02 seconds: 153.79 ops/second - unlinked 0 (time 1710103656 ; total 0 ; last 0) total: 10000 unlinks in 34 seconds: 294.117645 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg136-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg136-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.136@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.136@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg136-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1 Started lustre-OST0000 oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg136-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec oleg136-server: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg136-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg136-client.virtnet: -o user_xattr,flock oleg136-server@tcp:/lustre /mnt/lustre On OST0, 9143 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg136-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg136-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server stop mds service on oleg136-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server stop mds service on oleg136-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server unloading modules on: 'oleg136-server' oleg136-server: oleg136-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg136-client: oleg136-client: ssh exited with exit code 2 pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2 pdsh@oleg136-client: oleg136-client: ssh exited with exit code 2 pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2