== conf-sanity test 69: replace an OST with the same index ========================================================== 15:59:22 (1710100762) start mds service on oleg214-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg214-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-OST0000 oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg214-client.virtnet: -o user_xattr,flock oleg214-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99936. rc=0 - open/close 1662 (time 1710100790.16 total 10.00 last 166.14) - open/close 2999 (time 1710100800.17 total 20.01 last 133.59) - open/close 4165 (time 1710100810.18 total 30.02 last 116.52) - open/close 5168 (time 1710100820.19 total 40.03 last 100.22) - open/close 6168 (time 1710100830.19 total 50.03 last 99.95) - open/close 7170 (time 1710100840.19 total 60.03 last 100.20) - open/close 8233 (time 1710100850.19 total 70.03 last 106.30) - open/close 9120 (time 1710100860.20 total 80.04 last 88.60) - open/close 9984 (time 1710100870.22 total 90.06 last 86.29) total: 10000 open/close in 90.23 seconds: 110.82 ops/second - unlinked 0 (time 1710100871 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 1099 (time 1710100948.19 total 10.00 last 109.89) - open/close 2111 (time 1710100958.20 total 20.01 last 101.14) - open/close 3168 (time 1710100968.21 total 30.02 last 105.54) - open/close 4232 (time 1710100978.22 total 40.03 last 106.33) - open/close 5160 (time 1710100988.23 total 50.04 last 92.71) - open/close 6107 (time 1710100998.24 total 60.05 last 94.62) - open/close 7043 (time 1710101008.24 total 70.06 last 93.52) - open/close 7801 (time 1710101018.25 total 80.07 last 75.73) - open/close 8585 (time 1710101028.26 total 90.07 last 78.36) - open/close 9419 (time 1710101038.26 total 100.07 last 83.38) total: 10000 open/close in 108.50 seconds: 92.17 ops/second - unlinked 0 (time 1710101048 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 1104 (time 1710101147.29 total 10.00 last 110.35) - open/close 2023 (time 1710101157.29 total 20.01 last 91.86) - open/close 2880 (time 1710101167.30 total 30.02 last 85.62) - open/close 3772 (time 1710101177.31 total 40.02 last 89.14) - open/close 4590 (time 1710101187.31 total 50.03 last 81.77) - open/close 5321 (time 1710101197.32 total 60.04 last 73.04) - open/close 6048 (time 1710101207.33 total 70.05 last 72.58) - open/close 6803 (time 1710101217.34 total 80.05 last 75.49) - open/close 7428 (time 1710101227.34 total 90.06 last 62.47) - open/close 8036 (time 1710101237.35 total 100.07 last 60.73) - open/close 8688 (time 1710101247.36 total 110.08 last 65.15) - open/close 9357 (time 1710101257.36 total 120.08 last 66.88) - open/close 9988 (time 1710101267.37 total 130.09 last 63.03) total: 10000 open/close in 130.27 seconds: 76.76 ops/second - unlinked 0 (time 1710101269 ; total 0 ; last 0) total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second - open/close 1017 (time 1710101370.30 total 10.01 last 101.62) - open/close 1995 (time 1710101380.30 total 20.01 last 97.80) - open/close 2958 (time 1710101390.30 total 30.01 last 96.26) - open/close 3767 (time 1710101400.30 total 40.01 last 80.89) - open/close 4556 (time 1710101410.31 total 50.03 last 78.80) - open/close 5280 (time 1710101420.32 total 60.03 last 72.34) - open/close 6043 (time 1710101430.33 total 70.04 last 76.24) - open/close 6802 (time 1710101440.33 total 80.05 last 75.88) - open/close 7523 (time 1710101450.34 total 90.05 last 72.09) - open/close 8170 (time 1710101460.36 total 100.07 last 64.57) - open/close 8803 (time 1710101470.36 total 110.07 last 63.30) - open/close 9418 (time 1710101480.37 total 120.08 last 61.40) total: 10000 open/close in 129.97 seconds: 76.94 ops/second - unlinked 0 (time 1710101492 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 1126 (time 1710101585.50 total 10.00 last 112.60) - open/close 2156 (time 1710101595.50 total 20.01 last 102.92) - open/close 3068 (time 1710101605.51 total 30.02 last 91.13) - open/close 4083 (time 1710101615.51 total 40.02 last 101.49) - open/close 4914 (time 1710101625.52 total 50.03 last 83.03) - open/close 5771 (time 1710101635.52 total 60.03 last 85.66) - open/close 6547 (time 1710101645.54 total 70.04 last 77.49) - open/close 7279 (time 1710101655.57 total 80.07 last 72.99) - open/close 8095 (time 1710101665.64 total 90.14 last 81.04) - open/close 8798 (time 1710101675.64 total 100.15 last 70.28) - open/close 9481 (time 1710101685.64 total 110.15 last 68.30) total: 10000 open/close in 117.64 seconds: 85.00 ops/second - unlinked 0 (time 1710101694 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 1211 (time 1710101784.54 total 10.00 last 121.10) - open/close 2283 (time 1710101794.55 total 20.01 last 107.08) - open/close 3218 (time 1710101804.56 total 30.02 last 93.45) - open/close 4108 (time 1710101814.56 total 40.02 last 88.94) - open/close 4978 (time 1710101824.57 total 50.03 last 86.97) - open/close 5805 (time 1710101834.57 total 60.03 last 82.64) - open/close 6610 (time 1710101844.58 total 70.04 last 80.45) - open/close 7341 (time 1710101854.59 total 80.05 last 73.02) - open/close 8007 (time 1710101864.59 total 90.05 last 66.57) - open/close 8536 (time 1710101884.36 total 109.82 last 26.76) - open/close 9145 (time 1710101904.74 total 130.19 last 29.89) - open/close 9804 (time 1710101914.75 total 140.21 last 65.81) total: 10000 open/close in 143.27 seconds: 69.80 ops/second - unlinked 0 (time 1710101919 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 1128 (time 1710102013.37 total 10.01 last 112.73) - open/close 2108 (time 1710102023.37 total 20.01 last 97.99) - open/close 3089 (time 1710102033.38 total 30.02 last 98.03) - open/close 3620 (time 1710102049.93 total 46.56 last 32.09) - open/close 4542 (time 1710102059.94 total 56.57 last 92.10) - open/close 5358 (time 1710102069.94 total 66.58 last 81.54) - open/close 6159 (time 1710102079.95 total 76.59 last 80.02) - open/close 6919 (time 1710102089.96 total 86.60 last 75.97) - open/close 7350 (time 1710102106.80 total 103.44 last 25.58) - open/close 8075 (time 1710102116.81 total 113.45 last 72.47) - open/close 8734 (time 1710102126.82 total 123.46 last 65.82) - open/close 9403 (time 1710102136.83 total 133.46 last 66.87) total: 10000 open/close in 143.16 seconds: 69.85 ops/second - unlinked 0 (time 1710102148 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 1160 (time 1710102239.48 total 10.00 last 115.99) - open/close 2258 (time 1710102249.48 total 20.00 last 109.77) - open/close 3329 (time 1710102259.48 total 30.01 last 107.05) - open/close 4214 (time 1710102269.48 total 40.01 last 88.48) - open/close 5024 (time 1710102279.49 total 50.02 last 80.94) - open/close 5766 (time 1710102289.49 total 60.02 last 74.18) - open/close 6559 (time 1710102299.58 total 70.10 last 78.66) - open/close 7348 (time 1710102309.58 total 80.10 last 78.89) - open/close 8093 (time 1710102319.59 total 90.11 last 74.41) - open/close 8815 (time 1710102329.59 total 100.12 last 72.17) - open/close 9424 (time 1710102339.61 total 110.13 last 60.81) total: 10000 open/close in 118.10 seconds: 84.67 ops/second - unlinked 0 (time 1710102349 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 1199 (time 1710102438.09 total 10.00 last 119.86) - open/close 2209 (time 1710102448.10 total 20.01 last 100.94) - open/close 3156 (time 1710102458.10 total 30.02 last 94.64) - open/close 4072 (time 1710102468.11 total 40.02 last 91.58) - open/close 4930 (time 1710102478.11 total 50.02 last 85.78) - open/close 5772 (time 1710102488.11 total 60.02 last 84.17) - open/close 6559 (time 1710102498.11 total 70.02 last 78.69) - open/close 7298 (time 1710102508.12 total 80.03 last 73.86) - open/close 7981 (time 1710102518.13 total 90.04 last 68.24) - open/close 8620 (time 1710102528.14 total 100.05 last 63.82) - open/close 9341 (time 1710102538.15 total 110.07 last 72.01) total: 10000 open/close in 119.72 seconds: 83.53 ops/second - unlinked 0 (time 1710102549 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 1107 (time 1710102635.82 total 10.00 last 110.66) - open/close 2231 (time 1710102645.82 total 20.01 last 112.37) - open/close 3191 (time 1710102655.82 total 30.01 last 95.98) - open/close 4073 (time 1710102665.85 total 40.03 last 87.97) - open/close 4976 (time 1710102675.85 total 50.04 last 90.28) - open/close 5831 (time 1710102685.86 total 60.05 last 85.40) - open/close 6689 (time 1710102695.87 total 70.05 last 85.75) - open/close 7524 (time 1710102705.87 total 80.06 last 83.43) - open/close 8240 (time 1710102715.88 total 90.06 last 71.59) - open/close 8964 (time 1710102725.89 total 100.08 last 72.31) - open/close 9661 (time 1710102735.89 total 110.08 last 69.68) total: 10000 open/close in 114.69 seconds: 87.19 ops/second - unlinked 0 (time 1710102742 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg214-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg214-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg214-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.114@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.114@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg214-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-OST0000 oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg214-server: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg214-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec mount lustre on /mnt/lustre..... Starting client: oleg214-client.virtnet: -o user_xattr,flock oleg214-server@tcp:/lustre /mnt/lustre On OST0, 7189 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg214-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg214-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg214-server stop mds service on oleg214-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg214-server unloading modules on: 'oleg214-server' oleg214-server: oleg214-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg214-client: oleg214-client: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-client: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2