== conf-sanity test 69: replace an OST with the same index ========================================================== 12:12:24 (1709053944) start mds service on oleg342-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg342-server' oleg342-server: oleg342-server.virtnet: executing load_modules_local oleg342-server: Loading modules from /home/green/git/lustre-release/lustre oleg342-server: detected 4 online CPUs by sysfs oleg342-server: Force libcfs to create 2 CPU partitions oleg342-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg342-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg342-server: oleg342-server.virtnet: executing set_default_debug -1 all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg342-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg342-server: oleg342-server.virtnet: executing set_default_debug -1 all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Started lustre-OST0000 oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg342-client.virtnet: -o user_xattr,flock oleg342-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 4129 (time 1709053979.82 total 10.00 last 412.87) - open/close 6714 (time 1709053989.83 total 20.01 last 258.38) - open/close 8811 (time 1709053999.84 total 30.02 last 209.47) - open/close 9798 (time 1709054009.84 total 40.02 last 98.64) total: 10000 open/close in 41.92 seconds: 238.53 ops/second - unlinked 0 (time 1709054012 ; total 0 ; last 0) total: 10000 unlinks in 49 seconds: 204.081635 unlinks/second - open/close 1970 (time 1709054073.16 total 10.00 last 196.96) - open/close 3664 (time 1709054083.16 total 20.01 last 169.32) - open/close 5201 (time 1709054093.17 total 30.01 last 153.62) - open/close 5995 (time 1709054103.19 total 40.03 last 79.25) - open/close 6557 (time 1709054113.21 total 50.05 last 56.08) - open/close 7198 (time 1709054123.21 total 60.06 last 64.06) - open/close 8037 (time 1709054133.22 total 70.07 last 83.84) - open/close 8866 (time 1709054143.22 total 80.07 last 82.88) - open/close 9728 (time 1709054153.23 total 90.07 last 86.15) total: 10000 open/close in 93.43 seconds: 107.04 ops/second - unlinked 0 (time 1709054158 ; total 0 ; last 0) total: 10000 unlinks in 53 seconds: 188.679245 unlinks/second - open/close 1692 (time 1709054222.40 total 10.01 last 169.03) - open/close 2881 (time 1709054232.41 total 20.02 last 118.73) - open/close 3703 (time 1709054242.41 total 30.03 last 82.19) - open/close 4466 (time 1709054252.41 total 40.03 last 76.30) - open/close 5272 (time 1709054262.42 total 50.03 last 80.59) - open/close 6034 (time 1709054272.43 total 60.04 last 76.11) - open/close 6875 (time 1709054282.44 total 70.05 last 83.99) - open/close 7813 (time 1709054292.45 total 80.06 last 93.76) - open/close 8695 (time 1709054302.45 total 90.06 last 88.16) - open/close 9514 (time 1709054312.45 total 100.06 last 81.90) total: 10000 open/close in 106.59 seconds: 93.81 ops/second - unlinked 0 (time 1709054320 ; total 0 ; last 0) total: 10000 unlinks in 50 seconds: 200.000000 unlinks/second - open/close 1518 (time 1709054381.98 total 10.01 last 151.70) - open/close 2839 (time 1709054391.98 total 20.01 last 132.08) - open/close 4198 (time 1709054401.99 total 30.01 last 135.84) - open/close 5422 (time 1709054411.99 total 40.02 last 122.34) - open/close 6510 (time 1709054421.99 total 50.02 last 108.80) - open/close 7528 (time 1709054432.00 total 60.02 last 101.73) - open/close 8364 (time 1709054442.01 total 70.04 last 83.48) - open/close 9280 (time 1709054452.02 total 80.05 last 91.54) total: 10000 open/close in 88.66 seconds: 112.79 ops/second - unlinked 0 (time 1709054462 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 1209 (time 1709054541.01 total 10.01 last 120.84) - open/close 1537 (time 1709054555.04 total 24.04 last 23.37) - open/close 2748 (time 1709054565.05 total 34.05 last 121.01) - open/close 3329 (time 1709054581.79 total 50.79 last 34.71) - open/close 4590 (time 1709054591.80 total 60.79 last 126.04) - open/close 5573 (time 1709054601.80 total 70.80 last 98.21) - open/close 6518 (time 1709054611.81 total 80.80 last 94.48) - open/close 7481 (time 1709054621.81 total 90.81 last 96.25) - open/close 8442 (time 1709054631.82 total 100.82 last 96.04) - open/close 9330 (time 1709054641.82 total 110.82 last 88.77) total: 10000 open/close in 119.25 seconds: 83.86 ops/second - unlinked 0 (time 1709054651 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 1285 (time 1709054735.74 total 10.01 last 128.38) - open/close 2504 (time 1709054745.74 total 20.01 last 121.85) - open/close 3615 (time 1709054755.75 total 30.02 last 111.03) - open/close 4600 (time 1709054765.78 total 40.05 last 98.20) - open/close 5508 (time 1709054775.79 total 50.06 last 90.69) - open/close 6351 (time 1709054785.79 total 60.07 last 84.27) - open/close 7243 (time 1709054795.79 total 70.07 last 89.20) - open/close 8092 (time 1709054805.80 total 80.07 last 84.86) - open/close 8928 (time 1709054815.80 total 90.07 last 83.58) - open/close 9650 (time 1709054825.81 total 100.08 last 72.16) total: 10000 open/close in 104.82 seconds: 95.40 ops/second - unlinked 0 (time 1709054832 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 1270 (time 1709054919.29 total 10.00 last 126.99) - open/close 2458 (time 1709054929.29 total 20.01 last 118.74) - open/close 3506 (time 1709054939.29 total 30.01 last 104.79) - open/close 4427 (time 1709054949.31 total 40.02 last 91.96) - open/close 5347 (time 1709054959.31 total 50.03 last 91.94) - open/close 6334 (time 1709054969.32 total 60.04 last 98.63) - open/close 7311 (time 1709054979.32 total 70.04 last 97.69) - open/close 8390 (time 1709054989.33 total 80.05 last 107.80) - open/close 9423 (time 1709054999.33 total 90.05 last 103.28) total: 10000 open/close in 96.15 seconds: 104.00 ops/second - unlinked 0 (time 1709055006 ; total 0 ; last 0) total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second - open/close 1546 (time 1709055075.85 total 10.00 last 154.55) - open/close 2789 (time 1709055085.85 total 20.01 last 124.26) - open/close 3997 (time 1709055095.86 total 30.01 last 120.72) - open/close 5093 (time 1709055105.86 total 40.01 last 109.60) - open/close 5947 (time 1709055115.87 total 50.02 last 85.30) - open/close 6746 (time 1709055125.87 total 60.03 last 79.86) - open/close 7566 (time 1709055135.87 total 70.03 last 81.99) - open/close 8389 (time 1709055145.88 total 80.04 last 82.24) - open/close 9367 (time 1709055155.89 total 90.04 last 97.74) total: 10000 open/close in 95.90 seconds: 104.28 ops/second - unlinked 0 (time 1709055163 ; total 0 ; last 0) total: 10000 unlinks in 54 seconds: 185.185181 unlinks/second - open/close 1736 (time 1709055228.56 total 10.01 last 173.50) - open/close 3401 (time 1709055238.56 total 20.01 last 166.50) - open/close 5061 (time 1709055248.57 total 30.01 last 165.93) - open/close 6498 (time 1709055258.57 total 40.01 last 143.64) - open/close 7621 (time 1709055268.57 total 50.01 last 112.29) - open/close 8601 (time 1709055278.57 total 60.02 last 97.98) - open/close 9599 (time 1709055288.58 total 70.02 last 99.75) total: 10000 open/close in 75.05 seconds: 133.25 ops/second - unlinked 0 (time 1709055295 ; total 0 ; last 0) total: 10000 unlinks in 54 seconds: 185.185181 unlinks/second - open/close 1762 (time 1709055360.60 total 10.00 last 176.12) - open/close 3443 (time 1709055370.60 total 20.00 last 168.10) - open/close 4934 (time 1709055380.61 total 30.01 last 149.04) - open/close 5462 (time 1709055395.47 total 44.88 last 35.51) - open/close 6465 (time 1709055405.48 total 54.88 last 100.25) - open/close 7432 (time 1709055415.48 total 64.88 last 96.68) - open/close 8411 (time 1709055425.49 total 74.89 last 97.81) - open/close 9444 (time 1709055435.50 total 84.90 last 103.21) total: 10000 open/close in 90.76 seconds: 110.18 ops/second - unlinked 0 (time 1709055442 ; total 0 ; last 0) total: 10000 unlinks in 45 seconds: 222.222229 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg342-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg342-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg342-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.142@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.142@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg342-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg342-server: oleg342-server.virtnet: executing set_default_debug -1 all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Started lustre-OST0000 oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg342-server: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg342-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec mount lustre on /mnt/lustre..... Starting client: oleg342-client.virtnet: -o user_xattr,flock oleg342-server@tcp:/lustre /mnt/lustre On OST0, 18077 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg342-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg342-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg342-server stop mds service on oleg342-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg342-server unloading modules on: 'oleg342-server' oleg342-server: oleg342-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg342-client: oleg342-client: ssh exited with exit code 2 pdsh@oleg342-client: oleg342-server: ssh exited with exit code 2 pdsh@oleg342-client: oleg342-client: ssh exited with exit code 2 pdsh@oleg342-client: oleg342-server: ssh exited with exit code 2