== conf-sanity test 69: replace an OST with the same index ========================================================== 13:08:28 (1726852108) start mds service on oleg230-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: detected 4 online CPUs by sysfs oleg230-server: Force libcfs to create 2 CPU partitions oleg230-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg230-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg230-server: oleg230-server.virtnet: executing set_default_debug -1 all pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg230-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg230-server: oleg230-server.virtnet: executing set_default_debug -1 all pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Started lustre-OST0000 oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1357 (time 1726852148.46 total 10.02 last 135.49) - open/close 2760 (time 1726852158.47 total 20.02 last 140.26) - open/close 4198 (time 1726852168.47 total 30.02 last 143.73) - open/close 5480 (time 1726852178.48 total 40.03 last 128.09) - open/close 6782 (time 1726852188.48 total 50.03 last 130.19) - open/close 8096 (time 1726852198.49 total 60.04 last 131.30) - open/close 9411 (time 1726852208.49 total 70.04 last 131.45) total: 10000 open/close in 74.42 seconds: 134.37 ops/second - unlinked 0 (time 1726852214 ; total 0 ; last 0) total: 10000 unlinks in 36 seconds: 277.777771 unlinks/second - open/close 1738 (time 1726852262.07 total 10.00 last 173.78) - open/close 3630 (time 1726852272.08 total 20.01 last 189.12) - open/close 5403 (time 1726852282.08 total 30.01 last 177.17) - open/close 6920 (time 1726852292.08 total 40.01 last 151.67) - open/close 8386 (time 1726852302.08 total 50.01 last 146.59) - open/close 9776 (time 1726852312.09 total 60.02 last 138.95) total: 10000 open/close in 61.52 seconds: 162.56 ops/second - unlinked 0 (time 1726852315 ; total 0 ; last 0) total: 10000 unlinks in 49 seconds: 204.081635 unlinks/second - open/close 1287 (time 1726852376.97 total 10.00 last 128.66) - open/close 2595 (time 1726852386.97 total 20.01 last 130.77) - open/close 3813 (time 1726852396.98 total 30.01 last 121.74) - open/close 5009 (time 1726852406.99 total 40.02 last 119.51) - open/close 6446 (time 1726852416.99 total 50.02 last 143.66) - open/close 8016 (time 1726852427.00 total 60.03 last 156.88) - open/close 9621 (time 1726852437.00 total 70.03 last 160.43) total: 10000 open/close in 71.86 seconds: 139.16 ops/second - unlinked 0 (time 1726852440 ; total 0 ; last 0) total: 10000 unlinks in 37 seconds: 270.270264 unlinks/second - open/close 1399 (time 1726852489.62 total 10.01 last 139.80) - open/close 2903 (time 1726852499.62 total 20.01 last 150.35) - open/close 4312 (time 1726852509.63 total 30.01 last 140.84) - open/close 5724 (time 1726852519.63 total 40.02 last 141.16) - open/close 7273 (time 1726852529.64 total 50.02 last 154.84) - open/close 8679 (time 1726852539.64 total 60.02 last 140.56) total: 10000 open/close in 69.22 seconds: 144.47 ops/second - unlinked 0 (time 1726852550 ; total 0 ; last 0) total: 10000 unlinks in 40 seconds: 250.000000 unlinks/second - open/close 1660 (time 1726852601.46 total 10.01 last 165.89) - open/close 3398 (time 1726852611.46 total 20.01 last 173.79) - open/close 5723 (time 1726852621.47 total 30.01 last 232.38) - open/close 7779 (time 1726852631.47 total 40.01 last 205.58) - open/close 9505 (time 1726852641.48 total 50.02 last 172.42) total: 10000 open/close in 53.12 seconds: 188.26 ops/second - unlinked 0 (time 1726852646 ; total 0 ; last 0) total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second - open/close 1616 (time 1726852696.19 total 10.00 last 161.53) - open/close 3100 (time 1726852706.19 total 20.01 last 148.38) - open/close 4411 (time 1726852716.20 total 30.01 last 131.00) - open/close 5833 (time 1726852726.20 total 40.02 last 142.16) - open/close 7420 (time 1726852736.21 total 50.02 last 158.68) - open/close 8739 (time 1726852746.21 total 60.02 last 131.87) total: 10000 open/close in 69.52 seconds: 143.85 ops/second - unlinked 0 (time 1726852757 ; total 0 ; last 0) total: 10000 unlinks in 35 seconds: 285.714294 unlinks/second - open/close 1464 (time 1726852804.33 total 10.01 last 146.31) - open/close 2822 (time 1726852814.33 total 20.01 last 135.76) - open/close 4296 (time 1726852824.33 total 30.01 last 147.38) - open/close 5572 (time 1726852834.33 total 40.01 last 127.56) - open/close 6861 (time 1726852844.34 total 50.02 last 128.85) - open/close 8309 (time 1726852854.34 total 60.02 last 144.75) - open/close 9925 (time 1726852864.35 total 70.03 last 161.51) total: 10000 open/close in 70.54 seconds: 141.76 ops/second - unlinked 0 (time 1726852866 ; total 0 ; last 0) total: 10000 unlinks in 43 seconds: 232.558136 unlinks/second - open/close 1035 (time 1726852922.00 total 10.01 last 103.43) - open/close 2320 (time 1726852932.00 total 20.01 last 128.47) - open/close 3798 (time 1726852942.01 total 30.01 last 147.76) - open/close 5347 (time 1726852952.01 total 40.02 last 154.83) - open/close 6675 (time 1726852962.02 total 50.02 last 132.72) - open/close 7927 (time 1726852972.02 total 60.02 last 125.18) - open/close 9387 (time 1726852982.02 total 70.03 last 145.99) total: 10000 open/close in 74.83 seconds: 133.64 ops/second - unlinked 0 (time 1726852988 ; total 0 ; last 0) total: 10000 unlinks in 39 seconds: 256.410248 unlinks/second - open/close 1252 (time 1726853039.65 total 10.00 last 125.16) - open/close 2383 (time 1726853049.66 total 20.01 last 113.07) - open/close 3496 (time 1726853059.66 total 30.01 last 111.24) - open/close 4744 (time 1726853069.66 total 40.01 last 124.78) - open/close 5986 (time 1726853079.67 total 50.02 last 124.08) - open/close 7203 (time 1726853089.68 total 60.03 last 121.63) - open/close 8562 (time 1726853099.69 total 70.04 last 135.78) total: 10000 open/close in 79.97 seconds: 125.04 ops/second - unlinked 0 (time 1726853111 ; total 0 ; last 0) total: 10000 unlinks in 47 seconds: 212.765961 unlinks/second - open/close 1047 (time 1726853171.32 total 10.01 last 104.64) - open/close 2242 (time 1726853181.33 total 20.02 last 119.37) - open/close 3623 (time 1726853191.33 total 30.02 last 138.05) - open/close 4897 (time 1726853201.34 total 40.03 last 127.30) - open/close 6112 (time 1726853211.34 total 50.03 last 121.46) - open/close 7281 (time 1726853221.35 total 60.04 last 116.82) - open/close 8480 (time 1726853231.36 total 70.05 last 119.80) - open/close 9793 (time 1726853241.36 total 80.05 last 131.29) total: 10000 open/close in 81.67 seconds: 122.44 ops/second - unlinked 0 (time 1726853244 ; total 0 ; last 0) total: 10000 unlinks in 40 seconds: 250.000000 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg230-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg230-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg230-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.130@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.130@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg230-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg230-server: oleg230-server.virtnet: executing set_default_debug -1 all pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Started lustre-OST0000 oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg230-server: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg230-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec mount lustre on /mnt/lustre..... Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre On OST0, 12829 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg230-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg230-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg230-server stop mds service on oleg230-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg230-server unloading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg230-client: oleg230-client: ssh exited with exit code 2 pdsh@oleg230-client: oleg230-server: ssh exited with exit code 2 pdsh@oleg230-client: oleg230-client: ssh exited with exit code 2 pdsh@oleg230-client: oleg230-server: ssh exited with exit code 2