== conf-sanity test 69: replace an OST with the same index ========================================================== 05:14:04 (1734430444) start mds service on oleg331-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing load_modules_local oleg331-server: Loading modules from /home/green/git/lustre-release/lustre oleg331-server: detected 4 online CPUs by sysfs oleg331-server: MODOPTS_LIBCFS= oleg331-server: Force libcfs to create 2 CPU partitions oleg331-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg331-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg331-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-OST0000 oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1382 (time 1734430486.37 total 10.01 last 138.12) - open/close 2723 (time 1734430496.37 total 20.01 last 134.10) - open/close 4221 (time 1734430506.37 total 30.01 last 149.77) - open/close 5776 (time 1734430516.37 total 40.01 last 155.48) - open/close 7152 (time 1734430526.37 total 50.01 last 137.53) - open/close 8652 (time 1734430536.38 total 60.02 last 149.95) total: 10000 open/close in 69.64 seconds: 143.61 ops/second - unlinked 0 (time 1734430547 ; total 0 ; last 0) total: 10000 unlinks in 43 seconds: 232.558136 unlinks/second - open/close 1271 (time 1734430602.50 total 10.00 last 127.04) - open/close 2612 (time 1734430612.50 total 20.01 last 134.02) - open/close 3857 (time 1734430622.51 total 30.02 last 124.43) - open/close 5276 (time 1734430632.52 total 40.02 last 141.78) - open/close 6556 (time 1734430642.52 total 50.03 last 127.96) - open/close 7865 (time 1734430652.53 total 60.03 last 130.83) - open/close 9238 (time 1734430662.53 total 70.04 last 137.25) total: 10000 open/close in 75.57 seconds: 132.33 ops/second - unlinked 0 (time 1734430669 ; total 0 ; last 0) total: 10000 unlinks in 40 seconds: 250.000000 unlinks/second - open/close 1384 (time 1734430721.33 total 10.01 last 138.26) - open/close 2905 (time 1734430731.33 total 20.01 last 152.09) - open/close 4494 (time 1734430741.33 total 30.01 last 158.88) - open/close 5867 (time 1734430751.34 total 40.01 last 137.28) - open/close 7348 (time 1734430761.34 total 50.01 last 148.09) - open/close 8836 (time 1734430771.34 total 60.02 last 148.77) total: 10000 open/close in 68.14 seconds: 146.76 ops/second - unlinked 0 (time 1734430780 ; total 0 ; last 0) total: 10000 unlinks in 39 seconds: 256.410248 unlinks/second - open/close 1603 (time 1734430830.86 total 10.01 last 160.19) - open/close 3482 (time 1734430840.86 total 20.01 last 187.86) - open/close 5186 (time 1734430850.87 total 30.01 last 170.31) - open/close 7185 (time 1734430860.88 total 40.02 last 199.70) - open/close 9020 (time 1734430870.88 total 50.03 last 183.39) total: 10000 open/close in 55.46 seconds: 180.31 ops/second - unlinked 0 (time 1734430877 ; total 0 ; last 0) total: 10000 unlinks in 30 seconds: 333.333344 unlinks/second - open/close 1859 (time 1734430918.76 total 10.00 last 185.86) - open/close 3268 (time 1734430928.76 total 20.01 last 140.83) - open/close 4648 (time 1734430938.77 total 30.01 last 137.95) - open/close 6127 (time 1734430948.77 total 40.01 last 147.87) - open/close 7815 (time 1734430958.77 total 50.02 last 168.75) - open/close 9330 (time 1734430968.78 total 60.02 last 151.44) total: 10000 open/close in 64.64 seconds: 154.70 ops/second - unlinked 0 (time 1734430975 ; total 0 ; last 0) total: 10000 unlinks in 30 seconds: 333.333344 unlinks/second - open/close 1987 (time 1734431016.63 total 10.00 last 198.68) - open/close 3785 (time 1734431026.63 total 20.01 last 179.69) - open/close 5598 (time 1734431036.64 total 30.01 last 181.22) - open/close 7335 (time 1734431046.64 total 40.01 last 173.69) - open/close 9187 (time 1734431056.64 total 50.01 last 185.14) total: 10000 open/close in 53.78 seconds: 185.94 ops/second - unlinked 0 (time 1734431061 ; total 0 ; last 0) total: 10000 unlinks in 35 seconds: 285.714294 unlinks/second - open/close 1813 (time 1734431107.57 total 10.01 last 181.18) - open/close 3421 (time 1734431117.57 total 20.01 last 160.77) - open/close 5306 (time 1734431127.57 total 30.01 last 188.48) - open/close 7296 (time 1734431137.57 total 40.01 last 198.95) - open/close 9462 (time 1734431147.58 total 50.02 last 216.47) total: 10000 open/close in 52.12 seconds: 191.87 ops/second - unlinked 0 (time 1734431150 ; total 0 ; last 0) total: 10000 unlinks in 22 seconds: 454.545441 unlinks/second - open/close 1974 (time 1734431184.10 total 10.00 last 197.35) - open/close 4586 (time 1734431194.10 total 20.01 last 261.06) - open/close 6352 (time 1734431204.11 total 30.01 last 176.55) - open/close 8125 (time 1734431214.11 total 40.01 last 177.27) - open/close 9888 (time 1734431224.11 total 50.02 last 176.23) total: 10000 open/close in 50.69 seconds: 197.27 ops/second - unlinked 0 (time 1734431226 ; total 0 ; last 0) total: 10000 unlinks in 39 seconds: 256.410248 unlinks/second - open/close 1605 (time 1734431276.68 total 10.01 last 160.39) - open/close 3075 (time 1734431286.68 total 20.01 last 146.96) - open/close 4663 (time 1734431296.68 total 30.01 last 158.80) - open/close 6452 (time 1734431306.68 total 40.01 last 178.82) - open/close 8177 (time 1734431316.69 total 50.02 last 172.45) total: 10000 open/close in 59.76 seconds: 167.35 ops/second - unlinked 0 (time 1734431327 ; total 0 ; last 0) total: 10000 unlinks in 26 seconds: 384.615387 unlinks/second - open/close 2002 (time 1734431365.25 total 10.00 last 200.15) - open/close 3679 (time 1734431375.25 total 20.00 last 167.66) - open/close 5277 (time 1734431385.26 total 30.01 last 159.68) - open/close 6841 (time 1734431395.26 total 40.01 last 156.39) - open/close 8505 (time 1734431405.26 total 50.02 last 166.31) - open/close 9971 (time 1734431415.27 total 60.02 last 146.57) total: 10000 open/close in 60.22 seconds: 166.06 ops/second - unlinked 0 (time 1734431417 ; total 0 ; last 0) total: 10000 unlinks in 34 seconds: 294.117645 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg331-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg331-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg331-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.131@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.131@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg331-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-OST0000 oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg331-server: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg331-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec mount lustre on /mnt/lustre..... Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre On OST0, 4909 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg331-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg331-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg331-server stop mds service on oleg331-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg331-server unloading modules via unload_modules_local on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg331-client: oleg331-client: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-client: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2