== conf-sanity test 69: replace an OST with the same index ========================================================== 23:08:36 (1746155316) start mds service on oleg237-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions oleg237-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg237-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg237-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Started lustre-OST0000 oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1313 (time 1746155359.60 total 10.00 last 131.29) - open/close 2593 (time 1746155369.60 total 20.00 last 127.97) - open/close 3776 (time 1746155379.60 total 30.00 last 118.29) - open/close 5001 (time 1746155389.61 total 40.01 last 122.46) - open/close 6001 (time 1746155399.61 total 50.01 last 99.95) - open/close 7205 (time 1746155409.62 total 60.02 last 120.28) - open/close 8366 (time 1746155419.62 total 70.02 last 116.10) - open/close 9484 (time 1746155429.63 total 80.03 last 111.71) total: 10000 open/close in 84.72 seconds: 118.03 ops/second - unlinked 0 (time 1746155436 ; total 0 ; last 0) total: 10000 unlinks in 53 seconds: 188.679245 unlinks/second - open/close 1121 (time 1746155500.94 total 10.01 last 111.99) - open/close 2307 (time 1746155510.94 total 20.01 last 118.54) - open/close 3466 (time 1746155520.95 total 30.02 last 115.85) - open/close 4696 (time 1746155530.95 total 40.02 last 122.96) - open/close 5903 (time 1746155540.95 total 50.02 last 120.69) - open/close 6920 (time 1746155550.96 total 60.03 last 101.64) - open/close 8086 (time 1746155560.96 total 70.03 last 116.55) - open/close 9023 (time 1746155570.97 total 80.04 last 93.61) total: 10000 open/close in 89.57 seconds: 111.64 ops/second - unlinked 0 (time 1746155582 ; total 0 ; last 0) total: 10000 unlinks in 46 seconds: 217.391312 unlinks/second - open/close 1450 (time 1746155640.35 total 10.00 last 144.96) - open/close 2837 (time 1746155650.36 total 20.01 last 138.57) - open/close 4064 (time 1746155660.36 total 30.02 last 122.64) - open/close 5271 (time 1746155670.36 total 40.02 last 120.65) - open/close 6428 (time 1746155680.37 total 50.03 last 115.63) - open/close 7968 (time 1746155690.38 total 60.03 last 153.90) - open/close 9028 (time 1746155700.38 total 70.04 last 105.98) total: 10000 open/close in 79.46 seconds: 125.86 ops/second - unlinked 0 (time 1746155711 ; total 0 ; last 0) total: 10000 unlinks in 52 seconds: 192.307693 unlinks/second - open/close 1301 (time 1746155774.97 total 10.00 last 130.10) - open/close 2542 (time 1746155784.97 total 20.00 last 124.10) - open/close 4093 (time 1746155794.97 total 30.00 last 155.09) - open/close 5532 (time 1746155804.97 total 40.00 last 143.90) - open/close 7104 (time 1746155814.98 total 50.01 last 157.02) - open/close 8542 (time 1746155824.98 total 60.02 last 143.73) - open/close 9629 (time 1746155834.99 total 70.02 last 108.67) total: 10000 open/close in 73.69 seconds: 135.71 ops/second - unlinked 0 (time 1746155840 ; total 0 ; last 0) total: 10000 unlinks in 56 seconds: 178.571426 unlinks/second - open/close 1246 (time 1746155908.30 total 10.00 last 124.59) - open/close 2374 (time 1746155918.30 total 20.00 last 112.79) - open/close 3377 (time 1746155928.32 total 30.02 last 100.16) - open/close 4374 (time 1746155938.32 total 40.02 last 99.69) - open/close 5310 (time 1746155948.32 total 50.02 last 93.60) - open/close 6240 (time 1746155958.33 total 60.02 last 92.94) - open/close 7194 (time 1746155968.33 total 70.03 last 95.35) - open/close 8048 (time 1746155978.34 total 80.04 last 85.29) - open/close 8968 (time 1746155988.35 total 90.04 last 91.97) total: 10000 open/close in 100.04 seconds: 99.96 ops/second - unlinked 0 (time 1746156000 ; total 0 ; last 0) total: 10000 unlinks in 53 seconds: 188.679245 unlinks/second - open/close 1273 (time 1746156065.73 total 10.01 last 127.23) - open/close 2478 (time 1746156075.73 total 20.01 last 120.48) - open/close 3674 (time 1746156085.74 total 30.01 last 119.54) - open/close 4972 (time 1746156095.74 total 40.02 last 129.75) - open/close 6477 (time 1746156105.74 total 50.02 last 150.44) - open/close 7632 (time 1746156115.75 total 60.02 last 115.47) - open/close 9091 (time 1746156125.75 total 70.03 last 145.78) total: 10000 open/close in 76.94 seconds: 129.98 ops/second - unlinked 0 (time 1746156134 ; total 0 ; last 0) total: 10000 unlinks in 44 seconds: 227.272720 unlinks/second - open/close 1352 (time 1746156190.90 total 10.00 last 135.16) - open/close 2446 (time 1746156200.91 total 20.01 last 109.31) - open/close 3636 (time 1746156210.91 total 30.01 last 118.97) - open/close 4952 (time 1746156220.92 total 40.02 last 131.54) - open/close 6072 (time 1746156230.92 total 50.02 last 111.95) - open/close 7291 (time 1746156240.93 total 60.03 last 121.82) - open/close 8467 (time 1746156250.93 total 70.04 last 117.52) - open/close 9609 (time 1746156260.94 total 80.04 last 114.11) total: 10000 open/close in 83.00 seconds: 120.49 ops/second - unlinked 0 (time 1746156265 ; total 0 ; last 0) total: 10000 unlinks in 48 seconds: 208.333328 unlinks/second - open/close 1302 (time 1746156325.45 total 10.00 last 130.18) - open/close 2485 (time 1746156335.45 total 20.01 last 118.24) - open/close 3794 (time 1746156345.46 total 30.01 last 130.85) - open/close 4923 (time 1746156355.48 total 40.03 last 112.70) - open/close 6138 (time 1746156365.49 total 50.04 last 121.39) - open/close 7359 (time 1746156375.49 total 60.04 last 122.04) - open/close 8332 (time 1746156385.50 total 70.05 last 97.23) - open/close 9392 (time 1746156395.50 total 80.05 last 106.00) total: 10000 open/close in 86.70 seconds: 115.35 ops/second - unlinked 0 (time 1746156404 ; total 0 ; last 0) total: 10000 unlinks in 53 seconds: 188.679245 unlinks/second - open/close 1079 (time 1746156468.96 total 10.00 last 107.86) - open/close 2165 (time 1746156478.96 total 20.00 last 108.58) - open/close 3322 (time 1746156488.97 total 30.01 last 115.62) - open/close 4534 (time 1746156498.97 total 40.02 last 121.11) - open/close 5696 (time 1746156508.98 total 50.02 last 116.16) - open/close 6800 (time 1746156518.99 total 60.03 last 110.31) - open/close 7853 (time 1746156529.00 total 70.05 last 105.12) - open/close 8797 (time 1746156539.02 total 80.06 last 94.28) - open/close 9782 (time 1746156549.02 total 90.07 last 98.43) total: 10000 open/close in 92.31 seconds: 108.33 ops/second - unlinked 0 (time 1746156553 ; total 0 ; last 0) total: 10000 unlinks in 46 seconds: 217.391312 unlinks/second - open/close 1823 (time 1746156610.62 total 10.01 last 182.20) - open/close 3430 (time 1746156620.62 total 20.01 last 160.66) - open/close 4958 (time 1746156630.63 total 30.02 last 152.66) - open/close 6705 (time 1746156640.63 total 40.02 last 174.68) - open/close 8075 (time 1746156650.63 total 50.02 last 136.95) - open/close 9558 (time 1746156660.63 total 60.02 last 148.28) total: 10000 open/close in 62.90 seconds: 158.98 ops/second - unlinked 0 (time 1746156665 ; total 0 ; last 0) total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg237-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg237-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg237-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.137@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.137@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg237-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Started lustre-OST0000 oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg237-server: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg237-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec mount lustre on /mnt/lustre..... Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre On OST0, 17701 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg237-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg237-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg237-server stop mds service on oleg237-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg237-server unloading modules via unload_modules_local on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg237-client: oleg237-client: ssh exited with exit code 2 pdsh@oleg237-client: oleg237-server: ssh exited with exit code 2 pdsh@oleg237-client: oleg237-client: ssh exited with exit code 2 pdsh@oleg237-client: oleg237-server: ssh exited with exit code 2