== conf-sanity test 69: replace an OST with the same index ========================================================== 18:29:16 (1746138556) start mds service on oleg460-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg460-server' oleg460-server: oleg460-server.virtnet: executing load_modules_local oleg460-server: Loading modules from /home/green/git/lustre-release/lustre oleg460-server: detected 4 online CPUs by sysfs oleg460-server: MODOPTS_LIBCFS= oleg460-server: Force libcfs to create 2 CPU partitions oleg460-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg460-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg460-server: oleg460-server.virtnet: executing set_default_debug -1 all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg460-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg460-server: oleg460-server.virtnet: executing set_default_debug -1 all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Started lustre-OST0000 oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg460-client.virtnet: -o user_xattr,flock 192.168.204.160@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1347 (time 1746138599.08 total 10.00 last 134.66) - open/close 2785 (time 1746138609.08 total 20.01 last 143.76) - open/close 3982 (time 1746138619.10 total 30.02 last 119.48) - open/close 5345 (time 1746138629.10 total 40.02 last 136.29) - open/close 6777 (time 1746138639.10 total 50.03 last 143.14) - open/close 8422 (time 1746138649.11 total 60.03 last 164.44) total: 10000 open/close in 69.65 seconds: 143.58 ops/second - unlinked 0 (time 1746138660 ; total 0 ; last 0) total: 10000 unlinks in 42 seconds: 238.095245 unlinks/second - open/close 1175 (time 1746138713.76 total 10.00 last 117.45) - open/close 2232 (time 1746138723.77 total 20.01 last 105.68) - open/close 3243 (time 1746138733.77 total 30.01 last 101.06) - open/close 4455 (time 1746138743.77 total 40.01 last 121.19) - open/close 5690 (time 1746138753.77 total 50.01 last 123.48) - open/close 7069 (time 1746138763.78 total 60.02 last 137.78) - open/close 8344 (time 1746138773.78 total 70.02 last 127.48) - open/close 9748 (time 1746138783.79 total 80.03 last 140.33) total: 10000 open/close in 81.99 seconds: 121.96 ops/second - unlinked 0 (time 1746138787 ; total 0 ; last 0) total: 10000 unlinks in 48 seconds: 208.333328 unlinks/second - open/close 1296 (time 1746138847.10 total 10.00 last 129.57) - open/close 2608 (time 1746138857.10 total 20.01 last 131.14) - open/close 3830 (time 1746138867.10 total 30.01 last 122.19) - open/close 4967 (time 1746138877.11 total 40.01 last 113.65) - open/close 6138 (time 1746138887.11 total 50.01 last 117.08) - open/close 7268 (time 1746138897.11 total 60.02 last 112.94) - open/close 8335 (time 1746138907.13 total 70.03 last 106.57) - open/close 9365 (time 1746138917.13 total 80.03 last 102.97) total: 10000 open/close in 85.90 seconds: 116.42 ops/second - unlinked 0 (time 1746138925 ; total 0 ; last 0) total: 10000 unlinks in 49 seconds: 204.081635 unlinks/second - open/close 950 (time 1746138986.97 total 10.00 last 94.99) - open/close 1975 (time 1746138996.97 total 20.00 last 102.50) - open/close 3062 (time 1746139006.97 total 30.00 last 108.70) - open/close 4116 (time 1746139016.97 total 40.00 last 105.39) - open/close 5179 (time 1746139026.97 total 50.00 last 106.29) - open/close 6299 (time 1746139036.98 total 60.01 last 111.91) - open/close 7411 (time 1746139046.98 total 70.02 last 111.15) - open/close 8545 (time 1746139056.98 total 80.02 last 113.37) - open/close 9543 (time 1746139067.00 total 90.03 last 99.68) total: 10000 open/close in 94.83 seconds: 105.45 ops/second - unlinked 0 (time 1746139073 ; total 0 ; last 0) total: 10000 unlinks in 50 seconds: 200.000000 unlinks/second - open/close 1675 (time 1746139134.54 total 10.00 last 167.44) - open/close 2822 (time 1746139144.55 total 20.01 last 114.64) - open/close 3925 (time 1746139154.55 total 30.01 last 110.28) - open/close 5173 (time 1746139164.56 total 40.02 last 124.74) - open/close 6343 (time 1746139174.57 total 50.02 last 116.90) - open/close 7283 (time 1746139184.57 total 60.03 last 93.93) - open/close 8310 (time 1746139194.58 total 70.04 last 102.65) - open/close 9584 (time 1746139204.59 total 80.04 last 127.29) total: 10000 open/close in 83.83 seconds: 119.29 ops/second - unlinked 0 (time 1746139210 ; total 0 ; last 0) total: 10000 unlinks in 52 seconds: 192.307693 unlinks/second - open/close 1185 (time 1746139274.60 total 10.00 last 118.45) - open/close 2561 (time 1746139284.61 total 20.01 last 137.53) - open/close 3769 (time 1746139294.62 total 30.02 last 120.68) - open/close 4682 (time 1746139304.62 total 40.02 last 91.27) - open/close 5633 (time 1746139314.62 total 50.02 last 95.09) - open/close 6752 (time 1746139324.63 total 60.03 last 111.80) - open/close 8099 (time 1746139334.63 total 70.03 last 134.68) - open/close 9215 (time 1746139344.64 total 80.04 last 111.55) total: 10000 open/close in 86.86 seconds: 115.13 ops/second - unlinked 0 (time 1746139353 ; total 0 ; last 0) total: 10000 unlinks in 55 seconds: 181.818176 unlinks/second - open/close 1237 (time 1746139420.78 total 10.00 last 123.66) - open/close 2352 (time 1746139430.78 total 20.01 last 111.48) - open/close 3341 (time 1746139440.79 total 30.02 last 98.80) - open/close 4290 (time 1746139450.80 total 40.02 last 94.86) - open/close 5222 (time 1746139460.80 total 50.03 last 93.14) - open/close 6158 (time 1746139470.81 total 60.03 last 93.56) - open/close 7200 (time 1746139480.82 total 70.04 last 104.12) - open/close 8091 (time 1746139490.82 total 80.04 last 89.05) - open/close 8950 (time 1746139500.82 total 90.04 last 85.89) - open/close 9983 (time 1746139510.83 total 100.05 last 103.26) total: 10000 open/close in 100.14 seconds: 99.87 ops/second - unlinked 0 (time 1746139513 ; total 0 ; last 0) total: 10000 unlinks in 48 seconds: 208.333328 unlinks/second - open/close 1126 (time 1746139573.16 total 10.00 last 112.57) - open/close 2392 (time 1746139583.17 total 20.01 last 126.47) - open/close 3721 (time 1746139593.18 total 30.02 last 132.76) - open/close 4730 (time 1746139603.18 total 40.02 last 100.90) - open/close 5755 (time 1746139613.19 total 50.03 last 102.47) - open/close 7065 (time 1746139623.19 total 60.03 last 130.97) - open/close 8300 (time 1746139633.19 total 70.03 last 123.45) - open/close 9307 (time 1746139643.19 total 80.03 last 100.70) total: 10000 open/close in 85.96 seconds: 116.34 ops/second - unlinked 0 (time 1746139650 ; total 0 ; last 0) total: 10000 unlinks in 52 seconds: 192.307693 unlinks/second - open/close 1064 (time 1746139714.06 total 10.00 last 106.38) - open/close 2083 (time 1746139724.06 total 20.01 last 101.86) - open/close 3104 (time 1746139734.07 total 30.01 last 102.06) - open/close 4039 (time 1746139744.07 total 40.02 last 93.45) - open/close 4957 (time 1746139754.08 total 50.02 last 91.76) - open/close 5895 (time 1746139764.08 total 60.03 last 93.76) - open/close 6832 (time 1746139774.08 total 70.03 last 93.68) - open/close 7877 (time 1746139784.08 total 80.03 last 104.50) - open/close 9084 (time 1746139794.08 total 90.03 last 120.68) total: 10000 open/close in 99.04 seconds: 100.97 ops/second - unlinked 0 (time 1746139805 ; total 0 ; last 0) total: 10000 unlinks in 46 seconds: 217.391312 unlinks/second - open/close 1421 (time 1746139863.02 total 10.00 last 142.07) - open/close 2690 (time 1746139873.02 total 20.00 last 126.87) - open/close 3873 (time 1746139883.02 total 30.01 last 118.27) - open/close 5191 (time 1746139893.04 total 40.02 last 131.57) - open/close 6466 (time 1746139903.04 total 50.03 last 127.47) - open/close 7715 (time 1746139913.04 total 60.03 last 124.88) - open/close 8807 (time 1746139923.04 total 70.03 last 109.18) total: 10000 open/close in 79.90 seconds: 125.16 ops/second - unlinked 0 (time 1746139934 ; total 0 ; last 0) total: 10000 unlinks in 56 seconds: 178.571426 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg460-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg460-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg460-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.160@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.160@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg460-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg460-server: oleg460-server.virtnet: executing set_default_debug -1 all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Started lustre-OST0000 oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg460-server: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg460-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec mount lustre on /mnt/lustre..... Starting client: oleg460-client.virtnet: -o user_xattr,flock 192.168.204.160@tcp:/lustre /mnt/lustre On OST0, 5261 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg460-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg460-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg460-server stop mds service on oleg460-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg460-server unloading modules via unload_modules_local on: 'oleg460-server' oleg460-server: oleg460-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg460-client: oleg460-server: ssh exited with exit code 2 pdsh@oleg460-client: oleg460-client: ssh exited with exit code 2 pdsh@oleg460-client: oleg460-client: ssh exited with exit code 2 pdsh@oleg460-client: oleg460-server: ssh exited with exit code 2