== conf-sanity test 69: replace an OST with the same index ========================================================== 15:54:51 (1746129291) start mds service on oleg235-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg235-server' oleg235-server: oleg235-server.virtnet: executing load_modules_local oleg235-server: Loading modules from /home/green/git/lustre-release/lustre oleg235-server: detected 4 online CPUs by sysfs oleg235-server: MODOPTS_LIBCFS= oleg235-server: Force libcfs to create 2 CPU partitions oleg235-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg235-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg235-server: oleg235-server.virtnet: executing set_default_debug -1 all pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg235-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg235-server: oleg235-server.virtnet: executing set_default_debug -1 all pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1 Started lustre-OST0000 oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg235-client.virtnet: -o user_xattr,flock 192.168.202.135@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1306 (time 1746129332.84 total 10.00 last 130.58) - open/close 2785 (time 1746129342.84 total 20.00 last 147.87) - open/close 4200 (time 1746129352.84 total 30.01 last 141.41) - open/close 5404 (time 1746129362.85 total 40.02 last 120.27) - open/close 6461 (time 1746129372.87 total 50.03 last 105.57) - open/close 7617 (time 1746129382.87 total 60.03 last 115.60) - open/close 8884 (time 1746129392.87 total 70.04 last 126.63) total: 10000 open/close in 77.91 seconds: 128.36 ops/second - unlinked 0 (time 1746129402 ; total 0 ; last 0) total: 10000 unlinks in 50 seconds: 200.000000 unlinks/second - open/close 1226 (time 1746129463.99 total 10.01 last 122.53) - open/close 2188 (time 1746129473.99 total 20.01 last 96.14) - open/close 3415 (time 1746129484.00 total 30.02 last 122.66) - open/close 4682 (time 1746129494.00 total 40.02 last 126.62) - open/close 6109 (time 1746129504.01 total 50.03 last 142.62) - open/close 7177 (time 1746129514.01 total 60.03 last 106.75) - open/close 8246 (time 1746129524.01 total 70.03 last 106.89) - open/close 9439 (time 1746129534.02 total 80.03 last 119.28) total: 10000 open/close in 84.31 seconds: 118.62 ops/second - unlinked 0 (time 1746129539 ; total 0 ; last 0) total: 10000 unlinks in 55 seconds: 181.818176 unlinks/second - open/close 1127 (time 1746129606.15 total 10.00 last 112.68) - open/close 2142 (time 1746129616.15 total 20.01 last 101.46) - open/close 3392 (time 1746129626.15 total 30.01 last 124.99) - open/close 5133 (time 1746129636.16 total 40.01 last 174.00) - open/close 6584 (time 1746129646.16 total 50.02 last 145.01) - open/close 7878 (time 1746129656.17 total 60.02 last 129.31) - open/close 9247 (time 1746129666.17 total 70.03 last 136.84) total: 10000 open/close in 74.75 seconds: 133.77 ops/second - unlinked 0 (time 1746129672 ; total 0 ; last 0) total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second - open/close 1643 (time 1746129721.86 total 10.00 last 164.28) - open/close 3301 (time 1746129731.87 total 20.01 last 165.70) - open/close 4907 (time 1746129741.87 total 30.01 last 160.53) - open/close 6374 (time 1746129751.87 total 40.01 last 146.67) - open/close 7666 (time 1746129761.88 total 50.02 last 129.12) - open/close 9216 (time 1746129771.89 total 60.03 last 154.86) total: 10000 open/close in 66.47 seconds: 150.44 ops/second - unlinked 0 (time 1746129780 ; total 0 ; last 0) total: 10000 unlinks in 44 seconds: 227.272720 unlinks/second - open/close 1596 (time 1746129836.35 total 10.01 last 159.48) - open/close 2975 (time 1746129846.35 total 20.01 last 137.83) - open/close 4228 (time 1746129856.36 total 30.02 last 125.19) - open/close 5911 (time 1746129866.37 total 40.03 last 168.22) - open/close 7468 (time 1746129876.37 total 50.03 last 155.61) - open/close 9122 (time 1746129886.38 total 60.04 last 165.34) total: 10000 open/close in 65.09 seconds: 153.63 ops/second - unlinked 0 (time 1746129892 ; total 0 ; last 0) total: 10000 unlinks in 40 seconds: 250.000000 unlinks/second - open/close 1045 (time 1746129945.16 total 10.01 last 104.41) - open/close 2308 (time 1746129955.17 total 20.01 last 126.25) - open/close 3528 (time 1746129965.18 total 30.03 last 121.81) - open/close 4713 (time 1746129975.19 total 40.03 last 118.45) - open/close 5997 (time 1746129985.19 total 50.03 last 128.38) - open/close 7305 (time 1746129995.20 total 60.04 last 130.73) - open/close 8697 (time 1746130005.20 total 70.04 last 139.17) total: 10000 open/close in 78.80 seconds: 126.90 ops/second - unlinked 0 (time 1746130015 ; total 0 ; last 0) total: 10000 unlinks in 46 seconds: 217.391312 unlinks/second - open/close 1516 (time 1746130072.55 total 10.00 last 151.60) - open/close 3223 (time 1746130082.56 total 20.00 last 170.62) - open/close 4709 (time 1746130092.56 total 30.01 last 148.50) - open/close 6106 (time 1746130102.57 total 40.02 last 139.64) - open/close 7601 (time 1746130112.57 total 50.02 last 149.41) - open/close 9053 (time 1746130122.58 total 60.03 last 145.12) total: 10000 open/close in 66.15 seconds: 151.16 ops/second - unlinked 0 (time 1746130130 ; total 0 ; last 0) total: 10000 unlinks in 46 seconds: 217.391312 unlinks/second - open/close 1327 (time 1746130187.90 total 10.00 last 132.66) - open/close 3066 (time 1746130197.90 total 20.01 last 173.80) - open/close 4816 (time 1746130207.91 total 30.01 last 174.97) - open/close 6346 (time 1746130217.91 total 40.01 last 152.94) - open/close 7996 (time 1746130227.91 total 50.02 last 164.93) - open/close 9501 (time 1746130237.92 total 60.02 last 150.48) total: 10000 open/close in 63.61 seconds: 157.20 ops/second - unlinked 0 (time 1746130243 ; total 0 ; last 0) total: 10000 unlinks in 36 seconds: 277.777771 unlinks/second - open/close 1654 (time 1746130290.80 total 10.00 last 165.32) - open/close 3420 (time 1746130300.80 total 20.01 last 176.53) - open/close 4929 (time 1746130310.81 total 30.01 last 150.84) - open/close 6253 (time 1746130320.82 total 40.02 last 132.30) - open/close 7374 (time 1746130330.82 total 50.03 last 112.04) - open/close 8726 (time 1746130340.82 total 60.03 last 135.16) - open/close 9995 (time 1746130350.83 total 70.04 last 126.82) total: 10000 open/close in 70.07 seconds: 142.72 ops/second - unlinked 0 (time 1746130352 ; total 0 ; last 0) total: 10000 unlinks in 43 seconds: 232.558136 unlinks/second - open/close 1405 (time 1746130407.33 total 10.00 last 140.48) - open/close 2560 (time 1746130417.34 total 20.00 last 115.46) - open/close 3439 (time 1746130427.34 total 30.01 last 87.88) - open/close 4615 (time 1746130437.34 total 40.01 last 117.60) - open/close 5554 (time 1746130447.35 total 50.02 last 93.80) - open/close 6593 (time 1746130457.35 total 60.02 last 103.89) - open/close 7850 (time 1746130467.36 total 70.02 last 125.64) - open/close 8942 (time 1746130477.36 total 80.02 last 109.19) - open/close 9997 (time 1746130487.36 total 90.03 last 105.43) total: 10000 open/close in 90.05 seconds: 111.05 ops/second - unlinked 0 (time 1746130489 ; total 0 ; last 0) total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg235-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg235-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg235-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.135@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.135@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg235-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg235-server: oleg235-server.virtnet: executing set_default_debug -1 all pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1 Started lustre-OST0000 oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg235-server: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg235-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec mount lustre on /mnt/lustre..... Starting client: oleg235-client.virtnet: -o user_xattr,flock 192.168.202.135@tcp:/lustre /mnt/lustre On OST0, 18069 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg235-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg235-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg235-server stop mds service on oleg235-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg235-server unloading modules via unload_modules_local on: 'oleg235-server' oleg235-server: oleg235-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg235-client: oleg235-client: ssh exited with exit code 2 pdsh@oleg235-client: oleg235-server: ssh exited with exit code 2 pdsh@oleg235-client: oleg235-client: ssh exited with exit code 2 pdsh@oleg235-client: oleg235-server: ssh exited with exit code 2