== conf-sanity test 69: replace an OST with the same index ========================================================== 20:12:48 (1746144768) start mds service on oleg241-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg241-server' oleg241-server: oleg241-server.virtnet: executing load_modules_local oleg241-server: Loading modules from /home/green/git/lustre-release/lustre oleg241-server: detected 4 online CPUs by sysfs oleg241-server: MODOPTS_LIBCFS= oleg241-server: Force libcfs to create 2 CPU partitions oleg241-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg241-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg241-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg241-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1 Started lustre-OST0000 oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg241-client.virtnet: -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1342 (time 1746144808.89 total 10.00 last 134.16) - open/close 2627 (time 1746144818.90 total 20.01 last 128.43) - open/close 4071 (time 1746144828.90 total 30.01 last 144.37) - open/close 5622 (time 1746144838.91 total 40.02 last 154.97) - open/close 7088 (time 1746144848.91 total 50.02 last 146.53) - open/close 8465 (time 1746144858.92 total 60.03 last 137.61) - open/close 9604 (time 1746144868.92 total 70.03 last 113.86) total: 10000 open/close in 73.21 seconds: 136.59 ops/second - unlinked 0 (time 1746144874 ; total 0 ; last 0) total: 10000 unlinks in 46 seconds: 217.391312 unlinks/second - open/close 1341 (time 1746144931.74 total 10.00 last 134.09) - open/close 2696 (time 1746144941.75 total 20.01 last 135.43) - open/close 4164 (time 1746144951.75 total 30.01 last 146.80) - open/close 5552 (time 1746144961.75 total 40.01 last 138.79) - open/close 6865 (time 1746144971.75 total 50.01 last 131.28) - open/close 8104 (time 1746144981.75 total 60.01 last 123.86) - open/close 9484 (time 1746144991.76 total 70.01 last 137.96) total: 10000 open/close in 74.17 seconds: 134.83 ops/second - unlinked 0 (time 1746144997 ; total 0 ; last 0) total: 10000 unlinks in 44 seconds: 227.272720 unlinks/second - open/close 1270 (time 1746145053.50 total 10.00 last 126.96) - open/close 2654 (time 1746145063.50 total 20.00 last 138.39) - open/close 3912 (time 1746145073.51 total 30.01 last 125.71) - open/close 5366 (time 1746145083.52 total 40.01 last 145.34) - open/close 7102 (time 1746145093.52 total 50.02 last 173.57) - open/close 8609 (time 1746145103.52 total 60.02 last 150.63) - open/close 9995 (time 1746145113.52 total 70.02 last 138.56) total: 10000 open/close in 70.05 seconds: 142.75 ops/second - unlinked 0 (time 1746145115 ; total 0 ; last 0) total: 10000 unlinks in 35 seconds: 285.714294 unlinks/second - open/close 1494 (time 1746145162.39 total 10.00 last 149.38) - open/close 2927 (time 1746145172.39 total 20.00 last 143.30) - open/close 4511 (time 1746145182.40 total 30.01 last 158.31) - open/close 6457 (time 1746145192.40 total 40.01 last 194.58) - open/close 8055 (time 1746145202.40 total 50.01 last 159.75) - open/close 9598 (time 1746145212.40 total 60.01 last 154.26) total: 10000 open/close in 63.38 seconds: 157.78 ops/second - unlinked 0 (time 1746145217 ; total 0 ; last 0) total: 10000 unlinks in 41 seconds: 243.902435 unlinks/second - open/close 1184 (time 1746145269.76 total 10.00 last 118.40) - open/close 2496 (time 1746145279.77 total 20.02 last 130.97) - open/close 3743 (time 1746145289.78 total 30.02 last 124.64) - open/close 4858 (time 1746145299.79 total 40.03 last 111.43) - open/close 6129 (time 1746145309.79 total 50.04 last 127.01) - open/close 7554 (time 1746145319.80 total 60.04 last 142.44) - open/close 8900 (time 1746145329.80 total 70.04 last 134.55) total: 10000 open/close in 77.30 seconds: 129.36 ops/second - unlinked 0 (time 1746145338 ; total 0 ; last 0) total: 10000 unlinks in 42 seconds: 238.095245 unlinks/second - open/close 1358 (time 1746145392.07 total 10.01 last 135.61) - open/close 3059 (time 1746145402.07 total 20.02 last 170.00) - open/close 4343 (time 1746145412.08 total 30.02 last 128.35) - open/close 5482 (time 1746145422.08 total 40.03 last 113.81) - open/close 6677 (time 1746145432.09 total 50.04 last 119.43) - open/close 7894 (time 1746145442.10 total 60.04 last 121.62) - open/close 9244 (time 1746145452.10 total 70.04 last 135.00) total: 10000 open/close in 78.14 seconds: 127.97 ops/second - unlinked 0 (time 1746145462 ; total 0 ; last 0) total: 10000 unlinks in 48 seconds: 208.333328 unlinks/second - open/close 954 (time 1746145522.82 total 10.01 last 95.34) - open/close 1745 (time 1746145532.82 total 20.02 last 79.03) - open/close 2633 (time 1746145542.83 total 30.02 last 88.79) - open/close 3510 (time 1746145552.83 total 40.02 last 87.67) - open/close 4497 (time 1746145562.83 total 50.02 last 98.67) - open/close 5635 (time 1746145572.83 total 60.02 last 113.79) - open/close 6849 (time 1746145582.84 total 70.03 last 121.34) - open/close 8064 (time 1746145592.84 total 80.03 last 121.49) - open/close 8944 (time 1746145602.84 total 90.03 last 87.99) - open/close 9934 (time 1746145612.84 total 100.03 last 98.99) total: 10000 open/close in 100.74 seconds: 99.26 ops/second - unlinked 0 (time 1746145615 ; total 0 ; last 0) total: 10000 unlinks in 50 seconds: 200.000000 unlinks/second - open/close 1106 (time 1746145677.05 total 10.00 last 110.59) - open/close 2247 (time 1746145687.06 total 20.00 last 114.09) - open/close 3312 (time 1746145697.06 total 30.00 last 106.49) - open/close 4434 (time 1746145707.06 total 40.01 last 112.15) - open/close 5630 (time 1746145717.06 total 50.01 last 119.57) - open/close 6734 (time 1746145727.06 total 60.01 last 110.39) - open/close 7860 (time 1746145737.07 total 70.01 last 112.56) - open/close 9011 (time 1746145747.07 total 80.02 last 115.04) total: 10000 open/close in 89.09 seconds: 112.24 ops/second - unlinked 0 (time 1746145758 ; total 0 ; last 0) total: 10000 unlinks in 54 seconds: 185.185181 unlinks/second - open/close 1218 (time 1746145824.40 total 10.00 last 121.78) - open/close 2670 (time 1746145834.40 total 20.00 last 145.17) - open/close 3821 (time 1746145844.41 total 30.01 last 115.08) - open/close 4948 (time 1746145854.42 total 40.02 last 112.56) - open/close 6030 (time 1746145864.43 total 50.03 last 108.10) - open/close 7028 (time 1746145874.45 total 60.04 last 99.63) - open/close 8057 (time 1746145884.46 total 70.06 last 102.71) - open/close 9196 (time 1746145894.47 total 80.07 last 113.88) total: 10000 open/close in 87.94 seconds: 113.72 ops/second - unlinked 0 (time 1746145904 ; total 0 ; last 0) total: 10000 unlinks in 55 seconds: 181.818176 unlinks/second - open/close 1080 (time 1746145971.29 total 10.00 last 107.95) - open/close 2323 (time 1746145981.29 total 20.01 last 124.29) - open/close 3428 (time 1746145991.32 total 30.03 last 110.23) - open/close 4775 (time 1746146001.32 total 40.03 last 134.66) - open/close 5927 (time 1746146011.32 total 50.03 last 115.19) - open/close 7185 (time 1746146021.33 total 60.04 last 125.66) - open/close 8118 (time 1746146031.33 total 70.05 last 93.28) - open/close 9286 (time 1746146041.34 total 80.05 last 116.73) total: 10000 open/close in 84.48 seconds: 118.37 ops/second - unlinked 0 (time 1746146047 ; total 0 ; last 0) total: 10000 unlinks in 46 seconds: 217.391312 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg241-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg241-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.141@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.141@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg241-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1 Started lustre-OST0000 oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg241-server: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg241-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec mount lustre on /mnt/lustre..... Starting client: oleg241-client.virtnet: -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre On OST0, 20325 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg241-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg241-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server stop mds service on oleg241-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg241-server unloading modules via unload_modules_local on: 'oleg241-server' oleg241-server: oleg241-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg241-client: oleg241-client: ssh exited with exit code 2 pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2 pdsh@oleg241-client: oleg241-client: ssh exited with exit code 2 pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2