== conf-sanity test 69: replace an OST with the same index ========================================================== 15:02:28 (1746126148) start mds service on oleg210-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg210-server' oleg210-server: oleg210-server.virtnet: executing load_modules_local oleg210-server: Loading modules from /home/green/git/lustre-release/lustre oleg210-server: detected 4 online CPUs by sysfs oleg210-server: MODOPTS_LIBCFS= oleg210-server: Force libcfs to create 2 CPU partitions oleg210-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg210-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg210-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg210-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0000 oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1148 (time 1746126191.81 total 10.00 last 114.77) - open/close 2673 (time 1746126201.81 total 20.00 last 152.49) - open/close 4194 (time 1746126211.81 total 30.00 last 152.10) - open/close 5686 (time 1746126221.81 total 40.01 last 149.14) - open/close 6859 (time 1746126231.82 total 50.01 last 117.26) - open/close 8276 (time 1746126241.82 total 60.01 last 141.66) - open/close 9544 (time 1746126251.82 total 70.02 last 126.72) total: 10000 open/close in 74.07 seconds: 135.02 ops/second - unlinked 0 (time 1746126257 ; total 0 ; last 0) total: 10000 unlinks in 57 seconds: 175.438599 unlinks/second - open/close 1203 (time 1746126326.12 total 10.01 last 120.21) - open/close 2301 (time 1746126336.12 total 20.01 last 109.78) - open/close 3546 (time 1746126346.13 total 30.01 last 124.47) - open/close 4751 (time 1746126356.13 total 40.02 last 120.40) - open/close 6178 (time 1746126366.14 total 50.02 last 142.66) - open/close 7603 (time 1746126376.14 total 60.02 last 142.45) - open/close 8873 (time 1746126386.14 total 70.03 last 126.97) total: 10000 open/close in 79.14 seconds: 126.36 ops/second - unlinked 0 (time 1746126396 ; total 0 ; last 0) total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second - open/close 1561 (time 1746126445.48 total 10.00 last 156.05) - open/close 3189 (time 1746126455.49 total 20.02 last 162.58) - open/close 4568 (time 1746126465.50 total 30.02 last 137.84) - open/close 5825 (time 1746126475.50 total 40.03 last 125.60) - open/close 7056 (time 1746126485.50 total 50.03 last 123.09) - open/close 8370 (time 1746126495.52 total 60.04 last 131.24) - open/close 9454 (time 1746126505.53 total 70.05 last 108.29) total: 10000 open/close in 75.57 seconds: 132.34 ops/second - unlinked 0 (time 1746126513 ; total 0 ; last 0) total: 10000 unlinks in 53 seconds: 188.679245 unlinks/second - open/close 1440 (time 1746126578.45 total 10.00 last 143.95) - open/close 2637 (time 1746126588.45 total 20.01 last 119.65) - open/close 4154 (time 1746126598.45 total 30.01 last 151.64) - open/close 5508 (time 1746126608.45 total 40.01 last 135.40) - open/close 7040 (time 1746126618.46 total 50.02 last 153.14) - open/close 8441 (time 1746126628.48 total 60.03 last 139.85) - open/close 9815 (time 1746126638.48 total 70.04 last 137.35) total: 10000 open/close in 71.28 seconds: 140.30 ops/second - unlinked 0 (time 1746126641 ; total 0 ; last 0) total: 10000 unlinks in 35 seconds: 285.714294 unlinks/second - open/close 1344 (time 1746126688.46 total 10.00 last 134.35) - open/close 2648 (time 1746126698.46 total 20.00 last 130.39) - open/close 4296 (time 1746126708.47 total 30.01 last 164.77) - open/close 5860 (time 1746126718.47 total 40.01 last 156.33) - open/close 7246 (time 1746126728.48 total 50.02 last 138.52) - open/close 8652 (time 1746126738.48 total 60.02 last 140.58) total: 10000 open/close in 69.89 seconds: 143.09 ops/second - unlinked 0 (time 1746126749 ; total 0 ; last 0) total: 10000 unlinks in 49 seconds: 204.081635 unlinks/second - open/close 1204 (time 1746126810.48 total 10.00 last 120.35) - open/close 2293 (time 1746126820.49 total 20.01 last 108.82) - open/close 3485 (time 1746126830.50 total 30.02 last 119.13) - open/close 5001 (time 1746126840.50 total 40.02 last 151.58) - open/close 6395 (time 1746126850.50 total 50.02 last 139.32) - open/close 7603 (time 1746126860.51 total 60.02 last 120.79) - open/close 8833 (time 1746126870.51 total 70.03 last 123.00) total: 10000 open/close in 79.89 seconds: 125.18 ops/second - unlinked 0 (time 1746126882 ; total 0 ; last 0) total: 10000 unlinks in 39 seconds: 256.410248 unlinks/second - open/close 1610 (time 1746126932.61 total 10.01 last 160.89) - open/close 3256 (time 1746126942.61 total 20.01 last 164.59) - open/close 4777 (time 1746126952.62 total 30.01 last 152.00) - open/close 6190 (time 1746126962.62 total 40.02 last 141.24) - open/close 7819 (time 1746126972.62 total 50.02 last 162.88) - open/close 9258 (time 1746126982.63 total 60.02 last 143.87) total: 10000 open/close in 66.04 seconds: 151.43 ops/second - unlinked 0 (time 1746126990 ; total 0 ; last 0) total: 10000 unlinks in 46 seconds: 217.391312 unlinks/second - open/close 1357 (time 1746127048.48 total 10.01 last 135.63) - open/close 2536 (time 1746127058.49 total 20.01 last 117.78) - open/close 3689 (time 1746127068.49 total 30.02 last 115.28) - open/close 5079 (time 1746127078.49 total 40.02 last 138.99) - open/close 6582 (time 1746127088.49 total 50.02 last 150.30) - open/close 7815 (time 1746127098.50 total 60.02 last 123.25) - open/close 9060 (time 1746127108.50 total 70.03 last 124.43) total: 10000 open/close in 75.96 seconds: 131.65 ops/second - unlinked 0 (time 1746127116 ; total 0 ; last 0) total: 10000 unlinks in 42 seconds: 238.095245 unlinks/second - open/close 1491 (time 1746127170.02 total 10.01 last 148.99) - open/close 2799 (time 1746127180.02 total 20.01 last 130.80) - open/close 4152 (time 1746127190.02 total 30.01 last 135.28) - open/close 5678 (time 1746127200.02 total 40.01 last 152.58) - open/close 7137 (time 1746127210.03 total 50.01 last 145.85) - open/close 8531 (time 1746127220.03 total 60.02 last 139.34) - open/close 9882 (time 1746127230.03 total 70.02 last 135.08) total: 10000 open/close in 70.85 seconds: 141.14 ops/second - unlinked 0 (time 1746127232 ; total 0 ; last 0) total: 10000 unlinks in 36 seconds: 277.777771 unlinks/second - open/close 1672 (time 1746127279.36 total 10.01 last 167.11) - open/close 3368 (time 1746127289.37 total 20.01 last 169.51) - open/close 4675 (time 1746127299.38 total 30.02 last 130.58) - open/close 5821 (time 1746127309.38 total 40.02 last 114.59) - open/close 7127 (time 1746127319.38 total 50.03 last 130.54) - open/close 8395 (time 1746127329.38 total 60.03 last 126.80) - open/close 9901 (time 1746127339.39 total 70.03 last 150.56) total: 10000 open/close in 70.77 seconds: 141.31 ops/second - unlinked 0 (time 1746127341 ; total 0 ; last 0) total: 10000 unlinks in 45 seconds: 222.222229 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg210-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg210-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg210-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.110@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.110@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg210-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0000 oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg210-server: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg210-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec mount lustre on /mnt/lustre..... Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre On OST0, 17701 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg210-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg210-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg210-server stop mds service on oleg210-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg210-server unloading modules via unload_modules_local on: 'oleg210-server' oleg210-server: oleg210-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg210-client: oleg210-client: ssh exited with exit code 2 pdsh@oleg210-client: oleg210-server: ssh exited with exit code 2 pdsh@oleg210-client: oleg210-server: ssh exited with exit code 2 pdsh@oleg210-client: oleg210-client: ssh exited with exit code 2