== conf-sanity test 69: replace an OST with the same index ========================================================== 16:37:58 (1746131878) start mds service on oleg306-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg306-server' oleg306-server: oleg306-server.virtnet: executing load_modules_local oleg306-server: Loading modules from /home/green/git/lustre-release/lustre oleg306-server: detected 4 online CPUs by sysfs oleg306-server: MODOPTS_LIBCFS= oleg306-server: Force libcfs to create 2 CPU partitions oleg306-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg306-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg306-server: oleg306-server.virtnet: executing set_default_debug -1 all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg306-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg306-server: oleg306-server.virtnet: executing set_default_debug -1 all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-OST0000 oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg306-client.virtnet: -o user_xattr,flock 192.168.203.106@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1219 (time 1746131925.51 total 10.00 last 121.86) - open/close 2142 (time 1746131935.52 total 20.01 last 92.22) - open/close 3085 (time 1746131945.53 total 30.02 last 94.26) - open/close 4073 (time 1746131955.53 total 40.02 last 98.77) - open/close 5190 (time 1746131965.53 total 50.02 last 111.65) - open/close 6143 (time 1746131975.54 total 60.03 last 95.22) - open/close 7032 (time 1746131985.54 total 70.03 last 88.89) - open/close 7914 (time 1746131995.54 total 80.03 last 88.19) - open/close 8927 (time 1746132005.55 total 90.04 last 101.24) - open/close 9950 (time 1746132015.56 total 100.05 last 102.18) total: 10000 open/close in 100.60 seconds: 99.41 ops/second - unlinked 0 (time 1746132018 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 1084 (time 1746132093.95 total 10.01 last 108.33) - open/close 2145 (time 1746132103.95 total 20.01 last 106.02) - open/close 3076 (time 1746132113.97 total 30.02 last 93.00) - open/close 4027 (time 1746132123.97 total 40.03 last 95.04) - open/close 4989 (time 1746132133.98 total 50.03 last 96.17) - open/close 6008 (time 1746132143.98 total 60.04 last 101.85) - open/close 6947 (time 1746132154.00 total 70.05 last 93.76) - open/close 7833 (time 1746132164.00 total 80.06 last 88.55) - open/close 8736 (time 1746132174.01 total 90.07 last 90.24) - open/close 9710 (time 1746132184.01 total 100.07 last 97.36) total: 10000 open/close in 102.88 seconds: 97.20 ops/second - unlinked 0 (time 1746132188 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second - open/close 912 (time 1746132261.05 total 10.01 last 91.12) - open/close 1846 (time 1746132271.08 total 20.03 last 93.17) - open/close 3113 (time 1746132281.08 total 30.04 last 126.63) - open/close 4118 (time 1746132291.09 total 40.05 last 100.43) - open/close 5271 (time 1746132301.10 total 50.05 last 115.23) - open/close 6347 (time 1746132311.10 total 60.05 last 107.58) - open/close 7505 (time 1746132321.10 total 70.05 last 115.80) - open/close 8394 (time 1746132331.12 total 80.07 last 88.75) - open/close 9350 (time 1746132341.12 total 90.08 last 95.54) total: 10000 open/close in 96.53 seconds: 103.60 ops/second - unlinked 0 (time 1746132349 ; total 0 ; last 0) total: 10000 unlinks in 57 seconds: 175.438599 unlinks/second - open/close 897 (time 1746132418.10 total 10.00 last 89.67) - open/close 2119 (time 1746132428.10 total 20.01 last 122.17) - open/close 3164 (time 1746132438.11 total 30.02 last 104.39) - open/close 4025 (time 1746132448.12 total 40.02 last 86.03) - open/close 4992 (time 1746132458.12 total 50.03 last 96.68) - open/close 6199 (time 1746132468.14 total 60.05 last 120.46) - open/close 7098 (time 1746132478.14 total 70.05 last 89.89) - open/close 7994 (time 1746132488.16 total 80.07 last 89.42) - open/close 8917 (time 1746132498.17 total 90.07 last 92.26) total: 10000 open/close in 98.45 seconds: 101.58 ops/second - unlinked 0 (time 1746132508 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 983 (time 1746132582.55 total 10.00 last 98.28) - open/close 2114 (time 1746132592.55 total 20.01 last 113.04) - open/close 3048 (time 1746132602.56 total 30.01 last 93.39) - open/close 4049 (time 1746132612.57 total 40.02 last 99.98) - open/close 5136 (time 1746132622.57 total 50.03 last 108.63) - open/close 6218 (time 1746132632.58 total 60.03 last 108.19) - open/close 7299 (time 1746132642.58 total 70.03 last 108.08) - open/close 8476 (time 1746132652.58 total 80.04 last 117.63) - open/close 9645 (time 1746132662.58 total 90.04 last 116.89) total: 10000 open/close in 93.82 seconds: 106.59 ops/second - unlinked 0 (time 1746132668 ; total 0 ; last 0) total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second - open/close 1062 (time 1746132737.99 total 10.02 last 106.03) - open/close 2065 (time 1746132748.00 total 20.03 last 100.12) - open/close 3124 (time 1746132758.01 total 30.03 last 105.89) - open/close 4362 (time 1746132768.01 total 40.04 last 123.76) - open/close 5334 (time 1746132778.01 total 50.04 last 97.16) - open/close 6287 (time 1746132788.02 total 60.05 last 95.25) - open/close 7460 (time 1746132798.02 total 70.05 last 117.28) - open/close 8506 (time 1746132808.02 total 80.05 last 104.57) - open/close 9487 (time 1746132818.04 total 90.07 last 97.88) total: 10000 open/close in 95.23 seconds: 105.01 ops/second - unlinked 0 (time 1746132825 ; total 0 ; last 0) total: 10000 unlinks in 53 seconds: 188.679245 unlinks/second - open/close 955 (time 1746132890.44 total 10.00 last 95.49) - open/close 2031 (time 1746132900.45 total 20.01 last 107.56) - open/close 3334 (time 1746132910.45 total 30.01 last 130.29) - open/close 4432 (time 1746132920.45 total 40.01 last 109.73) - open/close 5619 (time 1746132930.45 total 50.01 last 118.68) - open/close 7012 (time 1746132940.47 total 60.03 last 139.09) - open/close 7984 (time 1746132950.48 total 70.04 last 97.13) - open/close 9050 (time 1746132960.48 total 80.04 last 106.59) total: 10000 open/close in 88.20 seconds: 113.38 ops/second - unlinked 0 (time 1746132969 ; total 0 ; last 0) total: 10000 unlinks in 49 seconds: 204.081635 unlinks/second - open/close 1371 (time 1746133030.55 total 10.01 last 137.00) - open/close 2514 (time 1746133040.56 total 20.01 last 114.24) - open/close 3565 (time 1746133050.56 total 30.02 last 105.07) - open/close 4695 (time 1746133060.57 total 40.02 last 112.93) - open/close 5955 (time 1746133070.57 total 50.03 last 125.95) - open/close 7128 (time 1746133080.57 total 60.03 last 117.29) - open/close 8402 (time 1746133090.58 total 70.03 last 127.37) - open/close 9877 (time 1746133100.60 total 80.05 last 147.19) total: 10000 open/close in 81.22 seconds: 123.12 ops/second - unlinked 0 (time 1746133103 ; total 0 ; last 0) total: 10000 unlinks in 53 seconds: 188.679245 unlinks/second - open/close 1202 (time 1746133168.71 total 10.00 last 120.19) - open/close 2211 (time 1746133178.71 total 20.01 last 100.85) - open/close 3244 (time 1746133188.72 total 30.01 last 103.20) - open/close 4344 (time 1746133198.72 total 40.02 last 109.98) - open/close 5750 (time 1746133208.72 total 50.02 last 140.56) - open/close 7055 (time 1746133218.74 total 60.04 last 130.25) - open/close 8398 (time 1746133228.75 total 70.04 last 134.26) - open/close 9493 (time 1746133238.75 total 80.05 last 109.44) total: 10000 open/close in 84.16 seconds: 118.82 ops/second - unlinked 0 (time 1746133244 ; total 0 ; last 0) total: 10000 unlinks in 45 seconds: 222.222229 unlinks/second - open/close 1089 (time 1746133301.70 total 10.00 last 108.90) - open/close 2308 (time 1746133311.70 total 20.00 last 121.88) - open/close 3605 (time 1746133321.71 total 30.01 last 129.60) - open/close 4776 (time 1746133331.72 total 40.02 last 117.00) - open/close 6102 (time 1746133341.72 total 50.02 last 132.57) - open/close 7521 (time 1746133351.72 total 60.02 last 141.90) - open/close 8808 (time 1746133361.72 total 70.02 last 128.68) total: 10000 open/close in 79.25 seconds: 126.18 ops/second - unlinked 0 (time 1746133372 ; total 0 ; last 0) total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg306-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg306-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg306-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.106@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.106@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg306-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg306-server: oleg306-server.virtnet: executing set_default_debug -1 all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-OST0000 oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg306-server: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg306-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec mount lustre on /mnt/lustre..... Starting client: oleg306-client.virtnet: -o user_xattr,flock 192.168.203.106@tcp:/lustre /mnt/lustre On OST0, 6773 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg306-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg306-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg306-server stop mds service on oleg306-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg306-server unloading modules via unload_modules_local on: 'oleg306-server' oleg306-server: oleg306-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg306-client: oleg306-client: ssh exited with exit code 2 pdsh@oleg306-client: oleg306-server: ssh exited with exit code 2 pdsh@oleg306-client: oleg306-server: ssh exited with exit code 2 pdsh@oleg306-client: oleg306-client: ssh exited with exit code 2