== conf-sanity test 69: replace an OST with the same index ========================================================== 15:30:29 (1746127829) start mds service on oleg210-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg210-server' oleg210-server: oleg210-server.virtnet: executing load_modules_local oleg210-server: Loading modules from /home/green/git/lustre-release/lustre oleg210-server: detected 4 online CPUs by sysfs oleg210-server: MODOPTS_LIBCFS= oleg210-server: Force libcfs to create 2 CPU partitions oleg210-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg210-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg210-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg210-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0000 oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1474 (time 1746127874.51 total 10.00 last 147.38) - open/close 2871 (time 1746127884.51 total 20.00 last 139.65) - open/close 4140 (time 1746127894.52 total 30.01 last 126.81) - open/close 5392 (time 1746127904.54 total 40.03 last 125.01) - open/close 6661 (time 1746127914.54 total 50.03 last 126.83) - open/close 7933 (time 1746127924.55 total 60.04 last 127.14) - open/close 9138 (time 1746127934.55 total 70.05 last 120.39) total: 10000 open/close in 77.96 seconds: 128.27 ops/second - unlinked 0 (time 1746127944 ; total 0 ; last 0) total: 10000 unlinks in 42 seconds: 238.095245 unlinks/second - open/close 1248 (time 1746127998.19 total 10.00 last 124.78) - open/close 2492 (time 1746128008.20 total 20.01 last 124.30) - open/close 3782 (time 1746128018.20 total 30.01 last 128.97) - open/close 4918 (time 1746128028.20 total 40.02 last 113.54) - open/close 6133 (time 1746128038.22 total 50.03 last 121.33) - open/close 7229 (time 1746128048.22 total 60.04 last 109.55) - open/close 8405 (time 1746128058.23 total 70.04 last 117.55) - open/close 9592 (time 1746128068.23 total 80.04 last 118.69) total: 10000 open/close in 82.84 seconds: 120.72 ops/second - unlinked 0 (time 1746128072 ; total 0 ; last 0) total: 10000 unlinks in 56 seconds: 178.571426 unlinks/second - open/close 1047 (time 1746128140.66 total 10.00 last 104.66) - open/close 2182 (time 1746128150.66 total 20.01 last 113.43) - open/close 3487 (time 1746128160.66 total 30.01 last 130.48) - open/close 4717 (time 1746128170.67 total 40.02 last 122.93) - open/close 6038 (time 1746128180.67 total 50.02 last 132.05) - open/close 7484 (time 1746128190.68 total 60.03 last 144.48) - open/close 8652 (time 1746128200.69 total 70.04 last 116.72) - open/close 9813 (time 1746128210.70 total 80.04 last 116.00) total: 10000 open/close in 81.79 seconds: 122.26 ops/second - unlinked 0 (time 1746128214 ; total 0 ; last 0) total: 10000 unlinks in 33 seconds: 303.030304 unlinks/second - open/close 2129 (time 1746128258.92 total 10.00 last 212.85) - open/close 3895 (time 1746128268.92 total 20.00 last 176.58) - open/close 5726 (time 1746128278.93 total 30.01 last 183.00) - open/close 7227 (time 1746128288.93 total 40.01 last 150.04) - open/close 9177 (time 1746128298.94 total 50.02 last 194.96) total: 10000 open/close in 53.52 seconds: 186.85 ops/second - unlinked 0 (time 1746128303 ; total 0 ; last 0) total: 10000 unlinks in 36 seconds: 277.777771 unlinks/second - open/close 1502 (time 1746128350.47 total 10.00 last 150.17) - open/close 2951 (time 1746128360.47 total 20.00 last 144.89) - open/close 4225 (time 1746128370.48 total 30.01 last 127.36) - open/close 5446 (time 1746128380.50 total 40.03 last 121.85) - open/close 6550 (time 1746128390.50 total 50.03 last 110.39) - open/close 7619 (time 1746128400.52 total 60.04 last 106.71) - open/close 8731 (time 1746128410.52 total 70.05 last 111.13) - open/close 9936 (time 1746128420.52 total 80.05 last 120.50) total: 10000 open/close in 80.54 seconds: 124.15 ops/second - unlinked 0 (time 1746128423 ; total 0 ; last 0) total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second - open/close 981 (time 1746128493.75 total 10.02 last 97.94) - open/close 1981 (time 1746128503.77 total 20.03 last 99.82) - open/close 2944 (time 1746128513.77 total 30.04 last 96.26) - open/close 3853 (time 1746128523.81 total 40.08 last 90.54) - open/close 4815 (time 1746128533.83 total 50.09 last 96.06) - open/close 5841 (time 1746128543.83 total 60.10 last 102.53) - open/close 6882 (time 1746128553.84 total 70.11 last 104.00) - open/close 7890 (time 1746128563.85 total 80.12 last 100.73) - open/close 8843 (time 1746128573.87 total 90.14 last 95.08) - open/close 9869 (time 1746128583.87 total 100.14 last 102.60) total: 10000 open/close in 101.30 seconds: 98.72 ops/second - unlinked 0 (time 1746128587 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 954 (time 1746128670.76 total 10.00 last 95.39) - open/close 1873 (time 1746128680.77 total 20.00 last 91.88) - open/close 2741 (time 1746128690.77 total 30.00 last 86.78) - open/close 3679 (time 1746128700.77 total 40.01 last 93.78) - open/close 4619 (time 1746128710.77 total 50.01 last 93.98) - open/close 5512 (time 1746128720.78 total 60.02 last 89.22) - open/close 6391 (time 1746128730.78 total 70.02 last 87.89) - open/close 7159 (time 1746128740.79 total 80.03 last 76.75) - open/close 8031 (time 1746128750.80 total 90.03 last 87.15) - open/close 8978 (time 1746128760.80 total 100.04 last 94.66) total: 10000 open/close in 109.43 seconds: 91.38 ops/second - unlinked 0 (time 1746128772 ; total 0 ; last 0) total: 10000 unlinks in 55 seconds: 181.818176 unlinks/second - open/close 1201 (time 1746128839.64 total 10.00 last 120.09) - open/close 2498 (time 1746128849.65 total 20.01 last 129.60) - open/close 3745 (time 1746128859.65 total 30.01 last 124.69) - open/close 5039 (time 1746128869.65 total 40.01 last 129.37) - open/close 6169 (time 1746128879.66 total 50.02 last 112.91) - open/close 7417 (time 1746128889.69 total 60.05 last 124.43) - open/close 8456 (time 1746128899.69 total 70.05 last 103.89) - open/close 9506 (time 1746128909.69 total 80.05 last 104.96) total: 10000 open/close in 84.03 seconds: 119.00 ops/second - unlinked 0 (time 1746128915 ; total 0 ; last 0) total: 10000 unlinks in 55 seconds: 181.818176 unlinks/second - open/close 1241 (time 1746128982.48 total 10.00 last 124.08) - open/close 2547 (time 1746128992.49 total 20.01 last 130.50) - open/close 3863 (time 1746129002.49 total 30.01 last 131.57) - open/close 5253 (time 1746129012.49 total 40.01 last 138.97) - open/close 6510 (time 1746129022.50 total 50.02 last 125.63) - open/close 7662 (time 1746129032.50 total 60.02 last 115.19) - open/close 8880 (time 1746129042.50 total 70.03 last 121.73) total: 10000 open/close in 79.46 seconds: 125.84 ops/second - unlinked 0 (time 1746129053 ; total 0 ; last 0) total: 10000 unlinks in 47 seconds: 212.765961 unlinks/second - open/close 1183 (time 1746129112.67 total 10.01 last 118.17) - open/close 2343 (time 1746129122.67 total 20.01 last 115.96) - open/close 3547 (time 1746129132.68 total 30.03 last 120.25) - open/close 4740 (time 1746129142.69 total 40.03 last 119.21) - open/close 6078 (time 1746129152.70 total 50.04 last 133.74) - open/close 7564 (time 1746129162.70 total 60.05 last 148.50) - open/close 8998 (time 1746129172.71 total 70.05 last 143.34) total: 10000 open/close in 77.05 seconds: 129.79 ops/second - unlinked 0 (time 1746129181 ; total 0 ; last 0) total: 10000 unlinks in 42 seconds: 238.095245 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg210-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg210-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg210-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.110@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.110@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg210-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0000 oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg210-server: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg210-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 6 sec mount lustre on /mnt/lustre..... Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre On OST0, 17693 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg210-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg210-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg210-server stop mds service on oleg210-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg210-server unloading modules via unload_modules_local on: 'oleg210-server' oleg210-server: oleg210-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg210-client: oleg210-client: ssh exited with exit code 2 pdsh@oleg210-client: oleg210-server: ssh exited with exit code 2 pdsh@oleg210-client: oleg210-client: ssh exited with exit code 2 pdsh@oleg210-client: oleg210-server: ssh exited with exit code 2