== conf-sanity test 69: replace an OST with the same index ========================================================== 19:38:08 (1746142688) start mds service on oleg239-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg239-server' oleg239-server: oleg239-server.virtnet: executing load_modules_local oleg239-server: Loading modules from /home/green/git/lustre-release/lustre oleg239-server: detected 4 online CPUs by sysfs oleg239-server: MODOPTS_LIBCFS= oleg239-server: Force libcfs to create 2 CPU partitions oleg239-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg239-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg239-server: oleg239-server.virtnet: executing set_default_debug -1 all pdsh@oleg239-client: oleg239-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg239-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg239-server: oleg239-server.virtnet: executing set_default_debug -1 all pdsh@oleg239-client: oleg239-server: ssh exited with exit code 1 Started lustre-OST0000 oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg239-client.virtnet: -o user_xattr,flock 192.168.202.139@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1250 (time 1746142732.69 total 10.00 last 124.97) - open/close 2368 (time 1746142742.69 total 20.01 last 111.76) - open/close 3260 (time 1746142752.70 total 30.01 last 89.16) - open/close 4181 (time 1746142762.70 total 40.01 last 92.09) - open/close 5094 (time 1746142772.70 total 50.02 last 91.26) - open/close 6173 (time 1746142782.71 total 60.02 last 107.87) - open/close 7351 (time 1746142792.71 total 70.02 last 117.74) - open/close 8358 (time 1746142802.71 total 80.03 last 100.69) - open/close 9450 (time 1746142812.72 total 90.03 last 109.13) total: 10000 open/close in 93.69 seconds: 106.73 ops/second - unlinked 0 (time 1746142818 ; total 0 ; last 0) total: 10000 unlinks in 50 seconds: 200.000000 unlinks/second - open/close 1242 (time 1746142880.18 total 10.00 last 124.17) - open/close 2467 (time 1746142890.18 total 20.00 last 122.48) - open/close 3746 (time 1746142900.18 total 30.01 last 127.85) - open/close 4968 (time 1746142910.19 total 40.02 last 122.10) - open/close 6232 (time 1746142920.20 total 50.02 last 126.37) - open/close 7188 (time 1746142930.22 total 60.04 last 95.40) - open/close 8178 (time 1746142940.23 total 70.05 last 98.90) - open/close 9220 (time 1746142950.23 total 80.05 last 104.15) total: 10000 open/close in 86.47 seconds: 115.64 ops/second - unlinked 0 (time 1746142958 ; total 0 ; last 0) total: 10000 unlinks in 53 seconds: 188.679245 unlinks/second - open/close 1055 (time 1746143023.81 total 10.01 last 105.40) - open/close 2071 (time 1746143033.81 total 20.01 last 101.59) - open/close 3198 (time 1746143043.81 total 30.01 last 112.69) - open/close 4283 (time 1746143053.81 total 40.02 last 108.45) - open/close 5306 (time 1746143063.82 total 50.02 last 102.22) - open/close 6331 (time 1746143073.83 total 60.03 last 102.42) - open/close 7275 (time 1746143083.84 total 70.04 last 94.29) - open/close 8178 (time 1746143093.85 total 80.05 last 90.24) - open/close 9186 (time 1746143103.85 total 90.05 last 100.78) total: 10000 open/close in 98.18 seconds: 101.86 ops/second - unlinked 0 (time 1746143114 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 964 (time 1746143190.67 total 10.01 last 96.34) - open/close 2111 (time 1746143200.67 total 20.01 last 114.64) - open/close 3059 (time 1746143210.69 total 30.02 last 94.68) - open/close 4060 (time 1746143220.69 total 40.03 last 100.02) - open/close 5157 (time 1746143230.70 total 50.04 last 109.65) - open/close 6149 (time 1746143240.70 total 60.04 last 99.16) - open/close 7105 (time 1746143250.71 total 70.05 last 95.55) - open/close 8099 (time 1746143260.71 total 80.05 last 99.36) - open/close 9120 (time 1746143270.73 total 90.07 last 101.94) total: 10000 open/close in 99.41 seconds: 100.60 ops/second - unlinked 0 (time 1746143282 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 1002 (time 1746143356.60 total 10.01 last 100.13) - open/close 2023 (time 1746143366.61 total 20.01 last 102.05) - open/close 3044 (time 1746143376.62 total 30.02 last 101.98) - open/close 4042 (time 1746143386.63 total 40.03 last 99.75) - open/close 5017 (time 1746143396.64 total 50.04 last 97.40) - open/close 6005 (time 1746143406.64 total 60.04 last 98.76) - open/close 7005 (time 1746143416.65 total 70.05 last 99.94) - open/close 8005 (time 1746143426.67 total 80.07 last 99.78) - open/close 9010 (time 1746143436.67 total 90.08 last 100.44) total: 10000 open/close in 99.24 seconds: 100.77 ops/second - unlinked 0 (time 1746143447 ; total 0 ; last 0) total: 10000 unlinks in 54 seconds: 185.185181 unlinks/second - open/close 1065 (time 1746143513.09 total 10.00 last 106.47) - open/close 2096 (time 1746143523.09 total 20.00 last 103.10) - open/close 3200 (time 1746143533.10 total 30.01 last 110.34) - open/close 4311 (time 1746143543.11 total 40.02 last 111.01) - open/close 5310 (time 1746143553.11 total 50.02 last 99.82) - open/close 6332 (time 1746143563.12 total 60.03 last 102.11) - open/close 7317 (time 1746143573.12 total 70.03 last 98.50) - open/close 8249 (time 1746143583.13 total 80.04 last 93.13) - open/close 9291 (time 1746143593.13 total 90.05 last 104.15) total: 10000 open/close in 97.05 seconds: 103.04 ops/second - unlinked 0 (time 1746143602 ; total 0 ; last 0) total: 10000 unlinks in 55 seconds: 181.818176 unlinks/second - open/close 988 (time 1746143669.58 total 10.00 last 98.77) - open/close 2032 (time 1746143679.58 total 20.00 last 104.38) - open/close 3044 (time 1746143689.60 total 30.02 last 101.08) - open/close 4028 (time 1746143699.60 total 40.02 last 98.39) - open/close 5080 (time 1746143709.61 total 50.03 last 105.06) - open/close 6228 (time 1746143719.62 total 60.04 last 114.71) - open/close 7190 (time 1746143729.63 total 70.05 last 96.08) - open/close 8234 (time 1746143739.64 total 80.06 last 104.32) - open/close 9366 (time 1746143749.64 total 90.06 last 113.15) total: 10000 open/close in 95.58 seconds: 104.62 ops/second - unlinked 0 (time 1746143756 ; total 0 ; last 0) total: 10000 unlinks in 45 seconds: 222.222229 unlinks/second - open/close 1478 (time 1746143812.62 total 10.00 last 147.77) - open/close 2848 (time 1746143822.62 total 20.00 last 136.97) - open/close 4317 (time 1746143832.62 total 30.01 last 146.88) - open/close 5510 (time 1746143842.62 total 40.01 last 119.29) - open/close 6683 (time 1746143852.62 total 50.01 last 117.28) - open/close 7845 (time 1746143862.62 total 60.01 last 116.19) - open/close 8982 (time 1746143872.63 total 70.02 last 113.60) total: 10000 open/close in 77.69 seconds: 128.72 ops/second - unlinked 0 (time 1746143882 ; total 0 ; last 0) total: 10000 unlinks in 43 seconds: 232.558136 unlinks/second - open/close 1231 (time 1746143937.28 total 10.01 last 123.04) - open/close 2467 (time 1746143947.28 total 20.01 last 123.55) - open/close 3652 (time 1746143957.29 total 30.01 last 118.45) - open/close 5105 (time 1746143967.29 total 40.02 last 145.27) - open/close 6243 (time 1746143977.30 total 50.03 last 113.67) - open/close 7598 (time 1746143987.30 total 60.03 last 135.49) - open/close 9007 (time 1746143997.31 total 70.03 last 140.82) total: 10000 open/close in 77.15 seconds: 129.62 ops/second - unlinked 0 (time 1746144006 ; total 0 ; last 0) total: 10000 unlinks in 51 seconds: 196.078430 unlinks/second - open/close 1074 (time 1746144069.76 total 10.00 last 107.35) - open/close 2189 (time 1746144079.76 total 20.01 last 111.47) - open/close 3357 (time 1746144089.77 total 30.01 last 116.73) - open/close 4433 (time 1746144099.77 total 40.01 last 107.59) - open/close 5562 (time 1746144109.78 total 50.03 last 112.74) - open/close 6775 (time 1746144119.79 total 60.03 last 121.25) - open/close 8012 (time 1746144129.79 total 70.04 last 123.62) - open/close 9138 (time 1746144139.80 total 80.04 last 112.56) total: 10000 open/close in 86.84 seconds: 115.15 ops/second - unlinked 0 (time 1746144148 ; total 0 ; last 0) total: 10000 unlinks in 48 seconds: 208.333328 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg239-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg239-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg239-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.139@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.139@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg239-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg239-server: oleg239-server.virtnet: executing set_default_debug -1 all pdsh@oleg239-client: oleg239-server: ssh exited with exit code 1 Started lustre-OST0000 oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg239-server: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg239-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg239-client.virtnet: -o user_xattr,flock 192.168.202.139@tcp:/lustre /mnt/lustre On OST0, 15117 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg239-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg239-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg239-server stop mds service on oleg239-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg239-server unloading modules via unload_modules_local on: 'oleg239-server' oleg239-server: oleg239-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg239-client: oleg239-client: ssh exited with exit code 2 pdsh@oleg239-client: oleg239-server: ssh exited with exit code 2 pdsh@oleg239-client: oleg239-client: ssh exited with exit code 2 pdsh@oleg239-client: oleg239-server: ssh exited with exit code 2