== conf-sanity test 69: replace an OST with the same index ========================================================== 19:20:28 (1746141628) start mds service on oleg424-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg424-server' oleg424-server: oleg424-server.virtnet: executing load_modules_local oleg424-server: Loading modules from /home/green/git/lustre-release/lustre oleg424-server: detected 4 online CPUs by sysfs oleg424-server: MODOPTS_LIBCFS= oleg424-server: Force libcfs to create 2 CPU partitions oleg424-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg424-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg424-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-OST0000 oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1122 (time 1746141671.73 total 10.01 last 112.14) - open/close 2984 (time 1746141681.73 total 20.01 last 186.15) - open/close 4266 (time 1746141691.73 total 30.01 last 128.19) - open/close 5435 (time 1746141701.74 total 40.01 last 116.84) - open/close 6678 (time 1746141711.74 total 50.02 last 124.25) - open/close 7862 (time 1746141721.75 total 60.02 last 118.34) - open/close 8944 (time 1746141731.75 total 70.02 last 108.18) total: 10000 open/close in 78.90 seconds: 126.75 ops/second - unlinked 0 (time 1746141741 ; total 0 ; last 0) total: 10000 unlinks in 53 seconds: 188.679245 unlinks/second - open/close 1449 (time 1746141806.23 total 10.00 last 144.85) - open/close 2791 (time 1746141816.23 total 20.01 last 134.18) - open/close 3964 (time 1746141826.24 total 30.01 last 117.25) - open/close 5228 (time 1746141836.24 total 40.01 last 126.38) - open/close 6473 (time 1746141846.24 total 50.02 last 124.41) - open/close 7540 (time 1746141856.26 total 60.03 last 106.57) - open/close 8588 (time 1746141866.27 total 70.04 last 104.70) - open/close 9677 (time 1746141876.27 total 80.04 last 108.87) total: 10000 open/close in 83.18 seconds: 120.22 ops/second - unlinked 0 (time 1746141881 ; total 0 ; last 0) total: 10000 unlinks in 54 seconds: 185.185181 unlinks/second - open/close 1002 (time 1746141947.79 total 10.00 last 100.20) - open/close 2035 (time 1746141957.80 total 20.01 last 103.23) - open/close 3174 (time 1746141967.81 total 30.01 last 113.82) - open/close 4265 (time 1746141977.81 total 40.01 last 109.09) - open/close 5476 (time 1746141987.81 total 50.02 last 121.03) - open/close 6881 (time 1746141997.83 total 60.03 last 140.32) - open/close 8139 (time 1746142007.83 total 70.04 last 125.73) - open/close 9479 (time 1746142017.84 total 80.04 last 133.94) total: 10000 open/close in 83.91 seconds: 119.17 ops/second - unlinked 0 (time 1746142023 ; total 0 ; last 0) total: 10000 unlinks in 46 seconds: 217.391312 unlinks/second - open/close 1328 (time 1746142080.73 total 10.00 last 132.78) - open/close 2549 (time 1746142090.73 total 20.00 last 122.07) - open/close 3702 (time 1746142100.73 total 30.01 last 115.29) - open/close 4812 (time 1746142110.73 total 40.01 last 111.00) - open/close 6192 (time 1746142120.75 total 50.02 last 137.84) - open/close 7502 (time 1746142130.75 total 60.02 last 130.97) - open/close 8790 (time 1746142140.75 total 70.02 last 128.79) - open/close 9981 (time 1746142150.76 total 80.03 last 119.03) total: 10000 open/close in 80.13 seconds: 124.80 ops/second - unlinked 0 (time 1746142152 ; total 0 ; last 0) total: 10000 unlinks in 50 seconds: 200.000000 unlinks/second - open/close 1143 (time 1746142213.79 total 10.01 last 114.19) - open/close 3121 (time 1746142223.79 total 20.01 last 197.75) - open/close 4694 (time 1746142233.80 total 30.02 last 157.17) - open/close 5868 (time 1746142243.80 total 40.02 last 117.38) - open/close 7025 (time 1746142253.81 total 50.03 last 115.65) - open/close 8245 (time 1746142263.81 total 60.03 last 121.99) - open/close 9439 (time 1746142273.81 total 70.03 last 119.38) total: 10000 open/close in 74.54 seconds: 134.15 ops/second - unlinked 0 (time 1746142280 ; total 0 ; last 0) total: 10000 unlinks in 37 seconds: 270.270264 unlinks/second - open/close 1261 (time 1746142329.60 total 10.01 last 125.98) - open/close 2493 (time 1746142339.61 total 20.01 last 123.14) - open/close 4031 (time 1746142349.61 total 30.02 last 153.71) - open/close 5385 (time 1746142359.62 total 40.02 last 135.33) - open/close 6691 (time 1746142369.63 total 50.03 last 130.51) - open/close 8033 (time 1746142379.63 total 60.03 last 134.17) - open/close 9239 (time 1746142389.64 total 70.05 last 120.47) total: 10000 open/close in 77.32 seconds: 129.33 ops/second - unlinked 0 (time 1746142398 ; total 0 ; last 0) total: 10000 unlinks in 47 seconds: 212.765961 unlinks/second - open/close 1296 (time 1746142456.98 total 10.00 last 129.60) - open/close 2940 (time 1746142466.99 total 20.00 last 164.38) - open/close 4737 (time 1746142476.99 total 30.00 last 179.69) - open/close 6109 (time 1746142486.99 total 40.01 last 137.12) - open/close 7538 (time 1746142496.99 total 50.01 last 142.86) - open/close 8845 (time 1746142507.00 total 60.01 last 130.68) total: 10000 open/close in 69.50 seconds: 143.89 ops/second - unlinked 0 (time 1746142518 ; total 0 ; last 0) total: 10000 unlinks in 50 seconds: 200.000000 unlinks/second - open/close 991 (time 1746142581.19 total 10.00 last 99.05) - open/close 2203 (time 1746142591.19 total 20.01 last 121.14) - open/close 3386 (time 1746142601.20 total 30.02 last 118.16) - open/close 4475 (time 1746142611.20 total 40.02 last 108.88) - open/close 5629 (time 1746142621.21 total 50.03 last 115.37) - open/close 6840 (time 1746142631.21 total 60.03 last 121.01) - open/close 8210 (time 1746142641.22 total 70.03 last 136.98) - open/close 9482 (time 1746142651.22 total 80.04 last 127.09) total: 10000 open/close in 83.42 seconds: 119.87 ops/second - unlinked 0 (time 1746142656 ; total 0 ; last 0) total: 10000 unlinks in 36 seconds: 277.777771 unlinks/second - open/close 1248 (time 1746142703.67 total 10.01 last 124.73) - open/close 2536 (time 1746142713.69 total 20.02 last 128.60) - open/close 3793 (time 1746142723.69 total 30.02 last 125.66) - open/close 5013 (time 1746142733.70 total 40.03 last 121.93) - open/close 6003 (time 1746142743.70 total 50.03 last 98.98) - open/close 7011 (time 1746142753.70 total 60.04 last 100.76) - open/close 8112 (time 1746142763.71 total 70.04 last 110.05) - open/close 9362 (time 1746142773.72 total 80.05 last 124.89) total: 10000 open/close in 85.52 seconds: 116.93 ops/second - unlinked 0 (time 1746142781 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 1114 (time 1746142854.44 total 10.00 last 111.38) - open/close 2263 (time 1746142864.45 total 20.01 last 114.84) - open/close 3391 (time 1746142874.45 total 30.01 last 112.79) - open/close 4548 (time 1746142884.45 total 40.01 last 115.69) - open/close 5959 (time 1746142894.45 total 50.01 last 141.06) - open/close 7184 (time 1746142904.46 total 60.02 last 122.43) - open/close 8404 (time 1746142914.46 total 70.02 last 121.94) - open/close 9329 (time 1746142924.47 total 80.03 last 92.46) total: 10000 open/close in 85.77 seconds: 116.59 ops/second - unlinked 0 (time 1746142932 ; total 0 ; last 0) total: 10000 unlinks in 42 seconds: 238.095245 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg424-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg424-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg424-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.124@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.124@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg424-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-OST0000 oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg424-server: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg424-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec mount lustre on /mnt/lustre..... Starting client: oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre On OST0, 17701 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg424-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg424-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg424-server stop mds service on oleg424-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg424-server unloading modules via unload_modules_local on: 'oleg424-server' oleg424-server: oleg424-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg424-client: oleg424-client: ssh exited with exit code 2 pdsh@oleg424-client: oleg424-server: ssh exited with exit code 2 pdsh@oleg424-client: oleg424-client: ssh exited with exit code 2 pdsh@oleg424-client: oleg424-server: ssh exited with exit code 2