== conf-sanity test 69: replace an OST with the same index ========================================================== 21:58:38 (1746064718) start mds service on oleg209-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg209-server' oleg209-server: oleg209-server.virtnet: executing load_modules_local oleg209-server: Loading modules from /home/green/git/lustre-release/lustre oleg209-server: detected 4 online CPUs by sysfs oleg209-server: MODOPTS_LIBCFS= oleg209-server: Force libcfs to create 2 CPU partitions oleg209-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg209-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg209-server: oleg209-server.virtnet: executing set_default_debug -1 all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg209-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg209-server: oleg209-server.virtnet: executing set_default_debug -1 all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-OST0000 oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1065 (time 1746064762.77 total 10.00 last 106.50) - open/close 2116 (time 1746064772.77 total 20.00 last 105.07) - open/close 3201 (time 1746064782.77 total 30.01 last 108.47) - open/close 4366 (time 1746064792.78 total 40.01 last 116.43) - open/close 5513 (time 1746064802.78 total 50.01 last 114.69) - open/close 6851 (time 1746064812.79 total 60.02 last 133.70) - open/close 8290 (time 1746064822.79 total 70.02 last 143.84) - open/close 9457 (time 1746064832.80 total 80.03 last 116.59) total: 10000 open/close in 84.47 seconds: 118.38 ops/second - unlinked 0 (time 1746064839 ; total 0 ; last 0) total: 10000 unlinks in 51 seconds: 196.078430 unlinks/second - open/close 1387 (time 1746064901.73 total 10.01 last 138.56) - open/close 2572 (time 1746064911.73 total 20.01 last 118.49) - open/close 3736 (time 1746064921.73 total 30.01 last 116.37) - open/close 5034 (time 1746064931.74 total 40.02 last 129.70) - open/close 6191 (time 1746064941.74 total 50.03 last 115.64) - open/close 7447 (time 1746064951.75 total 60.03 last 125.59) - open/close 8732 (time 1746064961.75 total 70.04 last 128.38) total: 10000 open/close in 79.17 seconds: 126.31 ops/second - unlinked 0 (time 1746064972 ; total 0 ; last 0) total: 10000 unlinks in 47 seconds: 212.765961 unlinks/second - open/close 1238 (time 1746065031.28 total 10.00 last 123.76) - open/close 2306 (time 1746065041.29 total 20.01 last 106.74) - open/close 3362 (time 1746065051.29 total 30.02 last 105.52) - open/close 4456 (time 1746065061.30 total 40.02 last 109.32) - open/close 5629 (time 1746065071.30 total 50.03 last 117.28) - open/close 6746 (time 1746065081.30 total 60.03 last 111.70) - open/close 7987 (time 1746065091.31 total 70.03 last 124.04) - open/close 9343 (time 1746065101.31 total 80.04 last 135.52) total: 10000 open/close in 85.14 seconds: 117.45 ops/second - unlinked 0 (time 1746065107 ; total 0 ; last 0) total: 10000 unlinks in 48 seconds: 208.333328 unlinks/second - open/close 1161 (time 1746065167.83 total 10.00 last 116.08) - open/close 2219 (time 1746065177.83 total 20.00 last 105.80) - open/close 3434 (time 1746065187.84 total 30.01 last 121.45) - open/close 4670 (time 1746065197.85 total 40.02 last 123.42) - open/close 5753 (time 1746065207.86 total 50.03 last 108.24) - open/close 6944 (time 1746065217.86 total 60.03 last 119.05) - open/close 8099 (time 1746065227.86 total 70.03 last 115.48) - open/close 9428 (time 1746065237.86 total 80.03 last 132.88) total: 10000 open/close in 83.79 seconds: 119.35 ops/second - unlinked 0 (time 1746065243 ; total 0 ; last 0) total: 10000 unlinks in 39 seconds: 256.410248 unlinks/second - open/close 1432 (time 1746065293.83 total 10.00 last 143.17) - open/close 2926 (time 1746065303.83 total 20.00 last 149.36) - open/close 4078 (time 1746065313.84 total 30.01 last 115.11) - open/close 5430 (time 1746065323.84 total 40.01 last 135.19) - open/close 6621 (time 1746065333.84 total 50.01 last 119.09) - open/close 7837 (time 1746065343.86 total 60.03 last 121.38) - open/close 9030 (time 1746065353.87 total 70.04 last 119.25) total: 10000 open/close in 78.89 seconds: 126.77 ops/second - unlinked 0 (time 1746065364 ; total 0 ; last 0) total: 10000 unlinks in 43 seconds: 232.558136 unlinks/second - open/close 1171 (time 1746065419.54 total 10.01 last 117.01) - open/close 2506 (time 1746065429.54 total 20.01 last 133.49) - open/close 3861 (time 1746065439.55 total 30.02 last 135.39) - open/close 5032 (time 1746065449.56 total 40.02 last 117.02) - open/close 6120 (time 1746065459.56 total 50.02 last 108.79) - open/close 7389 (time 1746065469.56 total 60.03 last 126.88) - open/close 8641 (time 1746065479.57 total 70.04 last 125.06) - open/close 9812 (time 1746065489.57 total 80.04 last 117.07) total: 10000 open/close in 81.78 seconds: 122.28 ops/second - unlinked 0 (time 1746065493 ; total 0 ; last 0) total: 10000 unlinks in 49 seconds: 204.081635 unlinks/second - open/close 1450 (time 1746065554.09 total 10.00 last 144.95) - open/close 2798 (time 1746065564.10 total 20.01 last 134.72) - open/close 4102 (time 1746065574.11 total 30.02 last 130.30) - open/close 5380 (time 1746065584.11 total 40.02 last 127.75) - open/close 6665 (time 1746065594.12 total 50.03 last 128.43) - open/close 8202 (time 1746065604.12 total 60.03 last 153.69) - open/close 9642 (time 1746065614.12 total 70.03 last 143.96) total: 10000 open/close in 72.46 seconds: 138.00 ops/second - unlinked 0 (time 1746065618 ; total 0 ; last 0) total: 10000 unlinks in 50 seconds: 200.000000 unlinks/second - open/close 1261 (time 1746065680.19 total 10.00 last 126.09) - open/close 2584 (time 1746065690.19 total 20.00 last 132.27) - open/close 3924 (time 1746065700.19 total 30.01 last 133.97) - open/close 5147 (time 1746065710.19 total 40.01 last 122.28) - open/close 6381 (time 1746065720.20 total 50.01 last 123.37) - open/close 7600 (time 1746065730.20 total 60.01 last 121.87) - open/close 8768 (time 1746065740.22 total 70.03 last 116.61) total: 10000 open/close in 78.89 seconds: 126.76 ops/second - unlinked 0 (time 1746065750 ; total 0 ; last 0) total: 10000 unlinks in 44 seconds: 227.272720 unlinks/second - open/close 1100 (time 1746065806.01 total 10.01 last 109.93) - open/close 2293 (time 1746065816.03 total 20.02 last 119.15) - open/close 3305 (time 1746065826.04 total 30.03 last 101.07) - open/close 4391 (time 1746065836.04 total 40.03 last 108.57) - open/close 5352 (time 1746065846.06 total 50.05 last 95.93) - open/close 6393 (time 1746065856.07 total 60.06 last 104.04) - open/close 7413 (time 1746065866.07 total 70.06 last 101.97) - open/close 8374 (time 1746065876.08 total 80.07 last 96.02) - open/close 9422 (time 1746065886.09 total 90.08 last 104.63) total: 10000 open/close in 95.99 seconds: 104.18 ops/second - unlinked 0 (time 1746065894 ; total 0 ; last 0) total: 10000 unlinks in 52 seconds: 192.307693 unlinks/second - open/close 1511 (time 1746065958.08 total 10.01 last 151.01) - open/close 2888 (time 1746065968.08 total 20.01 last 137.65) - open/close 4107 (time 1746065978.08 total 30.01 last 121.89) - open/close 5442 (time 1746065988.09 total 40.02 last 133.41) - open/close 6782 (time 1746065998.09 total 50.02 last 133.96) - open/close 7987 (time 1746066008.09 total 60.02 last 120.50) - open/close 9467 (time 1746066018.10 total 70.03 last 147.92) total: 10000 open/close in 73.84 seconds: 135.43 ops/second - unlinked 0 (time 1746066023 ; total 0 ; last 0) total: 10000 unlinks in 42 seconds: 238.095245 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg209-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg209-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg209-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.109@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.109@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg209-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg209-server: oleg209-server.virtnet: executing set_default_debug -1 all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-OST0000 oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg209-server: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg209-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec mount lustre on /mnt/lustre..... Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre On OST0, 6533 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg209-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg209-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg209-server stop mds service on oleg209-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg209-server unloading modules via unload_modules_local on: 'oleg209-server' oleg209-server: oleg209-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg209-client: oleg209-client: ssh exited with exit code 2 pdsh@oleg209-client: oleg209-server: ssh exited with exit code 2 pdsh@oleg209-client: oleg209-server: ssh exited with exit code 2 pdsh@oleg209-client: oleg209-client: ssh exited with exit code 2