== conf-sanity test 69: replace an OST with the same index ========================================================== 16:56:02 (1746132962) start mds service on oleg155-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg155-server' oleg155-server: oleg155-server.virtnet: executing load_modules_local oleg155-server: Loading modules from /home/green/git/lustre-release/lustre oleg155-server: detected 4 online CPUs by sysfs oleg155-server: MODOPTS_LIBCFS= oleg155-server: Force libcfs to create 2 CPU partitions oleg155-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg155-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg155-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg155-server: oleg155-server.virtnet: executing set_default_debug -1 all pdsh@oleg155-client: oleg155-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg155-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg155-server: oleg155-server.virtnet: executing set_default_debug -1 all pdsh@oleg155-client: oleg155-server: ssh exited with exit code 1 Started lustre-OST0000 oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg155-client.virtnet: -o user_xattr,flock 192.168.201.155@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1106 (time 1746133008.75 total 10.00 last 110.55) - open/close 2218 (time 1746133018.75 total 20.01 last 111.16) - open/close 3519 (time 1746133028.76 total 30.02 last 130.01) - open/close 4664 (time 1746133038.76 total 40.02 last 114.48) - open/close 5704 (time 1746133048.77 total 50.03 last 103.91) - open/close 6864 (time 1746133058.78 total 60.03 last 115.91) - open/close 8204 (time 1746133068.78 total 70.04 last 133.90) - open/close 9548 (time 1746133078.79 total 80.05 last 134.31) total: 10000 open/close in 83.66 seconds: 119.53 ops/second - unlinked 0 (time 1746133084 ; total 0 ; last 0) total: 10000 unlinks in 54 seconds: 185.185181 unlinks/second - open/close 1057 (time 1746133150.21 total 10.00 last 105.65) - open/close 2310 (time 1746133160.21 total 20.01 last 125.24) - open/close 3423 (time 1746133170.22 total 30.02 last 111.22) - open/close 4457 (time 1746133180.23 total 40.03 last 103.27) - open/close 5612 (time 1746133190.25 total 50.05 last 115.31) - open/close 6831 (time 1746133200.25 total 60.05 last 121.87) - open/close 7972 (time 1746133210.26 total 70.05 last 114.05) - open/close 9084 (time 1746133220.27 total 80.06 last 111.08) total: 10000 open/close in 87.88 seconds: 113.79 ops/second - unlinked 0 (time 1746133229 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second - open/close 994 (time 1746133301.41 total 10.01 last 99.30) - open/close 1964 (time 1746133311.41 total 20.01 last 96.95) - open/close 3010 (time 1746133321.43 total 30.03 last 104.42) - open/close 4102 (time 1746133331.43 total 40.03 last 109.17) - open/close 5080 (time 1746133341.44 total 50.04 last 97.70) - open/close 6080 (time 1746133351.47 total 60.07 last 99.73) - open/close 6975 (time 1746133361.47 total 70.07 last 89.49) - open/close 8084 (time 1746133371.48 total 80.08 last 110.87) - open/close 8972 (time 1746133381.49 total 90.09 last 88.68) - open/close 9976 (time 1746133391.49 total 100.09 last 100.38) total: 10000 open/close in 100.26 seconds: 99.74 ops/second - unlinked 0 (time 1746133393 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 1157 (time 1746133468.18 total 10.02 last 115.52) - open/close 2182 (time 1746133478.19 total 20.03 last 102.38) - open/close 3238 (time 1746133488.20 total 30.03 last 105.57) - open/close 4285 (time 1746133498.20 total 40.03 last 104.70) - open/close 5268 (time 1746133508.21 total 50.04 last 98.20) - open/close 6152 (time 1746133518.22 total 60.05 last 88.33) - open/close 7089 (time 1746133528.22 total 70.06 last 93.64) - open/close 8156 (time 1746133538.24 total 80.07 last 106.54) - open/close 9270 (time 1746133548.24 total 90.07 last 111.40) total: 10000 open/close in 96.96 seconds: 103.13 ops/second - unlinked 0 (time 1746133557 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 992 (time 1746133632.65 total 10.00 last 99.20) - open/close 2032 (time 1746133642.65 total 20.00 last 103.96) - open/close 3111 (time 1746133652.66 total 30.01 last 107.86) - open/close 4067 (time 1746133662.68 total 40.02 last 95.44) - open/close 5010 (time 1746133672.68 total 50.03 last 94.25) - open/close 5986 (time 1746133682.69 total 60.04 last 97.55) - open/close 6936 (time 1746133692.69 total 70.04 last 94.93) - open/close 7737 (time 1746133702.70 total 80.05 last 80.07) - open/close 8699 (time 1746133712.70 total 90.05 last 96.13) - open/close 9667 (time 1746133722.72 total 100.07 last 96.69) total: 10000 open/close in 103.20 seconds: 96.89 ops/second - unlinked 0 (time 1746133728 ; total 0 ; last 0) total: 10000 unlinks in 53 seconds: 188.679245 unlinks/second - open/close 1197 (time 1746133793.70 total 10.01 last 119.59) - open/close 2363 (time 1746133803.70 total 20.01 last 116.56) - open/close 3495 (time 1746133813.70 total 30.01 last 113.18) - open/close 4551 (time 1746133823.71 total 40.02 last 105.50) - open/close 5559 (time 1746133833.72 total 50.03 last 100.73) - open/close 6625 (time 1746133843.72 total 60.03 last 106.58) - open/close 7851 (time 1746133853.73 total 70.04 last 122.48) - open/close 8975 (time 1746133863.74 total 80.05 last 112.33) total: 10000 open/close in 88.42 seconds: 113.10 ops/second - unlinked 0 (time 1746133874 ; total 0 ; last 0) total: 10000 unlinks in 47 seconds: 212.765961 unlinks/second - open/close 1464 (time 1746133933.97 total 10.01 last 146.24) - open/close 2679 (time 1746133943.97 total 20.02 last 121.43) - open/close 3720 (time 1746133953.98 total 30.02 last 104.05) - open/close 4907 (time 1746133963.98 total 40.02 last 118.67) - open/close 6057 (time 1746133973.98 total 50.03 last 114.93) - open/close 7178 (time 1746133984.02 total 60.06 last 111.72) - open/close 8225 (time 1746133994.02 total 70.06 last 104.70) - open/close 9484 (time 1746134004.02 total 80.07 last 125.82) total: 10000 open/close in 85.40 seconds: 117.09 ops/second - unlinked 0 (time 1746134011 ; total 0 ; last 0) total: 10000 unlinks in 42 seconds: 238.095245 unlinks/second - open/close 1242 (time 1746134066.05 total 10.00 last 124.16) - open/close 2467 (time 1746134076.05 total 20.00 last 122.48) - open/close 3637 (time 1746134086.06 total 30.01 last 116.89) - open/close 4746 (time 1746134096.07 total 40.02 last 110.80) - open/close 5809 (time 1746134106.08 total 50.03 last 106.22) - open/close 6811 (time 1746134116.08 total 60.03 last 100.16) - open/close 7982 (time 1746134126.09 total 70.04 last 117.03) - open/close 9272 (time 1746134136.09 total 80.05 last 128.90) total: 10000 open/close in 86.41 seconds: 115.73 ops/second - unlinked 0 (time 1746134144 ; total 0 ; last 0) total: 10000 unlinks in 49 seconds: 204.081635 unlinks/second - open/close 1278 (time 1746134205.59 total 10.01 last 127.73) - open/close 2509 (time 1746134215.59 total 20.01 last 123.05) - open/close 3568 (time 1746134225.60 total 30.02 last 105.78) - open/close 4653 (time 1746134235.61 total 40.03 last 108.40) - open/close 5819 (time 1746134245.61 total 50.03 last 116.57) - open/close 6926 (time 1746134255.62 total 60.03 last 110.68) - open/close 8099 (time 1746134265.62 total 70.03 last 117.30) - open/close 9107 (time 1746134275.62 total 80.04 last 100.72) total: 10000 open/close in 87.29 seconds: 114.56 ops/second - unlinked 0 (time 1746134284 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 1056 (time 1746134359.16 total 10.00 last 105.56) - open/close 2118 (time 1746134369.16 total 20.00 last 106.19) - open/close 3229 (time 1746134379.16 total 30.01 last 111.04) - open/close 4427 (time 1746134389.17 total 40.02 last 119.71) - open/close 5607 (time 1746134399.18 total 50.02 last 117.93) - open/close 6746 (time 1746134409.19 total 60.04 last 113.74) - open/close 7613 (time 1746134419.20 total 70.05 last 86.63) - open/close 8508 (time 1746134429.21 total 80.06 last 89.41) - open/close 9352 (time 1746134439.21 total 90.06 last 84.35) total: 10000 open/close in 96.93 seconds: 103.16 ops/second - unlinked 0 (time 1746134448 ; total 0 ; last 0) total: 10000 unlinks in 57 seconds: 175.438599 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg155-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg155-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg155-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.155@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.155@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg155-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg155-server: oleg155-server.virtnet: executing set_default_debug -1 all pdsh@oleg155-client: oleg155-server: ssh exited with exit code 1 Started lustre-OST0000 oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg155-server: oleg155-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg155-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec mount lustre on /mnt/lustre..... Starting client: oleg155-client.virtnet: -o user_xattr,flock 192.168.201.155@tcp:/lustre /mnt/lustre On OST0, 13229 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg155-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg155-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg155-server stop mds service on oleg155-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg155-server unloading modules via unload_modules_local on: 'oleg155-server' oleg155-server: oleg155-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg155-client: oleg155-server: ssh exited with exit code 2 pdsh@oleg155-client: oleg155-client: ssh exited with exit code 2 pdsh@oleg155-client: oleg155-client: ssh exited with exit code 2 pdsh@oleg155-client: oleg155-server: ssh exited with exit code 2