== conf-sanity test 69: replace an OST with the same index ========================================================== 17:27:22 (1746134842) start mds service on oleg246-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg246-server' oleg246-server: oleg246-server.virtnet: executing load_modules_local oleg246-server: Loading modules from /home/green/git/lustre-release/lustre oleg246-server: detected 4 online CPUs by sysfs oleg246-server: MODOPTS_LIBCFS= oleg246-server: Force libcfs to create 2 CPU partitions oleg246-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg246-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg246-server: oleg246-server.virtnet: executing set_default_debug -1 all pdsh@oleg246-client: oleg246-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg246-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg246-server: oleg246-server.virtnet: executing set_default_debug -1 all pdsh@oleg246-client: oleg246-server: ssh exited with exit code 1 Started lustre-OST0000 oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg246-client.virtnet: -o user_xattr,flock 192.168.202.146@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 892 (time 1746134889.06 total 10.00 last 89.16) - open/close 1866 (time 1746134899.07 total 20.00 last 97.40) - open/close 2924 (time 1746134909.07 total 30.01 last 105.71) - open/close 3861 (time 1746134919.08 total 40.02 last 93.68) - open/close 4616 (time 1746134929.12 total 50.06 last 75.20) - open/close 5663 (time 1746134939.13 total 60.07 last 104.57) - open/close 6650 (time 1746134949.14 total 70.08 last 98.62) - open/close 7477 (time 1746134959.14 total 80.08 last 82.69) - open/close 8390 (time 1746134969.14 total 90.08 last 91.29) - open/close 9333 (time 1746134979.14 total 100.08 last 94.30) total: 10000 open/close in 106.40 seconds: 93.99 ops/second - unlinked 0 (time 1746134987 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 1006 (time 1746135061.75 total 10.01 last 100.55) - open/close 1933 (time 1746135071.75 total 20.01 last 92.69) - open/close 2743 (time 1746135081.76 total 30.01 last 80.99) - open/close 3620 (time 1746135091.76 total 40.01 last 87.69) - open/close 4668 (time 1746135101.76 total 50.01 last 104.77) - open/close 5619 (time 1746135111.78 total 60.04 last 94.87) - open/close 6399 (time 1746135121.79 total 70.04 last 77.97) - open/close 7209 (time 1746135131.80 total 80.05 last 80.91) - open/close 8010 (time 1746135141.80 total 90.05 last 80.08) - open/close 9074 (time 1746135151.80 total 100.05 last 106.39) total: 10000 open/close in 109.90 seconds: 90.99 ops/second - unlinked 0 (time 1746135163 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 1015 (time 1746135238.78 total 10.01 last 101.38) - open/close 1895 (time 1746135248.79 total 20.02 last 87.92) - open/close 2729 (time 1746135258.80 total 30.03 last 83.36) - open/close 3611 (time 1746135268.81 total 40.03 last 88.14) - open/close 4637 (time 1746135278.81 total 50.04 last 102.54) - open/close 5406 (time 1746135288.81 total 60.04 last 76.88) - open/close 6194 (time 1746135298.82 total 70.04 last 78.79) - open/close 6947 (time 1746135308.82 total 80.04 last 75.29) - open/close 7871 (time 1746135318.82 total 90.05 last 92.33) - open/close 8715 (time 1746135328.83 total 100.06 last 84.36) - open/close 9480 (time 1746135338.84 total 110.07 last 76.42) total: 10000 open/close in 118.02 seconds: 84.73 ops/second - unlinked 0 (time 1746135349 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 1025 (time 1746135435.46 total 10.00 last 102.46) - open/close 1941 (time 1746135445.47 total 20.02 last 91.47) - open/close 2796 (time 1746135455.48 total 30.02 last 85.49) - open/close 3610 (time 1746135465.48 total 40.02 last 81.39) - open/close 4600 (time 1746135475.48 total 50.02 last 98.99) - open/close 5835 (time 1746135485.48 total 60.03 last 123.46) - open/close 6658 (time 1746135495.50 total 70.05 last 82.12) - open/close 7520 (time 1746135505.51 total 80.06 last 86.12) - open/close 8461 (time 1746135515.51 total 90.06 last 94.10) - open/close 9692 (time 1746135525.51 total 100.06 last 123.08) total: 10000 open/close in 103.31 seconds: 96.80 ops/second - unlinked 0 (time 1746135531 ; total 0 ; last 0) total: 10000 unlinks in 55 seconds: 181.818176 unlinks/second - open/close 1249 (time 1746135598.84 total 10.00 last 124.85) - open/close 2275 (time 1746135608.84 total 20.01 last 102.58) - open/close 3337 (time 1746135618.85 total 30.01 last 106.14) - open/close 4447 (time 1746135628.85 total 40.01 last 110.98) - open/close 5559 (time 1746135638.86 total 50.02 last 111.07) - open/close 6600 (time 1746135648.86 total 60.03 last 104.09) - open/close 7509 (time 1746135658.86 total 70.03 last 90.87) - open/close 8587 (time 1746135668.87 total 80.03 last 107.78) - open/close 9630 (time 1746135678.87 total 90.04 last 104.21) total: 10000 open/close in 94.19 seconds: 106.16 ops/second - unlinked 0 (time 1746135684 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 996 (time 1746135762.28 total 10.01 last 99.55) - open/close 2104 (time 1746135772.28 total 20.01 last 110.79) - open/close 2951 (time 1746135782.28 total 30.01 last 84.69) - open/close 3796 (time 1746135792.29 total 40.01 last 84.47) - open/close 4829 (time 1746135802.29 total 50.01 last 103.29) - open/close 5954 (time 1746135812.29 total 60.02 last 112.45) - open/close 6955 (time 1746135822.31 total 70.03 last 99.97) - open/close 7908 (time 1746135832.31 total 80.04 last 95.23) - open/close 9061 (time 1746135842.32 total 90.04 last 115.26) total: 10000 open/close in 98.61 seconds: 101.41 ops/second - unlinked 0 (time 1746135852 ; total 0 ; last 0) total: 10000 unlinks in 48 seconds: 208.333328 unlinks/second - open/close 1414 (time 1746135912.22 total 10.00 last 141.34) - open/close 2660 (time 1746135922.23 total 20.01 last 124.55) - open/close 3898 (time 1746135932.23 total 30.01 last 123.79) - open/close 5044 (time 1746135942.23 total 40.01 last 114.57) - open/close 6361 (time 1746135952.24 total 50.02 last 131.59) - open/close 7421 (time 1746135962.24 total 60.02 last 105.99) - open/close 8719 (time 1746135972.24 total 70.02 last 129.77) total: 10000 open/close in 78.83 seconds: 126.86 ops/second - unlinked 0 (time 1746135982 ; total 0 ; last 0) total: 10000 unlinks in 41 seconds: 243.902435 unlinks/second - open/close 1131 (time 1746136035.05 total 10.00 last 113.06) - open/close 2533 (time 1746136045.06 total 20.01 last 140.09) - open/close 3954 (time 1746136055.06 total 30.01 last 142.08) - open/close 5215 (time 1746136065.06 total 40.02 last 126.03) - open/close 6600 (time 1746136075.07 total 50.02 last 138.42) - open/close 7798 (time 1746136085.07 total 60.03 last 119.77) - open/close 9482 (time 1746136095.07 total 70.03 last 168.37) total: 10000 open/close in 73.11 seconds: 136.78 ops/second - unlinked 0 (time 1746136099 ; total 0 ; last 0) total: 10000 unlinks in 41 seconds: 243.902435 unlinks/second - open/close 1615 (time 1746136151.70 total 10.00 last 161.47) - open/close 3570 (time 1746136161.71 total 20.01 last 195.40) - open/close 5335 (time 1746136171.71 total 30.01 last 176.37) - open/close 6782 (time 1746136181.72 total 40.02 last 144.55) - open/close 8068 (time 1746136191.73 total 50.03 last 128.56) - open/close 9575 (time 1746136201.73 total 60.03 last 150.64) total: 10000 open/close in 62.47 seconds: 160.08 ops/second - unlinked 0 (time 1746136205 ; total 0 ; last 0) total: 10000 unlinks in 48 seconds: 208.333328 unlinks/second - open/close 1480 (time 1746136265.62 total 10.00 last 147.93) - open/close 2890 (time 1746136275.62 total 20.01 last 141.00) - open/close 4520 (time 1746136285.62 total 30.01 last 162.99) - open/close 6138 (time 1746136295.63 total 40.01 last 161.76) - open/close 7712 (time 1746136305.63 total 50.01 last 157.32) - open/close 9092 (time 1746136315.63 total 60.02 last 137.97) total: 10000 open/close in 66.92 seconds: 149.43 ops/second - unlinked 0 (time 1746136324 ; total 0 ; last 0) total: 10000 unlinks in 40 seconds: 250.000000 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg246-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg246-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg246-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.146@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.146@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg246-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg246-server: oleg246-server.virtnet: executing set_default_debug -1 all pdsh@oleg246-client: oleg246-server: ssh exited with exit code 1 Started lustre-OST0000 oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg246-server: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg246-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 6 sec mount lustre on /mnt/lustre..... Starting client: oleg246-client.virtnet: -o user_xattr,flock 192.168.202.146@tcp:/lustre /mnt/lustre On OST0, 16141 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg246-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg246-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg246-server stop mds service on oleg246-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg246-server unloading modules via unload_modules_local on: 'oleg246-server' oleg246-server: oleg246-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg246-client: oleg246-client: ssh exited with exit code 2 pdsh@oleg246-client: oleg246-server: ssh exited with exit code 2 pdsh@oleg246-client: oleg246-client: ssh exited with exit code 2 pdsh@oleg246-client: oleg246-server: ssh exited with exit code 2