== conf-sanity test 69: replace an OST with the same index ========================================================== 23:57:53 (1746071873) start mds service on oleg144-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg144-server' oleg144-server: oleg144-server.virtnet: executing load_modules_local oleg144-server: Loading modules from /home/green/git/lustre-release/lustre oleg144-server: detected 4 online CPUs by sysfs oleg144-server: MODOPTS_LIBCFS= oleg144-server: Force libcfs to create 2 CPU partitions oleg144-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg144-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg144-server: oleg144-server.virtnet: executing set_default_debug -1 all pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg144-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg144-server: oleg144-server.virtnet: executing set_default_debug -1 all pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 Started lustre-OST0000 oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg144-client.virtnet: -o user_xattr,flock 192.168.201.144@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 750 (time 1746071937.64 total 10.00 last 74.99) - open/close 1484 (time 1746071947.64 total 20.01 last 73.34) - open/close 2190 (time 1746071957.66 total 30.02 last 70.51) - open/close 2979 (time 1746071967.66 total 40.03 last 78.86) - open/close 3791 (time 1746071977.67 total 50.04 last 81.13) - open/close 4532 (time 1746071987.68 total 60.05 last 74.03) - open/close 5323 (time 1746071997.69 total 70.05 last 79.02) - open/close 6135 (time 1746072007.70 total 80.06 last 81.14) - open/close 6909 (time 1746072017.72 total 90.09 last 77.20) - open/close 7783 (time 1746072027.73 total 100.10 last 87.33) - open/close 8541 (time 1746072037.73 total 110.10 last 75.76) - open/close 9284 (time 1746072047.75 total 120.12 last 74.19) total: 10000 open/close in 130.06 seconds: 76.89 ops/second - unlinked 0 (time 1746072060 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 883 (time 1746072144.46 total 10.00 last 88.28) - open/close 1696 (time 1746072154.47 total 20.01 last 81.22) - open/close 2439 (time 1746072164.47 total 30.02 last 74.25) - open/close 3458 (time 1746072174.48 total 40.03 last 101.81) - open/close 4548 (time 1746072184.49 total 50.03 last 108.95) - open/close 5544 (time 1746072194.49 total 60.04 last 99.55) - open/close 6270 (time 1746072204.51 total 70.05 last 72.49) - open/close 7026 (time 1746072214.51 total 80.06 last 75.54) - open/close 7861 (time 1746072224.52 total 90.07 last 83.45) - open/close 8685 (time 1746072234.52 total 100.07 last 82.37) - open/close 9508 (time 1746072244.53 total 110.08 last 82.24) total: 10000 open/close in 116.42 seconds: 85.89 ops/second - unlinked 0 (time 1746072253 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 826 (time 1746072351.23 total 10.00 last 82.56) - open/close 1765 (time 1746072361.25 total 20.02 last 93.72) - open/close 2496 (time 1746072371.26 total 30.03 last 73.07) - open/close 3340 (time 1746072381.26 total 40.03 last 84.38) - open/close 4095 (time 1746072391.26 total 50.03 last 75.47) - open/close 4868 (time 1746072401.27 total 60.04 last 77.22) - open/close 5733 (time 1746072411.29 total 70.06 last 86.37) - open/close 6512 (time 1746072421.30 total 80.07 last 77.80) - open/close 7282 (time 1746072431.31 total 90.09 last 76.91) - open/close 8157 (time 1746072441.33 total 100.10 last 87.36) - open/close 8950 (time 1746072451.33 total 110.11 last 79.27) - open/close 9725 (time 1746072461.33 total 120.11 last 77.49) total: 10000 open/close in 123.70 seconds: 80.84 ops/second - unlinked 0 (time 1746072467 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 764 (time 1746072562.40 total 10.00 last 76.37) - open/close 1690 (time 1746072572.41 total 20.01 last 92.57) - open/close 2492 (time 1746072582.41 total 30.01 last 80.15) - open/close 3236 (time 1746072592.42 total 40.02 last 74.33) - open/close 3928 (time 1746072602.42 total 50.02 last 69.19) - open/close 4629 (time 1746072612.43 total 60.03 last 70.05) - open/close 5424 (time 1746072622.44 total 70.04 last 79.44) - open/close 6264 (time 1746072632.45 total 80.05 last 83.90) - open/close 7017 (time 1746072642.45 total 90.05 last 75.29) - open/close 7809 (time 1746072652.46 total 100.06 last 79.13) - open/close 8601 (time 1746072662.46 total 110.06 last 79.17) - open/close 9366 (time 1746072672.47 total 120.07 last 76.50) total: 10000 open/close in 128.39 seconds: 77.89 ops/second - unlinked 0 (time 1746072683 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 760 (time 1746072775.50 total 10.00 last 75.97) - open/close 1541 (time 1746072785.50 total 20.01 last 78.09) - open/close 2493 (time 1746072795.51 total 30.01 last 95.14) - open/close 3381 (time 1746072805.51 total 40.01 last 88.80) - open/close 4247 (time 1746072815.51 total 50.02 last 86.57) - open/close 5024 (time 1746072825.53 total 60.03 last 77.59) - open/close 5843 (time 1746072835.53 total 70.03 last 81.87) - open/close 6583 (time 1746072845.54 total 80.04 last 73.97) - open/close 7396 (time 1746072855.54 total 90.04 last 81.25) - open/close 8111 (time 1746072865.55 total 100.05 last 71.47) - open/close 8824 (time 1746072875.57 total 110.07 last 71.16) - open/close 9497 (time 1746072885.57 total 120.08 last 67.25) total: 10000 open/close in 126.99 seconds: 78.75 ops/second - unlinked 0 (time 1746072895 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 701 (time 1746072980.51 total 10.01 last 70.02) - open/close 1423 (time 1746072990.52 total 20.01 last 72.19) - open/close 2233 (time 1746073000.53 total 30.03 last 80.90) - open/close 3055 (time 1746073010.53 total 40.03 last 82.20) - open/close 3912 (time 1746073020.53 total 50.03 last 85.66) - open/close 4689 (time 1746073030.53 total 60.03 last 77.69) - open/close 5473 (time 1746073040.55 total 70.04 last 78.30) - open/close 6234 (time 1746073050.56 total 80.05 last 76.04) - open/close 7061 (time 1746073060.56 total 90.06 last 82.63) - open/close 7821 (time 1746073070.57 total 100.07 last 75.95) - open/close 8715 (time 1746073080.57 total 110.07 last 89.40) - open/close 9741 (time 1746073090.58 total 120.08 last 102.48) total: 10000 open/close in 123.32 seconds: 81.09 ops/second - unlinked 0 (time 1746073096 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 976 (time 1746073183.44 total 10.00 last 97.57) - open/close 1923 (time 1746073193.45 total 20.01 last 94.60) - open/close 2922 (time 1746073203.46 total 30.02 last 99.85) - open/close 4057 (time 1746073213.47 total 40.03 last 113.34) - open/close 4956 (time 1746073223.47 total 50.04 last 89.88) - open/close 5786 (time 1746073233.48 total 60.04 last 82.96) - open/close 6669 (time 1746073243.49 total 70.05 last 88.22) - open/close 7701 (time 1746073253.49 total 80.06 last 103.12) - open/close 8700 (time 1746073263.50 total 90.06 last 99.81) - open/close 9792 (time 1746073273.51 total 100.07 last 109.14) total: 10000 open/close in 102.42 seconds: 97.64 ops/second - unlinked 0 (time 1746073278 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 939 (time 1746073353.52 total 10.00 last 93.87) - open/close 1838 (time 1746073363.53 total 20.02 last 89.77) - open/close 2761 (time 1746073373.54 total 30.02 last 92.25) - open/close 3579 (time 1746073383.55 total 40.03 last 81.72) - open/close 4433 (time 1746073393.55 total 50.04 last 85.37) - open/close 5248 (time 1746073403.56 total 60.05 last 81.43) - open/close 6072 (time 1746073413.57 total 70.05 last 82.36) - open/close 6751 (time 1746073423.57 total 80.06 last 67.85) - open/close 7480 (time 1746073433.58 total 90.07 last 72.84) - open/close 8242 (time 1746073443.58 total 100.07 last 76.20) - open/close 9024 (time 1746073453.59 total 110.07 last 78.17) - open/close 9916 (time 1746073463.60 total 120.08 last 89.10) total: 10000 open/close in 121.00 seconds: 82.65 ops/second - unlinked 0 (time 1746073466 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 854 (time 1746073547.21 total 10.00 last 85.38) - open/close 1668 (time 1746073557.21 total 20.01 last 81.35) - open/close 2438 (time 1746073567.21 total 30.01 last 76.99) - open/close 3205 (time 1746073577.22 total 40.01 last 76.67) - open/close 4067 (time 1746073587.22 total 50.02 last 86.17) - open/close 4844 (time 1746073597.23 total 60.03 last 77.64) - open/close 5692 (time 1746073607.25 total 70.04 last 84.64) - open/close 6572 (time 1746073617.26 total 80.06 last 87.87) - open/close 7392 (time 1746073627.27 total 90.07 last 81.93) - open/close 8319 (time 1746073637.27 total 100.07 last 92.68) - open/close 9104 (time 1746073647.28 total 110.08 last 78.42) - open/close 9913 (time 1746073657.29 total 120.09 last 80.86) total: 10000 open/close in 121.33 seconds: 82.42 ops/second - unlinked 0 (time 1746073661 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 971 (time 1746073748.32 total 10.01 last 96.98) - open/close 1911 (time 1746073758.33 total 20.02 last 93.90) - open/close 2708 (time 1746073768.34 total 30.03 last 79.67) - open/close 3431 (time 1746073778.34 total 40.03 last 72.28) - open/close 4168 (time 1746073788.35 total 50.04 last 73.66) - open/close 5040 (time 1746073798.35 total 60.04 last 87.18) - open/close 5890 (time 1746073808.36 total 70.05 last 84.92) - open/close 6636 (time 1746073818.37 total 80.06 last 74.53) - open/close 7586 (time 1746073828.38 total 90.07 last 94.86) - open/close 8336 (time 1746073838.38 total 100.07 last 74.99) - open/close 9077 (time 1746073848.41 total 110.10 last 73.94) - open/close 9857 (time 1746073858.42 total 120.11 last 77.92) total: 10000 open/close in 121.54 seconds: 82.28 ops/second - unlinked 0 (time 1746073862 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg144-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg144-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg144-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.144@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.144@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg144-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg144-server: oleg144-server.virtnet: executing set_default_debug -1 all pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 Started lustre-OST0000 oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg144-server: oleg144-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg144-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec mount lustre on /mnt/lustre..... Starting client: oleg144-client.virtnet: -o user_xattr,flock 192.168.201.144@tcp:/lustre /mnt/lustre On OST0, 3461 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg144-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg144-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg144-server stop mds service on oleg144-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg144-server unloading modules via unload_modules_local on: 'oleg144-server' oleg144-server: oleg144-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg144-client: oleg144-server: ssh exited with exit code 2 pdsh@oleg144-client: oleg144-client: ssh exited with exit code 2 pdsh@oleg144-client: oleg144-client: ssh exited with exit code 2 pdsh@oleg144-client: oleg144-server: ssh exited with exit code 2