== conf-sanity test 69: replace an OST with the same index ========================================================== 15:57:43 (1710100663) start mds service on oleg144-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg144-server: oleg144-server.virtnet: executing set_default_debug -1 all pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg144-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg144-server: oleg144-server.virtnet: executing set_default_debug -1 all pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 Started lustre-OST0000 oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg144-client.virtnet: -o user_xattr,flock oleg144-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99936. rc=0 - open/close 2601 (time 1710100690.00 total 10.00 last 260.01) - open/close 4314 (time 1710100700.01 total 20.01 last 171.27) - open/close 6174 (time 1710100710.01 total 30.01 last 185.97) - open/close 7898 (time 1710100720.01 total 40.01 last 172.30) - open/close 9638 (time 1710100730.02 total 50.02 last 173.90) total: 10000 open/close in 52.48 seconds: 190.54 ops/second - unlinked 0 (time 1710100733 ; total 0 ; last 0) total: 10000 unlinks in 47 seconds: 212.765961 unlinks/second - open/close 2825 (time 1710100791.62 total 10.00 last 282.47) - open/close 4937 (time 1710100801.62 total 20.01 last 211.10) - open/close 6219 (time 1710100811.62 total 30.01 last 128.15) - open/close 7155 (time 1710100821.64 total 40.03 last 93.44) - open/close 7926 (time 1710100831.66 total 50.05 last 76.96) - open/close 8691 (time 1710100841.67 total 60.05 last 76.45) - open/close 9403 (time 1710100851.68 total 70.06 last 71.13) total: 10000 open/close in 78.48 seconds: 127.41 ops/second - unlinked 0 (time 1710100861 ; total 0 ; last 0) total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second - open/close 1133 (time 1710100961.60 total 10.00 last 113.30) - open/close 2113 (time 1710100971.61 total 20.00 last 97.96) - open/close 2973 (time 1710100981.61 total 30.01 last 85.99) - open/close 3708 (time 1710100991.62 total 40.02 last 73.44) - open/close 4453 (time 1710101001.62 total 50.02 last 74.44) - open/close 5194 (time 1710101011.63 total 60.03 last 74.04) - open/close 5959 (time 1710101021.64 total 70.03 last 76.48) - open/close 6583 (time 1710101031.64 total 80.04 last 62.35) - open/close 7291 (time 1710101041.65 total 90.05 last 70.76) - open/close 7881 (time 1710101051.65 total 100.05 last 58.97) - open/close 8543 (time 1710101061.68 total 110.08 last 66.05) - open/close 9090 (time 1710101081.29 total 129.69 last 27.89) - open/close 9663 (time 1710101091.30 total 139.70 last 57.24) total: 10000 open/close in 144.91 seconds: 69.01 ops/second - unlinked 0 (time 1710101098 ; total 0 ; last 0) total: 10000 unlinks in 101 seconds: 99.009903 unlinks/second - open/close 812 (time 1710101223.54 total 20.83 last 38.98) - open/close 1543 (time 1710101233.54 total 30.83 last 73.08) - open/close 2290 (time 1710101243.55 total 40.84 last 74.65) - open/close 2953 (time 1710101253.56 total 50.85 last 66.25) - open/close 3582 (time 1710101263.56 total 60.85 last 62.86) - open/close 4136 (time 1710101273.57 total 70.86 last 55.38) - open/close 4742 (time 1710101283.57 total 80.86 last 60.59) - open/close 5415 (time 1710101293.58 total 90.87 last 67.21) - open/close 5980 (time 1710101303.64 total 100.93 last 56.15) - open/close 6483 (time 1710101313.66 total 110.95 last 50.23) - open/close 6958 (time 1710101323.67 total 120.95 last 47.46) - open/close 7004 (time 1710101335.50 total 132.79 last 3.89) - open/close 7465 (time 1710101345.52 total 142.81 last 45.98) - open/close 7892 (time 1710101355.53 total 152.82 last 42.68) - open/close 8343 (time 1710101365.53 total 162.82 last 45.09) - open/close 8774 (time 1710101375.54 total 172.82 last 43.08) - open/close 9251 (time 1710101385.54 total 182.83 last 47.66) - open/close 9658 (time 1710101395.57 total 192.86 last 40.59) total: 10000 open/close in 201.30 seconds: 49.68 ops/second - unlinked 0 (time 1710101406 ; total 0 ; last 0) total: 10000 unlinks in 115 seconds: 86.956520 unlinks/second - open/close 814 (time 1710101533.93 total 10.01 last 81.32) - open/close 1536 (time 1710101543.94 total 20.02 last 72.16) - open/close 2287 (time 1710101553.95 total 30.02 last 75.05) - open/close 2535 (time 1710101568.82 total 44.90 last 16.67) - open/close 3230 (time 1710101578.83 total 54.90 last 69.48) - open/close 3878 (time 1710101588.84 total 64.92 last 64.71) - open/close 4522 (time 1710101598.85 total 74.92 last 64.35) - open/close 5126 (time 1710101608.86 total 84.93 last 60.33) - open/close 5696 (time 1710101618.87 total 94.95 last 56.91) - open/close 5789 (time 1710101631.90 total 107.97 last 7.14) - open/close 6306 (time 1710101641.91 total 117.98 last 51.67) - open/close 6781 (time 1710101651.91 total 127.98 last 47.50) - open/close 7297 (time 1710101661.91 total 137.98 last 51.59) - open/close 7785 (time 1710101671.91 total 147.99 last 48.78) - open/close 8256 (time 1710101681.92 total 158.00 last 47.06) - open/close 8769 (time 1710101691.94 total 168.01 last 51.22) - open/close 9202 (time 1710101701.95 total 178.03 last 43.22) - open/close 9669 (time 1710101711.97 total 188.05 last 46.61) total: 10000 open/close in 196.19 seconds: 50.97 ops/second - unlinked 0 (time 1710101723 ; total 0 ; last 0) total: 10000 unlinks in 114 seconds: 87.719299 unlinks/second - open/close 901 (time 1710101850.07 total 10.01 last 90.05) - open/close 1670 (time 1710101860.08 total 20.02 last 76.80) - open/close 2437 (time 1710101870.09 total 30.02 last 76.67) - open/close 3101 (time 1710101880.10 total 40.03 last 66.36) - open/close 3788 (time 1710101890.10 total 50.04 last 68.64) - open/close 4477 (time 1710101900.11 total 60.04 last 68.85) - open/close 5113 (time 1710101910.11 total 70.05 last 63.59) - open/close 5732 (time 1710101920.13 total 80.07 last 61.76) - open/close 6308 (time 1710101930.14 total 90.07 last 57.57) - open/close 6854 (time 1710101940.15 total 100.08 last 54.55) - open/close 7374 (time 1710101950.15 total 110.08 last 51.99) - open/close 7915 (time 1710101960.16 total 120.09 last 54.04) - open/close 8384 (time 1710101970.17 total 130.11 last 46.84) - open/close 8849 (time 1710101980.20 total 140.14 last 46.36) - open/close 9350 (time 1710101990.22 total 150.15 last 50.04) - open/close 9816 (time 1710102000.23 total 160.16 last 46.54) total: 10000 open/close in 164.47 seconds: 60.80 ops/second - unlinked 0 (time 1710102007 ; total 0 ; last 0) total: 10000 unlinks in 111 seconds: 90.090088 unlinks/second - open/close 845 (time 1710102131.12 total 10.01 last 84.45) - open/close 1583 (time 1710102141.13 total 20.02 last 73.70) - open/close 2362 (time 1710102151.13 total 30.02 last 77.88) - open/close 3028 (time 1710102161.14 total 40.02 last 66.59) - open/close 3428 (time 1710102177.80 total 56.69 last 24.00) - open/close 4042 (time 1710102187.80 total 66.69 last 61.36) - open/close 4732 (time 1710102197.81 total 76.70 last 68.96) - open/close 5284 (time 1710102207.82 total 86.70 last 55.17) - open/close 5823 (time 1710102217.83 total 96.71 last 53.84) - open/close 6364 (time 1710102227.86 total 106.75 last 53.92) - open/close 6912 (time 1710102237.87 total 116.75 last 54.76) - open/close 7422 (time 1710102247.87 total 126.76 last 50.99) - open/close 7927 (time 1710102257.87 total 136.76 last 50.48) - open/close 8379 (time 1710102267.87 total 146.76 last 45.19) - open/close 8881 (time 1710102277.88 total 156.76 last 50.19) - open/close 8896 (time 1710102289.20 total 168.09 last 1.32) - open/close 9412 (time 1710102299.21 total 178.09 last 51.56) - open/close 9919 (time 1710102309.22 total 188.11 last 50.64) total: 10000 open/close in 189.66 seconds: 52.72 ops/second - unlinked 0 (time 1710102313 ; total 0 ; last 0) total: 10000 unlinks in 115 seconds: 86.956520 unlinks/second - open/close 885 (time 1710102441.04 total 10.00 last 88.49) - open/close 1610 (time 1710102451.04 total 20.00 last 72.47) - open/close 2343 (time 1710102461.05 total 30.01 last 73.26) - open/close 3034 (time 1710102471.05 total 40.02 last 69.06) - open/close 3639 (time 1710102481.12 total 50.08 last 60.08) - open/close 4353 (time 1710102491.13 total 60.09 last 71.38) - open/close 4917 (time 1710102501.13 total 70.09 last 56.38) - open/close 5440 (time 1710102511.15 total 80.11 last 52.20) - open/close 5980 (time 1710102521.15 total 90.12 last 53.97) - open/close 6526 (time 1710102531.16 total 100.12 last 54.55) - open/close 7051 (time 1710102541.17 total 110.13 last 52.46) - open/close 7513 (time 1710102551.18 total 120.14 last 46.15) - open/close 8005 (time 1710102561.19 total 130.15 last 49.17) - open/close 8462 (time 1710102571.38 total 140.34 last 44.84) - open/close 8992 (time 1710102581.38 total 150.34 last 52.98) - open/close 9425 (time 1710102591.40 total 160.36 last 43.22) - open/close 9857 (time 1710102601.41 total 170.37 last 43.18) total: 10000 open/close in 174.07 seconds: 57.45 ops/second - unlinked 0 (time 1710102607 ; total 0 ; last 0) total: 10000 unlinks in 127 seconds: 78.740158 unlinks/second - open/close 838 (time 1710102747.34 total 10.00 last 83.77) - open/close 1669 (time 1710102757.35 total 20.01 last 83.01) - open/close 2415 (time 1710102767.35 total 30.02 last 74.57) - open/close 3096 (time 1710102777.35 total 40.02 last 68.09) - open/close 3773 (time 1710102787.36 total 50.02 last 67.68) - open/close 4402 (time 1710102797.37 total 60.03 last 62.83) - open/close 4967 (time 1710102807.38 total 70.05 last 56.41) - open/close 5544 (time 1710102817.39 total 80.05 last 57.69) - open/close 6117 (time 1710102827.42 total 90.08 last 57.12) - open/close 6663 (time 1710102837.43 total 100.09 last 54.55) - open/close 7156 (time 1710102847.44 total 110.10 last 49.23) - open/close 7621 (time 1710102857.44 total 120.11 last 46.48) - open/close 8162 (time 1710102867.45 total 130.11 last 54.09) - open/close 8670 (time 1710102877.45 total 140.11 last 50.79) - open/close 9086 (time 1710102897.31 total 159.97 last 20.95) - open/close 9488 (time 1710102907.37 total 170.04 last 39.93) - open/close 9952 (time 1710102917.38 total 180.05 last 46.36) total: 10000 open/close in 180.94 seconds: 55.27 ops/second - unlinked 0 (time 1710102921 ; total 1 ; last 1) total: 10000 unlinks in 113 seconds: 88.495575 unlinks/second - open/close 793 (time 1710103046.46 total 10.02 last 79.16) - open/close 1567 (time 1710103056.46 total 20.02 last 77.40) - open/close 2307 (time 1710103066.47 total 30.03 last 73.92) - open/close 3035 (time 1710103076.49 total 40.04 last 72.69) - open/close 3725 (time 1710103086.49 total 50.05 last 68.98) - open/close 4402 (time 1710103096.50 total 60.06 last 67.58) - open/close 5047 (time 1710103106.52 total 70.08 last 64.40) - open/close 5646 (time 1710103116.53 total 80.09 last 59.83) - open/close 6241 (time 1710103126.55 total 90.11 last 59.38) - open/close 6760 (time 1710103136.56 total 100.12 last 51.85) - open/close 7297 (time 1710103146.58 total 110.14 last 53.57) - open/close 7851 (time 1710103156.59 total 120.15 last 55.34) - open/close 8325 (time 1710103166.60 total 130.16 last 47.35) - open/close 8839 (time 1710103176.62 total 140.18 last 51.32) - open/close 9398 (time 1710103186.62 total 150.18 last 55.89) - open/close 9919 (time 1710103196.63 total 160.18 last 52.08) total: 10000 open/close in 161.74 seconds: 61.83 ops/second - unlinked 0 (time 1710103200 ; total 0 ; last 0) total: 10000 unlinks in 110 seconds: 90.909088 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg144-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg144-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg144-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.144@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.144@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg144-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg144-server: oleg144-server.virtnet: executing set_default_debug -1 all pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 Started lustre-OST0000 oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg144-server: oleg144-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg144-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec mount lustre on /mnt/lustre..... Starting client: oleg144-client.virtnet: -o user_xattr,flock oleg144-server@tcp:/lustre /mnt/lustre On OST0, 20325 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg144-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg144-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg144-server stop mds service on oleg144-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg144-server unloading modules on: 'oleg144-server' oleg144-server: oleg144-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg144-client: oleg144-client: ssh exited with exit code 2 pdsh@oleg144-client: oleg144-server: ssh exited with exit code 2 pdsh@oleg144-client: oleg144-client: ssh exited with exit code 2 pdsh@oleg144-client: oleg144-server: ssh exited with exit code 2