== conf-sanity test 69: replace an OST with the same index ========================================================== 16:00:02 (1710100802) start mds service on oleg331-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg331-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-OST0000 oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg331-client.virtnet: -o user_xattr,flock oleg331-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99936. rc=0 - open/close 1777 (time 1710100828.55 total 10.00 last 177.61) - open/close 3314 (time 1710100838.56 total 20.02 last 153.53) - open/close 4799 (time 1710100848.57 total 30.02 last 148.42) - open/close 5906 (time 1710100858.57 total 40.02 last 110.69) - open/close 6938 (time 1710100868.57 total 50.03 last 103.16) - open/close 8061 (time 1710100878.58 total 60.03 last 112.26) - open/close 9107 (time 1710100888.58 total 70.04 last 104.51) total: 10000 open/close in 78.81 seconds: 126.88 ops/second - unlinked 0 (time 1710100898 ; total 0 ; last 0) total: 10000 unlinks in 51 seconds: 196.078430 unlinks/second - open/close 1654 (time 1710100960.33 total 10.00 last 165.39) - open/close 3032 (time 1710100970.34 total 20.01 last 137.69) - open/close 3998 (time 1710100980.35 total 30.02 last 96.49) - open/close 4857 (time 1710100990.37 total 40.04 last 85.70) - open/close 5758 (time 1710101000.37 total 50.04 last 90.09) - open/close 6725 (time 1710101010.38 total 60.06 last 96.59) - open/close 7545 (time 1710101020.38 total 70.06 last 81.99) - open/close 8317 (time 1710101030.39 total 80.06 last 77.14) - open/close 9114 (time 1710101040.39 total 90.06 last 79.70) - open/close 9859 (time 1710101050.39 total 100.07 last 74.48) total: 10000 open/close in 102.01 seconds: 98.03 ops/second - unlinked 0 (time 1710101054 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 1068 (time 1710101144.93 total 10.01 last 106.74) - open/close 2118 (time 1710101154.94 total 20.01 last 104.98) - open/close 3137 (time 1710101164.94 total 30.01 last 101.88) - open/close 4062 (time 1710101174.94 total 40.01 last 92.46) - open/close 4880 (time 1710101184.94 total 50.01 last 81.80) - open/close 5728 (time 1710101194.95 total 60.02 last 84.74) - open/close 6479 (time 1710101204.95 total 70.03 last 75.06) - open/close 7238 (time 1710101214.96 total 80.03 last 75.84) - open/close 7989 (time 1710101224.97 total 90.04 last 75.04) - open/close 8695 (time 1710101234.99 total 100.06 last 70.49) - open/close 9396 (time 1710101245.00 total 110.07 last 70.00) total: 10000 open/close in 118.64 seconds: 84.29 ops/second - unlinked 0 (time 1710101255 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 1189 (time 1710101344.29 total 10.00 last 118.90) - open/close 2312 (time 1710101354.30 total 20.01 last 112.21) - open/close 3301 (time 1710101364.31 total 30.02 last 98.80) - open/close 4294 (time 1710101374.31 total 40.03 last 99.22) - open/close 5194 (time 1710101384.32 total 50.03 last 89.96) - open/close 6059 (time 1710101394.32 total 60.03 last 86.49) - open/close 6868 (time 1710101404.32 total 70.03 last 80.90) - open/close 7633 (time 1710101414.33 total 80.04 last 76.46) - open/close 8381 (time 1710101424.34 total 90.05 last 74.72) - open/close 9083 (time 1710101434.35 total 100.06 last 70.13) - open/close 9765 (time 1710101444.35 total 110.06 last 68.16) total: 10000 open/close in 113.61 seconds: 88.02 ops/second - unlinked 0 (time 1710101449 ; total 0 ; last 0) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second - open/close 1167 (time 1710101551.90 total 10.01 last 116.62) - open/close 2283 (time 1710101561.91 total 20.01 last 111.57) - open/close 3335 (time 1710101571.91 total 30.01 last 105.15) - open/close 4318 (time 1710101581.91 total 40.02 last 98.29) - open/close 5225 (time 1710101591.92 total 50.03 last 90.62) - open/close 6072 (time 1710101601.93 total 60.04 last 84.60) - open/close 6877 (time 1710101611.94 total 70.04 last 80.45) - open/close 7642 (time 1710101621.94 total 80.05 last 76.47) - open/close 8417 (time 1710101631.95 total 90.05 last 77.48) - open/close 9151 (time 1710101641.95 total 100.05 last 73.39) - open/close 9830 (time 1710101651.96 total 110.06 last 67.83) total: 10000 open/close in 112.37 seconds: 88.99 ops/second - unlinked 0 (time 1710101655 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 1165 (time 1710101745.06 total 10.00 last 116.48) - open/close 2324 (time 1710101755.06 total 20.00 last 115.87) - open/close 3405 (time 1710101765.07 total 30.01 last 108.06) - open/close 4373 (time 1710101775.07 total 40.01 last 96.75) - open/close 5292 (time 1710101785.08 total 50.02 last 91.86) - open/close 6166 (time 1710101795.08 total 60.03 last 87.33) - open/close 7032 (time 1710101805.10 total 70.04 last 86.51) - open/close 7850 (time 1710101815.10 total 80.04 last 81.75) - open/close 8575 (time 1710101825.10 total 90.04 last 72.49) - open/close 9291 (time 1710101835.11 total 100.05 last 71.52) - open/close 9985 (time 1710101845.12 total 110.06 last 69.39) total: 10000 open/close in 110.23 seconds: 90.72 ops/second - unlinked 0 (time 1710101846 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 1139 (time 1710101938.95 total 10.01 last 113.76) - open/close 2263 (time 1710101948.95 total 20.01 last 112.38) - open/close 3329 (time 1710101958.96 total 30.02 last 106.54) - open/close 4282 (time 1710101968.97 total 40.03 last 95.22) - open/close 5228 (time 1710101978.98 total 50.04 last 94.50) - open/close 6100 (time 1710101988.99 total 60.05 last 87.14) - open/close 6967 (time 1710101998.99 total 70.05 last 86.69) - open/close 7762 (time 1710102009.00 total 80.06 last 79.39) - open/close 8554 (time 1710102019.00 total 90.06 last 79.19) - open/close 9278 (time 1710102029.01 total 100.07 last 72.35) - open/close 9964 (time 1710102039.02 total 110.08 last 68.54) total: 10000 open/close in 110.53 seconds: 90.47 ops/second - unlinked 0 (time 1710102041 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 1264 (time 1710102128.39 total 10.00 last 126.38) - open/close 2366 (time 1710102138.41 total 20.02 last 110.03) - open/close 3444 (time 1710102148.41 total 30.02 last 107.78) - open/close 4470 (time 1710102158.41 total 40.02 last 102.57) - open/close 5460 (time 1710102168.42 total 50.03 last 98.97) - open/close 6404 (time 1710102178.42 total 60.03 last 94.39) - open/close 7337 (time 1710102188.43 total 70.04 last 93.20) - open/close 8247 (time 1710102198.44 total 80.05 last 90.92) - open/close 9083 (time 1710102208.44 total 90.05 last 83.59) - open/close 9887 (time 1710102218.44 total 100.05 last 80.35) total: 10000 open/close in 101.48 seconds: 98.54 ops/second - unlinked 0 (time 1710102221 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 1334 (time 1710102302.87 total 10.01 last 133.31) - open/close 2663 (time 1710102312.88 total 20.01 last 132.83) - open/close 3873 (time 1710102322.88 total 30.02 last 120.90) - open/close 5050 (time 1710102332.89 total 40.03 last 117.62) - open/close 6158 (time 1710102342.90 total 50.04 last 110.69) - open/close 7170 (time 1710102352.91 total 60.05 last 101.07) - open/close 8128 (time 1710102362.92 total 70.05 last 95.76) - open/close 9064 (time 1710102372.92 total 80.05 last 93.60) - open/close 9954 (time 1710102382.92 total 90.05 last 89.00) total: 10000 open/close in 90.56 seconds: 110.42 ops/second - unlinked 0 (time 1710102384 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 1469 (time 1710102459.90 total 10.00 last 146.85) - open/close 2851 (time 1710102469.91 total 20.01 last 138.14) - open/close 4183 (time 1710102479.91 total 30.01 last 133.19) - open/close 5475 (time 1710102489.91 total 40.01 last 129.11) - open/close 6741 (time 1710102499.92 total 50.02 last 126.56) - open/close 7934 (time 1710102509.92 total 60.02 last 119.29) - open/close 8990 (time 1710102519.92 total 70.02 last 105.54) - open/close 9976 (time 1710102529.93 total 80.03 last 98.51) total: 10000 open/close in 80.31 seconds: 124.51 ops/second - unlinked 0 (time 1710102531 ; total 0 ; last 0) total: 10000 unlinks in 44 seconds: 227.272720 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg331-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg331-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg331-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.131@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.131@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg331-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-OST0000 oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg331-server: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg331-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec mount lustre on /mnt/lustre..... Starting client: oleg331-client.virtnet: -o user_xattr,flock oleg331-server@tcp:/lustre /mnt/lustre On OST0, 17717 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg331-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg331-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg331-server stop mds service on oleg331-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg331-server unloading modules on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg331-client: oleg331-client: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-client: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2