== conf-sanity test 69: replace an OST with the same index ========================================================== 03:42:04 (1785397324)
start mds service on oleg328-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg328-server'
oleg328-server: oleg328-server.virtnet: executing load_modules_local
oleg328-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg328-server: Loading modules from /home/green/git/lustre-release/lustre
oleg328-server: detected 4 online CPUs by sysfs
oleg328-server: MODOPTS_LIBCFS=
oleg328-server: Force libcfs to create 2 CPU partitions
oleg328-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg328-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg328-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all
pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg328-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all
pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1
Started lustre-OST0000
oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg328-client.virtnet:  -o user_xattr,flock 192.168.203.128@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 473 (time 1785397440.09 total 10.00 last 47.28)
 - open/close 897 (time 1785397450.09 total 20.01 last 42.38)
 - open/close 1356 (time 1785397460.10 total 30.01 last 45.88)
 - open/close 1877 (time 1785397470.10 total 40.02 last 52.08)
 - open/close 2310 (time 1785397480.10 total 50.02 last 43.29)
 - open/close 2751 (time 1785397490.11 total 60.03 last 44.05)
 - open/close 3426 (time 1785397500.13 total 70.04 last 67.39)
 - open/close 4027 (time 1785397510.14 total 80.06 last 60.02)
 - open/close 4511 (time 1785397520.17 total 90.09 last 48.24)
 - open/close 5106 (time 1785397530.22 total 100.13 last 59.24)
 - open/close 5580 (time 1785397540.23 total 110.14 last 47.35)
 - open/close 5934 (time 1785397550.24 total 120.16 last 35.34)
 - open/close 6409 (time 1785397560.25 total 130.16 last 47.47)
 - open/close 6874 (time 1785397570.25 total 140.16 last 46.50)
 - open/close 7263 (time 1785397580.25 total 150.17 last 38.89)
 - open/close 7799 (time 1785397590.26 total 160.18 last 53.54)
 - open/close 8354 (time 1785397600.28 total 170.20 last 55.38)
 - open/close 8753 (time 1785397610.30 total 180.22 last 39.83)
 - open/close 9217 (time 1785397620.31 total 190.23 last 46.35)
 - open/close 9713 (time 1785397630.34 total 200.26 last 49.46)
total: 10000 open/close in 204.32 seconds: 48.94 ops/second
 - unlinked 0 (time 1785397641 ; total 0 ; last 0)
total: 10000 unlinks in 123 seconds: 81.300812 unlinks/second
 - open/close 525 (time 1785397779.71 total 10.00 last 52.49)
 - open/close 1145 (time 1785397789.73 total 20.03 last 61.83)
 - open/close 1657 (time 1785397799.74 total 30.03 last 51.18)
 - open/close 2102 (time 1785397809.74 total 40.04 last 44.47)
 - open/close 2450 (time 1785397819.78 total 50.08 last 34.66)
 - open/close 2860 (time 1785397829.79 total 60.09 last 40.97)
 - open/close 3357 (time 1785397839.88 total 70.18 last 49.26)
 - open/close 3621 (time 1785397849.89 total 80.19 last 26.37)
 - open/close 3928 (time 1785397859.92 total 90.22 last 30.61)
 - open/close 4335 (time 1785397869.94 total 100.23 last 40.64)
 - open/close 4795 (time 1785397879.95 total 110.24 last 45.96)
 - open/close 5330 (time 1785397889.96 total 120.26 last 53.42)
 - open/close 5762 (time 1785397899.96 total 130.26 last 43.20)
 - open/close 6179 (time 1785397909.96 total 140.26 last 41.70)
 - open/close 6731 (time 1785397919.97 total 150.27 last 55.16)
 - open/close 7162 (time 1785397929.99 total 160.29 last 43.01)
 - open/close 7615 (time 1785397939.99 total 170.29 last 45.29)
 - open/close 7996 (time 1785397950.01 total 180.30 last 38.06)
 - open/close 8468 (time 1785397960.01 total 190.30 last 47.18)
 - open/close 8922 (time 1785397970.05 total 200.34 last 45.22)
 - open/close 9449 (time 1785397980.06 total 210.36 last 52.62)
 - open/close 9930 (time 1785397990.09 total 220.38 last 47.99)
total: 10000 open/close in 221.45 seconds: 45.16 ops/second
 - unlinked 0 (time 1785397997 ; total 0 ; last 0)
total: 10000 unlinks in 126 seconds: 79.365082 unlinks/second
 - open/close 314 (time 1785398140.22 total 10.06 last 31.21)
 - open/close 670 (time 1785398150.22 total 20.06 last 35.59)
 - open/close 1248 (time 1785398160.22 total 30.07 last 57.76)
 - open/close 1744 (time 1785398170.23 total 40.08 last 49.55)
 - open/close 2198 (time 1785398180.24 total 50.08 last 45.39)
 - open/close 2618 (time 1785398190.24 total 60.08 last 42.00)
 - open/close 3148 (time 1785398200.25 total 70.10 last 52.92)
 - open/close 3527 (time 1785398210.25 total 80.10 last 37.89)
 - open/close 3941 (time 1785398220.26 total 90.11 last 41.37)
 - open/close 4432 (time 1785398230.30 total 100.14 last 48.92)
 - open/close 4822 (time 1785398240.30 total 110.15 last 38.98)
 - open/close 5312 (time 1785398250.30 total 120.15 last 48.99)
 - open/close 5867 (time 1785398260.32 total 130.17 last 55.39)
 - open/close 6297 (time 1785398270.33 total 140.18 last 42.98)
 - open/close 6729 (time 1785398280.38 total 150.22 last 42.99)
 - open/close 7145 (time 1785398290.39 total 160.23 last 41.57)
 - open/close 7644 (time 1785398300.39 total 170.24 last 49.88)
 - open/close 8111 (time 1785398310.40 total 180.24 last 46.68)
 - open/close 8533 (time 1785398320.40 total 190.24 last 42.19)
 - open/close 9002 (time 1785398330.41 total 200.26 last 46.85)
 - open/close 9480 (time 1785398340.42 total 210.26 last 47.77)
total: 10000 open/close in 220.22 seconds: 45.41 ops/second
 - unlinked 0 (time 1785398356 ; total 0 ; last 0)
total: 10000 unlinks in 114 seconds: 87.719299 unlinks/second
 - open/close 424 (time 1785398486.72 total 10.03 last 42.29)
 - open/close 810 (time 1785398496.77 total 20.07 last 38.43)
 - open/close 1251 (time 1785398506.77 total 30.07 last 44.09)
 - open/close 1810 (time 1785398516.78 total 40.09 last 55.83)
 - open/close 2471 (time 1785398526.78 total 50.09 last 66.10)
 - open/close 2942 (time 1785398536.78 total 60.09 last 47.10)
 - open/close 3369 (time 1785398546.79 total 70.09 last 42.69)
 - open/close 3871 (time 1785398556.80 total 80.10 last 50.14)
 - open/close 4322 (time 1785398566.80 total 90.11 last 45.07)
 - open/close 4677 (time 1785398576.82 total 100.12 last 35.43)
 - open/close 5152 (time 1785398586.83 total 110.13 last 47.46)
 - open/close 5628 (time 1785398596.85 total 120.15 last 47.50)
 - open/close 6084 (time 1785398606.86 total 130.16 last 45.56)
 - open/close 6564 (time 1785398616.89 total 140.20 last 47.84)
 - open/close 7026 (time 1785398626.90 total 150.20 last 46.17)
 - open/close 7473 (time 1785398636.92 total 160.22 last 44.63)
 - open/close 7914 (time 1785398646.93 total 170.23 last 44.06)
 - open/close 8384 (time 1785398656.94 total 180.24 last 46.94)
 - open/close 8844 (time 1785398666.96 total 190.26 last 45.92)
 - open/close 9330 (time 1785398676.97 total 200.28 last 48.53)
 - open/close 9777 (time 1785398686.98 total 210.28 last 44.68)
total: 10000 open/close in 215.49 seconds: 46.40 ops/second
 - unlinked 0 (time 1785398697 ; total 0 ; last 0)
total: 10000 unlinks in 147 seconds: 68.027214 unlinks/second
 - open/close 416 (time 1785398862.44 total 10.01 last 41.57)
 - open/close 779 (time 1785398872.48 total 20.05 last 36.15)
 - open/close 1140 (time 1785398882.48 total 30.05 last 36.09)
 - open/close 1594 (time 1785398892.51 total 40.08 last 45.26)
 - open/close 2064 (time 1785398902.52 total 50.09 last 46.96)
 - open/close 2559 (time 1785398912.53 total 60.10 last 49.46)
 - open/close 3011 (time 1785398922.53 total 70.10 last 45.19)
 - open/close 3487 (time 1785398932.54 total 80.11 last 47.57)
 - open/close 4018 (time 1785398942.55 total 90.12 last 53.02)
 - open/close 4440 (time 1785398952.55 total 100.13 last 42.19)
 - open/close 4962 (time 1785398962.58 total 110.15 last 52.08)
 - open/close 5451 (time 1785398972.58 total 120.15 last 48.89)
 - open/close 5784 (time 1785398982.58 total 130.16 last 33.29)
 - open/close 6129 (time 1785398992.60 total 140.17 last 34.46)
 - open/close 6702 (time 1785399002.60 total 150.17 last 57.26)
 - open/close 7185 (time 1785399012.61 total 160.19 last 48.25)
 - open/close 7601 (time 1785399022.65 total 170.22 last 41.46)
 - open/close 8018 (time 1785399032.65 total 180.22 last 41.69)
 - open/close 8504 (time 1785399042.69 total 190.26 last 48.43)
 - open/close 9015 (time 1785399052.69 total 200.26 last 51.08)
 - open/close 9480 (time 1785399062.70 total 210.27 last 46.44)
 - open/close 9920 (time 1785399072.72 total 220.29 last 43.92)
total: 10000 open/close in 222.20 seconds: 45.00 ops/second
 - unlinked 0 (time 1785399079 ; total 0 ; last 0)
total: 10000 unlinks in 120 seconds: 83.333336 unlinks/second
 - open/close 492 (time 1785399214.54 total 10.01 last 49.17)
 - open/close 978 (time 1785399224.56 total 20.02 last 48.54)
 - open/close 1563 (time 1785399234.57 total 30.03 last 58.40)
 - open/close 1964 (time 1785399244.59 total 40.05 last 40.03)
 - open/close 2447 (time 1785399254.61 total 50.07 last 48.22)
 - open/close 3036 (time 1785399264.64 total 60.10 last 58.71)
 - open/close 3501 (time 1785399274.66 total 70.12 last 46.39)
 - open/close 3973 (time 1785399284.66 total 80.12 last 47.19)
 - open/close 4404 (time 1785399294.67 total 90.13 last 43.06)
 - open/close 4829 (time 1785399304.70 total 100.16 last 42.37)
 - open/close 5271 (time 1785399314.71 total 110.17 last 44.19)
 - open/close 5809 (time 1785399324.71 total 120.17 last 53.80)
 - open/close 6259 (time 1785399334.72 total 130.19 last 44.93)
 - open/close 6801 (time 1785399344.73 total 140.19 last 54.16)
 - open/close 7472 (time 1785399354.75 total 150.21 last 66.99)
 - open/close 8089 (time 1785399364.75 total 160.21 last 61.70)
 - open/close 8631 (time 1785399374.76 total 170.22 last 54.13)
 - open/close 9114 (time 1785399384.77 total 180.23 last 48.26)
 - open/close 9567 (time 1785399394.77 total 190.24 last 45.28)
total: 10000 open/close in 199.71 seconds: 50.07 ops/second
 - unlinked 0 (time 1785399409 ; total 0 ; last 0)
total: 10000 unlinks in 131 seconds: 76.335876 unlinks/second
 - open/close 430 (time 1785399556.55 total 10.02 last 42.91)
 - open/close 915 (time 1785399566.55 total 20.02 last 48.49)
 - open/close 1416 (time 1785399576.56 total 30.03 last 50.07)
 - open/close 1931 (time 1785399586.57 total 40.04 last 51.42)
 - open/close 2285 (time 1785399596.59 total 50.06 last 35.35)
 - open/close 2698 (time 1785399606.62 total 60.08 last 41.18)
 - open/close 3260 (time 1785399616.64 total 70.10 last 56.09)
 - open/close 3760 (time 1785399626.65 total 80.12 last 49.93)
 - open/close 4238 (time 1785399636.66 total 90.13 last 47.75)
 - open/close 4699 (time 1785399646.66 total 100.13 last 46.08)
 - open/close 5063 (time 1785399656.68 total 110.15 last 36.35)
 - open/close 5409 (time 1785399666.70 total 120.17 last 34.53)
 - open/close 5852 (time 1785399676.70 total 130.17 last 44.30)
 - open/close 6418 (time 1785399686.70 total 140.17 last 56.57)
 - open/close 6848 (time 1785399696.73 total 150.20 last 42.89)
 - open/close 7241 (time 1785399706.81 total 160.28 last 38.97)
 - open/close 7625 (time 1785399716.82 total 170.29 last 38.38)
 - open/close 8094 (time 1785399726.83 total 180.30 last 46.86)
 - open/close 8693 (time 1785399736.86 total 190.33 last 59.70)
 - open/close 9260 (time 1785399746.87 total 200.34 last 56.66)
 - open/close 9800 (time 1785399756.89 total 210.36 last 53.85)
total: 10000 open/close in 214.62 seconds: 46.59 ops/second
 - unlinked 0 (time 1785399768 ; total 0 ; last 0)
total: 10000 unlinks in 128 seconds: 78.125000 unlinks/second
 - open/close 429 (time 1785399912.46 total 10.01 last 42.86)
 - open/close 866 (time 1785399922.47 total 20.02 last 43.66)
 - open/close 1334 (time 1785399932.49 total 30.04 last 46.70)
 - open/close 1946 (time 1785399942.50 total 40.06 last 61.10)
 - open/close 2531 (time 1785399952.51 total 50.06 last 58.45)
 - open/close 2962 (time 1785399962.52 total 60.07 last 43.07)
 - open/close 3428 (time 1785399972.54 total 70.09 last 46.51)
 - open/close 3948 (time 1785399982.57 total 80.12 last 51.83)
 - open/close 4560 (time 1785399992.57 total 90.12 last 61.19)
 - open/close 5170 (time 1785400002.59 total 100.14 last 60.93)
 - open/close 5721 (time 1785400012.59 total 110.14 last 55.10)
 - open/close 6239 (time 1785400022.59 total 120.14 last 51.78)
 - open/close 6648 (time 1785400032.60 total 130.16 last 40.84)
 - open/close 7156 (time 1785400042.62 total 140.17 last 50.72)
 - open/close 7683 (time 1785400052.64 total 150.19 last 52.60)
 - open/close 8188 (time 1785400062.66 total 160.21 last 50.41)
 - open/close 8767 (time 1785400072.66 total 170.21 last 57.87)
 - open/close 9244 (time 1785400082.68 total 180.23 last 47.61)
 - open/close 9697 (time 1785400092.71 total 190.26 last 45.19)
total: 10000 open/close in 196.05 seconds: 51.01 ops/second
 - unlinked 0 (time 1785400104 ; total 0 ; last 0)
total: 10000 unlinks in 122 seconds: 81.967216 unlinks/second
 - open/close 629 (time 1785400242.54 total 10.01 last 62.84)
 - open/close 1427 (time 1785400252.54 total 20.01 last 79.76)
 - open/close 1938 (time 1785400262.57 total 30.04 last 50.97)
 - open/close 2461 (time 1785400272.58 total 40.05 last 52.25)
 - open/close 2943 (time 1785400282.59 total 50.06 last 48.15)
 - open/close 3471 (time 1785400292.62 total 60.09 last 52.62)
 - open/close 3858 (time 1785400302.62 total 70.09 last 38.70)
 - open/close 4258 (time 1785400312.63 total 80.10 last 39.96)
 - open/close 4653 (time 1785400322.63 total 90.11 last 39.48)
 - open/close 4968 (time 1785400332.65 total 100.13 last 31.44)
 - open/close 5530 (time 1785400342.66 total 110.13 last 56.18)
 - open/close 5966 (time 1785400352.67 total 120.14 last 43.55)
 - open/close 6490 (time 1785400362.69 total 130.16 last 52.31)
 - open/close 7000 (time 1785400372.71 total 140.19 last 50.86)
 - open/close 7453 (time 1785400382.73 total 150.20 last 45.23)
 - open/close 7759 (time 1785400392.74 total 160.22 last 30.55)
 - open/close 8276 (time 1785400402.76 total 170.23 last 51.63)
 - open/close 8736 (time 1785400412.77 total 180.25 last 45.94)
 - open/close 9276 (time 1785400422.79 total 190.26 last 53.91)
 - open/close 9737 (time 1785400432.80 total 200.28 last 46.04)
total: 10000 open/close in 207.82 seconds: 48.12 ops/second
 - unlinked 0 (time 1785400447 ; total 0 ; last 0)
total: 10000 unlinks in 119 seconds: 84.033615 unlinks/second
 - open/close 463 (time 1785400582.04 total 10.01 last 46.27)
 - open/close 940 (time 1785400592.04 total 20.01 last 47.69)
 - open/close 1351 (time 1785400602.05 total 30.02 last 41.04)
 - open/close 1801 (time 1785400612.06 total 40.02 last 44.99)
 - open/close 2286 (time 1785400622.06 total 50.03 last 48.48)
 - open/close 2552 (time 1785400632.11 total 60.08 last 26.46)
 - open/close 2934 (time 1785400642.14 total 70.11 last 38.09)
 - open/close 3498 (time 1785400652.16 total 80.13 last 56.30)
 - open/close 3937 (time 1785400662.16 total 90.13 last 43.90)
 - open/close 4390 (time 1785400672.16 total 100.13 last 45.30)
 - open/close 4942 (time 1785400682.18 total 110.15 last 55.09)
 - open/close 5440 (time 1785400692.19 total 120.16 last 49.77)
 - open/close 5947 (time 1785400702.20 total 130.16 last 50.66)
 - open/close 6479 (time 1785400712.23 total 140.19 last 53.05)
 - open/close 7069 (time 1785400722.24 total 150.20 last 58.94)
 - open/close 7526 (time 1785400732.26 total 160.22 last 45.61)
 - open/close 8008 (time 1785400742.27 total 170.24 last 48.12)
 - open/close 8449 (time 1785400752.28 total 180.25 last 44.08)
 - open/close 8900 (time 1785400762.28 total 190.25 last 45.09)
 - open/close 9400 (time 1785400772.30 total 200.27 last 49.91)
 - open/close 9914 (time 1785400782.33 total 210.30 last 51.22)
total: 10000 open/close in 211.67 seconds: 47.24 ops/second
 - unlinked 0 (time 1785400789 ; total 0 ; last 0)
total: 10000 unlinks in 121 seconds: 82.644630 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg328-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg328-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg328-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.128@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.203.128@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg328-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all
pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1
Started lustre-OST0000
oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg328-server: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg328-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg328-client.virtnet:  -o user_xattr,flock 192.168.203.128@tcp:/lustre /mnt/lustre
On OST0, 20326 used inodes rc=0
pdsh@oleg328-client: oleg328-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg328-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg328-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg328-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.128@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.203.128@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.128@tcp autodegrade=on sys.timeout=20

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.128@tcp autodegrade=on sys.timeout=20 autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.203.128@tcp
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg328-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all
pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1
Started lustre-OST0000
oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg328-server: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg328-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg328-client.virtnet:  -o user_xattr,flock 192.168.203.128@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg328-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg328-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg328-server
stop mds service on oleg328-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg328-server

unloading modules via unload_modules_local on: 'oleg328-server'
oleg328-server: oleg328-server.virtnet: executing unload_modules_local
oleg328-server: 
modules unloaded.
pdsh@oleg328-client: oleg328-client: ssh exited with exit code 2
pdsh@oleg328-client: oleg328-server: ssh exited with exit code 2
pdsh@oleg328-client: oleg328-server: ssh exited with exit code 2
pdsh@oleg328-client: oleg328-client: ssh exited with exit code 2