== conf-sanity test 69: replace an OST with the same index ========================================================== 04:25:30 (1743495930)
start mds service on oleg660-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg660-server'
oleg660-server: oleg660-server.virtnet: executing load_modules_local
oleg660-server: Loading modules from /home/green/git/lustre-release/lustre
oleg660-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy
oleg660-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy
oleg660-server: detected 4 online CPUs by sysfs
oleg660-server: MODOPTS_LIBCFS=
oleg660-server: Force libcfs to create 2 CPU partitions
oleg660-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg660-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg660-server: oleg660-server.virtnet: executing set_default_debug -1 all
pdsh@oleg660-client: oleg660-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg660-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg660-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg660-server: oleg660-server.virtnet: executing set_default_debug -1 all
pdsh@oleg660-client: oleg660-server: ssh exited with exit code 1
Started lustre-OST0000
oleg660-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg660-client.virtnet:  -o user_xattr,flock 192.168.206.160@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11872 inodes available. Want 99872. rc=0
 - open/close 380 (time 1743496093.78 total 10.02 last 37.93)
 - open/close 712 (time 1743496103.81 total 20.04 last 33.12)
 - open/close 1071 (time 1743496113.81 total 30.05 last 35.89)
 - open/close 1403 (time 1743496123.83 total 40.06 last 33.15)
 - open/close 1758 (time 1743496133.84 total 50.07 last 35.46)
 - open/close 2168 (time 1743496143.84 total 60.08 last 40.99)
 - open/close 2515 (time 1743496153.86 total 70.09 last 34.65)
 - open/close 2874 (time 1743496163.86 total 80.10 last 35.88)
 - open/close 3249 (time 1743496173.87 total 90.11 last 37.45)
 - open/close 3594 (time 1743496183.88 total 100.12 last 34.47)
 - open/close 3928 (time 1743496193.90 total 110.13 last 33.36)
 - open/close 4286 (time 1743496203.94 total 120.17 last 35.66)
 - open/close 4582 (time 1743496213.95 total 130.18 last 29.57)
 - open/close 4906 (time 1743496223.97 total 140.21 last 32.31)
 - open/close 5210 (time 1743496233.98 total 150.21 last 30.40)
 - open/close 5544 (time 1743496243.99 total 160.22 last 33.36)
 - open/close 5864 (time 1743496253.99 total 170.22 last 32.00)
 - open/close 6201 (time 1743496264.01 total 180.24 last 33.63)
 - open/close 6553 (time 1743496274.02 total 190.26 last 35.16)
 - open/close 6875 (time 1743496284.02 total 200.26 last 32.19)
 - open/close 7197 (time 1743496294.08 total 210.31 last 32.02)
 - open/close 7521 (time 1743496304.10 total 220.34 last 32.33)
 - open/close 7872 (time 1743496314.12 total 230.35 last 35.04)
 - open/close 8210 (time 1743496324.13 total 240.37 last 33.74)
 - open/close 8532 (time 1743496334.15 total 250.38 last 32.15)
 - open/close 8893 (time 1743496344.16 total 260.40 last 36.05)
 - open/close 9230 (time 1743496354.19 total 270.42 last 33.62)
 - open/close 9584 (time 1743496364.19 total 280.43 last 35.38)
 - open/close 9912 (time 1743496374.20 total 290.43 last 32.80)
total: 10000 open/close in 292.79 seconds: 34.15 ops/second
 - unlinked 0 (time 1743496387 ; total 0 ; last 0)
total: 10000 unlinks in 203 seconds: 49.261086 unlinks/second
 - open/close 358 (time 1743496610.90 total 10.01 last 35.77)
 - open/close 689 (time 1743496620.91 total 20.01 last 33.08)
 - open/close 1079 (time 1743496630.91 total 30.02 last 38.99)
 - open/close 1422 (time 1743496640.92 total 40.02 last 34.28)
 - open/close 1750 (time 1743496650.93 total 50.04 last 32.75)
 - open/close 2102 (time 1743496660.95 total 60.05 last 35.15)
 - open/close 2452 (time 1743496670.95 total 70.06 last 34.99)
 - open/close 2837 (time 1743496680.97 total 80.08 last 38.42)
 - open/close 3183 (time 1743496690.99 total 90.09 last 34.55)
 - open/close 3540 (time 1743496700.99 total 100.10 last 35.68)
 - open/close 3901 (time 1743496711.01 total 110.12 last 36.03)
 - open/close 4233 (time 1743496721.09 total 120.20 last 32.94)
 - open/close 4548 (time 1743496731.15 total 130.25 last 31.33)
 - open/close 4805 (time 1743496741.17 total 140.27 last 25.64)
 - open/close 5096 (time 1743496751.19 total 150.30 last 29.03)
 - open/close 5407 (time 1743496761.23 total 160.33 last 30.99)
 - open/close 5701 (time 1743496771.24 total 170.35 last 29.37)
 - open/close 6009 (time 1743496781.27 total 180.38 last 30.71)
 - open/close 6296 (time 1743496791.27 total 190.38 last 28.69)
 - open/close 6602 (time 1743496801.28 total 200.39 last 30.57)
 - open/close 6928 (time 1743496811.30 total 210.40 last 32.56)
 - open/close 7275 (time 1743496821.33 total 220.43 last 34.60)
 - open/close 7601 (time 1743496831.33 total 230.44 last 32.58)
 - open/close 7935 (time 1743496841.34 total 240.45 last 33.36)
 - open/close 8310 (time 1743496851.37 total 250.47 last 37.41)
 - open/close 8611 (time 1743496861.39 total 260.50 last 30.02)
 - open/close 8944 (time 1743496871.43 total 270.54 last 33.17)
 - open/close 9266 (time 1743496881.45 total 280.56 last 32.13)
 - open/close 9582 (time 1743496891.45 total 290.56 last 31.59)
 - open/close 9920 (time 1743496901.47 total 300.58 last 33.73)
total: 10000 open/close in 302.96 seconds: 33.01 ops/second
 - unlinked 0 (time 1743496915 ; total 0 ; last 0)
total: 10000 unlinks in 197 seconds: 50.761421 unlinks/second
 - open/close 399 (time 1743497134.53 total 10.01 last 39.85)
 - open/close 780 (time 1743497144.53 total 20.01 last 38.10)
 - open/close 1169 (time 1743497154.55 total 30.04 last 38.81)
 - open/close 1516 (time 1743497164.56 total 40.04 last 34.69)
 - open/close 1866 (time 1743497174.56 total 50.04 last 34.99)
 - open/close 2227 (time 1743497184.57 total 60.05 last 36.07)
 - open/close 2580 (time 1743497194.58 total 70.07 last 35.24)
 - open/close 2945 (time 1743497204.59 total 80.07 last 36.48)
 - open/close 3311 (time 1743497214.60 total 90.09 last 36.55)
 - open/close 3643 (time 1743497224.63 total 100.12 last 33.11)
 - open/close 4026 (time 1743497234.65 total 110.13 last 38.25)
 - open/close 4404 (time 1743497244.67 total 120.15 last 37.71)
 - open/close 4767 (time 1743497254.68 total 130.16 last 36.27)
 - open/close 5083 (time 1743497264.68 total 140.16 last 31.58)
 - open/close 5450 (time 1743497274.70 total 150.18 last 36.63)
 - open/close 5821 (time 1743497284.71 total 160.19 last 37.08)
 - open/close 6178 (time 1743497294.71 total 170.19 last 35.70)
 - open/close 6543 (time 1743497304.75 total 180.24 last 36.32)
 - open/close 6917 (time 1743497314.78 total 190.27 last 37.29)
 - open/close 7284 (time 1743497324.81 total 200.29 last 36.61)
 - open/close 7625 (time 1743497334.81 total 210.29 last 34.09)
 - open/close 7991 (time 1743497344.82 total 220.30 last 36.58)
 - open/close 8371 (time 1743497354.84 total 230.32 last 37.92)
 - open/close 8744 (time 1743497364.85 total 240.33 last 37.25)
 - open/close 9092 (time 1743497374.87 total 250.35 last 34.74)
 - open/close 9428 (time 1743497384.87 total 260.35 last 33.59)
 - open/close 9794 (time 1743497394.88 total 270.36 last 36.58)
total: 10000 open/close in 276.05 seconds: 36.23 ops/second
 - unlinked 0 (time 1743497409 ; total 0 ; last 0)
total: 10000 unlinks in 187 seconds: 53.475937 unlinks/second
 - open/close 399 (time 1743497615.68 total 10.02 last 39.84)
 - open/close 793 (time 1743497625.70 total 20.04 last 39.31)
 - open/close 1179 (time 1743497635.71 total 30.05 last 38.57)
 - open/close 1576 (time 1743497645.73 total 40.06 last 39.64)
 - open/close 1980 (time 1743497655.76 total 50.09 last 40.26)
 - open/close 2376 (time 1743497665.80 total 60.13 last 39.46)
 - open/close 2769 (time 1743497675.81 total 70.14 last 39.26)
 - open/close 3155 (time 1743497685.82 total 80.15 last 38.56)
 - open/close 3575 (time 1743497695.82 total 90.15 last 41.98)
 - open/close 3976 (time 1743497705.83 total 100.17 last 40.05)
 - open/close 4362 (time 1743497715.84 total 110.17 last 38.58)
 - open/close 4774 (time 1743497725.85 total 120.18 last 41.14)
 - open/close 5190 (time 1743497735.86 total 130.20 last 41.56)
 - open/close 5584 (time 1743497745.89 total 140.22 last 39.31)
 - open/close 6012 (time 1743497755.90 total 150.24 last 42.72)
 - open/close 6438 (time 1743497765.92 total 160.25 last 42.53)
 - open/close 6849 (time 1743497775.95 total 170.28 last 41.00)
 - open/close 7245 (time 1743497785.99 total 180.33 last 39.41)
 - open/close 7625 (time 1743497796.01 total 190.34 last 37.94)
 - open/close 8015 (time 1743497806.02 total 200.35 last 38.95)
 - open/close 8381 (time 1743497816.04 total 210.37 last 36.54)
 - open/close 8793 (time 1743497826.05 total 220.38 last 41.17)
 - open/close 9167 (time 1743497836.05 total 230.38 last 37.39)
 - open/close 9555 (time 1743497846.07 total 240.40 last 38.72)
 - open/close 9950 (time 1743497856.07 total 250.40 last 39.49)
total: 10000 open/close in 251.98 seconds: 39.69 ops/second
 - unlinked 0 (time 1743497866 ; total 0 ; last 0)
total: 10000 unlinks in 175 seconds: 57.142857 unlinks/second
 - open/close 401 (time 1743498059.68 total 10.01 last 40.08)
 - open/close 807 (time 1743498069.70 total 20.02 last 40.53)
 - open/close 1236 (time 1743498079.71 total 30.03 last 42.87)
 - open/close 1678 (time 1743498089.71 total 40.04 last 44.18)
 - open/close 2110 (time 1743498099.72 total 50.05 last 43.15)
 - open/close 2546 (time 1743498109.74 total 60.06 last 43.55)
 - open/close 2970 (time 1743498119.74 total 70.06 last 42.39)
 - open/close 3388 (time 1743498129.77 total 80.09 last 41.69)
 - open/close 3838 (time 1743498139.78 total 90.10 last 44.94)
 - open/close 4288 (time 1743498149.78 total 100.10 last 45.00)
 - open/close 4682 (time 1743498159.80 total 110.12 last 39.33)
 - open/close 5119 (time 1743498169.89 total 120.21 last 43.30)
 - open/close 5541 (time 1743498179.90 total 130.22 last 42.17)
 - open/close 5953 (time 1743498189.93 total 140.25 last 41.06)
 - open/close 6374 (time 1743498199.93 total 150.25 last 42.10)
 - open/close 6784 (time 1743498209.93 total 160.26 last 40.99)
 - open/close 7244 (time 1743498219.96 total 170.28 last 45.87)
 - open/close 7659 (time 1743498229.97 total 180.30 last 41.45)
 - open/close 8114 (time 1743498239.97 total 190.30 last 45.50)
 - open/close 8543 (time 1743498249.99 total 200.31 last 42.85)
 - open/close 8922 (time 1743498259.99 total 210.31 last 37.89)
 - open/close 9319 (time 1743498270.02 total 220.34 last 39.57)
 - open/close 9717 (time 1743498280.02 total 230.35 last 39.79)
total: 10000 open/close in 237.16 seconds: 42.17 ops/second
 - unlinked 0 (time 1743498294 ; total 0 ; last 0)
total: 10000 unlinks in 158 seconds: 63.291138 unlinks/second
 - open/close 452 (time 1743498470.45 total 10.01 last 45.16)
 - open/close 909 (time 1743498480.46 total 20.02 last 45.65)
 - open/close 1378 (time 1743498490.48 total 30.03 last 46.83)
 - open/close 1826 (time 1743498500.49 total 40.05 last 44.74)
 - open/close 2265 (time 1743498510.50 total 50.06 last 43.85)
 - open/close 2724 (time 1743498520.52 total 60.08 last 45.79)
 - open/close 3152 (time 1743498530.53 total 70.09 last 42.77)
 - open/close 3599 (time 1743498540.54 total 80.10 last 44.66)
 - open/close 4043 (time 1743498550.55 total 90.11 last 44.34)
 - open/close 4461 (time 1743498560.57 total 100.12 last 41.74)
 - open/close 4869 (time 1743498570.58 total 110.13 last 40.76)
 - open/close 5296 (time 1743498580.60 total 120.15 last 42.61)
 - open/close 5707 (time 1743498590.61 total 130.17 last 41.04)
 - open/close 6130 (time 1743498600.64 total 140.20 last 42.17)
 - open/close 6552 (time 1743498610.65 total 150.20 last 42.19)
 - open/close 6984 (time 1743498620.68 total 160.24 last 43.06)
 - open/close 7427 (time 1743498630.68 total 170.24 last 44.28)
 - open/close 7885 (time 1743498640.70 total 180.26 last 45.72)
 - open/close 8338 (time 1743498650.70 total 190.26 last 45.30)
 - open/close 8743 (time 1743498660.71 total 200.26 last 40.48)
 - open/close 9175 (time 1743498670.72 total 210.28 last 43.13)
 - open/close 9603 (time 1743498680.74 total 220.29 last 42.74)
total: 10000 open/close in 229.65 seconds: 43.54 ops/second
 - unlinked 0 (time 1743498697 ; total 0 ; last 0)
total: 10000 unlinks in 146 seconds: 68.493149 unlinks/second
 - open/close 478 (time 1743498861.27 total 10.01 last 47.75)
 - open/close 941 (time 1743498871.29 total 20.04 last 46.18)
 - open/close 1393 (time 1743498881.30 total 30.04 last 45.18)
 - open/close 1853 (time 1743498891.30 total 40.05 last 45.98)
 - open/close 2347 (time 1743498901.31 total 50.06 last 49.36)
 - open/close 2852 (time 1743498911.32 total 60.06 last 50.47)
 - open/close 3367 (time 1743498921.32 total 70.07 last 51.47)
 - open/close 3871 (time 1743498931.32 total 80.07 last 50.40)
 - open/close 4359 (time 1743498941.34 total 90.08 last 48.72)
 - open/close 4836 (time 1743498951.34 total 100.09 last 47.67)
 - open/close 5347 (time 1743498961.35 total 110.09 last 51.08)
 - open/close 5824 (time 1743498971.36 total 120.10 last 47.65)
 - open/close 6250 (time 1743498981.38 total 130.13 last 42.51)
 - open/close 6698 (time 1743498991.39 total 140.14 last 44.75)
 - open/close 7140 (time 1743499001.41 total 150.16 last 44.11)
 - open/close 7622 (time 1743499011.42 total 160.17 last 48.13)
 - open/close 8080 (time 1743499021.44 total 170.18 last 45.75)
 - open/close 8504 (time 1743499031.44 total 180.19 last 42.36)
 - open/close 8922 (time 1743499041.46 total 190.21 last 41.72)
 - open/close 9355 (time 1743499051.47 total 200.21 last 43.29)
 - open/close 9776 (time 1743499061.48 total 210.23 last 42.03)
total: 10000 open/close in 215.43 seconds: 46.42 ops/second
 - unlinked 0 (time 1743499073 ; total 0 ; last 0)
total: 10000 unlinks in 149 seconds: 67.114098 unlinks/second
 - open/close 513 (time 1743499238.72 total 10.01 last 51.23)
 - open/close 1000 (time 1743499248.73 total 20.03 last 48.64)
 - open/close 1569 (time 1743499258.75 total 30.04 last 56.79)
 - open/close 2128 (time 1743499268.76 total 40.05 last 55.84)
 - open/close 2683 (time 1743499278.77 total 50.06 last 55.45)
 - open/close 3247 (time 1743499288.78 total 60.07 last 56.34)
 - open/close 3774 (time 1743499298.80 total 70.10 last 52.59)
 - open/close 4264 (time 1743499308.80 total 80.10 last 48.98)
 - open/close 4774 (time 1743499318.82 total 90.11 last 50.92)
 - open/close 5266 (time 1743499328.83 total 100.12 last 49.17)
 - open/close 5792 (time 1743499338.84 total 110.13 last 52.54)
 - open/close 6312 (time 1743499348.84 total 120.14 last 51.97)
 - open/close 6818 (time 1743499358.86 total 130.15 last 50.52)
 - open/close 7319 (time 1743499368.87 total 140.17 last 50.04)
 - open/close 7899 (time 1743499378.88 total 150.17 last 57.96)
 - open/close 8431 (time 1743499388.88 total 160.17 last 53.20)
 - open/close 8998 (time 1743499398.90 total 170.20 last 56.55)
 - open/close 9605 (time 1743499408.95 total 180.25 last 60.40)
total: 10000 open/close in 187.58 seconds: 53.31 ops/second
 - unlinked 0 (time 1743499421 ; total 0 ; last 0)
total: 10000 unlinks in 123 seconds: 81.300812 unlinks/second
 - open/close 634 (time 1743499560.55 total 10.00 last 63.37)
 - open/close 1227 (time 1743499570.57 total 20.02 last 59.21)
 - open/close 1832 (time 1743499580.58 total 30.04 last 60.41)
 - open/close 2467 (time 1743499590.60 total 40.05 last 63.41)
 - open/close 3089 (time 1743499600.62 total 50.07 last 62.09)
 - open/close 3665 (time 1743499610.62 total 60.07 last 57.58)
 - open/close 4258 (time 1743499620.62 total 70.07 last 59.29)
 - open/close 4772 (time 1743499630.63 total 80.08 last 51.35)
 - open/close 5284 (time 1743499640.63 total 90.08 last 51.19)
 - open/close 5812 (time 1743499650.65 total 100.11 last 52.69)
 - open/close 6349 (time 1743499660.67 total 110.12 last 53.62)
 - open/close 6884 (time 1743499670.70 total 120.15 last 53.33)
 - open/close 7472 (time 1743499680.71 total 130.16 last 58.73)
 - open/close 8057 (time 1743499690.73 total 140.18 last 58.42)
 - open/close 8667 (time 1743499700.75 total 150.20 last 60.83)
 - open/close 9252 (time 1743499710.77 total 160.22 last 58.39)
 - open/close 9804 (time 1743499720.78 total 170.23 last 55.19)
total: 10000 open/close in 173.50 seconds: 57.64 ops/second
 - unlinked 0 (time 1743499729 ; total 0 ; last 0)
total: 10000 unlinks in 117 seconds: 85.470085 unlinks/second
 - open/close 570 (time 1743499862.49 total 10.01 last 56.97)
 - open/close 1157 (time 1743499872.50 total 20.01 last 58.68)
 - open/close 1747 (time 1743499882.51 total 30.02 last 58.94)
 - open/close 2296 (time 1743499892.52 total 40.03 last 54.83)
 - open/close 2843 (time 1743499902.52 total 50.03 last 54.70)
 - open/close 3402 (time 1743499912.52 total 60.04 last 55.89)
 - open/close 3974 (time 1743499922.52 total 70.04 last 57.19)
 - open/close 4562 (time 1743499932.53 total 80.05 last 58.74)
 - open/close 5096 (time 1743499942.54 total 90.06 last 53.35)
 - open/close 5639 (time 1743499952.56 total 100.07 last 54.24)
 - open/close 6136 (time 1743499962.56 total 110.08 last 49.67)
 - open/close 6682 (time 1743499972.57 total 120.08 last 54.55)
 - open/close 7308 (time 1743499982.58 total 130.10 last 62.51)
 - open/close 7948 (time 1743499992.63 total 140.15 last 63.70)
 - open/close 8593 (time 1743500002.64 total 150.15 last 64.46)
 - open/close 9170 (time 1743500012.65 total 160.16 last 57.63)
 - open/close 9701 (time 1743500022.67 total 170.19 last 52.98)
total: 10000 open/close in 176.46 seconds: 56.67 ops/second
 - unlinked 0 (time 1743500034 ; total 0 ; last 0)
total: 10000 unlinks in 113 seconds: 88.495575 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg660-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg660-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg660-server

   Permanent disk data:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.206.160@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.206.160@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg660-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg660-server: oleg660-server.virtnet: executing set_default_debug -1 all
pdsh@oleg660-client: oleg660-server: ssh exited with exit code 1
Started lustre-OST0000
oleg660-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg660-server: oleg660-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg660-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg660-client.virtnet:  -o user_xattr,flock 192.168.206.160@tcp:/lustre /mnt/lustre
On OST0, 16469 used inodes rc=0
umount lustre on /mnt/lustre.....
Stopping client oleg660-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg660-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg660-server
stop mds service on oleg660-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg660-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
unloading modules via unload_modules_local on: 'oleg660-server'
oleg660-server: oleg660-server.virtnet: executing unload_modules_local
oleg660-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
oleg660-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
modules unloaded.
pdsh@oleg660-client: oleg660-client: ssh exited with exit code 2
pdsh@oleg660-client: oleg660-server: ssh exited with exit code 2
pdsh@oleg660-client: oleg660-client: ssh exited with exit code 2
pdsh@oleg660-client: oleg660-server: ssh exited with exit code 2