== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 05:57:53 (1781603873)
Stopping clients: oleg649-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg649-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg649-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg649-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg649-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg649-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg649-server
oleg649-server: oleg649-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg649-server'
oleg649-server: oleg649-server.virtnet: executing load_modules_local
oleg649-server: Loading modules from /home/green/git/lustre-release/lustre
oleg649-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy
oleg649-server: detected 4 online CPUs by sysfs
oleg649-server: MODOPTS_LIBCFS=
oleg649-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg649-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg649-server: oleg649-server.virtnet: executing set_default_debug -1 all
pdsh@oleg649-client: oleg649-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg649-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg649-server: oleg649-server.virtnet: executing set_default_debug -1 all
pdsh@oleg649-client: oleg649-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg649-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg649-server: oleg649-server.virtnet: executing set_default_debug -1 all
pdsh@oleg649-client: oleg649-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg649-client.virtnet:  -o user_xattr,flock 192.168.206.149@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H fnv_1a_64  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 621 (time 1781604092.32 total 10.02 last 61.97)
 - open/close 1126 (time 1781604102.34 total 20.03 last 50.44)
 - open/close 1686 (time 1781604112.35 total 30.04 last 55.94)
 - open/close 2256 (time 1781604122.35 total 40.05 last 56.98)
 - open/close 2935 (time 1781604132.37 total 50.06 last 67.78)
 - open/close 3507 (time 1781604142.37 total 60.07 last 57.17)
 - open/close 4112 (time 1781604152.39 total 70.09 last 60.37)
total: 4500 open/close in 76.25 seconds: 59.01 ops/second
 - /unlink 1113 (time 1781604173.48 total 10.00 last 111.28)
 - /unlink 1982 (time 1781604183.48 total 20.00 last 86.89)
 - /unlink 3029 (time 1781604193.48 total 30.00 last 104.68)
 - /unlink 4076 (time 1781604203.48 total 40.01 last 104.64)
total: 4500 /unlink in 44.52 seconds: 101.08 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 580 (time 1781604223.29 total 10.02 last 57.91)
 - open/close 1167 (time 1781604233.30 total 20.03 last 58.62)
 - open/close 1764 (time 1781604243.31 total 30.04 last 59.62)
 - open/close 2309 (time 1781604253.32 total 40.05 last 54.48)
 - open/close 2873 (time 1781604263.32 total 50.05 last 56.37)
 - open/close 3381 (time 1781604273.33 total 60.06 last 50.79)
 - open/close 3906 (time 1781604283.33 total 70.06 last 52.50)
 - open/close 4468 (time 1781604293.35 total 80.08 last 56.05)
total: 4500 open/close in 80.72 seconds: 55.75 ops/second
 - /unlink 1094 (time 1781604309.66 total 10.01 last 109.32)
 - /unlink 2078 (time 1781604319.66 total 20.01 last 98.36)
 - /unlink 3099 (time 1781604329.66 total 30.01 last 102.08)
 - /unlink 4158 (time 1781604339.66 total 40.02 last 105.87)
total: 4500 /unlink in 43.54 seconds: 103.35 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 542 (time 1781604358.80 total 10.01 last 54.17)
 - open/close 1077 (time 1781604368.81 total 20.01 last 53.49)
 - open/close 1673 (time 1781604378.83 total 30.03 last 59.47)
 - open/close 2173 (time 1781604388.85 total 40.05 last 49.90)
 - open/close 2724 (time 1781604398.85 total 50.05 last 55.10)
 - open/close 3440 (time 1781604408.86 total 60.06 last 71.55)
 - open/close 3984 (time 1781604418.86 total 70.07 last 54.36)
total: 4500 open/close in 79.37 seconds: 56.70 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1094 (time 1781604443.09 total 10.00 last 109.37)
 - /unlink 2070 (time 1781604453.10 total 20.01 last 97.54)
 - /unlink 3249 (time 1781604463.10 total 30.01 last 117.86)
 - /unlink 4428 (time 1781604473.10 total 40.01 last 117.89)
total: 4500 /unlink in 40.84 seconds: 110.20 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 500 (time 1781604489.44 total 10.01 last 49.93)
 - open/close 1073 (time 1781604499.45 total 20.02 last 57.25)
 - open/close 1653 (time 1781604509.46 total 30.03 last 57.93)
 - open/close 2251 (time 1781604519.49 total 40.06 last 59.64)
 - open/close 2775 (time 1781604529.51 total 50.08 last 52.31)
 - open/close 3366 (time 1781604539.51 total 60.08 last 59.09)
 - open/close 3933 (time 1781604549.52 total 70.09 last 56.64)
total: 4500 open/close in 79.53 seconds: 56.58 ops/second
 - /unlink 1033 (time 1781604574.10 total 10.01 last 103.16)
 - /unlink 2091 (time 1781604584.11 total 20.02 last 105.72)
 - /unlink 3220 (time 1781604594.12 total 30.03 last 112.83)
 - /unlink 4310 (time 1781604604.12 total 40.03 last 108.92)
total: 4500 /unlink in 41.99 seconds: 107.16 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 588 (time 1781604620.75 total 10.02 last 58.69)
 - open/close 1362 (time 1781604630.76 total 20.03 last 77.33)
 - open/close 2057 (time 1781604640.76 total 30.03 last 69.48)
 - open/close 2852 (time 1781604650.77 total 40.04 last 79.45)
 - open/close 3545 (time 1781604660.78 total 50.05 last 69.23)
 - open/close 4257 (time 1781604670.78 total 60.05 last 71.16)
total: 4500 open/close in 63.51 seconds: 70.85 ops/second
 - /unlink 1119 (time 1781604688.71 total 10.01 last 111.80)
 - /unlink 2258 (time 1781604698.72 total 20.01 last 113.86)
 - /unlink 3482 (time 1781604708.72 total 30.01 last 122.38)
total: 4500 /unlink in 39.24 seconds: 114.68 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 714 (time 1781604732.97 total 10.01 last 71.36)
 - open/close 1336 (time 1781604742.97 total 20.01 last 62.19)
 - open/close 1907 (time 1781604752.97 total 30.01 last 57.08)
 - open/close 2544 (time 1781604762.99 total 40.03 last 63.61)
 - open/close 3161 (time 1781604773.01 total 50.04 last 61.58)
 - open/close 3744 (time 1781604783.01 total 60.05 last 58.25)
 - open/close 4344 (time 1781604793.02 total 70.06 last 59.98)
total: 4500 open/close in 72.11 seconds: 62.41 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1256 (time 1781604809.15 total 10.01 last 125.53)
 - /unlink 2611 (time 1781604819.15 total 20.01 last 135.50)
 - /unlink 4064 (time 1781604829.15 total 30.01 last 145.25)
total: 4500 /unlink in 33.17 seconds: 135.65 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 706 (time 1781604846.61 total 10.01 last 70.54)
 - open/close 1363 (time 1781604856.64 total 20.04 last 65.52)
 - open/close 2020 (time 1781604866.65 total 30.05 last 65.62)
 - open/close 2701 (time 1781604876.65 total 40.05 last 68.06)
 - open/close 3387 (time 1781604886.66 total 50.06 last 68.58)
 - open/close 4102 (time 1781604896.69 total 60.09 last 71.27)
total: 4500 open/close in 65.37 seconds: 68.84 ops/second
 - /unlink 1158 (time 1781604917.26 total 10.02 last 115.51)
 - /unlink 2284 (time 1781604927.27 total 20.03 last 112.54)
 - /unlink 3591 (time 1781604937.27 total 30.03 last 130.67)
total: 4500 /unlink in 37.00 seconds: 121.62 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 534 (time 1781604959.70 total 10.01 last 53.35)
 - open/close 1025 (time 1781604969.72 total 20.03 last 49.01)
 - open/close 1603 (time 1781604979.74 total 30.05 last 57.67)
 - open/close 2157 (time 1781604989.75 total 40.06 last 55.34)
 - open/close 2697 (time 1781604999.75 total 50.06 last 53.99)
 - open/close 3231 (time 1781605009.78 total 60.09 last 53.26)
 - open/close 3752 (time 1781605019.79 total 70.10 last 52.04)
 - open/close 4337 (time 1781605029.80 total 80.11 last 58.46)
total: 4500 open/close in 82.87 seconds: 54.30 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 972 (time 1781605047.18 total 10.00 last 97.20)
 - /unlink 1989 (time 1781605057.18 total 20.00 last 101.67)
 - /unlink 2971 (time 1781605067.19 total 30.00 last 98.19)
 - /unlink 4009 (time 1781605077.19 total 40.01 last 103.76)
total: 4500 /unlink in 44.71 seconds: 100.64 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 543 (time 1781605096.89 total 10.00 last 54.28)
 - open/close 1083 (time 1781605106.89 total 20.01 last 53.98)
 - open/close 1632 (time 1781605116.90 total 30.01 last 54.86)
 - open/close 2244 (time 1781605126.93 total 40.05 last 61.01)
 - open/close 2883 (time 1781605136.94 total 50.06 last 63.83)
 - open/close 3465 (time 1781605146.95 total 60.07 last 58.15)
 - open/close 3936 (time 1781605157.00 total 70.11 last 46.88)
 - open/close 4366 (time 1781605167.00 total 80.12 last 42.96)
total: 4500 open/close in 82.63 seconds: 54.46 ops/second
 - /unlink 865 (time 1781605185.79 total 10.00 last 86.47)
 - /unlink 1904 (time 1781605195.80 total 20.02 last 103.77)
 - /unlink 2899 (time 1781605205.81 total 30.02 last 99.47)
 - /unlink 3820 (time 1781605215.82 total 40.03 last 91.98)
total: 4500 /unlink in 48.04 seconds: 93.67 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 476 (time 1781605241.38 total 10.00 last 47.59)
 - open/close 1040 (time 1781605251.39 total 20.01 last 56.34)
 - open/close 1584 (time 1781605261.39 total 30.01 last 54.40)
 - open/close 2106 (time 1781605271.40 total 40.02 last 52.15)
 - open/close 2613 (time 1781605281.41 total 50.04 last 50.62)
 - open/close 3150 (time 1781605291.42 total 60.05 last 53.64)
 - open/close 3560 (time 1781605301.45 total 70.07 last 40.91)
 - open/close 4001 (time 1781605311.46 total 80.09 last 44.03)
 - open/close 4480 (time 1781605321.47 total 90.09 last 47.87)
total: 4500 open/close in 90.42 seconds: 49.77 ops/second
 - /unlink 835 (time 1781605337.58 total 10.01 last 83.40)
 - /unlink 1712 (time 1781605347.59 total 20.02 last 87.64)
 - /unlink 2652 (time 1781605357.60 total 30.03 last 93.88)
 - /unlink 3588 (time 1781605367.60 total 40.03 last 93.58)
total: 4500 /unlink in 49.25 seconds: 91.37 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 421 (time 1781605395.59 total 10.00 last 42.09)
 - open/close 854 (time 1781605405.60 total 20.02 last 43.25)
 - open/close 1297 (time 1781605415.64 total 30.05 last 44.15)
 - open/close 1738 (time 1781605425.64 total 40.05 last 44.07)
 - open/close 2234 (time 1781605435.65 total 50.07 last 49.54)
 - open/close 2776 (time 1781605445.67 total 60.08 last 54.13)
 - open/close 3315 (time 1781605455.67 total 70.09 last 53.87)
 - open/close 3846 (time 1781605465.70 total 80.12 last 52.94)
 - open/close 4386 (time 1781605475.70 total 90.12 last 54.00)
total: 4500 open/close in 92.23 seconds: 48.79 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1064 (time 1781605493.68 total 10.00 last 106.37)
 - /unlink 1897 (time 1781605503.68 total 20.00 last 83.29)
 - /unlink 2833 (time 1781605513.70 total 30.02 last 93.47)
 - /unlink 3705 (time 1781605523.71 total 40.03 last 87.12)
total: 4500 /unlink in 49.82 seconds: 90.32 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 484 (time 1781605549.27 total 10.02 last 48.31)
 - open/close 1021 (time 1781605559.27 total 20.02 last 53.69)
 - open/close 1521 (time 1781605569.27 total 30.03 last 49.97)
 - open/close 1950 (time 1781605579.29 total 40.04 last 42.84)
 - open/close 2533 (time 1781605589.30 total 50.05 last 58.23)
 - open/close 3064 (time 1781605599.32 total 60.07 last 52.99)
 - open/close 3589 (time 1781605609.34 total 70.09 last 52.42)
 - open/close 4133 (time 1781605619.36 total 80.11 last 54.28)
total: 4500 open/close in 87.28 seconds: 51.56 ops/second
 - /unlink 947 (time 1781605641.42 total 10.00 last 94.69)
 - /unlink 1868 (time 1781605651.43 total 20.01 last 92.06)
 - /unlink 2799 (time 1781605661.43 total 30.01 last 93.08)
 - /unlink 3743 (time 1781605671.44 total 40.02 last 94.32)
total: 4500 /unlink in 49.19 seconds: 91.48 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 432 (time 1781605696.85 total 10.00 last 43.18)
 - open/close 985 (time 1781605706.86 total 20.01 last 55.29)
 - open/close 1514 (time 1781605716.86 total 30.01 last 52.90)
 - open/close 1985 (time 1781605726.87 total 40.02 last 47.04)
 - open/close 2460 (time 1781605736.88 total 50.03 last 47.47)
 - open/close 2881 (time 1781605746.89 total 60.04 last 42.02)
 - open/close 3303 (time 1781605756.92 total 70.07 last 42.11)
 - open/close 3756 (time 1781605766.92 total 80.07 last 45.29)
 - open/close 4204 (time 1781605776.93 total 90.09 last 44.72)
total: 4500 open/close in 96.24 seconds: 46.76 ops/second
 - /unlink 795 (time 1781605798.70 total 10.01 last 79.44)
 - /unlink 1761 (time 1781605808.70 total 20.01 last 96.60)
 - /unlink 2645 (time 1781605818.71 total 30.01 last 88.35)
 - /unlink 3538 (time 1781605828.71 total 40.02 last 89.28)
 - /unlink 4412 (time 1781605838.72 total 50.02 last 87.36)
total: 4500 /unlink in 51.12 seconds: 88.02 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 494 (time 1781605856.32 total 10.01 last 49.36)
 - open/close 1035 (time 1781605866.35 total 20.04 last 53.94)
 - open/close 1445 (time 1781605876.40 total 30.09 last 40.78)
 - open/close 1805 (time 1781605886.41 total 40.10 last 35.96)
 - open/close 2177 (time 1781605896.42 total 50.11 last 37.17)
 - open/close 2577 (time 1781605906.42 total 60.11 last 39.99)
 - open/close 3022 (time 1781605916.43 total 70.12 last 44.46)
 - open/close 3397 (time 1781605926.43 total 80.12 last 37.49)
 - open/close 3754 (time 1781605936.44 total 90.13 last 35.66)
 - open/close 4225 (time 1781605946.46 total 100.15 last 47.04)
total: 4500 open/close in 106.44 seconds: 42.28 ops/second
 - /unlink 783 (time 1781605969.52 total 10.00 last 78.29)
 - /unlink 1575 (time 1781605979.52 total 20.01 last 79.17)
 - /unlink 2453 (time 1781605989.53 total 30.02 last 87.70)
 - /unlink 3242 (time 1781605999.54 total 40.03 last 78.83)
 - /unlink 4081 (time 1781606009.55 total 50.04 last 83.79)
total: 4500 /unlink in 55.22 seconds: 81.49 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 372 (time 1781606030.22 total 10.00 last 37.20)
 - open/close 867 (time 1781606040.24 total 20.02 last 49.40)
 - open/close 1324 (time 1781606050.26 total 30.04 last 45.62)
 - open/close 1826 (time 1781606060.27 total 40.05 last 50.15)
 - open/close 2299 (time 1781606070.27 total 50.05 last 47.30)
 - open/close 2807 (time 1781606080.28 total 60.06 last 50.73)
 - open/close 3301 (time 1781606090.29 total 70.07 last 49.37)
 - open/close 3814 (time 1781606100.30 total 80.08 last 51.22)
 - open/close 4313 (time 1781606110.31 total 90.09 last 49.87)
total: 4500 open/close in 93.28 seconds: 48.24 ops/second
 - /unlink 1051 (time 1781606128.55 total 10.00 last 105.07)
 - /unlink 2140 (time 1781606138.55 total 20.01 last 108.87)
 - /unlink 3273 (time 1781606148.55 total 30.01 last 113.30)
 - /unlink 4344 (time 1781606158.56 total 40.02 last 106.98)
total: 4500 /unlink in 41.41 seconds: 108.66 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 631 (time 1781606175.09 total 10.01 last 63.06)
 - open/close 1204 (time 1781606185.09 total 20.01 last 57.30)
 - open/close 1872 (time 1781606195.11 total 30.03 last 66.67)
 - open/close 2514 (time 1781606205.11 total 40.03 last 64.19)
 - open/close 3133 (time 1781606215.12 total 50.04 last 61.79)
 - open/close 3738 (time 1781606225.14 total 60.06 last 60.42)
 - open/close 4366 (time 1781606235.17 total 70.09 last 62.61)
total: 4500 open/close in 72.48 seconds: 62.09 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1121 (time 1781606252.05 total 10.01 last 112.04)
 - /unlink 2364 (time 1781606262.05 total 20.01 last 124.27)
 - /unlink 3502 (time 1781606272.05 total 30.01 last 113.76)
total: 4500 /unlink in 37.71 seconds: 119.32 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 617 (time 1781606295.02 total 10.01 last 61.63)
 - open/close 1247 (time 1781606305.02 total 20.01 last 62.99)
 - open/close 1910 (time 1781606315.02 total 30.02 last 66.28)
 - open/close 2551 (time 1781606325.03 total 40.02 last 64.04)
 - open/close 3170 (time 1781606335.04 total 50.03 last 61.87)
 - open/close 3801 (time 1781606345.04 total 60.04 last 63.05)
 - open/close 4438 (time 1781606355.05 total 70.04 last 63.67)
total: 4500 open/close in 70.90 seconds: 63.47 ops/second
 - /unlink 1300 (time 1781606370.77 total 10.00 last 129.96)
 - /unlink 2373 (time 1781606380.78 total 20.01 last 107.24)
 - /unlink 3536 (time 1781606390.78 total 30.01 last 116.30)
total: 4500 /unlink in 38.91 seconds: 115.66 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 612 (time 1781606414.81 total 10.02 last 61.06)
 - open/close 1230 (time 1781606424.82 total 20.03 last 61.76)
 - open/close 1936 (time 1781606434.83 total 30.04 last 70.54)
 - open/close 2584 (time 1781606444.83 total 40.04 last 64.79)
 - open/close 3280 (time 1781606454.83 total 50.04 last 69.59)
 - open/close 3953 (time 1781606464.84 total 60.05 last 67.23)
total: 4500 open/close in 67.95 seconds: 66.22 ops/second
 - /unlink 1189 (time 1781606486.92 total 10.01 last 118.84)
 - /unlink 2457 (time 1781606496.93 total 20.01 last 126.73)
 - /unlink 3694 (time 1781606506.93 total 30.01 last 123.65)
total: 4500 /unlink in 36.78 seconds: 122.35 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11225: 95133 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg649-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg649-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg649-server
stop mds service on oleg649-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg649-server
stop mds service on oleg649-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg649-server

unloading modules via unload_modules_local on: 'oleg649-server'
oleg649-server: oleg649-server.virtnet: executing unload_modules_local
oleg649-server: 
modules unloaded.