== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 05:40:29 (1774518029) Stopping clients: oleg154-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg154-client.virtnet /mnt/lustre2 (opts:-f) oleg154-server: oleg154-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg154-server: detected 4 online CPUs by sysfs oleg154-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg154-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg154-server: oleg154-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg154-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg154-server: oleg154-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16166 358 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 531 (time 1774518174.79 total 10.02 last 52.99) - open/close 912 (time 1774518184.80 total 20.03 last 38.06) - open/close 1515 (time 1774518194.82 total 30.05 last 60.21) - open/close 2243 (time 1774518204.83 total 40.06 last 72.70) - open/close 2886 (time 1774518214.84 total 50.07 last 64.24) - open/close 3575 (time 1774518224.84 total 60.07 last 68.90) - open/close 4210 (time 1774518234.86 total 70.08 last 63.41) - open/close 5082 (time 1774518244.86 total 80.09 last 87.14) - open/close 5804 (time 1774518254.87 total 90.09 last 72.16) - open/close 6339 (time 1774518264.88 total 100.11 last 53.44) - open/close 6832 (time 1774518274.88 total 110.11 last 49.30) - open/close 7448 (time 1774518284.90 total 120.13 last 61.45) - open/close 7952 (time 1774518294.92 total 130.15 last 50.30) - open/close 8494 (time 1774518304.93 total 140.15 last 54.17) - open/close 9202 (time 1774518314.95 total 150.17 last 70.67) - open/close 9846 (time 1774518324.95 total 160.18 last 64.36) - open/close 10000 (time 1774518327.26 total 162.49 last 66.63) - open/close 10730 (time 1774518337.27 total 172.50 last 72.94) - open/close 11421 (time 1774518347.29 total 182.51 last 68.99) - open/close 12011 (time 1774518357.32 total 192.55 last 58.79) - open/close 12702 (time 1774518367.33 total 202.56 last 69.05) - open/close 13351 (time 1774518377.34 total 212.57 last 64.83) - open/close 13977 (time 1774518387.35 total 222.58 last 62.53) - open/close 14691 (time 1774518397.38 total 232.60 last 71.24) - open/close 15314 (time 1774518407.38 total 242.60 last 62.29) - open/close 15902 (time 1774518417.38 total 252.61 last 58.78) - open/close 16509 (time 1774518427.39 total 262.62 last 60.64) - open/close 17132 (time 1774518437.41 total 272.63 last 62.19) - open/close 17612 (time 1774518447.41 total 282.64 last 47.99) - open/close 18109 (time 1774518457.42 total 292.65 last 49.65) - open/close 18761 (time 1774518467.43 total 302.65 last 65.15) - open/close 19528 (time 1774518477.43 total 312.66 last 76.69) - open/close 20000 (time 1774518484.15 total 319.38 last 70.20) - open/close 20635 (time 1774518494.16 total 329.39 last 63.45) - open/close 21296 (time 1774518504.16 total 339.39 last 66.08) - open/close 22074 (time 1774518514.19 total 349.41 last 77.62) - open/close 22806 (time 1774518524.21 total 359.43 last 73.06) - open/close 23568 (time 1774518534.22 total 369.45 last 76.08) - open/close 24313 (time 1774518544.23 total 379.46 last 74.40) - open/close 25089 (time 1774518554.24 total 389.46 last 77.57) - open/close 25807 (time 1774518564.27 total 399.49 last 71.60) - open/close 26527 (time 1774518574.28 total 409.51 last 71.89) - open/close 27213 (time 1774518584.29 total 419.52 last 68.54) - open/close 27944 (time 1774518594.29 total 429.52 last 73.07) - open/close 28670 (time 1774518604.30 total 439.53 last 72.56) - open/close 29520 (time 1774518614.30 total 449.53 last 84.96) - open/close 30000 (time 1774518619.32 total 454.55 last 95.65) - open/close 30764 (time 1774518629.33 total 464.56 last 76.34) - open/close 31525 (time 1774518639.34 total 474.57 last 76.01) - open/close 32368 (time 1774518649.34 total 484.57 last 84.30) - open/close 33283 (time 1774518659.35 total 494.57 last 91.47) - open/close 34205 (time 1774518669.35 total 504.58 last 92.15) - open/close 35078 (time 1774518679.35 total 514.58 last 87.27) - open/close 36002 (time 1774518689.36 total 524.59 last 92.34) - open/close 36948 (time 1774518699.36 total 534.59 last 94.57) - open/close 37912 (time 1774518709.37 total 544.60 last 96.34) - open/close 38847 (time 1774518719.37 total 554.60 last 93.50) - open/close 39904 (time 1774518729.37 total 564.60 last 105.66) - open/close 40873 (time 1774518739.38 total 574.61 last 96.82) - open/close 41778 (time 1774518749.39 total 584.61 last 90.45) - open/close 42940 (time 1774518759.40 total 594.62 last 116.09) - open/close 44030 (time 1774518769.40 total 604.63 last 108.99) - open/close 44987 (time 1774518779.40 total 614.63 last 95.65) - open/close 45947 (time 1774518789.42 total 624.65 last 95.85) - open/close 46900 (time 1774518799.42 total 634.65 last 95.29) - open/close 47845 (time 1774518809.43 total 644.65 last 94.44) - open/close 48816 (time 1774518819.43 total 654.66 last 97.04) - open/close 49819 (time 1774518829.44 total 664.67 last 100.21) - open/close 50822 (time 1774518839.45 total 674.67 last 100.27) - open/close 51974 (time 1774518849.45 total 684.68 last 115.11) - open/close 52998 (time 1774518859.45 total 694.68 last 102.40) - open/close 53908 (time 1774518869.46 total 704.69 last 90.94) - open/close 54872 (time 1774518879.46 total 714.69 last 96.39) - open/close 55948 (time 1774518889.46 total 724.69 last 107.59) - open/close 57068 (time 1774518899.47 total 734.69 last 111.98) - open/close 58263 (time 1774518909.47 total 744.70 last 119.46) - open/close 59265 (time 1774518919.47 total 754.70 last 100.15) - open/close 60000 (time 1774518927.07 total 762.29 last 96.80) - open/close 61084 (time 1774518937.07 total 772.30 last 108.38) - open/close 62099 (time 1774518947.07 total 782.30 last 101.49) - open/close 63194 (time 1774518957.08 total 792.31 last 109.36) - open/close 64268 (time 1774518967.09 total 802.32 last 107.29) total: 64768 open/close in 807.04 seconds: 80.25 ops/second - /unlink 1799 (time 1774518984.41 total 10.00 last 179.87) - /unlink 3594 (time 1774518994.41 total 20.01 last 179.42) - /unlink 5250 (time 1774519004.41 total 30.01 last 165.57) - /unlink 7245 (time 1774519014.42 total 40.01 last 199.43) - /unlink 9287 (time 1774519024.42 total 50.02 last 204.12) - /unlink 10000 (time 1774519027.98 total 53.57 last 200.35) - /unlink 12017 (time 1774519037.98 total 63.58 last 201.68) - /unlink 13926 (time 1774519047.99 total 73.58 last 190.82) - /unlink 15761 (time 1774519057.99 total 83.58 last 183.47) - /unlink 17639 (time 1774519067.99 total 93.59 last 187.69) - /unlink 19826 (time 1774519077.99 total 103.59 last 218.70) - /unlink 22047 (time 1774519088.00 total 113.59 last 222.06) - /unlink 24808 (time 1774519098.00 total 123.59 last 275.99) - /unlink 27035 (time 1774519108.01 total 133.60 last 222.57) - /unlink 29469 (time 1774519118.01 total 143.60 last 243.27) - /unlink 30000 (time 1774519120.49 total 146.08 last 214.19) - /unlink 32244 (time 1774519130.49 total 156.08 last 224.37) - /unlink 34722 (time 1774519140.49 total 166.09 last 247.72) - /unlink 37238 (time 1774519150.50 total 176.09 last 251.56) - /unlink 39235 (time 1774519160.50 total 186.09 last 199.65) - /unlink 40000 (time 1774519165.03 total 190.63 last 168.73) - /unlink 41948 (time 1774519175.04 total 200.63 last 194.72) - /unlink 43970 (time 1774519185.04 total 210.63 last 202.17) - /unlink 46218 (time 1774519195.04 total 220.63 last 224.78) - /unlink 48397 (time 1774519205.04 total 230.63 last 217.86) - /unlink 50000 (time 1774519211.84 total 237.43 last 235.82) - /unlink 52235 (time 1774519221.84 total 247.43 last 223.45) - /unlink 54641 (time 1774519231.84 total 257.44 last 240.54) - /unlink 57269 (time 1774519241.84 total 267.44 last 262.78) - /unlink 59767 (time 1774519251.85 total 277.44 last 249.71) - /unlink 62186 (time 1774519261.85 total 287.44 last 241.85) - /unlink 64567 (time 1774519271.85 total 297.45 last 237.98) total: 64768 /unlink in 298.50 seconds: 216.98 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1437 (time 1774519303.79 total 10.01 last 143.59) - open/close 2802 (time 1774519313.79 total 20.01 last 136.50) - open/close 4070 (time 1774519323.79 total 30.01 last 126.77) - open/close 5154 (time 1774519333.80 total 40.01 last 108.37) - open/close 6243 (time 1774519343.80 total 50.02 last 108.86) - open/close 7507 (time 1774519353.81 total 60.02 last 126.32) - open/close 8825 (time 1774519363.81 total 70.03 last 131.78) - open/close 10000 (time 1774519373.21 total 79.42 last 125.02) - open/close 11290 (time 1774519383.21 total 89.43 last 128.95) - open/close 12526 (time 1774519393.22 total 99.43 last 123.53) - open/close 13916 (time 1774519403.22 total 109.44 last 138.95) - open/close 15256 (time 1774519413.22 total 119.44 last 134.00) - open/close 16803 (time 1774519423.22 total 129.44 last 154.69) - open/close 18312 (time 1774519433.23 total 139.44 last 150.84) - open/close 19731 (time 1774519443.23 total 149.44 last 141.89) - open/close 20000 (time 1774519445.28 total 151.50 last 130.76) - open/close 21319 (time 1774519455.29 total 161.50 last 131.88) - open/close 22897 (time 1774519465.29 total 171.51 last 157.74) - open/close 24337 (time 1774519475.30 total 181.52 last 143.78) - open/close 25639 (time 1774519485.31 total 191.52 last 130.19) - open/close 27030 (time 1774519495.31 total 201.52 last 139.10) - open/close 28661 (time 1774519505.31 total 211.52 last 163.06) - open/close 30000 (time 1774519514.21 total 220.43 last 150.41) - open/close 31563 (time 1774519524.21 total 230.43 last 156.29) - open/close 33147 (time 1774519534.21 total 240.43 last 158.37) - open/close 34642 (time 1774519544.22 total 250.44 last 149.36) - open/close 35865 (time 1774519554.22 total 260.44 last 122.30) - open/close 37113 (time 1774519564.23 total 270.45 last 124.72) - open/close 38364 (time 1774519574.24 total 280.45 last 125.02) - open/close 39648 (time 1774519584.24 total 290.46 last 128.35) - open/close 40000 (time 1774519586.94 total 293.15 last 130.46) - open/close 41274 (time 1774519596.95 total 303.16 last 127.28) - open/close 42532 (time 1774519606.95 total 313.17 last 125.76) - open/close 43906 (time 1774519616.95 total 323.17 last 137.38) - open/close 45614 (time 1774519626.95 total 333.17 last 170.76) - open/close 46874 (time 1774519636.95 total 343.17 last 126.00) - open/close 48257 (time 1774519646.96 total 353.17 last 138.28) - open/close 49997 (time 1774519656.96 total 363.17 last 173.96) - open/close 51827 (time 1774519666.96 total 373.17 last 183.00) - open/close 53322 (time 1774519676.96 total 383.18 last 149.44) - open/close 55358 (time 1774519686.96 total 393.18 last 203.58) - open/close 57009 (time 1774519696.97 total 403.18 last 165.04) - open/close 58598 (time 1774519706.97 total 413.18 last 158.87) - open/close 60000 (time 1774519715.04 total 421.26 last 173.66) - open/close 62312 (time 1774519725.04 total 431.26 last 231.18) - open/close 64642 (time 1774519735.04 total 441.26 last 232.98) total: 64768 open/close in 441.91 seconds: 146.56 ops/second - /unlink 3611 (time 1774519747.52 total 10.00 last 361.08) - /unlink 6568 (time 1774519757.52 total 20.00 last 295.69) - /unlink 9646 (time 1774519767.52 total 30.00 last 307.78) - /unlink 13032 (time 1774519777.52 total 40.00 last 338.57) - /unlink 16534 (time 1774519787.52 total 50.00 last 350.15) - /unlink 20000 (time 1774519795.96 total 58.44 last 410.89) - /unlink 24492 (time 1774519805.96 total 68.44 last 449.19) - /unlink 27524 (time 1774519815.96 total 78.44 last 303.13) - /unlink 30000 (time 1774519824.44 total 86.92 last 292.00) - /unlink 32918 (time 1774519834.44 total 96.92 last 291.74) - /unlink 35990 (time 1774519844.44 total 106.93 last 307.10) - /unlink 38765 (time 1774519854.45 total 116.93 last 277.45) - /unlink 40000 (time 1774519858.30 total 120.78 last 320.33) - /unlink 43702 (time 1774519868.30 total 130.79 last 370.15) - /unlink 47136 (time 1774519878.30 total 140.79 last 343.36) - /unlink 50000 (time 1774519888.30 total 150.78 last 286.48) - /unlink 53328 (time 1774519898.30 total 160.78 last 332.80) - /unlink 56534 (time 1774519908.30 total 170.79 last 320.54) - /unlink 59917 (time 1774519918.30 total 180.79 last 338.26) - /unlink 62986 (time 1774519928.30 total 190.79 last 306.89) total: 64768 /unlink in 195.95 seconds: 330.54 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1993 (time 1774519965.06 total 10.00 last 199.26) - open/close 3751 (time 1774519975.06 total 20.00 last 175.78) - open/close 5755 (time 1774519985.07 total 30.01 last 200.33) - open/close 7998 (time 1774519995.07 total 40.01 last 224.22) - open/close 9639 (time 1774520005.07 total 50.01 last 164.05) - open/close 11374 (time 1774520015.07 total 60.01 last 173.49) - open/close 12942 (time 1774520025.08 total 70.02 last 156.76) - open/close 15079 (time 1774520035.08 total 80.02 last 213.69) - open/close 17240 (time 1774520045.08 total 90.02 last 216.06) - open/close 19690 (time 1774520055.08 total 100.02 last 244.84) - open/close 22462 (time 1774520065.09 total 110.03 last 277.01) - open/close 24820 (time 1774520075.09 total 120.03 last 235.75) - open/close 27270 (time 1774520085.09 total 130.03 last 244.97) - open/close 30000 (time 1774520094.78 total 139.72 last 281.92) - open/close 33284 (time 1774520104.78 total 149.72 last 328.35) - open/close 36928 (time 1774520114.78 total 159.72 last 364.37) - open/close 40000 (time 1774520122.93 total 167.88 last 376.71) - open/close 42425 (time 1774520132.94 total 177.88 last 242.40) - open/close 44710 (time 1774520142.94 total 187.88 last 228.43) - open/close 46785 (time 1774520152.94 total 197.88 last 207.46) - open/close 48967 (time 1774520162.94 total 207.89 last 218.19) - open/close 50000 (time 1774520167.52 total 212.46 last 225.86) - open/close 52604 (time 1774520177.52 total 222.46 last 260.33) - open/close 55906 (time 1774520187.52 total 232.46 last 330.16) - open/close 59164 (time 1774520197.52 total 242.46 last 325.80) - open/close 60000 (time 1774520199.63 total 244.57 last 396.17) - open/close 63282 (time 1774520209.64 total 254.58 last 328.04) total: 64768 open/close in 259.85 seconds: 249.26 ops/second - /unlink 5610 (time 1774520226.42 total 10.00 last 560.91) - /unlink 10000 (time 1774520235.13 total 18.71 last 504.29) - /unlink 14750 (time 1774520245.13 total 28.71 last 474.91) - /unlink 19230 (time 1774520255.13 total 38.71 last 447.96) - /unlink 24192 (time 1774520265.13 total 48.71 last 496.13) - /unlink 29271 (time 1774520275.13 total 58.71 last 507.84) - /unlink 34755 (time 1774520285.14 total 68.71 last 548.31) - /unlink 40000 (time 1774520294.53 total 78.11 last 558.47) - /unlink 45247 (time 1774520304.53 total 88.11 last 524.66) - /unlink 50000 (time 1774520311.35 total 94.93 last 696.35) - /unlink 56106 (time 1774520321.35 total 104.93 last 610.56) - /unlink 60000 (time 1774520327.02 total 110.60 last 686.98) total: 64768 /unlink in 119.19 seconds: 543.40 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4475 (time 1774520363.13 total 10.00 last 447.50) - open/close 6822 (time 1774520373.13 total 20.00 last 234.66) - open/close 9390 (time 1774520383.14 total 30.01 last 256.58) - open/close 10000 (time 1774520386.01 total 32.88 last 212.29) - open/close 12339 (time 1774520396.02 total 42.89 last 233.86) - open/close 14998 (time 1774520406.02 total 52.89 last 265.89) - open/close 17927 (time 1774520416.02 total 62.89 last 292.85) - open/close 20000 (time 1774520420.45 total 67.32 last 467.54) - open/close 24758 (time 1774520430.45 total 77.32 last 475.78) - open/close 29634 (time 1774520440.45 total 87.32 last 487.60) - open/close 34222 (time 1774520450.45 total 97.32 last 458.77) - open/close 39001 (time 1774520460.45 total 107.32 last 477.85) - open/close 40000 (time 1774520462.68 total 109.55 last 448.61) - open/close 44356 (time 1774520472.68 total 119.55 last 435.57) - open/close 48490 (time 1774520482.68 total 129.55 last 413.37) - open/close 50000 (time 1774520485.71 total 132.58 last 498.17) - open/close 54649 (time 1774520495.72 total 142.58 last 464.82) - open/close 59327 (time 1774520505.72 total 152.59 last 467.74) - open/close 63847 (time 1774520515.72 total 162.59 last 451.99) total: 64768 open/close in 164.51 seconds: 393.71 ops/second - /unlink 7165 (time 1774520528.96 total 10.00 last 716.48) - /unlink 10000 (time 1774520532.54 total 13.57 last 793.22) - /unlink 17269 (time 1774520542.54 total 23.57 last 726.86) - /unlink 20000 (time 1774520546.28 total 27.32 last 729.50) - /unlink 27938 (time 1774520556.28 total 37.32 last 793.76) - /unlink 30000 (time 1774520558.60 total 39.64 last 888.11) - /unlink 37186 (time 1774520568.61 total 49.64 last 718.58) - /unlink 40000 (time 1774520573.23 total 54.27 last 607.90) - /unlink 46930 (time 1774520583.23 total 64.27 last 693.00) - /unlink 50000 (time 1774520587.57 total 68.61 last 708.04) - /unlink 57524 (time 1774520597.57 total 78.61 last 752.36) - /unlink 60000 (time 1774520600.87 total 81.91 last 750.18) total: 64768 /unlink in 87.40 seconds: 741.02 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4881 (time 1774520634.79 total 10.00 last 488.02) - open/close 9893 (time 1774520644.79 total 20.00 last 501.18) - open/close 14745 (time 1774520654.79 total 30.00 last 485.11) - open/close 19769 (time 1774520664.79 total 40.01 last 502.30) - open/close 24830 (time 1774520674.79 total 50.01 last 506.03) - open/close 29989 (time 1774520684.80 total 60.01 last 515.62) - open/close 35026 (time 1774520694.80 total 70.01 last 503.64) - open/close 40000 (time 1774520704.50 total 79.71 last 512.73) - open/close 45161 (time 1774520714.50 total 89.72 last 516.09) - open/close 50000 (time 1774520724.40 total 99.61 last 489.02) - open/close 54344 (time 1774520734.40 total 109.61 last 434.33) - open/close 58997 (time 1774520744.40 total 119.61 last 465.26) - open/close 64122 (time 1774520754.40 total 129.61 last 512.43) total: 64768 open/close in 131.01 seconds: 494.36 ops/second - /unlink 7913 (time 1774520767.12 total 10.00 last 791.25) - /unlink 10000 (time 1774520769.58 total 12.46 last 847.07) - /unlink 17680 (time 1774520779.58 total 22.46 last 767.98) - /unlink 20000 (time 1774520782.88 total 25.76 last 703.51) - /unlink 26874 (time 1774520792.88 total 35.77 last 687.20) - /unlink 30000 (time 1774520797.53 total 40.42 last 672.07) - /unlink 37095 (time 1774520807.53 total 50.42 last 709.43) - /unlink 40000 (time 1774520810.87 total 53.75 last 871.15) - /unlink 48062 (time 1774520820.87 total 63.75 last 806.15) - /unlink 50000 (time 1774520823.61 total 66.50 last 706.71) - /unlink 58164 (time 1774520833.61 total 76.50 last 816.35) total: 64768 /unlink in 84.05 seconds: 770.57 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg154-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg154-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg154-server stop mds service on oleg154-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg154-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing unload_modules_local oleg154-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg154-server: LNET ready to unload oleg154-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg154-client: oleg154-client: ssh exited with exit code 2 pdsh@oleg154-client: oleg154-server: ssh exited with exit code 2 pdsh@oleg154-client: oleg154-client: ssh exited with exit code 2 pdsh@oleg154-client: oleg154-server: ssh exited with exit code 2