== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 16:28:22 (1764106102) Stopping clients: oleg302-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg302-client.virtnet /mnt/lustre2 (opts:-f) oleg302-server: oleg302-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg302-server' oleg302-server: oleg302-server.virtnet: executing load_modules_local oleg302-server: Loading modules from /home/green/git/lustre-release/lustre oleg302-server: detected 4 online CPUs by sysfs oleg302-server: MODOPTS_LIBCFS= oleg302-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg302-server: Force libcfs to create 2 CPU partitions oleg302-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg302-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg302-server: oleg302-server.virtnet: executing set_default_debug -1 all pdsh@oleg302-client: oleg302-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg302-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg302-server: oleg302-server.virtnet: executing set_default_debug -1 all pdsh@oleg302-client: oleg302-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg302-client.virtnet: -o user_xattr,flock 192.168.203.102@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32648 392 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1211 (time 1764106170.10 total 10.00 last 121.06) - open/close 2384 (time 1764106180.10 total 20.01 last 117.21) - open/close 3785 (time 1764106190.11 total 30.01 last 140.05) - open/close 5045 (time 1764106200.11 total 40.02 last 125.98) - open/close 6240 (time 1764106210.11 total 50.02 last 119.44) - open/close 7557 (time 1764106220.12 total 60.03 last 131.65) - open/close 9045 (time 1764106230.12 total 70.03 last 148.73) - open/close 10000 (time 1764106236.47 total 76.38 last 150.41) - open/close 11782 (time 1764106246.48 total 86.39 last 178.07) - open/close 13147 (time 1764106256.48 total 96.39 last 136.44) - open/close 14383 (time 1764106266.49 total 106.39 last 123.56) - open/close 15836 (time 1764106276.50 total 116.40 last 145.17) - open/close 17555 (time 1764106286.50 total 126.40 last 171.88) - open/close 19068 (time 1764106296.50 total 136.41 last 151.22) - open/close 20000 (time 1764106302.89 total 142.80 last 145.86) - open/close 21628 (time 1764106312.90 total 152.81 last 162.68) - open/close 22971 (time 1764106322.90 total 162.81 last 134.28) - open/close 24671 (time 1764106332.90 total 172.81 last 169.96) - open/close 26778 (time 1764106342.91 total 182.82 last 210.54) - open/close 28712 (time 1764106352.91 total 192.82 last 193.34) - open/close 30000 (time 1764106359.33 total 199.24 last 200.69) - open/close 32615 (time 1764106369.33 total 209.24 last 261.43) - open/close 34500 (time 1764106379.34 total 219.25 last 188.39) - open/close 36668 (time 1764106389.34 total 229.25 last 216.77) - open/close 38742 (time 1764106399.34 total 239.25 last 207.37) - open/close 40000 (time 1764106406.28 total 246.19 last 181.33) - open/close 42074 (time 1764106416.28 total 256.19 last 207.32) - open/close 43538 (time 1764106426.29 total 266.19 last 146.38) - open/close 44909 (time 1764106436.29 total 276.20 last 137.01) - open/close 46721 (time 1764106446.29 total 286.20 last 181.16) - open/close 48647 (time 1764106456.30 total 296.21 last 192.54) - open/close 50000 (time 1764106462.69 total 302.60 last 211.50) - open/close 52476 (time 1764106472.70 total 312.61 last 247.50) - open/close 54653 (time 1764106482.70 total 322.61 last 217.68) - open/close 56653 (time 1764106492.70 total 332.61 last 199.97) - open/close 58901 (time 1764106502.70 total 342.61 last 224.75) - open/close 60000 (time 1764106507.90 total 347.81 last 211.36) - open/close 62105 (time 1764106517.90 total 357.81 last 210.50) - open/close 64452 (time 1764106527.90 total 367.81 last 234.68) total: 64768 open/close in 369.00 seconds: 175.52 ops/second - /unlink 3649 (time 1764106540.92 total 10.00 last 364.82) - /unlink 6282 (time 1764106550.93 total 20.01 last 263.12) - /unlink 9272 (time 1764106560.93 total 30.01 last 298.84) - /unlink 10000 (time 1764106563.47 total 32.55 last 286.94) - /unlink 12649 (time 1764106573.47 total 42.55 last 264.85) - /unlink 15618 (time 1764106583.47 total 52.55 last 296.85) - /unlink 18805 (time 1764106593.48 total 62.56 last 318.62) - /unlink 20000 (time 1764106596.81 total 65.89 last 358.42) - /unlink 22701 (time 1764106606.86 total 75.94 last 268.67) - /unlink 26484 (time 1764106616.87 total 85.95 last 378.26) - /unlink 30000 (time 1764106626.05 total 95.13 last 382.67) - /unlink 34730 (time 1764106636.06 total 105.14 last 472.92) - /unlink 39313 (time 1764106646.06 total 115.14 last 458.27) - /unlink 44601 (time 1764106656.06 total 125.14 last 528.74) - /unlink 50000 (time 1764106665.72 total 134.80 last 558.85) - /unlink 55200 (time 1764106675.72 total 144.80 last 519.95) - /unlink 60000 (time 1764106684.67 total 153.75 last 536.04) total: 64768 /unlink in 161.49 seconds: 401.05 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3391 (time 1764106724.31 total 10.00 last 339.02) - open/close 6813 (time 1764106734.31 total 20.00 last 342.14) - open/close 10000 (time 1764106742.48 total 28.18 last 389.82) - open/close 13958 (time 1764106752.48 total 38.18 last 395.78) - open/close 16688 (time 1764106762.48 total 48.18 last 273.00) - open/close 19843 (time 1764106772.49 total 58.18 last 315.45) - open/close 23260 (time 1764106782.49 total 68.19 last 341.50) - open/close 26550 (time 1764106792.49 total 78.19 last 328.92) - open/close 29674 (time 1764106802.50 total 88.19 last 312.34) - open/close 32234 (time 1764106812.50 total 98.19 last 255.92) - open/close 33970 (time 1764106822.50 total 108.20 last 173.53) - open/close 36195 (time 1764106832.50 total 118.20 last 222.47) - open/close 38555 (time 1764106842.51 total 128.20 last 235.90) - open/close 40000 (time 1764106848.82 total 134.52 last 228.76) - open/close 42403 (time 1764106858.83 total 144.53 last 240.20) - open/close 44871 (time 1764106868.83 total 154.53 last 246.70) - open/close 46979 (time 1764106878.84 total 164.53 last 210.72) - open/close 49212 (time 1764106888.84 total 174.53 last 223.29) - open/close 50000 (time 1764106891.59 total 177.29 last 286.33) - open/close 52754 (time 1764106901.59 total 187.29 last 275.34) - open/close 55677 (time 1764106911.59 total 197.29 last 292.22) - open/close 58389 (time 1764106921.63 total 207.32 last 270.33) - open/close 59336 (time 1764106931.64 total 217.33 last 94.60) - open/close 60000 (time 1764106940.12 total 225.82 last 78.27) - open/close 60630 (time 1764106950.13 total 235.83 last 62.94) - open/close 61324 (time 1764106960.13 total 245.83 last 69.36) - open/close 61988 (time 1764106970.14 total 255.84 last 66.35) - open/close 62698 (time 1764106980.15 total 265.85 last 70.91) - open/close 63239 (time 1764106990.16 total 275.85 last 54.08) - open/close 63673 (time 1764107000.16 total 285.86 last 43.39) - open/close 64107 (time 1764107010.19 total 295.89 last 43.28) - open/close 64674 (time 1764107020.20 total 305.89 last 56.65) total: 64768 open/close in 307.67 seconds: 210.51 ops/second - /unlink 1251 (time 1764107036.48 total 10.00 last 125.05) - /unlink 2642 (time 1764107046.48 total 20.00 last 139.09) - /unlink 4050 (time 1764107056.49 total 30.01 last 140.73) - /unlink 5379 (time 1764107066.50 total 40.02 last 132.79) - /unlink 6712 (time 1764107076.50 total 50.03 last 133.20) - /unlink 8178 (time 1764107086.50 total 60.03 last 146.58) - /unlink 9564 (time 1764107096.51 total 70.03 last 138.57) - /unlink 10000 (time 1764107099.48 total 73.00 last 146.65) - /unlink 11449 (time 1764107109.48 total 83.01 last 144.85) - /unlink 12834 (time 1764107119.49 total 93.01 last 138.44) - /unlink 14270 (time 1764107129.49 total 103.01 last 143.53) - /unlink 15843 (time 1764107139.49 total 113.02 last 157.26) - /unlink 17556 (time 1764107149.50 total 123.02 last 171.20) - /unlink 18983 (time 1764107159.50 total 133.02 last 142.68) - /unlink 20000 (time 1764107165.13 total 138.65 last 180.82) - /unlink 21781 (time 1764107175.13 total 148.65 last 178.03) - /unlink 23155 (time 1764107185.14 total 158.66 last 137.30) - /unlink 24394 (time 1764107195.14 total 168.66 last 123.89) - /unlink 25679 (time 1764107205.14 total 178.66 last 128.47) - /unlink 27206 (time 1764107215.14 total 188.67 last 152.66) - /unlink 28700 (time 1764107225.14 total 198.67 last 149.40) - /unlink 30000 (time 1764107233.23 total 206.75 last 160.70) - /unlink 32014 (time 1764107243.23 total 216.76 last 201.37) - /unlink 33734 (time 1764107253.24 total 226.76 last 171.96) - /unlink 35298 (time 1764107263.24 total 236.76 last 156.37) - /unlink 38058 (time 1764107273.24 total 246.76 last 275.95) - /unlink 40000 (time 1764107279.73 total 253.26 last 299.01) - /unlink 43185 (time 1764107289.74 total 263.26 last 318.45) - /unlink 48136 (time 1764107299.74 total 273.26 last 494.99) - /unlink 50000 (time 1764107303.44 total 276.96 last 503.29) - /unlink 54903 (time 1764107313.44 total 286.97 last 490.29) - /unlink 59612 (time 1764107323.44 total 296.97 last 470.85) - /unlink 64563 (time 1764107333.44 total 306.97 last 495.09) total: 64768 /unlink in 307.38 seconds: 210.71 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3839 (time 1764107361.68 total 10.00 last 383.87) - open/close 7993 (time 1764107371.68 total 20.00 last 415.36) - open/close 10000 (time 1764107375.77 total 24.09 last 490.84) - open/close 14383 (time 1764107385.77 total 34.09 last 438.18) - open/close 18587 (time 1764107395.77 total 44.09 last 420.36) - open/close 20000 (time 1764107399.11 total 47.43 last 423.23) - open/close 24413 (time 1764107409.11 total 57.43 last 441.29) - open/close 28983 (time 1764107419.11 total 67.43 last 456.96) - open/close 30000 (time 1764107421.43 total 69.75 last 438.83) - open/close 34224 (time 1764107431.43 total 79.75 last 422.38) - open/close 38711 (time 1764107441.43 total 89.75 last 448.66) - open/close 40000 (time 1764107444.12 total 92.44 last 479.93) - open/close 43727 (time 1764107454.12 total 102.44 last 372.65) - open/close 47278 (time 1764107464.12 total 112.44 last 355.04) - open/close 50000 (time 1764107472.03 total 120.35 last 344.08) - open/close 53774 (time 1764107482.03 total 130.35 last 377.37) - open/close 58116 (time 1764107492.06 total 140.38 last 433.24) - open/close 60000 (time 1764107496.69 total 145.01 last 406.24) - open/close 64516 (time 1764107506.69 total 155.01 last 451.58) total: 64768 open/close in 155.84 seconds: 415.60 ops/second - /unlink 5904 (time 1764107519.03 total 10.00 last 590.35) - /unlink 10000 (time 1764107525.26 total 16.24 last 656.90) - /unlink 15897 (time 1764107535.26 total 26.24 last 589.66) - /unlink 20000 (time 1764107542.07 total 33.04 last 603.02) - /unlink 27069 (time 1764107552.07 total 43.04 last 706.85) - /unlink 30000 (time 1764107556.57 total 47.55 last 650.48) - /unlink 37027 (time 1764107566.57 total 57.55 last 702.64) - /unlink 40000 (time 1764107570.93 total 61.90 last 682.75) - /unlink 46666 (time 1764107580.93 total 71.90 last 666.55) - /unlink 50000 (time 1764107585.36 total 76.34 last 751.65) - /unlink 58894 (time 1764107595.36 total 86.34 last 889.35) total: 64768 /unlink in 93.56 seconds: 692.26 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4290 (time 1764107639.78 total 10.00 last 428.95) - open/close 8880 (time 1764107649.78 total 20.00 last 458.94) - open/close 10000 (time 1764107652.26 total 22.48 last 452.15) - open/close 14415 (time 1764107662.26 total 32.48 last 441.48) - open/close 18517 (time 1764107672.26 total 42.48 last 410.20) - open/close 20000 (time 1764107676.02 total 46.24 last 394.66) - open/close 24340 (time 1764107686.02 total 56.24 last 433.92) - open/close 28833 (time 1764107696.02 total 66.24 last 449.20) - open/close 30000 (time 1764107698.54 total 68.76 last 464.00) - open/close 34481 (time 1764107708.54 total 78.76 last 447.97) - open/close 38969 (time 1764107718.54 total 88.76 last 448.75) - open/close 43769 (time 1764107728.54 total 98.76 last 479.98) - open/close 48124 (time 1764107738.54 total 108.76 last 435.43) - open/close 50000 (time 1764107742.19 total 112.41 last 514.60) - open/close 54307 (time 1764107752.19 total 122.41 last 430.64) - open/close 55440 (time 1764107762.20 total 132.41 last 113.25) - open/close 56342 (time 1764107772.21 total 142.43 last 90.10) - open/close 57198 (time 1764107782.22 total 152.44 last 85.49) - open/close 57691 (time 1764107792.23 total 162.45 last 49.24) - open/close 58473 (time 1764107802.23 total 172.45 last 78.18) - open/close 59304 (time 1764107812.24 total 182.46 last 83.01) - open/close 60000 (time 1764107817.11 total 187.32 last 143.16) - open/close 61907 (time 1764107827.11 total 197.33 last 190.66) - open/close 64403 (time 1764107837.11 total 207.33 last 249.54) total: 64768 open/close in 208.65 seconds: 310.41 ops/second - /unlink 4461 (time 1764107850.09 total 10.00 last 446.03) - /unlink 7278 (time 1764107860.09 total 20.00 last 281.66) - /unlink 10000 (time 1764107869.24 total 29.16 last 297.28) - /unlink 13599 (time 1764107879.25 total 39.16 last 359.83) - /unlink 15634 (time 1764107889.25 total 49.16 last 203.44) - /unlink 16878 (time 1764107899.25 total 59.17 last 124.36) - /unlink 18168 (time 1764107909.25 total 69.17 last 128.97) - /unlink 19405 (time 1764107919.26 total 79.17 last 123.66) - /unlink 20000 (time 1764107923.48 total 83.40 last 140.81) - /unlink 21340 (time 1764107933.49 total 93.41 last 133.89) - /unlink 23914 (time 1764107943.49 total 103.41 last 257.35) - /unlink 26376 (time 1764107953.50 total 113.41 last 246.08) - /unlink 28997 (time 1764107963.50 total 123.41 last 262.07) - /unlink 30000 (time 1764107967.49 total 127.40 last 251.67) - /unlink 32860 (time 1764107977.49 total 137.40 last 285.94) - /unlink 35829 (time 1764107987.49 total 147.40 last 296.90) - /unlink 38805 (time 1764107997.49 total 157.40 last 297.55) - /unlink 40000 (time 1764108001.94 total 161.86 last 268.29) - /unlink 43477 (time 1764108011.94 total 171.86 last 347.64) - /unlink 46411 (time 1764108021.95 total 181.86 last 293.37) - /unlink 49406 (time 1764108031.95 total 191.86 last 299.46) - /unlink 52930 (time 1764108041.95 total 201.86 last 352.33) - /unlink 57252 (time 1764108051.95 total 211.87 last 432.09) - /unlink 59395 (time 1764108061.96 total 221.87 last 214.22) - /unlink 60000 (time 1764108066.12 total 226.03 last 145.36) - /unlink 61857 (time 1764108076.13 total 236.04 last 185.54) - /unlink 64038 (time 1764108086.13 total 246.05 last 218.00) total: 64768 /unlink in 249.02 seconds: 260.09 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1222 (time 1764108120.46 total 10.00 last 122.14) - open/close 2395 (time 1764108130.46 total 20.00 last 117.30) - open/close 3596 (time 1764108140.46 total 30.01 last 120.05) - open/close 5194 (time 1764108150.47 total 40.01 last 159.75) - open/close 7034 (time 1764108160.47 total 50.02 last 183.93) - open/close 8637 (time 1764108170.47 total 60.02 last 160.29) - open/close 10000 (time 1764108177.91 total 67.46 last 183.22) - open/close 11821 (time 1764108187.92 total 77.46 last 181.97) - open/close 13001 (time 1764108197.92 total 87.47 last 117.97) - open/close 14152 (time 1764108207.92 total 97.47 last 115.07) - open/close 15513 (time 1764108217.93 total 107.47 last 136.01) - open/close 16639 (time 1764108227.94 total 117.48 last 112.54) - open/close 17794 (time 1764108237.95 total 127.49 last 115.39) - open/close 18941 (time 1764108247.95 total 137.49 last 114.66) - open/close 19947 (time 1764108257.95 total 147.50 last 100.55) - open/close 20945 (time 1764108267.96 total 157.50 last 99.75) - open/close 22262 (time 1764108277.97 total 167.51 last 131.60) - open/close 23899 (time 1764108287.97 total 177.51 last 163.69) - open/close 25348 (time 1764108297.97 total 187.51 last 144.89) - open/close 26752 (time 1764108307.97 total 197.51 last 140.39) - open/close 28047 (time 1764108317.97 total 207.51 last 129.49) - open/close 29488 (time 1764108327.97 total 217.51 last 144.10) - open/close 30000 (time 1764108331.93 total 221.48 last 129.20) - open/close 31455 (time 1764108341.96 total 231.50 last 145.09) - open/close 32716 (time 1764108351.96 total 241.51 last 126.06) - open/close 34369 (time 1764108361.96 total 251.51 last 165.29) - open/close 35533 (time 1764108371.96 total 261.51 last 116.40) - open/close 36738 (time 1764108381.97 total 271.51 last 120.43) - open/close 37794 (time 1764108391.97 total 281.52 last 105.59) - open/close 38893 (time 1764108401.98 total 291.52 last 109.83) - open/close 40000 (time 1764108410.44 total 299.99 last 130.79) - open/close 41589 (time 1764108420.44 total 309.99 last 158.87) - open/close 42703 (time 1764108430.45 total 319.99 last 111.36) - open/close 44001 (time 1764108440.45 total 330.00 last 129.73) - open/close 45516 (time 1764108450.45 total 340.00 last 151.49) - open/close 46917 (time 1764108460.46 total 350.00 last 140.06) - open/close 48280 (time 1764108470.48 total 360.02 last 135.99) - open/close 49886 (time 1764108480.48 total 370.02 last 160.59) - open/close 51382 (time 1764108490.48 total 380.03 last 149.52) - open/close 52503 (time 1764108500.49 total 390.04 last 112.00) - open/close 53668 (time 1764108510.50 total 400.04 last 116.46) - open/close 54855 (time 1764108520.51 total 410.05 last 118.57) - open/close 56289 (time 1764108530.51 total 420.05 last 143.39) - open/close 57630 (time 1764108540.52 total 430.06 last 133.99) - open/close 58633 (time 1764108550.53 total 440.07 last 100.19) - open/close 59345 (time 1764108560.54 total 450.09 last 71.10) - open/close 59823 (time 1764108570.54 total 460.09 last 47.79) - open/close 60000 (time 1764108574.24 total 463.78 last 47.91) - open/close 60558 (time 1764108584.25 total 473.79 last 55.74) - open/close 61419 (time 1764108594.25 total 483.80 last 86.08) - open/close 62419 (time 1764108604.25 total 493.80 last 99.98) - open/close 63735 (time 1764108614.26 total 503.80 last 131.58) total: 64768 open/close in 510.92 seconds: 126.77 ops/second - /unlink 2689 (time 1764108633.70 total 10.00 last 268.84) - /unlink 5148 (time 1764108643.70 total 20.00 last 245.88) - /unlink 7545 (time 1764108653.71 total 30.01 last 239.58) - /unlink 9972 (time 1764108663.71 total 40.01 last 242.68) - /unlink 12016 (time 1764108673.71 total 50.01 last 204.33) - /unlink 14609 (time 1764108683.71 total 60.01 last 259.29) - /unlink 16721 (time 1764108693.71 total 70.01 last 211.16) - /unlink 18932 (time 1764108703.72 total 80.02 last 221.01) - /unlink 20000 (time 1764108708.82 total 85.12 last 209.27) - /unlink 22062 (time 1764108718.82 total 95.12 last 206.17) - /unlink 24601 (time 1764108728.82 total 105.12 last 253.89) - /unlink 27194 (time 1764108738.82 total 115.13 last 259.24) - /unlink 29149 (time 1764108748.83 total 125.13 last 195.40) - /unlink 30000 (time 1764108753.37 total 129.67 last 187.55) - /unlink 31940 (time 1764108763.37 total 139.67 last 193.95) - /unlink 34166 (time 1764108773.37 total 149.67 last 222.58) - /unlink 36437 (time 1764108783.37 total 159.68 last 227.02) - /unlink 38641 (time 1764108793.38 total 169.68 last 220.33) - /unlink 40000 (time 1764108799.14 total 175.44 last 235.99) - /unlink 41864 (time 1764108809.14 total 185.44 last 186.31) - /unlink 43816 (time 1764108819.16 total 195.46 last 194.93) - /unlink 46298 (time 1764108829.16 total 205.46 last 248.17) - /unlink 48343 (time 1764108839.16 total 215.46 last 204.36) - /unlink 50000 (time 1764108846.97 total 223.27 last 212.29) - /unlink 51602 (time 1764108856.97 total 233.27 last 160.15) - /unlink 53045 (time 1764108866.98 total 243.28 last 144.15) - /unlink 54501 (time 1764108876.98 total 253.28 last 145.59) - /unlink 55879 (time 1764108886.99 total 263.29 last 137.77) - /unlink 57284 (time 1764108896.99 total 273.29 last 140.42) - /unlink 58412 (time 1764108907.00 total 283.30 last 112.75) - /unlink 59735 (time 1764108917.00 total 293.30 last 132.25) - /unlink 61250 (time 1764108927.00 total 303.30 last 151.46) - /unlink 62848 (time 1764108937.01 total 313.31 last 159.73) total: 64768 /unlink in 323.22 seconds: 200.38 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg302-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg302-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg302-server stop mds service on oleg302-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg302-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg302-server' oleg302-server: oleg302-server.virtnet: executing unload_modules_local oleg302-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg302-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg302-client: oleg302-client: ssh exited with exit code 2 pdsh@oleg302-client: oleg302-server: ssh exited with exit code 2 pdsh@oleg302-client: oleg302-client: ssh exited with exit code 2 pdsh@oleg302-client: oleg302-server: ssh exited with exit code 2