== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 09:44:10 (1753364650) Stopping clients: oleg221-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg221-client.virtnet /mnt/lustre2 (opts:-f) oleg221-server: oleg221-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg221-server' oleg221-server: oleg221-server.virtnet: executing load_modules_local oleg221-server: Loading modules from /home/green/git/lustre-release/lustre oleg221-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg221-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg221-server: detected 4 online CPUs by sysfs oleg221-server: MODOPTS_LIBCFS= oleg221-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg221-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg221-server: oleg221-server.virtnet: executing set_default_debug -1 all pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg221-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg221-server: oleg221-server.virtnet: executing set_default_debug -1 all pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg221-client.virtnet: -o user_xattr,flock 192.168.202.121@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1432 (time 1753364700.97 total 10.01 last 143.10) - open/close 2813 (time 1753364710.97 total 20.01 last 138.06) - open/close 4070 (time 1753364720.98 total 30.01 last 125.64) - open/close 5447 (time 1753364730.99 total 40.02 last 137.61) - open/close 6765 (time 1753364740.99 total 50.02 last 131.77) - open/close 8192 (time 1753364751.00 total 60.03 last 142.58) - open/close 9254 (time 1753364761.00 total 70.04 last 106.17) - open/close 10000 (time 1753364767.19 total 76.23 last 120.43) - open/close 11448 (time 1753364777.19 total 86.23 last 144.78) - open/close 13208 (time 1753364787.20 total 96.23 last 175.94) - open/close 14996 (time 1753364797.20 total 106.24 last 178.75) - open/close 16845 (time 1753364807.21 total 116.24 last 184.77) - open/close 18282 (time 1753364817.21 total 126.25 last 143.64) - open/close 19681 (time 1753364827.22 total 136.25 last 139.83) - open/close 20000 (time 1753364829.66 total 138.70 last 130.61) - open/close 21659 (time 1753364839.66 total 148.70 last 165.89) - open/close 23472 (time 1753364849.66 total 158.70 last 181.23) - open/close 25185 (time 1753364859.66 total 168.70 last 171.28) - open/close 26988 (time 1753364869.67 total 178.70 last 180.28) - open/close 29164 (time 1753364879.67 total 188.70 last 217.57) - open/close 30000 (time 1753364885.90 total 194.94 last 134.11) - open/close 31459 (time 1753364895.90 total 204.94 last 145.85) - open/close 33081 (time 1753364905.91 total 214.94 last 162.17) - open/close 35085 (time 1753364915.91 total 224.95 last 200.34) - open/close 37227 (time 1753364925.91 total 234.95 last 214.11) - open/close 39364 (time 1753364935.91 total 244.95 last 213.68) - open/close 40000 (time 1753364940.05 total 249.08 last 153.86) - open/close 41194 (time 1753364950.09 total 259.12 last 118.94) - open/close 41922 (time 1753364960.09 total 269.12 last 72.80) - open/close 42588 (time 1753364970.11 total 279.15 last 66.45) - open/close 43282 (time 1753364980.11 total 289.15 last 69.37) - open/close 44237 (time 1753364990.12 total 299.16 last 95.43) - open/close 45474 (time 1753365000.12 total 309.16 last 123.67) - open/close 46731 (time 1753365010.13 total 319.17 last 125.57) - open/close 48223 (time 1753365020.14 total 329.17 last 149.15) - open/close 49479 (time 1753365030.14 total 339.17 last 125.58) - open/close 50000 (time 1753365034.43 total 343.47 last 121.27) - open/close 51141 (time 1753365044.44 total 353.48 last 114.04) - open/close 52418 (time 1753365054.44 total 363.48 last 127.63) - open/close 53744 (time 1753365064.45 total 373.49 last 132.54) - open/close 54933 (time 1753365074.45 total 383.49 last 118.88) - open/close 56121 (time 1753365084.45 total 393.49 last 118.77) - open/close 57313 (time 1753365094.46 total 403.49 last 119.14) - open/close 58415 (time 1753365104.46 total 413.50 last 110.18) - open/close 59582 (time 1753365114.47 total 423.50 last 116.62) - open/close 60000 (time 1753365117.17 total 426.21 last 154.47) - open/close 61518 (time 1753365127.17 total 436.21 last 151.77) - open/close 63210 (time 1753365137.18 total 446.21 last 169.13) total: 64768 open/close in 454.96 seconds: 142.36 ops/second - /unlink 3024 (time 1753365157.60 total 10.00 last 302.38) - /unlink 5543 (time 1753365167.61 total 20.01 last 251.79) - /unlink 8270 (time 1753365177.61 total 30.01 last 272.69) - /unlink 10000 (time 1753365183.55 total 35.95 last 290.92) - /unlink 12883 (time 1753365193.55 total 45.95 last 288.27) - /unlink 16300 (time 1753365203.55 total 55.95 last 341.65) - /unlink 19846 (time 1753365213.56 total 65.96 last 354.55) - /unlink 23258 (time 1753365223.56 total 75.96 last 341.15) - /unlink 26585 (time 1753365233.56 total 85.96 last 332.60) - /unlink 29881 (time 1753365243.56 total 95.96 last 329.58) - /unlink 33201 (time 1753365253.56 total 105.96 last 331.92) - /unlink 36688 (time 1753365263.57 total 115.97 last 348.59) - /unlink 40000 (time 1753365273.47 total 125.87 last 334.29) - /unlink 43950 (time 1753365283.47 total 135.87 last 394.99) - /unlink 47533 (time 1753365293.48 total 145.88 last 358.25) - /unlink 50000 (time 1753365300.53 total 152.93 last 349.95) - /unlink 53473 (time 1753365310.53 total 162.93 last 347.29) - /unlink 57737 (time 1753365320.53 total 172.93 last 426.37) - /unlink 60000 (time 1753365326.96 total 179.36 last 351.49) - /unlink 63770 (time 1753365336.97 total 189.37 last 376.94) total: 64768 /unlink in 192.15 seconds: 337.06 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2598 (time 1753365369.60 total 10.00 last 259.73) - open/close 5007 (time 1753365379.60 total 20.00 last 240.87) - open/close 7693 (time 1753365389.61 total 30.00 last 268.59) - open/close 10000 (time 1753365399.04 total 39.44 last 244.62) - open/close 12400 (time 1753365409.04 total 49.44 last 239.90) - open/close 14824 (time 1753365419.04 total 59.44 last 242.38) - open/close 17907 (time 1753365429.04 total 69.44 last 308.23) - open/close 20000 (time 1753365435.84 total 76.24 last 307.91) - open/close 22930 (time 1753365445.84 total 86.24 last 292.95) - open/close 25145 (time 1753365455.84 total 96.24 last 221.48) - open/close 27228 (time 1753365465.85 total 106.25 last 208.23) - open/close 29677 (time 1753365475.85 total 116.25 last 244.82) - open/close 32091 (time 1753365485.85 total 126.25 last 241.36) - open/close 34160 (time 1753365495.86 total 136.26 last 206.80) - open/close 36235 (time 1753365505.86 total 146.26 last 207.47) - open/close 38932 (time 1753365515.86 total 156.26 last 269.65) - open/close 40000 (time 1753365520.20 total 160.60 last 246.18) - open/close 42508 (time 1753365530.20 total 170.60 last 250.75) - open/close 44858 (time 1753365540.20 total 180.60 last 234.94) - open/close 47241 (time 1753365550.20 total 190.60 last 238.28) - open/close 49661 (time 1753365560.21 total 200.61 last 241.95) - open/close 52623 (time 1753365570.21 total 210.61 last 296.13) - open/close 55085 (time 1753365580.21 total 220.61 last 246.13) - open/close 57660 (time 1753365590.21 total 230.61 last 257.50) - open/close 60000 (time 1753365596.91 total 237.30 last 349.56) - open/close 63454 (time 1753365606.91 total 247.31 last 345.38) total: 64768 open/close in 251.59 seconds: 257.44 ops/second - /unlink 5330 (time 1753365622.64 total 10.00 last 532.92) - /unlink 10000 (time 1753365631.10 total 18.46 last 551.81) - /unlink 15111 (time 1753365641.10 total 28.47 last 511.00) - /unlink 20000 (time 1753365650.12 total 37.48 last 542.30) - /unlink 26436 (time 1753365660.12 total 47.48 last 643.57) - /unlink 30000 (time 1753365665.89 total 53.25 last 617.88) - /unlink 36653 (time 1753365675.89 total 63.25 last 665.25) - /unlink 40000 (time 1753365680.98 total 68.35 last 656.71) - /unlink 45677 (time 1753365690.99 total 78.35 last 567.62) - /unlink 50000 (time 1753365699.15 total 86.52 last 529.38) - /unlink 55759 (time 1753365709.15 total 96.52 last 575.89) - /unlink 60000 (time 1753365716.01 total 103.38 last 618.09) - /unlink 62016 (time 1753365726.02 total 113.39 last 201.41) - /unlink 63145 (time 1753365736.03 total 123.39 last 112.86) - /unlink 64554 (time 1753365746.04 total 133.40 last 140.78) total: 64768 /unlink in 135.50 seconds: 478.00 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 399 (time 1753365787.10 total 10.01 last 39.85) - open/close 768 (time 1753365797.11 total 20.02 last 36.88) - open/close 1171 (time 1753365807.12 total 30.03 last 40.25) - open/close 1713 (time 1753365817.13 total 40.04 last 54.13) - open/close 2346 (time 1753365827.13 total 50.04 last 63.29) - open/close 2989 (time 1753365837.14 total 60.05 last 64.25) - open/close 3735 (time 1753365847.14 total 70.05 last 74.59) - open/close 4527 (time 1753365857.15 total 80.06 last 79.13) - open/close 5318 (time 1753365867.16 total 90.07 last 79.04) - open/close 6070 (time 1753365877.17 total 100.08 last 75.10) - open/close 6702 (time 1753365887.18 total 110.09 last 63.15) - open/close 7417 (time 1753365897.19 total 120.10 last 71.41) - open/close 8007 (time 1753365907.19 total 130.11 last 58.97) - open/close 8718 (time 1753365917.21 total 140.12 last 71.02) - open/close 9433 (time 1753365927.22 total 150.13 last 71.43) - open/close 10000 (time 1753365934.71 total 157.63 last 75.62) - open/close 10722 (time 1753365944.72 total 167.63 last 72.19) - open/close 11505 (time 1753365954.73 total 177.64 last 78.21) - open/close 12255 (time 1753365964.74 total 187.65 last 74.92) - open/close 13155 (time 1753365974.74 total 197.66 last 89.94) - open/close 13972 (time 1753365984.75 total 207.66 last 81.63) - open/close 14740 (time 1753365994.76 total 217.67 last 76.76) - open/close 15565 (time 1753366004.79 total 227.70 last 82.23) - open/close 16350 (time 1753366014.80 total 237.71 last 78.41) - open/close 17015 (time 1753366024.81 total 247.72 last 66.46) - open/close 17785 (time 1753366034.82 total 257.73 last 76.90) - open/close 18645 (time 1753366044.82 total 267.74 last 85.98) - open/close 19632 (time 1753366054.84 total 277.75 last 98.59) - open/close 20000 (time 1753366059.19 total 282.10 last 84.47) - open/close 20927 (time 1753366069.20 total 292.11 last 92.66) - open/close 21991 (time 1753366079.20 total 302.11 last 106.36) - open/close 23055 (time 1753366089.21 total 312.12 last 106.34) - open/close 24228 (time 1753366099.21 total 322.12 last 117.25) - open/close 25546 (time 1753366109.22 total 332.13 last 131.74) - open/close 26941 (time 1753366119.22 total 342.13 last 139.47) - open/close 28840 (time 1753366129.22 total 352.13 last 189.85) - open/close 30000 (time 1753366134.24 total 357.15 last 231.14) - open/close 32053 (time 1753366144.24 total 367.16 last 205.18) - open/close 33777 (time 1753366154.24 total 377.16 last 172.40) - open/close 35940 (time 1753366164.25 total 387.16 last 216.28) - open/close 38159 (time 1753366174.25 total 397.16 last 221.89) - open/close 40000 (time 1753366179.96 total 402.87 last 322.00) - open/close 42281 (time 1753366189.97 total 412.88 last 228.06) - open/close 45459 (time 1753366199.97 total 422.88 last 317.72) - open/close 48520 (time 1753366209.97 total 432.88 last 306.02) - open/close 50000 (time 1753366214.36 total 437.27 last 337.11) - open/close 54193 (time 1753366224.36 total 447.27 last 419.27) - open/close 57439 (time 1753366234.36 total 457.27 last 324.56) - open/close 60000 (time 1753366244.25 total 467.16 last 259.12) - open/close 63263 (time 1753366254.25 total 477.16 last 326.23) total: 64768 open/close in 481.04 seconds: 134.64 ops/second - /unlink 5428 (time 1753366269.65 total 10.00 last 542.78) - /unlink 10000 (time 1753366277.88 total 18.23 last 555.58) - /unlink 15302 (time 1753366287.88 total 28.23 last 530.18) - /unlink 20000 (time 1753366297.41 total 37.76 last 493.16) - /unlink 26108 (time 1753366307.41 total 47.76 last 610.76) - /unlink 30000 (time 1753366315.88 total 56.22 last 459.71) - /unlink 35414 (time 1753366325.88 total 66.22 last 541.35) - /unlink 40000 (time 1753366334.69 total 75.04 last 520.29) - /unlink 45370 (time 1753366344.69 total 85.04 last 536.97) - /unlink 49536 (time 1753366354.69 total 95.04 last 416.55) - /unlink 55700 (time 1753366364.69 total 105.04 last 616.39) - /unlink 60000 (time 1753366371.12 total 111.47 last 668.65) total: 64768 /unlink in 119.05 seconds: 544.06 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3951 (time 1753366408.90 total 10.00 last 395.06) - open/close 8188 (time 1753366418.90 total 20.00 last 423.66) - open/close 10000 (time 1753366422.84 total 23.94 last 460.25) - open/close 14785 (time 1753366432.84 total 33.94 last 478.49) - open/close 19728 (time 1753366442.84 total 43.94 last 494.25) - open/close 24351 (time 1753366452.84 total 53.94 last 462.22) - open/close 28901 (time 1753366462.84 total 63.94 last 454.93) - open/close 30000 (time 1753366464.86 total 65.96 last 545.81) - open/close 34990 (time 1753366474.86 total 75.96 last 498.92) - open/close 39695 (time 1753366484.86 total 85.96 last 470.48) - open/close 44427 (time 1753366494.86 total 95.96 last 473.14) - open/close 49121 (time 1753366504.86 total 105.96 last 469.39) - open/close 50000 (time 1753366507.14 total 108.24 last 385.49) - open/close 54755 (time 1753366517.14 total 118.24 last 475.41) - open/close 59527 (time 1753366527.14 total 128.24 last 477.18) - open/close 64429 (time 1753366537.14 total 138.24 last 490.15) total: 64768 open/close in 139.12 seconds: 465.56 ops/second - /unlink 7325 (time 1753366549.30 total 10.00 last 732.46) - /unlink 10000 (time 1753366552.87 total 13.57 last 748.99) - /unlink 17262 (time 1753366562.87 total 23.57 last 726.12) - /unlink 20000 (time 1753366567.15 total 27.85 last 640.16) - /unlink 26252 (time 1753366577.15 total 37.85 last 625.18) - /unlink 30000 (time 1753366582.21 total 42.91 last 741.12) - /unlink 36267 (time 1753366592.21 total 52.91 last 626.66) - /unlink 40000 (time 1753366598.30 total 59.00 last 613.08) - /unlink 46157 (time 1753366608.30 total 69.00 last 615.66) - /unlink 50000 (time 1753366614.13 total 74.83 last 659.10) - /unlink 56411 (time 1753366624.13 total 84.83 last 641.06) - /unlink 60000 (time 1753366630.92 total 91.62 last 528.33) - /unlink 61652 (time 1753366640.92 total 101.62 last 165.17) - /unlink 63343 (time 1753366650.94 total 111.64 last 168.83) total: 64768 /unlink in 120.46 seconds: 537.68 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 848 (time 1753366693.84 total 10.00 last 84.80) - open/close 1767 (time 1753366703.85 total 20.01 last 91.85) - open/close 2695 (time 1753366713.86 total 30.01 last 92.73) - open/close 3753 (time 1753366723.87 total 40.02 last 105.68) - open/close 5085 (time 1753366733.87 total 50.02 last 133.20) - open/close 6181 (time 1753366743.87 total 60.03 last 109.56) - open/close 7228 (time 1753366753.88 total 70.03 last 104.63) - open/close 8380 (time 1753366763.88 total 80.04 last 115.14) - open/close 9582 (time 1753366773.90 total 90.05 last 120.04) - open/close 10000 (time 1753366776.91 total 93.07 last 138.70) - open/close 11415 (time 1753366786.91 total 103.07 last 141.45) - open/close 12631 (time 1753366796.92 total 113.07 last 121.56) - open/close 13681 (time 1753366806.92 total 123.08 last 104.96) - open/close 14690 (time 1753366816.92 total 133.08 last 100.86) - open/close 15900 (time 1753366826.93 total 143.09 last 120.93) - open/close 16860 (time 1753366836.93 total 153.09 last 95.97) - open/close 17912 (time 1753366846.93 total 163.09 last 105.18) - open/close 19018 (time 1753366856.94 total 173.09 last 110.56) - open/close 20000 (time 1753366866.46 total 182.62 last 103.11) - open/close 21061 (time 1753366876.46 total 192.62 last 106.09) - open/close 22112 (time 1753366886.47 total 202.62 last 105.05) - open/close 23031 (time 1753366896.47 total 212.63 last 91.87) - open/close 23880 (time 1753366906.47 total 222.63 last 84.88) - open/close 24745 (time 1753366916.48 total 232.64 last 86.42) - open/close 25677 (time 1753366926.49 total 242.65 last 93.14) - open/close 26744 (time 1753366936.50 total 252.65 last 106.63) - open/close 27775 (time 1753366946.50 total 262.65 last 103.09) - open/close 28968 (time 1753366956.50 total 272.66 last 119.28) - open/close 30000 (time 1753366965.19 total 281.35 last 118.73) - open/close 30982 (time 1753366975.20 total 291.35 last 98.15) - open/close 31889 (time 1753366985.21 total 301.36 last 90.62) - open/close 32818 (time 1753366995.21 total 311.37 last 92.86) - open/close 33759 (time 1753367005.22 total 321.37 last 94.03) - open/close 34774 (time 1753367015.22 total 331.38 last 101.46) - open/close 36020 (time 1753367025.23 total 341.39 last 124.50) - open/close 37091 (time 1753367035.24 total 351.39 last 107.04) - open/close 38144 (time 1753367045.24 total 361.39 last 105.30) - open/close 39552 (time 1753367055.24 total 371.39 last 140.77) - open/close 40000 (time 1753367059.41 total 375.56 last 107.48) - open/close 41296 (time 1753367069.41 total 385.57 last 129.53) - open/close 43100 (time 1753367079.41 total 395.57 last 180.34) - open/close 44728 (time 1753367089.42 total 405.58 last 162.70) - open/close 45947 (time 1753367099.42 total 415.58 last 121.85) - open/close 47288 (time 1753367109.43 total 425.59 last 133.98) - open/close 48459 (time 1753367119.45 total 435.61 last 116.86) - open/close 49666 (time 1753367129.46 total 445.61 last 120.68) - open/close 50000 (time 1753367132.63 total 448.79 last 105.10) - open/close 51248 (time 1753367142.64 total 458.79 last 124.77) - open/close 52977 (time 1753367152.64 total 468.79 last 172.86) - open/close 54881 (time 1753367162.64 total 478.79 last 190.39) - open/close 56639 (time 1753367172.64 total 488.80 last 175.71) - open/close 58179 (time 1753367182.65 total 498.81 last 153.86) - open/close 59912 (time 1753367192.65 total 508.81 last 173.28) - open/close 61563 (time 1753367202.66 total 518.81 last 165.07) - open/close 63211 (time 1753367212.66 total 528.82 last 164.69) - open/close 64709 (time 1753367222.67 total 538.83 last 149.70) total: 64768 open/close in 539.36 seconds: 120.08 ops/second - /unlink 1962 (time 1753367235.21 total 10.00 last 196.16) - /unlink 3817 (time 1753367245.21 total 20.00 last 185.47) - /unlink 5919 (time 1753367255.21 total 30.00 last 210.18) - /unlink 7773 (time 1753367265.21 total 40.00 last 185.39) - /unlink 9565 (time 1753367275.21 total 50.01 last 179.18) - /unlink 10000 (time 1753367277.54 total 52.33 last 187.21) - /unlink 12148 (time 1753367287.54 total 62.33 last 214.79) - /unlink 14616 (time 1753367297.54 total 72.33 last 246.72) - /unlink 16541 (time 1753367307.54 total 82.33 last 192.47) - /unlink 18637 (time 1753367317.54 total 92.34 last 209.59) - /unlink 20000 (time 1753367324.04 total 98.84 last 209.68) - /unlink 22174 (time 1753367334.05 total 108.84 last 217.30) - /unlink 24395 (time 1753367344.05 total 118.84 last 222.00) - /unlink 26576 (time 1753367354.05 total 128.85 last 218.07) - /unlink 28367 (time 1753367364.06 total 138.85 last 178.99) - /unlink 30000 (time 1753367373.36 total 148.16 last 175.53) - /unlink 31940 (time 1753367383.37 total 158.16 last 193.90) - /unlink 33838 (time 1753367393.37 total 168.17 last 189.70) - /unlink 35774 (time 1753367403.38 total 178.17 last 193.49) - /unlink 38707 (time 1753367413.38 total 188.17 last 293.24) - /unlink 40000 (time 1753367418.06 total 192.86 last 276.16) - /unlink 42750 (time 1753367428.06 total 202.86 last 274.97) - /unlink 45015 (time 1753367438.06 total 212.86 last 226.46) - /unlink 47141 (time 1753367448.07 total 222.86 last 212.58) - /unlink 49030 (time 1753367458.07 total 232.86 last 188.87) - /unlink 50000 (time 1753367464.21 total 239.00 last 157.91) - /unlink 51848 (time 1753367474.23 total 249.02 last 184.46) - /unlink 53674 (time 1753367484.23 total 259.02 last 182.55) - /unlink 55490 (time 1753367494.23 total 269.03 last 181.58) - /unlink 57421 (time 1753367504.23 total 279.03 last 193.09) - /unlink 59823 (time 1753367514.24 total 289.03 last 240.13) - /unlink 62191 (time 1753367524.24 total 299.03 last 236.76) total: 64768 /unlink in 308.73 seconds: 209.79 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg221-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg221-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg221-server stop mds service on oleg221-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg221-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg221-server' oleg221-server: oleg221-server.virtnet: executing unload_modules_local oleg221-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg221-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg221-client: oleg221-client: ssh exited with exit code 2 pdsh@oleg221-client: oleg221-server: ssh exited with exit code 2 pdsh@oleg221-client: oleg221-client: ssh exited with exit code 2 pdsh@oleg221-client: oleg221-server: ssh exited with exit code 2