== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 06:46:28 (1753440388) Stopping clients: oleg137-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg137-client.virtnet /mnt/lustre2 (opts:-f) oleg137-server: oleg137-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg137-server' oleg137-server: oleg137-server.virtnet: executing load_modules_local oleg137-server: Loading modules from /home/green/git/lustre-release/lustre oleg137-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg137-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg137-server: detected 4 online CPUs by sysfs oleg137-server: MODOPTS_LIBCFS= oleg137-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg137-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg137-server: oleg137-server.virtnet: executing set_default_debug -1 all pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg137-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg137-server: oleg137-server.virtnet: executing set_default_debug -1 all pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg137-client.virtnet: -o user_xattr,flock 192.168.201.137@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32648 392 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 827 (time 1753440466.33 total 10.01 last 82.63) - open/close 1618 (time 1753440476.33 total 20.01 last 79.09) - open/close 2416 (time 1753440486.34 total 30.02 last 79.72) - open/close 3243 (time 1753440496.34 total 40.03 last 82.64) - open/close 4060 (time 1753440506.36 total 50.04 last 81.57) - open/close 5079 (time 1753440516.37 total 60.06 last 101.76) - open/close 6100 (time 1753440526.39 total 70.07 last 101.96) - open/close 7025 (time 1753440536.39 total 80.08 last 92.44) - open/close 7936 (time 1753440546.40 total 90.08 last 91.07) - open/close 8894 (time 1753440556.40 total 100.09 last 95.76) - open/close 10000 (time 1753440565.74 total 109.42 last 118.45) - open/close 11402 (time 1753440575.74 total 119.42 last 140.18) - open/close 12726 (time 1753440585.75 total 129.43 last 132.34) - open/close 13953 (time 1753440595.75 total 139.43 last 122.63) - open/close 14934 (time 1753440605.76 total 149.44 last 98.05) - open/close 15836 (time 1753440615.76 total 159.44 last 90.20) - open/close 16756 (time 1753440625.76 total 169.44 last 91.99) - open/close 17918 (time 1753440635.76 total 179.45 last 116.12) - open/close 19221 (time 1753440645.77 total 189.45 last 130.29) - open/close 20000 (time 1753440652.74 total 196.43 last 111.61) - open/close 21188 (time 1753440662.75 total 206.43 last 118.77) - open/close 22195 (time 1753440672.75 total 216.43 last 100.67) - open/close 23721 (time 1753440682.76 total 226.44 last 152.49) - open/close 25126 (time 1753440692.76 total 236.44 last 140.50) - open/close 26349 (time 1753440702.76 total 246.44 last 122.29) - open/close 27595 (time 1753440712.76 total 256.45 last 124.54) - open/close 28848 (time 1753440722.77 total 266.45 last 125.20) - open/close 30000 (time 1753440730.25 total 273.93 last 154.09) - open/close 31313 (time 1753440740.25 total 283.93 last 131.26) - open/close 32911 (time 1753440750.25 total 293.94 last 159.74) - open/close 34580 (time 1753440760.26 total 303.94 last 166.83) - open/close 36714 (time 1753440770.26 total 313.94 last 213.36) - open/close 38295 (time 1753440780.26 total 323.95 last 158.08) - open/close 39864 (time 1753440790.26 total 333.95 last 156.89) - open/close 41572 (time 1753440800.27 total 343.95 last 170.74) - open/close 43520 (time 1753440810.27 total 353.96 last 194.66) - open/close 45438 (time 1753440820.27 total 363.96 last 191.79) - open/close 47214 (time 1753440830.28 total 373.96 last 177.48) - open/close 48741 (time 1753440840.28 total 383.96 last 152.69) - open/close 50000 (time 1753440848.61 total 392.29 last 151.24) - open/close 51644 (time 1753440858.61 total 402.29 last 164.37) - open/close 53736 (time 1753440868.61 total 412.30 last 209.12) - open/close 55533 (time 1753440878.62 total 422.30 last 179.63) - open/close 57025 (time 1753440888.63 total 432.31 last 148.98) - open/close 58462 (time 1753440898.63 total 442.32 last 143.66) - open/close 59961 (time 1753440908.64 total 452.32 last 149.86) - open/close 61408 (time 1753440918.64 total 462.32 last 144.68) - open/close 62993 (time 1753440928.64 total 472.32 last 158.49) - open/close 64236 (time 1753440938.64 total 482.32 last 124.26) total: 64768 open/close in 487.05 seconds: 132.98 ops/second - /unlink 2558 (time 1753440955.48 total 10.00 last 255.77) - /unlink 5373 (time 1753440965.52 total 20.04 last 280.39) - /unlink 7969 (time 1753440975.52 total 30.04 last 259.54) - /unlink 9818 (time 1753440985.52 total 40.04 last 184.86) - /unlink 11992 (time 1753440995.53 total 50.05 last 217.28) - /unlink 13783 (time 1753441005.53 total 60.05 last 179.02) - /unlink 14977 (time 1753441015.54 total 70.06 last 119.34) - /unlink 15961 (time 1753441025.54 total 80.06 last 98.37) - /unlink 16532 (time 1753441035.56 total 90.08 last 56.98) - /unlink 17330 (time 1753441045.56 total 100.08 last 79.79) - /unlink 18434 (time 1753441055.57 total 110.09 last 110.35) - /unlink 19590 (time 1753441065.57 total 120.09 last 115.56) - /unlink 20000 (time 1753441068.86 total 123.38 last 124.70) - /unlink 21156 (time 1753441078.86 total 133.38 last 115.57) - /unlink 22551 (time 1753441088.87 total 143.39 last 139.39) - /unlink 23909 (time 1753441098.87 total 153.39 last 135.75) - /unlink 25264 (time 1753441108.88 total 163.41 last 135.32) - /unlink 26584 (time 1753441118.89 total 173.41 last 131.97) - /unlink 27953 (time 1753441128.89 total 183.41 last 136.88) - /unlink 29350 (time 1753441138.89 total 193.42 last 139.61) - /unlink 30000 (time 1753441143.77 total 198.29 last 133.33) - /unlink 31350 (time 1753441153.77 total 208.29 last 134.99) - /unlink 32634 (time 1753441163.78 total 218.30 last 128.29) - /unlink 33857 (time 1753441173.78 total 228.30 last 122.30) - /unlink 35117 (time 1753441183.78 total 238.30 last 125.96) - /unlink 36388 (time 1753441193.79 total 248.31 last 127.05) - /unlink 37695 (time 1753441203.79 total 258.31 last 130.69) - /unlink 39073 (time 1753441213.79 total 268.31 last 137.77) - /unlink 40000 (time 1753441220.27 total 274.79 last 143.04) - /unlink 41362 (time 1753441230.27 total 284.80 last 136.14) - /unlink 42659 (time 1753441240.27 total 294.80 last 129.69) - /unlink 43940 (time 1753441250.28 total 304.80 last 128.09) - /unlink 45230 (time 1753441260.28 total 314.80 last 128.95) - /unlink 46447 (time 1753441270.28 total 324.81 last 121.66) - /unlink 47531 (time 1753441280.29 total 334.82 last 108.28) - /unlink 48712 (time 1753441290.29 total 344.82 last 118.09) - /unlink 49994 (time 1753441300.30 total 354.82 last 128.15) - /unlink 51267 (time 1753441310.30 total 364.82 last 127.29) - /unlink 52576 (time 1753441320.30 total 374.83 last 130.83) - /unlink 53800 (time 1753441330.31 total 384.83 last 122.32) - /unlink 54925 (time 1753441340.32 total 394.84 last 112.44) - /unlink 56298 (time 1753441350.32 total 404.84 last 137.24) - /unlink 57690 (time 1753441360.32 total 414.85 last 139.16) - /unlink 59021 (time 1753441370.33 total 424.86 last 132.96) - /unlink 60000 (time 1753441378.35 total 432.87 last 122.14) - /unlink 61245 (time 1753441388.36 total 442.88 last 124.36) - /unlink 62502 (time 1753441398.37 total 452.89 last 125.59) - /unlink 63832 (time 1753441408.37 total 462.89 last 133.00) total: 64768 /unlink in 470.08 seconds: 137.78 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 815 (time 1753441450.51 total 10.00 last 81.47) - open/close 1623 (time 1753441460.52 total 20.01 last 80.77) - open/close 2515 (time 1753441470.53 total 30.02 last 89.06) - open/close 3465 (time 1753441480.54 total 40.03 last 94.97) - open/close 4348 (time 1753441490.54 total 50.03 last 88.27) - open/close 5276 (time 1753441500.54 total 60.03 last 92.76) - open/close 6460 (time 1753441510.55 total 70.04 last 118.39) - open/close 7599 (time 1753441520.56 total 80.05 last 113.77) - open/close 8565 (time 1753441530.57 total 90.06 last 96.51) - open/close 9461 (time 1753441540.57 total 100.06 last 89.53) - open/close 10000 (time 1753441545.53 total 105.02 last 108.67) - open/close 11117 (time 1753441555.55 total 115.04 last 111.56) - open/close 12035 (time 1753441565.56 total 125.05 last 91.71) - open/close 13117 (time 1753441575.56 total 135.05 last 108.17) - open/close 14108 (time 1753441585.57 total 145.06 last 99.03) - open/close 15045 (time 1753441595.57 total 155.06 last 93.70) - open/close 16365 (time 1753441605.57 total 165.06 last 131.93) - open/close 17680 (time 1753441615.57 total 175.06 last 131.47) - open/close 18981 (time 1753441625.58 total 185.06 last 130.09) - open/close 20000 (time 1753441633.54 total 193.03 last 127.89) - open/close 21384 (time 1753441643.55 total 203.04 last 138.36) - open/close 22761 (time 1753441653.55 total 213.04 last 137.69) - open/close 24073 (time 1753441663.55 total 223.04 last 131.19) - open/close 25279 (time 1753441673.55 total 233.04 last 120.56) - open/close 26513 (time 1753441683.56 total 243.04 last 123.33) - open/close 27867 (time 1753441693.56 total 253.05 last 135.34) - open/close 29414 (time 1753441703.56 total 263.05 last 154.67) - open/close 30000 (time 1753441707.60 total 267.09 last 145.07) - open/close 31406 (time 1753441717.61 total 277.09 last 140.54) - open/close 32707 (time 1753441727.61 total 287.10 last 130.06) - open/close 33932 (time 1753441737.61 total 297.10 last 122.48) - open/close 35257 (time 1753441747.61 total 307.10 last 132.45) - open/close 36530 (time 1753441757.62 total 317.11 last 127.21) - open/close 38013 (time 1753441767.62 total 327.11 last 148.25) - open/close 39412 (time 1753441777.63 total 337.11 last 139.87) - open/close 40000 (time 1753441781.24 total 340.73 last 162.53) - open/close 41422 (time 1753441791.25 total 350.74 last 142.13) - open/close 42526 (time 1753441801.25 total 360.74 last 110.37) - open/close 43707 (time 1753441811.25 total 370.74 last 118.06) - open/close 45089 (time 1753441821.26 total 380.74 last 138.17) - open/close 46549 (time 1753441831.26 total 390.75 last 145.95) - open/close 47966 (time 1753441841.26 total 400.75 last 141.65) - open/close 49126 (time 1753441851.27 total 410.76 last 115.92) - open/close 50000 (time 1753441858.89 total 418.37 last 114.76) - open/close 51549 (time 1753441868.89 total 428.38 last 154.86) - open/close 53064 (time 1753441878.89 total 438.38 last 151.49) - open/close 54816 (time 1753441888.89 total 448.38 last 175.20) - open/close 56482 (time 1753441898.89 total 458.38 last 166.59) - open/close 58257 (time 1753441908.89 total 468.38 last 177.49) - open/close 60000 (time 1753441918.51 total 478.00 last 181.13) - open/close 62053 (time 1753441928.51 total 488.00 last 205.29) - open/close 64174 (time 1753441938.52 total 498.01 last 212.00) total: 64768 open/close in 501.47 seconds: 129.16 ops/second - /unlink 2743 (time 1753441953.78 total 10.00 last 274.30) - /unlink 5385 (time 1753441963.78 total 20.00 last 264.11) - /unlink 8256 (time 1753441973.78 total 30.01 last 287.04) - /unlink 10000 (time 1753441980.39 total 36.61 last 264.03) - /unlink 13115 (time 1753441990.39 total 46.61 last 311.44) - /unlink 16311 (time 1753442000.39 total 56.61 last 319.55) - /unlink 19030 (time 1753442010.39 total 66.62 last 271.81) - /unlink 20000 (time 1753442014.29 total 70.51 last 249.22) - /unlink 22391 (time 1753442024.29 total 80.51 last 239.10) - /unlink 25171 (time 1753442034.29 total 90.51 last 277.96) - /unlink 27173 (time 1753442044.29 total 100.52 last 200.09) - /unlink 29241 (time 1753442054.29 total 110.52 last 206.77) - /unlink 30000 (time 1753442057.72 total 113.94 last 221.71) - /unlink 32431 (time 1753442067.72 total 123.94 last 243.04) - /unlink 34874 (time 1753442077.72 total 133.94 last 244.29) - /unlink 37387 (time 1753442087.72 total 143.95 last 251.20) - /unlink 39721 (time 1753442097.73 total 153.95 last 233.34) - /unlink 43038 (time 1753442107.73 total 163.95 last 331.62) - /unlink 45985 (time 1753442117.73 total 173.95 last 294.69) - /unlink 49263 (time 1753442127.73 total 183.95 last 327.77) - /unlink 50000 (time 1753442130.34 total 186.56 last 282.62) - /unlink 52038 (time 1753442140.34 total 196.56 last 203.77) - /unlink 54208 (time 1753442150.34 total 206.56 last 217.00) - /unlink 56999 (time 1753442160.34 total 216.57 last 279.02) - /unlink 59423 (time 1753442170.35 total 226.57 last 242.32) - /unlink 60000 (time 1753442172.60 total 228.83 last 255.70) - /unlink 62616 (time 1753442182.61 total 238.83 last 261.50) total: 64768 /unlink in 245.95 seconds: 263.34 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2053 (time 1753442220.93 total 10.00 last 205.23) - open/close 3806 (time 1753442230.93 total 20.01 last 175.26) - open/close 5600 (time 1753442240.93 total 30.01 last 179.37) - open/close 7413 (time 1753442250.94 total 40.01 last 181.23) - open/close 9261 (time 1753442260.94 total 50.01 last 184.73) - open/close 10000 (time 1753442265.00 total 54.08 last 181.95) - open/close 12028 (time 1753442275.01 total 64.08 last 202.73) - open/close 14523 (time 1753442285.01 total 74.08 last 249.39) - open/close 16694 (time 1753442295.01 total 84.09 last 217.00) - open/close 18983 (time 1753442305.02 total 94.09 last 228.85) - open/close 20000 (time 1753442308.59 total 97.66 last 284.96) - open/close 22394 (time 1753442318.59 total 107.66 last 239.30) - open/close 24851 (time 1753442328.59 total 117.67 last 245.59) - open/close 26494 (time 1753442338.60 total 127.68 last 164.18) - open/close 28795 (time 1753442348.60 total 137.68 last 230.03) - open/close 30000 (time 1753442352.81 total 141.89 last 286.44) - open/close 32611 (time 1753442362.81 total 151.89 last 261.03) - open/close 35232 (time 1753442372.82 total 161.89 last 262.05) - open/close 37880 (time 1753442382.82 total 171.89 last 264.77) - open/close 40000 (time 1753442391.82 total 180.89 last 235.59) - open/close 42856 (time 1753442401.82 total 190.89 last 285.54) - open/close 45123 (time 1753442411.82 total 200.89 last 226.69) - open/close 47379 (time 1753442421.82 total 210.89 last 225.60) - open/close 49447 (time 1753442431.82 total 220.89 last 206.79) - open/close 50000 (time 1753442434.35 total 223.43 last 218.36) - open/close 52714 (time 1753442444.36 total 233.43 last 271.27) - open/close 55399 (time 1753442454.36 total 243.43 last 268.41) - open/close 57733 (time 1753442464.36 total 253.44 last 233.37) - open/close 60000 (time 1753442472.85 total 261.93 last 267.00) - open/close 63723 (time 1753442482.86 total 271.93 last 372.17) total: 64768 open/close in 275.78 seconds: 234.86 ops/second - /unlink 4698 (time 1753442498.29 total 10.00 last 469.71) - /unlink 8683 (time 1753442508.29 total 20.00 last 398.48) - /unlink 10000 (time 1753442511.09 total 22.81 last 468.93) - /unlink 14486 (time 1753442521.10 total 32.81 last 448.55) - /unlink 18871 (time 1753442531.10 total 42.81 last 438.42) - /unlink 20000 (time 1753442533.16 total 44.88 last 546.44) - /unlink 25582 (time 1753442543.16 total 54.88 last 558.13) - /unlink 30000 (time 1753442552.30 total 64.02 last 483.63) - /unlink 35365 (time 1753442562.30 total 74.02 last 536.41) - /unlink 40000 (time 1753442569.24 total 80.96 last 668.16) - /unlink 45562 (time 1753442579.24 total 90.96 last 556.18) - /unlink 49993 (time 1753442589.24 total 100.96 last 443.06) - /unlink 54654 (time 1753442599.24 total 110.96 last 466.08) - /unlink 60000 (time 1753442607.15 total 118.87 last 675.66) total: 64768 /unlink in 125.85 seconds: 514.66 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3392 (time 1753442642.91 total 10.01 last 339.00) - open/close 6607 (time 1753442652.91 total 20.01 last 321.49) - open/close 9543 (time 1753442662.91 total 30.01 last 293.60) - open/close 13956 (time 1753442672.91 total 40.01 last 441.25) - open/close 17535 (time 1753442682.91 total 50.01 last 357.81) - open/close 20000 (time 1753442692.26 total 59.36 last 263.74) - open/close 23460 (time 1753442702.26 total 69.36 last 345.94) - open/close 26418 (time 1753442712.26 total 79.36 last 295.77) - open/close 30000 (time 1753442721.68 total 88.78 last 380.22) - open/close 33013 (time 1753442731.68 total 98.78 last 301.30) - open/close 36220 (time 1753442741.68 total 108.78 last 320.66) - open/close 39984 (time 1753442751.68 total 118.78 last 376.34) - open/close 44226 (time 1753442761.74 total 128.84 last 421.65) - open/close 48520 (time 1753442771.74 total 138.84 last 429.39) - open/close 50000 (time 1753442774.98 total 142.08 last 458.01) - open/close 54362 (time 1753442784.98 total 152.08 last 436.12) - open/close 58280 (time 1753442794.98 total 162.08 last 391.77) - open/close 60000 (time 1753442799.19 total 166.29 last 408.80) - open/close 62522 (time 1753442809.19 total 176.29 last 252.12) total: 64768 open/close in 183.80 seconds: 352.38 ops/second - /unlink 8075 (time 1753442827.99 total 10.00 last 807.44) - /unlink 10000 (time 1753442830.76 total 12.77 last 696.16) - /unlink 17164 (time 1753442840.76 total 22.77 last 716.35) - /unlink 20000 (time 1753442844.81 total 26.82 last 699.16) - /unlink 26445 (time 1753442854.81 total 36.82 last 644.45) - /unlink 30000 (time 1753442860.59 total 42.60 last 615.56) - /unlink 36283 (time 1753442870.59 total 52.60 last 628.20) - /unlink 40000 (time 1753442876.21 total 58.22 last 661.55) - /unlink 46093 (time 1753442886.21 total 68.22 last 609.23) - /unlink 50000 (time 1753442891.69 total 73.70 last 713.27) - /unlink 56459 (time 1753442901.69 total 83.70 last 645.87) - /unlink 60000 (time 1753442907.91 total 89.92 last 569.11) total: 64768 /unlink in 97.65 seconds: 663.30 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3021 (time 1753442947.39 total 10.00 last 302.07) - open/close 5967 (time 1753442957.39 total 20.00 last 294.56) - open/close 8402 (time 1753442967.39 total 30.00 last 243.45) - open/close 10000 (time 1753442972.86 total 35.47 last 292.37) - open/close 12903 (time 1753442982.86 total 45.47 last 290.25) - open/close 15335 (time 1753442992.86 total 55.47 last 243.18) - open/close 18193 (time 1753443002.87 total 65.48 last 285.70) - open/close 20000 (time 1753443009.29 total 71.90 last 281.49) - open/close 22924 (time 1753443019.29 total 81.90 last 292.37) - open/close 26757 (time 1753443029.29 total 91.90 last 383.27) - open/close 29415 (time 1753443039.29 total 101.90 last 265.68) - open/close 30000 (time 1753443045.25 total 107.86 last 98.17) - open/close 31005 (time 1753443055.25 total 117.86 last 100.46) - open/close 32424 (time 1753443065.26 total 127.87 last 141.80) - open/close 33319 (time 1753443075.27 total 137.88 last 89.39) - open/close 34028 (time 1753443085.29 total 147.90 last 70.81) - open/close 34556 (time 1753443095.30 total 157.91 last 52.74) - open/close 35104 (time 1753443105.31 total 167.92 last 54.75) - open/close 35738 (time 1753443115.32 total 177.93 last 63.35) - open/close 36442 (time 1753443125.32 total 187.93 last 70.36) - open/close 37251 (time 1753443135.32 total 197.93 last 80.89) - open/close 38034 (time 1753443145.32 total 207.93 last 78.30) - open/close 38886 (time 1753443155.33 total 217.94 last 85.18) - open/close 39737 (time 1753443165.33 total 227.94 last 85.08) - open/close 40000 (time 1753443168.60 total 231.21 last 80.37) - open/close 40883 (time 1753443178.61 total 241.22 last 88.25) - open/close 41730 (time 1753443188.61 total 251.22 last 84.69) - open/close 42581 (time 1753443198.61 total 261.22 last 85.05) - open/close 43425 (time 1753443208.62 total 271.23 last 84.35) - open/close 44268 (time 1753443218.63 total 281.24 last 84.17) - open/close 45075 (time 1753443228.64 total 291.25 last 80.70) - open/close 45878 (time 1753443238.65 total 301.26 last 80.18) - open/close 46740 (time 1753443248.66 total 311.27 last 86.14) - open/close 47676 (time 1753443258.66 total 321.27 last 93.57) - open/close 48484 (time 1753443268.67 total 331.28 last 80.73) - open/close 49332 (time 1753443278.68 total 341.29 last 84.74) - open/close 50000 (time 1753443286.75 total 349.36 last 82.71) - open/close 50799 (time 1753443296.76 total 359.37 last 79.86) - open/close 51612 (time 1753443306.79 total 369.40 last 81.08) - open/close 52364 (time 1753443316.79 total 379.40 last 75.20) - open/close 53065 (time 1753443326.79 total 389.40 last 70.05) - open/close 53784 (time 1753443336.80 total 399.41 last 71.88) - open/close 54615 (time 1753443346.80 total 409.41 last 83.03) - open/close 55371 (time 1753443356.81 total 419.42 last 75.57) - open/close 56122 (time 1753443366.81 total 429.42 last 75.08) - open/close 56909 (time 1753443376.82 total 439.43 last 78.65) - open/close 57712 (time 1753443386.83 total 449.44 last 80.18) - open/close 58505 (time 1753443396.84 total 459.45 last 79.26) - open/close 59108 (time 1753443406.85 total 469.46 last 60.23) - open/close 59696 (time 1753443416.85 total 479.46 last 58.76) - open/close 60000 (time 1753443422.80 total 485.41 last 51.10) - open/close 60559 (time 1753443432.81 total 495.42 last 55.87) - open/close 61230 (time 1753443442.82 total 505.43 last 67.05) - open/close 61941 (time 1753443452.82 total 515.43 last 71.10) - open/close 62701 (time 1753443462.83 total 525.44 last 75.93) - open/close 63423 (time 1753443472.83 total 535.44 last 72.17) - open/close 64150 (time 1753443482.84 total 545.45 last 72.67) total: 64768 open/close in 553.70 seconds: 116.97 ops/second - /unlink 1317 (time 1753443504.16 total 10.00 last 131.64) - /unlink 2601 (time 1753443514.17 total 20.02 last 128.24) - /unlink 3868 (time 1753443524.18 total 30.03 last 126.53) - /unlink 5135 (time 1753443534.18 total 40.03 last 126.65) - /unlink 6392 (time 1753443544.19 total 50.04 last 125.62) - /unlink 7615 (time 1753443554.20 total 60.05 last 122.16) - /unlink 8862 (time 1753443564.22 total 70.07 last 124.54) - /unlink 9965 (time 1753443574.22 total 80.07 last 110.26) - /unlink 10681 (time 1753443584.23 total 90.08 last 71.56) - /unlink 11400 (time 1753443594.23 total 100.08 last 71.87) - /unlink 12125 (time 1753443604.23 total 110.08 last 72.49) - /unlink 12795 (time 1753443614.23 total 120.08 last 66.99) - /unlink 13507 (time 1753443624.25 total 130.10 last 71.09) - /unlink 14201 (time 1753443634.27 total 140.12 last 69.28) - /unlink 14970 (time 1753443644.27 total 150.12 last 76.86) - /unlink 15681 (time 1753443654.28 total 160.12 last 71.07) - /unlink 16371 (time 1753443664.28 total 170.13 last 68.99) - /unlink 17087 (time 1753443674.28 total 180.13 last 71.59) - /unlink 17824 (time 1753443684.28 total 190.13 last 73.67) - /unlink 18561 (time 1753443694.29 total 200.14 last 73.65) - /unlink 19328 (time 1753443704.30 total 210.14 last 76.65) - /unlink 20000 (time 1753443714.04 total 219.89 last 68.97) - /unlink 20721 (time 1753443724.04 total 229.89 last 72.05) - /unlink 21438 (time 1753443734.05 total 239.90 last 71.68) - /unlink 22139 (time 1753443744.05 total 249.90 last 70.08) - /unlink 22910 (time 1753443754.06 total 259.90 last 77.06) - /unlink 23650 (time 1753443764.07 total 269.91 last 73.93) - /unlink 24370 (time 1753443774.08 total 279.92 last 71.92) - /unlink 25122 (time 1753443784.09 total 289.94 last 75.11) - /unlink 25879 (time 1753443794.09 total 299.94 last 75.69) - /unlink 26612 (time 1753443804.10 total 309.94 last 73.26) - /unlink 27362 (time 1753443814.10 total 319.95 last 74.97) - /unlink 28163 (time 1753443824.11 total 329.96 last 80.03) - /unlink 28938 (time 1753443834.15 total 340.00 last 77.16) - /unlink 29751 (time 1753443844.15 total 350.00 last 81.28) - /unlink 30000 (time 1753443847.73 total 353.58 last 69.53) - /unlink 30786 (time 1753443857.73 total 363.58 last 78.60) - /unlink 31498 (time 1753443867.74 total 373.58 last 71.19) - /unlink 32239 (time 1753443877.74 total 383.58 last 74.10) - /unlink 33040 (time 1753443887.75 total 393.59 last 80.03) - /unlink 33785 (time 1753443897.75 total 403.60 last 74.46) - /unlink 34534 (time 1753443907.76 total 413.61 last 74.85) - /unlink 35277 (time 1753443917.76 total 423.61 last 74.29) - /unlink 36022 (time 1753443927.76 total 433.61 last 74.47) - /unlink 36766 (time 1753443937.77 total 443.62 last 74.37) - /unlink 37505 (time 1753443947.77 total 453.62 last 73.86) - /unlink 38196 (time 1753443957.78 total 463.63 last 69.07) - /unlink 38898 (time 1753443967.79 total 473.63 last 70.13) - /unlink 39635 (time 1753443977.79 total 483.64 last 73.66) - /unlink 40000 (time 1753443982.40 total 488.25 last 79.18) - /unlink 40741 (time 1753443992.41 total 498.26 last 74.02) - /unlink 41453 (time 1753444002.42 total 508.27 last 71.11) - /unlink 42172 (time 1753444012.43 total 518.28 last 71.87) - /unlink 42895 (time 1753444022.43 total 528.28 last 72.28) - /unlink 43613 (time 1753444032.44 total 538.29 last 71.76) - /unlink 44359 (time 1753444042.44 total 548.29 last 74.59) - /unlink 45122 (time 1753444052.44 total 558.29 last 76.25) - /unlink 45832 (time 1753444062.46 total 568.30 last 70.92) - /unlink 46505 (time 1753444072.48 total 578.33 last 67.13) - /unlink 47224 (time 1753444082.50 total 588.35 last 71.76) - /unlink 47977 (time 1753444092.52 total 598.37 last 75.17) - /unlink 48722 (time 1753444102.53 total 608.38 last 74.41) - /unlink 49437 (time 1753444112.53 total 618.38 last 71.48) - /unlink 50000 (time 1753444120.38 total 626.23 last 71.75) - /unlink 50753 (time 1753444130.39 total 636.24 last 75.24) - /unlink 51473 (time 1753444140.39 total 646.24 last 71.96) - /unlink 52144 (time 1753444150.41 total 656.26 last 66.96) - /unlink 52870 (time 1753444160.42 total 666.27 last 72.55) - /unlink 53628 (time 1753444170.42 total 676.27 last 75.77) - /unlink 54377 (time 1753444180.43 total 686.28 last 74.84) - /unlink 55158 (time 1753444190.44 total 696.28 last 78.08) - /unlink 55905 (time 1753444200.44 total 706.29 last 74.66) - /unlink 56663 (time 1753444210.45 total 716.30 last 75.74) - /unlink 57396 (time 1753444220.46 total 726.31 last 73.21) - /unlink 58163 (time 1753444230.47 total 736.32 last 76.64) - /unlink 58903 (time 1753444240.48 total 746.33 last 73.88) - /unlink 59672 (time 1753444250.49 total 756.34 last 76.87) - /unlink 60000 (time 1753444254.94 total 760.79 last 73.62) - /unlink 60728 (time 1753444264.96 total 770.81 last 72.71) - /unlink 61420 (time 1753444274.96 total 780.81 last 69.16) - /unlink 62180 (time 1753444284.97 total 790.81 last 75.97) - /unlink 62870 (time 1753444294.99 total 800.83 last 68.86) - /unlink 63641 (time 1753444304.99 total 810.83 last 77.10) - /unlink 64399 (time 1753444315.00 total 820.85 last 75.71) total: 64768 /unlink in 825.68 seconds: 78.44 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg137-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg137-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg137-server stop mds service on oleg137-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg137-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg137-server' oleg137-server: oleg137-server.virtnet: executing unload_modules_local oleg137-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg137-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg137-client: oleg137-client: ssh exited with exit code 2 pdsh@oleg137-client: oleg137-server: ssh exited with exit code 2 pdsh@oleg137-client: oleg137-client: ssh exited with exit code 2 pdsh@oleg137-client: oleg137-server: ssh exited with exit code 2