== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 05:45:39 (1765104339) Stopping clients: oleg441-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg441-client.virtnet /mnt/lustre2 (opts:-f) oleg441-server: oleg441-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg441-server' oleg441-server: oleg441-server.virtnet: executing load_modules_local oleg441-server: Loading modules from /home/green/git/lustre-release/lustre oleg441-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg441-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg441-server: detected 4 online CPUs by sysfs oleg441-server: MODOPTS_LIBCFS= oleg441-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg441-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg441-server Start mds2: mount -t lustre -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg441-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg441-client.virtnet: -o user_xattr,flock 192.168.204.141@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86890 331 86559 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 87086 312 86774 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32714 458 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32899 643 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 917 (time 1765104428.93 total 10.02 last 91.55) - open/close 1900 (time 1765104438.94 total 20.03 last 98.21) - open/close 2936 (time 1765104448.94 total 30.03 last 103.56) - open/close 3823 (time 1765104458.95 total 40.04 last 88.65) - open/close 4830 (time 1765104468.95 total 50.04 last 100.68) - open/close 5836 (time 1765104478.95 total 60.04 last 100.55) - open/close 6837 (time 1765104488.95 total 70.04 last 100.09) - open/close 7832 (time 1765104498.96 total 80.05 last 99.48) - open/close 8893 (time 1765104508.96 total 90.05 last 106.07) - open/close 10000 (time 1765104518.68 total 99.77 last 113.92) - open/close 11114 (time 1765104528.68 total 109.77 last 111.38) - open/close 12210 (time 1765104538.68 total 119.77 last 109.57) - open/close 13257 (time 1765104548.69 total 129.78 last 104.62) - open/close 14445 (time 1765104558.70 total 139.79 last 118.66) - open/close 15534 (time 1765104568.70 total 149.79 last 108.87) - open/close 16593 (time 1765104578.71 total 159.80 last 105.87) - open/close 17636 (time 1765104588.71 total 169.80 last 104.26) - open/close 18560 (time 1765104598.72 total 179.81 last 92.34) - open/close 19615 (time 1765104608.73 total 189.82 last 105.38) - open/close 20000 (time 1765104612.75 total 193.84 last 95.68) - open/close 20855 (time 1765104622.76 total 203.85 last 85.46) - open/close 21672 (time 1765104632.76 total 213.85 last 81.66) - open/close 22575 (time 1765104642.77 total 223.86 last 90.20) - open/close 23426 (time 1765104652.78 total 233.87 last 85.02) - open/close 24314 (time 1765104662.79 total 243.88 last 88.76) - open/close 25205 (time 1765104672.80 total 253.88 last 89.03) - open/close 26069 (time 1765104682.80 total 263.89 last 86.38) - open/close 26904 (time 1765104692.81 total 273.89 last 83.43) - open/close 27737 (time 1765104702.81 total 283.90 last 83.27) - open/close 28590 (time 1765104712.82 total 293.91 last 85.20) - open/close 29423 (time 1765104722.82 total 303.91 last 83.29) - open/close 30000 (time 1765104730.12 total 311.21 last 79.08) - open/close 30871 (time 1765104740.12 total 321.21 last 87.07) - open/close 31750 (time 1765104750.13 total 331.22 last 87.86) - open/close 32711 (time 1765104760.13 total 341.22 last 96.06) - open/close 33627 (time 1765104770.13 total 351.22 last 91.57) - open/close 34612 (time 1765104780.13 total 361.22 last 98.48) - open/close 35613 (time 1765104790.14 total 371.22 last 100.08) - open/close 36568 (time 1765104800.14 total 381.23 last 95.48) - open/close 37484 (time 1765104810.14 total 391.23 last 91.55) - open/close 38354 (time 1765104820.15 total 401.24 last 86.96) - open/close 39159 (time 1765104830.15 total 411.24 last 80.48) - open/close 40000 (time 1765104839.62 total 420.71 last 88.81) - open/close 40863 (time 1765104849.62 total 430.71 last 86.28) - open/close 41679 (time 1765104859.63 total 440.72 last 81.54) - open/close 42496 (time 1765104869.64 total 450.73 last 81.58) - open/close 43391 (time 1765104879.65 total 460.74 last 89.45) - open/close 44190 (time 1765104889.66 total 470.75 last 79.82) - open/close 45051 (time 1765104899.67 total 480.76 last 86.02) - open/close 45945 (time 1765104909.67 total 490.76 last 89.37) - open/close 46796 (time 1765104919.68 total 500.77 last 85.04) - open/close 47579 (time 1765104929.68 total 510.77 last 78.27) - open/close 48324 (time 1765104939.69 total 520.78 last 74.47) - open/close 49204 (time 1765104949.69 total 530.78 last 87.98) - open/close 50000 (time 1765104957.91 total 539.00 last 96.88) - open/close 51027 (time 1765104967.91 total 549.00 last 102.62) - open/close 51841 (time 1765104977.92 total 559.01 last 81.32) - open/close 52672 (time 1765104987.93 total 569.01 last 83.09) - open/close 53553 (time 1765104997.94 total 579.03 last 87.98) - open/close 54375 (time 1765105007.94 total 589.03 last 82.17) - open/close 55234 (time 1765105017.96 total 599.05 last 85.76) - open/close 56130 (time 1765105027.97 total 609.06 last 89.52) - open/close 57103 (time 1765105037.98 total 619.07 last 97.22) - open/close 58018 (time 1765105047.98 total 629.07 last 91.43) - open/close 58945 (time 1765105057.98 total 639.07 last 92.69) - open/close 59969 (time 1765105067.98 total 649.07 last 102.40) - open/close 61081 (time 1765105077.99 total 659.08 last 111.10) - open/close 62069 (time 1765105088.00 total 669.09 last 98.77) - open/close 63035 (time 1765105098.00 total 679.09 last 96.55) - open/close 64080 (time 1765105108.01 total 689.10 last 104.39) total: 64768 open/close in 695.60 seconds: 93.11 ops/second - /unlink 1101 (time 1765105126.81 total 10.00 last 110.10) - /unlink 2193 (time 1765105136.81 total 20.00 last 109.16) - /unlink 3196 (time 1765105146.81 total 30.00 last 100.30) - /unlink 4457 (time 1765105156.82 total 40.01 last 126.03) - /unlink 5630 (time 1765105166.82 total 50.02 last 117.24) - /unlink 6536 (time 1765105176.82 total 60.02 last 90.59) - /unlink 7588 (time 1765105186.83 total 70.02 last 105.16) - /unlink 8741 (time 1765105196.83 total 80.03 last 115.23) - /unlink 9740 (time 1765105206.84 total 90.03 last 99.86) - /unlink 10000 (time 1765105209.91 total 93.10 last 84.57) - /unlink 10851 (time 1765105219.92 total 103.11 last 85.05) - /unlink 11825 (time 1765105229.92 total 113.12 last 97.34) - /unlink 12940 (time 1765105239.93 total 123.12 last 111.48) - /unlink 14130 (time 1765105249.94 total 133.13 last 118.87) - /unlink 15204 (time 1765105259.94 total 143.13 last 107.40) - /unlink 16260 (time 1765105269.94 total 153.14 last 105.53) - /unlink 17232 (time 1765105279.95 total 163.14 last 97.13) - /unlink 18406 (time 1765105289.95 total 173.15 last 117.35) - /unlink 19477 (time 1765105299.96 total 183.15 last 107.02) - /unlink 20000 (time 1765105305.24 total 188.43 last 99.04) - /unlink 20911 (time 1765105315.24 total 198.44 last 91.09) - /unlink 21981 (time 1765105325.25 total 208.44 last 106.93) - /unlink 22983 (time 1765105335.25 total 218.45 last 100.16) - /unlink 24057 (time 1765105345.27 total 228.46 last 107.22) - /unlink 25160 (time 1765105355.27 total 238.46 last 110.28) - /unlink 26355 (time 1765105365.27 total 248.47 last 119.47) - /unlink 27463 (time 1765105375.28 total 258.47 last 110.73) - /unlink 28483 (time 1765105385.28 total 268.47 last 102.00) - /unlink 29342 (time 1765105395.28 total 278.47 last 85.89) - /unlink 30000 (time 1765105402.16 total 285.35 last 95.64) - /unlink 31250 (time 1765105412.17 total 295.36 last 124.94) - /unlink 32359 (time 1765105422.17 total 305.36 last 110.87) - /unlink 33327 (time 1765105432.17 total 315.37 last 96.74) - /unlink 34384 (time 1765105442.18 total 325.37 last 105.66) - /unlink 35370 (time 1765105452.18 total 335.38 last 98.53) - /unlink 36367 (time 1765105462.19 total 345.38 last 99.65) - /unlink 37392 (time 1765105472.20 total 355.39 last 102.45) - /unlink 38368 (time 1765105482.21 total 365.40 last 97.48) - /unlink 39389 (time 1765105492.22 total 375.41 last 102.02) - /unlink 40000 (time 1765105498.78 total 381.98 last 93.04) - /unlink 40920 (time 1765105508.79 total 391.98 last 91.98) - /unlink 41901 (time 1765105518.80 total 401.99 last 98.00) - /unlink 42887 (time 1765105528.80 total 411.99 last 98.57) - /unlink 43894 (time 1765105538.81 total 422.00 last 100.60) - /unlink 44827 (time 1765105548.82 total 432.01 last 93.22) - /unlink 45696 (time 1765105558.83 total 442.02 last 86.81) - /unlink 46612 (time 1765105568.83 total 452.02 last 91.59) - /unlink 47741 (time 1765105578.83 total 462.03 last 112.84) - /unlink 48874 (time 1765105588.84 total 472.04 last 113.19) - /unlink 49756 (time 1765105598.85 total 482.04 last 88.17) - /unlink 50000 (time 1765105601.49 total 484.69 last 92.21) - /unlink 51009 (time 1765105611.53 total 494.73 last 100.50) - /unlink 52005 (time 1765105621.53 total 504.73 last 99.60) - /unlink 53234 (time 1765105631.53 total 514.73 last 122.87) - /unlink 54468 (time 1765105641.54 total 524.73 last 123.33) - /unlink 56005 (time 1765105651.55 total 534.74 last 153.61) - /unlink 57147 (time 1765105661.56 total 544.75 last 114.08) - /unlink 58113 (time 1765105671.56 total 554.75 last 96.58) - /unlink 59205 (time 1765105681.57 total 564.76 last 109.11) - /unlink 60000 (time 1765105690.26 total 573.45 last 91.48) - /unlink 61048 (time 1765105700.26 total 583.46 last 104.74) - /unlink 62116 (time 1765105710.27 total 593.46 last 106.74) - /unlink 63203 (time 1765105720.28 total 603.47 last 108.61) - /unlink 64412 (time 1765105730.28 total 613.48 last 120.82) total: 64768 /unlink in 615.86 seconds: 105.17 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1211 (time 1765105767.58 total 10.00 last 121.04) - open/close 2518 (time 1765105777.58 total 20.01 last 130.66) - open/close 3752 (time 1765105787.58 total 30.01 last 123.39) - open/close 5048 (time 1765105797.59 total 40.01 last 129.55) - open/close 6596 (time 1765105807.59 total 50.02 last 154.74) - open/close 8160 (time 1765105817.59 total 60.02 last 156.39) - open/close 9823 (time 1765105827.59 total 70.02 last 166.29) - open/close 11187 (time 1765105837.59 total 80.02 last 136.40) - open/close 12553 (time 1765105847.60 total 90.02 last 136.50) - open/close 14356 (time 1765105857.60 total 100.03 last 180.20) - open/close 15801 (time 1765105867.61 total 110.03 last 144.47) - open/close 17241 (time 1765105877.61 total 120.03 last 143.98) - open/close 18607 (time 1765105887.61 total 130.04 last 136.58) - open/close 20000 (time 1765105896.23 total 138.65 last 161.66) - open/close 21596 (time 1765105906.23 total 148.65 last 159.58) - open/close 23017 (time 1765105916.23 total 158.66 last 142.06) - open/close 24353 (time 1765105926.23 total 168.66 last 133.57) - open/close 25550 (time 1765105936.23 total 178.66 last 119.69) - open/close 26812 (time 1765105946.24 total 188.66 last 126.15) - open/close 28317 (time 1765105956.24 total 198.67 last 150.43) - open/close 30000 (time 1765105964.50 total 206.93 last 203.73) - open/close 32152 (time 1765105974.50 total 216.93 last 215.18) - open/close 33911 (time 1765105984.50 total 226.93 last 175.87) - open/close 35165 (time 1765105994.51 total 236.94 last 125.31) - open/close 36610 (time 1765106004.51 total 246.94 last 144.47) - open/close 37964 (time 1765106014.51 total 256.94 last 135.40) - open/close 39367 (time 1765106024.52 total 266.95 last 140.21) - open/close 40000 (time 1765106028.57 total 271.00 last 156.34) - open/close 41794 (time 1765106038.57 total 281.00 last 179.38) - open/close 43547 (time 1765106048.57 total 291.00 last 175.25) - open/close 44969 (time 1765106058.58 total 301.01 last 142.12) - open/close 46434 (time 1765106068.58 total 311.01 last 146.48) - open/close 48014 (time 1765106078.59 total 321.01 last 157.91) - open/close 49738 (time 1765106088.59 total 331.02 last 172.35) - open/close 51074 (time 1765106098.60 total 341.02 last 133.52) - open/close 52373 (time 1765106108.60 total 351.03 last 129.85) - open/close 53690 (time 1765106118.61 total 361.03 last 131.59) - open/close 55501 (time 1765106128.62 total 371.04 last 180.91) - open/close 57299 (time 1765106138.62 total 381.05 last 179.73) - open/close 59406 (time 1765106148.63 total 391.05 last 210.63) - open/close 60000 (time 1765106152.04 total 394.46 last 174.13) - open/close 61921 (time 1765106162.04 total 404.47 last 192.03) - open/close 63484 (time 1765106172.04 total 414.47 last 156.28) total: 64768 open/close in 422.53 seconds: 153.28 ops/second - /unlink 2720 (time 1765106192.01 total 10.00 last 271.96) - /unlink 5574 (time 1765106202.01 total 20.00 last 285.35) - /unlink 9035 (time 1765106212.01 total 30.01 last 346.03) - /unlink 10000 (time 1765106215.04 total 33.03 last 318.67) - /unlink 13021 (time 1765106225.04 total 43.03 last 302.08) - /unlink 16452 (time 1765106235.04 total 53.04 last 343.03) - /unlink 19828 (time 1765106245.04 total 63.04 last 337.52) - /unlink 23083 (time 1765106255.05 total 73.04 last 325.41) - /unlink 26622 (time 1765106265.05 total 83.04 last 353.84) - /unlink 30000 (time 1765106274.93 total 92.93 last 341.68) - /unlink 34183 (time 1765106284.93 total 102.93 last 418.27) - /unlink 38112 (time 1765106294.94 total 112.93 last 392.85) - /unlink 40000 (time 1765106300.25 total 118.25 last 355.16) - /unlink 43494 (time 1765106310.25 total 128.25 last 349.37) - /unlink 47723 (time 1765106320.25 total 138.25 last 422.88) - /unlink 50000 (time 1765106326.63 total 144.62 last 357.23) - /unlink 52552 (time 1765106336.63 total 154.62 last 255.16) - /unlink 55665 (time 1765106346.63 total 164.63 last 311.22) - /unlink 59531 (time 1765106356.63 total 174.63 last 386.54) - /unlink 63724 (time 1765106366.64 total 184.63 last 419.20) total: 64768 /unlink in 186.82 seconds: 346.68 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2440 (time 1765106400.33 total 10.00 last 243.93) - open/close 4524 (time 1765106410.33 total 20.01 last 208.34) - open/close 6801 (time 1765106420.34 total 30.01 last 227.60) - open/close 9240 (time 1765106430.34 total 40.01 last 243.86) - open/close 10000 (time 1765106432.97 total 42.64 last 289.25) - open/close 12603 (time 1765106442.97 total 52.64 last 260.18) - open/close 14895 (time 1765106452.97 total 62.65 last 229.17) - open/close 17677 (time 1765106462.97 total 72.65 last 278.17) - open/close 20000 (time 1765106468.55 total 78.22 last 416.79) - open/close 23821 (time 1765106478.55 total 88.22 last 382.07) - open/close 28284 (time 1765106488.55 total 98.22 last 446.21) - open/close 30000 (time 1765106493.08 total 102.75 last 379.16) - open/close 33901 (time 1765106503.08 total 112.75 last 389.98) - open/close 36977 (time 1765106513.08 total 122.75 last 307.57) - open/close 40000 (time 1765106521.02 total 130.69 last 380.87) - open/close 43498 (time 1765106531.02 total 140.69 last 349.79) - open/close 47003 (time 1765106541.02 total 150.69 last 350.49) - open/close 50000 (time 1765106550.12 total 159.80 last 329.14) - open/close 52782 (time 1765106560.13 total 169.80 last 278.16) - open/close 56799 (time 1765106570.13 total 179.80 last 401.63) - open/close 60000 (time 1765106578.05 total 187.72 last 404.12) - open/close 64214 (time 1765106588.05 total 197.72 last 421.34) total: 64768 open/close in 199.10 seconds: 325.31 ops/second - /unlink 6606 (time 1765106600.81 total 10.00 last 660.47) - /unlink 10000 (time 1765106606.46 total 15.65 last 600.54) - /unlink 16424 (time 1765106616.46 total 25.65 last 642.34) - /unlink 20000 (time 1765106620.53 total 29.73 last 878.10) - /unlink 26076 (time 1765106630.53 total 39.73 last 607.52) - /unlink 30000 (time 1765106637.18 total 46.37 last 590.39) - /unlink 35514 (time 1765106647.18 total 56.38 last 551.32) - /unlink 40000 (time 1765106653.88 total 63.08 last 669.65) - /unlink 45655 (time 1765106663.88 total 73.08 last 565.47) - /unlink 50000 (time 1765106670.79 total 79.98 last 629.18) - /unlink 54355 (time 1765106680.79 total 89.98 last 435.44) - /unlink 58048 (time 1765106690.79 total 99.99 last 369.21) - /unlink 60000 (time 1765106696.29 total 105.48 last 355.01) - /unlink 63585 (time 1765106706.29 total 115.49 last 358.44) total: 64768 /unlink in 118.68 seconds: 545.73 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2508 (time 1765106735.71 total 10.00 last 250.76) - open/close 5133 (time 1765106745.72 total 20.00 last 262.46) - open/close 8001 (time 1765106755.72 total 30.00 last 286.77) - open/close 10000 (time 1765106761.68 total 35.96 last 335.36) - open/close 12720 (time 1765106771.68 total 45.97 last 271.96) - open/close 15317 (time 1765106781.68 total 55.97 last 259.62) - open/close 17819 (time 1765106791.68 total 65.97 last 250.17) - open/close 20000 (time 1765106797.84 total 72.13 last 354.11) - open/close 23116 (time 1765106807.84 total 82.13 last 311.59) - open/close 25503 (time 1765106817.85 total 92.13 last 238.65) - open/close 28030 (time 1765106827.85 total 102.13 last 252.66) - open/close 30000 (time 1765106834.41 total 108.69 last 300.34) - open/close 33302 (time 1765106844.41 total 118.70 last 330.12) - open/close 36527 (time 1765106854.41 total 128.70 last 322.47) - open/close 40000 (time 1765106864.02 total 138.31 last 361.40) - open/close 44401 (time 1765106874.02 total 148.31 last 440.06) - open/close 48130 (time 1765106884.02 total 158.31 last 372.83) - open/close 50000 (time 1765106888.77 total 163.06 last 393.48) - open/close 54299 (time 1765106898.78 total 173.06 last 429.87) - open/close 58433 (time 1765106908.79 total 183.08 last 412.83) - open/close 60000 (time 1765106912.24 total 186.53 last 453.67) - open/close 64161 (time 1765106922.24 total 196.53 last 416.09) total: 64768 open/close in 198.12 seconds: 326.91 ops/second - /unlink 5167 (time 1765106935.31 total 10.00 last 516.62) - /unlink 10000 (time 1765106942.46 total 17.15 last 675.89) - /unlink 16169 (time 1765106952.46 total 27.15 last 616.83) - /unlink 20000 (time 1765106957.97 total 32.66 last 695.18) - /unlink 26427 (time 1765106967.97 total 42.66 last 642.66) - /unlink 30000 (time 1765106973.20 total 47.90 last 683.14) - /unlink 36237 (time 1765106983.20 total 57.90 last 623.67) - /unlink 40000 (time 1765106989.40 total 64.09 last 607.78) - /unlink 45937 (time 1765106999.40 total 74.09 last 593.66) - /unlink 50000 (time 1765107005.88 total 80.57 last 626.98) - /unlink 55979 (time 1765107015.88 total 90.57 last 597.78) - /unlink 60000 (time 1765107022.97 total 97.66 last 567.28) total: 64768 /unlink in 104.02 seconds: 622.66 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3498 (time 1765107060.78 total 10.00 last 349.75) - open/close 7385 (time 1765107070.78 total 20.00 last 388.68) - open/close 10000 (time 1765107077.96 total 27.18 last 364.21) - open/close 12555 (time 1765107087.96 total 37.18 last 255.47) - open/close 17207 (time 1765107097.96 total 47.18 last 465.20) - open/close 20000 (time 1765107105.01 total 54.24 last 396.01) - open/close 23830 (time 1765107115.01 total 64.24 last 382.95) - open/close 27687 (time 1765107125.01 total 74.24 last 385.67) - open/close 30000 (time 1765107130.89 total 80.12 last 393.41) - open/close 34305 (time 1765107140.89 total 90.12 last 430.48) - open/close 39042 (time 1765107150.89 total 100.12 last 473.66) - open/close 40000 (time 1765107153.15 total 102.37 last 425.06) - open/close 44235 (time 1765107163.15 total 112.37 last 423.46) - open/close 48561 (time 1765107173.15 total 122.38 last 432.54) - open/close 50000 (time 1765107176.74 total 125.96 last 400.91) - open/close 54246 (time 1765107186.74 total 135.97 last 424.57) - open/close 58818 (time 1765107196.74 total 145.97 last 457.17) - open/close 60000 (time 1765107199.45 total 148.67 last 436.92) - open/close 64541 (time 1765107209.45 total 158.67 last 454.07) total: 64768 open/close in 159.23 seconds: 406.75 ops/second - /unlink 7135 (time 1765107221.33 total 10.00 last 713.47) - /unlink 10000 (time 1765107225.06 total 13.73 last 767.69) - /unlink 17223 (time 1765107235.06 total 23.73 last 722.25) - /unlink 20000 (time 1765107238.82 total 27.49 last 739.07) - /unlink 27236 (time 1765107248.82 total 37.49 last 723.55) - /unlink 30000 (time 1765107253.24 total 41.91 last 625.57) - /unlink 37156 (time 1765107263.24 total 51.91 last 715.58) - /unlink 40000 (time 1765107267.38 total 56.06 last 685.83) - /unlink 47031 (time 1765107277.38 total 66.06 last 703.09) - /unlink 50000 (time 1765107281.22 total 69.90 last 773.14) - /unlink 56484 (time 1765107291.22 total 79.90 last 648.39) - /unlink 60000 (time 1765107296.61 total 85.28 last 652.76) total: 64768 /unlink in 92.25 seconds: 702.13 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg441-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg441-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg441-server stop mds service on oleg441-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg441-server stop mds service on oleg441-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg441-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg441-server' oleg441-server: oleg441-server.virtnet: executing unload_modules_local oleg441-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg441-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg441-client: oleg441-client: ssh exited with exit code 2 pdsh@oleg441-client: oleg441-server: ssh exited with exit code 2 pdsh@oleg441-client: oleg441-client: ssh exited with exit code 2 pdsh@oleg441-client: oleg441-server: ssh exited with exit code 2