== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 20:37:30 (1746146250) Stopping clients: oleg229-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg229-client.virtnet /mnt/lustre2 (opts:-f) oleg229-server: oleg229-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory loading modules on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing load_modules_local oleg229-server: Loading modules from /home/green/git/lustre-release/lustre oleg229-server: detected 4 online CPUs by sysfs oleg229-server: MODOPTS_LIBCFS= oleg229-server: Force libcfs to create 2 CPU partitions oleg229-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg229-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg229-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg229-client.virtnet: -o user_xattr,flock 192.168.202.129@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1282 (time 1746146302.96 total 10.01 last 128.11) - open/close 2575 (time 1746146312.96 total 20.01 last 129.28) - open/close 4045 (time 1746146322.96 total 30.01 last 146.98) - open/close 5312 (time 1746146332.96 total 40.01 last 126.69) - open/close 6649 (time 1746146342.97 total 50.02 last 133.62) - open/close 7968 (time 1746146352.97 total 60.02 last 131.84) - open/close 9241 (time 1746146362.97 total 70.02 last 127.26) - open/close 10000 (time 1746146369.06 total 76.11 last 124.78) - open/close 11212 (time 1746146379.06 total 86.11 last 121.13) - open/close 12417 (time 1746146389.08 total 96.13 last 120.36) - open/close 13488 (time 1746146399.09 total 106.14 last 106.99) - open/close 14721 (time 1746146409.09 total 116.14 last 123.23) - open/close 15805 (time 1746146419.10 total 126.15 last 108.32) - open/close 16837 (time 1746146429.10 total 136.15 last 103.18) - open/close 18043 (time 1746146439.11 total 146.16 last 120.53) - open/close 19262 (time 1746146449.11 total 156.16 last 121.84) - open/close 20000 (time 1746146454.79 total 161.84 last 129.91) - open/close 21307 (time 1746146464.79 total 171.84 last 130.69) - open/close 22568 (time 1746146474.80 total 181.85 last 126.00) - open/close 23711 (time 1746146484.80 total 191.85 last 114.30) - open/close 24914 (time 1746146494.80 total 201.85 last 120.27) - open/close 26061 (time 1746146504.81 total 211.86 last 114.66) - open/close 27058 (time 1746146514.81 total 221.86 last 99.66) - open/close 28200 (time 1746146524.81 total 231.86 last 114.20) - open/close 29294 (time 1746146534.82 total 241.87 last 109.36) - open/close 30000 (time 1746146542.23 total 249.28 last 95.22) - open/close 31010 (time 1746146552.23 total 259.28 last 100.99) - open/close 32078 (time 1746146562.23 total 269.28 last 106.78) - open/close 33058 (time 1746146572.23 total 279.28 last 98.00) - open/close 33994 (time 1746146582.24 total 289.29 last 93.50) - open/close 34955 (time 1746146592.25 total 299.30 last 96.06) - open/close 35970 (time 1746146602.25 total 309.30 last 101.46) - open/close 36987 (time 1746146612.27 total 319.32 last 101.56) - open/close 37888 (time 1746146622.27 total 329.32 last 90.02) - open/close 38809 (time 1746146632.28 total 339.33 last 92.07) - open/close 39750 (time 1746146642.29 total 349.34 last 94.03) - open/close 40000 (time 1746146645.04 total 352.09 last 90.87) - open/close 41003 (time 1746146655.05 total 362.10 last 100.22) - open/close 42026 (time 1746146665.05 total 372.10 last 102.22) - open/close 43107 (time 1746146675.06 total 382.11 last 108.07) - open/close 44127 (time 1746146685.06 total 392.11 last 101.95) - open/close 45087 (time 1746146695.07 total 402.12 last 95.93) - open/close 46123 (time 1746146705.07 total 412.12 last 103.55) - open/close 47147 (time 1746146715.08 total 422.13 last 102.38) - open/close 48191 (time 1746146725.08 total 432.13 last 104.39) - open/close 49345 (time 1746146735.08 total 442.13 last 115.36) - open/close 50000 (time 1746146739.64 total 446.69 last 143.59) - open/close 51386 (time 1746146749.65 total 456.70 last 138.50) - open/close 52627 (time 1746146759.65 total 466.70 last 124.03) - open/close 53878 (time 1746146769.66 total 476.71 last 124.99) - open/close 54978 (time 1746146779.67 total 486.72 last 109.90) - open/close 56001 (time 1746146789.69 total 496.74 last 102.15) - open/close 57007 (time 1746146799.70 total 506.75 last 100.51) - open/close 57967 (time 1746146809.70 total 516.75 last 95.92) - open/close 58931 (time 1746146819.71 total 526.76 last 96.34) - open/close 60000 (time 1746146829.27 total 536.32 last 111.77) - open/close 61123 (time 1746146839.29 total 546.34 last 112.17) - open/close 62477 (time 1746146849.29 total 556.34 last 135.37) - open/close 63557 (time 1746146859.29 total 566.34 last 107.95) total: 64768 open/close in 576.00 seconds: 112.45 ops/second - /unlink 2113 (time 1746146880.33 total 10.00 last 211.27) - /unlink 4532 (time 1746146890.33 total 20.00 last 241.89) - /unlink 7196 (time 1746146900.33 total 30.00 last 266.38) - /unlink 9351 (time 1746146910.33 total 40.00 last 215.46) - /unlink 10000 (time 1746146913.53 total 43.20 last 202.92) - /unlink 12098 (time 1746146923.54 total 53.21 last 209.65) - /unlink 13889 (time 1746146933.54 total 63.21 last 179.07) - /unlink 16080 (time 1746146943.54 total 73.21 last 219.05) - /unlink 17830 (time 1746146953.54 total 83.21 last 174.99) - /unlink 19535 (time 1746146963.54 total 93.22 last 170.44) - /unlink 20000 (time 1746146966.35 total 96.03 last 165.36) - /unlink 21818 (time 1746146976.35 total 106.03 last 181.80) - /unlink 23930 (time 1746146986.36 total 116.03 last 211.17) - /unlink 25740 (time 1746146996.36 total 126.03 last 180.98) - /unlink 27624 (time 1746147006.37 total 136.04 last 188.23) - /unlink 29737 (time 1746147016.37 total 146.05 last 211.21) - /unlink 31829 (time 1746147026.38 total 156.05 last 209.09) - /unlink 33586 (time 1746147036.38 total 166.06 last 175.62) - /unlink 35359 (time 1746147046.38 total 176.06 last 177.28) - /unlink 37120 (time 1746147056.39 total 186.07 last 175.92) - /unlink 39154 (time 1746147066.39 total 196.07 last 203.34) - /unlink 40000 (time 1746147070.63 total 200.30 last 199.86) - /unlink 41874 (time 1746147080.63 total 210.30 last 187.37) - /unlink 43736 (time 1746147090.63 total 220.30 last 186.19) - /unlink 45679 (time 1746147100.64 total 230.31 last 194.15) - /unlink 47599 (time 1746147110.64 total 240.32 last 191.94) - /unlink 49456 (time 1746147120.64 total 250.32 last 185.64) - /unlink 50000 (time 1746147123.81 total 253.48 last 171.82) - /unlink 51719 (time 1746147133.81 total 263.49 last 171.82) - /unlink 53623 (time 1746147143.82 total 273.49 last 190.30) - /unlink 55236 (time 1746147153.82 total 283.50 last 161.28) - /unlink 57197 (time 1746147163.82 total 293.50 last 196.07) - /unlink 58954 (time 1746147173.82 total 303.50 last 175.66) - /unlink 60000 (time 1746147179.27 total 308.95 last 192.01) - /unlink 61806 (time 1746147189.28 total 318.95 last 180.53) - /unlink 63831 (time 1746147199.28 total 328.95 last 202.44) total: 64768 /unlink in 332.69 seconds: 194.68 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1094 (time 1746147230.29 total 10.00 last 109.36) - open/close 2163 (time 1746147240.30 total 20.01 last 106.85) - open/close 3206 (time 1746147250.30 total 30.01 last 104.28) - open/close 4240 (time 1746147260.32 total 40.03 last 103.20) - open/close 5349 (time 1746147270.33 total 50.03 last 110.84) - open/close 6442 (time 1746147280.33 total 60.04 last 109.26) - open/close 7550 (time 1746147290.33 total 70.04 last 110.79) - open/close 8546 (time 1746147300.34 total 80.05 last 99.54) - open/close 9665 (time 1746147310.34 total 90.05 last 111.84) - open/close 10000 (time 1746147313.23 total 92.94 last 116.04) - open/close 11271 (time 1746147323.24 total 102.95 last 126.99) - open/close 12412 (time 1746147333.24 total 112.95 last 114.08) - open/close 13550 (time 1746147343.25 total 122.96 last 113.67) - open/close 14647 (time 1746147353.25 total 132.96 last 109.69) - open/close 15721 (time 1746147363.26 total 142.97 last 107.35) - open/close 16720 (time 1746147373.26 total 152.97 last 99.85) - open/close 17739 (time 1746147383.26 total 162.97 last 101.90) - open/close 18885 (time 1746147393.27 total 172.98 last 114.56) - open/close 19948 (time 1746147403.27 total 182.98 last 106.27) - open/close 20989 (time 1746147413.27 total 192.98 last 104.09) - open/close 22108 (time 1746147423.27 total 202.98 last 111.87) - open/close 23292 (time 1746147433.27 total 212.98 last 118.38) - open/close 24414 (time 1746147443.28 total 222.99 last 112.18) - open/close 25565 (time 1746147453.29 total 233.00 last 114.95) - open/close 26686 (time 1746147463.30 total 243.01 last 111.99) - open/close 27825 (time 1746147473.30 total 253.01 last 113.85) - open/close 28852 (time 1746147483.31 total 263.02 last 102.64) - open/close 29906 (time 1746147493.32 total 273.03 last 105.28) - open/close 31272 (time 1746147503.33 total 283.04 last 136.53) - open/close 32653 (time 1746147513.33 total 293.04 last 138.02) - open/close 33997 (time 1746147523.33 total 303.04 last 134.39) - open/close 35217 (time 1746147533.34 total 313.04 last 121.98) - open/close 36459 (time 1746147543.34 total 323.05 last 124.16) - open/close 37651 (time 1746147553.34 total 333.05 last 119.16) - open/close 38842 (time 1746147563.34 total 343.05 last 119.07) - open/close 40000 (time 1746147572.19 total 351.90 last 130.90) - open/close 41326 (time 1746147582.19 total 361.90 last 132.59) - open/close 42657 (time 1746147592.19 total 371.90 last 133.08) - open/close 43968 (time 1746147602.20 total 381.91 last 131.03) - open/close 45236 (time 1746147612.20 total 391.91 last 126.77) - open/close 46765 (time 1746147622.20 total 401.91 last 152.83) - open/close 47849 (time 1746147632.21 total 411.92 last 108.29) - open/close 48919 (time 1746147642.22 total 421.93 last 106.95) - open/close 49968 (time 1746147652.23 total 431.94 last 104.82) - open/close 50992 (time 1746147662.23 total 441.94 last 102.40) - open/close 52058 (time 1746147672.23 total 451.94 last 106.52) - open/close 53079 (time 1746147682.24 total 461.95 last 102.05) - open/close 54158 (time 1746147692.24 total 471.95 last 107.90) - open/close 55316 (time 1746147702.24 total 481.95 last 115.79) - open/close 56370 (time 1746147712.25 total 491.95 last 105.36) - open/close 57432 (time 1746147722.25 total 501.96 last 106.15) - open/close 58490 (time 1746147732.26 total 511.97 last 105.69) - open/close 59643 (time 1746147742.27 total 521.98 last 115.22) - open/close 60000 (time 1746147745.02 total 524.73 last 129.85) - open/close 61108 (time 1746147755.02 total 534.73 last 110.72) - open/close 62216 (time 1746147765.04 total 544.74 last 110.68) - open/close 63353 (time 1746147775.04 total 554.75 last 113.67) - open/close 64513 (time 1746147785.04 total 564.75 last 115.95) total: 64768 open/close in 567.07 seconds: 114.21 ops/second - /unlink 2037 (time 1746147799.23 total 10.00 last 203.65) - /unlink 4465 (time 1746147809.23 total 20.00 last 242.76) - /unlink 7819 (time 1746147819.23 total 30.00 last 335.39) - /unlink 10000 (time 1746147826.24 total 37.01 last 311.40) - /unlink 12870 (time 1746147836.24 total 47.01 last 286.87) - /unlink 15296 (time 1746147846.24 total 57.01 last 242.58) - /unlink 17388 (time 1746147856.24 total 67.01 last 209.17) - /unlink 19653 (time 1746147866.25 total 77.02 last 226.42) - /unlink 22082 (time 1746147876.25 total 87.02 last 242.76) - /unlink 24734 (time 1746147886.25 total 97.03 last 265.14) - /unlink 27522 (time 1746147896.26 total 107.03 last 278.75) - /unlink 30000 (time 1746147906.16 total 116.94 last 250.12) - /unlink 32374 (time 1746147916.17 total 126.94 last 237.30) - /unlink 34802 (time 1746147926.17 total 136.94 last 242.72) - /unlink 37482 (time 1746147936.17 total 146.94 last 267.97) - /unlink 40000 (time 1746147945.38 total 156.15 last 273.56) - /unlink 42726 (time 1746147955.38 total 166.15 last 272.51) - /unlink 46247 (time 1746147965.38 total 176.15 last 352.05) - /unlink 48758 (time 1746147975.39 total 186.16 last 250.92) - /unlink 50000 (time 1746147979.50 total 190.27 last 302.37) - /unlink 52339 (time 1746147989.50 total 200.27 last 233.89) - /unlink 54636 (time 1746147999.50 total 210.27 last 229.61) - /unlink 57099 (time 1746148009.50 total 220.27 last 246.29) - /unlink 59825 (time 1746148019.50 total 230.27 last 272.60) - /unlink 62369 (time 1746148029.51 total 240.28 last 254.24) total: 64768 /unlink in 249.83 seconds: 259.24 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1679 (time 1746148065.53 total 10.00 last 167.87) - open/close 3287 (time 1746148075.54 total 20.01 last 160.68) - open/close 4739 (time 1746148085.54 total 30.01 last 145.17) - open/close 6214 (time 1746148095.55 total 40.01 last 147.46) - open/close 7589 (time 1746148105.55 total 50.02 last 137.48) - open/close 8931 (time 1746148115.55 total 60.02 last 134.13) - open/close 10000 (time 1746148122.77 total 67.24 last 148.13) - open/close 11704 (time 1746148132.78 total 77.25 last 170.27) - open/close 13277 (time 1746148142.78 total 87.25 last 157.27) - open/close 14869 (time 1746148152.78 total 97.25 last 159.16) - open/close 16606 (time 1746148162.79 total 107.25 last 173.62) - open/close 18080 (time 1746148172.79 total 117.26 last 147.33) - open/close 19401 (time 1746148182.81 total 127.28 last 131.86) - open/close 20000 (time 1746148187.16 total 131.63 last 137.74) - open/close 21389 (time 1746148197.16 total 141.63 last 138.89) - open/close 22814 (time 1746148207.16 total 151.63 last 142.47) - open/close 24336 (time 1746148217.17 total 161.63 last 152.13) - open/close 25919 (time 1746148227.17 total 171.64 last 158.27) - open/close 27469 (time 1746148237.17 total 181.64 last 154.98) - open/close 28792 (time 1746148247.18 total 191.64 last 132.22) - open/close 30000 (time 1746148255.32 total 199.79 last 148.26) - open/close 31415 (time 1746148265.33 total 209.79 last 141.45) - open/close 33038 (time 1746148275.33 total 219.79 last 162.30) - open/close 34696 (time 1746148285.33 total 229.80 last 165.71) - open/close 36324 (time 1746148295.33 total 239.80 last 162.80) - open/close 37859 (time 1746148305.33 total 249.80 last 153.48) - open/close 39271 (time 1746148315.34 total 259.80 last 141.16) - open/close 40000 (time 1746148320.63 total 265.10 last 137.66) - open/close 41370 (time 1746148330.64 total 275.10 last 136.95) - open/close 42833 (time 1746148340.64 total 285.11 last 146.21) - open/close 44443 (time 1746148350.65 total 295.11 last 160.95) - open/close 46126 (time 1746148360.65 total 305.11 last 168.27) - open/close 47496 (time 1746148370.65 total 315.12 last 136.93) - open/close 49119 (time 1746148380.66 total 325.12 last 162.24) - open/close 50000 (time 1746148386.44 total 330.90 last 152.39) - open/close 51569 (time 1746148396.45 total 340.91 last 156.78) - open/close 53302 (time 1746148406.45 total 350.92 last 173.25) - open/close 54689 (time 1746148416.45 total 360.92 last 138.68) - open/close 56123 (time 1746148426.45 total 370.92 last 143.33) - open/close 57477 (time 1746148436.46 total 380.92 last 135.39) - open/close 58804 (time 1746148446.46 total 390.93 last 132.66) - open/close 60000 (time 1746148455.00 total 399.46 last 140.08) - open/close 61276 (time 1746148465.00 total 409.47 last 127.51) - open/close 62658 (time 1746148475.00 total 419.47 last 138.19) - open/close 63850 (time 1746148485.00 total 429.47 last 119.19) total: 64768 open/close in 435.93 seconds: 148.57 ops/second - /unlink 2174 (time 1746148502.99 total 10.01 last 217.19) - /unlink 4364 (time 1746148513.05 total 20.07 last 217.78) - /unlink 6432 (time 1746148523.05 total 30.07 last 206.69) - /unlink 8509 (time 1746148533.05 total 40.07 last 207.68) - /unlink 10000 (time 1746148540.57 total 47.59 last 198.37) - /unlink 12282 (time 1746148550.57 total 57.59 last 228.19) - /unlink 14422 (time 1746148560.57 total 67.59 last 213.91) - /unlink 16913 (time 1746148570.57 total 77.59 last 249.10) - /unlink 18961 (time 1746148580.58 total 87.60 last 204.74) - /unlink 20000 (time 1746148585.23 total 92.25 last 223.09) - /unlink 22301 (time 1746148595.24 total 102.26 last 230.03) - /unlink 25170 (time 1746148605.24 total 112.26 last 286.90) - /unlink 27206 (time 1746148615.24 total 122.26 last 203.54) - /unlink 29023 (time 1746148625.24 total 132.26 last 181.66) - /unlink 30000 (time 1746148630.00 total 137.02 last 205.47) - /unlink 32061 (time 1746148640.00 total 147.02 last 206.04) - /unlink 34023 (time 1746148650.01 total 157.03 last 196.07) - /unlink 35930 (time 1746148660.02 total 167.04 last 190.45) - /unlink 38191 (time 1746148670.02 total 177.04 last 226.10) - /unlink 40000 (time 1746148678.21 total 185.23 last 220.96) - /unlink 42043 (time 1746148688.21 total 195.23 last 204.28) - /unlink 44119 (time 1746148698.22 total 205.23 last 207.43) - /unlink 46210 (time 1746148708.22 total 215.24 last 208.97) - /unlink 48570 (time 1746148718.23 total 225.24 last 235.94) - /unlink 50000 (time 1746148723.23 total 230.25 last 285.77) - /unlink 52138 (time 1746148733.23 total 240.25 last 213.74) - /unlink 54171 (time 1746148743.23 total 250.25 last 203.28) - /unlink 56103 (time 1746148753.24 total 260.25 last 193.13) - /unlink 57922 (time 1746148763.24 total 270.26 last 181.88) - /unlink 59756 (time 1746148773.24 total 280.26 last 183.39) - /unlink 61822 (time 1746148783.24 total 290.26 last 206.57) - /unlink 63958 (time 1746148793.24 total 300.26 last 213.53) total: 64768 /unlink in 304.44 seconds: 212.74 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1365 (time 1746148827.68 total 10.00 last 136.46) - open/close 2634 (time 1746148837.68 total 20.01 last 126.86) - open/close 3877 (time 1746148847.69 total 30.02 last 124.17) - open/close 5057 (time 1746148857.71 total 40.03 last 117.84) - open/close 6255 (time 1746148867.71 total 50.03 last 119.76) - open/close 7502 (time 1746148877.72 total 60.04 last 124.63) - open/close 8716 (time 1746148887.72 total 70.04 last 121.33) - open/close 10000 (time 1746148896.88 total 79.20 last 140.20) - open/close 11514 (time 1746148906.88 total 89.21 last 151.35) - open/close 12785 (time 1746148916.88 total 99.21 last 127.10) - open/close 13979 (time 1746148926.89 total 109.21 last 119.35) - open/close 15186 (time 1746148936.89 total 119.21 last 120.68) - open/close 16441 (time 1746148946.89 total 129.22 last 125.46) - open/close 17917 (time 1746148956.89 total 139.22 last 147.58) - open/close 19485 (time 1746148966.90 total 149.22 last 156.70) - open/close 20000 (time 1746148970.43 total 152.75 last 146.01) - open/close 21337 (time 1746148980.43 total 162.75 last 133.65) - open/close 22557 (time 1746148990.43 total 172.76 last 121.97) - open/close 23921 (time 1746149000.44 total 182.76 last 136.38) - open/close 25160 (time 1746149010.44 total 192.76 last 123.85) - open/close 26380 (time 1746149020.44 total 202.77 last 121.93) - open/close 27629 (time 1746149030.45 total 212.77 last 124.87) - open/close 28965 (time 1746149040.45 total 222.77 last 133.56) - open/close 30000 (time 1746149048.11 total 230.43 last 135.13) - open/close 31344 (time 1746149058.12 total 240.44 last 134.23) - open/close 32724 (time 1746149068.13 total 250.45 last 137.91) - open/close 33912 (time 1746149078.13 total 260.45 last 118.80) - open/close 34974 (time 1746149088.13 total 270.45 last 106.19) - open/close 35989 (time 1746149098.14 total 280.46 last 101.44) - open/close 37082 (time 1746149108.14 total 290.46 last 109.28) - open/close 38334 (time 1746149118.14 total 300.47 last 125.11) - open/close 39469 (time 1746149128.15 total 310.48 last 113.40) - open/close 40000 (time 1746149132.84 total 315.16 last 113.39) - open/close 41129 (time 1746149142.84 total 325.16 last 112.83) - open/close 42359 (time 1746149152.85 total 335.17 last 122.94) - open/close 43517 (time 1746149162.85 total 345.18 last 115.74) - open/close 44462 (time 1746149172.86 total 355.18 last 94.47) - open/close 45479 (time 1746149182.86 total 365.18 last 101.64) - open/close 46628 (time 1746149192.86 total 375.19 last 114.90) - open/close 47768 (time 1746149202.87 total 385.19 last 113.93) - open/close 49024 (time 1746149212.87 total 395.19 last 125.59) - open/close 50000 (time 1746149219.86 total 402.18 last 139.66) - open/close 51492 (time 1746149229.86 total 412.18 last 149.18) - open/close 52638 (time 1746149239.87 total 422.19 last 114.53) - open/close 53773 (time 1746149249.87 total 432.20 last 113.41) - open/close 54906 (time 1746149259.88 total 442.20 last 113.22) - open/close 56043 (time 1746149269.89 total 452.21 last 113.60) - open/close 57135 (time 1746149279.89 total 462.22 last 109.17) - open/close 58280 (time 1746149289.90 total 472.22 last 114.46) - open/close 59353 (time 1746149299.90 total 482.23 last 107.23) - open/close 60000 (time 1746149305.82 total 488.14 last 109.44) - open/close 61107 (time 1746149315.82 total 498.14 last 110.64) - open/close 62348 (time 1746149325.83 total 508.15 last 123.99) - open/close 63510 (time 1746149335.83 total 518.15 last 116.17) - open/close 64605 (time 1746149345.84 total 528.16 last 109.43) total: 64768 open/close in 529.52 seconds: 122.32 ops/second - /unlink 1590 (time 1746149359.13 total 10.00 last 158.94) - /unlink 3440 (time 1746149369.14 total 20.01 last 184.94) - /unlink 5158 (time 1746149379.14 total 30.01 last 171.78) - /unlink 7018 (time 1746149389.14 total 40.01 last 185.95) - /unlink 9258 (time 1746149399.14 total 50.01 last 223.96) - /unlink 10000 (time 1746149403.41 total 54.28 last 173.83) - /unlink 11784 (time 1746149413.41 total 64.28 last 178.35) - /unlink 13625 (time 1746149423.42 total 74.29 last 184.03) - /unlink 15888 (time 1746149433.42 total 84.29 last 226.25) - /unlink 18262 (time 1746149443.42 total 94.29 last 237.37) - /unlink 20000 (time 1746149451.18 total 102.05 last 224.02) - /unlink 21776 (time 1746149461.19 total 112.06 last 177.46) - /unlink 23517 (time 1746149471.19 total 122.06 last 174.06) - /unlink 25125 (time 1746149481.19 total 132.06 last 160.79) - /unlink 26597 (time 1746149491.19 total 142.07 last 147.12) - /unlink 28134 (time 1746149501.20 total 152.07 last 153.67) - /unlink 29861 (time 1746149511.20 total 162.07 last 172.68) - /unlink 31824 (time 1746149521.20 total 172.07 last 196.23) - /unlink 33958 (time 1746149531.20 total 182.07 last 213.40) - /unlink 35934 (time 1746149541.20 total 192.07 last 197.58) - /unlink 37799 (time 1746149551.21 total 202.08 last 186.45) - /unlink 39413 (time 1746149561.21 total 212.08 last 161.34) - /unlink 40000 (time 1746149564.48 total 215.35 last 179.66) - /unlink 41682 (time 1746149574.48 total 225.35 last 168.13) - /unlink 43679 (time 1746149584.49 total 235.36 last 199.60) - /unlink 45836 (time 1746149594.50 total 245.37 last 215.43) - /unlink 47993 (time 1746149604.50 total 255.37 last 215.67) - /unlink 50000 (time 1746149612.98 total 263.85 last 236.74) - /unlink 52357 (time 1746149622.98 total 273.85 last 235.65) - /unlink 54599 (time 1746149632.98 total 283.85 last 224.14) - /unlink 56842 (time 1746149642.99 total 293.86 last 224.17) - /unlink 58912 (time 1746149652.99 total 303.87 last 206.90) - /unlink 60000 (time 1746149658.27 total 309.14 last 206.15) - /unlink 62132 (time 1746149668.27 total 319.14 last 213.16) - /unlink 64612 (time 1746149678.27 total 329.15 last 247.95) total: 64768 /unlink in 329.57 seconds: 196.52 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1190 (time 1746149703.82 total 10.01 last 118.92) - open/close 2438 (time 1746149713.82 total 20.01 last 124.79) - open/close 3719 (time 1746149723.83 total 30.01 last 128.03) - open/close 5013 (time 1746149733.83 total 40.02 last 129.29) - open/close 6389 (time 1746149743.83 total 50.02 last 137.59) - open/close 7748 (time 1746149753.84 total 60.02 last 135.89) - open/close 9177 (time 1746149763.85 total 70.03 last 142.76) - open/close 10000 (time 1746149768.77 total 74.96 last 167.13) - open/close 11348 (time 1746149778.78 total 84.96 last 134.71) - open/close 12708 (time 1746149788.78 total 94.97 last 135.94) - open/close 13978 (time 1746149798.78 total 104.97 last 126.98) - open/close 15377 (time 1746149808.79 total 114.97 last 139.85) - open/close 16727 (time 1746149818.79 total 124.98 last 134.90) - open/close 18133 (time 1746149828.79 total 134.98 last 140.58) - open/close 19448 (time 1746149838.80 total 144.99 last 131.39) - open/close 20000 (time 1746149842.87 total 149.06 last 135.55) - open/close 21477 (time 1746149852.88 total 159.07 last 147.64) - open/close 23009 (time 1746149862.88 total 169.07 last 153.19) - open/close 24576 (time 1746149872.88 total 179.07 last 156.66) - open/close 25978 (time 1746149882.89 total 189.07 last 140.14) - open/close 27368 (time 1746149892.89 total 199.08 last 138.92) - open/close 29386 (time 1746149902.90 total 209.09 last 201.59) - open/close 30000 (time 1746149906.09 total 212.28 last 192.63) - open/close 31697 (time 1746149916.10 total 222.28 last 169.61) - open/close 33214 (time 1746149926.10 total 232.29 last 151.59) - open/close 34895 (time 1746149936.11 total 242.30 last 167.99) - open/close 36129 (time 1746149946.11 total 252.30 last 123.39) - open/close 37294 (time 1746149956.11 total 262.30 last 116.46) - open/close 38322 (time 1746149966.12 total 272.31 last 102.72) - open/close 39333 (time 1746149976.12 total 282.31 last 101.06) - open/close 40000 (time 1746149981.74 total 287.93 last 118.81) - open/close 41062 (time 1746149991.75 total 297.94 last 106.05) - open/close 42124 (time 1746150001.76 total 307.95 last 106.13) - open/close 43190 (time 1746150011.76 total 317.95 last 106.54) - open/close 44269 (time 1746150021.77 total 327.96 last 107.81) - open/close 45312 (time 1746150031.78 total 337.97 last 104.21) - open/close 46488 (time 1746150041.79 total 347.98 last 117.48) - open/close 47689 (time 1746150051.79 total 357.98 last 120.09) - open/close 48939 (time 1746150061.80 total 367.99 last 124.91) - open/close 50000 (time 1746150070.30 total 376.48 last 124.89) - open/close 51100 (time 1746150080.32 total 386.50 last 109.76) - open/close 52140 (time 1746150090.34 total 396.53 last 103.77) - open/close 53243 (time 1746150100.35 total 406.54 last 110.20) - open/close 54431 (time 1746150110.36 total 416.55 last 118.65) - open/close 55541 (time 1746150120.36 total 426.55 last 110.99) - open/close 56793 (time 1746150130.36 total 436.55 last 125.19) - open/close 58032 (time 1746150140.39 total 446.57 last 123.61) - open/close 59249 (time 1746150150.39 total 456.58 last 121.67) - open/close 60000 (time 1746150155.70 total 461.88 last 141.51) - open/close 61415 (time 1746150165.70 total 471.88 last 141.47) - open/close 62734 (time 1746150175.70 total 481.89 last 131.87) - open/close 64017 (time 1746150185.70 total 491.89 last 128.29) total: 64768 open/close in 497.91 seconds: 130.08 ops/second - /unlink 2209 (time 1746150203.52 total 10.00 last 220.89) - /unlink 4191 (time 1746150213.53 total 20.01 last 197.94) - /unlink 6053 (time 1746150223.54 total 30.02 last 186.05) - /unlink 7913 (time 1746150233.54 total 40.02 last 185.94) - /unlink 9658 (time 1746150243.55 total 50.03 last 174.42) - /unlink 10000 (time 1746150245.61 total 52.09 last 165.86) - /unlink 11506 (time 1746150255.61 total 62.10 last 150.52) - /unlink 12987 (time 1746150265.62 total 72.11 last 147.98) - /unlink 14612 (time 1746150275.62 total 82.11 last 162.45) - /unlink 16309 (time 1746150285.63 total 92.11 last 169.63) - /unlink 17851 (time 1746150295.63 total 102.11 last 154.19) - /unlink 19669 (time 1746150305.63 total 112.12 last 181.74) - /unlink 21269 (time 1746150315.64 total 122.12 last 159.95) - /unlink 22830 (time 1746150325.64 total 132.12 last 156.09) - /unlink 24481 (time 1746150335.64 total 142.12 last 165.04) - /unlink 26163 (time 1746150345.68 total 152.17 last 167.47) - /unlink 27908 (time 1746150355.68 total 162.17 last 174.49) - /unlink 29851 (time 1746150365.69 total 172.17 last 194.20) - /unlink 32183 (time 1746150375.69 total 182.18 last 233.08) - /unlink 33952 (time 1746150385.70 total 192.18 last 176.82) - /unlink 35826 (time 1746150395.70 total 202.19 last 187.34) - /unlink 37603 (time 1746150405.70 total 212.19 last 177.67) - /unlink 39515 (time 1746150415.71 total 222.19 last 191.06) - /unlink 40000 (time 1746150418.04 total 224.53 last 207.86) - /unlink 42481 (time 1746150428.04 total 234.53 last 248.09) - /unlink 45000 (time 1746150438.05 total 244.53 last 251.87) - /unlink 47114 (time 1746150448.05 total 254.53 last 211.37) - /unlink 49341 (time 1746150458.05 total 264.53 last 222.67) - /unlink 50000 (time 1746150461.25 total 267.73 last 205.99) - /unlink 52115 (time 1746150471.25 total 277.74 last 211.39) - /unlink 54428 (time 1746150481.25 total 287.74 last 231.29) - /unlink 56899 (time 1746150491.26 total 297.74 last 246.99) - /unlink 59283 (time 1746150501.26 total 307.74 last 238.34) - /unlink 60000 (time 1746150504.32 total 310.80 last 234.43) - /unlink 62369 (time 1746150514.32 total 320.80 last 236.88) - /unlink 64245 (time 1746150524.32 total 330.80 last 187.57) total: 64768 /unlink in 333.23 seconds: 194.36 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg229-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg229-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg229-server unloading modules via unload_modules_local on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg229-client: oleg229-server: ssh exited with exit code 2 pdsh@oleg229-client: oleg229-client: ssh exited with exit code 2 pdsh@oleg229-client: oleg229-client: ssh exited with exit code 2 pdsh@oleg229-client: oleg229-server: ssh exited with exit code 2