== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 18:20:16 (1746138016) Stopping clients: oleg419-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg419-client.virtnet /mnt/lustre2 (opts:-f) oleg419-server: oleg419-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg419-server' oleg419-server: oleg419-server.virtnet: executing load_modules_local oleg419-server: Loading modules from /home/green/git/lustre-release/lustre oleg419-server: detected 4 online CPUs by sysfs oleg419-server: MODOPTS_LIBCFS= oleg419-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg419-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg419-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg419-client.virtnet: -o user_xattr,flock 192.168.204.119@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1455 (time 1746138064.31 total 10.00 last 145.46) - open/close 2696 (time 1746138074.32 total 20.01 last 124.01) - open/close 4281 (time 1746138084.32 total 30.01 last 158.46) - open/close 5833 (time 1746138094.32 total 40.02 last 155.14) - open/close 7268 (time 1746138104.33 total 50.02 last 143.47) - open/close 8893 (time 1746138114.33 total 60.02 last 162.48) - open/close 10000 (time 1746138122.01 total 67.70 last 144.10) - open/close 11734 (time 1746138132.01 total 77.71 last 173.32) - open/close 13598 (time 1746138142.02 total 87.71 last 186.33) - open/close 14996 (time 1746138152.02 total 97.71 last 139.76) - open/close 16355 (time 1746138162.02 total 107.72 last 135.85) - open/close 17702 (time 1746138172.03 total 117.72 last 134.63) - open/close 19007 (time 1746138182.03 total 127.73 last 130.43) - open/close 20000 (time 1746138189.63 total 135.32 last 130.74) - open/close 21237 (time 1746138199.63 total 145.33 last 123.66) - open/close 22614 (time 1746138209.64 total 155.33 last 137.66) - open/close 24147 (time 1746138219.65 total 165.34 last 153.16) - open/close 25735 (time 1746138229.65 total 175.35 last 158.65) - open/close 27778 (time 1746138239.66 total 185.36 last 204.13) - open/close 29669 (time 1746138249.67 total 195.37 last 188.90) - open/close 30000 (time 1746138251.75 total 197.45 last 159.09) - open/close 31561 (time 1746138261.76 total 207.45 last 156.03) - open/close 33268 (time 1746138271.76 total 217.45 last 170.70) - open/close 34779 (time 1746138281.76 total 227.46 last 151.03) - open/close 36292 (time 1746138291.77 total 237.46 last 151.24) - open/close 37459 (time 1746138301.77 total 247.46 last 116.66) - open/close 38668 (time 1746138311.78 total 257.47 last 120.83) - open/close 39875 (time 1746138321.78 total 267.47 last 120.64) - open/close 41082 (time 1746138331.78 total 277.48 last 120.68) - open/close 42207 (time 1746138341.79 total 287.48 last 112.48) - open/close 43607 (time 1746138351.79 total 297.48 last 139.99) - open/close 45130 (time 1746138361.79 total 307.48 last 152.22) - open/close 46561 (time 1746138371.79 total 317.49 last 143.08) - open/close 48011 (time 1746138381.79 total 327.49 last 144.99) - open/close 49539 (time 1746138391.79 total 337.49 last 152.79) - open/close 50000 (time 1746138394.23 total 339.92 last 189.25) - open/close 51712 (time 1746138404.23 total 349.93 last 171.13) - open/close 53725 (time 1746138414.24 total 359.93 last 201.23) - open/close 55462 (time 1746138424.24 total 369.93 last 173.63) - open/close 57196 (time 1746138434.25 total 379.94 last 173.34) - open/close 59050 (time 1746138444.25 total 389.94 last 185.40) - open/close 60000 (time 1746138449.20 total 394.89 last 191.88) - open/close 61548 (time 1746138459.20 total 404.89 last 154.70) - open/close 63165 (time 1746138469.20 total 414.90 last 161.66) total: 64768 open/close in 423.59 seconds: 152.90 ops/second - /unlink 2268 (time 1746138489.31 total 10.01 last 226.66) - /unlink 4853 (time 1746138499.31 total 20.01 last 258.42) - /unlink 7215 (time 1746138509.31 total 30.01 last 236.20) - /unlink 9620 (time 1746138519.32 total 40.01 last 240.44) - /unlink 11987 (time 1746138529.32 total 50.02 last 236.62) - /unlink 14312 (time 1746138539.32 total 60.02 last 232.44) - /unlink 16948 (time 1746138549.32 total 70.02 last 263.60) - /unlink 19096 (time 1746138559.32 total 80.02 last 214.79) - /unlink 20000 (time 1746138563.08 total 83.77 last 240.69) - /unlink 22928 (time 1746138573.08 total 93.78 last 292.74) - /unlink 25176 (time 1746138583.08 total 103.78 last 224.71) - /unlink 27573 (time 1746138593.09 total 113.79 last 239.52) - /unlink 29619 (time 1746138603.09 total 123.79 last 204.57) - /unlink 32112 (time 1746138613.10 total 133.79 last 249.18) - /unlink 34038 (time 1746138623.11 total 143.80 last 192.44) - /unlink 36914 (time 1746138633.11 total 153.80 last 287.56) - /unlink 39358 (time 1746138643.11 total 163.81 last 244.21) - /unlink 40000 (time 1746138645.29 total 165.99 last 294.94) - /unlink 43424 (time 1746138655.29 total 175.99 last 342.37) - /unlink 46062 (time 1746138665.29 total 185.99 last 263.80) - /unlink 48743 (time 1746138675.30 total 195.99 last 267.99) - /unlink 50000 (time 1746138679.88 total 200.58 last 273.97) - /unlink 52390 (time 1746138689.89 total 210.58 last 238.95) - /unlink 54437 (time 1746138699.89 total 220.59 last 204.56) - /unlink 56750 (time 1746138709.89 total 230.59 last 231.27) - /unlink 58570 (time 1746138719.90 total 240.60 last 181.90) - /unlink 60000 (time 1746138727.25 total 247.95 last 194.50) - /unlink 61999 (time 1746138737.26 total 257.96 last 199.76) - /unlink 64655 (time 1746138747.27 total 267.96 last 265.39) total: 64768 /unlink in 268.49 seconds: 241.23 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1290 (time 1746138776.31 total 10.00 last 128.94) - open/close 2818 (time 1746138786.31 total 20.01 last 152.77) - open/close 4164 (time 1746138796.31 total 30.01 last 134.55) - open/close 5327 (time 1746138806.32 total 40.01 last 116.25) - open/close 6648 (time 1746138816.32 total 50.02 last 132.03) - open/close 7864 (time 1746138826.32 total 60.02 last 121.60) - open/close 9095 (time 1746138836.33 total 70.02 last 123.07) - open/close 10000 (time 1746138842.68 total 76.38 last 142.42) - open/close 11354 (time 1746138852.69 total 86.38 last 135.33) - open/close 12516 (time 1746138862.69 total 96.39 last 116.15) - open/close 13760 (time 1746138872.70 total 106.40 last 124.25) - open/close 14930 (time 1746138882.71 total 116.40 last 116.96) - open/close 16176 (time 1746138892.72 total 126.41 last 124.48) - open/close 17319 (time 1746138902.72 total 136.41 last 114.28) - open/close 18507 (time 1746138912.73 total 146.42 last 118.67) - open/close 19594 (time 1746138922.74 total 156.43 last 108.59) - open/close 20000 (time 1746138926.90 total 160.60 last 97.55) - open/close 21236 (time 1746138936.91 total 170.61 last 123.48) - open/close 22532 (time 1746138946.91 total 180.61 last 129.59) - open/close 23825 (time 1746138956.91 total 190.61 last 129.28) - open/close 25230 (time 1746138966.91 total 200.61 last 140.48) - open/close 26362 (time 1746138976.92 total 210.61 last 113.16) - open/close 27385 (time 1746138986.92 total 220.62 last 102.29) - open/close 28448 (time 1746138996.93 total 230.62 last 106.22) - open/close 29640 (time 1746139006.93 total 240.62 last 119.19) - open/close 30000 (time 1746139010.00 total 243.69 last 117.37) - open/close 31209 (time 1746139020.00 total 253.69 last 120.87) - open/close 32517 (time 1746139030.00 total 263.69 last 130.79) - open/close 33820 (time 1746139040.00 total 273.70 last 130.27) - open/close 35072 (time 1746139050.00 total 283.70 last 125.17) - open/close 36234 (time 1746139060.01 total 293.71 last 116.09) - open/close 37343 (time 1746139070.02 total 303.72 last 110.82) - open/close 38496 (time 1746139080.02 total 313.72 last 115.28) - open/close 39779 (time 1746139090.02 total 323.72 last 128.29) - open/close 41250 (time 1746139100.03 total 333.72 last 147.05) - open/close 42611 (time 1746139110.03 total 343.73 last 136.04) - open/close 44237 (time 1746139120.03 total 353.73 last 162.55) - open/close 46183 (time 1746139130.04 total 363.73 last 194.56) - open/close 47469 (time 1746139140.04 total 373.74 last 128.55) - open/close 48684 (time 1746139150.05 total 383.74 last 121.42) - open/close 50000 (time 1746139159.32 total 393.02 last 141.83) - open/close 51551 (time 1746139169.33 total 403.02 last 155.10) - open/close 52764 (time 1746139179.33 total 413.03 last 121.25) - open/close 53950 (time 1746139189.33 total 423.03 last 118.58) - open/close 55253 (time 1746139199.33 total 433.03 last 130.26) - open/close 56555 (time 1746139209.34 total 443.03 last 130.19) - open/close 57691 (time 1746139219.34 total 453.03 last 113.56) - open/close 59006 (time 1746139229.34 total 463.04 last 131.43) - open/close 60000 (time 1746139236.86 total 470.55 last 132.29) - open/close 61294 (time 1746139246.86 total 480.56 last 129.39) - open/close 62620 (time 1746139256.86 total 490.56 last 132.59) - open/close 63874 (time 1746139266.86 total 500.56 last 125.35) total: 64768 open/close in 506.33 seconds: 127.92 ops/second - /unlink 2676 (time 1746139284.14 total 10.00 last 267.58) - /unlink 4993 (time 1746139294.14 total 20.00 last 231.65) - /unlink 6747 (time 1746139304.14 total 30.00 last 175.40) - /unlink 8594 (time 1746139314.15 total 40.01 last 184.59) - /unlink 10000 (time 1746139321.35 total 47.22 last 195.04) - /unlink 12467 (time 1746139331.36 total 57.22 last 246.68) - /unlink 14793 (time 1746139341.36 total 67.22 last 232.57) - /unlink 16830 (time 1746139351.36 total 77.22 last 203.66) - /unlink 18724 (time 1746139361.37 total 87.23 last 189.27) - /unlink 20000 (time 1746139368.40 total 94.26 last 181.38) - /unlink 21786 (time 1746139378.41 total 104.27 last 178.47) - /unlink 23679 (time 1746139388.41 total 114.27 last 189.25) - /unlink 25806 (time 1746139398.41 total 124.28 last 212.64) - /unlink 28148 (time 1746139408.42 total 134.28 last 234.13) - /unlink 30000 (time 1746139416.30 total 142.17 last 234.84) - /unlink 32189 (time 1746139426.30 total 152.17 last 218.88) - /unlink 34089 (time 1746139436.31 total 162.17 last 189.98) - /unlink 36012 (time 1746139446.31 total 172.17 last 192.25) - /unlink 37757 (time 1746139456.33 total 182.19 last 174.17) - /unlink 39534 (time 1746139466.33 total 192.19 last 177.64) - /unlink 40000 (time 1746139468.63 total 194.49 last 202.55) - /unlink 41995 (time 1746139478.64 total 204.50 last 199.37) - /unlink 43540 (time 1746139488.64 total 214.50 last 154.49) - /unlink 45297 (time 1746139498.64 total 224.50 last 175.67) - /unlink 47072 (time 1746139508.64 total 234.50 last 177.47) - /unlink 49068 (time 1746139518.64 total 244.51 last 199.59) - /unlink 50000 (time 1746139522.60 total 248.46 last 235.60) - /unlink 51982 (time 1746139532.61 total 258.47 last 197.96) - /unlink 54013 (time 1746139542.61 total 268.47 last 203.09) - /unlink 56229 (time 1746139552.61 total 278.47 last 221.58) - /unlink 59190 (time 1746139562.62 total 288.48 last 295.99) - /unlink 60000 (time 1746139566.00 total 291.86 last 239.43) - /unlink 62425 (time 1746139576.00 total 301.86 last 242.44) total: 64768 /unlink in 309.67 seconds: 209.15 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1157 (time 1746139613.93 total 10.01 last 115.61) - open/close 2567 (time 1746139623.94 total 20.02 last 140.87) - open/close 3924 (time 1746139633.95 total 30.02 last 135.61) - open/close 5104 (time 1746139643.95 total 40.02 last 118.00) - open/close 6423 (time 1746139653.95 total 50.02 last 131.88) - open/close 7797 (time 1746139663.95 total 60.03 last 137.35) - open/close 9085 (time 1746139673.96 total 70.03 last 128.75) - open/close 10000 (time 1746139680.81 total 76.89 last 133.48) - open/close 11312 (time 1746139690.81 total 86.89 last 131.19) - open/close 12557 (time 1746139700.82 total 96.89 last 124.46) - open/close 13820 (time 1746139710.82 total 106.89 last 126.29) - open/close 14994 (time 1746139720.83 total 116.90 last 117.30) - open/close 16135 (time 1746139730.83 total 126.90 last 114.05) - open/close 17250 (time 1746139740.84 total 136.91 last 111.42) - open/close 18438 (time 1746139750.84 total 146.92 last 118.73) - open/close 19660 (time 1746139760.85 total 156.92 last 122.17) - open/close 20000 (time 1746139763.80 total 159.87 last 115.24) - open/close 21255 (time 1746139773.80 total 169.88 last 125.41) - open/close 22474 (time 1746139783.81 total 179.88 last 121.82) - open/close 23833 (time 1746139793.82 total 189.89 last 135.80) - open/close 24944 (time 1746139803.82 total 199.90 last 111.03) - open/close 26173 (time 1746139813.83 total 209.90 last 122.89) - open/close 27757 (time 1746139823.83 total 219.91 last 158.27) - open/close 29252 (time 1746139833.84 total 229.91 last 149.43) - open/close 30000 (time 1746139839.49 total 235.56 last 132.39) - open/close 31282 (time 1746139849.49 total 245.56 last 128.19) - open/close 32730 (time 1746139859.50 total 255.57 last 144.70) - open/close 34087 (time 1746139869.51 total 265.58 last 135.55) - open/close 35445 (time 1746139879.51 total 275.58 last 135.79) - open/close 36942 (time 1746139889.51 total 285.58 last 149.69) - open/close 38326 (time 1746139899.51 total 295.59 last 138.34) - open/close 39774 (time 1746139909.51 total 305.59 last 144.79) - open/close 40975 (time 1746139919.51 total 315.59 last 120.08) - open/close 42209 (time 1746139929.52 total 325.59 last 123.36) - open/close 43567 (time 1746139939.52 total 335.60 last 135.72) - open/close 44687 (time 1746139949.53 total 345.60 last 111.99) - open/close 45748 (time 1746139959.54 total 355.61 last 105.99) - open/close 46723 (time 1746139969.54 total 365.61 last 97.47) - open/close 47747 (time 1746139979.55 total 375.62 last 102.32) - open/close 48764 (time 1746139989.55 total 385.63 last 101.64) - open/close 49916 (time 1746139999.56 total 395.63 last 115.14) - open/close 51053 (time 1746140009.56 total 405.63 last 113.65) - open/close 52220 (time 1746140019.57 total 415.64 last 116.60) - open/close 53239 (time 1746140029.57 total 425.64 last 101.89) - open/close 54303 (time 1746140039.57 total 435.65 last 106.36) - open/close 55305 (time 1746140049.58 total 445.65 last 100.13) - open/close 56448 (time 1746140059.59 total 455.66 last 114.25) - open/close 57439 (time 1746140069.59 total 465.66 last 99.05) - open/close 58431 (time 1746140079.63 total 475.70 last 98.83) - open/close 59521 (time 1746140089.64 total 485.71 last 108.93) - open/close 60000 (time 1746140093.70 total 489.77 last 117.82) - open/close 61088 (time 1746140103.71 total 499.78 last 108.73) - open/close 62170 (time 1746140113.71 total 509.79 last 108.13) - open/close 63224 (time 1746140123.72 total 519.79 last 105.37) - open/close 64285 (time 1746140133.72 total 529.79 last 106.09) total: 64768 open/close in 535.11 seconds: 121.04 ops/second - /unlink 1621 (time 1746140151.14 total 10.01 last 161.96) - /unlink 3150 (time 1746140161.15 total 20.02 last 152.80) - /unlink 4692 (time 1746140171.16 total 30.03 last 153.95) - /unlink 6305 (time 1746140181.16 total 40.03 last 161.28) - /unlink 8059 (time 1746140191.21 total 50.08 last 174.63) - /unlink 9604 (time 1746140201.21 total 60.08 last 154.45) - /unlink 10000 (time 1746140204.04 total 62.91 last 139.91) - /unlink 11562 (time 1746140214.05 total 72.91 last 156.14) - /unlink 13166 (time 1746140224.05 total 82.91 last 160.40) - /unlink 14844 (time 1746140234.05 total 92.91 last 167.79) - /unlink 16521 (time 1746140244.05 total 102.91 last 167.70) - /unlink 18027 (time 1746140254.05 total 112.92 last 150.54) - /unlink 19621 (time 1746140264.05 total 122.92 last 159.36) - /unlink 20000 (time 1746140266.50 total 125.37 last 154.71) - /unlink 21601 (time 1746140276.50 total 135.37 last 160.09) - /unlink 23215 (time 1746140286.51 total 145.37 last 161.35) - /unlink 24851 (time 1746140296.51 total 155.37 last 163.60) - /unlink 26469 (time 1746140306.51 total 165.38 last 161.75) - /unlink 28117 (time 1746140316.51 total 175.38 last 164.79) - /unlink 29643 (time 1746140326.51 total 185.38 last 152.54) - /unlink 30000 (time 1746140328.77 total 187.63 last 158.56) - /unlink 31566 (time 1746140338.77 total 197.64 last 156.57) - /unlink 33157 (time 1746140348.77 total 207.64 last 159.09) - /unlink 34795 (time 1746140358.77 total 217.64 last 163.71) - /unlink 36447 (time 1746140368.78 total 227.64 last 165.18) - /unlink 38012 (time 1746140378.78 total 237.65 last 156.45) - /unlink 39736 (time 1746140388.78 total 247.65 last 172.33) - /unlink 41579 (time 1746140398.79 total 257.65 last 184.23) - /unlink 43157 (time 1746140408.79 total 267.65 last 157.78) - /unlink 44743 (time 1746140418.79 total 277.66 last 158.53) - /unlink 46439 (time 1746140428.80 total 287.67 last 169.46) - /unlink 48064 (time 1746140438.80 total 297.67 last 162.47) - /unlink 49789 (time 1746140448.80 total 307.67 last 172.47) - /unlink 51631 (time 1746140458.81 total 317.68 last 184.06) - /unlink 53545 (time 1746140468.81 total 327.68 last 191.35) - /unlink 55269 (time 1746140478.82 total 337.68 last 172.38) - /unlink 57074 (time 1746140488.82 total 347.68 last 180.48) - /unlink 58812 (time 1746140498.82 total 357.69 last 173.77) - /unlink 60000 (time 1746140505.50 total 364.37 last 177.80) - /unlink 61733 (time 1746140515.50 total 374.37 last 173.25) - /unlink 63452 (time 1746140525.51 total 384.38 last 171.78) total: 64768 /unlink in 392.63 seconds: 164.96 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1095 (time 1746140564.38 total 10.01 last 109.42) - open/close 2173 (time 1746140574.38 total 20.01 last 107.79) - open/close 3308 (time 1746140584.39 total 30.02 last 113.41) - open/close 4320 (time 1746140594.40 total 40.02 last 101.11) - open/close 5312 (time 1746140604.40 total 50.03 last 99.18) - open/close 6302 (time 1746140614.41 total 60.04 last 98.87) - open/close 7379 (time 1746140624.43 total 70.05 last 107.54) - open/close 8409 (time 1746140634.44 total 80.06 last 102.93) - open/close 9436 (time 1746140644.44 total 90.07 last 102.60) - open/close 10000 (time 1746140649.08 total 94.70 last 121.81) - open/close 11282 (time 1746140659.08 total 104.70 last 128.15) - open/close 12496 (time 1746140669.08 total 114.71 last 121.35) - open/close 13913 (time 1746140679.09 total 124.71 last 141.61) - open/close 15508 (time 1746140689.10 total 134.72 last 159.40) - open/close 16783 (time 1746140699.10 total 144.73 last 127.42) - open/close 17942 (time 1746140709.11 total 154.73 last 115.85) - open/close 19012 (time 1746140719.11 total 164.73 last 106.98) - open/close 20000 (time 1746140727.27 total 172.89 last 121.06) - open/close 21094 (time 1746140737.27 total 182.90 last 109.35) - open/close 22266 (time 1746140747.28 total 192.90 last 117.16) - open/close 23515 (time 1746140757.28 total 202.91 last 124.83) - open/close 24891 (time 1746140767.28 total 212.91 last 137.59) - open/close 26153 (time 1746140777.29 total 222.91 last 126.17) - open/close 27413 (time 1746140787.29 total 232.92 last 125.91) - open/close 28654 (time 1746140797.30 total 242.92 last 124.03) - open/close 30000 (time 1746140806.44 total 252.06 last 147.31) - open/close 31647 (time 1746140816.44 total 262.06 last 164.66) - open/close 33059 (time 1746140826.44 total 272.07 last 141.14) - open/close 34444 (time 1746140836.44 total 282.07 last 138.48) - open/close 35660 (time 1746140846.45 total 292.07 last 121.58) - open/close 37157 (time 1746140856.45 total 302.07 last 149.67) - open/close 38328 (time 1746140866.45 total 312.08 last 117.06) - open/close 39850 (time 1746140876.46 total 322.08 last 152.10) - open/close 41326 (time 1746140886.46 total 332.08 last 147.57) - open/close 43151 (time 1746140896.46 total 342.09 last 182.42) - open/close 44650 (time 1746140906.47 total 352.09 last 149.88) - open/close 46098 (time 1746140916.47 total 362.09 last 144.79) - open/close 47394 (time 1746140926.47 total 372.09 last 129.56) - open/close 48656 (time 1746140936.47 total 382.10 last 126.19) - open/close 49975 (time 1746140946.48 total 392.10 last 131.83) - open/close 51447 (time 1746140956.48 total 402.10 last 147.14) - open/close 52827 (time 1746140966.48 total 412.11 last 137.99) - open/close 54769 (time 1746140976.48 total 422.11 last 194.17) - open/close 56359 (time 1746140986.49 total 432.11 last 158.91) - open/close 57915 (time 1746140996.49 total 442.12 last 155.52) - open/close 59761 (time 1746141006.50 total 452.12 last 184.54) - open/close 61385 (time 1746141016.50 total 462.13 last 162.35) - open/close 62937 (time 1746141026.50 total 472.13 last 155.16) total: 64768 open/close in 481.74 seconds: 134.45 ops/second - /unlink 2109 (time 1746141047.71 total 10.00 last 210.88) - /unlink 3878 (time 1746141057.72 total 20.01 last 176.72) - /unlink 5951 (time 1746141067.72 total 30.01 last 207.30) - /unlink 8103 (time 1746141077.72 total 40.01 last 215.18) - /unlink 10000 (time 1746141087.49 total 49.78 last 194.29) - /unlink 11950 (time 1746141097.50 total 59.78 last 194.84) - /unlink 13632 (time 1746141107.50 total 69.79 last 168.15) - /unlink 15259 (time 1746141117.50 total 79.79 last 162.69) - /unlink 17061 (time 1746141127.51 total 89.80 last 180.04) - /unlink 18879 (time 1746141137.51 total 99.80 last 181.76) - /unlink 20000 (time 1746141143.55 total 105.84 last 185.71) - /unlink 22148 (time 1746141153.55 total 115.84 last 214.74) - /unlink 24251 (time 1746141163.56 total 125.85 last 210.10) - /unlink 26205 (time 1746141173.56 total 135.85 last 195.31) - /unlink 28234 (time 1746141183.57 total 145.86 last 202.81) - /unlink 29992 (time 1746141193.59 total 155.88 last 175.37) - /unlink 32134 (time 1746141203.60 total 165.88 last 214.16) - /unlink 34091 (time 1746141213.60 total 175.89 last 195.59) - /unlink 35934 (time 1746141223.60 total 185.89 last 184.25) - /unlink 37741 (time 1746141233.61 total 195.90 last 180.62) - /unlink 39948 (time 1746141243.61 total 205.90 last 220.70) - /unlink 42224 (time 1746141253.61 total 215.90 last 227.58) - /unlink 44443 (time 1746141263.61 total 225.90 last 221.89) - /unlink 46664 (time 1746141273.61 total 235.90 last 221.97) - /unlink 48742 (time 1746141283.62 total 245.90 last 207.79) - /unlink 50000 (time 1746141290.03 total 252.32 last 196.18) - /unlink 51785 (time 1746141300.03 total 262.32 last 178.46) - /unlink 54012 (time 1746141310.03 total 272.32 last 222.64) - /unlink 56314 (time 1746141320.04 total 282.32 last 230.13) - /unlink 58787 (time 1746141330.04 total 292.33 last 247.23) - /unlink 60000 (time 1746141334.82 total 297.11 last 253.72) - /unlink 62551 (time 1746141344.82 total 307.11 last 255.07) total: 64768 /unlink in 315.50 seconds: 205.29 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1429 (time 1746141386.26 total 10.01 last 142.79) - open/close 3049 (time 1746141396.26 total 20.01 last 161.97) - open/close 4808 (time 1746141406.26 total 30.01 last 175.87) - open/close 6208 (time 1746141416.26 total 40.01 last 139.98) - open/close 7532 (time 1746141426.26 total 50.02 last 132.37) - open/close 9034 (time 1746141436.27 total 60.02 last 150.16) - open/close 10000 (time 1746141443.00 total 66.75 last 143.58) - open/close 11244 (time 1746141453.00 total 76.75 last 124.36) - open/close 12747 (time 1746141463.00 total 86.75 last 150.21) - open/close 14548 (time 1746141473.01 total 96.76 last 180.02) - open/close 16061 (time 1746141483.02 total 106.77 last 151.19) - open/close 17550 (time 1746141493.02 total 116.77 last 148.81) - open/close 18989 (time 1746141503.02 total 126.77 last 143.87) - open/close 20000 (time 1746141509.99 total 133.74 last 145.19) - open/close 21519 (time 1746141519.99 total 143.74 last 151.89) - open/close 23165 (time 1746141529.99 total 153.74 last 164.57) - open/close 24421 (time 1746141539.99 total 163.74 last 125.57) - open/close 25874 (time 1746141549.99 total 173.74 last 145.27) - open/close 27245 (time 1746141560.00 total 183.75 last 136.96) - open/close 28657 (time 1746141570.01 total 193.76 last 141.18) - open/close 30000 (time 1746141579.95 total 203.71 last 134.98) - open/close 31229 (time 1746141589.96 total 213.71 last 122.84) - open/close 32464 (time 1746141599.97 total 223.72 last 123.37) - open/close 33670 (time 1746141609.97 total 233.72 last 120.60) - open/close 34918 (time 1746141619.98 total 243.73 last 124.73) - open/close 36205 (time 1746141629.98 total 253.73 last 128.65) - open/close 37566 (time 1746141639.99 total 263.74 last 136.01) - open/close 39085 (time 1746141649.99 total 273.74 last 151.84) - open/close 40000 (time 1746141657.63 total 281.38 last 119.85) - open/close 41146 (time 1746141667.63 total 291.38 last 114.53) - open/close 42783 (time 1746141677.63 total 301.38 last 163.67) - open/close 44386 (time 1746141687.64 total 311.39 last 160.26) - open/close 45641 (time 1746141697.64 total 321.39 last 125.49) - open/close 46920 (time 1746141707.64 total 331.39 last 127.88) - open/close 48103 (time 1746141717.64 total 341.39 last 118.29) - open/close 49331 (time 1746141727.64 total 351.39 last 122.76) - open/close 50000 (time 1746141732.97 total 356.72 last 125.62) - open/close 51366 (time 1746141742.98 total 366.73 last 136.52) - open/close 52661 (time 1746141752.98 total 376.73 last 129.43) - open/close 53870 (time 1746141762.98 total 386.73 last 120.88) - open/close 55093 (time 1746141772.99 total 396.74 last 122.21) - open/close 56152 (time 1746141783.00 total 406.75 last 105.83) - open/close 57167 (time 1746141793.00 total 416.75 last 101.44) - open/close 58445 (time 1746141803.00 total 426.75 last 127.79) - open/close 60000 (time 1746141812.80 total 436.55 last 158.68) - open/close 61227 (time 1746141822.80 total 446.55 last 122.68) - open/close 62511 (time 1746141832.81 total 456.56 last 128.35) - open/close 63733 (time 1746141842.81 total 466.56 last 122.12) total: 64768 open/close in 474.87 seconds: 136.39 ops/second - /unlink 1787 (time 1746141862.92 total 10.00 last 178.66) - /unlink 3573 (time 1746141872.93 total 20.01 last 178.48) - /unlink 5399 (time 1746141882.93 total 30.01 last 182.57) - /unlink 7135 (time 1746141892.93 total 40.01 last 173.58) - /unlink 9687 (time 1746141902.94 total 50.01 last 255.14) - /unlink 11321 (time 1746141912.94 total 60.02 last 163.34) - /unlink 12874 (time 1746141922.95 total 70.03 last 155.13) - /unlink 14333 (time 1746141932.95 total 80.03 last 145.89) - /unlink 16002 (time 1746141942.95 total 90.03 last 166.88) - /unlink 17523 (time 1746141952.95 total 100.03 last 152.05) - /unlink 19044 (time 1746141962.96 total 110.04 last 152.08) - /unlink 20000 (time 1746141967.31 total 114.39 last 219.43) - /unlink 21828 (time 1746141977.31 total 124.39 last 182.77) - /unlink 23987 (time 1746141987.32 total 134.40 last 215.86) - /unlink 26478 (time 1746141997.32 total 144.40 last 249.04) - /unlink 28599 (time 1746142007.32 total 154.40 last 212.05) - /unlink 30000 (time 1746142013.12 total 160.20 last 241.70) - /unlink 32354 (time 1746142023.12 total 170.20 last 235.36) - /unlink 34246 (time 1746142033.12 total 180.20 last 189.15) - /unlink 36337 (time 1746142043.13 total 190.21 last 208.99) - /unlink 38833 (time 1746142053.13 total 200.21 last 249.55) - /unlink 40000 (time 1746142056.83 total 203.91 last 315.68) - /unlink 42839 (time 1746142066.83 total 213.91 last 283.81) - /unlink 45427 (time 1746142076.83 total 223.91 last 258.74) - /unlink 47630 (time 1746142086.83 total 233.91 last 220.26) - /unlink 49802 (time 1746142096.84 total 243.92 last 217.11) - /unlink 51723 (time 1746142106.84 total 253.92 last 192.09) - /unlink 54170 (time 1746142116.84 total 263.92 last 244.62) - /unlink 56544 (time 1746142126.85 total 273.92 last 237.32) - /unlink 59508 (time 1746142136.85 total 283.93 last 296.18) - /unlink 61799 (time 1746142146.85 total 293.93 last 229.08) - /unlink 63875 (time 1746142156.86 total 303.93 last 207.56) total: 64768 /unlink in 308.54 seconds: 209.92 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg419-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg419-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg419-server stop mds service on oleg419-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg419-server unloading modules via unload_modules_local on: 'oleg419-server' oleg419-server: oleg419-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg419-client: oleg419-client: ssh exited with exit code 2 pdsh@oleg419-client: oleg419-server: ssh exited with exit code 2 pdsh@oleg419-client: oleg419-client: ssh exited with exit code 2 pdsh@oleg419-client: oleg419-server: ssh exited with exit code 2