== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:19:08 (1734257948) Stopping clients: oleg256-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg256-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg256-client: oleg256-server: ssh exited with exit code 2 oleg256-server: oleg256-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg256-server' oleg256-server: oleg256-server.virtnet: executing load_modules_local oleg256-server: Loading modules from /home/green/git/lustre-release/lustre oleg256-server: detected 4 online CPUs by sysfs oleg256-server: Force libcfs to create 2 CPU partitions oleg256-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg256-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg256-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg256-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg256-client.virtnet: -o user_xattr,flock 192.168.202.156@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16200 392 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1259 (time 1734258011.75 total 10.00 last 125.86) - open/close 2455 (time 1734258021.75 total 20.01 last 119.56) - open/close 3785 (time 1734258031.75 total 30.01 last 133.00) - open/close 4993 (time 1734258041.76 total 40.01 last 120.77) - open/close 6228 (time 1734258051.76 total 50.01 last 123.43) - open/close 7534 (time 1734258061.76 total 60.02 last 130.57) - open/close 8887 (time 1734258071.77 total 70.02 last 135.24) - open/close 10000 (time 1734258080.65 total 78.90 last 125.33) - open/close 11344 (time 1734258090.65 total 88.90 last 134.38) - open/close 12605 (time 1734258100.66 total 98.91 last 126.01) - open/close 13820 (time 1734258110.66 total 108.91 last 121.48) - open/close 15066 (time 1734258120.66 total 118.91 last 124.59) - open/close 16360 (time 1734258130.67 total 128.92 last 129.30) - open/close 17493 (time 1734258140.67 total 138.92 last 113.27) - open/close 18925 (time 1734258150.68 total 148.93 last 143.09) - open/close 20000 (time 1734258157.85 total 156.10 last 149.91) - open/close 21206 (time 1734258167.85 total 166.10 last 120.59) - open/close 22451 (time 1734258177.86 total 176.11 last 124.44) - open/close 24031 (time 1734258187.86 total 186.11 last 157.93) - open/close 25401 (time 1734258197.87 total 196.12 last 136.86) - open/close 26693 (time 1734258207.87 total 206.12 last 129.18) - open/close 28017 (time 1734258217.87 total 216.13 last 132.38) - open/close 29382 (time 1734258227.88 total 226.13 last 136.45) - open/close 30000 (time 1734258232.04 total 230.29 last 148.45) - open/close 31541 (time 1734258242.04 total 240.30 last 154.06) - open/close 32776 (time 1734258252.05 total 250.31 last 123.37) - open/close 34213 (time 1734258262.06 total 260.31 last 143.66) - open/close 35586 (time 1734258272.06 total 270.31 last 137.23) - open/close 36904 (time 1734258282.07 total 280.32 last 131.74) - open/close 38339 (time 1734258292.07 total 290.32 last 143.46) - open/close 39789 (time 1734258302.07 total 300.32 last 144.96) - open/close 41212 (time 1734258312.08 total 310.33 last 142.24) - open/close 42610 (time 1734258322.09 total 320.34 last 139.68) - open/close 43818 (time 1734258332.09 total 330.34 last 120.74) - open/close 45132 (time 1734258342.09 total 340.34 last 131.40) - open/close 46487 (time 1734258352.10 total 350.35 last 135.42) - open/close 47839 (time 1734258362.10 total 360.35 last 135.14) - open/close 49214 (time 1734258372.11 total 370.36 last 137.42) - open/close 50000 (time 1734258377.71 total 375.96 last 140.28) - open/close 51405 (time 1734258387.72 total 385.97 last 140.42) - open/close 53033 (time 1734258397.72 total 395.97 last 162.77) - open/close 54660 (time 1734258407.72 total 405.97 last 162.63) - open/close 56250 (time 1734258417.72 total 415.98 last 158.95) - open/close 57980 (time 1734258427.73 total 425.98 last 172.95) - open/close 59499 (time 1734258437.73 total 435.98 last 151.89) - open/close 60000 (time 1734258440.85 total 439.10 last 160.56) - open/close 61552 (time 1734258450.85 total 449.11 last 155.11) - open/close 62979 (time 1734258460.86 total 459.11 last 142.65) - open/close 64631 (time 1734258470.86 total 469.11 last 165.12) total: 64768 open/close in 470.05 seconds: 137.79 ops/second - /unlink 2152 (time 1734258483.54 total 10.00 last 215.20) - /unlink 4496 (time 1734258493.54 total 20.01 last 234.28) - /unlink 7121 (time 1734258503.54 total 30.01 last 262.41) - /unlink 9298 (time 1734258513.55 total 40.01 last 217.66) - /unlink 10000 (time 1734258516.57 total 43.03 last 232.50) - /unlink 12511 (time 1734258526.57 total 53.03 last 251.06) - /unlink 14483 (time 1734258536.57 total 63.03 last 197.19) - /unlink 16650 (time 1734258546.57 total 73.03 last 216.67) - /unlink 19087 (time 1734258556.57 total 83.04 last 243.64) - /unlink 20000 (time 1734258560.83 total 87.30 last 214.29) - /unlink 22010 (time 1734258570.84 total 97.30 last 200.83) - /unlink 24429 (time 1734258580.84 total 107.31 last 241.89) - /unlink 26676 (time 1734258590.84 total 117.31 last 224.61) - /unlink 29059 (time 1734258600.85 total 127.31 last 238.21) - /unlink 30000 (time 1734258604.51 total 130.97 last 257.20) - /unlink 32483 (time 1734258614.51 total 140.97 last 248.30) - /unlink 34949 (time 1734258624.51 total 150.97 last 246.59) - /unlink 37221 (time 1734258634.51 total 160.98 last 227.10) - /unlink 39306 (time 1734258644.52 total 170.99 last 208.29) - /unlink 40000 (time 1734258647.47 total 173.93 last 235.47) - /unlink 42414 (time 1734258657.48 total 183.94 last 241.25) - /unlink 44547 (time 1734258667.48 total 193.94 last 213.27) - /unlink 46709 (time 1734258677.48 total 203.94 last 216.15) - /unlink 49720 (time 1734258687.48 total 213.95 last 301.05) - /unlink 52168 (time 1734258697.48 total 223.95 last 244.74) - /unlink 55062 (time 1734258707.48 total 233.95 last 289.39) - /unlink 57931 (time 1734258717.49 total 243.95 last 286.84) - /unlink 60000 (time 1734258726.53 total 253.00 last 228.66) - /unlink 62351 (time 1734258736.53 total 263.00 last 235.09) - /unlink 64710 (time 1734258746.54 total 273.00 last 235.84) total: 64768 /unlink in 273.25 seconds: 237.02 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1817 (time 1734258773.52 total 10.01 last 181.60) - open/close 3449 (time 1734258783.52 total 20.01 last 163.14) - open/close 5256 (time 1734258793.52 total 30.01 last 180.64) - open/close 7069 (time 1734258803.53 total 40.02 last 181.18) - open/close 8436 (time 1734258813.53 total 50.02 last 136.64) - open/close 9927 (time 1734258823.53 total 60.02 last 149.09) - open/close 11587 (time 1734258833.54 total 70.02 last 165.99) - open/close 13031 (time 1734258843.54 total 80.03 last 144.34) - open/close 14897 (time 1734258853.54 total 90.03 last 186.56) - open/close 16874 (time 1734258863.54 total 100.03 last 197.68) - open/close 19324 (time 1734258873.54 total 110.03 last 244.95) - open/close 20000 (time 1734258876.95 total 113.44 last 198.63) - open/close 21772 (time 1734258886.95 total 123.44 last 177.18) - open/close 23762 (time 1734258896.95 total 133.44 last 198.93) - open/close 25318 (time 1734258906.96 total 143.45 last 155.50) - open/close 26943 (time 1734258916.96 total 153.45 last 162.48) - open/close 28392 (time 1734258926.96 total 163.45 last 144.84) - open/close 30000 (time 1734258935.51 total 172.00 last 188.21) - open/close 31806 (time 1734258945.51 total 182.00 last 180.58) - open/close 33795 (time 1734258955.51 total 192.00 last 198.83) - open/close 35968 (time 1734258965.51 total 202.00 last 217.24) - open/close 37819 (time 1734258975.52 total 212.01 last 185.04) - open/close 39767 (time 1734258985.52 total 222.01 last 194.75) - open/close 41728 (time 1734258995.52 total 232.01 last 196.03) - open/close 43957 (time 1734259005.53 total 242.02 last 222.83) - open/close 46104 (time 1734259015.53 total 252.02 last 214.62) - open/close 47804 (time 1734259025.53 total 262.02 last 169.95) - open/close 49568 (time 1734259035.54 total 272.03 last 176.32) - open/close 50000 (time 1734259038.14 total 274.62 last 166.36) - open/close 51812 (time 1734259048.14 total 284.62 last 181.19) - open/close 53646 (time 1734259058.14 total 294.63 last 183.36) - open/close 55549 (time 1734259068.14 total 304.63 last 190.24) - open/close 57575 (time 1734259078.14 total 314.63 last 202.58) - open/close 59224 (time 1734259088.15 total 324.64 last 164.82) - open/close 60000 (time 1734259092.76 total 329.24 last 168.40) - open/close 61863 (time 1734259102.76 total 339.25 last 186.25) - open/close 63545 (time 1734259112.76 total 349.25 last 168.12) total: 64768 open/close in 355.01 seconds: 182.44 ops/second - /unlink 3356 (time 1734259129.61 total 10.00 last 335.50) - /unlink 6475 (time 1734259139.61 total 20.00 last 311.87) - /unlink 9925 (time 1734259149.61 total 30.00 last 344.98) - /unlink 13018 (time 1734259159.62 total 40.01 last 309.27) - /unlink 16512 (time 1734259169.62 total 50.01 last 349.31) - /unlink 19694 (time 1734259179.62 total 60.01 last 318.18) - /unlink 22893 (time 1734259189.62 total 70.01 last 319.83) - /unlink 26286 (time 1734259199.62 total 80.01 last 339.27) - /unlink 29238 (time 1734259209.62 total 90.01 last 295.15) - /unlink 30000 (time 1734259211.83 total 92.22 last 345.72) - /unlink 33739 (time 1734259221.83 total 102.22 last 373.87) - /unlink 36361 (time 1734259231.83 total 112.22 last 262.16) - /unlink 39372 (time 1734259241.83 total 122.22 last 300.99) - /unlink 42804 (time 1734259251.84 total 132.23 last 343.13) - /unlink 46190 (time 1734259261.84 total 142.23 last 338.56) - /unlink 49725 (time 1734259271.84 total 152.23 last 353.49) - /unlink 53660 (time 1734259281.84 total 162.23 last 393.43) - /unlink 58079 (time 1734259291.84 total 172.23 last 441.90) - /unlink 60000 (time 1734259296.88 total 177.27 last 381.15) - /unlink 63305 (time 1734259306.88 total 187.27 last 330.50) total: 64768 /unlink in 191.58 seconds: 338.07 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1946 (time 1734259338.86 total 10.00 last 194.56) - open/close 4027 (time 1734259348.87 total 20.01 last 207.90) - open/close 6193 (time 1734259358.87 total 30.01 last 216.55) - open/close 8346 (time 1734259368.87 total 40.02 last 215.24) - open/close 10000 (time 1734259377.87 total 49.02 last 183.77) - open/close 12107 (time 1734259387.87 total 59.02 last 210.64) - open/close 14059 (time 1734259397.88 total 69.02 last 195.12) - open/close 16127 (time 1734259407.88 total 79.03 last 206.75) - open/close 18069 (time 1734259417.89 total 89.03 last 194.13) - open/close 19900 (time 1734259427.89 total 99.03 last 183.03) - open/close 21842 (time 1734259437.89 total 109.04 last 194.13) - open/close 23525 (time 1734259447.89 total 119.04 last 168.29) - open/close 25196 (time 1734259457.89 total 129.04 last 167.07) - open/close 27114 (time 1734259467.90 total 139.04 last 191.79) - open/close 29212 (time 1734259477.90 total 149.05 last 209.71) - open/close 30000 (time 1734259481.88 total 153.02 last 198.18) - open/close 31810 (time 1734259491.88 total 163.02 last 180.94) - open/close 33767 (time 1734259501.89 total 173.03 last 195.54) - open/close 35609 (time 1734259511.89 total 183.04 last 184.15) - open/close 37611 (time 1734259521.89 total 193.04 last 200.13) - open/close 39414 (time 1734259531.90 total 203.04 last 180.25) - open/close 40000 (time 1734259535.03 total 206.18 last 186.73) - open/close 41801 (time 1734259545.04 total 216.18 last 180.08) - open/close 43661 (time 1734259555.04 total 226.18 last 185.96) - open/close 45603 (time 1734259565.04 total 236.19 last 194.15) - open/close 47617 (time 1734259575.04 total 246.19 last 201.32) - open/close 49443 (time 1734259585.05 total 256.19 last 182.58) - open/close 50000 (time 1734259587.62 total 258.77 last 216.22) - open/close 52136 (time 1734259597.63 total 268.77 last 213.49) - open/close 54249 (time 1734259607.63 total 278.77 last 211.27) - open/close 56390 (time 1734259617.63 total 288.78 last 214.05) - open/close 58483 (time 1734259627.63 total 298.78 last 209.25) - open/close 60000 (time 1734259633.93 total 305.08 last 240.90) - open/close 62606 (time 1734259643.93 total 315.08 last 260.57) total: 64768 open/close in 324.47 seconds: 199.61 ops/second - /unlink 3984 (time 1734259664.38 total 10.00 last 398.25) - /unlink 7400 (time 1734259674.38 total 20.01 last 341.55) - /unlink 10000 (time 1734259681.22 total 26.85 last 380.00) - /unlink 14118 (time 1734259691.22 total 36.85 last 411.73) - /unlink 18887 (time 1734259701.22 total 46.85 last 476.85) - /unlink 20000 (time 1734259703.61 total 49.24 last 466.35) - /unlink 24557 (time 1734259713.61 total 59.24 last 455.56) - /unlink 28227 (time 1734259723.61 total 69.24 last 366.95) - /unlink 30000 (time 1734259727.52 total 73.14 last 454.32) - /unlink 33891 (time 1734259737.52 total 83.14 last 389.07) - /unlink 37111 (time 1734259747.52 total 93.15 last 321.98) - /unlink 40000 (time 1734259754.87 total 100.50 last 392.93) - /unlink 43314 (time 1734259764.87 total 110.50 last 331.38) - /unlink 46416 (time 1734259774.87 total 120.50 last 310.17) - /unlink 49631 (time 1734259784.88 total 130.50 last 321.41) - /unlink 53078 (time 1734259794.88 total 140.50 last 344.63) - /unlink 57236 (time 1734259804.88 total 150.51 last 415.72) - /unlink 60000 (time 1734259810.80 total 156.43 last 466.98) - /unlink 64448 (time 1734259820.80 total 166.43 last 444.80) total: 64768 /unlink in 167.08 seconds: 387.64 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2409 (time 1734259849.93 total 10.00 last 240.81) - open/close 4242 (time 1734259859.93 total 20.01 last 183.23) - open/close 6408 (time 1734259869.94 total 30.01 last 216.48) - open/close 8388 (time 1734259879.94 total 40.02 last 197.91) - open/close 10000 (time 1734259887.70 total 47.78 last 207.67) - open/close 12419 (time 1734259897.78 total 57.85 last 240.08) - open/close 14523 (time 1734259907.78 total 67.86 last 210.36) - open/close 16848 (time 1734259917.79 total 77.86 last 232.38) - open/close 19555 (time 1734259927.79 total 87.86 last 270.69) - open/close 22437 (time 1734259937.79 total 97.86 last 288.15) - open/close 25286 (time 1734259947.79 total 107.87 last 284.87) - open/close 28315 (time 1734259957.79 total 117.87 last 302.89) - open/close 30000 (time 1734259963.78 total 123.85 last 281.41) - open/close 33012 (time 1734259973.78 total 133.85 last 301.20) - open/close 36161 (time 1734259983.78 total 143.85 last 314.87) - open/close 39107 (time 1734259993.78 total 153.86 last 294.54) - open/close 40000 (time 1734259996.87 total 156.95 last 288.54) - open/close 43105 (time 1734260006.88 total 166.95 last 310.44) - open/close 46039 (time 1734260016.88 total 176.96 last 293.33) - open/close 49035 (time 1734260026.88 total 186.96 last 299.53) - open/close 50000 (time 1734260029.98 total 190.06 last 311.15) - open/close 53046 (time 1734260039.99 total 200.06 last 304.52) - open/close 55624 (time 1734260049.99 total 210.06 last 257.79) - open/close 58461 (time 1734260059.99 total 220.07 last 283.63) - open/close 60000 (time 1734260064.68 total 224.75 last 328.40) - open/close 63420 (time 1734260074.68 total 234.75 last 341.96) total: 64768 open/close in 239.18 seconds: 270.79 ops/second - /unlink 5155 (time 1734260090.11 total 10.00 last 515.45) - /unlink 10000 (time 1734260099.28 total 19.17 last 528.39) - /unlink 15076 (time 1734260109.28 total 29.17 last 507.53) - /unlink 20000 (time 1734260118.07 total 37.96 last 559.98) - /unlink 25111 (time 1734260128.07 total 47.97 last 511.10) - /unlink 29958 (time 1734260138.07 total 57.97 last 484.69) - /unlink 35402 (time 1734260148.07 total 67.97 last 544.38) - /unlink 40000 (time 1734260156.75 total 76.65 last 529.46) - /unlink 45790 (time 1734260166.75 total 86.65 last 579.00) - /unlink 50000 (time 1734260174.48 total 94.38 last 544.76) - /unlink 55176 (time 1734260184.48 total 104.38 last 517.54) - /unlink 59859 (time 1734260194.48 total 114.38 last 468.27) total: 64768 /unlink in 123.04 seconds: 526.41 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2546 (time 1734260229.68 total 10.00 last 254.55) - open/close 5409 (time 1734260239.68 total 20.00 last 286.30) - open/close 8307 (time 1734260249.68 total 30.00 last 289.74) - open/close 10000 (time 1734260255.55 total 35.87 last 288.69) - open/close 12931 (time 1734260265.55 total 45.87 last 293.06) - open/close 16056 (time 1734260275.55 total 55.87 last 312.41) - open/close 19076 (time 1734260285.55 total 65.87 last 302.00) - open/close 20000 (time 1734260288.37 total 68.69 last 328.44) - open/close 23017 (time 1734260298.37 total 78.69 last 301.62) - open/close 25567 (time 1734260308.37 total 88.69 last 254.92) - open/close 28036 (time 1734260318.37 total 98.69 last 246.84) - open/close 30000 (time 1734260325.94 total 106.27 last 259.41) - open/close 32607 (time 1734260335.95 total 116.27 last 260.58) - open/close 34932 (time 1734260345.95 total 126.27 last 232.46) - open/close 37390 (time 1734260355.96 total 136.28 last 245.62) - open/close 39901 (time 1734260365.96 total 146.28 last 251.03) - open/close 42546 (time 1734260375.96 total 156.28 last 264.49) - open/close 45048 (time 1734260385.96 total 166.28 last 250.19) - open/close 47718 (time 1734260395.96 total 176.28 last 266.98) - open/close 50000 (time 1734260403.56 total 183.88 last 300.48) - open/close 52780 (time 1734260413.56 total 193.88 last 277.94) - open/close 55431 (time 1734260423.56 total 203.88 last 265.10) - open/close 58151 (time 1734260433.56 total 213.88 last 271.98) - open/close 60000 (time 1734260441.35 total 221.67 last 237.48) - open/close 63092 (time 1734260451.35 total 231.67 last 309.17) total: 64768 open/close in 237.05 seconds: 273.22 ops/second - /unlink 5761 (time 1734260467.71 total 10.00 last 576.01) - /unlink 10000 (time 1734260475.89 total 18.18 last 518.13) - /unlink 14658 (time 1734260485.90 total 28.18 last 465.76) - /unlink 19356 (time 1734260495.90 total 38.18 last 469.76) - /unlink 25038 (time 1734260505.90 total 48.19 last 568.15) - /unlink 30000 (time 1734260515.23 total 57.51 last 531.87) - /unlink 34727 (time 1734260525.23 total 67.52 last 472.60) - /unlink 39687 (time 1734260535.23 total 77.52 last 495.96) - /unlink 44543 (time 1734260545.23 total 87.52 last 485.51) - /unlink 49402 (time 1734260555.23 total 97.52 last 485.85) - /unlink 54368 (time 1734260565.23 total 107.52 last 496.55) - /unlink 59827 (time 1734260575.23 total 117.52 last 545.90) total: 64768 /unlink in 126.58 seconds: 511.68 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg256-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg256-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg256-server stop mds service on oleg256-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg256-server unloading modules via unload_modules_local on: 'oleg256-server' oleg256-server: oleg256-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg256-client: oleg256-client: ssh exited with exit code 2 pdsh@oleg256-client: oleg256-server: ssh exited with exit code 2 pdsh@oleg256-client: oleg256-client: ssh exited with exit code 2 pdsh@oleg256-client: oleg256-server: ssh exited with exit code 2