== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:34:08 (1734258848) Stopping clients: oleg209-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg209-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg209-client: oleg209-server: ssh exited with exit code 2 oleg209-server: oleg209-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg209-server' oleg209-server: oleg209-server.virtnet: executing load_modules_local oleg209-server: Loading modules from /home/green/git/lustre-release/lustre oleg209-server: detected 4 online CPUs by sysfs oleg209-server: Force libcfs to create 2 CPU partitions oleg209-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg209-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg209-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg209-server: oleg209-server.virtnet: executing set_default_debug -1 all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg209-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg209-server: oleg209-server.virtnet: executing set_default_debug -1 all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg209-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg209-server: oleg209-server.virtnet: executing set_default_debug -1 all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86890 331 86559 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 87086 312 86774 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 16131 291 15840 2% /mnt/lustre[OST:0] filesystem_summary: 16483 643 15840 4% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1970 (time 1734258906.66 total 10.00 last 196.98) - open/close 3727 (time 1734258916.67 total 20.01 last 175.59) - open/close 5602 (time 1734258926.67 total 30.01 last 187.49) - open/close 7838 (time 1734258936.67 total 40.01 last 223.52) - open/close 10000 (time 1734258945.84 total 49.18 last 235.90) - open/close 12463 (time 1734258955.84 total 59.18 last 246.20) - open/close 14930 (time 1734258965.85 total 69.19 last 246.56) - open/close 17168 (time 1734258975.85 total 79.19 last 223.64) - open/close 19367 (time 1734258985.85 total 89.19 last 219.89) - open/close 20000 (time 1734258988.74 total 92.08 last 219.64) - open/close 22526 (time 1734258998.74 total 102.08 last 252.49) - open/close 25105 (time 1734259008.74 total 112.08 last 257.87) - open/close 27514 (time 1734259018.74 total 122.08 last 240.86) - open/close 29438 (time 1734259028.75 total 132.09 last 192.36) - open/close 30000 (time 1734259031.78 total 135.12 last 185.36) - open/close 32075 (time 1734259041.78 total 145.12 last 207.46) - open/close 34246 (time 1734259051.78 total 155.12 last 217.06) - open/close 36335 (time 1734259061.78 total 165.12 last 208.85) - open/close 38590 (time 1734259071.79 total 175.13 last 225.43) - open/close 40000 (time 1734259078.15 total 181.49 last 221.76) - open/close 42037 (time 1734259088.15 total 191.49 last 203.61) - open/close 44085 (time 1734259098.15 total 201.49 last 204.79) - open/close 45996 (time 1734259108.15 total 211.49 last 191.09) - open/close 48341 (time 1734259118.15 total 221.49 last 234.48) - open/close 50000 (time 1734259125.81 total 229.15 last 216.64) - open/close 52056 (time 1734259135.81 total 239.15 last 205.58) - open/close 54141 (time 1734259145.81 total 249.15 last 208.46) - open/close 56191 (time 1734259155.82 total 259.16 last 204.84) - open/close 58380 (time 1734259165.82 total 269.16 last 218.84) - open/close 60000 (time 1734259173.52 total 276.86 last 210.50) - open/close 61925 (time 1734259183.52 total 286.86 last 192.50) - open/close 64259 (time 1734259193.52 total 296.86 last 233.36) total: 64768 open/close in 299.40 seconds: 216.32 ops/second - /unlink 3464 (time 1734259207.21 total 10.00 last 346.38) - /unlink 7532 (time 1734259217.21 total 20.00 last 406.77) - /unlink 10000 (time 1734259224.58 total 27.37 last 334.96) - /unlink 13090 (time 1734259234.58 total 37.37 last 309.00) - /unlink 16469 (time 1734259244.58 total 47.37 last 337.83) - /unlink 19989 (time 1734259254.58 total 57.37 last 351.90) - /unlink 23315 (time 1734259264.58 total 67.38 last 332.56) - /unlink 27006 (time 1734259274.59 total 77.38 last 369.07) - /unlink 30000 (time 1734259281.15 total 83.95 last 455.78) - /unlink 34283 (time 1734259291.16 total 93.95 last 428.27) - /unlink 37941 (time 1734259301.16 total 103.95 last 365.73) - /unlink 40000 (time 1734259307.37 total 110.16 last 331.45) - /unlink 42892 (time 1734259317.37 total 120.16 last 289.17) - /unlink 45954 (time 1734259327.37 total 130.16 last 306.11) - /unlink 48873 (time 1734259337.37 total 140.16 last 291.89) - /unlink 50000 (time 1734259340.51 total 143.31 last 358.85) - /unlink 53860 (time 1734259350.51 total 153.31 last 386.00) - /unlink 57623 (time 1734259360.52 total 163.31 last 376.20) - /unlink 60000 (time 1734259366.58 total 169.37 last 391.85) - /unlink 63568 (time 1734259376.58 total 179.38 last 356.75) total: 64768 /unlink in 182.24 seconds: 355.39 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2221 (time 1734259403.51 total 10.00 last 222.07) - open/close 4521 (time 1734259413.52 total 20.01 last 229.91) - open/close 6626 (time 1734259423.52 total 30.01 last 210.44) - open/close 8660 (time 1734259433.52 total 40.01 last 203.38) - open/close 10000 (time 1734259439.56 total 46.05 last 221.76) - open/close 11925 (time 1734259449.56 total 56.05 last 192.46) - open/close 13883 (time 1734259459.57 total 66.06 last 195.71) - open/close 16112 (time 1734259469.57 total 76.06 last 222.88) - open/close 18415 (time 1734259479.57 total 86.06 last 230.29) - open/close 20000 (time 1734259487.62 total 94.11 last 196.78) - open/close 22264 (time 1734259497.63 total 104.12 last 226.38) - open/close 24181 (time 1734259507.63 total 114.12 last 191.64) - open/close 26423 (time 1734259517.63 total 124.12 last 224.18) - open/close 28481 (time 1734259527.63 total 134.12 last 205.79) - open/close 30000 (time 1734259535.81 total 142.30 last 185.72) - open/close 31990 (time 1734259545.81 total 152.30 last 198.99) - open/close 34037 (time 1734259555.82 total 162.31 last 204.43) - open/close 36195 (time 1734259565.83 total 172.32 last 215.74) - open/close 38350 (time 1734259575.83 total 182.32 last 215.35) - open/close 40000 (time 1734259583.28 total 189.77 last 221.55) - open/close 42338 (time 1734259593.29 total 199.78 last 233.61) - open/close 44790 (time 1734259603.29 total 209.78 last 245.13) - open/close 47122 (time 1734259613.29 total 219.78 last 233.15) - open/close 49447 (time 1734259623.30 total 229.79 last 232.44) - open/close 50000 (time 1734259625.52 total 232.01 last 249.08) - open/close 52446 (time 1734259635.52 total 242.01 last 244.59) - open/close 54993 (time 1734259645.52 total 252.01 last 254.66) - open/close 57587 (time 1734259655.52 total 262.01 last 259.33) - open/close 59886 (time 1734259665.53 total 272.02 last 229.78) - open/close 62256 (time 1734259675.53 total 282.02 last 236.92) total: 64768 open/close in 291.95 seconds: 221.85 ops/second - /unlink 4823 (time 1734259696.60 total 10.00 last 482.27) - /unlink 9982 (time 1734259706.60 total 20.00 last 515.82) - /unlink 14285 (time 1734259716.60 total 30.00 last 430.27) - /unlink 18486 (time 1734259726.60 total 40.00 last 420.07) - /unlink 20000 (time 1734259729.95 total 43.35 last 452.32) - /unlink 24004 (time 1734259739.95 total 53.35 last 400.28) - /unlink 27992 (time 1734259749.96 total 63.36 last 398.73) - /unlink 30000 (time 1734259754.56 total 67.96 last 435.99) - /unlink 34102 (time 1734259764.56 total 77.96 last 410.15) - /unlink 37682 (time 1734259774.57 total 87.96 last 357.93) - /unlink 40000 (time 1734259780.66 total 94.06 last 380.18) - /unlink 44059 (time 1734259790.66 total 104.06 last 405.87) - /unlink 48175 (time 1734259800.66 total 114.06 last 411.57) - /unlink 50000 (time 1734259804.17 total 117.57 last 520.46) - /unlink 54963 (time 1734259814.17 total 127.57 last 496.20) - /unlink 59727 (time 1734259824.17 total 137.57 last 476.37) total: 64768 /unlink in 147.54 seconds: 438.98 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2260 (time 1734259859.68 total 10.00 last 225.99) - open/close 4874 (time 1734259869.68 total 20.00 last 261.31) - open/close 7263 (time 1734259879.69 total 30.01 last 238.83) - open/close 9869 (time 1734259889.69 total 40.01 last 260.57) - open/close 12438 (time 1734259899.69 total 50.01 last 256.80) - open/close 15200 (time 1734259909.69 total 60.02 last 276.11) - open/close 17668 (time 1734259919.70 total 70.02 last 246.75) - open/close 20000 (time 1734259927.54 total 77.86 last 297.50) - open/close 22895 (time 1734259937.54 total 87.86 last 289.48) - open/close 25762 (time 1734259947.54 total 97.86 last 286.66) - open/close 28778 (time 1734259957.54 total 107.86 last 301.51) - open/close 30000 (time 1734259961.95 total 112.27 last 277.07) - open/close 33040 (time 1734259971.95 total 122.27 last 303.91) - open/close 36340 (time 1734259981.96 total 132.28 last 329.96) - open/close 39568 (time 1734259991.96 total 142.28 last 322.71) - open/close 42298 (time 1734260001.96 total 152.28 last 272.94) - open/close 45301 (time 1734260011.96 total 162.28 last 300.30) - open/close 48090 (time 1734260021.96 total 172.28 last 278.82) - open/close 50000 (time 1734260028.21 total 178.53 last 305.60) - open/close 52927 (time 1734260038.21 total 188.53 last 292.69) - open/close 55470 (time 1734260048.21 total 198.53 last 254.29) - open/close 58038 (time 1734260058.22 total 208.54 last 256.72) - open/close 60000 (time 1734260064.71 total 215.04 last 301.97) - open/close 63131 (time 1734260074.72 total 225.04 last 313.06) total: 64768 open/close in 231.30 seconds: 280.02 ops/second - /unlink 4632 (time 1734260092.06 total 10.00 last 463.11) - /unlink 9029 (time 1734260102.06 total 20.00 last 439.64) - /unlink 10000 (time 1734260104.21 total 22.15 last 451.60) - /unlink 15014 (time 1734260114.22 total 32.16 last 501.30) - /unlink 19519 (time 1734260124.22 total 42.16 last 450.48) - /unlink 23880 (time 1734260134.22 total 52.16 last 436.05) - /unlink 28304 (time 1734260144.22 total 62.16 last 442.39) - /unlink 30000 (time 1734260147.75 total 65.69 last 480.34) - /unlink 34516 (time 1734260157.75 total 75.69 last 451.56) - /unlink 39331 (time 1734260167.75 total 85.69 last 481.48) - /unlink 43870 (time 1734260177.75 total 95.69 last 453.85) - /unlink 47855 (time 1734260187.75 total 105.69 last 398.46) - /unlink 50000 (time 1734260193.63 total 111.57 last 365.11) - /unlink 54460 (time 1734260203.63 total 121.57 last 445.96) - /unlink 58863 (time 1734260213.63 total 131.57 last 440.30) - /unlink 60000 (time 1734260216.39 total 134.33 last 411.97) - /unlink 64173 (time 1734260226.39 total 144.33 last 417.23) total: 64768 /unlink in 146.12 seconds: 443.25 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2663 (time 1734260255.01 total 10.00 last 266.25) - open/close 5326 (time 1734260265.01 total 20.00 last 266.22) - open/close 8209 (time 1734260275.02 total 30.01 last 288.27) - open/close 10000 (time 1734260281.20 total 36.19 last 289.73) - open/close 12967 (time 1734260291.20 total 46.19 last 296.70) - open/close 15676 (time 1734260301.20 total 56.19 last 270.87) - open/close 17955 (time 1734260311.20 total 66.19 last 227.85) - open/close 20000 (time 1734260319.73 total 74.72 last 239.71) - open/close 22615 (time 1734260329.73 total 84.72 last 261.46) - open/close 24942 (time 1734260339.73 total 94.72 last 232.68) - open/close 27261 (time 1734260349.73 total 104.72 last 231.90) - open/close 29605 (time 1734260359.73 total 114.73 last 234.39) - open/close 32201 (time 1734260369.74 total 124.73 last 259.53) - open/close 34627 (time 1734260379.74 total 134.73 last 242.55) - open/close 36957 (time 1734260389.74 total 144.73 last 232.99) - open/close 39932 (time 1734260399.74 total 154.73 last 297.47) - open/close 42726 (time 1734260409.74 total 164.73 last 279.39) - open/close 45350 (time 1734260419.74 total 174.73 last 262.34) - open/close 48093 (time 1734260429.75 total 184.74 last 274.16) - open/close 50000 (time 1734260437.78 total 192.77 last 237.31) - open/close 52717 (time 1734260447.78 total 202.78 last 271.67) - open/close 56130 (time 1734260457.79 total 212.78 last 341.26) - open/close 59170 (time 1734260467.79 total 222.78 last 303.99) - open/close 60000 (time 1734260470.66 total 225.65 last 288.98) - open/close 62686 (time 1734260480.66 total 235.65 last 268.53) total: 64768 open/close in 244.39 seconds: 265.02 ops/second - /unlink 4621 (time 1734260500.44 total 10.00 last 462.06) - /unlink 9870 (time 1734260510.44 total 20.00 last 524.84) - /unlink 14522 (time 1734260520.44 total 30.00 last 465.17) - /unlink 19200 (time 1734260530.44 total 40.00 last 467.78) - /unlink 24120 (time 1734260540.44 total 50.00 last 491.94) - /unlink 28666 (time 1734260550.44 total 60.01 last 454.51) - /unlink 30000 (time 1734260553.52 total 63.08 last 433.75) - /unlink 33668 (time 1734260563.52 total 73.08 last 366.72) - /unlink 37451 (time 1734260573.52 total 83.09 last 378.19) - /unlink 40000 (time 1734260579.53 total 89.09 last 424.30) - /unlink 43290 (time 1734260589.53 total 99.10 last 328.96) - /unlink 48030 (time 1734260599.53 total 109.10 last 473.99) - /unlink 50000 (time 1734260603.64 total 113.20 last 479.73) - /unlink 53442 (time 1734260613.64 total 123.20 last 344.20) - /unlink 58055 (time 1734260623.64 total 133.20 last 461.25) - /unlink 60000 (time 1734260628.18 total 137.75 last 428.27) - /unlink 64749 (time 1734260638.18 total 147.75 last 474.87) total: 64768 /unlink in 147.78 seconds: 438.27 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3025 (time 1734260665.97 total 10.00 last 302.40) - open/close 5609 (time 1734260675.98 total 20.00 last 258.36) - open/close 8382 (time 1734260685.98 total 30.01 last 277.16) - open/close 10000 (time 1734260692.82 total 36.85 last 236.56) - open/close 12549 (time 1734260702.82 total 46.85 last 254.89) - open/close 14999 (time 1734260712.82 total 56.85 last 244.96) - open/close 17826 (time 1734260722.82 total 66.85 last 282.63) - open/close 20000 (time 1734260731.69 total 75.72 last 245.17) - open/close 22488 (time 1734260741.70 total 85.72 last 248.72) - open/close 24812 (time 1734260751.70 total 95.73 last 232.37) - open/close 27395 (time 1734260761.70 total 105.73 last 258.20) - open/close 29882 (time 1734260771.70 total 115.73 last 248.66) - open/close 32949 (time 1734260781.70 total 125.73 last 306.66) - open/close 35732 (time 1734260791.71 total 135.74 last 278.23) - open/close 39102 (time 1734260801.71 total 145.74 last 336.93) - open/close 40000 (time 1734260804.61 total 148.64 last 309.30) - open/close 43086 (time 1734260814.61 total 158.64 last 308.50) - open/close 46376 (time 1734260824.62 total 168.65 last 328.89) - open/close 49270 (time 1734260834.62 total 178.65 last 289.35) - open/close 50000 (time 1734260837.05 total 181.08 last 300.52) - open/close 52897 (time 1734260847.05 total 191.08 last 289.67) - open/close 55888 (time 1734260857.05 total 201.08 last 299.09) - open/close 58788 (time 1734260867.05 total 211.08 last 289.91) - open/close 60000 (time 1734260871.41 total 215.44 last 278.43) - open/close 63330 (time 1734260881.47 total 225.50 last 330.93) total: 64768 open/close in 229.15 seconds: 282.65 ops/second - /unlink 6707 (time 1734260896.06 total 10.00 last 670.67) - /unlink 10000 (time 1734260901.21 total 15.15 last 639.23) - /unlink 15878 (time 1734260911.21 total 25.15 last 587.71) - /unlink 20000 (time 1734260918.92 total 32.86 last 534.92) - /unlink 24889 (time 1734260928.92 total 42.86 last 488.83) - /unlink 29859 (time 1734260938.92 total 52.86 last 496.96) - /unlink 34917 (time 1734260948.92 total 62.86 last 505.73) - /unlink 40000 (time 1734260958.76 total 72.70 last 516.48) - /unlink 44850 (time 1734260968.77 total 82.71 last 484.96) - /unlink 50000 (time 1734260977.47 total 91.41 last 591.34) - /unlink 56278 (time 1734260987.47 total 101.41 last 627.79) - /unlink 60000 (time 1734260993.05 total 106.99 last 667.74) total: 64768 /unlink in 116.71 seconds: 554.96 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg209-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg209-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg209-server stop mds service on oleg209-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg209-server stop mds service on oleg209-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg209-server unloading modules via unload_modules_local on: 'oleg209-server' oleg209-server: oleg209-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg209-client: oleg209-client: ssh exited with exit code 2 pdsh@oleg209-client: oleg209-server: ssh exited with exit code 2 pdsh@oleg209-client: oleg209-client: ssh exited with exit code 2 pdsh@oleg209-client: oleg209-server: ssh exited with exit code 2