== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:34:16 (1734258856) Stopping clients: oleg131-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg131-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg131-client: oleg131-server: ssh exited with exit code 2 oleg131-server: oleg131-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg131-server' oleg131-server: oleg131-server.virtnet: executing load_modules_local oleg131-server: Loading modules from /home/green/git/lustre-release/lustre oleg131-server: detected 4 online CPUs by sysfs oleg131-server: Force libcfs to create 2 CPU partitions oleg131-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg131-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg131-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg131-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg131-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86890 331 86559 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 87086 312 86774 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 16266 458 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16451 643 15808 4% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1692 (time 1734258933.17 total 10.00 last 169.12) - open/close 3418 (time 1734258943.17 total 20.00 last 172.60) - open/close 5144 (time 1734258953.18 total 30.01 last 172.52) - open/close 6730 (time 1734258963.18 total 40.01 last 158.59) - open/close 8065 (time 1734258973.18 total 50.01 last 133.44) - open/close 9453 (time 1734258983.19 total 60.02 last 138.73) - open/close 10000 (time 1734258987.21 total 64.04 last 136.12) - open/close 11336 (time 1734258997.21 total 74.04 last 133.56) - open/close 12748 (time 1734259007.21 total 84.04 last 141.18) - open/close 14184 (time 1734259017.21 total 94.05 last 143.55) - open/close 15686 (time 1734259027.22 total 104.05 last 150.11) - open/close 17010 (time 1734259037.23 total 114.06 last 132.32) - open/close 18388 (time 1734259047.23 total 124.06 last 137.79) - open/close 19823 (time 1734259057.24 total 134.07 last 143.39) - open/close 21147 (time 1734259067.24 total 144.08 last 132.29) - open/close 22454 (time 1734259077.25 total 154.08 last 130.65) - open/close 23919 (time 1734259087.26 total 164.09 last 146.36) - open/close 25306 (time 1734259097.26 total 174.09 last 138.62) - open/close 26712 (time 1734259107.27 total 184.10 last 140.53) - open/close 28013 (time 1734259117.27 total 194.10 last 130.04) - open/close 29308 (time 1734259127.27 total 204.10 last 129.48) - open/close 30000 (time 1734259133.01 total 209.84 last 120.64) - open/close 31271 (time 1734259143.01 total 219.85 last 127.04) - open/close 32511 (time 1734259153.02 total 229.85 last 123.98) - open/close 33703 (time 1734259163.02 total 239.85 last 119.19) - open/close 34949 (time 1734259173.02 total 249.85 last 124.60) - open/close 36203 (time 1734259183.03 total 259.86 last 125.30) - open/close 37507 (time 1734259193.04 total 269.87 last 130.28) - open/close 38762 (time 1734259203.04 total 279.87 last 125.50) - open/close 40000 (time 1734259212.77 total 289.60 last 127.15) - open/close 41239 (time 1734259222.78 total 299.61 last 123.85) - open/close 42548 (time 1734259232.78 total 309.61 last 130.86) - open/close 43773 (time 1734259242.78 total 319.62 last 122.44) - open/close 45149 (time 1734259252.79 total 329.62 last 137.53) - open/close 46490 (time 1734259262.80 total 339.63 last 134.01) - open/close 47838 (time 1734259272.80 total 349.63 last 134.75) - open/close 49382 (time 1734259282.80 total 359.63 last 154.36) - open/close 50000 (time 1734259286.98 total 363.81 last 148.00) - open/close 51297 (time 1734259296.98 total 373.81 last 129.65) - open/close 52729 (time 1734259306.98 total 383.81 last 143.18) - open/close 54060 (time 1734259316.99 total 393.82 last 133.05) - open/close 55422 (time 1734259326.99 total 403.83 last 136.09) - open/close 56647 (time 1734259337.00 total 413.83 last 122.43) - open/close 57924 (time 1734259347.00 total 423.83 last 127.69) - open/close 59332 (time 1734259357.01 total 433.84 last 140.67) - open/close 60000 (time 1734259361.92 total 438.75 last 135.97) - open/close 61271 (time 1734259371.92 total 448.75 last 127.09) - open/close 62780 (time 1734259381.93 total 458.76 last 150.84) - open/close 64320 (time 1734259391.93 total 468.76 last 153.97) total: 64768 open/close in 471.83 seconds: 137.27 ops/second - /unlink 2588 (time 1734259406.40 total 10.00 last 258.70) - /unlink 5327 (time 1734259416.40 total 20.01 last 273.86) - /unlink 8106 (time 1734259426.41 total 30.01 last 277.80) - /unlink 10000 (time 1734259433.36 total 36.96 last 272.32) - /unlink 12649 (time 1734259443.37 total 46.97 last 264.82) - /unlink 15142 (time 1734259453.37 total 56.97 last 249.25) - /unlink 17995 (time 1734259463.37 total 66.97 last 285.26) - /unlink 20000 (time 1734259469.82 total 73.42 last 310.78) - /unlink 23119 (time 1734259479.82 total 83.42 last 311.88) - /unlink 26391 (time 1734259489.82 total 93.42 last 327.15) - /unlink 29615 (time 1734259499.83 total 103.43 last 322.35) - /unlink 32810 (time 1734259509.83 total 113.43 last 319.43) - /unlink 35903 (time 1734259519.83 total 123.43 last 309.25) - /unlink 38980 (time 1734259529.83 total 133.43 last 307.65) - /unlink 40000 (time 1734259533.44 total 137.04 last 282.59) - /unlink 42653 (time 1734259543.44 total 147.04 last 265.21) - /unlink 44975 (time 1734259553.45 total 157.05 last 232.09) - /unlink 47338 (time 1734259563.45 total 167.05 last 236.21) - /unlink 49839 (time 1734259573.45 total 177.05 last 250.05) - /unlink 52386 (time 1734259583.46 total 187.06 last 254.64) - /unlink 54796 (time 1734259593.46 total 197.06 last 240.96) - /unlink 57363 (time 1734259603.46 total 207.06 last 256.65) - /unlink 60000 (time 1734259612.49 total 216.09 last 291.92) - /unlink 62872 (time 1734259622.50 total 226.10 last 287.13) total: 64768 /unlink in 233.35 seconds: 277.55 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1584 (time 1734259658.99 total 10.00 last 158.35) - open/close 3111 (time 1734259668.99 total 20.01 last 152.62) - open/close 4647 (time 1734259678.99 total 30.01 last 153.59) - open/close 6325 (time 1734259689.00 total 40.02 last 167.68) - open/close 7896 (time 1734259699.00 total 50.02 last 157.05) - open/close 9447 (time 1734259709.01 total 60.02 last 155.09) - open/close 10000 (time 1734259712.67 total 63.68 last 150.99) - open/close 11392 (time 1734259722.67 total 73.69 last 139.14) - open/close 12850 (time 1734259732.68 total 83.69 last 145.74) - open/close 14422 (time 1734259742.68 total 93.69 last 157.15) - open/close 16024 (time 1734259752.69 total 103.70 last 160.10) - open/close 17562 (time 1734259762.69 total 113.70 last 153.78) - open/close 19126 (time 1734259772.69 total 123.70 last 156.36) - open/close 20000 (time 1734259777.45 total 128.46 last 183.69) - open/close 21778 (time 1734259787.45 total 138.46 last 177.75) - open/close 23657 (time 1734259797.45 total 148.47 last 187.82) - open/close 25326 (time 1734259807.46 total 158.47 last 166.84) - open/close 27145 (time 1734259817.46 total 168.48 last 181.84) - open/close 28834 (time 1734259827.46 total 178.48 last 168.84) - open/close 30000 (time 1734259834.98 total 186.00 last 155.06) - open/close 31520 (time 1734259844.99 total 196.00 last 151.94) - open/close 33219 (time 1734259854.99 total 206.01 last 169.81) - open/close 34759 (time 1734259865.00 total 216.01 last 153.94) - open/close 36240 (time 1734259875.01 total 226.02 last 147.97) - open/close 37754 (time 1734259885.01 total 236.02 last 151.39) - open/close 39389 (time 1734259895.01 total 246.02 last 163.48) - open/close 40000 (time 1734259898.99 total 250.01 last 153.30) - open/close 41748 (time 1734259908.99 total 260.01 last 174.78) - open/close 43569 (time 1734259919.00 total 270.01 last 182.07) - open/close 45386 (time 1734259929.00 total 280.01 last 181.65) - open/close 47158 (time 1734259939.00 total 290.01 last 177.19) - open/close 48954 (time 1734259949.00 total 300.02 last 179.51) - open/close 50000 (time 1734259954.46 total 305.48 last 191.68) - open/close 51940 (time 1734259964.47 total 315.48 last 193.90) - open/close 53781 (time 1734259974.47 total 325.48 last 184.09) - open/close 55715 (time 1734259984.47 total 335.48 last 193.38) - open/close 57613 (time 1734259994.47 total 345.49 last 189.72) - open/close 59595 (time 1734260004.47 total 355.49 last 198.17) - open/close 60000 (time 1734260006.88 total 357.90 last 168.18) - open/close 61931 (time 1734260016.88 total 367.90 last 193.10) - open/close 63791 (time 1734260026.88 total 377.90 last 185.98) total: 64768 open/close in 382.98 seconds: 169.12 ops/second - /unlink 3170 (time 1734260043.14 total 10.00 last 316.96) - /unlink 6396 (time 1734260053.15 total 20.00 last 322.50) - /unlink 9418 (time 1734260063.15 total 30.01 last 302.09) - /unlink 12226 (time 1734260073.15 total 40.01 last 280.79) - /unlink 15039 (time 1734260083.15 total 50.01 last 281.28) - /unlink 18207 (time 1734260093.15 total 60.01 last 316.69) - /unlink 20000 (time 1734260099.16 total 66.02 last 298.47) - /unlink 23091 (time 1734260109.16 total 76.02 last 309.02) - /unlink 26178 (time 1734260119.17 total 86.03 last 308.56) - /unlink 29516 (time 1734260129.17 total 96.03 last 333.75) - /unlink 33020 (time 1734260139.17 total 106.03 last 350.39) - /unlink 36510 (time 1734260149.17 total 116.03 last 348.90) - /unlink 40000 (time 1734260157.99 total 124.85 last 395.61) - /unlink 43813 (time 1734260168.00 total 134.86 last 381.23) - /unlink 47668 (time 1734260178.00 total 144.86 last 385.38) - /unlink 50000 (time 1734260183.39 total 150.25 last 432.47) - /unlink 54494 (time 1734260193.39 total 160.25 last 449.33) - /unlink 58731 (time 1734260203.40 total 170.25 last 423.62) - /unlink 60000 (time 1734260205.81 total 172.67 last 524.97) total: 64768 /unlink in 182.45 seconds: 355.00 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2562 (time 1734260244.88 total 10.00 last 256.18) - open/close 5121 (time 1734260254.89 total 20.00 last 255.83) - open/close 7813 (time 1734260264.89 total 30.01 last 269.12) - open/close 10000 (time 1734260272.78 total 37.89 last 277.26) - open/close 12958 (time 1734260282.78 total 47.90 last 295.78) - open/close 15741 (time 1734260292.78 total 57.90 last 278.19) - open/close 18310 (time 1734260302.78 total 67.90 last 256.89) - open/close 20000 (time 1734260309.65 total 74.76 last 246.29) - open/close 22237 (time 1734260319.65 total 84.76 last 223.64) - open/close 24627 (time 1734260329.65 total 94.76 last 238.99) - open/close 26913 (time 1734260339.65 total 104.77 last 228.54) - open/close 29336 (time 1734260349.66 total 114.77 last 242.16) - open/close 30000 (time 1734260352.31 total 117.43 last 250.39) - open/close 33003 (time 1734260362.31 total 127.43 last 300.26) - open/close 35953 (time 1734260372.31 total 137.43 last 294.93) - open/close 38829 (time 1734260382.31 total 147.43 last 287.55) - open/close 40000 (time 1734260386.63 total 151.75 last 271.32) - open/close 42840 (time 1734260396.63 total 161.75 last 283.95) - open/close 45735 (time 1734260406.63 total 171.75 last 289.46) - open/close 48257 (time 1734260416.64 total 181.75 last 252.09) - open/close 50000 (time 1734260424.04 total 189.16 last 235.35) - open/close 52489 (time 1734260434.05 total 199.16 last 248.84) - open/close 54689 (time 1734260444.05 total 209.16 last 219.99) - open/close 56868 (time 1734260454.05 total 219.16 last 217.87) - open/close 59149 (time 1734260464.05 total 229.17 last 228.01) - open/close 60000 (time 1734260467.90 total 233.02 last 220.99) - open/close 62034 (time 1734260477.90 total 243.02 last 203.38) - open/close 64271 (time 1734260487.90 total 253.02 last 223.70) total: 64768 open/close in 255.58 seconds: 253.41 ops/second - /unlink 3657 (time 1734260501.58 total 10.00 last 365.60) - /unlink 7851 (time 1734260511.58 total 20.01 last 419.29) - /unlink 10000 (time 1734260516.28 total 24.70 last 457.55) - /unlink 14920 (time 1734260526.28 total 34.70 last 492.00) - /unlink 19879 (time 1734260536.28 total 44.70 last 495.81) - /unlink 24387 (time 1734260546.28 total 54.71 last 450.72) - /unlink 28909 (time 1734260556.28 total 64.71 last 452.13) - /unlink 30000 (time 1734260558.92 total 67.34 last 413.77) - /unlink 35237 (time 1734260568.92 total 77.35 last 523.62) - /unlink 40000 (time 1734260577.81 total 86.23 last 536.18) - /unlink 45546 (time 1734260587.81 total 96.23 last 554.55) - /unlink 50000 (time 1734260597.63 total 106.05 last 453.57) - /unlink 55315 (time 1734260607.63 total 116.05 last 531.45) - /unlink 60000 (time 1734260614.60 total 123.02 last 672.41) total: 64768 /unlink in 131.30 seconds: 493.30 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2662 (time 1734260650.64 total 10.00 last 266.19) - open/close 5236 (time 1734260660.64 total 20.00 last 257.38) - open/close 7840 (time 1734260670.64 total 30.00 last 260.32) - open/close 10000 (time 1734260679.02 total 38.38 last 258.01) - open/close 13000 (time 1734260689.02 total 48.38 last 299.98) - open/close 15969 (time 1734260699.02 total 58.38 last 296.75) - open/close 18957 (time 1734260709.02 total 68.38 last 298.78) - open/close 20000 (time 1734260712.42 total 71.78 last 306.58) - open/close 22999 (time 1734260722.43 total 81.79 last 299.86) - open/close 26444 (time 1734260732.43 total 91.79 last 344.48) - open/close 29687 (time 1734260742.43 total 101.79 last 324.25) - open/close 33239 (time 1734260752.43 total 111.79 last 355.15) - open/close 37456 (time 1734260762.43 total 121.79 last 421.66) - open/close 40000 (time 1734260768.23 total 127.59 last 438.50) - open/close 44343 (time 1734260778.23 total 137.59 last 434.23) - open/close 48631 (time 1734260788.24 total 147.60 last 428.72) - open/close 50000 (time 1734260791.43 total 150.79 last 428.59) - open/close 55160 (time 1734260801.43 total 160.79 last 516.00) - open/close 60000 (time 1734260809.87 total 169.23 last 573.45) total: 64768 open/close in 178.11 seconds: 363.65 ops/second - /unlink 8654 (time 1734260829.56 total 10.00 last 865.38) - /unlink 16989 (time 1734260839.56 total 20.00 last 833.43) - /unlink 20000 (time 1734260842.83 total 23.27 last 921.05) - /unlink 28056 (time 1734260852.83 total 33.27 last 805.59) - /unlink 30000 (time 1734260855.22 total 35.65 last 815.80) - /unlink 37299 (time 1734260865.22 total 45.65 last 729.83) - /unlink 40000 (time 1734260868.90 total 49.33 last 733.83) - /unlink 47081 (time 1734260878.90 total 59.34 last 708.09) - /unlink 50000 (time 1734260882.82 total 63.26 last 743.51) - /unlink 57787 (time 1734260892.82 total 73.26 last 778.65) - /unlink 60000 (time 1734260895.40 total 75.84 last 858.11) total: 64768 /unlink in 80.71 seconds: 802.49 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5369 (time 1734260930.74 total 10.00 last 536.90) - open/close 10000 (time 1734260940.08 total 19.33 last 496.11) - open/close 15228 (time 1734260950.08 total 29.34 last 522.76) - open/close 20000 (time 1734260958.98 total 38.24 last 535.82) - open/close 25511 (time 1734260968.99 total 48.24 last 551.00) - open/close 30000 (time 1734260976.96 total 56.22 last 563.11) - open/close 35361 (time 1734260986.96 total 66.22 last 536.06) - open/close 40000 (time 1734260995.97 total 75.23 last 514.51) - open/close 45071 (time 1734261005.97 total 85.23 last 507.08) - open/close 50000 (time 1734261015.85 total 95.11 last 498.99) - open/close 54852 (time 1734261025.85 total 105.11 last 485.19) - open/close 59545 (time 1734261035.85 total 115.11 last 469.25) - open/close 64473 (time 1734261045.86 total 125.11 last 492.72) total: 64768 open/close in 125.71 seconds: 515.21 ops/second - /unlink 8114 (time 1734261057.34 total 10.00 last 811.36) - /unlink 10000 (time 1734261059.65 total 12.32 last 814.44) - /unlink 17685 (time 1734261069.65 total 22.32 last 768.49) - /unlink 20000 (time 1734261072.50 total 25.16 last 813.11) - /unlink 27733 (time 1734261082.50 total 35.16 last 773.23) - /unlink 30000 (time 1734261084.81 total 37.47 last 984.00) - /unlink 38486 (time 1734261094.81 total 47.47 last 848.56) - /unlink 46764 (time 1734261104.81 total 57.47 last 827.75) - /unlink 50000 (time 1734261108.58 total 61.24 last 858.77) - /unlink 58300 (time 1734261118.58 total 71.24 last 829.94) - /unlink 60000 (time 1734261120.65 total 73.32 last 818.01) total: 64768 /unlink in 79.25 seconds: 817.31 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg131-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg131-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg131-server stop mds service on oleg131-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg131-server stop mds service on oleg131-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg131-server unloading modules via unload_modules_local on: 'oleg131-server' oleg131-server: oleg131-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg131-client: oleg131-client: ssh exited with exit code 2 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 2 pdsh@oleg131-client: oleg131-client: ssh exited with exit code 2 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 2