== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:33:14 (1734258794) Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2 oleg136-server: oleg136-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg136-server' oleg136-server: oleg136-server.virtnet: executing load_modules_local oleg136-server: Loading modules from /home/green/git/lustre-release/lustre oleg136-server: detected 4 online CPUs by sysfs oleg136-server: Force libcfs to create 2 CPU partitions oleg136-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory oleg136-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg136-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg136-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg136-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg136-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg136-client.virtnet: -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86890 331 86559 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 87086 312 86774 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 16266 458 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16451 643 15808 4% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1487 (time 1734258870.00 total 10.00 last 148.66) - open/close 2908 (time 1734258880.00 total 20.01 last 142.01) - open/close 4391 (time 1734258890.01 total 30.01 last 148.23) - open/close 5906 (time 1734258900.01 total 40.02 last 151.42) - open/close 7309 (time 1734258910.02 total 50.02 last 140.24) - open/close 8789 (time 1734258920.02 total 60.02 last 147.97) - open/close 10000 (time 1734258928.02 total 68.02 last 151.39) - open/close 11650 (time 1734258938.02 total 78.03 last 164.94) - open/close 13332 (time 1734258948.03 total 88.03 last 168.14) - open/close 14923 (time 1734258958.03 total 98.03 last 159.04) - open/close 16323 (time 1734258968.03 total 108.04 last 139.96) - open/close 17736 (time 1734258978.03 total 118.04 last 141.27) - open/close 19148 (time 1734258988.04 total 128.04 last 141.16) - open/close 20000 (time 1734258993.96 total 133.96 last 143.91) - open/close 21427 (time 1734259003.96 total 143.97 last 142.63) - open/close 22883 (time 1734259013.97 total 153.97 last 145.50) - open/close 24397 (time 1734259023.97 total 163.98 last 151.34) - open/close 25866 (time 1734259033.98 total 173.98 last 146.84) - open/close 27208 (time 1734259043.98 total 183.98 last 134.19) - open/close 28588 (time 1734259053.98 total 193.98 last 137.99) - open/close 30000 (time 1734259063.95 total 203.95 last 141.66) - open/close 31322 (time 1734259073.95 total 213.95 last 132.19) - open/close 32739 (time 1734259083.95 total 223.95 last 141.68) - open/close 34155 (time 1734259093.95 total 233.96 last 141.58) - open/close 35466 (time 1734259103.96 total 243.96 last 131.04) - open/close 36792 (time 1734259113.96 total 253.96 last 132.56) - open/close 38077 (time 1734259123.97 total 263.97 last 128.37) - open/close 39314 (time 1734259133.97 total 273.98 last 123.64) - open/close 40000 (time 1734259139.20 total 279.21 last 131.24) - open/close 41261 (time 1734259149.20 total 289.21 last 126.06) - open/close 42428 (time 1734259159.21 total 299.22 last 116.59) - open/close 43687 (time 1734259169.22 total 309.22 last 125.82) - open/close 44932 (time 1734259179.22 total 319.23 last 124.44) - open/close 46279 (time 1734259189.23 total 329.23 last 134.63) - open/close 47534 (time 1734259199.23 total 339.24 last 125.44) - open/close 48823 (time 1734259209.23 total 349.24 last 128.90) - open/close 50000 (time 1734259218.74 total 358.74 last 123.84) - open/close 51248 (time 1734259228.75 total 368.75 last 124.69) - open/close 52498 (time 1734259238.75 total 378.76 last 124.96) - open/close 53812 (time 1734259248.75 total 388.76 last 131.40) - open/close 55197 (time 1734259258.76 total 398.76 last 138.45) - open/close 56539 (time 1734259268.76 total 408.76 last 134.15) - open/close 58023 (time 1734259278.76 total 418.76 last 148.39) - open/close 59617 (time 1734259288.76 total 428.77 last 159.34) - open/close 60000 (time 1734259291.59 total 431.60 last 135.30) - open/close 61404 (time 1734259301.59 total 441.60 last 140.40) - open/close 62816 (time 1734259311.60 total 451.61 last 141.07) - open/close 64138 (time 1734259321.61 total 461.62 last 132.09) total: 64768 open/close in 466.36 seconds: 138.88 ops/second - /unlink 2082 (time 1734259338.11 total 10.00 last 208.15) - /unlink 4341 (time 1734259348.11 total 20.00 last 225.89) - /unlink 6735 (time 1734259358.11 total 30.01 last 239.35) - /unlink 8919 (time 1734259368.11 total 40.01 last 218.38) - /unlink 10000 (time 1734259373.05 total 44.95 last 218.80) - /unlink 12556 (time 1734259383.05 total 54.95 last 255.59) - /unlink 15204 (time 1734259393.05 total 64.95 last 264.78) - /unlink 17803 (time 1734259403.05 total 74.95 last 259.87) - /unlink 20000 (time 1734259411.50 total 83.39 last 260.20) - /unlink 22795 (time 1734259421.50 total 93.39 last 279.49) - /unlink 25535 (time 1734259431.50 total 103.40 last 273.94) - /unlink 28162 (time 1734259441.50 total 113.40 last 262.61) - /unlink 30000 (time 1734259449.23 total 121.12 last 237.95) - /unlink 32708 (time 1734259459.23 total 131.13 last 270.74) - /unlink 35732 (time 1734259469.23 total 141.13 last 302.34) - /unlink 38840 (time 1734259479.23 total 151.13 last 310.79) - /unlink 40000 (time 1734259483.11 total 155.00 last 299.43) - /unlink 43181 (time 1734259493.11 total 165.00 last 318.02) - /unlink 46211 (time 1734259503.11 total 175.01 last 302.94) - /unlink 49433 (time 1734259513.11 total 185.01 last 322.11) - /unlink 52545 (time 1734259523.12 total 195.01 last 311.14) - /unlink 55593 (time 1734259533.12 total 205.02 last 304.66) - /unlink 58258 (time 1734259543.12 total 215.02 last 266.43) - /unlink 60000 (time 1734259550.54 total 222.44 last 234.86) - /unlink 62411 (time 1734259560.54 total 232.44 last 241.10) total: 64768 /unlink in 241.69 seconds: 267.98 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1534 (time 1734259599.34 total 10.00 last 153.33) - open/close 3142 (time 1734259609.34 total 20.01 last 160.71) - open/close 4857 (time 1734259619.35 total 30.01 last 171.45) - open/close 6296 (time 1734259629.35 total 40.01 last 143.88) - open/close 7882 (time 1734259639.35 total 50.02 last 158.52) - open/close 9425 (time 1734259649.36 total 60.02 last 154.24) - open/close 10000 (time 1734259652.86 total 63.53 last 164.07) - open/close 11524 (time 1734259662.86 total 73.53 last 152.34) - open/close 13082 (time 1734259672.86 total 83.53 last 155.78) - open/close 14596 (time 1734259682.87 total 93.54 last 151.34) - open/close 16246 (time 1734259692.87 total 103.54 last 164.99) - open/close 17765 (time 1734259702.87 total 113.54 last 151.86) - open/close 19303 (time 1734259712.88 total 123.55 last 153.72) - open/close 20000 (time 1734259718.13 total 128.80 last 132.73) - open/close 21506 (time 1734259728.13 total 138.80 last 150.56) - open/close 22997 (time 1734259738.14 total 148.81 last 148.98) - open/close 24675 (time 1734259748.14 total 158.81 last 167.74) - open/close 26275 (time 1734259758.15 total 168.82 last 159.91) - open/close 27827 (time 1734259768.16 total 178.82 last 155.07) - open/close 29600 (time 1734259778.16 total 188.83 last 177.28) - open/close 30000 (time 1734259780.21 total 190.88 last 194.70) - open/close 31851 (time 1734259790.22 total 200.88 last 185.02) - open/close 33627 (time 1734259800.22 total 210.89 last 177.53) - open/close 35373 (time 1734259810.22 total 220.89 last 174.55) - open/close 37191 (time 1734259820.23 total 230.89 last 181.78) - open/close 38841 (time 1734259830.23 total 240.89 last 165.00) - open/close 40000 (time 1734259838.13 total 248.80 last 146.58) - open/close 41468 (time 1734259848.14 total 258.80 last 146.73) - open/close 43086 (time 1734259858.14 total 268.81 last 161.78) - open/close 44529 (time 1734259868.14 total 278.81 last 144.24) - open/close 45972 (time 1734259878.15 total 288.81 last 144.24) - open/close 47576 (time 1734259888.15 total 298.82 last 160.38) - open/close 49192 (time 1734259898.15 total 308.82 last 161.51) - open/close 50000 (time 1734259903.22 total 313.88 last 159.62) - open/close 51772 (time 1734259913.22 total 323.89 last 177.12) - open/close 53628 (time 1734259923.22 total 333.89 last 185.57) - open/close 55409 (time 1734259933.22 total 343.89 last 178.05) - open/close 57243 (time 1734259943.23 total 353.89 last 183.35) - open/close 59102 (time 1734259953.23 total 363.90 last 185.86) - open/close 60000 (time 1734259957.85 total 368.52 last 194.26) - open/close 61849 (time 1734259967.85 total 378.52 last 184.84) - open/close 63673 (time 1734259977.86 total 388.53 last 182.33) total: 64768 open/close in 394.50 seconds: 164.18 ops/second - /unlink 3166 (time 1734259995.05 total 10.00 last 316.58) - /unlink 6456 (time 1734260005.05 total 20.00 last 328.95) - /unlink 9684 (time 1734260015.05 total 30.00 last 322.77) - /unlink 12889 (time 1734260025.05 total 40.01 last 320.43) - /unlink 16196 (time 1734260035.05 total 50.01 last 330.62) - /unlink 19352 (time 1734260045.06 total 60.01 last 315.53) - /unlink 22607 (time 1734260055.06 total 70.01 last 325.48) - /unlink 25572 (time 1734260065.06 total 80.01 last 296.46) - /unlink 28235 (time 1734260075.06 total 90.02 last 266.17) - /unlink 30000 (time 1734260081.42 total 96.37 last 277.60) - /unlink 33020 (time 1734260091.42 total 106.38 last 301.90) - /unlink 35989 (time 1734260101.43 total 116.38 last 296.81) - /unlink 39022 (time 1734260111.43 total 126.38 last 303.29) - /unlink 40000 (time 1734260114.48 total 129.43 last 320.88) - /unlink 43006 (time 1734260124.48 total 139.43 last 300.54) - /unlink 46249 (time 1734260134.48 total 149.43 last 324.27) - /unlink 49639 (time 1734260144.48 total 159.44 last 338.89) - /unlink 53036 (time 1734260154.48 total 169.44 last 339.67) - /unlink 56838 (time 1734260164.48 total 179.44 last 380.14) - /unlink 60000 (time 1734260173.37 total 188.33 last 355.72) - /unlink 64134 (time 1734260183.37 total 198.33 last 413.36) total: 64768 /unlink in 199.82 seconds: 324.13 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2713 (time 1734260214.58 total 10.00 last 271.22) - open/close 5371 (time 1734260224.58 total 20.00 last 265.74) - open/close 8227 (time 1734260234.58 total 30.01 last 285.50) - open/close 10000 (time 1734260241.85 total 37.28 last 243.77) - open/close 12396 (time 1734260251.86 total 47.29 last 239.53) - open/close 14871 (time 1734260261.86 total 57.29 last 247.48) - open/close 17389 (time 1734260271.86 total 67.29 last 251.79) - open/close 20000 (time 1734260281.38 total 76.81 last 274.23) - open/close 22568 (time 1734260291.38 total 86.81 last 256.73) - open/close 25110 (time 1734260301.39 total 96.82 last 254.06) - open/close 27328 (time 1734260311.39 total 106.82 last 221.73) - open/close 29376 (time 1734260321.39 total 116.82 last 204.78) - open/close 30000 (time 1734260324.73 total 120.16 last 186.88) - open/close 32273 (time 1734260334.73 total 130.16 last 227.29) - open/close 34274 (time 1734260344.73 total 140.16 last 200.04) - open/close 36594 (time 1734260354.73 total 150.16 last 231.99) - open/close 39423 (time 1734260364.74 total 160.17 last 282.81) - open/close 40000 (time 1734260366.96 total 162.39 last 259.52) - open/close 42454 (time 1734260376.96 total 172.39 last 245.33) - open/close 44884 (time 1734260386.97 total 182.40 last 242.93) - open/close 47503 (time 1734260396.97 total 192.40 last 261.84) - open/close 50000 (time 1734260405.68 total 201.10 last 286.80) - open/close 52531 (time 1734260415.68 total 211.10 last 253.07) - open/close 54814 (time 1734260425.68 total 221.11 last 228.27) - open/close 57174 (time 1734260435.68 total 231.11 last 235.96) - open/close 59310 (time 1734260445.68 total 241.11 last 213.54) - open/close 60000 (time 1734260449.36 total 244.79 last 187.60) - open/close 62182 (time 1734260459.36 total 254.79 last 218.14) - open/close 64361 (time 1734260469.36 total 264.79 last 217.89) total: 64768 open/close in 266.96 seconds: 242.61 ops/second - /unlink 3482 (time 1734260482.73 total 10.00 last 348.12) - /unlink 6951 (time 1734260492.73 total 20.00 last 346.83) - /unlink 10000 (time 1734260501.46 total 28.73 last 349.27) - /unlink 13957 (time 1734260511.46 total 38.73 last 395.68) - /unlink 18357 (time 1734260521.47 total 48.74 last 439.95) - /unlink 20000 (time 1734260524.96 total 52.23 last 469.99) - /unlink 24502 (time 1734260534.96 total 62.23 last 450.08) - /unlink 28638 (time 1734260544.96 total 72.24 last 413.55) - /unlink 30000 (time 1734260548.18 total 75.45 last 423.70) - /unlink 34067 (time 1734260558.18 total 85.45 last 406.66) - /unlink 38963 (time 1734260568.18 total 95.45 last 489.55) - /unlink 40000 (time 1734260570.23 total 97.50 last 506.26) - /unlink 45199 (time 1734260580.23 total 107.50 last 519.80) - /unlink 50000 (time 1734260589.99 total 117.26 last 491.82) - /unlink 54290 (time 1734260599.99 total 127.27 last 428.94) - /unlink 59728 (time 1734260610.00 total 137.27 last 543.75) total: 64768 /unlink in 145.81 seconds: 444.21 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2511 (time 1734260650.32 total 10.00 last 251.01) - open/close 4988 (time 1734260660.32 total 20.01 last 247.59) - open/close 7508 (time 1734260670.32 total 30.01 last 252.00) - open/close 10000 (time 1734260679.99 total 39.67 last 257.87) - open/close 12863 (time 1734260689.99 total 49.67 last 286.27) - open/close 15539 (time 1734260699.99 total 59.68 last 267.54) - open/close 18214 (time 1734260709.99 total 69.68 last 267.46) - open/close 20000 (time 1734260716.59 total 76.28 last 270.62) - open/close 22701 (time 1734260726.59 total 86.28 last 270.08) - open/close 25484 (time 1734260736.59 total 96.28 last 278.26) - open/close 28383 (time 1734260746.60 total 106.28 last 289.86) - open/close 30000 (time 1734260751.98 total 111.67 last 300.18) - open/close 33595 (time 1734260761.99 total 121.67 last 359.38) - open/close 37558 (time 1734260771.99 total 131.67 last 396.24) - open/close 40000 (time 1734260778.90 total 138.59 last 352.99) - open/close 44018 (time 1734260788.91 total 148.59 last 401.76) - open/close 48698 (time 1734260798.91 total 158.59 last 467.98) - open/close 50000 (time 1734260801.34 total 161.02 last 535.80) - open/close 55646 (time 1734260811.34 total 171.02 last 564.59) - open/close 60000 (time 1734260819.97 total 179.65 last 504.47) - open/close 64751 (time 1734260829.97 total 189.65 last 475.07) total: 64768 open/close in 189.69 seconds: 341.44 ops/second - /unlink 8479 (time 1734260840.91 total 10.00 last 847.90) - /unlink 16371 (time 1734260850.91 total 20.00 last 789.13) - /unlink 20000 (time 1734260855.37 total 24.45 last 815.08) - /unlink 27098 (time 1734260865.37 total 34.45 last 709.74) - /unlink 30000 (time 1734260869.45 total 38.54 last 710.80) - /unlink 36903 (time 1734260879.45 total 48.54 last 690.26) - /unlink 40000 (time 1734260883.93 total 53.02 last 691.40) - /unlink 47869 (time 1734260893.93 total 63.02 last 786.84) - /unlink 50000 (time 1734260896.41 total 65.50 last 858.78) - /unlink 59562 (time 1734260906.41 total 75.50 last 956.12) total: 64768 /unlink in 81.09 seconds: 798.74 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5090 (time 1734260945.44 total 10.00 last 508.98) - open/close 10000 (time 1734260955.09 total 19.65 last 508.94) - open/close 15588 (time 1734260965.09 total 29.65 last 558.72) - open/close 20000 (time 1734260973.23 total 37.78 last 542.42) - open/close 25596 (time 1734260983.23 total 47.78 last 559.60) - open/close 30000 (time 1734260991.98 total 56.54 last 503.02) - open/close 35132 (time 1734261001.98 total 66.54 last 513.12) - open/close 40000 (time 1734261011.75 total 76.30 last 498.52) - open/close 45159 (time 1734261021.75 total 86.31 last 515.82) - open/close 49893 (time 1734261031.75 total 96.31 last 473.33) - open/close 54495 (time 1734261041.75 total 106.31 last 460.15) - open/close 59459 (time 1734261051.75 total 116.31 last 496.40) - open/close 64057 (time 1734261061.75 total 126.31 last 459.76) total: 64768 open/close in 127.88 seconds: 506.47 ops/second - /unlink 7868 (time 1734261074.25 total 10.00 last 786.76) - /unlink 10000 (time 1734261076.91 total 12.67 last 799.40) - /unlink 18396 (time 1734261086.91 total 22.67 last 839.57) - /unlink 26851 (time 1734261096.92 total 32.67 last 845.41) - /unlink 30000 (time 1734261100.67 total 36.43 last 838.00) - /unlink 38317 (time 1734261110.67 total 46.43 last 831.64) - /unlink 40000 (time 1734261112.68 total 48.43 last 839.83) - /unlink 47991 (time 1734261122.68 total 58.43 last 799.04) - /unlink 50000 (time 1734261125.40 total 61.15 last 737.92) - /unlink 58190 (time 1734261135.40 total 71.16 last 818.91) - /unlink 60000 (time 1734261137.45 total 73.21 last 882.64) total: 64768 /unlink in 78.64 seconds: 823.58 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg136-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg136-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server stop mds service on oleg136-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server stop mds service on oleg136-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server unloading modules via unload_modules_local on: 'oleg136-server' oleg136-server: oleg136-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg136-client: oleg136-client: ssh exited with exit code 2 pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2 pdsh@oleg136-client: oleg136-client: ssh exited with exit code 2 pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2