== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:34:12 (1734258852) Stopping clients: oleg340-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg340-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg340-client: oleg340-server: ssh exited with exit code 2 oleg340-server: oleg340-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions oleg340-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg340-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg340-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg340-server: oleg340-server.virtnet: executing set_default_debug -1 all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg340-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg340-server: oleg340-server.virtnet: executing set_default_debug -1 all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg340-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg340-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg340-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg340-server: oleg340-server.virtnet: executing set_default_debug -1 all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg340-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg340-client.virtnet: -o user_xattr,flock 192.168.203.140@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86890 331 86559 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 87086 312 86774 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 16266 458 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16451 643 15808 4% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1567 (time 1734258928.33 total 10.01 last 156.61) - open/close 3217 (time 1734258938.33 total 20.01 last 164.92) - open/close 4903 (time 1734258948.33 total 30.01 last 168.60) - open/close 6537 (time 1734258958.33 total 40.01 last 163.39) - open/close 8231 (time 1734258968.34 total 50.02 last 169.33) - open/close 9916 (time 1734258978.34 total 60.02 last 168.45) - open/close 11646 (time 1734258988.34 total 70.02 last 172.99) - open/close 13594 (time 1734258998.34 total 80.02 last 194.79) - open/close 15567 (time 1734259008.35 total 90.02 last 197.24) - open/close 17485 (time 1734259018.35 total 100.03 last 191.75) - open/close 19232 (time 1734259028.35 total 110.03 last 174.62) - open/close 20000 (time 1734259033.15 total 114.83 last 160.07) - open/close 21654 (time 1734259043.15 total 124.83 last 165.35) - open/close 23278 (time 1734259053.15 total 134.83 last 162.38) - open/close 24979 (time 1734259063.16 total 144.84 last 170.03) - open/close 26538 (time 1734259073.16 total 154.84 last 155.85) - open/close 28060 (time 1734259083.17 total 164.85 last 152.07) - open/close 29638 (time 1734259093.17 total 174.85 last 157.73) - open/close 30000 (time 1734259095.60 total 177.28 last 148.97) - open/close 31544 (time 1734259105.61 total 187.29 last 154.32) - open/close 33215 (time 1734259115.61 total 197.29 last 167.05) - open/close 35137 (time 1734259125.62 total 207.29 last 192.12) - open/close 36779 (time 1734259135.62 total 217.29 last 164.20) - open/close 38364 (time 1734259145.62 total 227.30 last 158.47) - open/close 40000 (time 1734259155.44 total 237.11 last 166.63) - open/close 41744 (time 1734259165.44 total 247.12 last 174.40) - open/close 43597 (time 1734259175.44 total 257.12 last 185.27) - open/close 45340 (time 1734259185.44 total 267.12 last 174.29) - open/close 46951 (time 1734259195.44 total 277.12 last 161.05) - open/close 48704 (time 1734259205.44 total 287.12 last 175.29) - open/close 50000 (time 1734259213.54 total 295.21 last 160.18) - open/close 51666 (time 1734259223.54 total 305.22 last 166.53) - open/close 53427 (time 1734259233.54 total 315.22 last 176.07) - open/close 55225 (time 1734259243.54 total 325.22 last 179.78) - open/close 57479 (time 1734259253.54 total 335.22 last 225.36) - open/close 59623 (time 1734259263.54 total 345.22 last 214.39) - open/close 60000 (time 1734259265.59 total 347.27 last 184.17) - open/close 61715 (time 1734259275.60 total 357.27 last 171.41) - open/close 63639 (time 1734259285.60 total 367.28 last 192.37) total: 64768 open/close in 372.92 seconds: 173.68 ops/second - /unlink 2811 (time 1734259302.53 total 10.00 last 281.05) - /unlink 5827 (time 1734259312.53 total 20.00 last 301.52) - /unlink 8665 (time 1734259322.53 total 30.01 last 283.72) - /unlink 10000 (time 1734259326.25 total 33.72 last 359.15) - /unlink 13465 (time 1734259336.25 total 43.72 last 346.47) - /unlink 16808 (time 1734259346.25 total 53.73 last 334.24) - /unlink 19614 (time 1734259356.25 total 63.73 last 280.57) - /unlink 22524 (time 1734259366.26 total 73.73 last 290.82) - /unlink 25528 (time 1734259376.26 total 83.74 last 300.32) - /unlink 28411 (time 1734259386.26 total 93.74 last 288.25) - /unlink 30000 (time 1734259391.50 total 98.97 last 303.63) - /unlink 33136 (time 1734259401.50 total 108.97 last 313.54) - /unlink 35972 (time 1734259411.50 total 118.97 last 283.58) - /unlink 38901 (time 1734259421.50 total 128.98 last 292.87) - /unlink 40000 (time 1734259425.30 total 132.77 last 289.40) - /unlink 42943 (time 1734259435.30 total 142.78 last 294.22) - /unlink 45911 (time 1734259445.30 total 152.78 last 296.71) - /unlink 48935 (time 1734259455.30 total 162.78 last 302.36) - /unlink 50000 (time 1734259459.04 total 166.51 last 285.43) - /unlink 53107 (time 1734259469.04 total 176.51 last 310.68) - /unlink 56111 (time 1734259479.04 total 186.51 last 300.38) - /unlink 59656 (time 1734259489.04 total 196.51 last 354.46) - /unlink 63005 (time 1734259499.04 total 206.52 last 334.85) total: 64768 /unlink in 212.76 seconds: 304.42 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2426 (time 1734259533.13 total 10.00 last 242.51) - open/close 4502 (time 1734259543.13 total 20.01 last 207.51) - open/close 6370 (time 1734259553.13 total 30.01 last 186.79) - open/close 8317 (time 1734259563.13 total 40.01 last 194.65) - open/close 10000 (time 1734259572.27 total 49.15 last 184.24) - open/close 11898 (time 1734259582.27 total 59.15 last 189.80) - open/close 14009 (time 1734259592.27 total 69.15 last 211.02) - open/close 16196 (time 1734259602.28 total 79.16 last 218.53) - open/close 18603 (time 1734259612.28 total 89.16 last 240.62) - open/close 20000 (time 1734259617.84 total 94.71 last 251.61) - open/close 22371 (time 1734259627.84 total 104.71 last 237.08) - open/close 24621 (time 1734259637.84 total 114.72 last 224.95) - open/close 26872 (time 1734259647.84 total 124.72 last 225.06) - open/close 29084 (time 1734259657.84 total 134.72 last 221.13) - open/close 30000 (time 1734259662.16 total 139.03 last 212.39) - open/close 32178 (time 1734259672.16 total 149.03 last 217.80) - open/close 34375 (time 1734259682.16 total 159.04 last 219.68) - open/close 36604 (time 1734259692.16 total 169.04 last 222.84) - open/close 39064 (time 1734259702.16 total 179.04 last 245.92) - open/close 40000 (time 1734259705.70 total 182.57 last 264.97) - open/close 42405 (time 1734259715.70 total 192.57 last 240.50) - open/close 44602 (time 1734259725.70 total 202.58 last 219.65) - open/close 46598 (time 1734259735.70 total 212.58 last 199.54) - open/close 48742 (time 1734259745.71 total 222.58 last 214.29) - open/close 50000 (time 1734259751.86 total 228.74 last 204.37) - open/close 52039 (time 1734259761.87 total 238.74 last 203.84) - open/close 54202 (time 1734259771.87 total 248.75 last 216.24) - open/close 56450 (time 1734259781.87 total 258.75 last 224.78) - open/close 58759 (time 1734259791.89 total 268.77 last 230.45) - open/close 60000 (time 1734259797.42 total 274.30 last 224.19) - open/close 61913 (time 1734259807.42 total 284.30 last 191.30) - open/close 63900 (time 1734259817.43 total 294.30 last 198.67) total: 64768 open/close in 298.47 seconds: 217.00 ops/second - /unlink 4243 (time 1734259832.73 total 10.00 last 424.20) - /unlink 8479 (time 1734259842.73 total 20.00 last 423.58) - /unlink 10000 (time 1734259846.35 total 23.63 last 419.41) - /unlink 14277 (time 1734259856.35 total 33.63 last 427.69) - /unlink 18430 (time 1734259866.35 total 43.63 last 415.28) - /unlink 20000 (time 1734259870.73 total 48.01 last 358.50) - /unlink 23726 (time 1734259880.74 total 58.01 last 372.56) - /unlink 27173 (time 1734259890.74 total 68.01 last 344.69) - /unlink 30000 (time 1734259899.15 total 76.43 last 335.93) - /unlink 33484 (time 1734259909.15 total 86.43 last 348.28) - /unlink 37363 (time 1734259919.16 total 96.43 last 387.83) - /unlink 40000 (time 1734259925.49 total 102.77 last 416.13) - /unlink 43855 (time 1734259935.49 total 112.77 last 385.49) - /unlink 47604 (time 1734259945.50 total 122.77 last 374.82) - /unlink 50000 (time 1734259952.21 total 129.49 last 356.80) - /unlink 53719 (time 1734259962.21 total 139.49 last 371.83) - /unlink 57514 (time 1734259972.21 total 149.49 last 379.43) - /unlink 60000 (time 1734259978.68 total 155.95 last 384.76) - /unlink 64074 (time 1734259988.68 total 165.95 last 407.36) total: 64768 /unlink in 167.74 seconds: 386.12 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2742 (time 1734260018.78 total 10.00 last 274.16) - open/close 5385 (time 1734260028.78 total 20.00 last 264.28) - open/close 8193 (time 1734260038.79 total 30.00 last 280.76) - open/close 10000 (time 1734260046.14 total 37.36 last 245.69) - open/close 12895 (time 1734260056.14 total 47.36 last 289.43) - open/close 16409 (time 1734260066.14 total 57.36 last 351.38) - open/close 19334 (time 1734260076.15 total 67.36 last 292.42) - open/close 20000 (time 1734260078.36 total 69.58 last 300.66) - open/close 22857 (time 1734260088.36 total 79.58 last 285.65) - open/close 25707 (time 1734260098.37 total 89.58 last 284.95) - open/close 28562 (time 1734260108.37 total 99.59 last 285.44) - open/close 30000 (time 1734260113.30 total 104.51 last 291.72) - open/close 32707 (time 1734260123.30 total 114.52 last 270.63) - open/close 35368 (time 1734260133.30 total 124.52 last 266.06) - open/close 37961 (time 1734260143.30 total 134.52 last 259.26) - open/close 40000 (time 1734260150.28 total 141.50 last 292.05) - open/close 42940 (time 1734260160.29 total 151.51 last 293.90) - open/close 45800 (time 1734260170.29 total 161.51 last 285.94) - open/close 48605 (time 1734260180.29 total 171.51 last 280.41) - open/close 50000 (time 1734260185.77 total 176.98 last 254.92) - open/close 52840 (time 1734260195.77 total 186.98 last 283.98) - open/close 55342 (time 1734260205.77 total 196.99 last 250.13) - open/close 57942 (time 1734260215.77 total 206.99 last 260.00) - open/close 60000 (time 1734260223.87 total 215.09 last 254.10) - open/close 62447 (time 1734260233.87 total 225.09 last 244.70) total: 64768 open/close in 234.14 seconds: 276.62 ops/second - /unlink 4751 (time 1734260253.97 total 10.00 last 475.10) - /unlink 10000 (time 1734260263.67 total 19.69 last 541.56) - /unlink 14750 (time 1734260273.67 total 29.69 last 474.97) - /unlink 18952 (time 1734260283.67 total 39.69 last 420.14) - /unlink 20000 (time 1734260286.10 total 42.12 last 431.72) - /unlink 24474 (time 1734260296.10 total 52.12 last 447.38) - /unlink 28341 (time 1734260306.10 total 62.12 last 386.64) - /unlink 30000 (time 1734260309.82 total 65.85 last 445.67) - /unlink 34272 (time 1734260319.82 total 75.85 last 427.20) - /unlink 38943 (time 1734260329.82 total 85.85 last 467.07) - /unlink 40000 (time 1734260332.04 total 88.07 last 476.32) - /unlink 44608 (time 1734260342.04 total 98.07 last 460.79) - /unlink 49366 (time 1734260352.04 total 108.07 last 475.71) - /unlink 53885 (time 1734260362.04 total 118.07 last 451.83) - /unlink 58385 (time 1734260372.05 total 128.07 last 449.92) - /unlink 60000 (time 1734260375.89 total 131.91 last 420.52) - /unlink 64289 (time 1734260385.89 total 141.91 last 428.89) total: 64768 /unlink in 143.01 seconds: 452.89 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2275 (time 1734260412.62 total 10.00 last 227.45) - open/close 4794 (time 1734260422.62 total 20.00 last 251.84) - open/close 7252 (time 1734260432.62 total 30.01 last 245.71) - open/close 9584 (time 1734260442.63 total 40.01 last 233.16) - open/close 11879 (time 1734260452.63 total 50.01 last 229.48) - open/close 14284 (time 1734260462.63 total 60.02 last 240.41) - open/close 16936 (time 1734260472.63 total 70.02 last 265.16) - open/close 19897 (time 1734260482.63 total 80.02 last 296.03) - open/close 22737 (time 1734260492.64 total 90.02 last 283.96) - open/close 25644 (time 1734260502.64 total 100.02 last 290.70) - open/close 28517 (time 1734260512.64 total 110.02 last 287.17) - open/close 30000 (time 1734260517.55 total 114.93 last 302.17) - open/close 32999 (time 1734260527.55 total 124.93 last 299.85) - open/close 36378 (time 1734260537.55 total 134.94 last 337.80) - open/close 38893 (time 1734260547.55 total 144.94 last 251.49) - open/close 40000 (time 1734260552.21 total 149.59 last 237.84) - open/close 42626 (time 1734260562.21 total 159.60 last 262.47) - open/close 45443 (time 1734260572.21 total 169.60 last 281.67) - open/close 48280 (time 1734260582.22 total 179.60 last 283.64) - open/close 50000 (time 1734260587.96 total 185.34 last 299.55) - open/close 53002 (time 1734260597.96 total 195.34 last 300.16) - open/close 55743 (time 1734260608.01 total 205.39 last 272.79) - open/close 58542 (time 1734260618.01 total 215.39 last 279.83) - open/close 60000 (time 1734260622.86 total 220.24 last 300.68) - open/close 62817 (time 1734260632.86 total 230.25 last 281.64) total: 64768 open/close in 237.68 seconds: 272.50 ops/second - /unlink 5075 (time 1734260651.32 total 10.00 last 507.41) - /unlink 10000 (time 1734260660.60 total 19.29 last 530.44) - /unlink 15326 (time 1734260670.60 total 29.29 last 532.53) - /unlink 20000 (time 1734260680.29 total 38.98 last 482.48) - /unlink 25145 (time 1734260690.29 total 48.98 last 514.49) - /unlink 30000 (time 1734260699.35 total 58.03 last 536.09) - /unlink 35250 (time 1734260709.35 total 68.03 last 524.90) - /unlink 40000 (time 1734260716.89 total 75.58 last 629.76) - /unlink 46378 (time 1734260726.90 total 85.58 last 637.63) - /unlink 50000 (time 1734260732.40 total 91.08 last 658.30) - /unlink 56638 (time 1734260742.40 total 101.08 last 663.80) - /unlink 60000 (time 1734260746.85 total 105.54 last 754.79) total: 64768 /unlink in 112.43 seconds: 576.09 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5264 (time 1734260781.77 total 10.00 last 526.31) - open/close 10000 (time 1734260791.28 total 19.51 last 497.97) - open/close 16060 (time 1734260801.28 total 29.51 last 606.00) - open/close 20000 (time 1734260808.56 total 36.79 last 541.10) - open/close 24575 (time 1734260818.63 total 46.87 last 454.24) - open/close 29494 (time 1734260828.63 total 56.87 last 491.86) - open/close 34486 (time 1734260838.63 total 66.87 last 499.18) - open/close 39190 (time 1734260848.63 total 76.87 last 470.38) - open/close 43665 (time 1734260858.63 total 86.87 last 447.49) - open/close 48780 (time 1734260868.63 total 96.87 last 511.49) - open/close 50000 (time 1734260870.72 total 98.96 last 582.94) - open/close 55755 (time 1734260880.72 total 108.96 last 575.50) - open/close 60000 (time 1734260889.07 total 117.31 last 508.67) - open/close 64760 (time 1734260899.07 total 127.31 last 475.92) total: 64768 open/close in 127.32 seconds: 508.70 ops/second - /unlink 8249 (time 1734260910.02 total 10.00 last 824.86) - /unlink 10000 (time 1734260912.13 total 12.12 last 827.44) - /unlink 17746 (time 1734260922.13 total 22.12 last 774.56) - /unlink 20000 (time 1734260924.63 total 24.62 last 902.10) - /unlink 28999 (time 1734260934.63 total 34.62 last 899.85) - /unlink 38629 (time 1734260944.63 total 44.62 last 962.99) - /unlink 48143 (time 1734260954.63 total 54.62 last 951.32) - /unlink 50000 (time 1734260956.90 total 56.89 last 817.26) - /unlink 57940 (time 1734260966.90 total 66.89 last 793.97) - /unlink 60000 (time 1734260969.30 total 69.28 last 861.33) total: 64768 /unlink in 74.85 seconds: 865.29 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg340-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg340-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg340-server stop mds service on oleg340-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg340-server stop mds service on oleg340-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg340-server unloading modules via unload_modules_local on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg340-client: oleg340-client: ssh exited with exit code 2 pdsh@oleg340-client: oleg340-server: ssh exited with exit code 2 pdsh@oleg340-client: oleg340-client: ssh exited with exit code 2 pdsh@oleg340-client: oleg340-server: ssh exited with exit code 2