== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 16:33:18 (1741897998) Stopping clients: oleg320-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg320-client.virtnet /mnt/lustre2 (opts:-f) oleg320-server: oleg320-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg320-server' oleg320-server: oleg320-server.virtnet: executing load_modules_local oleg320-server: Loading modules from /home/green/git/lustre-release/lustre oleg320-server: detected 4 online CPUs by sysfs oleg320-server: MODOPTS_LIBCFS= oleg320-server: Force libcfs to create 2 CPU partitions oleg320-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg320-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg320-server: oleg320-server.virtnet: executing set_default_debug -1 all pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg320-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg320-server: oleg320-server.virtnet: executing set_default_debug -1 all pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg320-client.virtnet: -o user_xattr,flock 192.168.203.120@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1341 (time 1741898055.68 total 10.00 last 134.03) - open/close 2583 (time 1741898065.68 total 20.01 last 124.18) - open/close 3772 (time 1741898075.68 total 30.01 last 118.88) - open/close 5024 (time 1741898085.68 total 40.01 last 125.19) - open/close 6120 (time 1741898095.69 total 50.02 last 109.50) - open/close 7158 (time 1741898105.69 total 60.02 last 103.80) - open/close 8175 (time 1741898115.69 total 70.02 last 101.67) - open/close 9500 (time 1741898125.70 total 80.03 last 132.37) - open/close 10000 (time 1741898129.36 total 83.69 last 136.73) - open/close 11459 (time 1741898139.36 total 93.69 last 145.83) - open/close 12907 (time 1741898149.37 total 103.70 last 144.74) - open/close 14308 (time 1741898159.37 total 113.70 last 140.07) - open/close 15690 (time 1741898169.37 total 123.70 last 138.15) - open/close 17299 (time 1741898179.38 total 133.71 last 160.84) - open/close 18673 (time 1741898189.38 total 143.71 last 137.32) - open/close 20000 (time 1741898198.20 total 152.53 last 150.51) - open/close 21556 (time 1741898208.20 total 162.53 last 155.50) - open/close 23146 (time 1741898218.21 total 172.54 last 158.94) - open/close 24940 (time 1741898228.21 total 182.54 last 179.35) - open/close 26650 (time 1741898238.21 total 192.54 last 170.99) - open/close 28221 (time 1741898248.21 total 202.54 last 157.09) - open/close 29674 (time 1741898258.22 total 212.55 last 145.25) - open/close 30000 (time 1741898260.29 total 214.62 last 157.44) - open/close 31436 (time 1741898270.29 total 224.62 last 143.59) - open/close 32790 (time 1741898280.29 total 234.62 last 135.38) - open/close 34057 (time 1741898290.29 total 244.62 last 126.67) - open/close 35442 (time 1741898300.30 total 254.63 last 138.37) - open/close 37095 (time 1741898310.30 total 264.63 last 165.24) - open/close 38796 (time 1741898320.31 total 274.64 last 170.01) - open/close 40000 (time 1741898329.96 total 284.28 last 124.82) - open/close 41468 (time 1741898339.96 total 294.29 last 146.74) - open/close 42892 (time 1741898349.96 total 304.29 last 142.39) - open/close 44256 (time 1741898359.96 total 314.29 last 136.34) - open/close 45703 (time 1741898369.97 total 324.30 last 144.64) - open/close 47204 (time 1741898379.97 total 334.30 last 150.02) - open/close 48528 (time 1741898389.98 total 344.31 last 132.33) - open/close 49961 (time 1741898399.99 total 354.32 last 143.19) - open/close 51348 (time 1741898409.99 total 364.32 last 138.64) - open/close 52665 (time 1741898420.00 total 374.32 last 131.65) - open/close 54089 (time 1741898430.00 total 384.33 last 142.37) - open/close 55446 (time 1741898440.00 total 394.33 last 135.65) - open/close 56900 (time 1741898450.00 total 404.33 last 145.40) - open/close 58352 (time 1741898460.00 total 414.33 last 145.18) - open/close 59721 (time 1741898470.01 total 424.34 last 136.82) - open/close 61027 (time 1741898480.01 total 434.34 last 130.56) - open/close 62276 (time 1741898490.02 total 444.35 last 124.82) - open/close 63563 (time 1741898500.03 total 454.36 last 128.60) total: 64768 open/close in 463.95 seconds: 139.60 ops/second - /unlink 2649 (time 1741898521.13 total 10.00 last 264.85) - /unlink 5188 (time 1741898531.13 total 20.01 last 253.78) - /unlink 7671 (time 1741898541.14 total 30.01 last 248.20) - /unlink 10000 (time 1741898550.96 total 39.84 last 237.00) - /unlink 12439 (time 1741898560.97 total 49.84 last 243.80) - /unlink 15002 (time 1741898570.97 total 59.84 last 256.29) - /unlink 17863 (time 1741898580.97 total 69.84 last 286.03) - /unlink 20000 (time 1741898587.67 total 76.55 last 318.84) - /unlink 22875 (time 1741898597.67 total 86.55 last 287.49) - /unlink 25935 (time 1741898607.67 total 96.55 last 305.95) - /unlink 28919 (time 1741898617.67 total 106.55 last 298.35) - /unlink 30000 (time 1741898621.14 total 110.01 last 312.26) - /unlink 33219 (time 1741898631.14 total 120.02 last 321.75) - /unlink 36639 (time 1741898641.15 total 130.02 last 341.81) - /unlink 39957 (time 1741898651.15 total 140.02 last 331.77) - /unlink 43043 (time 1741898661.15 total 150.02 last 308.56) - /unlink 45787 (time 1741898671.15 total 160.02 last 274.40) - /unlink 48511 (time 1741898681.15 total 170.03 last 272.38) - /unlink 50000 (time 1741898686.27 total 175.15 last 290.71) - /unlink 52546 (time 1741898696.27 total 185.15 last 254.54) - /unlink 55191 (time 1741898706.28 total 195.16 last 264.32) - /unlink 58080 (time 1741898716.28 total 205.16 last 288.90) - /unlink 60000 (time 1741898722.43 total 211.30 last 312.32) - /unlink 63960 (time 1741898732.43 total 221.30 last 395.99) total: 64768 /unlink in 222.99 seconds: 290.45 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2716 (time 1741898762.18 total 10.00 last 271.57) - open/close 5570 (time 1741898772.18 total 20.00 last 285.34) - open/close 8201 (time 1741898782.18 total 30.01 last 263.02) - open/close 10000 (time 1741898789.31 total 37.14 last 252.32) - open/close 12909 (time 1741898799.31 total 47.14 last 290.90) - open/close 15915 (time 1741898809.31 total 57.14 last 300.52) - open/close 18279 (time 1741898819.32 total 67.14 last 236.36) - open/close 20000 (time 1741898828.29 total 76.11 last 191.80) - open/close 21816 (time 1741898838.29 total 86.11 last 181.57) - open/close 23607 (time 1741898848.29 total 96.12 last 179.09) - open/close 25399 (time 1741898858.30 total 106.12 last 179.06) - open/close 27226 (time 1741898868.30 total 116.12 last 182.67) - open/close 29296 (time 1741898878.30 total 126.13 last 206.94) - open/close 30000 (time 1741898881.09 total 128.92 last 252.39) - open/close 31941 (time 1741898891.10 total 138.92 last 194.02) - open/close 33755 (time 1741898901.10 total 148.93 last 181.31) - open/close 35514 (time 1741898911.11 total 158.93 last 175.77) - open/close 37332 (time 1741898921.11 total 168.94 last 181.76) - open/close 39173 (time 1741898931.12 total 178.94 last 184.00) - open/close 40000 (time 1741898935.30 total 183.13 last 197.65) - open/close 41950 (time 1741898945.30 total 193.13 last 194.99) - open/close 44234 (time 1741898955.30 total 203.13 last 228.39) - open/close 46713 (time 1741898965.30 total 213.13 last 247.82) - open/close 48942 (time 1741898975.31 total 223.13 last 222.84) - open/close 50000 (time 1741898981.02 total 228.85 last 185.08) - open/close 51809 (time 1741898991.03 total 238.85 last 180.83) - open/close 53719 (time 1741899001.03 total 248.85 last 190.97) - open/close 55563 (time 1741899011.03 total 258.85 last 184.40) - open/close 57807 (time 1741899021.03 total 268.86 last 224.38) - open/close 59863 (time 1741899031.03 total 278.86 last 205.55) - open/close 61912 (time 1741899041.03 total 288.86 last 204.90) - open/close 64312 (time 1741899051.04 total 298.86 last 239.96) total: 64768 open/close in 301.29 seconds: 214.97 ops/second - /unlink 3447 (time 1741899064.57 total 10.00 last 344.65) - /unlink 7687 (time 1741899074.57 total 20.00 last 423.95) - /unlink 10000 (time 1741899079.92 total 25.35 last 432.72) - /unlink 14901 (time 1741899089.92 total 35.35 last 490.08) - /unlink 20000 (time 1741899099.67 total 45.11 last 522.60) - /unlink 24660 (time 1741899109.67 total 55.11 last 466.00) - /unlink 29495 (time 1741899119.68 total 65.11 last 483.48) - /unlink 33433 (time 1741899129.68 total 75.11 last 393.79) - /unlink 37509 (time 1741899139.68 total 85.11 last 407.58) - /unlink 40000 (time 1741899145.22 total 90.65 last 449.08) - /unlink 44477 (time 1741899155.22 total 100.65 last 447.64) - /unlink 48719 (time 1741899165.22 total 110.65 last 424.20) - /unlink 50000 (time 1741899168.84 total 114.27 last 353.91) - /unlink 54296 (time 1741899178.84 total 124.27 last 429.59) - /unlink 58895 (time 1741899188.84 total 134.28 last 459.87) - /unlink 63824 (time 1741899198.85 total 144.28 last 492.79) total: 64768 /unlink in 146.32 seconds: 442.63 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2782 (time 1741899228.67 total 10.00 last 278.17) - open/close 5685 (time 1741899238.67 total 20.00 last 290.29) - open/close 9076 (time 1741899248.67 total 30.00 last 339.08) - open/close 10000 (time 1741899251.41 total 32.73 last 338.32) - open/close 12152 (time 1741899261.41 total 42.74 last 215.16) - open/close 14311 (time 1741899271.41 total 52.74 last 215.87) - open/close 16439 (time 1741899281.41 total 62.74 last 212.77) - open/close 18662 (time 1741899291.41 total 72.74 last 222.28) - open/close 20000 (time 1741899296.62 total 77.95 last 256.97) - open/close 22594 (time 1741899306.62 total 87.95 last 259.35) - open/close 25528 (time 1741899316.62 total 97.95 last 293.38) - open/close 28747 (time 1741899326.62 total 107.95 last 321.83) - open/close 30000 (time 1741899330.24 total 111.57 last 346.07) - open/close 33324 (time 1741899340.24 total 121.57 last 332.40) - open/close 36834 (time 1741899350.24 total 131.57 last 350.96) - open/close 40000 (time 1741899356.09 total 137.42 last 541.27) - open/close 44396 (time 1741899366.09 total 147.42 last 439.58) - open/close 49348 (time 1741899376.10 total 157.42 last 495.14) - open/close 54617 (time 1741899386.10 total 167.42 last 526.86) - open/close 60000 (time 1741899395.87 total 177.20 last 550.56) total: 64768 open/close in 186.32 seconds: 347.62 ops/second - /unlink 8967 (time 1741899415.78 total 10.00 last 896.69) - /unlink 17088 (time 1741899425.78 total 20.00 last 812.07) - /unlink 20000 (time 1741899429.14 total 23.36 last 866.42) - /unlink 28373 (time 1741899439.14 total 33.36 last 837.25) - /unlink 37247 (time 1741899449.14 total 43.36 last 887.36) - /unlink 40000 (time 1741899453.00 total 47.22 last 713.62) - /unlink 46892 (time 1741899463.00 total 57.22 last 689.06) - /unlink 50000 (time 1741899467.80 total 62.03 last 646.98) - /unlink 55981 (time 1741899477.80 total 72.03 last 598.07) - /unlink 60000 (time 1741899482.85 total 77.07 last 796.16) total: 64768 /unlink in 82.31 seconds: 786.90 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5100 (time 1741899518.64 total 10.00 last 510.00) - open/close 10000 (time 1741899528.43 total 19.79 last 500.56) - open/close 13856 (time 1741899538.43 total 29.79 last 385.53) - open/close 17156 (time 1741899548.43 total 39.79 last 329.96) - open/close 20000 (time 1741899556.09 total 47.46 last 370.96) - open/close 23069 (time 1741899566.10 total 57.46 last 306.87) - open/close 26667 (time 1741899576.10 total 67.46 last 359.74) - open/close 29935 (time 1741899586.10 total 77.46 last 326.73) - open/close 32604 (time 1741899596.10 total 87.46 last 266.87) - open/close 35846 (time 1741899606.10 total 97.47 last 324.14) - open/close 38908 (time 1741899616.10 total 107.47 last 306.13) - open/close 40000 (time 1741899621.03 total 112.40 last 221.52) - open/close 42635 (time 1741899631.04 total 122.40 last 263.44) - open/close 45321 (time 1741899641.04 total 132.40 last 268.51) - open/close 48684 (time 1741899651.04 total 142.41 last 336.21) - open/close 50000 (time 1741899656.35 total 147.71 last 248.14) - open/close 54731 (time 1741899666.37 total 157.73 last 471.92) - open/close 58858 (time 1741899676.37 total 167.74 last 412.66) - open/close 60000 (time 1741899678.95 total 170.31 last 443.12) total: 64768 open/close in 179.84 seconds: 360.13 ops/second - /unlink 7922 (time 1741899699.37 total 10.00 last 792.11) - /unlink 10000 (time 1741899701.55 total 12.18 last 952.45) - /unlink 18763 (time 1741899711.55 total 22.18 last 876.22) - /unlink 26233 (time 1741899721.55 total 32.18 last 746.96) - /unlink 30000 (time 1741899727.52 total 38.15 last 631.05) - /unlink 36274 (time 1741899737.52 total 48.15 last 627.34) - /unlink 40000 (time 1741899742.08 total 52.71 last 818.26) - /unlink 48110 (time 1741899752.08 total 62.71 last 810.96) - /unlink 50000 (time 1741899754.58 total 65.21 last 755.54) - /unlink 54714 (time 1741899764.58 total 75.21 last 471.24) - /unlink 59771 (time 1741899774.58 total 85.21 last 505.68) total: 64768 /unlink in 91.15 seconds: 710.58 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3740 (time 1741899808.74 total 10.00 last 373.93) - open/close 7852 (time 1741899818.74 total 20.00 last 411.10) - open/close 10000 (time 1741899823.92 total 25.18 last 415.13) - open/close 14503 (time 1741899833.92 total 35.18 last 450.22) - open/close 19269 (time 1741899843.92 total 45.18 last 476.59) - open/close 24025 (time 1741899853.92 total 55.18 last 475.53) - open/close 28915 (time 1741899863.92 total 65.18 last 488.96) - open/close 30000 (time 1741899866.26 total 67.53 last 462.98) - open/close 34612 (time 1741899876.27 total 77.53 last 461.12) - open/close 40000 (time 1741899886.03 total 87.29 last 551.86) - open/close 45338 (time 1741899896.03 total 97.29 last 533.71) - open/close 50000 (time 1741899904.96 total 106.22 last 522.03) - open/close 54329 (time 1741899914.96 total 116.23 last 432.78) - open/close 59218 (time 1741899924.97 total 126.23 last 488.85) - open/close 62970 (time 1741899934.97 total 136.23 last 375.13) total: 64768 open/close in 141.22 seconds: 458.63 ops/second - /unlink 7397 (time 1741899950.85 total 10.00 last 739.70) - /unlink 10000 (time 1741899954.78 total 13.93 last 662.79) - /unlink 17275 (time 1741899964.78 total 23.93 last 727.44) - /unlink 20000 (time 1741899967.91 total 27.05 last 872.52) - /unlink 29251 (time 1741899977.91 total 37.05 last 925.04) - /unlink 37183 (time 1741899987.91 total 47.05 last 793.16) - /unlink 40000 (time 1741899992.52 total 51.66 last 610.87) - /unlink 47401 (time 1741900002.52 total 61.66 last 740.10) - /unlink 50000 (time 1741900006.22 total 65.36 last 702.94) - /unlink 57340 (time 1741900016.22 total 75.36 last 734.00) - /unlink 60000 (time 1741900019.49 total 78.63 last 813.51) total: 64768 /unlink in 84.34 seconds: 767.97 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg320-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg320-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg320-server stop mds service on oleg320-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg320-server unloading modules via unload_modules_local on: 'oleg320-server' oleg320-server: oleg320-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg320-client: oleg320-client: ssh exited with exit code 2 pdsh@oleg320-client: oleg320-server: ssh exited with exit code 2 pdsh@oleg320-client: oleg320-client: ssh exited with exit code 2 pdsh@oleg320-client: oleg320-server: ssh exited with exit code 2