== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 07:49:06 (1746791346) Stopping clients: oleg405-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg405-client.virtnet /mnt/lustre2 (opts:-f) oleg405-server: oleg405-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg405-server' oleg405-server: oleg405-server.virtnet: executing load_modules_local oleg405-server: Loading modules from /home/green/git/lustre-release/lustre oleg405-server: detected 4 online CPUs by sysfs oleg405-server: MODOPTS_LIBCFS= oleg405-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg405-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg405-server: oleg405-server.virtnet: executing set_default_debug -1 all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg405-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg405-server: oleg405-server.virtnet: executing set_default_debug -1 all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg405-client.virtnet: -o user_xattr,flock 192.168.204.105@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86890 331 86559 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32587 331 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 5003 (time 1746791380.22 total 10.00 last 500.18) - open/close 10000 (time 1746791389.88 total 19.67 last 517.10) - open/close 15208 (time 1746791399.88 total 29.67 last 520.75) - open/close 20000 (time 1746791409.04 total 38.83 last 523.22) - open/close 25295 (time 1746791419.04 total 48.83 last 529.42) - open/close 30000 (time 1746791427.74 total 57.53 last 540.70) - open/close 35386 (time 1746791437.75 total 67.53 last 538.54) - open/close 40000 (time 1746791446.18 total 75.96 last 547.35) - open/close 45016 (time 1746791456.18 total 85.96 last 501.54) - open/close 49832 (time 1746791466.18 total 95.96 last 481.51) - open/close 54581 (time 1746791476.18 total 105.96 last 474.84) - open/close 59764 (time 1746791486.18 total 115.96 last 518.26) total: 64768 open/close in 125.32 seconds: 516.83 ops/second - /unlink 8988 (time 1746791506.37 total 10.00 last 898.78) - /unlink 18311 (time 1746791516.37 total 20.00 last 932.21) - /unlink 26020 (time 1746791526.37 total 30.00 last 770.84) - /unlink 30000 (time 1746791530.70 total 34.33 last 919.05) - /unlink 38921 (time 1746791540.70 total 44.33 last 892.03) - /unlink 47257 (time 1746791550.70 total 54.34 last 833.45) - /unlink 50000 (time 1746791554.07 total 57.71 last 813.38) - /unlink 58184 (time 1746791564.07 total 67.71 last 818.39) total: 64768 /unlink in 74.62 seconds: 868.01 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5426 (time 1746791603.78 total 10.00 last 542.58) - open/close 10000 (time 1746791612.67 total 18.88 last 514.82) - open/close 15171 (time 1746791622.67 total 28.89 last 517.09) - open/close 20000 (time 1746791631.98 total 38.20 last 518.47) - open/close 25158 (time 1746791641.98 total 48.20 last 515.77) - open/close 30000 (time 1746791651.80 total 58.02 last 493.07) - open/close 35048 (time 1746791661.80 total 68.02 last 504.79) - open/close 40000 (time 1746791671.52 total 77.74 last 509.66) - open/close 45158 (time 1746791681.52 total 87.74 last 515.72) - open/close 50000 (time 1746791691.07 total 97.29 last 506.96) - open/close 55198 (time 1746791701.07 total 107.29 last 519.70) - open/close 60000 (time 1746791710.45 total 116.67 last 511.94) total: 64768 open/close in 125.76 seconds: 515.00 ops/second - /unlink 9037 (time 1746791730.38 total 10.00 last 903.68) - /unlink 17971 (time 1746791740.38 total 20.00 last 893.34) - /unlink 20000 (time 1746791742.58 total 22.20 last 922.25) - /unlink 29050 (time 1746791752.58 total 32.20 last 904.95) - /unlink 38196 (time 1746791762.59 total 42.20 last 914.46) - /unlink 40000 (time 1746791764.59 total 44.20 last 901.55) - /unlink 48965 (time 1746791774.59 total 54.21 last 896.38) - /unlink 57400 (time 1746791784.59 total 64.21 last 843.43) - /unlink 60000 (time 1746791787.30 total 66.92 last 959.11) total: 64768 /unlink in 72.84 seconds: 889.20 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5288 (time 1746791829.89 total 10.00 last 528.75) - open/close 10000 (time 1746791839.39 total 19.50 last 496.17) - open/close 15259 (time 1746791849.39 total 29.50 last 525.81) - open/close 19870 (time 1746791859.39 total 39.50 last 461.08) - open/close 24972 (time 1746791869.39 total 49.50 last 510.12) - open/close 30000 (time 1746791879.02 total 59.13 last 521.94) - open/close 34729 (time 1746791889.03 total 69.14 last 472.74) - open/close 38636 (time 1746791899.03 total 79.14 last 390.61) - open/close 40000 (time 1746791903.19 total 83.30 last 327.97) - open/close 43485 (time 1746791913.19 total 93.30 last 348.45) - open/close 48009 (time 1746791923.19 total 103.30 last 452.31) - open/close 50000 (time 1746791927.77 total 107.88 last 435.18) - open/close 54878 (time 1746791937.77 total 117.88 last 487.72) - open/close 59704 (time 1746791947.77 total 127.88 last 482.56) - open/close 64533 (time 1746791957.77 total 137.88 last 482.81) total: 64768 open/close in 138.39 seconds: 468.01 ops/second - /unlink 8432 (time 1746791969.11 total 10.00 last 843.14) - /unlink 17514 (time 1746791979.11 total 20.00 last 908.15) - /unlink 20000 (time 1746791981.99 total 22.88 last 864.97) - /unlink 28756 (time 1746791991.99 total 32.88 last 875.45) - /unlink 37990 (time 1746792001.99 total 42.88 last 923.32) - /unlink 40000 (time 1746792004.22 total 45.11 last 902.37) - /unlink 49120 (time 1746792014.22 total 55.11 last 911.97) - /unlink 57480 (time 1746792024.22 total 65.11 last 835.92) - /unlink 60000 (time 1746792027.13 total 68.02 last 866.37) total: 64768 /unlink in 72.98 seconds: 887.48 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5021 (time 1746792070.23 total 10.00 last 502.08) - open/close 9449 (time 1746792080.23 total 20.00 last 442.77) - open/close 13966 (time 1746792090.23 total 30.00 last 451.63) - open/close 18435 (time 1746792100.23 total 40.00 last 446.90) - open/close 20000 (time 1746792103.22 total 42.99 last 523.97) - open/close 25057 (time 1746792113.22 total 52.99 last 505.62) - open/close 29939 (time 1746792123.22 total 62.99 last 488.18) - open/close 35118 (time 1746792133.22 total 72.99 last 517.86) - open/close 40000 (time 1746792142.83 total 82.60 last 508.27) - open/close 45115 (time 1746792152.83 total 92.60 last 511.42) - open/close 49809 (time 1746792162.83 total 102.60 last 469.37) - open/close 53853 (time 1746792172.83 total 112.60 last 404.38) - open/close 58958 (time 1746792182.83 total 122.60 last 510.50) - open/close 63382 (time 1746792192.83 total 132.60 last 442.34) total: 64768 open/close in 135.48 seconds: 478.07 ops/second - /unlink 8520 (time 1746792206.55 total 10.00 last 851.96) - /unlink 17281 (time 1746792216.55 total 20.00 last 876.06) - /unlink 20000 (time 1746792220.46 total 23.91 last 695.62) - /unlink 26355 (time 1746792230.46 total 33.91 last 635.38) - /unlink 30000 (time 1746792235.15 total 38.59 last 778.35) - /unlink 37852 (time 1746792245.15 total 48.60 last 785.11) - /unlink 40000 (time 1746792248.18 total 51.63 last 708.55) - /unlink 47091 (time 1746792258.18 total 61.63 last 709.08) - /unlink 50000 (time 1746792263.15 total 66.60 last 585.19) - /unlink 56479 (time 1746792273.15 total 76.60 last 647.89) - /unlink 60000 (time 1746792277.87 total 81.31 last 746.63) total: 64768 /unlink in 88.57 seconds: 731.30 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5038 (time 1746792315.51 total 10.00 last 503.69) - open/close 10000 (time 1746792324.93 total 19.43 last 526.43) - open/close 15135 (time 1746792334.93 total 29.43 last 513.49) - open/close 19350 (time 1746792344.94 total 39.43 last 421.45) - open/close 23716 (time 1746792354.94 total 49.43 last 436.59) - open/close 27936 (time 1746792364.94 total 59.43 last 421.96) - open/close 30000 (time 1746792370.01 total 64.51 last 406.65) - open/close 33585 (time 1746792380.01 total 74.51 last 358.48) - open/close 38069 (time 1746792390.01 total 84.51 last 448.39) - open/close 40000 (time 1746792395.21 total 89.70 last 371.52) - open/close 44094 (time 1746792405.21 total 99.71 last 409.33) - open/close 47122 (time 1746792415.22 total 109.71 last 302.68) - open/close 49774 (time 1746792425.22 total 119.71 last 265.12) - open/close 52963 (time 1746792435.22 total 129.71 last 318.86) - open/close 56051 (time 1746792445.22 total 139.72 last 308.74) - open/close 58781 (time 1746792455.22 total 149.72 last 272.96) - open/close 60000 (time 1746792459.21 total 153.70 last 306.11) - open/close 63634 (time 1746792469.21 total 163.70 last 363.33) total: 64768 open/close in 167.05 seconds: 387.71 ops/second - /unlink 6246 (time 1746792483.66 total 10.00 last 624.52) - /unlink 10000 (time 1746792489.15 total 15.50 last 683.21) - /unlink 16013 (time 1746792499.16 total 25.50 last 601.25) - /unlink 20000 (time 1746792504.37 total 30.71 last 765.11) - /unlink 27197 (time 1746792514.37 total 40.71 last 719.66) - /unlink 30000 (time 1746792517.92 total 44.26 last 788.76) - /unlink 37687 (time 1746792527.92 total 54.26 last 768.67) - /unlink 40000 (time 1746792530.73 total 57.07 last 822.87) - /unlink 48551 (time 1746792540.73 total 67.07 last 854.96) - /unlink 50000 (time 1746792543.32 total 69.66 last 561.18) - /unlink 56251 (time 1746792553.32 total 79.66 last 625.09) - /unlink 60000 (time 1746792558.37 total 84.71 last 742.03) total: 64768 /unlink in 90.26 seconds: 717.57 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg405-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg405-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg405-server stop mds service on oleg405-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg405-server unloading modules via unload_modules_local on: 'oleg405-server' oleg405-server: oleg405-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg405-client: oleg405-client: ssh exited with exit code 2 pdsh@oleg405-client: oleg405-server: ssh exited with exit code 2 pdsh@oleg405-client: oleg405-client: ssh exited with exit code 2 pdsh@oleg405-client: oleg405-server: ssh exited with exit code 2