== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 13:11:06 (1744045866) Stopping clients: oleg451-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg451-client.virtnet /mnt/lustre2 (opts:-f) oleg451-server: oleg451-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg451-server' oleg451-server: oleg451-server.virtnet: executing load_modules_local oleg451-server: Loading modules from /home/green/git/lustre-release/lustre oleg451-server: detected 4 online CPUs by sysfs oleg451-server: MODOPTS_LIBCFS= oleg451-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg451-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg451-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg451-client.virtnet: -o user_xattr,flock 192.168.204.151@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4943 (time 1744045900.23 total 10.00 last 494.26) - open/close 9733 (time 1744045910.23 total 20.00 last 478.93) - open/close 15430 (time 1744045920.23 total 30.00 last 569.66) - open/close 20000 (time 1744045928.68 total 38.46 last 540.59) - open/close 24303 (time 1744045938.68 total 48.46 last 430.26) - open/close 28401 (time 1744045948.68 total 58.46 last 409.78) - open/close 30000 (time 1744045952.51 total 62.28 last 418.09) - open/close 34389 (time 1744045962.51 total 72.28 last 438.88) - open/close 39219 (time 1744045972.51 total 82.29 last 482.92) - open/close 44615 (time 1744045982.51 total 92.29 last 539.52) - open/close 49713 (time 1744045992.51 total 102.29 last 509.78) - open/close 54270 (time 1744046002.51 total 112.29 last 455.64) - open/close 59410 (time 1744046012.52 total 122.29 last 513.93) - open/close 64061 (time 1744046022.52 total 132.29 last 465.02) total: 64768 open/close in 133.79 seconds: 484.09 ops/second - /unlink 9244 (time 1744046034.85 total 10.00 last 924.30) - /unlink 17238 (time 1744046044.85 total 20.00 last 799.36) - /unlink 20000 (time 1744046048.20 total 23.35 last 824.23) - /unlink 28905 (time 1744046058.20 total 33.35 last 890.44) - /unlink 38293 (time 1744046068.20 total 43.35 last 938.80) - /unlink 48246 (time 1744046078.20 total 53.35 last 995.25) - /unlink 50000 (time 1744046080.23 total 55.39 last 863.00) - /unlink 58419 (time 1744046090.23 total 65.39 last 841.80) - /unlink 60000 (time 1744046092.35 total 67.51 last 746.13) total: 64768 /unlink in 72.31 seconds: 895.66 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4929 (time 1744046133.54 total 10.00 last 492.87) - open/close 9597 (time 1744046143.54 total 20.00 last 466.80) - open/close 14769 (time 1744046153.54 total 30.00 last 517.14) - open/close 20000 (time 1744046162.83 total 39.29 last 562.95) - open/close 25368 (time 1744046172.83 total 49.30 last 536.73) - open/close 30000 (time 1744046181.11 total 57.57 last 559.80) - open/close 35587 (time 1744046191.11 total 67.57 last 558.70) - open/close 40000 (time 1744046200.66 total 77.13 last 461.77) - open/close 45708 (time 1744046210.67 total 87.13 last 570.64) - open/close 50000 (time 1744046218.37 total 94.83 last 557.38) - open/close 54839 (time 1744046228.37 total 104.83 last 483.85) - open/close 59546 (time 1744046238.37 total 114.83 last 470.64) - open/close 64695 (time 1744046248.37 total 124.83 last 514.89) total: 64768 open/close in 124.97 seconds: 518.27 ops/second - /unlink 9582 (time 1744046259.29 total 10.00 last 958.13) - /unlink 19775 (time 1744046269.29 total 20.00 last 1019.27) - /unlink 29995 (time 1744046279.29 total 30.00 last 1021.99) - /unlink 40000 (time 1744046289.26 total 39.97 last 1004.09) - /unlink 48981 (time 1744046299.26 total 49.97 last 898.03) - /unlink 57905 (time 1744046309.26 total 59.97 last 892.35) - /unlink 60000 (time 1744046311.55 total 62.26 last 912.87) total: 64768 /unlink in 67.66 seconds: 957.31 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5044 (time 1744046358.91 total 10.00 last 504.39) - open/close 10000 (time 1744046368.71 total 19.80 last 505.67) - open/close 15373 (time 1744046378.71 total 29.80 last 537.22) - open/close 20000 (time 1744046387.29 total 38.39 last 539.03) - open/close 25252 (time 1744046397.29 total 48.39 last 525.04) - open/close 30000 (time 1744046406.12 total 57.22 last 537.82) - open/close 35027 (time 1744046416.12 total 67.22 last 502.68) - open/close 40000 (time 1744046425.87 total 76.97 last 509.98) - open/close 45183 (time 1744046435.88 total 86.97 last 518.30) - open/close 50000 (time 1744046445.43 total 96.52 last 504.21) - open/close 55122 (time 1744046455.43 total 106.52 last 512.14) - open/close 60000 (time 1744046464.93 total 116.03 last 513.39) total: 64768 open/close in 125.31 seconds: 516.85 ops/second - /unlink 8823 (time 1744046485.03 total 10.00 last 882.28) - /unlink 17717 (time 1744046495.03 total 20.00 last 889.37) - /unlink 20000 (time 1744046497.57 total 22.54 last 899.70) - /unlink 28313 (time 1744046507.57 total 32.54 last 831.16) - /unlink 30000 (time 1744046509.62 total 34.58 last 825.85) - /unlink 37972 (time 1744046519.62 total 44.58 last 797.17) - /unlink 40000 (time 1744046521.96 total 46.93 last 863.51) - /unlink 48521 (time 1744046531.97 total 56.93 last 852.04) - /unlink 57599 (time 1744046541.97 total 66.93 last 907.80) - /unlink 60000 (time 1744046544.69 total 69.66 last 880.67) total: 64768 /unlink in 74.89 seconds: 864.84 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5206 (time 1744046584.47 total 10.00 last 520.52) - open/close 10000 (time 1744046593.86 total 19.40 last 510.09) - open/close 15133 (time 1744046603.87 total 29.40 last 513.25) - open/close 20000 (time 1744046613.53 total 39.06 last 503.79) - open/close 25059 (time 1744046623.53 total 49.06 last 505.89) - open/close 30000 (time 1744046633.45 total 58.99 last 497.78) - open/close 34982 (time 1744046643.45 total 68.99 last 498.11) - open/close 40000 (time 1744046653.31 total 78.84 last 509.28) - open/close 45170 (time 1744046663.31 total 88.84 last 516.98) - open/close 50000 (time 1744046672.77 total 98.31 last 510.46) - open/close 55061 (time 1744046682.77 total 108.31 last 506.07) - open/close 60000 (time 1744046692.60 total 118.13 last 502.51) total: 64768 open/close in 127.43 seconds: 508.27 ops/second - /unlink 9038 (time 1744046712.74 total 10.00 last 903.74) - /unlink 18164 (time 1744046722.74 total 20.00 last 912.58) - /unlink 20000 (time 1744046724.94 total 22.20 last 834.52) - /unlink 29058 (time 1744046734.94 total 32.20 last 905.71) - /unlink 38165 (time 1744046744.94 total 42.20 last 910.64) - /unlink 47282 (time 1744046754.94 total 52.20 last 911.61) - /unlink 50000 (time 1744046757.90 total 55.16 last 918.19) - /unlink 58882 (time 1744046767.90 total 65.16 last 888.16) total: 64768 /unlink in 71.75 seconds: 902.67 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4175 (time 1744046814.81 total 10.00 last 417.41) - open/close 9182 (time 1744046824.81 total 20.00 last 500.61) - open/close 13558 (time 1744046834.81 total 30.01 last 437.53) - open/close 18175 (time 1744046844.81 total 40.01 last 461.63) - open/close 20000 (time 1744046849.18 total 44.38 last 417.68) - open/close 24504 (time 1744046859.18 total 54.38 last 450.32) - open/close 29253 (time 1744046869.19 total 64.38 last 474.86) - open/close 34175 (time 1744046879.19 total 74.38 last 492.19) - open/close 38938 (time 1744046889.19 total 84.38 last 476.15) - open/close 40000 (time 1744046891.84 total 87.04 last 400.11) - open/close 44919 (time 1744046901.84 total 97.04 last 491.90) - open/close 49903 (time 1744046911.84 total 107.04 last 498.38) - open/close 54398 (time 1744046921.84 total 117.04 last 449.48) - open/close 58616 (time 1744046931.85 total 127.04 last 421.76) - open/close 60000 (time 1744046935.63 total 130.82 last 365.74) - open/close 64460 (time 1744046945.63 total 140.82 last 445.97) total: 64768 open/close in 141.39 seconds: 458.08 ops/second - /unlink 8694 (time 1744046957.04 total 10.00 last 869.14) - /unlink 14746 (time 1744046967.04 total 20.00 last 605.20) - /unlink 20000 (time 1744046976.84 total 29.81 last 536.00) - /unlink 25565 (time 1744046986.84 total 39.81 last 556.49) - /unlink 30000 (time 1744046994.05 total 47.01 last 615.17) - /unlink 35614 (time 1744047004.05 total 57.02 last 561.28) - /unlink 40000 (time 1744047010.38 total 63.34 last 693.46) - /unlink 48804 (time 1744047020.38 total 73.34 last 880.36) - /unlink 57030 (time 1744047030.38 total 83.34 last 822.58) - /unlink 60000 (time 1744047033.85 total 86.81 last 855.43) total: 64768 /unlink in 92.53 seconds: 699.97 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg451-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg451-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg451-server stop mds service on oleg451-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg451-server unloading modules via unload_modules_local on: 'oleg451-server' oleg451-server: oleg451-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg451-client: oleg451-client: ssh exited with exit code 2 pdsh@oleg451-client: oleg451-server: ssh exited with exit code 2 pdsh@oleg451-client: oleg451-client: ssh exited with exit code 2 pdsh@oleg451-client: oleg451-server: ssh exited with exit code 2