== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 17:09:13 (1747256953) Stopping clients: oleg319-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg319-client.virtnet /mnt/lustre2 (opts:-f) oleg319-server: oleg319-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg319-server' oleg319-server: oleg319-server.virtnet: executing load_modules_local oleg319-server: Loading modules from /home/green/git/lustre-release/lustre oleg319-server: detected 4 online CPUs by sysfs oleg319-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg319-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg319-server: oleg319-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg319-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg319-server: oleg319-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg319-client.virtnet: -o user_xattr,flock oleg319-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16166 358 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 2257 (time 1747256994.91 total 10.00 last 225.64) - open/close 4182 (time 1747257004.91 total 20.01 last 192.43) - open/close 6231 (time 1747257014.91 total 30.01 last 204.83) - open/close 8187 (time 1747257024.92 total 40.01 last 195.55) - open/close 10000 (time 1747257032.91 total 48.00 last 226.86) - open/close 12159 (time 1747257042.91 total 58.01 last 215.89) - open/close 15026 (time 1747257052.91 total 68.01 last 286.68) - open/close 18561 (time 1747257062.91 total 78.01 last 353.38) - open/close 20000 (time 1747257067.68 total 82.78 last 301.73) - open/close 23271 (time 1747257077.68 total 92.78 last 327.08) - open/close 25948 (time 1747257087.68 total 102.78 last 267.69) - open/close 29273 (time 1747257097.68 total 112.78 last 332.41) - open/close 30000 (time 1747257100.26 total 115.36 last 281.76) - open/close 32633 (time 1747257110.27 total 125.37 last 263.14) - open/close 35559 (time 1747257120.27 total 135.37 last 292.55) - open/close 38171 (time 1747257130.27 total 145.37 last 261.13) - open/close 40000 (time 1747257136.80 total 151.90 last 280.09) - open/close 42306 (time 1747257146.81 total 161.90 last 230.55) - open/close 44901 (time 1747257156.81 total 171.90 last 259.49) - open/close 47267 (time 1747257166.81 total 181.91 last 236.52) - open/close 50000 (time 1747257176.30 total 191.40 last 287.87) - open/close 52881 (time 1747257186.31 total 201.40 last 288.06) - open/close 55701 (time 1747257196.31 total 211.40 last 281.95) - open/close 58539 (time 1747257206.31 total 221.41 last 283.77) - open/close 60000 (time 1747257210.66 total 225.76 last 335.60) - open/close 63069 (time 1747257220.66 total 235.76 last 306.87) total: 64768 open/close in 240.39 seconds: 269.43 ops/second - /unlink 6070 (time 1747257236.24 total 10.00 last 606.94) - /unlink 10000 (time 1747257244.00 total 17.76 last 506.45) - /unlink 15188 (time 1747257254.00 total 27.76 last 518.79) - /unlink 20000 (time 1747257263.32 total 37.08 last 516.48) - /unlink 25070 (time 1747257273.32 total 47.08 last 506.97) - /unlink 30000 (time 1747257280.50 total 54.27 last 686.02) - /unlink 35674 (time 1747257290.50 total 64.27 last 567.38) - /unlink 40000 (time 1747257299.21 total 72.97 last 496.72) - /unlink 45691 (time 1747257309.22 total 82.98 last 569.01) - /unlink 50000 (time 1747257315.92 total 89.68 last 643.13) - /unlink 55882 (time 1747257325.92 total 99.68 last 588.20) - /unlink 60000 (time 1747257332.73 total 106.49 last 604.54) total: 64768 /unlink in 113.51 seconds: 570.57 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 3227 (time 1747257373.09 total 10.00 last 322.66) - open/close 6785 (time 1747257383.09 total 20.00 last 355.71) - open/close 9929 (time 1747257393.09 total 30.00 last 314.37) - open/close 13027 (time 1747257403.09 total 40.01 last 309.76) - open/close 16314 (time 1747257413.09 total 50.01 last 328.57) - open/close 19603 (time 1747257423.10 total 60.01 last 328.79) - open/close 22984 (time 1747257433.10 total 70.01 last 338.04) - open/close 26806 (time 1747257443.10 total 80.02 last 382.14) - open/close 30000 (time 1747257450.67 total 87.59 last 421.90) - open/close 33747 (time 1747257460.67 total 97.59 last 374.67) - open/close 37064 (time 1747257470.67 total 107.59 last 331.69) - open/close 40000 (time 1747257479.77 total 116.69 last 322.65) - open/close 43619 (time 1747257489.77 total 126.69 last 361.87) - open/close 47229 (time 1747257499.78 total 136.69 last 360.90) - open/close 50000 (time 1747257507.02 total 143.94 last 382.52) - open/close 53786 (time 1747257517.02 total 153.94 last 378.53) - open/close 57903 (time 1747257527.02 total 163.94 last 411.69) - open/close 60000 (time 1747257531.91 total 168.83 last 428.54) - open/close 64661 (time 1747257541.92 total 178.83 last 466.00) total: 64768 open/close in 179.06 seconds: 361.71 ops/second - /unlink 7246 (time 1747257552.99 total 10.00 last 724.56) - /unlink 10000 (time 1747257556.75 total 13.77 last 731.07) - /unlink 17748 (time 1747257566.75 total 23.77 last 774.76) - /unlink 20000 (time 1747257569.09 total 26.11 last 962.90) - /unlink 28532 (time 1747257579.09 total 36.11 last 853.12) - /unlink 37064 (time 1747257589.09 total 46.11 last 853.19) - /unlink 40000 (time 1747257592.81 total 49.82 last 790.89) - /unlink 48904 (time 1747257602.81 total 59.82 last 890.34) - /unlink 57881 (time 1747257612.81 total 69.82 last 897.65) - /unlink 60000 (time 1747257615.32 total 72.33 last 844.55) total: 64768 /unlink in 77.34 seconds: 837.44 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5028 (time 1747257653.86 total 10.00 last 502.75) - open/close 9855 (time 1747257663.87 total 20.00 last 482.67) - open/close 14832 (time 1747257673.87 total 30.00 last 497.70) - open/close 20000 (time 1747257682.53 total 38.67 last 596.42) - open/close 25709 (time 1747257692.53 total 48.67 last 570.78) - open/close 30000 (time 1747257699.78 total 55.92 last 591.77) - open/close 36083 (time 1747257709.78 total 65.92 last 608.25) - open/close 40000 (time 1747257716.45 total 72.58 last 588.02) - open/close 46166 (time 1747257726.45 total 82.58 last 616.58) - open/close 50000 (time 1747257732.84 total 88.98 last 599.58) - open/close 55626 (time 1747257742.84 total 98.98 last 562.51) - open/close 60000 (time 1747257751.96 total 108.10 last 479.69) total: 64768 open/close in 117.62 seconds: 550.63 ops/second - /unlink 9347 (time 1747257772.34 total 10.00 last 934.65) - /unlink 19197 (time 1747257782.34 total 20.00 last 984.96) - /unlink 28004 (time 1747257792.34 total 30.00 last 880.67) - /unlink 30000 (time 1747257794.45 total 32.11 last 945.96) - /unlink 39358 (time 1747257804.45 total 42.11 last 935.77) - /unlink 48673 (time 1747257814.45 total 52.11 last 931.49) - /unlink 57919 (time 1747257824.45 total 62.11 last 924.52) - /unlink 60000 (time 1747257826.90 total 64.56 last 848.73) total: 64768 /unlink in 69.61 seconds: 930.43 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5368 (time 1747257873.70 total 10.00 last 536.73) - open/close 10000 (time 1747257883.03 total 19.33 last 496.38) - open/close 15184 (time 1747257893.03 total 29.33 last 518.39) - open/close 20000 (time 1747257902.25 total 38.55 last 522.45) - open/close 25266 (time 1747257912.25 total 48.55 last 526.55) - open/close 30000 (time 1747257921.45 total 57.75 last 514.93) - open/close 35220 (time 1747257931.45 total 67.75 last 521.99) - open/close 40000 (time 1747257940.51 total 76.82 last 527.05) - open/close 45363 (time 1747257950.51 total 86.82 last 536.30) - open/close 50000 (time 1747257959.31 total 95.61 last 527.11) - open/close 55291 (time 1747257969.31 total 105.61 last 529.06) - open/close 60000 (time 1747257978.56 total 114.86 last 509.19) total: 64768 open/close in 123.80 seconds: 523.18 ops/second - /unlink 9310 (time 1747257998.33 total 10.00 last 930.92) - /unlink 18864 (time 1747258008.33 total 20.00 last 955.40) - /unlink 27874 (time 1747258018.33 total 30.00 last 900.90) - /unlink 30000 (time 1747258020.57 total 32.24 last 948.90) - /unlink 39480 (time 1747258030.57 total 42.24 last 947.95) - /unlink 47341 (time 1747258040.58 total 52.24 last 786.06) - /unlink 50000 (time 1747258045.47 total 57.14 last 542.86) - /unlink 52883 (time 1747258055.47 total 67.14 last 288.27) - /unlink 59868 (time 1747258065.48 total 77.14 last 698.41) total: 64768 /unlink in 83.18 seconds: 778.69 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 4228 (time 1747258099.62 total 10.00 last 422.78) - open/close 10000 (time 1747258109.28 total 19.66 last 597.40) - open/close 15591 (time 1747258119.28 total 29.66 last 558.99) - open/close 19963 (time 1747258129.29 total 39.67 last 437.13) - open/close 23539 (time 1747258139.29 total 49.67 last 357.51) - open/close 26013 (time 1747258149.29 total 59.67 last 247.37) - open/close 28653 (time 1747258159.29 total 69.67 last 263.96) - open/close 30000 (time 1747258164.02 total 74.40 last 284.77) - open/close 32745 (time 1747258174.02 total 84.40 last 274.48) - open/close 35619 (time 1747258184.03 total 94.41 last 287.28) - open/close 38014 (time 1747258194.03 total 104.41 last 239.46) - open/close 40000 (time 1747258201.04 total 111.42 last 283.39) - open/close 44841 (time 1747258211.04 total 121.42 last 484.07) - open/close 48736 (time 1747258221.04 total 131.42 last 389.47) - open/close 50000 (time 1747258223.93 total 134.31 last 437.02) - open/close 53838 (time 1747258233.93 total 144.31 last 383.73) - open/close 57895 (time 1747258243.93 total 154.31 last 405.64) - open/close 60000 (time 1747258249.60 total 159.98 last 371.62) - open/close 63972 (time 1747258259.60 total 169.98 last 397.18) total: 64768 open/close in 172.19 seconds: 376.15 ops/second - /unlink 7158 (time 1747258272.79 total 10.00 last 715.73) - /unlink 10000 (time 1747258277.04 total 14.25 last 668.34) - /unlink 17251 (time 1747258287.04 total 24.25 last 725.09) - /unlink 20000 (time 1747258290.35 total 27.56 last 831.86) - /unlink 28014 (time 1747258300.35 total 37.56 last 801.38) - /unlink 30000 (time 1747258303.07 total 40.28 last 728.42) - /unlink 36914 (time 1747258313.07 total 50.29 last 691.34) - /unlink 40000 (time 1747258317.60 total 54.81 last 682.53) - /unlink 46633 (time 1747258327.60 total 64.81 last 663.18) - /unlink 50000 (time 1747258331.75 total 68.96 last 811.38) - /unlink 58185 (time 1747258341.75 total 78.96 last 818.48) total: 64768 /unlink in 86.74 seconds: 746.73 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg319-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg319-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg319-server stop mds service on oleg319-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg319-server LNET ready to unload unloading modules on: 'oleg319-server' oleg319-server: oleg319-server.virtnet: executing unload_modules_local oleg319-server: LNET ready to unload modules unloaded. pdsh@oleg319-client: oleg319-client: ssh exited with exit code 2 pdsh@oleg319-client: oleg319-server: ssh exited with exit code 2 pdsh@oleg319-client: oleg319-client: ssh exited with exit code 2 pdsh@oleg319-client: oleg319-server: ssh exited with exit code 2