== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 13:20:29 (1778347229) start mds service on oleg329-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg329-server: oleg329-server.virtnet: executing set_default_debug -1 all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg329-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg329-server: oleg329-server.virtnet: executing set_default_debug -1 all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Started lustre-OST0000 oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg329-client.virtnet: -o user_xattr,flock 192.168.203.129@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 92553 329 92224 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 12328 424 11904 4% /mnt/lustre[OST:0] filesystem_summary: 12233 329 11904 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close/unlink 402 (time 1778347277.58 total 10.01 last 40.18) - open/close/unlink 938 (time 1778347287.59 total 20.01 last 53.58) - open/close/unlink 1251 (time 1778347297.62 total 30.04 last 31.21) - open/close/unlink 1596 (time 1778347307.63 total 40.05 last 34.45) - open/close/unlink 1968 (time 1778347317.66 total 50.08 last 37.10) - open/close/unlink 2407 (time 1778347327.67 total 60.09 last 43.84) - open/close/unlink 2879 (time 1778347337.69 total 70.11 last 47.10) - open/close/unlink 3340 (time 1778347347.71 total 80.13 last 46.02) - open/close/unlink 3896 (time 1778347357.72 total 90.14 last 55.54) - open/close/unlink 4424 (time 1778347367.74 total 100.16 last 52.72) - open/close/unlink 4987 (time 1778347377.74 total 110.16 last 56.28) - open/close/unlink 5550 (time 1778347387.74 total 120.16 last 56.30) - open/close/unlink 6479 (time 1778347397.75 total 130.17 last 92.84) - open/close/unlink 7399 (time 1778347407.75 total 140.17 last 91.99) - open/close/unlink 8249 (time 1778347417.75 total 150.17 last 84.94) - open/close/unlink 9065 (time 1778347427.77 total 160.19 last 81.49) - open/close/unlink 9993 (time 1778347437.77 total 170.19 last 92.79) - open/close/unlink 11155 (time 1778347447.77 total 180.19 last 116.13) - open/close/unlink 12513 (time 1778347457.78 total 190.20 last 135.73) - open/close/unlink 13798 (time 1778347467.79 total 200.21 last 128.38) - open/close/unlink 15105 (time 1778347477.79 total 210.21 last 130.65) - open/close/unlink 16767 (time 1778347487.80 total 220.22 last 166.11) - open/close/unlink 18506 (time 1778347497.80 total 230.22 last 173.78) - open/close/unlink 20000 (time 1778347506.48 total 238.90 last 172.25) - open/close/unlink 21470 (time 1778347516.48 total 248.90 last 146.98) - open/close/unlink 23046 (time 1778347526.48 total 258.90 last 157.55) - open/close/unlink 24542 (time 1778347536.48 total 268.90 last 149.56) - open/close/unlink 26085 (time 1778347546.48 total 278.91 last 154.29) - open/close/unlink 27806 (time 1778347556.49 total 288.91 last 172.04) - open/close/unlink 29516 (time 1778347566.49 total 298.91 last 170.97) - open/close/unlink 30000 (time 1778347569.53 total 301.95 last 159.12) - open/close/unlink 31672 (time 1778347579.53 total 311.96 last 167.15) - open/close/unlink 33469 (time 1778347589.55 total 321.97 last 179.46) - open/close/unlink 35316 (time 1778347599.55 total 331.97 last 184.68) - open/close/unlink 37201 (time 1778347609.55 total 341.98 last 188.40) - open/close/unlink 39023 (time 1778347619.60 total 352.02 last 181.42) - open/close/unlink 40000 (time 1778347625.14 total 357.56 last 176.39) - open/close/unlink 41737 (time 1778347635.15 total 367.57 last 173.55) - open/close/unlink 43395 (time 1778347645.15 total 377.57 last 165.69) - open/close/unlink 44955 (time 1778347655.15 total 387.57 last 155.97) - open/close/unlink 46595 (time 1778347665.15 total 397.57 last 164.00) - open/close/unlink 48234 (time 1778347675.16 total 407.58 last 163.84) - open/close/unlink 49744 (time 1778347685.16 total 417.58 last 151.00) - open/close/unlink 51261 (time 1778347695.16 total 427.58 last 151.65) - open/close/unlink 52810 (time 1778347705.17 total 437.59 last 154.83) - open/close/unlink 54483 (time 1778347715.17 total 447.59 last 167.29) - open/close/unlink 56100 (time 1778347725.17 total 457.59 last 161.65) - open/close/unlink 57784 (time 1778347735.17 total 467.59 last 168.35) - open/close/unlink 59341 (time 1778347745.18 total 477.60 last 155.64) - open/close/unlink 60000 (time 1778347749.37 total 481.79 last 157.02) - open/close/unlink 61589 (time 1778347759.38 total 491.80 last 158.83) - open/close/unlink 63159 (time 1778347769.39 total 501.81 last 156.89) total: 64768 open/close/unlink in 511.01 seconds: 126.75 ops/second - open/close/unlink 413 (time 1778347277.81 total 10.01 last 41.26) - open/close/unlink 934 (time 1778347287.82 total 20.02 last 52.07) - open/close/unlink 1240 (time 1778347297.85 total 30.05 last 30.51) - open/close/unlink 1592 (time 1778347307.86 total 40.06 last 35.14) - open/close/unlink 1950 (time 1778347317.87 total 50.07 last 35.78) - open/close/unlink 2388 (time 1778347327.88 total 60.08 last 43.75) - open/close/unlink 2850 (time 1778347337.88 total 70.08 last 46.18) - open/close/unlink 3312 (time 1778347347.90 total 80.10 last 46.11) - open/close/unlink 3870 (time 1778347357.92 total 90.12 last 55.69) - open/close/unlink 4402 (time 1778347367.94 total 100.14 last 53.13) - open/close/unlink 4978 (time 1778347377.94 total 110.14 last 57.57) - open/close/unlink 5531 (time 1778347387.95 total 120.15 last 55.25) - open/close/unlink 6473 (time 1778347397.95 total 130.15 last 94.20) - open/close/unlink 7386 (time 1778347407.96 total 140.16 last 91.27) - open/close/unlink 8245 (time 1778347417.96 total 150.16 last 85.89) - open/close/unlink 9054 (time 1778347427.96 total 160.16 last 80.85) - open/close/unlink 10000 (time 1778347437.93 total 170.13 last 94.93) - open/close/unlink 11164 (time 1778347447.93 total 180.13 last 116.39) - open/close/unlink 12552 (time 1778347457.93 total 190.13 last 138.76) - open/close/unlink 13825 (time 1778347467.94 total 200.14 last 127.21) - open/close/unlink 15134 (time 1778347477.94 total 210.14 last 130.88) - open/close/unlink 16803 (time 1778347487.95 total 220.15 last 166.83) - open/close/unlink 18543 (time 1778347497.95 total 230.15 last 173.95) - open/close/unlink 20000 (time 1778347506.42 total 238.63 last 171.89) - open/close/unlink 21462 (time 1778347516.43 total 248.63 last 146.11) - open/close/unlink 23032 (time 1778347526.43 total 258.63 last 157.00) - open/close/unlink 24527 (time 1778347536.44 total 268.64 last 149.44) - open/close/unlink 26067 (time 1778347546.44 total 278.64 last 154.00) - open/close/unlink 27786 (time 1778347556.44 total 288.64 last 171.84) - open/close/unlink 29477 (time 1778347566.44 total 298.64 last 169.02) - open/close/unlink 30000 (time 1778347569.64 total 301.84 last 163.79) - open/close/unlink 31666 (time 1778347579.64 total 311.84 last 166.57) - open/close/unlink 33426 (time 1778347589.64 total 321.84 last 175.99) - open/close/unlink 35267 (time 1778347599.65 total 331.85 last 183.85) - open/close/unlink 37147 (time 1778347609.66 total 341.86 last 187.84) - open/close/unlink 38964 (time 1778347619.67 total 351.87 last 181.56) - open/close/unlink 40000 (time 1778347625.95 total 358.15 last 164.83) - open/close/unlink 41729 (time 1778347635.96 total 368.16 last 172.78) - open/close/unlink 43363 (time 1778347645.96 total 378.16 last 163.35) - open/close/unlink 44940 (time 1778347655.98 total 388.18 last 157.41) - open/close/unlink 46599 (time 1778347665.98 total 398.18 last 165.88) - open/close/unlink 48257 (time 1778347676.01 total 408.21 last 165.41) - open/close/unlink 49809 (time 1778347686.01 total 418.21 last 155.19) - open/close/unlink 51324 (time 1778347696.01 total 428.21 last 151.48) - open/close/unlink 52877 (time 1778347706.01 total 438.21 last 155.26) - open/close/unlink 54564 (time 1778347716.01 total 448.21 last 168.66) - open/close/unlink 56220 (time 1778347726.02 total 458.22 last 165.56) - open/close/unlink 57864 (time 1778347736.02 total 468.22 last 164.35) - open/close/unlink 59427 (time 1778347746.02 total 478.22 last 156.23) - open/close/unlink 60000 (time 1778347749.78 total 481.98 last 152.67) - open/close/unlink 61588 (time 1778347759.79 total 491.99 last 158.63) - open/close/unlink 63164 (time 1778347769.79 total 501.99 last 157.59) total: 64768 open/close/unlink in 511.63 seconds: 126.59 ops/second - open/close/unlink 404 (time 1778347277.54 total 10.01 last 40.37) - open/close/unlink 939 (time 1778347287.55 total 20.02 last 53.44) - open/close/unlink 1247 (time 1778347297.58 total 30.05 last 30.70) - open/close/unlink 1596 (time 1778347307.60 total 40.06 last 34.85) - open/close/unlink 1971 (time 1778347317.61 total 50.08 last 37.45) - open/close/unlink 2408 (time 1778347327.61 total 60.08 last 43.68) - open/close/unlink 2866 (time 1778347337.62 total 70.09 last 45.77) - open/close/unlink 3315 (time 1778347347.62 total 80.09 last 44.87) - open/close/unlink 3875 (time 1778347357.64 total 90.11 last 55.91) - open/close/unlink 4404 (time 1778347367.65 total 100.12 last 52.85) - open/close/unlink 4973 (time 1778347377.66 total 110.12 last 56.87) - open/close/unlink 5524 (time 1778347387.66 total 120.13 last 55.06) - open/close/unlink 6452 (time 1778347397.67 total 130.14 last 92.73) - open/close/unlink 7380 (time 1778347407.67 total 140.14 last 92.78) - open/close/unlink 8239 (time 1778347417.67 total 150.14 last 85.90) - open/close/unlink 9051 (time 1778347427.68 total 160.15 last 81.16) - open/close/unlink 9979 (time 1778347437.68 total 170.15 last 92.80) - open/close/unlink 11142 (time 1778347447.68 total 180.15 last 116.29) - open/close/unlink 12507 (time 1778347457.68 total 190.15 last 136.50) - open/close/unlink 13794 (time 1778347467.68 total 200.15 last 128.68) - open/close/unlink 15092 (time 1778347477.68 total 210.15 last 129.74) - open/close/unlink 16751 (time 1778347487.69 total 220.15 last 165.86) - open/close/unlink 18467 (time 1778347497.69 total 230.16 last 171.52) - open/close/unlink 20000 (time 1778347506.61 total 239.08 last 171.90) - open/close/unlink 21471 (time 1778347516.61 total 249.08 last 147.09) - open/close/unlink 23061 (time 1778347526.61 total 259.08 last 158.97) - open/close/unlink 24548 (time 1778347536.62 total 269.08 last 148.65) - open/close/unlink 26112 (time 1778347546.62 total 279.09 last 156.35) - open/close/unlink 27826 (time 1778347556.62 total 289.09 last 171.37) - open/close/unlink 29539 (time 1778347566.62 total 299.09 last 171.23) - open/close/unlink 30000 (time 1778347569.54 total 302.01 last 158.02) - open/close/unlink 31651 (time 1778347579.54 total 312.01 last 165.06) - open/close/unlink 33421 (time 1778347589.56 total 322.03 last 176.71) - open/close/unlink 35266 (time 1778347599.57 total 332.04 last 184.31) - open/close/unlink 37143 (time 1778347609.57 total 342.04 last 187.68) - open/close/unlink 38981 (time 1778347619.58 total 352.05 last 183.70) - open/close/unlink 40000 (time 1778347625.68 total 358.15 last 166.94) - open/close/unlink 41716 (time 1778347635.69 total 368.15 last 171.54) - open/close/unlink 43374 (time 1778347645.69 total 378.16 last 165.74) - open/close/unlink 44939 (time 1778347655.69 total 388.16 last 156.49) - open/close/unlink 46588 (time 1778347665.70 total 398.17 last 164.75) - open/close/unlink 48235 (time 1778347675.70 total 408.17 last 164.68) - open/close/unlink 49763 (time 1778347685.70 total 418.17 last 152.78) - open/close/unlink 51282 (time 1778347695.70 total 428.17 last 151.87) - open/close/unlink 52840 (time 1778347705.71 total 438.18 last 155.75) - open/close/unlink 54535 (time 1778347715.71 total 448.18 last 169.48) - open/close/unlink 56164 (time 1778347725.71 total 458.18 last 162.85) - open/close/unlink 57819 (time 1778347735.72 total 468.18 last 165.45) - open/close/unlink 59381 (time 1778347745.72 total 478.19 last 156.14) - open/close/unlink 60000 (time 1778347749.78 total 482.25 last 152.48) - open/close/unlink 61578 (time 1778347759.78 total 492.25 last 157.76) - open/close/unlink 63154 (time 1778347769.78 total 502.25 last 157.58) total: 64768 open/close/unlink in 512.04 seconds: 126.49 ops/second - open/close/unlink 394 (time 1778347277.54 total 10.00 last 39.39) - open/close/unlink 930 (time 1778347287.55 total 20.02 last 53.51) - open/close/unlink 1235 (time 1778347297.56 total 30.03 last 30.48) - open/close/unlink 1580 (time 1778347307.58 total 40.05 last 34.44) - open/close/unlink 1951 (time 1778347317.60 total 50.07 last 37.03) - open/close/unlink 2393 (time 1778347327.62 total 60.09 last 44.09) - open/close/unlink 2851 (time 1778347337.64 total 70.11 last 45.74) - open/close/unlink 3305 (time 1778347347.64 total 80.11 last 45.39) - open/close/unlink 3859 (time 1778347357.64 total 90.11 last 55.40) - open/close/unlink 4388 (time 1778347367.65 total 100.12 last 52.83) - open/close/unlink 4954 (time 1778347377.66 total 110.13 last 56.54) - open/close/unlink 5512 (time 1778347387.68 total 120.14 last 55.73) - open/close/unlink 6442 (time 1778347397.69 total 130.16 last 92.88) - open/close/unlink 7375 (time 1778347407.69 total 140.16 last 93.25) - open/close/unlink 8234 (time 1778347417.69 total 150.16 last 85.88) - open/close/unlink 9048 (time 1778347427.71 total 160.17 last 81.32) - open/close/unlink 9976 (time 1778347437.71 total 170.18 last 92.71) - open/close/unlink 11141 (time 1778347447.72 total 180.18 last 116.49) - open/close/unlink 12515 (time 1778347457.72 total 190.18 last 137.40) - open/close/unlink 13786 (time 1778347467.72 total 200.19 last 127.03) - open/close/unlink 15077 (time 1778347477.72 total 210.19 last 129.07) - open/close/unlink 16736 (time 1778347487.73 total 220.20 last 165.82) - open/close/unlink 18454 (time 1778347497.73 total 230.20 last 171.80) - open/close/unlink 20000 (time 1778347506.70 total 239.17 last 172.32) - open/close/unlink 21455 (time 1778347516.70 total 249.17 last 145.45) - open/close/unlink 23036 (time 1778347526.71 total 259.17 last 158.07) - open/close/unlink 24507 (time 1778347536.71 total 269.18 last 147.06) - open/close/unlink 26059 (time 1778347546.71 total 279.18 last 155.13) - open/close/unlink 27789 (time 1778347556.71 total 289.18 last 172.97) - open/close/unlink 29495 (time 1778347566.72 total 299.19 last 170.55) - open/close/unlink 30000 (time 1778347569.84 total 302.31 last 161.51) - open/close/unlink 31641 (time 1778347579.84 total 312.31 last 164.09) - open/close/unlink 33380 (time 1778347589.85 total 322.31 last 173.87) - open/close/unlink 35241 (time 1778347599.86 total 332.32 last 185.92) - open/close/unlink 37099 (time 1778347609.86 total 342.33 last 185.76) - open/close/unlink 38916 (time 1778347619.86 total 352.33 last 181.69) - open/close/unlink 40000 (time 1778347626.29 total 358.76 last 168.55) - open/close/unlink 41737 (time 1778347636.29 total 368.76 last 173.66) - open/close/unlink 43393 (time 1778347646.29 total 378.76 last 165.59) - open/close/unlink 44940 (time 1778347656.30 total 388.77 last 154.62) - open/close/unlink 46612 (time 1778347666.30 total 398.77 last 167.13) - open/close/unlink 48250 (time 1778347676.31 total 408.78 last 163.60) - open/close/unlink 49808 (time 1778347686.32 total 418.79 last 155.72) - open/close/unlink 51322 (time 1778347696.32 total 428.79 last 151.37) - open/close/unlink 52900 (time 1778347706.33 total 438.79 last 157.74) - open/close/unlink 54603 (time 1778347716.33 total 448.79 last 170.28) - open/close/unlink 56254 (time 1778347726.33 total 458.80 last 165.09) - open/close/unlink 57897 (time 1778347736.33 total 468.80 last 164.22) - open/close/unlink 59463 (time 1778347746.33 total 478.80 last 156.57) - open/close/unlink 60000 (time 1778347749.92 total 482.39 last 149.66) - open/close/unlink 61593 (time 1778347759.92 total 492.39 last 159.29) - open/close/unlink 63165 (time 1778347769.92 total 502.39 last 157.17) total: 64768 open/close/unlink in 512.34 seconds: 126.42 ops/second - open/close/unlink 411 (time 1778347277.92 total 10.01 last 41.07) - open/close/unlink 935 (time 1778347287.92 total 20.01 last 52.39) - open/close/unlink 1238 (time 1778347297.93 total 30.02 last 30.26) - open/close/unlink 1591 (time 1778347307.95 total 40.04 last 35.25) - open/close/unlink 1958 (time 1778347317.96 total 50.05 last 36.65) - open/close/unlink 2400 (time 1778347327.96 total 60.05 last 44.19) - open/close/unlink 2865 (time 1778347337.97 total 70.06 last 46.44) - open/close/unlink 3328 (time 1778347347.98 total 80.07 last 46.28) - open/close/unlink 3890 (time 1778347358.00 total 90.09 last 56.10) - open/close/unlink 4420 (time 1778347368.00 total 100.10 last 52.95) - open/close/unlink 4992 (time 1778347378.02 total 110.11 last 57.13) - open/close/unlink 5545 (time 1778347388.02 total 120.11 last 55.27) - open/close/unlink 6487 (time 1778347398.03 total 130.12 last 94.17) - open/close/unlink 7403 (time 1778347408.03 total 140.12 last 91.60) - open/close/unlink 8256 (time 1778347418.04 total 150.13 last 85.22) - open/close/unlink 9075 (time 1778347428.04 total 160.13 last 81.85) - open/close/unlink 10000 (time 1778347437.96 total 170.05 last 93.31) - open/close/unlink 11167 (time 1778347447.96 total 180.05 last 116.70) - open/close/unlink 12546 (time 1778347457.96 total 190.05 last 137.86) - open/close/unlink 13802 (time 1778347467.96 total 200.06 last 125.53) - open/close/unlink 15114 (time 1778347477.97 total 210.06 last 131.19) - open/close/unlink 16766 (time 1778347487.97 total 220.06 last 165.19) - open/close/unlink 18498 (time 1778347497.97 total 230.06 last 173.18) - open/close/unlink 20000 (time 1778347506.72 total 238.81 last 171.55) - open/close/unlink 21467 (time 1778347516.72 total 248.82 last 146.67) - open/close/unlink 23048 (time 1778347526.73 total 258.82 last 158.00) - open/close/unlink 24524 (time 1778347536.73 total 268.82 last 147.58) - open/close/unlink 26079 (time 1778347546.73 total 278.83 last 155.46) - open/close/unlink 27819 (time 1778347556.74 total 288.83 last 173.86) - open/close/unlink 29527 (time 1778347566.74 total 298.83 last 170.78) - open/close/unlink 30000 (time 1778347569.68 total 301.77 last 161.09) - open/close/unlink 31649 (time 1778347579.68 total 311.77 last 164.89) - open/close/unlink 33404 (time 1778347589.68 total 321.78 last 175.43) - open/close/unlink 35254 (time 1778347599.69 total 331.78 last 184.92) - open/close/unlink 37133 (time 1778347609.70 total 341.79 last 187.75) - open/close/unlink 38955 (time 1778347619.70 total 351.79 last 182.15) - open/close/unlink 40000 (time 1778347625.94 total 358.03 last 167.50) - open/close/unlink 41715 (time 1778347635.94 total 368.03 last 171.47) - open/close/unlink 43371 (time 1778347645.94 total 378.04 last 165.53) - open/close/unlink 44947 (time 1778347655.95 total 388.05 last 157.43) - open/close/unlink 46592 (time 1778347665.96 total 398.05 last 164.48) - open/close/unlink 48240 (time 1778347675.96 total 408.05 last 164.69) - open/close/unlink 49782 (time 1778347685.97 total 418.06 last 154.13) - open/close/unlink 51297 (time 1778347695.97 total 428.06 last 151.44) - open/close/unlink 52846 (time 1778347705.97 total 438.07 last 154.84) - open/close/unlink 54530 (time 1778347715.98 total 448.07 last 168.37) - open/close/unlink 56167 (time 1778347725.98 total 458.07 last 163.66) - open/close/unlink 57816 (time 1778347735.98 total 468.07 last 164.85) - open/close/unlink 59382 (time 1778347745.99 total 478.08 last 156.55) - open/close/unlink 60000 (time 1778347749.97 total 482.07 last 154.94) - open/close/unlink 61588 (time 1778347759.98 total 492.07 last 158.78) - open/close/unlink 63180 (time 1778347769.98 total 502.07 last 159.17) total: 64768 open/close/unlink in 512.04 seconds: 126.49 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg329-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg329-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg329-server stop mds service on oleg329-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg329-server unloading modules via unload_modules_local on: 'oleg329-server' oleg329-server: oleg329-server.virtnet: executing unload_modules_local oleg329-server: modules unloaded. pdsh@oleg329-client: oleg329-client: ssh exited with exit code 2 pdsh@oleg329-client: oleg329-server: ssh exited with exit code 2 pdsh@oleg329-client: oleg329-client: ssh exited with exit code 2 pdsh@oleg329-client: oleg329-server: ssh exited with exit code 2