== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 10:48:36 (1786114116) start mds service on oleg405-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg405-server: oleg405-server.virtnet: executing set_default_debug -1 all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg405-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg405-server: oleg405-server.virtnet: executing set_default_debug -1 all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Started lustre-OST0000 oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg405-client.virtnet: -o user_xattr,flock 192.168.204.105@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 92553 329 92224 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 12329 425 11904 4% /mnt/lustre[OST:0] filesystem_summary: 12233 329 11904 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close/unlink 381 (time 1786114167.79 total 10.02 last 38.04) - open/close/unlink 707 (time 1786114177.80 total 20.03 last 32.56) - open/close/unlink 1033 (time 1786114187.82 total 30.05 last 32.53) - open/close/unlink 1389 (time 1786114197.83 total 40.06 last 35.58) - open/close/unlink 1697 (time 1786114207.83 total 50.06 last 30.78) - open/close/unlink 2060 (time 1786114217.85 total 60.08 last 36.23) - open/close/unlink 2562 (time 1786114227.86 total 70.09 last 50.14) - open/close/unlink 2887 (time 1786114237.93 total 80.17 last 32.27) - open/close/unlink 3207 (time 1786114247.96 total 90.19 last 31.92) - open/close/unlink 3510 (time 1786114257.96 total 100.20 last 30.29) - open/close/unlink 3857 (time 1786114267.96 total 110.20 last 34.70) - open/close/unlink 4251 (time 1786114277.98 total 120.21 last 39.33) - open/close/unlink 4639 (time 1786114287.99 total 130.22 last 38.77) - open/close/unlink 5034 (time 1786114297.99 total 140.22 last 39.49) - open/close/unlink 5485 (time 1786114308.02 total 150.25 last 44.99) - open/close/unlink 5923 (time 1786114318.04 total 160.27 last 43.70) - open/close/unlink 6346 (time 1786114328.05 total 170.28 last 42.27) - open/close/unlink 6794 (time 1786114338.05 total 180.28 last 44.79) - open/close/unlink 7254 (time 1786114348.06 total 190.29 last 45.93) - open/close/unlink 7749 (time 1786114358.07 total 200.31 last 49.45) - open/close/unlink 8181 (time 1786114368.08 total 210.31 last 43.17) - open/close/unlink 8569 (time 1786114378.09 total 220.32 last 38.76) - open/close/unlink 9013 (time 1786114388.09 total 230.33 last 44.39) - open/close/unlink 9541 (time 1786114398.11 total 240.34 last 52.74) - open/close/unlink 9963 (time 1786114408.11 total 250.34 last 42.19) - open/close/unlink 10366 (time 1786114418.13 total 260.36 last 40.22) - open/close/unlink 10839 (time 1786114428.14 total 270.37 last 47.25) - open/close/unlink 11344 (time 1786114438.15 total 280.38 last 50.44) - open/close/unlink 11842 (time 1786114448.16 total 290.39 last 49.77) - open/close/unlink 12279 (time 1786114458.16 total 300.40 last 43.67) - open/close/unlink 12779 (time 1786114468.17 total 310.41 last 49.95) - open/close/unlink 13250 (time 1786114478.18 total 320.41 last 47.07) - open/close/unlink 13670 (time 1786114488.20 total 330.43 last 41.91) - open/close/unlink 14120 (time 1786114498.22 total 340.45 last 44.92) - open/close/unlink 14641 (time 1786114508.23 total 350.46 last 52.04) - open/close/unlink 15094 (time 1786114518.23 total 360.47 last 45.27) - open/close/unlink 15536 (time 1786114528.24 total 370.47 last 44.17) - open/close/unlink 16036 (time 1786114538.24 total 380.47 last 50.00) - open/close/unlink 16473 (time 1786114548.26 total 390.49 last 43.64) - open/close/unlink 16858 (time 1786114558.29 total 400.52 last 38.38) - open/close/unlink 17239 (time 1786114568.32 total 410.55 last 37.99) - open/close/unlink 17585 (time 1786114578.32 total 420.55 last 34.60) - open/close/unlink 17952 (time 1786114588.33 total 430.56 last 36.65) - open/close/unlink 18380 (time 1786114598.33 total 440.56 last 42.78) - open/close/unlink 18844 (time 1786114608.33 total 450.57 last 46.40) - open/close/unlink 19320 (time 1786114618.35 total 460.58 last 47.54) - open/close/unlink 19780 (time 1786114628.36 total 470.59 last 45.95) - open/close/unlink 20000 (time 1786114633.75 total 475.98 last 40.79) - open/close/unlink 20455 (time 1786114643.77 total 486.00 last 45.44) - open/close/unlink 20879 (time 1786114653.77 total 496.00 last 42.37) - open/close/unlink 21398 (time 1786114663.78 total 506.01 last 51.86) - open/close/unlink 21999 (time 1786114673.78 total 516.01 last 60.08) - open/close/unlink 22608 (time 1786114683.79 total 526.02 last 60.88) - open/close/unlink 23152 (time 1786114693.80 total 536.03 last 54.33) - open/close/unlink 23762 (time 1786114703.80 total 546.03 last 61.00) - open/close/unlink 24339 (time 1786114713.80 total 556.03 last 57.68) - open/close/unlink 24957 (time 1786114723.81 total 566.04 last 61.78) - open/close/unlink 25564 ( - open/close/unlink 377 (time 1786114167.46 total 10.01 last 37.64) - open/close/unlink 706 (time 1786114177.48 total 20.03 last 32.84) - open/close/unlink 1029 (time 1786114187.50 total 30.05 last 32.25) - open/close/unlink 1383 (time 1786114197.52 total 40.07 last 35.33) - open/close/unlink 1696 (time 1786114207.54 total 50.09 last 31.24) - open/close/unlink 2065 (time 1786114217.54 total 60.09 last 36.89) - open/close/unlink 2567 (time 1786114227.56 total 70.12 last 50.08) - open/close/unlink 2893 (time 1786114237.57 total 80.12 last 32.57) - open/close/unlink 3218 (time 1786114247.60 total 90.16 last 32.39) - open/close/unlink 3514 (time 1786114257.61 total 100.16 last 29.58) - open/close/unlink 3859 (time 1786114267.61 total 110.17 last 34.49) - open/close/unlink 4246 (time 1786114277.62 total 120.17 last 38.69) - open/close/unlink 4632 (time 1786114287.62 total 130.17 last 38.58) - open/close/unlink 5019 (time 1786114297.62 total 140.17 last 38.70) - open/close/unlink 5475 (time 1786114307.62 total 150.18 last 45.59) - open/close/unlink 5918 (time 1786114317.63 total 160.18 last 44.27) - open/close/unlink 6340 (time 1786114327.64 total 170.20 last 42.13) - open/close/unlink 6784 (time 1786114337.65 total 180.21 last 44.36) - open/close/unlink 7247 (time 1786114347.67 total 190.23 last 46.21) - open/close/unlink 7753 (time 1786114357.70 total 200.26 last 50.45) - open/close/unlink 8180 (time 1786114367.71 total 210.26 last 42.68) - open/close/unlink 8573 (time 1786114377.71 total 220.27 last 39.28) - open/close/unlink 9012 (time 1786114387.73 total 230.28 last 43.84) - open/close/unlink 9539 (time 1786114397.75 total 240.31 last 52.57) - open/close/unlink 9959 (time 1786114407.76 total 250.32 last 41.96) - open/close/unlink 10355 (time 1786114417.78 total 260.33 last 39.54) - open/close/unlink 10821 (time 1786114427.78 total 270.34 last 46.58) - open/close/unlink 11341 (time 1786114437.80 total 280.35 last 51.91) - open/close/unlink 11844 (time 1786114447.83 total 290.39 last 50.13) - open/close/unlink 12282 (time 1786114457.84 total 300.40 last 43.77) - open/close/unlink 12778 (time 1786114467.85 total 310.40 last 49.58) - open/close/unlink 13255 (time 1786114477.85 total 320.40 last 47.68) - open/close/unlink 13676 (time 1786114487.86 total 330.42 last 42.04) - open/close/unlink 14117 (time 1786114497.87 total 340.42 last 44.09) - open/close/unlink 14635 (time 1786114507.87 total 350.42 last 51.79) - open/close/unlink 15093 (time 1786114517.89 total 360.45 last 45.68) - open/close/unlink 15531 (time 1786114527.89 total 370.45 last 43.79) - open/close/unlink 16041 (time 1786114537.92 total 380.47 last 50.88) - open/close/unlink 16475 (time 1786114547.92 total 390.48 last 43.37) - open/close/unlink 16863 (time 1786114557.93 total 400.48 last 38.78) - open/close/unlink 17236 (time 1786114567.95 total 410.50 last 37.23) - open/close/unlink 17575 (time 1786114577.96 total 420.51 last 33.86) - open/close/unlink 17941 (time 1786114587.96 total 430.51 last 36.60) - open/close/unlink 18366 (time 1786114597.98 total 440.54 last 42.40) - open/close/unlink 18829 (time 1786114607.99 total 450.54 last 46.28) - open/close/unlink 19293 (time 1786114618.00 total 460.55 last 46.36) - open/close/unlink 19752 (time 1786114628.00 total 470.56 last 45.87) - open/close/unlink 20000 (time 1786114633.96 total 476.52 last 41.61) - open/close/unlink 20454 (time 1786114643.97 total 486.53 last 45.35) - open/close/unlink 20877 (time 1786114653.98 total 496.53 last 42.27) - open/close/unlink 21400 (time 1786114663.99 total 506.54 last 52.27) - open/close/unlink 22008 (time 1786114673.99 total 516.55 last 60.76) - open/close/unlink 22615 (time 1786114684.00 total 526.55 last 60.67) - open/close/unlink 23158 (time 1786114694.01 total 536.57 last 54.22) - open/close/unlink 23766 (time 1786114704.03 total 546.59 last 60.68) - open/close/unlink 24357 (time 1786114714.06 total 556.61 last 58.96) - open/close/unlink 24981 (time 1786114724.06 total 566.61 last 62.39) - open/close/unlink 25589 ( - open/close/unlink 374 (time 1786114167.19 total 10.01 last 37.35) - open/close/unlink 705 (time 1786114177.21 total 20.04 last 33.03) - open/close/unlink 1019 (time 1786114187.21 total 30.04 last 31.39) - open/close/unlink 1377 (time 1786114197.24 total 40.07 last 35.71) - open/close/unlink 1687 (time 1786114207.24 total 50.07 last 30.98) - open/close/unlink 2062 (time 1786114217.26 total 60.09 last 37.44) - open/close/unlink 2555 (time 1786114227.27 total 70.10 last 49.23) - open/close/unlink 2880 (time 1786114237.29 total 80.11 last 32.46) - open/close/unlink 3197 (time 1786114247.31 total 90.14 last 31.64) - open/close/unlink 3495 (time 1786114257.31 total 100.14 last 29.78) - open/close/unlink 3847 (time 1786114267.32 total 110.15 last 35.17) - open/close/unlink 4237 (time 1786114277.34 total 120.17 last 38.93) - open/close/unlink 4628 (time 1786114287.35 total 130.18 last 39.07) - open/close/unlink 5012 (time 1786114297.35 total 140.18 last 38.38) - open/close/unlink 5468 (time 1786114307.36 total 150.19 last 45.58) - open/close/unlink 5904 (time 1786114317.36 total 160.19 last 43.58) - open/close/unlink 6335 (time 1786114327.36 total 170.19 last 43.09) - open/close/unlink 6776 (time 1786114337.38 total 180.21 last 44.05) - open/close/unlink 7240 (time 1786114347.38 total 190.21 last 46.38) - open/close/unlink 7741 (time 1786114357.39 total 200.22 last 50.07) - open/close/unlink 8166 (time 1786114367.39 total 210.22 last 42.47) - open/close/unlink 8558 (time 1786114377.41 total 220.24 last 39.14) - open/close/unlink 8996 (time 1786114387.42 total 230.25 last 43.76) - open/close/unlink 9523 (time 1786114397.43 total 240.26 last 52.63) - open/close/unlink 9937 (time 1786114407.45 total 250.28 last 41.32) - open/close/unlink 10330 (time 1786114417.46 total 260.29 last 39.27) - open/close/unlink 10797 (time 1786114427.48 total 270.31 last 46.60) - open/close/unlink 11312 (time 1786114437.48 total 280.31 last 51.48) - open/close/unlink 11814 (time 1786114447.50 total 290.33 last 50.12) - open/close/unlink 12249 (time 1786114457.52 total 300.35 last 43.43) - open/close/unlink 12739 (time 1786114467.53 total 310.36 last 48.92) - open/close/unlink 13215 (time 1786114477.54 total 320.37 last 47.56) - open/close/unlink 13642 (time 1786114487.56 total 330.39 last 42.60) - open/close/unlink 14081 (time 1786114497.57 total 340.40 last 43.88) - open/close/unlink 14596 (time 1786114507.58 total 350.41 last 51.43) - open/close/unlink 15060 (time 1786114517.61 total 360.44 last 46.28) - open/close/unlink 15493 (time 1786114527.62 total 370.45 last 43.26) - open/close/unlink 15998 (time 1786114537.63 total 380.45 last 50.45) - open/close/unlink 16433 (time 1786114547.64 total 390.47 last 43.43) - open/close/unlink 16824 (time 1786114557.68 total 400.51 last 38.96) - open/close/unlink 17199 (time 1786114567.69 total 410.52 last 37.47) - open/close/unlink 17539 (time 1786114577.70 total 420.53 last 33.94) - open/close/unlink 17902 (time 1786114587.72 total 430.55 last 36.24) - open/close/unlink 18330 (time 1786114597.74 total 440.57 last 42.72) - open/close/unlink 18797 (time 1786114607.76 total 450.58 last 46.62) - open/close/unlink 19265 (time 1786114617.76 total 460.58 last 46.80) - open/close/unlink 19727 (time 1786114627.78 total 470.61 last 46.09) - open/close/unlink 20000 (time 1786114634.34 total 477.17 last 41.63) - open/close/unlink 20454 (time 1786114644.35 total 487.18 last 45.33) - open/close/unlink 20878 (time 1786114654.37 total 497.20 last 42.32) - open/close/unlink 21398 (time 1786114664.39 total 507.22 last 51.93) - open/close/unlink 22013 (time 1786114674.39 total 517.22 last 61.46) - open/close/unlink 22613 (time 1786114684.40 total 527.23 last 59.96) - open/close/unlink 23148 (time 1786114694.43 total 537.25 last 53.36) - open/close/unlink 23766 (time 1786114704.44 total 547.27 last 61.72) - open/close/unlink 24351 (time 1786114714.44 total 557.27 last 58.49) - open/close/unlink 24981 (time 1786114724.45 total 567.28 last 62.95) - open/close/unlink 25587 ( - open/close/unlink 387 (time 1786114167.81 total 10.00 last 38.69) - open/close/unlink 711 (time 1786114177.82 total 20.01 last 32.38) - open/close/unlink 1035 (time 1786114187.83 total 30.02 last 32.37) - open/close/unlink 1378 (time 1786114197.83 total 40.02 last 34.28) - open/close/unlink 1689 (time 1786114207.83 total 50.03 last 31.10) - open/close/unlink 2057 (time 1786114217.85 total 60.04 last 36.76) - open/close/unlink 2558 (time 1786114227.86 total 70.05 last 50.03) - open/close/unlink 2875 (time 1786114237.87 total 80.06 last 31.67) - open/close/unlink 3196 (time 1786114247.90 total 90.09 last 32.00) - open/close/unlink 3499 (time 1786114257.91 total 100.10 last 30.28) - open/close/unlink 3851 (time 1786114267.92 total 110.11 last 35.15) - open/close/unlink 4244 (time 1786114277.93 total 120.13 last 39.25) - open/close/unlink 4633 (time 1786114287.94 total 130.13 last 38.88) - open/close/unlink 5025 (time 1786114297.95 total 140.14 last 39.17) - open/close/unlink 5479 (time 1786114307.95 total 150.14 last 45.39) - open/close/unlink 5924 (time 1786114317.96 total 160.15 last 44.45) - open/close/unlink 6343 (time 1786114327.98 total 170.17 last 41.83) - open/close/unlink 6797 (time 1786114337.99 total 180.18 last 45.35) - open/close/unlink 7262 (time 1786114348.01 total 190.20 last 46.43) - open/close/unlink 7767 (time 1786114358.03 total 200.22 last 50.36) - open/close/unlink 8195 (time 1786114368.04 total 210.23 last 42.77) - open/close/unlink 8583 (time 1786114378.05 total 220.25 last 38.75) - open/close/unlink 9027 (time 1786114388.06 total 230.25 last 44.38) - open/close/unlink 9551 (time 1786114398.08 total 240.27 last 52.31) - open/close/unlink 9967 (time 1786114408.10 total 250.29 last 41.52) - open/close/unlink 10366 (time 1786114418.11 total 260.30 last 39.85) - open/close/unlink 10832 (time 1786114428.12 total 270.31 last 46.56) - open/close/unlink 11335 (time 1786114438.12 total 280.31 last 50.30) - open/close/unlink 11833 (time 1786114448.14 total 290.33 last 49.68) - open/close/unlink 12267 (time 1786114458.16 total 300.35 last 43.33) - open/close/unlink 12767 (time 1786114468.16 total 310.36 last 49.96) - open/close/unlink 13242 (time 1786114478.17 total 320.36 last 47.46) - open/close/unlink 13658 (time 1786114488.18 total 330.37 last 41.57) - open/close/unlink 14104 (time 1786114498.18 total 340.37 last 44.60) - open/close/unlink 14617 (time 1786114508.18 total 350.37 last 51.30) - open/close/unlink 15065 (time 1786114518.20 total 360.39 last 44.72) - open/close/unlink 15510 (time 1786114528.20 total 370.39 last 44.47) - open/close/unlink 16010 (time 1786114538.20 total 380.39 last 50.00) - open/close/unlink 16444 (time 1786114548.21 total 390.40 last 43.37) - open/close/unlink 16831 (time 1786114558.21 total 400.40 last 38.70) - open/close/unlink 17206 (time 1786114568.22 total 410.41 last 37.47) - open/close/unlink 17547 (time 1786114578.23 total 420.42 last 34.07) - open/close/unlink 17917 (time 1786114588.25 total 430.45 last 36.91) - open/close/unlink 18342 (time 1786114598.27 total 440.46 last 42.43) - open/close/unlink 18800 (time 1786114608.28 total 450.47 last 45.75) - open/close/unlink 19272 (time 1786114618.29 total 460.48 last 47.17) - open/close/unlink 19727 (time 1786114628.31 total 470.50 last 45.42) - open/close/unlink 20000 (time 1786114634.82 total 477.01 last 41.93) - open/close/unlink 20458 (time 1786114644.82 total 487.01 last 45.79) - open/close/unlink 20875 (time 1786114654.83 total 497.02 last 41.65) - open/close/unlink 21401 (time 1786114664.84 total 507.04 last 52.53) - open/close/unlink 22025 (time 1786114674.85 total 517.04 last 62.37) - open/close/unlink 22631 (time 1786114684.85 total 527.04 last 60.58) - open/close/unlink 23150 (time 1786114694.87 total 537.06 last 51.83) - open/close/unlink 23774 (time 1786114704.88 total 547.07 last 62.31) - open/close/unlink 24357 (time 1786114714.89 total 557.08 last 58.27) - open/close/unlink 24994 (time 1786114724.89 total 567.08 last 63.70) - open/close/unlink 25601 ( - open/close/unlink 381 (time 1786114167.71 total 10.02 last 38.04) - open/close/unlink 704 (time 1786114177.71 total 20.02 last 32.30) - open/close/unlink 1024 (time 1786114187.72 total 30.03 last 31.96) - open/close/unlink 1380 (time 1786114197.73 total 40.03 last 35.59) - open/close/unlink 1687 (time 1786114207.75 total 50.06 last 30.63) - open/close/unlink 2048 (time 1786114217.75 total 60.06 last 36.08) - open/close/unlink 2548 (time 1786114227.77 total 70.08 last 49.93) - open/close/unlink 2873 (time 1786114237.79 total 80.09 last 32.44) - open/close/unlink 3194 (time 1786114247.79 total 90.10 last 32.09) - open/close/unlink 3494 (time 1786114257.81 total 100.12 last 29.95) - open/close/unlink 3840 (time 1786114267.81 total 110.12 last 34.59) - open/close/unlink 4228 (time 1786114277.82 total 120.13 last 38.78) - open/close/unlink 4618 (time 1786114287.84 total 130.15 last 38.90) - open/close/unlink 5011 (time 1786114297.85 total 140.16 last 39.26) - open/close/unlink 5459 (time 1786114307.86 total 150.17 last 44.76) - open/close/unlink 5899 (time 1786114317.87 total 160.18 last 43.97) - open/close/unlink 6318 (time 1786114327.88 total 170.18 last 41.87) - open/close/unlink 6760 (time 1786114337.88 total 180.19 last 44.17) - open/close/unlink 7221 (time 1786114347.90 total 190.20 last 46.04) - open/close/unlink 7716 (time 1786114357.91 total 200.22 last 49.43) - open/close/unlink 8147 (time 1786114367.93 total 210.24 last 42.99) - open/close/unlink 8535 (time 1786114377.94 total 220.25 last 38.76) - open/close/unlink 8977 (time 1786114387.95 total 230.25 last 44.19) - open/close/unlink 9497 (time 1786114397.96 total 240.27 last 51.93) - open/close/unlink 9921 (time 1786114407.97 total 250.28 last 42.36) - open/close/unlink 10317 (time 1786114417.98 total 260.29 last 39.56) - open/close/unlink 10791 (time 1786114427.99 total 270.30 last 47.35) - open/close/unlink 11299 (time 1786114438.00 total 280.31 last 50.76) - open/close/unlink 11799 (time 1786114448.02 total 290.32 last 49.91) - open/close/unlink 12234 (time 1786114458.03 total 300.34 last 43.45) - open/close/unlink 12737 (time 1786114468.03 total 310.34 last 50.30) - open/close/unlink 13218 (time 1786114478.04 total 320.35 last 48.03) - open/close/unlink 13634 (time 1786114488.05 total 330.35 last 41.58) - open/close/unlink 14081 (time 1786114498.07 total 340.37 last 44.62) - open/close/unlink 14594 (time 1786114508.07 total 350.38 last 51.27) - open/close/unlink 15047 (time 1786114518.11 total 360.42 last 45.12) - open/close/unlink 15494 (time 1786114528.14 total 370.44 last 44.59) - open/close/unlink 15997 (time 1786114538.16 total 380.46 last 50.20) - open/close/unlink 16430 (time 1786114548.18 total 390.49 last 43.18) - open/close/unlink 16818 (time 1786114558.22 total 400.52 last 38.68) - open/close/unlink 17196 (time 1786114568.22 total 410.53 last 37.79) - open/close/unlink 17536 (time 1786114578.23 total 420.54 last 33.96) - open/close/unlink 17905 (time 1786114588.25 total 430.55 last 36.84) - open/close/unlink 18327 (time 1786114598.26 total 440.57 last 42.14) - open/close/unlink 18790 (time 1786114608.28 total 450.59 last 46.19) - open/close/unlink 19259 (time 1786114618.29 total 460.59 last 46.89) - open/close/unlink 19714 (time 1786114628.30 total 470.61 last 45.42) - open/close/unlink 20000 (time 1786114635.07 total 477.38 last 42.26) - open/close/unlink 20470 (time 1786114645.07 total 487.38 last 46.98) - open/close/unlink 20888 (time 1786114655.08 total 497.39 last 41.76) - open/close/unlink 21424 (time 1786114665.09 total 507.40 last 53.55) - open/close/unlink 22043 (time 1786114675.11 total 517.41 last 61.81) - open/close/unlink 22647 (time 1786114685.11 total 527.42 last 60.35) - open/close/unlink 23165 (time 1786114695.13 total 537.44 last 51.72) - open/close/unlink 23785 (time 1786114705.13 total 547.44 last 61.99) - open/close/unlink 24375 (time 1786114715.14 total 557.45 last 58.95) - open/close/unlink 25008 (time 1786114725.15 total 567.45 last 63.27) - open/close/unlink 25607 (time 1786114733.81 total 576.04 last 60.67) - open/close/unlink 26203 (time 1786114743.82 total 586.05 last 63.86) - open/close/unlink 26861 (time 1786114753.83 total 596.06 last 65.75) - open/close/unlink 27547 (time 1786114763.83 total 606.06 last 68.60) - open/close/unlink 28185 (time 1786114773.83 total 616.06 last 63.79) - open/close/unlink 28870 (time 1786114783.84 total 626.07 last 68.44) - open/close/unlink 29585 (time 1786114793.85 total 636.08 last 71.44) - open/close/unlink 30000 (time 1786114800.11 total 642.34 last 66.26) - open/close/unlink 30622 (time 1786114810.11 total 652.34 last 62.19) - open/close/unlink 31342 (time 1786114820.11 total 662.34 last 71.99) - open/close/unlink 32073 (time 1786114830.13 total 672.36 last 72.99) - open/close/unlink 32778 (time 1786114840.14 total 682.37 last 70.44) - open/close/unlink 33453 (time 1786114850.14 total 692.37 last 67.48) - open/close/unlink 34146 (time 1786114860.14 total 702.37 last 69.29) - open/close/unlink 34764 (time 1786114870.15 total 712.38 last 61.77) - open/close/unlink 35479 (time 1786114880.16 total 722.39 last 71.43) - open/close/unlink 36178 (time 1786114890.16 total 732.39 last 69.87) - open/close/unlink 36880 (time 1786114900.16 total 742.39 last 70.18) - open/close/unlink 37666 (time 1786114910.17 total 752.40 last 78.56) - open/close/unlink 38376 (time 1786114920.17 total 762.41 last 70.94) - open/close/unlink 39094 (time 1786114930.18 total 772.41 last 71.75) - open/close/unlink 39845 (time 1786114940.19 total 782.42 last 75.05) - open/close/unlink 40000 (time 1786114942.20 total 784.43 last 76.93) - open/close/unlink 40799 (time 1786114952.21 total 794.44 last 79.82) - open/close/unlink 41563 (time 1786114962.22 total 804.45 last 76.33) - open/close/unlink 42403 (time 1786114972.23 total 814.46 last 83.92) - open/close/unlink 43246 (time 1786114982.25 total 824.48 last 84.16) - open/close/unlink 44079 (time 1786114992.26 total 834.49 last 83.24) - open/close/unlink 44819 (time 1786115002.26 total 844.49 last 73.96) - open/close/unlink 45553 (time 1786115012.27 total 854.50 last 73.37) - open/close/unlink 46262 (time 1786115022.27 total 864.50 last 70.84) - open/close/unlink 46981 (time 1786115032.29 total 874.52 last 71.80) - open/close/unlink 47712 (time 1786115042.30 total 884.53 last 73.03) - open/close/unlink 48662 (time 1786115052.30 total 894.53 last 94.96) - open/close/unlink 49537 (time 1786115062.30 total 904.53 last 87.50) - open/close/unlink 50000 (time 1786115068.75 total 910.98 last 71.82) - open/close/unlink 50828 (time 1786115078.76 total 920.99 last 82.71) - open/close/unlink 51649 (time 1786115088.77 total 931.00 last 81.98) - open/close/unlink 52363 (time 1786115098.78 total 941.01 last 71.38) - open/close/unlink 53076 (time 1786115108.78 total 951.01 last 71.26) - open/close/unlink 53811 (time 1786115118.78 total 961.01 last 73.50) - open/close/unlink 54634 (time 1786115128.79 total 971.02 last 82.25) - open/close/unlink 55465 (time 1786115138.79 total 981.03 last 83.05) - open/close/unlink 56224 (time 1786115148.80 total 991.03 last 75.88) - open/close/unlink 57005 (time 1786115158.80 total 1001.03 last 78.10) - open/close/unlink 57834 (time 1786115168.81 total 1011.04 last 82.82) - open/close/unlink 58640 (time 1786115178.81 total 1021.04 last 80.55) - open/close/unlink 59473 (time 1786115188.81 total 1031.05 last 83.28) - open/close/unlink 60000 (time 1786115195.00 total 1037.23 last 85.21) - open/close/unlink 60910 (time 1786115205.00 total 1047.24 last 90.96) - open/close/unlink 62009 (time 1786115215.00 total 1057.24 last 109.90) - open/close/unlink 63246 (time 1786115225.01 total 1067.24 last 123.65) - open/close/unlink 64485 (time 1786115235.01 total 1077.24 last 123.89) total: 64768 open/close/unlink in 1079.87 seconds: 59.98 ops/second time 1786114734.90 total 577.09 last 60.65) - open/close/unlink 26255 (time 1786114744.91 total 587.10 last 65.29) - open/close/unlink 26915 (time 1786114754.92 total 597.11 last 65.95) - open/close/unlink 27602 (time 1786114764.93 total 607.12 last 68.64) - open/close/unlink 28244 (time 1786114774.94 total 617.13 last 64.14) - open/close/unlink 28952 (time 1786114784.95 total 627.14 last 70.71) - open/close/unlink 29659 (time 1786114794.96 total 637.15 last 70.66) - open/close/unlink 30000 (time 1786114800.15 total 642.34 last 65.65) - open/close/unlink 30612 (time 1786114810.15 total 652.34 last 61.18) - open/close/unlink 31331 (time 1786114820.16 total 662.35 last 71.82) - open/close/unlink 32058 (time 1786114830.17 total 672.36 last 72.66) - open/close/unlink 32765 (time 1786114840.18 total 682.37 last 70.65) - open/close/unlink 33436 (time 1786114850.18 total 692.38 last 67.04) - open/close/unlink 34143 (time 1786114860.19 total 702.38 last 70.69) - open/close/unlink 34762 (time 1786114870.19 total 712.38 last 61.89) - open/close/unlink 35473 (time 1786114880.19 total 722.38 last 71.09) - open/close/unlink 36170 (time 1786114890.19 total 732.38 last 69.69) - open/close/unlink 36867 (time 1786114900.19 total 742.38 last 69.70) - open/close/unlink 37648 (time 1786114910.20 total 752.39 last 78.05) - open/close/unlink 38359 (time 1786114920.20 total 762.39 last 71.10) - open/close/unlink 39074 (time 1786114930.21 total 772.40 last 71.43) - open/close/unlink 39821 (time 1786114940.21 total 782.41 last 74.65) - open/close/unlink 40000 (time 1786114942.59 total 784.78 last 75.45) - open/close/unlink 40802 (time 1786114952.59 total 794.78 last 80.18) - open/close/unlink 41563 (time 1786114962.61 total 804.80 last 75.97) - open/close/unlink 42399 (time 1786114972.61 total 814.80 last 83.60) - open/close/unlink 43247 (time 1786114982.62 total 824.81 last 84.71) - open/close/unlink 44087 (time 1786114992.62 total 834.81 last 83.97) - open/close/unlink 44827 (time 1786115002.62 total 844.81 last 73.99) - open/close/unlink 45560 (time 1786115012.62 total 854.81 last 73.29) - open/close/unlink 46279 (time 1786115022.63 total 864.82 last 71.86) - open/close/unlink 46989 (time 1786115032.64 total 874.83 last 70.92) - open/close/unlink 47720 (time 1786115042.64 total 884.84 last 73.06) - open/close/unlink 48657 (time 1786115052.65 total 894.84 last 93.63) - open/close/unlink 49528 (time 1786115062.66 total 904.85 last 87.04) - open/close/unlink 50000 (time 1786115069.30 total 911.49 last 71.12) - open/close/unlink 50842 (time 1786115079.30 total 921.49 last 84.13) - open/close/unlink 51644 (time 1786115089.31 total 931.50 last 80.12) - open/close/unlink 52350 (time 1786115099.32 total 941.51 last 70.56) - open/close/unlink 53070 (time 1786115109.33 total 951.52 last 71.94) - open/close/unlink 53808 (time 1786115119.33 total 961.52 last 73.78) - open/close/unlink 54629 (time 1786115129.33 total 971.53 last 82.07) - open/close/unlink 55448 (time 1786115139.35 total 981.54 last 81.78) - open/close/unlink 56203 (time 1786115149.35 total 991.54 last 75.47) - open/close/unlink 57003 (time 1786115159.36 total 1001.55 last 79.92) - open/close/unlink 57820 (time 1786115169.37 total 1011.56 last 81.64) - open/close/unlink 58623 (time 1786115179.37 total 1021.57 last 80.25) - open/close/unlink 59476 (time 1786115189.38 total 1031.57 last 85.27) - open/close/unlink 60000 (time 1786115195.99 total 1038.18 last 79.25) - open/close/unlink 60945 (time 1786115206.00 total 1048.19 last 94.45) - open/close/unlink 62075 (time 1786115216.00 total 1058.19 last 113.00) - open/close/unlink 63330 (time 1786115226.02 total 1068.21 last 125.26) - open/close/unlink 64566 (time 1786115236.02 total 1078.21 last 123.59) total: 64768 open/close/unlink in 1080.44 seconds: 59.95 ops/second time 1786114734.06 total 576.62 last 60.77) - open/close/unlink 26229 (time 1786114744.07 total 586.62 last 63.96) - open/close/unlink 26883 (time 1786114754.07 total 596.63 last 65.37) - open/close/unlink 27574 (time 1786114764.08 total 606.63 last 69.09) - open/close/unlink 28205 (time 1786114774.08 total 616.63 last 63.07) - open/close/unlink 28901 (time 1786114784.08 total 626.63 last 69.59) - open/close/unlink 29606 (time 1786114794.09 total 636.64 last 70.44) - open/close/unlink 30000 (time 1786114800.11 total 642.67 last 65.42) - open/close/unlink 30629 (time 1786114810.12 total 652.67 last 62.89) - open/close/unlink 31350 (time 1786114820.12 total 662.67 last 72.07) - open/close/unlink 32086 (time 1786114830.12 total 672.68 last 73.57) - open/close/unlink 32797 (time 1786114840.12 total 682.68 last 71.09) - open/close/unlink 33467 (time 1786114850.13 total 692.68 last 66.96) - open/close/unlink 34166 (time 1786114860.14 total 702.69 last 69.85) - open/close/unlink 34787 (time 1786114870.14 total 712.70 last 62.06) - open/close/unlink 35499 (time 1786114880.15 total 722.70 last 71.15) - open/close/unlink 36194 (time 1786114890.16 total 732.71 last 69.44) - open/close/unlink 36900 (time 1786114900.16 total 742.71 last 70.59) - open/close/unlink 37683 (time 1786114910.17 total 752.72 last 78.24) - open/close/unlink 38395 (time 1786114920.17 total 762.72 last 71.19) - open/close/unlink 39110 (time 1786114930.18 total 772.74 last 71.40) - open/close/unlink 39857 (time 1786114940.19 total 782.74 last 74.67) - open/close/unlink 40647 (time 1786114950.19 total 792.74 last 78.99) - open/close/unlink 41434 (time 1786114960.21 total 802.76 last 78.56) - open/close/unlink 42241 (time 1786114970.22 total 812.77 last 80.62) - open/close/unlink 43092 (time 1786114980.22 total 822.77 last 85.07) - open/close/unlink 43944 (time 1786114990.23 total 832.78 last 85.12) - open/close/unlink 44682 (time 1786115000.24 total 842.79 last 73.72) - open/close/unlink 45418 (time 1786115010.25 total 852.80 last 73.52) - open/close/unlink 46141 (time 1786115020.25 total 862.81 last 72.25) - open/close/unlink 46849 (time 1786115030.26 total 872.82 last 70.73) - open/close/unlink 47554 (time 1786115040.27 total 882.82 last 70.47) - open/close/unlink 48469 (time 1786115050.28 total 892.83 last 91.43) - open/close/unlink 49357 (time 1786115060.28 total 902.83 last 88.77) - open/close/unlink 50000 (time 1786115069.16 total 911.71 last 72.40) - open/close/unlink 50836 (time 1786115079.16 total 921.72 last 83.58) - open/close/unlink 51648 (time 1786115089.16 total 931.72 last 81.18) - open/close/unlink 52353 (time 1786115099.18 total 941.73 last 70.39) - open/close/unlink 53073 (time 1786115109.19 total 951.75 last 71.91) - open/close/unlink 53806 (time 1786115119.20 total 961.75 last 73.27) - open/close/unlink 54623 (time 1786115129.21 total 971.76 last 81.63) - open/close/unlink 55458 (time 1786115139.21 total 981.76 last 83.48) - open/close/unlink 56212 (time 1786115149.21 total 991.76 last 75.38) - open/close/unlink 56997 (time 1786115159.22 total 1001.77 last 78.43) - open/close/unlink 57814 (time 1786115169.23 total 1011.78 last 81.61) - open/close/unlink 58612 (time 1786115179.23 total 1021.79 last 79.79) - open/close/unlink 59458 (time 1786115189.24 total 1031.79 last 84.53) - open/close/unlink 60000 (time 1786115196.09 total 1038.64 last 79.19) - open/close/unlink 60950 (time 1786115206.09 total 1048.64 last 94.95) - open/close/unlink 62068 (time 1786115216.10 total 1058.65 last 111.75) - open/close/unlink 63311 (time 1786115226.11 total 1068.66 last 124.16) - open/close/unlink 64551 (time 1786115236.11 total 1078.66 last 124.00) total: 64768 open/close/unlink in 1081.31 seconds: 59.90 ops/second time 1786114734.45 total 577.28 last 60.60) - open/close/unlink 26223 (time 1786114744.45 total 587.28 last 63.59) - open/close/unlink 26877 (time 1786114754.45 total 597.28 last 65.38) - open/close/unlink 27569 (time 1786114764.46 total 607.29 last 69.13) - open/close/unlink 28206 (time 1786114774.48 total 617.31 last 63.59) - open/close/unlink 28913 (time 1786114784.49 total 627.32 last 70.63) - open/close/unlink 29627 (time 1786114794.49 total 637.32 last 71.37) - open/close/unlink 30000 (time 1786114800.11 total 642.94 last 66.45) - open/close/unlink 30622 (time 1786114810.11 total 652.94 last 62.15) - open/close/unlink 31345 (time 1786114820.12 total 662.95 last 72.25) - open/close/unlink 32075 (time 1786114830.12 total 672.95 last 72.99) - open/close/unlink 32779 (time 1786114840.12 total 682.95 last 70.39) - open/close/unlink 33460 (time 1786114850.13 total 692.96 last 68.06) - open/close/unlink 34162 (time 1786114860.14 total 702.97 last 70.13) - open/close/unlink 34777 (time 1786114870.15 total 712.98 last 61.45) - open/close/unlink 35487 (time 1786114880.15 total 722.98 last 70.98) - open/close/unlink 36182 (time 1786114890.16 total 732.99 last 69.43) - open/close/unlink 36880 (time 1786114900.17 total 743.00 last 69.71) - open/close/unlink 37662 (time 1786114910.18 total 753.01 last 78.14) - open/close/unlink 38370 (time 1786114920.19 total 763.02 last 70.73) - open/close/unlink 39082 (time 1786114930.19 total 773.02 last 71.17) - open/close/unlink 39826 (time 1786114940.21 total 783.03 last 74.32) - open/close/unlink 40000 (time 1786114942.48 total 785.31 last 76.40) - open/close/unlink 40796 (time 1786114952.49 total 795.32 last 79.57) - open/close/unlink 41557 (time 1786114962.49 total 805.32 last 76.05) - open/close/unlink 42388 (time 1786114972.50 total 815.33 last 83.02) - open/close/unlink 43231 (time 1786114982.51 total 825.34 last 84.26) - open/close/unlink 44064 (time 1786114992.51 total 835.34 last 83.26) - open/close/unlink 44799 (time 1786115002.53 total 845.35 last 73.42) - open/close/unlink 45540 (time 1786115012.53 total 855.36 last 74.03) - open/close/unlink 46263 (time 1786115022.54 total 865.37 last 72.24) - open/close/unlink 46979 (time 1786115032.56 total 875.38 last 71.51) - open/close/unlink 47712 (time 1786115042.56 total 885.39 last 73.29) - open/close/unlink 48652 (time 1786115052.56 total 895.39 last 93.96) - open/close/unlink 49522 (time 1786115062.56 total 905.39 last 86.99) - open/close/unlink 50000 (time 1786115069.27 total 912.10 last 71.22) - open/close/unlink 50845 (time 1786115079.28 total 922.11 last 84.45) - open/close/unlink 51647 (time 1786115089.29 total 932.11 last 80.15) - open/close/unlink 52353 (time 1786115099.30 total 942.13 last 70.47) - open/close/unlink 53072 (time 1786115109.30 total 952.13 last 71.89) - open/close/unlink 53806 (time 1786115119.31 total 962.14 last 73.34) - open/close/unlink 54626 (time 1786115129.32 total 972.15 last 81.97) - open/close/unlink 55456 (time 1786115139.33 total 982.16 last 82.87) - open/close/unlink 56207 (time 1786115149.34 total 992.17 last 75.00) - open/close/unlink 56996 (time 1786115159.36 total 1002.19 last 78.81) - open/close/unlink 57809 (time 1786115169.37 total 1012.19 last 81.22) - open/close/unlink 58601 (time 1786115179.37 total 1022.20 last 79.14) - open/close/unlink 59452 (time 1786115189.38 total 1032.21 last 85.06) - open/close/unlink 60000 (time 1786115196.24 total 1039.07 last 79.88) - open/close/unlink 60946 (time 1786115206.24 total 1049.07 last 94.58) - open/close/unlink 62064 (time 1786115216.25 total 1059.07 last 111.75) - open/close/unlink 63306 (time 1786115226.26 total 1069.09 last 124.03) - open/close/unlink 64553 (time 1786115236.27 total 1079.10 last 124.61) total: 64768 open/close/unlink in 1081.99 seconds: 59.86 ops/second time 1786114735.16 total 577.47 last 59.82) - open/close/unlink 26259 (time 1786114745.17 total 587.48 last 65.11) - open/close/unlink 26914 (time 1786114755.18 total 597.49 last 65.45) - open/close/unlink 27591 (time 1786114765.18 total 607.49 last 67.68) - open/close/unlink 28219 (time 1786114775.18 total 617.49 last 62.80) - open/close/unlink 28931 (time 1786114785.18 total 627.49 last 71.19) - open/close/unlink 29643 (time 1786114795.19 total 637.50 last 71.18) - open/close/unlink 30000 (time 1786114800.72 total 643.03 last 64.55) - open/close/unlink 30628 (time 1786114810.73 total 653.04 last 62.72) - open/close/unlink 31353 (time 1786114820.74 total 663.05 last 72.43) - open/close/unlink 32078 (time 1786114830.75 total 673.06 last 72.44) - open/close/unlink 32794 (time 1786114840.76 total 683.07 last 71.53) - open/close/unlink 33472 (time 1786114850.77 total 693.08 last 67.71) - open/close/unlink 34168 (time 1786114860.78 total 703.09 last 69.52) - open/close/unlink 34785 (time 1786114870.80 total 713.10 last 61.63) - open/close/unlink 35501 (time 1786114880.81 total 723.11 last 71.52) - open/close/unlink 36201 (time 1786114890.81 total 733.12 last 69.98) - open/close/unlink 36897 (time 1786114900.82 total 743.13 last 69.49) - open/close/unlink 37695 (time 1786114910.84 total 753.14 last 79.72) - open/close/unlink 38405 (time 1786114920.84 total 763.15 last 70.97) - open/close/unlink 39099 (time 1786114930.84 total 773.15 last 69.36) - open/close/unlink 39853 (time 1786114940.86 total 783.17 last 75.25) - open/close/unlink 40641 (time 1786114950.87 total 793.18 last 78.75) - open/close/unlink 41410 (time 1786114960.88 total 803.19 last 76.83) - open/close/unlink 42227 (time 1786114970.89 total 813.20 last 81.61) - open/close/unlink 43080 (time 1786114980.90 total 823.21 last 85.23) - open/close/unlink 43928 (time 1786114990.90 total 833.21 last 84.75) - open/close/unlink 44677 (time 1786115000.92 total 843.23 last 74.81) - open/close/unlink 45422 (time 1786115010.93 total 853.24 last 74.38) - open/close/unlink 46139 (time 1786115020.94 total 863.24 last 71.67) - open/close/unlink 46853 (time 1786115030.95 total 873.25 last 71.33) - open/close/unlink 47563 (time 1786115040.96 total 883.27 last 70.89) - open/close/unlink 48511 (time 1786115050.96 total 893.27 last 94.78) - open/close/unlink 49386 (time 1786115060.96 total 903.27 last 87.50) - open/close/unlink 50000 (time 1786115069.48 total 911.79 last 72.10) - open/close/unlink 50850 (time 1786115079.48 total 921.79 last 84.99) - open/close/unlink 51656 (time 1786115089.49 total 931.79 last 80.56) - open/close/unlink 52361 (time 1786115099.49 total 941.80 last 70.45) - open/close/unlink 53081 (time 1786115109.50 total 951.81 last 71.95) - open/close/unlink 53822 (time 1786115119.51 total 961.82 last 74.05) - open/close/unlink 54648 (time 1786115129.51 total 971.82 last 82.59) - open/close/unlink 55467 (time 1786115139.51 total 981.82 last 81.88) - open/close/unlink 56219 (time 1786115149.53 total 991.83 last 75.09) - open/close/unlink 57017 (time 1786115159.55 total 1001.85 last 79.63) - open/close/unlink 57841 (time 1786115169.55 total 1011.85 last 82.39) - open/close/unlink 58638 (time 1786115179.55 total 1021.86 last 79.68) - open/close/unlink 59485 (time 1786115189.56 total 1031.87 last 84.62) - open/close/unlink 60000 (time 1786115196.05 total 1038.36 last 79.32) - open/close/unlink 60935 (time 1786115206.06 total 1048.36 last 93.46) - open/close/unlink 62061 (time 1786115216.06 total 1058.37 last 112.58) - open/close/unlink 63295 (time 1786115226.06 total 1068.37 last 123.32) - open/close/unlink 64524 (time 1786115236.07 total 1078.37 last 122.89) total: 64768 open/close/unlink in 1081.64 seconds: 59.88 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete oleg405-server: cannot open 'lustre-ost2': no such pool pdsh@oleg405-client: oleg405-server: ssh exited with exit code 255 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg405-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg405-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg405-server stop mds service on oleg405-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg405-server unloading modules via unload_modules_local on: 'oleg405-server' oleg405-server: oleg405-server.virtnet: executing unload_modules_local oleg405-server: modules unloaded. pdsh@oleg405-client: oleg405-client: ssh exited with exit code 2 pdsh@oleg405-client: oleg405-server: ssh exited with exit code 2 pdsh@oleg405-client: oleg405-client: ssh exited with exit code 2 pdsh@oleg405-client: oleg405-server: ssh exited with exit code 2