== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 20:18:05 (1789517885) start mds service on oleg613-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg613-server: oleg613-server.virtnet: executing set_default_debug -1 all pdsh@oleg613-client: oleg613-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg613-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg613-server: oleg613-server.virtnet: executing set_default_debug -1 all pdsh@oleg613-client: oleg613-server: ssh exited with exit code 1 Started lustre-OST0000 oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg613-client.virtnet: -o user_xattr,flock 192.168.206.113@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 92553 329 92224 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 12297 393 11904 4% /mnt/lustre[OST:0] filesystem_summary: 12233 329 11904 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close/unlink 646 (time 1789517922.86 total 10.01 last 64.54) - open/close/unlink 1263 (time 1789517932.87 total 20.02 last 61.63) - open/close/unlink 1932 (time 1789517942.88 total 30.03 last 66.85) - open/close/unlink 2490 (time 1789517952.88 total 40.03 last 55.79) - open/close/unlink 2947 (time 1789517962.90 total 50.04 last 45.63) - open/close/unlink 3444 (time 1789517972.91 total 60.06 last 49.63) - open/close/unlink 4013 (time 1789517982.91 total 70.06 last 56.89) - open/close/unlink 4527 (time 1789517992.92 total 80.07 last 51.36) - open/close/unlink 5074 (time 1789518002.94 total 90.09 last 54.59) - open/close/unlink 5553 (time 1789518012.94 total 100.09 last 47.89) - open/close/unlink 6137 (time 1789518022.96 total 110.11 last 58.33) - open/close/unlink 6715 (time 1789518032.96 total 120.11 last 57.76) - open/close/unlink 7313 (time 1789518042.97 total 130.12 last 59.73) - open/close/unlink 8014 (time 1789518052.98 total 140.13 last 70.05) - open/close/unlink 8693 (time 1789518062.99 total 150.14 last 67.86) - open/close/unlink 9240 (time 1789518072.99 total 160.14 last 54.67) - open/close/unlink 9733 (time 1789518083.00 total 170.15 last 49.29) - open/close/unlink 10000 (time 1789518088.31 total 175.46 last 50.24) - open/close/unlink 10549 (time 1789518098.34 total 185.49 last 54.74) - open/close/unlink 11145 (time 1789518108.35 total 195.50 last 59.54) - open/close/unlink 11701 (time 1789518118.37 total 205.52 last 55.50) - open/close/unlink 12313 (time 1789518128.39 total 215.54 last 61.08) - open/close/unlink 12772 (time 1789518138.42 total 225.57 last 45.77) - open/close/unlink 13310 (time 1789518148.43 total 235.58 last 53.75) - open/close/unlink 13863 (time 1789518158.44 total 245.58 last 55.25) - open/close/unlink 14448 (time 1789518168.44 total 255.59 last 58.49) - open/close/unlink 15122 (time 1789518178.45 total 265.60 last 67.32) - open/close/unlink 15687 (time 1789518188.47 total 275.62 last 56.40) - open/close/unlink 16250 (time 1789518198.48 total 285.63 last 56.21) - open/close/unlink 16833 (time 1789518208.49 total 295.64 last 58.26) - open/close/unlink 17486 (time 1789518218.50 total 305.64 last 65.26) - open/close/unlink 18032 (time 1789518228.50 total 315.65 last 54.56) - open/close/unlink 18644 (time 1789518238.51 total 325.66 last 61.15) - open/close/unlink 19268 (time 1789518248.52 total 335.67 last 62.35) - open/close/unlink 19742 (time 1789518258.52 total 345.67 last 47.37) - open/close/unlink 20000 (time 1789518264.74 total 351.89 last 41.47) - open/close/unlink 20579 (time 1789518274.75 total 361.90 last 57.84) - open/close/unlink 21192 (time 1789518284.75 total 371.90 last 61.29) - open/close/unlink 21822 (time 1789518294.77 total 381.92 last 62.88) - open/close/unlink 22525 (time 1789518304.78 total 391.93 last 70.28) - open/close/unlink 23176 (time 1789518314.79 total 401.94 last 65.00) - open/close/unlink 23890 (time 1789518324.80 total 411.95 last 71.34) - open/close/unlink 24550 (time 1789518334.81 total 421.96 last 65.94) - open/close/unlink 25255 (time 1789518344.81 total 431.96 last 70.49) - open/close/unlink 25893 (time 1789518354.82 total 441.97 last 63.76) - open/close/unlink 26470 (time 1789518364.82 total 451.97 last 57.68) - open/close/unlink 27109 (time 1789518374.83 total 461.98 last 63.82) - open/close/unlink 27624 (time 1789518384.84 total 471.99 last 51.45) - open/close/unlink 28369 (time 1789518394.84 total 481.99 last 74.48) - open/close/unlink 29160 (time 1789518404.85 total 492.00 last 79.07) - open/close/unlink 29838 (time 1789518414.85 total 502.00 last 67.79) - open/close/unlink 30000 (time 1789518417.69 total 504.84 last 57.10) - open/close/unlink 30582 (time 1789518427.71 total 514.86 last 58.04) - open/close/unlink 31203 (time 1789518437.72 total 524.87 last 62.05) - open/close/unlink 31742 (time 1789518447.74 total 534.89 last 53.81) - open/close/unlink 32308 (time 1789518457.75 total 544.90 last 56.55) - open/close/unlink 32917 (time 1789518467.76 total 554.91 last 60.84) - open/close/unlin - open/close/unlink 653 (time 1789517922.69 total 10.01 last 65.23) - open/close/unlink 1265 (time 1789517932.70 total 20.02 last 61.15) - open/close/unlink 1939 (time 1789517942.71 total 30.03 last 67.35) - open/close/unlink 2506 (time 1789517952.71 total 40.03 last 56.67) - open/close/unlink 2960 (time 1789517962.71 total 50.03 last 45.40) - open/close/unlink 3449 (time 1789517972.72 total 60.04 last 48.85) - open/close/unlink 4010 (time 1789517982.73 total 70.05 last 56.07) - open/close/unlink 4524 (time 1789517992.74 total 80.06 last 51.34) - open/close/unlink 5059 (time 1789518002.74 total 90.06 last 53.49) - open/close/unlink 5538 (time 1789518012.75 total 100.07 last 47.87) - open/close/unlink 6123 (time 1789518022.76 total 110.08 last 58.45) - open/close/unlink 6706 (time 1789518032.78 total 120.10 last 58.15) - open/close/unlink 7309 (time 1789518042.80 total 130.12 last 60.17) - open/close/unlink 8003 (time 1789518052.80 total 140.12 last 69.39) - open/close/unlink 8684 (time 1789518062.82 total 150.14 last 68.01) - open/close/unlink 9237 (time 1789518072.82 total 160.14 last 55.27) - open/close/unlink 9738 (time 1789518082.89 total 170.21 last 49.77) - open/close/unlink 10000 (time 1789518088.12 total 175.44 last 50.12) - open/close/unlink 10546 (time 1789518098.12 total 185.44 last 54.57) - open/close/unlink 11148 (time 1789518108.14 total 195.46 last 60.08) - open/close/unlink 11705 (time 1789518118.18 total 205.50 last 55.48) - open/close/unlink 12309 (time 1789518128.19 total 215.51 last 60.34) - open/close/unlink 12771 (time 1789518138.21 total 225.53 last 46.13) - open/close/unlink 13304 (time 1789518148.21 total 235.53 last 53.29) - open/close/unlink 13866 (time 1789518158.22 total 245.54 last 56.13) - open/close/unlink 14438 (time 1789518168.23 total 255.55 last 57.14) - open/close/unlink 15123 (time 1789518178.24 total 265.56 last 68.47) - open/close/unlink 15687 (time 1789518188.24 total 275.56 last 56.39) - open/close/unlink 16242 (time 1789518198.24 total 285.56 last 55.48) - open/close/unlink 16820 (time 1789518208.26 total 295.58 last 57.71) - open/close/unlink 17475 (time 1789518218.27 total 305.59 last 65.45) - open/close/unlink 18031 (time 1789518228.27 total 315.59 last 55.58) - open/close/unlink 18646 (time 1789518238.28 total 325.60 last 61.43) - open/close/unlink 19275 (time 1789518248.29 total 335.61 last 62.82) - open/close/unlink 19746 (time 1789518258.30 total 345.62 last 47.06) - open/close/unlink 20000 (time 1789518264.56 total 351.88 last 40.60) - open/close/unlink 20576 (time 1789518274.57 total 361.89 last 57.54) - open/close/unlink 21184 (time 1789518284.58 total 371.90 last 60.74) - open/close/unlink 21807 (time 1789518294.58 total 381.91 last 62.26) - open/close/unlink 22525 (time 1789518304.59 total 391.91 last 71.74) - open/close/unlink 23166 (time 1789518314.60 total 401.92 last 64.04) - open/close/unlink 23871 (time 1789518324.61 total 411.93 last 70.42) - open/close/unlink 24534 (time 1789518334.62 total 421.95 last 66.23) - open/close/unlink 25231 (time 1789518344.63 total 431.95 last 69.69) - open/close/unlink 25862 (time 1789518354.65 total 441.97 last 62.98) - open/close/unlink 26436 (time 1789518364.65 total 451.97 last 57.38) - open/close/unlink 27075 (time 1789518374.66 total 461.98 last 63.83) - open/close/unlink 27589 (time 1789518384.67 total 471.99 last 51.36) - open/close/unlink 28335 (time 1789518394.69 total 482.01 last 74.46) - open/close/unlink 29126 (time 1789518404.69 total 492.01 last 79.06) - open/close/unlink 29809 (time 1789518414.69 total 502.01 last 68.28) - open/close/unlink 30000 (time 1789518417.97 total 505.29 last 58.28) - open/close/unlink 30575 (time 1789518427.97 total 515.29 last 57.49) - open/close/unlink 31199 (time 1789518437.97 total 525.29 last 62.39) - open/close/unlink 31728 (time 1789518447.98 total 535.30 last 52.85) - open/close/unlink 32286 (time 1789518458.00 total 545.32 last 55.73) - open/close/unlink 32900 (time 1789518468.00 total 555.32 last 61.40) - open/close/unlin - open/close/unlink 643 (time 1789517922.80 total 10.00 last 64.27) - open/close/unlink 1253 (time 1789517932.80 total 20.01 last 60.99) - open/close/unlink 1920 (time 1789517942.81 total 30.02 last 66.62) - open/close/unlink 2476 (time 1789517952.83 total 40.04 last 55.50) - open/close/unlink 2931 (time 1789517962.85 total 50.05 last 45.42) - open/close/unlink 3422 (time 1789517972.85 total 60.06 last 49.09) - open/close/unlink 3977 (time 1789517982.86 total 70.06 last 55.45) - open/close/unlink 4495 (time 1789517992.87 total 80.08 last 51.73) - open/close/unlink 5031 (time 1789518002.87 total 90.08 last 53.60) - open/close/unlink 5508 (time 1789518012.89 total 100.10 last 47.60) - open/close/unlink 6084 (time 1789518022.90 total 110.11 last 57.54) - open/close/unlink 6662 (time 1789518032.91 total 120.12 last 57.75) - open/close/unlink 7264 (time 1789518042.92 total 130.13 last 60.12) - open/close/unlink 7961 (time 1789518052.94 total 140.15 last 69.59) - open/close/unlink 8635 (time 1789518062.95 total 150.16 last 67.33) - open/close/unlink 9189 (time 1789518072.96 total 160.17 last 55.33) - open/close/unlink 9681 (time 1789518082.98 total 170.18 last 49.14) - open/close/unlink 10000 (time 1789518089.24 total 176.45 last 50.89) - open/close/unlink 10528 (time 1789518099.25 total 186.46 last 52.78) - open/close/unlink 11146 (time 1789518109.26 total 196.47 last 61.70) - open/close/unlink 11700 (time 1789518119.27 total 206.47 last 55.39) - open/close/unlink 12295 (time 1789518129.29 total 216.49 last 59.38) - open/close/unlink 12765 (time 1789518139.30 total 226.51 last 46.93) - open/close/unlink 13315 (time 1789518149.32 total 236.53 last 54.89) - open/close/unlink 13861 (time 1789518159.34 total 246.55 last 54.49) - open/close/unlink 14460 (time 1789518169.36 total 256.57 last 59.79) - open/close/unlink 15114 (time 1789518179.36 total 266.57 last 65.39) - open/close/unlink 15661 (time 1789518189.38 total 276.59 last 54.61) - open/close/unlink 16235 (time 1789518199.39 total 286.60 last 57.35) - open/close/unlink 16825 (time 1789518209.40 total 296.61 last 58.94) - open/close/unlink 17454 (time 1789518219.40 total 306.61 last 62.88) - open/close/unlink 18024 (time 1789518229.41 total 316.61 last 56.97) - open/close/unlink 18629 (time 1789518239.41 total 326.62 last 60.49) - open/close/unlink 19252 (time 1789518249.41 total 336.62 last 62.30) - open/close/unlink 19703 (time 1789518259.45 total 346.66 last 44.89) - open/close/unlink 20000 (time 1789518266.27 total 353.48 last 43.59) - open/close/unlink 20628 (time 1789518276.28 total 363.49 last 62.72) - open/close/unlink 21215 (time 1789518286.28 total 373.49 last 58.69) - open/close/unlink 21836 (time 1789518296.30 total 383.51 last 62.01) - open/close/unlink 22556 (time 1789518306.30 total 393.51 last 71.96) - open/close/unlink 23203 (time 1789518316.31 total 403.52 last 64.66) - open/close/unlink 23926 (time 1789518326.31 total 413.52 last 72.29) - open/close/unlink 24583 (time 1789518336.31 total 423.52 last 65.68) - open/close/unlink 25288 (time 1789518346.32 total 433.53 last 70.42) - open/close/unlink 25893 (time 1789518356.35 total 443.56 last 60.34) - open/close/unlink 26480 (time 1789518366.37 total 453.57 last 58.60) - open/close/unlink 27099 (time 1789518376.38 total 463.59 last 61.82) - open/close/unlink 27632 (time 1789518386.38 total 473.59 last 53.28) - open/close/unlink 28378 (time 1789518396.39 total 483.60 last 74.55) - open/close/unlink 29167 (time 1789518406.39 total 493.60 last 78.88) - open/close/unlink 29836 (time 1789518416.41 total 503.61 last 66.79) - open/close/unlink 30000 (time 1789518419.32 total 506.53 last 56.35) - open/close/unlink 30585 (time 1789518429.32 total 516.53 last 58.47) - open/close/unlink 31203 (time 1789518439.32 total 526.53 last 61.78) - open/close/unlink 31726 (time 1789518449.33 total 536.54 last 52.26) - open/close/unlink 32314 (time 1789518459.33 total 546.54 last 58.79) - open/close/unlink 32942 (time 1789518469.35 total 556.55 last 62.73) - open/close/unlin - open/close/unlink 646 (time 1789517922.95 total 10.01 last 64.54) - open/close/unlink 1261 (time 1789517932.96 total 20.01 last 61.49) - open/close/unlink 1933 (time 1789517942.96 total 30.02 last 67.15) - open/close/unlink 2490 (time 1789517952.98 total 40.03 last 55.63) - open/close/unlink 2940 (time 1789517962.99 total 50.04 last 44.96) - open/close/unlink 3435 (time 1789517972.99 total 60.04 last 49.50) - open/close/unlink 3997 (time 1789517982.99 total 70.04 last 56.18) - open/close/unlink 4521 (time 1789517993.00 total 80.06 last 52.32) - open/close/unlink 5058 (time 1789518003.02 total 90.08 last 53.60) - open/close/unlink 5542 (time 1789518013.04 total 100.10 last 48.29) - open/close/unlink 6113 (time 1789518023.06 total 110.12 last 57.01) - open/close/unlink 6694 (time 1789518033.06 total 120.12 last 58.09) - open/close/unlink 7293 (time 1789518043.07 total 130.13 last 59.86) - open/close/unlink 7991 (time 1789518053.08 total 140.13 last 69.77) - open/close/unlink 8667 (time 1789518063.08 total 150.14 last 67.56) - open/close/unlink 9217 (time 1789518073.10 total 160.16 last 54.88) - open/close/unlink 9708 (time 1789518083.11 total 170.17 last 49.06) - open/close/unlink 10000 (time 1789518088.87 total 175.93 last 50.68) - open/close/unlink 10542 (time 1789518098.89 total 185.94 last 54.12) - open/close/unlink 11158 (time 1789518108.89 total 195.94 last 61.60) - open/close/unlink 11695 (time 1789518118.89 total 205.95 last 53.68) - open/close/unlink 12294 (time 1789518128.90 total 215.95 last 59.86) - open/close/unlink 12759 (time 1789518138.90 total 225.96 last 46.48) - open/close/unlink 13302 (time 1789518148.92 total 235.98 last 54.20) - open/close/unlink 13850 (time 1789518158.93 total 245.99 last 54.75) - open/close/unlink 14449 (time 1789518168.94 total 256.00 last 59.83) - open/close/unlink 15112 (time 1789518178.96 total 266.01 last 66.19) - open/close/unlink 15669 (time 1789518188.97 total 276.02 last 55.64) - open/close/unlink 16232 (time 1789518198.99 total 286.04 last 56.21) - open/close/unlink 16818 (time 1789518209.00 total 296.05 last 58.53) - open/close/unlink 17456 (time 1789518219.02 total 306.07 last 63.67) - open/close/unlink 18019 (time 1789518229.02 total 316.07 last 56.28) - open/close/unlink 18629 (time 1789518239.03 total 326.09 last 60.91) - open/close/unlink 19247 (time 1789518249.04 total 336.10 last 61.75) - open/close/unlink 19714 (time 1789518259.06 total 346.11 last 46.64) - open/close/unlink 20000 (time 1789518265.86 total 352.92 last 42.01) - open/close/unlink 20604 (time 1789518275.87 total 362.92 last 60.39) - open/close/unlink 21194 (time 1789518285.88 total 372.93 last 58.91) - open/close/unlink 21822 (time 1789518295.89 total 382.94 last 62.75) - open/close/unlink 22529 (time 1789518305.90 total 392.95 last 70.63) - open/close/unlink 23172 (time 1789518315.90 total 402.96 last 64.27) - open/close/unlink 23896 (time 1789518325.91 total 412.97 last 72.31) - open/close/unlink 24556 (time 1789518335.92 total 422.97 last 65.96) - open/close/unlink 25262 (time 1789518345.93 total 432.98 last 70.53) - open/close/unlink 25877 (time 1789518355.94 total 442.99 last 61.44) - open/close/unlink 26458 (time 1789518365.95 total 453.00 last 58.04) - open/close/unlink 27080 (time 1789518375.95 total 463.01 last 62.19) - open/close/unlink 27603 (time 1789518385.96 total 473.01 last 52.28) - open/close/unlink 28347 (time 1789518395.96 total 483.01 last 74.39) - open/close/unlink 29138 (time 1789518405.97 total 493.02 last 79.04) - open/close/unlink 29806 (time 1789518415.97 total 503.03 last 66.74) - open/close/unlink 30000 (time 1789518419.38 total 506.44 last 56.90) - open/close/unlink 30578 (time 1789518429.40 total 516.45 last 57.71) - open/close/unlink 31187 (time 1789518439.40 total 526.46 last 60.87) - open/close/unlink 31713 (time 1789518449.43 total 536.48 last 52.49) - open/close/unlink 32288 (time 1789518459.43 total 546.49 last 57.45) - open/close/unlink 32924 (time 1789518469.44 total 556.49 last 63.58) - open/close/unlin - open/close/unlink 646 (time 1789517922.92 total 10.00 last 64.59) - open/close/unlink 1257 (time 1789517932.92 total 20.00 last 61.10) - open/close/unlink 1924 (time 1789517942.93 total 30.01 last 66.63) - open/close/unlink 2482 (time 1789517952.93 total 40.01 last 55.79) - open/close/unlink 2929 (time 1789517962.96 total 50.04 last 44.58) - open/close/unlink 3417 (time 1789517972.96 total 60.04 last 48.80) - open/close/unlink 3975 (time 1789517982.96 total 70.05 last 55.77) - open/close/unlink 4488 (time 1789517992.97 total 80.05 last 51.27) - open/close/unlink 5027 (time 1789518002.97 total 90.05 last 53.90) - open/close/unlink 5515 (time 1789518012.97 total 100.06 last 48.77) - open/close/unlink 6097 (time 1789518022.98 total 110.06 last 58.19) - open/close/unlink 6674 (time 1789518032.99 total 120.07 last 57.63) - open/close/unlink 7271 (time 1789518042.99 total 130.08 last 59.66) - open/close/unlink 7966 (time 1789518053.00 total 140.08 last 69.47) - open/close/unlink 8640 (time 1789518063.02 total 150.10 last 67.26) - open/close/unlink 9191 (time 1789518073.03 total 160.11 last 55.07) - open/close/unlink 9685 (time 1789518083.03 total 170.12 last 49.37) - open/close/unlink 10000 (time 1789518089.19 total 176.27 last 51.15) - open/close/unlink 10533 (time 1789518099.22 total 186.30 last 53.15) - open/close/unlink 11149 (time 1789518109.23 total 196.32 last 61.50) - open/close/unlink 11699 (time 1789518119.24 total 206.33 last 54.95) - open/close/unlink 12296 (time 1789518129.24 total 216.33 last 59.70) - open/close/unlink 12758 (time 1789518139.25 total 226.33 last 46.19) - open/close/unlink 13309 (time 1789518149.25 total 236.34 last 55.05) - open/close/unlink 13860 (time 1789518159.26 total 246.35 last 55.06) - open/close/unlink 14463 (time 1789518169.27 total 256.35 last 60.26) - open/close/unlink 15124 (time 1789518179.28 total 266.36 last 66.04) - open/close/unlink 15683 (time 1789518189.28 total 276.36 last 55.88) - open/close/unlink 16251 (time 1789518199.28 total 286.37 last 56.78) - open/close/unlink 16832 (time 1789518209.29 total 296.38 last 58.04) - open/close/unlink 17468 (time 1789518219.30 total 306.38 last 63.59) - open/close/unlink 18027 (time 1789518229.30 total 316.38 last 55.89) - open/close/unlink 18621 (time 1789518239.30 total 326.39 last 59.38) - open/close/unlink 19248 (time 1789518249.31 total 336.39 last 62.65) - open/close/unlink 19703 (time 1789518259.32 total 346.40 last 45.45) - open/close/unlink 20000 (time 1789518266.26 total 353.34 last 42.79) - open/close/unlink 20614 (time 1789518276.26 total 363.35 last 61.40) - open/close/unlink 21203 (time 1789518286.26 total 373.35 last 58.88) - open/close/unlink 21827 (time 1789518296.27 total 383.36 last 62.35) - open/close/unlink 22540 (time 1789518306.27 total 393.36 last 71.30) - open/close/unlink 23184 (time 1789518316.27 total 403.36 last 64.40) - open/close/unlink 23906 (time 1789518326.28 total 413.36 last 72.17) - open/close/unlink 24567 (time 1789518336.28 total 423.37 last 66.08) - open/close/unlink 25274 (time 1789518346.28 total 433.37 last 70.68) - open/close/unlink 25876 (time 1789518356.30 total 443.38 last 60.10) - open/close/unlink 26461 (time 1789518366.30 total 453.38 last 58.50) - open/close/unlink 27084 (time 1789518376.31 total 463.40 last 62.22) - open/close/unlink 27616 (time 1789518386.32 total 473.41 last 53.16) - open/close/unlink 28356 (time 1789518396.32 total 483.41 last 73.98) - open/close/unlink 29142 (time 1789518406.33 total 493.41 last 78.56) - open/close/unlink 29819 (time 1789518416.34 total 503.42 last 67.65) - open/close/unlink 30000 (time 1789518419.66 total 506.74 last 54.45) - open/close/unlink 30587 (time 1789518429.68 total 516.77 last 58.56) - open/close/unlink 31204 (time 1789518439.70 total 526.79 last 61.59) - open/close/unlink 31720 (time 1789518449.72 total 536.80 last 51.51) - open/close/unlink 32310 (time 1789518459.74 total 546.82 last 58.89) - open/close/unlink 32936 (time 1789518469.75 total 556.83 last 62.54) - open/close/unlink 33487 (time 1789518477.77 total 564.92 last 56.93) - open/close/unlink 34033 (time 1789518487.77 total 574.92 last 54.58) - open/close/unlink 34566 (time 1789518497.77 total 584.92 last 53.29) - open/close/unlink 35149 (time 1789518507.79 total 594.94 last 58.21) - open/close/unlink 35667 (time 1789518517.80 total 604.95 last 51.74) - open/close/unlink 36155 (time 1789518527.80 total 614.95 last 48.79) - open/close/unlink 36796 (time 1789518537.80 total 624.95 last 64.10) - open/close/unlink 37372 (time 1789518547.81 total 634.96 last 57.53) - open/close/unlink 37961 (time 1789518557.83 total 644.98 last 58.82) - open/close/unlink 38533 (time 1789518567.83 total 654.98 last 57.17) - open/close/unlink 39276 (time 1789518577.84 total 664.99 last 74.21) - open/close/unlink 39908 (time 1789518587.86 total 675.01 last 63.11) - open/close/unlink 40530 (time 1789518597.86 total 685.01 last 62.16) - open/close/unlink 41182 (time 1789518607.87 total 695.02 last 65.20) - open/close/unlink 41892 (time 1789518617.88 total 705.03 last 70.91) - open/close/unlink 42542 (time 1789518627.88 total 715.03 last 64.96) - open/close/unlink 43097 (time 1789518637.89 total 725.04 last 55.46) - open/close/unlink 43832 (time 1789518647.90 total 735.05 last 73.44) - open/close/unlink 44439 (time 1789518657.92 total 745.07 last 60.59) - open/close/unlink 45054 (time 1789518667.93 total 755.08 last 61.45) - open/close/unlink 45666 (time 1789518677.94 total 765.08 last 61.15) - open/close/unlink 46320 (time 1789518687.94 total 775.09 last 65.36) - open/close/unlink 46997 (time 1789518697.94 total 785.09 last 67.68) - open/close/unlink 47710 (time 1789518707.95 total 795.10 last 71.28) - open/close/unlink 48381 (time 1789518717.95 total 805.10 last 67.09) - open/close/unlink 49148 (time 1789518727.96 total 815.11 last 76.63) - open/close/unlink 49948 (time 1789518737.97 total 825.12 last 79.91) - open/close/unlink 50643 (time 1789518747.97 total 835.12 last 69.48) - open/close/unlink 51253 (time 1789518757.98 total 845.13 last 60.93) - open/close/unlink 51935 (time 1789518767.99 total 855.14 last 68.16) - open/close/unlink 52589 (time 1789518778.00 total 865.15 last 65.34) - open/close/unlink 53208 (time 1789518788.02 total 875.17 last 61.78) - open/close/unlink 53762 (time 1789518798.03 total 885.18 last 55.34) - open/close/unlink 54269 (time 1789518808.03 total 895.18 last 50.68) - open/close/unlink 54733 (time 1789518818.05 total 905.20 last 46.32) - open/close/unlink 55156 (time 1789518828.06 total 915.21 last 42.27) - open/close/unlink 55660 (time 1789518838.06 total 925.21 last 50.38) - open/close/unlink 56222 (time 1789518848.07 total 935.22 last 56.16) - open/close/unlink 56786 (time 1789518858.08 total 945.23 last 56.34) - open/close/unlink 57377 (time 1789518868.09 total 955.24 last 59.03) - open/close/unlink 57954 (time 1789518878.09 total 965.24 last 57.70) - open/close/unlink 58543 (time 1789518888.09 total 975.24 last 58.90) - open/close/unlink 59173 (time 1789518898.10 total 985.25 last 62.95) - open/close/unlink 59692 (time 1789518908.11 total 995.26 last 51.86) - open/close/unlink 60000 (time 1789518912.95 total 1000.10 last 63.66) - open/close/unlink 60588 (time 1789518922.95 total 1010.10 last 58.75) - open/close/unlink 61187 (time 1789518932.96 total 1020.11 last 59.83) - open/close/unlink 61845 (time 1789518942.99 total 1030.14 last 65.66) - open/close/unlink 62438 (time 1789518953.02 total 1040.17 last 59.12) - open/close/unlink 62992 (time 1789518963.02 total 1050.17 last 55.37) - open/close/unlink 63530 (time 1789518973.02 total 1060.17 last 53.80) - open/close/unlink 64049 (time 1789518983.05 total 1070.20 last 51.77) - open/close/unlink 64546 (time 1789518993.06 total 1080.21 last 49.65) total: 64768 open/close/unlink in 1083.40 seconds: 59.78 ops/second k 33463 (time 1789518478.02 total 565.34 last 56.19) - open/close/unlink 34014 (time 1789518488.02 total 575.34 last 55.10) - open/close/unlink 34553 (time 1789518498.02 total 585.34 last 53.89) - open/close/unlink 35119 (time 1789518508.03 total 595.35 last 56.51) - open/close/unlink 35631 (time 1789518518.05 total 605.37 last 51.10) - open/close/unlink 36127 (time 1789518528.06 total 615.38 last 49.58) - open/close/unlink 36776 (time 1789518538.06 total 625.38 last 64.87) - open/close/unlink 37344 (time 1789518548.07 total 635.39 last 56.77) - open/close/unlink 37941 (time 1789518558.07 total 645.39 last 59.66) - open/close/unlink 38520 (time 1789518568.08 total 655.40 last 57.88) - open/close/unlink 39261 (time 1789518578.09 total 665.41 last 74.03) - open/close/unlink 39882 (time 1789518588.09 total 675.41 last 62.10) - open/close/unlink 40506 (time 1789518598.09 total 685.41 last 62.35) - open/close/unlink 41159 (time 1789518608.10 total 695.42 last 65.28) - open/close/unlink 41864 (time 1789518618.10 total 705.42 last 70.49) - open/close/unlink 42513 (time 1789518628.11 total 715.43 last 64.81) - open/close/unlink 43072 (time 1789518638.11 total 725.43 last 55.89) - open/close/unlink 43813 (time 1789518648.12 total 735.44 last 74.07) - open/close/unlink 44428 (time 1789518658.12 total 745.44 last 61.48) - open/close/unlink 45031 (time 1789518668.12 total 755.45 last 60.28) - open/close/unlink 45641 (time 1789518678.13 total 765.45 last 60.95) - open/close/unlink 46295 (time 1789518688.15 total 775.47 last 65.32) - open/close/unlink 46965 (time 1789518698.15 total 785.47 last 66.96) - open/close/unlink 47671 (time 1789518708.15 total 795.47 last 70.59) - open/close/unlink 48355 (time 1789518718.16 total 805.48 last 68.34) - open/close/unlink 49118 (time 1789518728.17 total 815.49 last 76.27) - open/close/unlink 49932 (time 1789518738.18 total 825.51 last 81.25) - open/close/unlink 50616 (time 1789518748.19 total 835.51 last 68.35) - open/close/unlink 51232 (time 1789518758.20 total 845.52 last 61.55) - open/close/unlink 51915 (time 1789518768.20 total 855.52 last 68.29) - open/close/unlink 52569 (time 1789518778.20 total 865.52 last 65.40) - open/close/unlink 53180 (time 1789518788.20 total 875.52 last 61.10) - open/close/unlink 53730 (time 1789518798.20 total 885.52 last 54.99) - open/close/unlink 54243 (time 1789518808.22 total 895.55 last 51.19) - open/close/unlink 54721 (time 1789518818.23 total 905.55 last 47.76) - open/close/unlink 55131 (time 1789518828.24 total 915.56 last 40.97) - open/close/unlink 55630 (time 1789518838.25 total 925.57 last 49.85) - open/close/unlink 56191 (time 1789518848.25 total 935.57 last 56.07) - open/close/unlink 56757 (time 1789518858.26 total 945.58 last 56.59) - open/close/unlink 57341 (time 1789518868.26 total 955.58 last 58.40) - open/close/unlink 57913 (time 1789518878.26 total 965.58 last 57.18) - open/close/unlink 58507 (time 1789518888.28 total 975.60 last 59.29) - open/close/unlink 59132 (time 1789518898.29 total 985.61 last 62.41) - open/close/unlink 59649 (time 1789518908.30 total 995.63 last 51.63) - open/close/unlink 60000 (time 1789518913.77 total 1001.10 last 64.18) - open/close/unlink 60587 (time 1789518923.78 total 1011.10 last 58.66) - open/close/unlink 61178 (time 1789518933.79 total 1021.11 last 59.05) - open/close/unlink 61843 (time 1789518943.81 total 1031.13 last 66.35) - open/close/unlink 62435 (time 1789518953.81 total 1041.13 last 59.20) - open/close/unlink 62967 (time 1789518963.83 total 1051.15 last 53.13) - open/close/unlink 63486 (time 1789518973.86 total 1061.18 last 51.71) - open/close/unlink 64009 (time 1789518983.87 total 1071.19 last 52.28) - open/close/unlink 64537 (time 1789518993.88 total 1081.20 last 52.74) total: 64768 open/close/unlink in 1085.16 seconds: 59.69 ops/second k 33469 (time 1789518479.35 total 566.56 last 52.69) - open/close/unlink 34024 (time 1789518489.36 total 576.57 last 55.43) - open/close/unlink 34581 (time 1789518499.37 total 586.58 last 55.63) - open/close/unlink 35131 (time 1789518509.38 total 596.59 last 54.96) - open/close/unlink 35619 (time 1789518519.38 total 606.59 last 48.79) - open/close/unlink 36136 (time 1789518529.38 total 616.59 last 51.69) - open/close/unlink 36785 (time 1789518539.39 total 626.59 last 64.89) - open/close/unlink 37352 (time 1789518549.39 total 636.60 last 56.68) - open/close/unlink 37934 (time 1789518559.40 total 646.61 last 58.11) - open/close/unlink 38544 (time 1789518569.42 total 656.63 last 60.92) - open/close/unlink 39277 (time 1789518579.42 total 666.63 last 73.29) - open/close/unlink 39907 (time 1789518589.43 total 676.64 last 62.90) - open/close/unlink 40542 (time 1789518599.45 total 686.66 last 63.38) - open/close/unlink 41201 (time 1789518609.47 total 696.67 last 65.81) - open/close/unlink 41903 (time 1789518619.48 total 706.69 last 70.10) - open/close/unlink 42540 (time 1789518629.48 total 716.69 last 63.70) - open/close/unlink 43111 (time 1789518639.48 total 726.69 last 57.08) - open/close/unlink 43842 (time 1789518649.48 total 736.69 last 73.09) - open/close/unlink 44472 (time 1789518659.49 total 746.69 last 62.99) - open/close/unlink 45066 (time 1789518669.50 total 756.71 last 59.30) - open/close/unlink 45690 (time 1789518679.50 total 766.71 last 62.39) - open/close/unlink 46355 (time 1789518689.51 total 776.72 last 66.45) - open/close/unlink 47033 (time 1789518699.52 total 786.73 last 67.73) - open/close/unlink 47749 (time 1789518709.52 total 796.73 last 71.59) - open/close/unlink 48449 (time 1789518719.53 total 806.74 last 69.95) - open/close/unlink 49204 (time 1789518729.54 total 816.75 last 75.42) - open/close/unlink 50000 (time 1789518739.32 total 826.53 last 81.38) - open/close/unlink 50660 (time 1789518749.33 total 836.54 last 65.93) - open/close/unlink 51279 (time 1789518759.34 total 846.55 last 61.86) - open/close/unlink 51965 (time 1789518769.34 total 856.55 last 68.59) - open/close/unlink 52621 (time 1789518779.35 total 866.56 last 65.52) - open/close/unlink 53223 (time 1789518789.37 total 876.58 last 60.09) - open/close/unlink 53783 (time 1789518799.39 total 886.60 last 55.91) - open/close/unlink 54289 (time 1789518809.41 total 896.62 last 50.51) - open/close/unlink 54747 (time 1789518819.42 total 906.63 last 45.75) - open/close/unlink 55172 (time 1789518829.44 total 916.65 last 42.41) - open/close/unlink 55675 (time 1789518839.45 total 926.65 last 50.28) - open/close/unlink 56251 (time 1789518849.45 total 936.66 last 57.59) - open/close/unlink 56816 (time 1789518859.46 total 946.67 last 56.44) - open/close/unlink 57399 (time 1789518869.46 total 956.67 last 58.28) - open/close/unlink 57974 (time 1789518879.48 total 966.68 last 57.41) - open/close/unlink 58562 (time 1789518889.49 total 976.70 last 58.70) - open/close/unlink 59188 (time 1789518899.50 total 986.70 last 62.58) - open/close/unlink 59717 (time 1789518909.51 total 996.72 last 52.81) - open/close/unlink 60000 (time 1789518914.03 total 1001.24 last 62.67) - open/close/unlink 60586 (time 1789518924.05 total 1011.26 last 58.48) - open/close/unlink 61186 (time 1789518934.05 total 1021.26 last 59.98) - open/close/unlink 61844 (time 1789518944.06 total 1031.27 last 65.76) - open/close/unlink 62428 (time 1789518954.07 total 1041.28 last 58.34) - open/close/unlink 62973 (time 1789518964.08 total 1051.28 last 54.46) - open/close/unlink 63482 (time 1789518974.09 total 1061.30 last 50.83) - open/close/unlink 63998 (time 1789518984.10 total 1071.31 last 51.54) - open/close/unlink 64520 (time 1789518994.11 total 1081.32 last 52.15) total: 64768 open/close/unlink in 1085.83 seconds: 59.65 ops/second k 33467 (time 1789518479.76 total 566.85 last 53.03) - open/close/unlink 34024 (time 1789518489.77 total 576.86 last 55.63) - open/close/unlink 34581 (time 1789518499.78 total 586.86 last 55.67) - open/close/unlink 35120 (time 1789518509.80 total 596.88 last 53.80) - open/close/unlink 35607 (time 1789518519.80 total 606.89 last 48.68) - open/close/unlink 36137 (time 1789518529.80 total 616.89 last 53.00) - open/close/unlink 36777 (time 1789518539.81 total 626.89 last 63.97) - open/close/unlink 37348 (time 1789518549.82 total 636.90 last 57.05) - open/close/unlink 37928 (time 1789518559.83 total 646.92 last 57.91) - open/close/unlink 38550 (time 1789518569.84 total 656.92 last 62.17) - open/close/unlink 39279 (time 1789518579.84 total 666.92 last 72.89) - open/close/unlink 39898 (time 1789518589.86 total 676.94 last 61.75) - open/close/unlink 40512 (time 1789518599.88 total 686.96 last 61.31) - open/close/unlink 41180 (time 1789518609.89 total 696.97 last 66.70) - open/close/unlink 41902 (time 1789518619.90 total 706.99 last 72.10) - open/close/unlink 42539 (time 1789518629.91 total 717.00 last 63.65) - open/close/unlink 43115 (time 1789518639.91 total 727.00 last 57.59) - open/close/unlink 43843 (time 1789518649.92 total 737.00 last 72.76) - open/close/unlink 44467 (time 1789518659.92 total 747.01 last 62.38) - open/close/unlink 45055 (time 1789518669.93 total 757.02 last 58.74) - open/close/unlink 45680 (time 1789518679.94 total 767.03 last 62.44) - open/close/unlink 46345 (time 1789518689.95 total 777.03 last 66.46) - open/close/unlink 47021 (time 1789518699.96 total 787.04 last 67.55) - open/close/unlink 47750 (time 1789518709.96 total 797.05 last 72.83) - open/close/unlink 48450 (time 1789518719.97 total 807.05 last 69.99) - open/close/unlink 49183 (time 1789518729.98 total 817.06 last 73.22) - open/close/unlink 50000 (time 1789518739.86 total 826.95 last 82.63) - open/close/unlink 50657 (time 1789518749.87 total 836.96 last 65.66) - open/close/unlink 51272 (time 1789518759.89 total 846.98 last 61.36) - open/close/unlink 51959 (time 1789518769.89 total 856.98 last 68.69) - open/close/unlink 52622 (time 1789518779.90 total 866.99 last 66.25) - open/close/unlink 53214 (time 1789518789.92 total 877.00 last 59.11) - open/close/unlink 53775 (time 1789518799.93 total 887.02 last 56.00) - open/close/unlink 54268 (time 1789518809.94 total 897.02 last 49.29) - open/close/unlink 54712 (time 1789518819.94 total 907.02 last 44.40) - open/close/unlink 55147 (time 1789518829.95 total 917.03 last 43.45) - open/close/unlink 55653 (time 1789518839.95 total 927.04 last 50.57) - open/close/unlink 56232 (time 1789518849.96 total 937.04 last 57.88) - open/close/unlink 56793 (time 1789518859.96 total 947.04 last 56.10) - open/close/unlink 57373 (time 1789518869.97 total 957.05 last 57.92) - open/close/unlink 57949 (time 1789518879.98 total 967.06 last 57.54) - open/close/unlink 58543 (time 1789518889.99 total 977.08 last 59.32) - open/close/unlink 59163 (time 1789518900.01 total 987.10 last 61.89) - open/close/unlink 59698 (time 1789518910.02 total 997.11 last 53.43) - open/close/unlink 60000 (time 1789518914.95 total 1002.04 last 61.27) - open/close/unlink 60591 (time 1789518924.97 total 1012.05 last 59.00) - open/close/unlink 61183 (time 1789518934.98 total 1022.07 last 59.12) - open/close/unlink 61854 (time 1789518944.99 total 1032.07 last 67.08) - open/close/unlink 62453 (time 1789518955.00 total 1042.08 last 59.82) - open/close/unlink 62996 (time 1789518965.00 total 1052.09 last 54.30) - open/close/unlink 63493 (time 1789518975.01 total 1062.09 last 49.67) - open/close/unlink 64009 (time 1789518985.01 total 1072.10 last 51.57) - open/close/unlink 64566 (time 1789518995.02 total 1082.11 last 55.63) total: 64768 open/close/unlink in 1087.19 seconds: 59.57 ops/second k 33463 (time 1789518479.45 total 566.51 last 53.81) - open/close/unlink 34021 (time 1789518489.47 total 576.52 last 55.72) - open/close/unlink 34580 (time 1789518499.47 total 586.53 last 55.89) - open/close/unlink 35123 (time 1789518509.48 total 596.53 last 54.26) - open/close/unlink 35615 (time 1789518519.49 total 606.55 last 49.13) - open/close/unlink 36135 (time 1789518529.50 total 616.56 last 51.95) - open/close/unlink 36775 (time 1789518539.51 total 626.56 last 63.95) - open/close/unlink 37342 (time 1789518549.51 total 636.57 last 56.68) - open/close/unlink 37919 (time 1789518559.52 total 646.57 last 57.69) - open/close/unlink 38527 (time 1789518569.52 total 656.58 last 60.75) - open/close/unlink 39265 (time 1789518579.53 total 666.58 last 73.77) - open/close/unlink 39885 (time 1789518589.54 total 676.60 last 61.91) - open/close/unlink 40513 (time 1789518599.56 total 686.61 last 62.70) - open/close/unlink 41184 (time 1789518609.57 total 696.62 last 67.02) - open/close/unlink 41895 (time 1789518619.58 total 706.63 last 71.03) - open/close/unlink 42533 (time 1789518629.59 total 716.65 last 63.72) - open/close/unlink 43100 (time 1789518639.60 total 726.65 last 56.68) - open/close/unlink 43830 (time 1789518649.60 total 736.65 last 72.99) - open/close/unlink 44460 (time 1789518659.61 total 746.66 last 62.93) - open/close/unlink 45049 (time 1789518669.61 total 756.67 last 58.88) - open/close/unlink 45668 (time 1789518679.62 total 766.67 last 61.86) - open/close/unlink 46333 (time 1789518689.63 total 776.68 last 66.45) - open/close/unlink 47006 (time 1789518699.63 total 786.68 last 67.26) - open/close/unlink 47720 (time 1789518709.64 total 796.69 last 71.34) - open/close/unlink 48417 (time 1789518719.64 total 806.70 last 69.67) - open/close/unlink 49157 (time 1789518729.65 total 816.71 last 73.94) - open/close/unlink 49974 (time 1789518739.65 total 826.71 last 81.70) - open/close/unlink 50617 (time 1789518749.66 total 836.72 last 64.24) - open/close/unlink 51236 (time 1789518759.67 total 846.73 last 61.83) - open/close/unlink 51929 (time 1789518769.68 total 856.74 last 69.22) - open/close/unlink 52590 (time 1789518779.69 total 866.74 last 66.06) - open/close/unlink 53180 (time 1789518789.69 total 876.75 last 58.99) - open/close/unlink 53741 (time 1789518799.71 total 886.77 last 55.98) - open/close/unlink 54252 (time 1789518809.73 total 896.79 last 51.00) - open/close/unlink 54702 (time 1789518819.77 total 906.83 last 44.83) - open/close/unlink 55132 (time 1789518829.79 total 916.84 last 42.94) - open/close/unlink 55636 (time 1789518839.80 total 926.85 last 50.34) - open/close/unlink 56219 (time 1789518849.80 total 936.85 last 58.29) - open/close/unlink 56789 (time 1789518859.80 total 946.86 last 56.98) - open/close/unlink 57367 (time 1789518869.81 total 956.86 last 57.79) - open/close/unlink 57941 (time 1789518879.82 total 966.88 last 57.31) - open/close/unlink 58531 (time 1789518889.82 total 976.88 last 58.99) - open/close/unlink 59149 (time 1789518899.84 total 986.90 last 61.67) - open/close/unlink 59676 (time 1789518909.86 total 996.91 last 52.65) - open/close/unlink 60000 (time 1789518915.13 total 1002.18 last 61.47) - open/close/unlink 60589 (time 1789518925.13 total 1012.18 last 58.90) - open/close/unlink 61182 (time 1789518935.13 total 1022.19 last 59.27) - open/close/unlink 61853 (time 1789518945.14 total 1032.20 last 67.04) - open/close/unlink 62449 (time 1789518955.14 total 1042.20 last 59.59) - open/close/unlink 62993 (time 1789518965.15 total 1052.21 last 54.34) - open/close/unlink 63491 (time 1789518975.17 total 1062.22 last 49.72) - open/close/unlink 64009 (time 1789518985.18 total 1072.24 last 51.73) - open/close/unlink 64566 (time 1789518995.18 total 1082.24 last 55.69) total: 64768 open/close/unlink in 1088.12 seconds: 59.52 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete oleg613-server: cannot open 'lustre-ost2': no such pool pdsh@oleg613-client: oleg613-server: ssh exited with exit code 255 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg613-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg613-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg613-server stop mds service on oleg613-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg613-server unloading modules via unload_modules_local on: 'oleg613-server' oleg613-server: oleg613-server.virtnet: executing unload_modules_local oleg613-server: kmemleak disabled modules unloaded. pdsh@oleg613-client: oleg613-client: ssh exited with exit code 2 pdsh@oleg613-client: oleg613-server: ssh exited with exit code 2 pdsh@oleg613-client: oleg613-client: ssh exited with exit code 2 pdsh@oleg613-client: oleg613-server: ssh exited with exit code 2