== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 21:42:40 (1789263760) start mds service on oleg119-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg119-server: oleg119-server.virtnet: executing set_default_debug -1 all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg119-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg119-server: oleg119-server.virtnet: executing set_default_debug -1 all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-OST0000 oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg119-client.virtnet: -o user_xattr,flock 192.168.201.119@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 92553 329 92224 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 12297 393 11904 4% /mnt/lustre[OST:0] filesystem_summary: 12233 329 11904 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close/unlink 1137 (time 1789263789.05 total 10.01 last 113.63) - open/close/unlink 2275 (time 1789263799.06 total 20.02 last 113.69) - open/close/unlink 3486 (time 1789263809.07 total 30.02 last 121.02) - open/close/unlink 4920 (time 1789263819.07 total 40.03 last 143.35) - open/close/unlink 6256 (time 1789263829.07 total 50.03 last 133.57) - open/close/unlink 7622 (time 1789263839.08 total 60.03 last 136.53) - open/close/unlink 9349 (time 1789263849.09 total 70.04 last 172.59) - open/close/unlink 10000 (time 1789263852.91 total 73.86 last 170.40) - open/close/unlink 11494 (time 1789263862.91 total 83.87 last 149.30) - open/close/unlink 13102 (time 1789263872.91 total 93.87 last 160.78) - open/close/unlink 14633 (time 1789263882.93 total 103.88 last 152.92) - open/close/unlink 15999 (time 1789263892.93 total 113.88 last 136.54) - open/close/unlink 17516 (time 1789263902.93 total 123.89 last 151.65) - open/close/unlink 19071 (time 1789263912.93 total 133.89 last 155.48) - open/close/unlink 19910 (time 1789263922.94 total 143.89 last 83.87) - open/close/unlink 20000 (time 1789263925.29 total 146.25 last 38.23) - open/close/unlink 20410 (time 1789263935.31 total 156.26 last 40.94) - open/close/unlink 20707 (time 1789263945.31 total 166.26 last 29.69) - open/close/unlink 21010 (time 1789263955.31 total 176.27 last 30.30) - open/close/unlink 21544 (time 1789263965.32 total 186.27 last 53.39) - open/close/unlink 22151 (time 1789263975.32 total 196.27 last 60.67) - open/close/unlink 22652 (time 1789263985.32 total 206.27 last 50.10) - open/close/unlink 22924 (time 1789263995.32 total 216.28 last 27.19) - open/close/unlink 23335 (time 1789264005.34 total 226.30 last 41.01) - open/close/unlink 23755 (time 1789264015.35 total 236.30 last 41.99) - open/close/unlink 24342 (time 1789264025.35 total 246.31 last 58.67) - open/close/unlink 24963 (time 1789264035.37 total 256.32 last 62.01) - open/close/unlink 25600 (time 1789264045.37 total 266.32 last 63.67) - open/close/unlink 26182 (time 1789264055.38 total 276.33 last 58.15) - open/close/unlink 26927 (time 1789264065.39 total 286.34 last 74.46) - open/close/unlink 27563 (time 1789264075.39 total 296.34 last 63.57) - open/close/unlink 28192 (time 1789264085.40 total 306.36 last 62.82) - open/close/unlink 28854 (time 1789264095.41 total 316.36 last 66.15) - open/close/unlink 29683 (time 1789264105.42 total 326.37 last 82.83) - open/close/unlink 30000 (time 1789264108.57 total 329.53 last 100.53) - open/close/unlink 30894 (time 1789264118.59 total 339.55 last 89.21) - open/close/unlink 31544 (time 1789264128.61 total 349.56 last 64.92) - open/close/unlink 32180 (time 1789264138.61 total 359.56 last 63.58) - open/close/unlink 32723 (time 1789264148.61 total 369.57 last 54.29) - open/close/unlink 33285 (time 1789264158.62 total 379.57 last 56.16) - open/close/unlink 33914 (time 1789264168.62 total 389.57 last 62.90) - open/close/unlink 34567 (time 1789264178.63 total 399.58 last 65.23) - open/close/unlink 35128 (time 1789264188.64 total 409.60 last 56.01) - open/close/unlink 35726 (time 1789264198.65 total 419.60 last 59.78) - open/close/unlink 36351 (time 1789264208.65 total 429.61 last 62.46) - open/close/unlink 37160 (time 1789264218.66 total 439.61 last 80.86) - open/close/unlink 37903 (time 1789264228.67 total 449.62 last 74.25) - open/close/unlink 38622 (time 1789264238.68 total 459.63 last 71.83) - open/close/unlink 39299 (time 1789264248.68 total 469.64 last 67.65) - open/close/unlink 39936 (time 1789264258.69 total 479.64 last 63.68) - open/close/unlink 40529 (time 1789264268.69 total 489.65 last 59.26) - open/close/unlink 41305 (time 1789264278.70 total 499.65 last 77.55) - open/close/unlink 41809 (time 1789264288.71 total 509.66 last 50.35) - open/close/unlink 42268 (time 1789264298.73 total 519.68 last 45.83) - open/close/unlink 42658 (time 1789264308.74 total 529.69 last 38.95) - open/close/unlink 43140 (time 1789264318.74 total 539.69 last 48.19) - open/close/unlink 43600 (time 1789264328.75 total 549.70 last 4 - open/close/unlink 1155 (time 1789263789.17 total 10.00 last 115.48) - open/close/unlink 2309 (time 1789263799.17 total 20.01 last 115.33) - open/close/unlink 3529 (time 1789263809.17 total 30.01 last 121.99) - open/close/unlink 4950 (time 1789263819.18 total 40.01 last 142.07) - open/close/unlink 6280 (time 1789263829.18 total 50.01 last 133.00) - open/close/unlink 7655 (time 1789263839.18 total 60.02 last 137.42) - open/close/unlink 9399 (time 1789263849.19 total 70.02 last 174.34) - open/close/unlink 10000 (time 1789263852.61 total 73.45 last 175.31) - open/close/unlink 11526 (time 1789263862.62 total 83.46 last 152.49) - open/close/unlink 13141 (time 1789263872.64 total 93.47 last 161.21) - open/close/unlink 14687 (time 1789263882.64 total 103.48 last 154.54) - open/close/unlink 16054 (time 1789263892.65 total 113.48 last 136.64) - open/close/unlink 17552 (time 1789263902.65 total 123.49 last 149.73) - open/close/unlink 19135 (time 1789263912.65 total 133.49 last 158.26) - open/close/unlink 19995 (time 1789263922.67 total 143.51 last 85.86) - open/close/unlink 20418 (time 1789263932.68 total 153.52 last 42.25) - open/close/unlink 20706 (time 1789263942.70 total 163.54 last 28.74) - open/close/unlink 21004 (time 1789263952.73 total 173.56 last 29.72) - open/close/unlink 21504 (time 1789263962.74 total 183.58 last 49.94) - open/close/unlink 22130 (time 1789263972.74 total 193.58 last 62.60) - open/close/unlink 22662 (time 1789263982.76 total 203.60 last 53.09) - open/close/unlink 22951 (time 1789263992.77 total 213.60 last 28.88) - open/close/unlink 23339 (time 1789264002.77 total 223.61 last 38.78) - open/close/unlink 23739 (time 1789264012.79 total 233.62 last 39.96) - open/close/unlink 24326 (time 1789264022.79 total 243.63 last 58.65) - open/close/unlink 24935 (time 1789264032.81 total 253.64 last 60.81) - open/close/unlink 25557 (time 1789264042.82 total 263.66 last 62.10) - open/close/unlink 26133 (time 1789264052.84 total 273.67 last 57.53) - open/close/unlink 26811 (time 1789264062.85 total 283.68 last 67.74) - open/close/unlink 27520 (time 1789264072.86 total 293.69 last 70.81) - open/close/unlink 28104 (time 1789264082.86 total 303.70 last 58.37) - open/close/unlink 28768 (time 1789264092.87 total 313.71 last 66.36) - open/close/unlink 29554 (time 1789264102.87 total 323.71 last 78.56) - open/close/unlink 30000 (time 1789264107.43 total 328.26 last 97.96) - open/close/unlink 30915 (time 1789264117.44 total 338.27 last 91.43) - open/close/unlink 31587 (time 1789264127.45 total 348.29 last 67.08) - open/close/unlink 32219 (time 1789264137.46 total 358.29 last 63.19) - open/close/unlink 32771 (time 1789264147.46 total 368.29 last 55.19) - open/close/unlink 33326 (time 1789264157.46 total 378.30 last 55.46) - open/close/unlink 33949 (time 1789264167.47 total 388.31 last 62.25) - open/close/unlink 34606 (time 1789264177.48 total 398.31 last 65.67) - open/close/unlink 35186 (time 1789264187.48 total 408.31 last 58.00) - open/close/unlink 35765 (time 1789264197.48 total 418.32 last 57.86) - open/close/unlink 36383 (time 1789264207.49 total 428.32 last 61.77) - open/close/unlink 37160 (time 1789264217.50 total 438.33 last 77.63) - open/close/unlink 37939 (time 1789264227.50 total 448.34 last 77.88) - open/close/unlink 38662 (time 1789264237.50 total 458.34 last 72.28) - open/close/unlink 39355 (time 1789264247.52 total 468.36 last 69.18) - open/close/unlink 39987 (time 1789264257.53 total 478.36 last 63.15) - open/close/unlink 40579 (time 1789264267.55 total 488.38 last 59.09) - open/close/unlink 41360 (time 1789264277.55 total 498.39 last 78.07) - open/close/unlink 41887 (time 1789264287.56 total 508.40 last 52.64) - open/close/unlink 42351 (time 1789264297.62 total 518.45 last 46.15) - open/close/unlink 42755 (time 1789264307.64 total 528.47 last 40.32) - open/close/unlink 43240 (time 1789264317.66 total 538.49 last 48.39) - open/close/unlink 43707 (time 1789264327.67 total 548.50 last 46.66) - open/close/unlink 44115 (time 1789264337.68 total 558.51 last 40 - open/close/unlink 1143 (time 1789263788.94 total 10.01 last 114.23) - open/close/unlink 2299 (time 1789263798.95 total 20.01 last 115.57) - open/close/unlink 3520 (time 1789263808.95 total 30.01 last 122.09) - open/close/unlink 4940 (time 1789263818.95 total 40.01 last 141.93) - open/close/unlink 6272 (time 1789263828.96 total 50.02 last 133.09) - open/close/unlink 7634 (time 1789263838.97 total 60.03 last 136.14) - open/close/unlink 9363 (time 1789263848.97 total 70.03 last 172.88) - open/close/unlink 10000 (time 1789263852.58 total 73.64 last 176.46) - open/close/unlink 11507 (time 1789263862.58 total 83.64 last 150.67) - open/close/unlink 13107 (time 1789263872.59 total 93.65 last 159.89) - open/close/unlink 14653 (time 1789263882.59 total 103.65 last 154.58) - open/close/unlink 16011 (time 1789263892.59 total 113.65 last 135.72) - open/close/unlink 17526 (time 1789263902.60 total 123.66 last 151.41) - open/close/unlink 19119 (time 1789263912.60 total 133.66 last 159.30) - open/close/unlink 19981 (time 1789263922.62 total 143.68 last 86.06) - open/close/unlink 20412 (time 1789263932.64 total 153.70 last 43.00) - open/close/unlink 20696 (time 1789263942.64 total 163.70 last 28.40) - open/close/unlink 20995 (time 1789263952.67 total 173.73 last 29.80) - open/close/unlink 21479 (time 1789263962.68 total 183.74 last 48.35) - open/close/unlink 22107 (time 1789263972.69 total 193.75 last 62.74) - open/close/unlink 22632 (time 1789263982.70 total 203.76 last 52.45) - open/close/unlink 22925 (time 1789263992.72 total 213.78 last 29.25) - open/close/unlink 23316 (time 1789264002.74 total 223.80 last 39.04) - open/close/unlink 23707 (time 1789264012.75 total 233.81 last 39.05) - open/close/unlink 24299 (time 1789264022.77 total 243.83 last 59.07) - open/close/unlink 24908 (time 1789264032.78 total 253.84 last 60.87) - open/close/unlink 25530 (time 1789264042.78 total 263.85 last 62.15) - open/close/unlink 26121 (time 1789264052.79 total 273.85 last 59.09) - open/close/unlink 26801 (time 1789264062.79 total 283.85 last 67.99) - open/close/unlink 27504 (time 1789264072.79 total 293.85 last 70.26) - open/close/unlink 28083 (time 1789264082.79 total 303.85 last 57.90) - open/close/unlink 28755 (time 1789264092.80 total 313.86 last 67.14) - open/close/unlink 29539 (time 1789264102.81 total 323.87 last 78.35) - open/close/unlink 30000 (time 1789264107.53 total 328.59 last 97.70) - open/close/unlink 30915 (time 1789264117.53 total 338.59 last 91.49) - open/close/unlink 31576 (time 1789264127.53 total 348.59 last 66.09) - open/close/unlink 32217 (time 1789264137.53 total 358.59 last 64.07) - open/close/unlink 32765 (time 1789264147.54 total 368.60 last 54.77) - open/close/unlink 33318 (time 1789264157.54 total 378.60 last 55.30) - open/close/unlink 33944 (time 1789264167.55 total 388.61 last 62.52) - open/close/unlink 34602 (time 1789264177.56 total 398.63 last 65.71) - open/close/unlink 35169 (time 1789264187.57 total 408.63 last 56.65) - open/close/unlink 35753 (time 1789264197.59 total 418.65 last 58.31) - open/close/unlink 36368 (time 1789264207.59 total 428.65 last 61.49) - open/close/unlink 37151 (time 1789264217.59 total 438.65 last 78.29) - open/close/unlink 37929 (time 1789264227.60 total 448.66 last 77.73) - open/close/unlink 38648 (time 1789264237.61 total 458.67 last 71.81) - open/close/unlink 39340 (time 1789264247.61 total 468.67 last 69.19) - open/close/unlink 39978 (time 1789264257.62 total 478.69 last 63.72) - open/close/unlink 40568 (time 1789264267.62 total 488.69 last 59.00) - open/close/unlink 41353 (time 1789264277.63 total 498.69 last 78.44) - open/close/unlink 41882 (time 1789264287.65 total 508.72 last 52.78) - open/close/unlink 42341 (time 1789264297.66 total 518.72 last 45.86) - open/close/unlink 42741 (time 1789264307.67 total 528.74 last 39.95) - open/close/unlink 43211 (time 1789264317.70 total 538.76 last 46.86) - open/close/unlink 43691 (time 1789264327.71 total 548.77 last 47.99) - open/close/unlink 44098 (time 1789264337.73 total 558.79 last 40 - open/close/unlink 1151 (time 1789263789.11 total 10.00 last 115.07) - open/close/unlink 2303 (time 1789263799.11 total 20.00 last 115.19) - open/close/unlink 3521 (time 1789263809.11 total 30.01 last 121.78) - open/close/unlink 4947 (time 1789263819.12 total 40.01 last 142.56) - open/close/unlink 6279 (time 1789263829.12 total 50.01 last 133.19) - open/close/unlink 7632 (time 1789263839.12 total 60.01 last 135.29) - open/close/unlink 9371 (time 1789263849.12 total 70.01 last 173.88) - open/close/unlink 10000 (time 1789263852.76 total 73.66 last 172.60) - open/close/unlink 11516 (time 1789263862.77 total 83.66 last 151.47) - open/close/unlink 13138 (time 1789263872.78 total 93.67 last 162.06) - open/close/unlink 14676 (time 1789263882.80 total 103.70 last 153.45) - open/close/unlink 16046 (time 1789263892.85 total 113.74 last 136.34) - open/close/unlink 17580 (time 1789263902.86 total 123.75 last 153.35) - open/close/unlink 19139 (time 1789263912.86 total 133.75 last 155.80) - open/close/unlink 19984 (time 1789263922.87 total 143.76 last 84.44) - open/close/unlink 20405 (time 1789263932.88 total 153.77 last 42.07) - open/close/unlink 20684 (time 1789263942.90 total 163.80 last 27.82) - open/close/unlink 20981 (time 1789263952.91 total 173.81 last 29.67) - open/close/unlink 21473 (time 1789263962.93 total 183.82 last 49.12) - open/close/unlink 22086 (time 1789263972.94 total 193.84 last 61.22) - open/close/unlink 22618 (time 1789263982.95 total 203.84 last 53.16) - open/close/unlink 22909 (time 1789263992.97 total 213.86 last 29.05) - open/close/unlink 23299 (time 1789264002.99 total 223.88 last 38.92) - open/close/unlink 23695 (time 1789264013.01 total 233.90 last 39.54) - open/close/unlink 24273 (time 1789264023.02 total 243.92 last 57.69) - open/close/unlink 24886 (time 1789264033.04 total 253.93 last 61.20) - open/close/unlink 25508 (time 1789264043.05 total 263.94 last 62.15) - open/close/unlink 26089 (time 1789264053.06 total 273.95 last 58.07) - open/close/unlink 26771 (time 1789264063.06 total 283.96 last 68.14) - open/close/unlink 27480 (time 1789264073.08 total 293.97 last 70.79) - open/close/unlink 28065 (time 1789264083.09 total 303.98 last 58.43) - open/close/unlink 28724 (time 1789264093.10 total 313.99 last 65.83) - open/close/unlink 29509 (time 1789264103.10 total 324.00 last 78.48) - open/close/unlink 30000 (time 1789264108.22 total 329.11 last 95.95) - open/close/unlink 30903 (time 1789264118.22 total 339.12 last 90.28) - open/close/unlink 31547 (time 1789264128.22 total 349.12 last 64.40) - open/close/unlink 32187 (time 1789264138.24 total 359.13 last 63.93) - open/close/unlink 32726 (time 1789264148.26 total 369.15 last 53.80) - open/close/unlink 33294 (time 1789264158.27 total 379.16 last 56.71) - open/close/unlink 33932 (time 1789264168.28 total 389.17 last 63.77) - open/close/unlink 34572 (time 1789264178.29 total 399.18 last 63.91) - open/close/unlink 35136 (time 1789264188.29 total 409.19 last 56.38) - open/close/unlink 35716 (time 1789264198.30 total 419.19 last 57.96) - open/close/unlink 36347 (time 1789264208.31 total 429.20 last 63.04) - open/close/unlink 37151 (time 1789264218.31 total 439.21 last 80.36) - open/close/unlink 37902 (time 1789264228.32 total 449.21 last 75.04) - open/close/unlink 38620 (time 1789264238.33 total 459.22 last 71.74) - open/close/unlink 39305 (time 1789264248.34 total 469.23 last 68.46) - open/close/unlink 39939 (time 1789264258.35 total 479.24 last 63.34) - open/close/unlink 40526 (time 1789264268.35 total 489.24 last 58.68) - open/close/unlink 41307 (time 1789264278.35 total 499.24 last 78.08) - open/close/unlink 41820 (time 1789264288.35 total 509.24 last 51.30) - open/close/unlink 42278 (time 1789264298.36 total 519.25 last 45.75) - open/close/unlink 42663 (time 1789264308.39 total 529.29 last 38.38) - open/close/unlink 43152 (time 1789264318.41 total 539.31 last 48.80) - open/close/unlink 43622 (time 1789264328.42 total 549.31 last 46.98) - open/close/unlink 44037 (time 1789264338.43 total 559.32 last 41 - open/close/unlink 1153 (time 1789263789.13 total 10.00 last 115.27) - open/close/unlink 2301 (time 1789263799.13 total 20.01 last 114.77) - open/close/unlink 3534 (time 1789263809.14 total 30.01 last 123.23) - open/close/unlink 4968 (time 1789263819.14 total 40.02 last 143.33) - open/close/unlink 6299 (time 1789263829.15 total 50.02 last 133.04) - open/close/unlink 7669 (time 1789263839.15 total 60.02 last 136.99) - open/close/unlink 9411 (time 1789263849.16 total 70.03 last 174.05) - open/close/unlink 10000 (time 1789263852.50 total 73.37 last 176.19) - open/close/unlink 11501 (time 1789263862.50 total 83.37 last 150.07) - open/close/unlink 13097 (time 1789263872.50 total 93.38 last 159.54) - open/close/unlink 14627 (time 1789263882.51 total 103.38 last 152.92) - open/close/unlink 15989 (time 1789263892.51 total 113.39 last 136.16) - open/close/unlink 17481 (time 1789263902.51 total 123.39 last 149.18) - open/close/unlink 19055 (time 1789263912.52 total 133.39 last 157.33) - open/close/unlink 19925 (time 1789263922.52 total 143.39 last 86.99) - open/close/unlink 20355 (time 1789263932.52 total 153.40 last 42.99) - open/close/unlink 20640 (time 1789263942.53 total 163.40 last 28.49) - open/close/unlink 20939 (time 1789263952.57 total 173.45 last 29.76) - open/close/unlink 21416 (time 1789263962.58 total 183.45 last 47.69) - open/close/unlink 22046 (time 1789263972.59 total 193.47 last 62.89) - open/close/unlink 22571 (time 1789263982.61 total 203.48 last 52.42) - open/close/unlink 22867 (time 1789263992.63 total 213.50 last 29.55) - open/close/unlink 23255 (time 1789264002.64 total 223.51 last 38.74) - open/close/unlink 23644 (time 1789264012.65 total 233.52 last 38.87) - open/close/unlink 24223 (time 1789264022.67 total 243.54 last 57.80) - open/close/unlink 24832 (time 1789264032.67 total 253.55 last 60.85) - open/close/unlink 25457 (time 1789264042.68 total 263.55 last 62.46) - open/close/unlink 26044 (time 1789264052.70 total 273.57 last 58.61) - open/close/unlink 26715 (time 1789264062.70 total 283.57 last 67.08) - open/close/unlink 27421 (time 1789264072.71 total 293.59 last 70.51) - open/close/unlink 28002 (time 1789264082.71 total 303.59 last 58.08) - open/close/unlink 28668 (time 1789264092.72 total 313.59 last 66.56) - open/close/unlink 29450 (time 1789264102.73 total 323.60 last 78.11) - open/close/unlink 30000 (time 1789264108.57 total 329.45 last 94.17) - open/close/unlink 30900 (time 1789264118.58 total 339.45 last 89.92) - open/close/unlink 31559 (time 1789264128.58 total 349.46 last 65.87) - open/close/unlink 32195 (time 1789264138.60 total 359.47 last 63.50) - open/close/unlink 32734 (time 1789264148.61 total 369.48 last 53.85) - open/close/unlink 33301 (time 1789264158.62 total 379.49 last 56.66) - open/close/unlink 33935 (time 1789264168.62 total 389.49 last 63.36) - open/close/unlink 34580 (time 1789264178.63 total 399.51 last 64.42) - open/close/unlink 35137 (time 1789264188.64 total 409.51 last 55.68) - open/close/unlink 35727 (time 1789264198.64 total 419.51 last 58.98) - open/close/unlink 36355 (time 1789264208.65 total 429.53 last 62.72) - open/close/unlink 37165 (time 1789264218.66 total 439.53 last 80.98) - open/close/unlink 37915 (time 1789264228.66 total 449.53 last 74.99) - open/close/unlink 38628 (time 1789264238.67 total 459.54 last 71.24) - open/close/unlink 39314 (time 1789264248.67 total 469.54 last 68.59) - open/close/unlink 39951 (time 1789264258.67 total 479.54 last 63.69) - open/close/unlink 40543 (time 1789264268.68 total 489.55 last 59.16) - open/close/unlink 41321 (time 1789264278.68 total 499.55 last 77.78) - open/close/unlink 41833 (time 1789264288.69 total 509.56 last 51.15) - open/close/unlink 42295 (time 1789264298.70 total 519.58 last 46.13) - open/close/unlink 42670 (time 1789264308.71 total 529.58 last 37.49) - open/close/unlink 43148 (time 1789264318.72 total 539.59 last 47.75) - open/close/unlink 43610 (time 1789264328.74 total 549.62 last 46.07) - open/close/unlink 44020 (time 1789264338.75 total 559.62 last 40.76) - open/close/unlink 44575 (time 1789264347.70 total 568.53 last 45.91) - open/close/unlink 45143 (time 1789264357.70 total 578.54 last 56.77) - open/close/unlink 45671 (time 1789264367.71 total 588.55 last 52.76) - open/close/unlink 46165 (time 1789264377.71 total 598.55 last 49.39) - open/close/unlink 46783 (time 1789264387.72 total 608.56 last 61.73) - open/close/unlink 47403 (time 1789264397.73 total 618.56 last 61.98) - open/close/unlink 48083 (time 1789264407.74 total 628.57 last 67.93) - open/close/unlink 48705 (time 1789264417.74 total 638.58 last 62.17) - open/close/unlink 49458 (time 1789264427.75 total 648.58 last 75.26) - open/close/unlink 50000 (time 1789264435.59 total 656.42 last 69.11) - open/close/unlink 50657 (time 1789264445.60 total 666.44 last 65.60) - open/close/unlink 51326 (time 1789264455.61 total 676.44 last 66.86) - open/close/unlink 51997 (time 1789264465.61 total 686.45 last 67.09) - open/close/unlink 52719 (time 1789264475.62 total 696.46 last 72.11) - open/close/unlink 53462 (time 1789264485.64 total 706.47 last 74.20) - open/close/unlink 53984 (time 1789264495.64 total 716.48 last 52.17) - open/close/unlink 54554 (time 1789264505.64 total 726.48 last 57.00) - open/close/unlink 55181 (time 1789264515.65 total 736.49 last 62.65) - open/close/unlink 55964 (time 1789264525.66 total 746.49 last 78.23) - open/close/unlink 56766 (time 1789264535.66 total 756.50 last 80.15) - open/close/unlink 57520 (time 1789264545.67 total 766.51 last 75.36) - open/close/unlink 58131 (time 1789264555.67 total 776.51 last 61.08) - open/close/unlink 58771 (time 1789264565.69 total 786.52 last 63.92) - open/close/unlink 59405 (time 1789264575.69 total 796.52 last 63.40) - open/close/unlink 60000 (time 1789264584.84 total 805.68 last 64.99) - open/close/unlink 60611 (time 1789264594.85 total 815.68 last 61.07) - open/close/unlink 61057 (time 1789264604.87 total 825.71 last 44.48) - open/close/unlink 61620 (time 1789264614.90 total 835.73 last 56.17) - open/close/unlink 62198 (time 1789264624.92 total 845.75 last 57.69) - open/close/unlink 62827 (time 1789264634.92 total 855.75 last 62.88) - open/close/unlink 63432 (time 1789264644.92 total 865.75 last 60.50) - open/close/unlink 63952 (time 1789264654.92 total 875.76 last 51.99) - open/close/unlink 64510 (time 1789264664.94 total 885.77 last 55.71) total: 64768 open/close/unlink in 891.04 seconds: 72.69 ops/second .59) - open/close/unlink 44562 (time 1789264347.75 total 568.81 last 46.34) - open/close/unlink 45136 (time 1789264357.76 total 578.82 last 57.30) - open/close/unlink 45662 (time 1789264367.77 total 588.83 last 52.58) - open/close/unlink 46160 (time 1789264377.78 total 598.84 last 49.73) - open/close/unlink 46779 (time 1789264387.79 total 608.85 last 61.84) - open/close/unlink 47398 (time 1789264397.79 total 618.86 last 61.88) - open/close/unlink 48074 (time 1789264407.81 total 628.87 last 67.49) - open/close/unlink 48695 (time 1789264417.82 total 638.88 last 62.05) - open/close/unlink 49448 (time 1789264427.83 total 648.89 last 75.21) - open/close/unlink 50000 (time 1789264435.97 total 657.03 last 67.83) - open/close/unlink 50654 (time 1789264445.98 total 667.04 last 65.35) - open/close/unlink 51326 (time 1789264455.99 total 677.05 last 67.15) - open/close/unlink 52012 (time 1789264465.99 total 687.05 last 68.56) - open/close/unlink 52727 (time 1789264475.99 total 697.05 last 71.50) - open/close/unlink 53463 (time 1789264486.01 total 707.08 last 73.43) - open/close/unlink 53989 (time 1789264496.03 total 717.09 last 52.50) - open/close/unlink 54554 (time 1789264506.04 total 727.10 last 56.48) - open/close/unlink 55195 (time 1789264516.04 total 737.10 last 64.08) - open/close/unlink 55980 (time 1789264526.05 total 747.11 last 78.41) - open/close/unlink 56800 (time 1789264536.05 total 757.11 last 82.00) - open/close/unlink 57532 (time 1789264546.07 total 767.13 last 73.06) - open/close/unlink 58143 (time 1789264556.09 total 777.15 last 61.00) - open/close/unlink 58788 (time 1789264566.10 total 787.16 last 64.41) - open/close/unlink 59421 (time 1789264576.11 total 797.17 last 63.24) - open/close/unlink 60000 (time 1789264585.00 total 806.06 last 65.14) - open/close/unlink 60615 (time 1789264595.01 total 816.07 last 61.46) - open/close/unlink 61053 (time 1789264605.03 total 826.09 last 43.69) - open/close/unlink 61606 (time 1789264615.05 total 836.11 last 55.18) - open/close/unlink 62177 (time 1789264625.07 total 846.13 last 57.02) - open/close/unlink 62808 (time 1789264635.07 total 856.13 last 63.08) - open/close/unlink 63415 (time 1789264645.07 total 866.14 last 60.67) - open/close/unlink 63944 (time 1789264655.09 total 876.15 last 52.82) - open/close/unlink 64502 (time 1789264665.10 total 886.16 last 55.74) total: 64768 open/close/unlink in 891.76 seconds: 72.63 ops/second .47) - open/close/unlink 44511 (time 1789264348.45 total 569.34 last 47.29) - open/close/unlink 45089 (time 1789264358.48 total 579.37 last 57.65) - open/close/unlink 45607 (time 1789264368.49 total 589.39 last 51.71) - open/close/unlink 46118 (time 1789264378.50 total 599.39 last 51.09) - open/close/unlink 46747 (time 1789264388.51 total 609.40 last 62.83) - open/close/unlink 47357 (time 1789264398.52 total 619.42 last 60.89) - open/close/unlink 48034 (time 1789264408.53 total 629.42 last 67.67) - open/close/unlink 48668 (time 1789264418.53 total 639.43 last 63.37) - open/close/unlink 49411 (time 1789264428.53 total 649.43 last 74.30) - open/close/unlink 50000 (time 1789264437.33 total 658.22 last 66.95) - open/close/unlink 50653 (time 1789264447.34 total 668.23 last 65.24) - open/close/unlink 51325 (time 1789264457.35 total 678.24 last 67.15) - open/close/unlink 52022 (time 1789264467.37 total 688.26 last 69.58) - open/close/unlink 52736 (time 1789264477.37 total 698.27 last 71.34) - open/close/unlink 53444 (time 1789264487.39 total 708.28 last 70.69) - open/close/unlink 53980 (time 1789264497.41 total 718.30 last 53.50) - open/close/unlink 54565 (time 1789264507.42 total 728.31 last 58.41) - open/close/unlink 55225 (time 1789264517.43 total 738.32 last 65.97) - open/close/unlink 56021 (time 1789264527.44 total 748.33 last 79.52) - open/close/unlink 56857 (time 1789264537.44 total 758.33 last 83.58) - open/close/unlink 57557 (time 1789264547.45 total 768.34 last 69.92) - open/close/unlink 58157 (time 1789264557.45 total 778.34 last 60.00) - open/close/unlink 58797 (time 1789264567.46 total 788.35 last 63.93) - open/close/unlink 59431 (time 1789264577.47 total 798.36 last 63.35) - open/close/unlink 60000 (time 1789264585.88 total 806.77 last 67.66) - open/close/unlink 60571 (time 1789264595.89 total 816.78 last 57.03) - open/close/unlink 61017 (time 1789264605.90 total 826.79 last 44.57) - open/close/unlink 61597 (time 1789264615.91 total 836.80 last 57.96) - open/close/unlink 62164 (time 1789264625.91 total 846.80 last 56.68) - open/close/unlink 62793 (time 1789264635.91 total 856.80 last 62.88) - open/close/unlink 63400 (time 1789264645.91 total 866.81 last 60.69) - open/close/unlink 63919 (time 1789264655.93 total 876.82 last 51.82) - open/close/unlink 64482 (time 1789264665.95 total 886.84 last 56.18) total: 64768 open/close/unlink in 894.01 seconds: 72.45 ops/second .98) - open/close/unlink 44488 (time 1789264348.75 total 569.62 last 46.80) - open/close/unlink 45063 (time 1789264358.75 total 579.62 last 57.49) - open/close/unlink 45583 (time 1789264368.76 total 589.64 last 51.93) - open/close/unlink 46093 (time 1789264378.76 total 599.64 last 51.00) - open/close/unlink 46733 (time 1789264388.78 total 609.65 last 63.93) - open/close/unlink 47344 (time 1789264398.79 total 619.66 last 61.02) - open/close/unlink 48019 (time 1789264408.79 total 629.66 last 67.50) - open/close/unlink 48663 (time 1789264418.80 total 639.67 last 64.34) - open/close/unlink 49399 (time 1789264428.81 total 649.69 last 73.49) - open/close/unlink 50000 (time 1789264437.71 total 658.58 last 67.55) - open/close/unlink 50648 (time 1789264447.72 total 668.59 last 64.76) - open/close/unlink 51326 (time 1789264457.72 total 678.60 last 67.76) - open/close/unlink 52018 (time 1789264467.74 total 688.61 last 69.08) - open/close/unlink 52742 (time 1789264477.76 total 698.64 last 72.22) - open/close/unlink 53430 (time 1789264487.77 total 708.65 last 68.74) - open/close/unlink 53964 (time 1789264497.79 total 718.66 last 53.32) - open/close/unlink 54543 (time 1789264507.80 total 728.67 last 57.85) - open/close/unlink 55208 (time 1789264517.81 total 738.68 last 66.44) - open/close/unlink 56013 (time 1789264527.81 total 748.69 last 80.42) - open/close/unlink 56851 (time 1789264537.84 total 758.71 last 83.63) - open/close/unlink 57551 (time 1789264547.85 total 768.72 last 69.91) - open/close/unlink 58157 (time 1789264557.86 total 778.73 last 60.53) - open/close/unlink 58797 (time 1789264567.88 total 788.75 last 63.90) - open/close/unlink 59432 (time 1789264577.88 total 798.76 last 63.45) - open/close/unlink 60000 (time 1789264586.30 total 807.18 last 67.46) - open/close/unlink 60561 (time 1789264596.31 total 817.18 last 56.07) - open/close/unlink 61014 (time 1789264606.32 total 827.19 last 45.27) - open/close/unlink 61590 (time 1789264616.32 total 837.19 last 57.58) - open/close/unlink 62155 (time 1789264626.33 total 847.21 last 56.42) - open/close/unlink 62789 (time 1789264636.33 total 857.21 last 63.39) - open/close/unlink 63394 (time 1789264646.34 total 867.21 last 60.47) - open/close/unlink 63908 (time 1789264656.34 total 877.22 last 51.38) - open/close/unlink 64473 (time 1789264666.36 total 887.23 last 56.41) total: 64768 open/close/unlink in 894.82 seconds: 72.38 ops/second 5.97) - open/close/unlink 44006 (time 1789264338.75 total 559.70 last 40.58) - open/close/unlink 44488 (time 1789264348.75 total 569.71 last 48.17) - open/close/unlink 45070 (time 1789264358.76 total 579.72 last 58.16) - open/close/unlink 45590 (time 1789264368.79 total 589.75 last 51.84) - open/close/unlink 46103 (time 1789264378.80 total 599.75 last 51.29) - open/close/unlink 46736 (time 1789264388.80 total 609.76 last 63.25) - open/close/unlink 47334 (time 1789264398.81 total 619.76 last 59.79) - open/close/unlink 48009 (time 1789264408.81 total 629.77 last 67.44) - open/close/unlink 48651 (time 1789264418.82 total 639.77 last 64.16) - open/close/unlink 49392 (time 1789264428.83 total 649.78 last 74.03) - open/close/unlink 50000 (time 1789264437.77 total 658.72 last 68.00) - open/close/unlink 50643 (time 1789264447.77 total 668.73 last 64.30) - open/close/unlink 51312 (time 1789264457.78 total 678.74 last 66.81) - open/close/unlink 52007 (time 1789264467.79 total 688.75 last 69.43) - open/close/unlink 52726 (time 1789264477.80 total 698.75 last 71.88) - open/close/unlink 53421 (time 1789264487.81 total 708.76 last 69.43) - open/close/unlink 53949 (time 1789264497.81 total 718.76 last 52.79) - open/close/unlink 54533 (time 1789264507.81 total 728.77 last 58.37) - open/close/unlink 55184 (time 1789264517.83 total 738.78 last 65.03) - open/close/unlink 55977 (time 1789264527.83 total 748.79 last 79.23) - open/close/unlink 56809 (time 1789264537.86 total 758.81 last 83.00) - open/close/unlink 57509 (time 1789264547.87 total 768.83 last 69.89) - open/close/unlink 58108 (time 1789264557.89 total 778.85 last 59.79) - open/close/unlink 58741 (time 1789264567.89 total 788.85 last 63.29) - open/close/unlink 59383 (time 1789264577.90 total 798.85 last 64.18) - open/close/unlink 60000 (time 1789264587.12 total 808.07 last 66.93) - open/close/unlink 60552 (time 1789264597.13 total 818.08 last 55.14) - open/close/unlink 61001 (time 1789264607.13 total 828.08 last 44.90) - open/close/unlink 61585 (time 1789264617.13 total 838.08 last 58.39) - open/close/unlink 62159 (time 1789264627.13 total 848.09 last 57.37) - open/close/unlink 62788 (time 1789264637.15 total 858.10 last 62.79) - open/close/unlink 63401 (time 1789264647.16 total 868.11 last 61.25) - open/close/unlink 63910 (time 1789264657.16 total 878.12 last 50.88) - open/close/unlink 64470 (time 1789264667.18 total 888.14 last 55.88) total: 64768 open/close/unlink in 896.12 seconds: 72.28 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete oleg119-server: cannot open 'lustre-ost2': no such pool pdsh@oleg119-client: oleg119-server: ssh exited with exit code 255 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg119-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg119-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg119-server stop mds service on oleg119-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg119-server unloading modules via unload_modules_local on: 'oleg119-server' oleg119-server: oleg119-server.virtnet: executing unload_modules_local oleg119-server: kmemleak disabled modules unloaded. pdsh@oleg119-client: oleg119-server: ssh exited with exit code 2 pdsh@oleg119-client: oleg119-client: ssh exited with exit code 2 pdsh@oleg119-client: oleg119-client: ssh exited with exit code 2 pdsh@oleg119-client: oleg119-server: ssh exited with exit code 2