== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 12:36:21 (1788712581) start mds service on oleg155-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg155-server: oleg155-server.virtnet: executing set_default_debug -1 all pdsh@oleg155-client: oleg155-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg155-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg155-server: oleg155-server.virtnet: executing set_default_debug -1 all pdsh@oleg155-client: oleg155-server: ssh exited with exit code 1 Started lustre-OST0000 oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg155-client.virtnet: -o user_xattr,flock 192.168.201.155@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 92553 329 92224 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 12297 393 11904 4% /mnt/lustre[OST:0] filesystem_summary: 12233 329 11904 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close/unlink 582 (time 1788712628.63 total 10.01 last 58.16) - open/close/unlink 1858 (time 1788712638.63 total 20.01 last 127.58) - open/close/unlink 3335 (time 1788712648.63 total 30.01 last 147.67) - open/close/unlink 4750 (time 1788712658.64 total 40.02 last 141.41) - open/close/unlink 6124 (time 1788712668.64 total 50.02 last 137.38) - open/close/unlink 7390 (time 1788712678.65 total 60.03 last 126.45) - open/close/unlink 8825 (time 1788712688.66 total 70.04 last 143.40) - open/close/unlink 10000 (time 1788712697.85 total 79.23 last 127.87) - open/close/unlink 11301 (time 1788712707.86 total 89.24 last 129.93) - open/close/unlink 12738 (time 1788712717.86 total 99.24 last 143.64) - open/close/unlink 14152 (time 1788712727.87 total 109.25 last 141.28) - open/close/unlink 15490 (time 1788712737.88 total 119.26 last 133.69) - open/close/unlink 17010 (time 1788712747.89 total 129.27 last 151.91) - open/close/unlink 18854 (time 1788712757.90 total 139.28 last 184.13) - open/close/unlink 20000 (time 1788712765.22 total 146.60 last 156.53) - open/close/unlink 21509 (time 1788712775.22 total 156.60 last 150.88) - open/close/unlink 23074 (time 1788712785.23 total 166.61 last 156.43) - open/close/unlink 24606 (time 1788712795.23 total 176.61 last 153.14) - open/close/unlink 25963 (time 1788712805.23 total 186.61 last 135.68) - open/close/unlink 27384 (time 1788712815.24 total 196.62 last 142.03) - open/close/unlink 28871 (time 1788712825.24 total 206.62 last 148.67) - open/close/unlink 30000 (time 1788712834.00 total 215.38 last 128.85) - open/close/unlink 31179 (time 1788712844.01 total 225.39 last 117.81) - open/close/unlink 32440 (time 1788712854.02 total 235.39 last 126.04) - open/close/unlink 33602 (time 1788712864.02 total 245.40 last 116.19) - open/close/unlink 35111 (time 1788712874.02 total 255.40 last 150.89) - open/close/unlink 36642 (time 1788712884.02 total 265.40 last 153.09) - open/close/unlink 38165 (time 1788712894.02 total 275.40 last 152.27) - open/close/unlink 39690 (time 1788712904.02 total 285.40 last 152.47) - open/close/unlink 41177 (time 1788712914.02 total 295.40 last 148.69) - open/close/unlink 42678 (time 1788712924.02 total 305.40 last 150.08) - open/close/unlink 44093 (time 1788712934.02 total 315.40 last 141.50) - open/close/unlink 45378 (time 1788712944.02 total 325.40 last 128.49) - open/close/unlink 46523 (time 1788712954.03 total 335.40 last 114.49) - open/close/unlink 47681 (time 1788712964.03 total 345.41 last 115.76) - open/close/unlink 48973 (time 1788712974.03 total 355.41 last 129.19) - open/close/unlink 50000 (time 1788712982.47 total 363.85 last 121.64) - open/close/unlink 51175 (time 1788712992.47 total 373.85 last 117.49) - open/close/unlink 52561 (time 1788713002.48 total 383.86 last 138.57) - open/close/unlink 53928 (time 1788713012.48 total 393.86 last 136.68) - open/close/unlink 55301 (time 1788713022.48 total 403.86 last 137.28) - open/close/unlink 56704 (time 1788713032.49 total 413.87 last 140.14) - open/close/unlink 58240 (time 1788713042.49 total 423.87 last 153.54) - open/close/unlink 59573 (time 1788713052.51 total 433.89 last 133.09) - open/close/unlink 60000 (time 1788713056.53 total 437.91 last 106.29) - open/close/unlink 61256 (time 1788713066.53 total 447.91 last 125.60) - open/close/unlink 62713 (time 1788713076.53 total 457.91 last 145.63) - open/close/unlink 64038 (time 1788713086.54 total 467.92 last 132.43) total: 64768 open/close/unlink in 474.09 seconds: 136.62 ops/second - open/close/unlink 598 (time 1788712628.93 total 10.01 last 59.76) - open/close/unlink 1885 (time 1788712638.93 total 20.01 last 128.65) - open/close/unlink 3357 (time 1788712648.94 total 30.02 last 147.11) - open/close/unlink 4769 (time 1788712658.94 total 40.02 last 141.16) - open/close/unlink 6138 (time 1788712668.94 total 50.02 last 136.86) - open/close/unlink 7396 (time 1788712678.95 total 60.03 last 125.76) - open/close/unlink 8820 (time 1788712688.95 total 70.03 last 142.32) - open/close/unlink 10000 (time 1788712698.06 total 79.14 last 129.51) - open/close/unlink 11292 (time 1788712708.07 total 89.14 last 129.19) - open/close/unlink 12696 (time 1788712718.07 total 99.15 last 140.37) - open/close/unlink 14109 (time 1788712728.08 total 109.15 last 141.17) - open/close/unlink 15415 (time 1788712738.08 total 119.16 last 130.52) - open/close/unlink 16964 (time 1788712748.09 total 129.16 last 154.84) - open/close/unlink 18786 (time 1788712758.09 total 139.17 last 182.12) - open/close/unlink 20000 (time 1788712765.88 total 146.96 last 155.89) - open/close/unlink 21519 (time 1788712775.88 total 156.96 last 151.83) - open/close/unlink 23075 (time 1788712785.89 total 166.97 last 155.55) - open/close/unlink 24626 (time 1788712795.89 total 176.97 last 155.02) - open/close/unlink 25928 (time 1788712805.89 total 186.97 last 130.18) - open/close/unlink 27381 (time 1788712815.90 total 196.97 last 145.27) - open/close/unlink 28815 (time 1788712825.90 total 206.98 last 143.37) - open/close/unlink 30000 (time 1788712835.62 total 216.70 last 121.86) - open/close/unlink 31207 (time 1788712845.63 total 226.71 last 120.60) - open/close/unlink 32462 (time 1788712855.63 total 236.71 last 125.46) - open/close/unlink 33641 (time 1788712865.63 total 246.71 last 117.89) - open/close/unlink 35237 (time 1788712875.65 total 256.73 last 159.31) - open/close/unlink 36796 (time 1788712885.65 total 266.73 last 155.88) - open/close/unlink 38321 (time 1788712895.66 total 276.74 last 152.45) - open/close/unlink 39843 (time 1788712905.66 total 286.74 last 152.09) - open/close/unlink 41334 (time 1788712915.67 total 296.75 last 149.03) - open/close/unlink 42795 (time 1788712925.67 total 306.75 last 146.06) - open/close/unlink 44247 (time 1788712935.68 total 316.75 last 145.12) - open/close/unlink 45525 (time 1788712945.68 total 326.76 last 127.73) - open/close/unlink 46639 (time 1788712955.69 total 336.77 last 111.33) - open/close/unlink 47812 (time 1788712965.69 total 346.77 last 117.26) - open/close/unlink 49141 (time 1788712975.70 total 356.77 last 132.84) - open/close/unlink 50000 (time 1788712983.20 total 364.28 last 114.49) - open/close/unlink 51196 (time 1788712993.20 total 374.28 last 119.59) - open/close/unlink 52575 (time 1788713003.21 total 384.29 last 137.75) - open/close/unlink 53967 (time 1788713013.22 total 394.29 last 139.12) - open/close/unlink 55380 (time 1788713023.22 total 404.29 last 141.30) - open/close/unlink 56787 (time 1788713033.22 total 414.30 last 140.69) - open/close/unlink 58329 (time 1788713043.22 total 424.30 last 154.14) - open/close/unlink 59615 (time 1788713053.23 total 434.31 last 128.53) - open/close/unlink 60000 (time 1788713056.76 total 437.84 last 108.96) - open/close/unlink 61259 (time 1788713066.76 total 447.84 last 125.88) - open/close/unlink 62700 (time 1788713076.77 total 457.84 last 144.06) - open/close/unlink 64003 (time 1788713086.77 total 467.85 last 130.26) total: 64768 open/close/unlink in 474.48 seconds: 136.50 ops/second - open/close/unlink 618 (time 1788712629.08 total 10.01 last 61.73) - open/close/unlink 1923 (time 1788712639.08 total 20.01 last 130.49) - open/close/unlink 3387 (time 1788712649.09 total 30.02 last 146.29) - open/close/unlink 4820 (time 1788712659.09 total 40.02 last 143.24) - open/close/unlink 6189 (time 1788712669.09 total 50.02 last 136.89) - open/close/unlink 7441 (time 1788712679.10 total 60.02 last 125.18) - open/close/unlink 8870 (time 1788712689.10 total 70.03 last 142.82) - open/close/unlink 10000 (time 1788712697.83 total 78.76 last 129.39) - open/close/unlink 11280 (time 1788712707.86 total 88.79 last 127.62) - open/close/unlink 12694 (time 1788712717.87 total 98.80 last 141.36) - open/close/unlink 14096 (time 1788712727.88 total 108.80 last 140.08) - open/close/unlink 15416 (time 1788712737.88 total 118.81 last 131.95) - open/close/unlink 16938 (time 1788712747.89 total 128.82 last 152.04) - open/close/unlink 18783 (time 1788712757.90 total 138.83 last 184.30) - open/close/unlink 20000 (time 1788712765.78 total 146.71 last 154.49) - open/close/unlink 21516 (time 1788712775.78 total 156.71 last 151.54) - open/close/unlink 23068 (time 1788712785.79 total 166.71 last 155.14) - open/close/unlink 24628 (time 1788712795.79 total 176.72 last 155.99) - open/close/unlink 25933 (time 1788712805.79 total 186.72 last 130.45) - open/close/unlink 27373 (time 1788712815.79 total 196.72 last 143.98) - open/close/unlink 28811 (time 1788712825.80 total 206.73 last 143.62) - open/close/unlink 30000 (time 1788712835.64 total 216.57 last 120.85) - open/close/unlink 31203 (time 1788712845.65 total 226.58 last 120.25) - open/close/unlink 32468 (time 1788712855.66 total 236.59 last 126.34) - open/close/unlink 33639 (time 1788712865.67 total 246.59 last 117.04) - open/close/unlink 35222 (time 1788712875.67 total 256.60 last 158.26) - open/close/unlink 36773 (time 1788712885.67 total 266.60 last 155.03) - open/close/unlink 38298 (time 1788712895.70 total 276.63 last 152.14) - open/close/unlink 39823 (time 1788712905.70 total 286.63 last 152.45) - open/close/unlink 41302 (time 1788712915.70 total 296.63 last 147.90) - open/close/unlink 42794 (time 1788712925.71 total 306.63 last 149.13) - open/close/unlink 44231 (time 1788712935.71 total 316.63 last 143.69) - open/close/unlink 45502 (time 1788712945.71 total 326.64 last 127.04) - open/close/unlink 46636 (time 1788712955.71 total 336.64 last 113.40) - open/close/unlink 47792 (time 1788712965.72 total 346.64 last 115.54) - open/close/unlink 49102 (time 1788712975.72 total 356.65 last 130.93) - open/close/unlink 50000 (time 1788712983.92 total 364.85 last 109.48) - open/close/unlink 51204 (time 1788712993.94 total 374.86 last 120.26) - open/close/unlink 52586 (time 1788713003.95 total 384.88 last 138.04) - open/close/unlink 53983 (time 1788713013.96 total 394.89 last 139.48) - open/close/unlink 55412 (time 1788713023.98 total 404.91 last 142.65) - open/close/unlink 56871 (time 1788713033.98 total 414.91 last 145.84) - open/close/unlink 58428 (time 1788713043.99 total 424.92 last 155.59) - open/close/unlink 59681 (time 1788713053.99 total 434.92 last 125.26) - open/close/unlink 60000 (time 1788713057.06 total 437.98 last 104.20) - open/close/unlink 61279 (time 1788713067.06 total 447.99 last 127.86) - open/close/unlink 62711 (time 1788713077.06 total 457.99 last 143.19) - open/close/unlink 63999 (time 1788713087.06 total 467.99 last 128.74) total: 64768 open/close/unlink in 474.81 seconds: 136.41 ops/second - open/close/unlink 612 (time 1788712628.99 total 10.00 last 61.17) - open/close/unlink 1914 (time 1788712639.00 total 20.01 last 130.16) - open/close/unlink 3388 (time 1788712649.00 total 30.01 last 147.32) - open/close/unlink 4806 (time 1788712659.01 total 40.02 last 141.70) - open/close/unlink 6175 (time 1788712669.01 total 50.03 last 136.83) - open/close/unlink 7435 (time 1788712679.02 total 60.03 last 125.92) - open/close/unlink 8871 (time 1788712689.02 total 70.03 last 143.57) - open/close/unlink 10000 (time 1788712697.73 total 78.75 last 129.59) - open/close/unlink 11300 (time 1788712707.74 total 88.75 last 129.92) - open/close/unlink 12703 (time 1788712717.74 total 98.76 last 140.25) - open/close/unlink 14107 (time 1788712727.75 total 108.76 last 140.32) - open/close/unlink 15429 (time 1788712737.76 total 118.78 last 132.00) - open/close/unlink 16948 (time 1788712747.78 total 128.79 last 151.67) - open/close/unlink 18783 (time 1788712757.79 total 138.80 last 183.35) - open/close/unlink 20000 (time 1788712765.66 total 146.67 last 154.63) - open/close/unlink 21524 (time 1788712775.66 total 156.67 last 152.37) - open/close/unlink 23071 (time 1788712785.66 total 166.68 last 154.63) - open/close/unlink 24619 (time 1788712795.67 total 176.68 last 154.77) - open/close/unlink 25922 (time 1788712805.67 total 186.68 last 130.24) - open/close/unlink 27356 (time 1788712815.67 total 196.69 last 143.33) - open/close/unlink 28812 (time 1788712825.68 total 206.69 last 145.59) - open/close/unlink 30000 (time 1788712835.52 total 216.54 last 120.64) - open/close/unlink 31197 (time 1788712845.53 total 226.54 last 119.67) - open/close/unlink 32457 (time 1788712855.53 total 236.54 last 125.91) - open/close/unlink 33632 (time 1788712865.55 total 246.57 last 117.26) - open/close/unlink 35216 (time 1788712875.56 total 256.58 last 158.22) - open/close/unlink 36759 (time 1788712885.57 total 266.59 last 154.15) - open/close/unlink 38301 (time 1788712895.58 total 276.59 last 154.16) - open/close/unlink 39832 (time 1788712905.58 total 286.59 last 153.10) - open/close/unlink 41308 (time 1788712915.58 total 296.59 last 147.55) - open/close/unlink 42791 (time 1788712925.58 total 306.59 last 148.30) - open/close/unlink 44228 (time 1788712935.58 total 316.60 last 143.64) - open/close/unlink 45496 (time 1788712945.59 total 326.60 last 126.77) - open/close/unlink 46633 (time 1788712955.59 total 336.60 last 113.70) - open/close/unlink 47796 (time 1788712965.59 total 346.61 last 116.23) - open/close/unlink 49110 (time 1788712975.60 total 356.61 last 131.31) - open/close/unlink 50000 (time 1788712983.56 total 364.57 last 111.83) - open/close/unlink 51191 (time 1788712993.57 total 374.58 last 118.95) - open/close/unlink 52568 (time 1788713003.57 total 384.58 last 137.69) - open/close/unlink 53969 (time 1788713013.58 total 394.59 last 140.00) - open/close/unlink 55411 (time 1788713023.59 total 404.60 last 144.04) - open/close/unlink 56847 (time 1788713033.59 total 414.61 last 143.56) - open/close/unlink 58366 (time 1788713043.60 total 424.61 last 151.84) - open/close/unlink 59635 (time 1788713053.60 total 434.61 last 126.84) - open/close/unlink 60000 (time 1788713057.03 total 438.04 last 106.51) - open/close/unlink 61269 (time 1788713067.03 total 448.04 last 126.86) - open/close/unlink 62721 (time 1788713077.03 total 458.05 last 145.16) - open/close/unlink 64017 (time 1788713087.04 total 468.05 last 129.53) total: 64768 open/close/unlink in 474.90 seconds: 136.38 ops/second - open/close/unlink 611 (time 1788712628.99 total 10.00 last 61.08) - open/close/unlink 1900 (time 1788712639.00 total 20.01 last 128.82) - open/close/unlink 3370 (time 1788712649.00 total 30.01 last 146.96) - open/close/unlink 4791 (time 1788712659.00 total 40.02 last 142.05) - open/close/unlink 6158 (time 1788712669.01 total 50.02 last 136.64) - open/close/unlink 7421 (time 1788712679.02 total 60.03 last 126.21) - open/close/unlink 8868 (time 1788712689.02 total 70.03 last 144.68) - open/close/unlink 10000 (time 1788712697.83 total 78.84 last 128.45) - open/close/unlink 11297 (time 1788712707.84 total 88.85 last 129.54) - open/close/unlink 12716 (time 1788712717.85 total 98.86 last 141.79) - open/close/unlink 14128 (time 1788712727.85 total 108.86 last 141.18) - open/close/unlink 15452 (time 1788712737.86 total 118.87 last 132.25) - open/close/unlink 16988 (time 1788712747.87 total 128.88 last 153.54) - open/close/unlink 18818 (time 1788712757.87 total 138.88 last 182.95) - open/close/unlink 20000 (time 1788712765.34 total 146.35 last 158.26) - open/close/unlink 21500 (time 1788712775.34 total 156.35 last 149.96) - open/close/unlink 23046 (time 1788712785.34 total 166.36 last 154.55) - open/close/unlink 24570 (time 1788712795.35 total 176.36 last 152.39) - open/close/unlink 25913 (time 1788712805.35 total 186.36 last 134.23) - open/close/unlink 27344 (time 1788712815.35 total 196.36 last 143.07) - open/close/unlink 28824 (time 1788712825.36 total 206.37 last 147.92) - open/close/unlink 30000 (time 1788712834.92 total 215.93 last 122.99) - open/close/unlink 31199 (time 1788712844.92 total 225.94 last 119.83) - open/close/unlink 32481 (time 1788712854.94 total 235.95 last 128.07) - open/close/unlink 33647 (time 1788712864.97 total 245.98 last 116.17) - open/close/unlink 35218 (time 1788712874.98 total 255.99 last 157.03) - open/close/unlink 36739 (time 1788712884.98 total 265.99 last 152.09) - open/close/unlink 38236 (time 1788712894.98 total 275.99 last 149.68) - open/close/unlink 39757 (time 1788712904.98 total 285.99 last 152.06) - open/close/unlink 40000 (time 1788712907.15 total 288.16 last 112.20) - open/close/unlink 41540 (time 1788712917.15 total 298.16 last 153.98) - open/close/unlink 42986 (time 1788712927.16 total 308.17 last 144.51) - open/close/unlink 44418 (time 1788712937.16 total 318.17 last 143.13) - open/close/unlink 45673 (time 1788712947.17 total 328.18 last 125.39) - open/close/unlink 46819 (time 1788712957.18 total 338.20 last 114.44) - open/close/unlink 48031 (time 1788712967.19 total 348.20 last 121.15) - open/close/unlink 49291 (time 1788712977.19 total 358.21 last 125.92) - open/close/unlink 50000 (time 1788712983.85 total 364.87 last 106.45) - open/close/unlink 51193 (time 1788712993.86 total 374.87 last 119.24) - open/close/unlink 52575 (time 1788713003.87 total 384.88 last 138.06) - open/close/unlink 53976 (time 1788713013.89 total 394.90 last 139.78) - open/close/unlink 55407 (time 1788713023.90 total 404.92 last 142.92) - open/close/unlink 56842 (time 1788713033.92 total 414.94 last 143.21) - open/close/unlink 58396 (time 1788713043.93 total 424.94 last 155.33) - open/close/unlink 59638 (time 1788713053.94 total 434.95 last 124.12) - open/close/unlink 60000 (time 1788713057.34 total 438.36 last 106.20) - open/close/unlink 61292 (time 1788713067.35 total 448.36 last 129.15) - open/close/unlink 62715 (time 1788713077.36 total 458.37 last 142.18) - open/close/unlink 63997 (time 1788713087.36 total 468.37 last 128.16) total: 64768 open/close/unlink in 475.67 seconds: 136.16 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete oleg155-server: cannot open 'lustre-ost2': no such pool pdsh@oleg155-client: oleg155-server: ssh exited with exit code 255 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg155-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg155-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg155-server stop mds service on oleg155-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg155-server unloading modules via unload_modules_local on: 'oleg155-server' oleg155-server: oleg155-server.virtnet: executing unload_modules_local oleg155-server: kmemleak disabled modules unloaded. pdsh@oleg155-client: oleg155-client: ssh exited with exit code 2 pdsh@oleg155-client: oleg155-server: ssh exited with exit code 2 pdsh@oleg155-client: oleg155-client: ssh exited with exit code 2 pdsh@oleg155-client: oleg155-server: ssh exited with exit code 2