== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 07:41:01 (1784288461) start mds service on oleg457-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug -1 all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg457-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug -1 all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0000 oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 92553 329 92224 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 12329 425 11904 4% /mnt/lustre[OST:0] filesystem_summary: 12233 329 11904 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close/unlink 1487 (time 1784288487.03 total 10.01 last 148.56) - open/close/unlink 2925 (time 1784288497.04 total 20.01 last 143.75) - open/close/unlink 4344 (time 1784288507.04 total 30.01 last 141.89) - open/close/unlink 5763 (time 1784288517.04 total 40.02 last 141.86) - open/close/unlink 7091 (time 1784288527.04 total 50.02 last 132.76) - open/close/unlink 8462 (time 1784288537.05 total 60.02 last 137.05) - open/close/unlink 9843 (time 1784288547.05 total 70.03 last 138.03) - open/close/unlink 11241 (time 1784288557.06 total 80.03 last 139.72) - open/close/unlink 12362 (time 1784288567.06 total 90.04 last 112.06) - open/close/unlink 13697 (time 1784288577.07 total 100.05 last 133.39) - open/close/unlink 15250 (time 1784288587.07 total 110.05 last 155.29) - open/close/unlink 16764 (time 1784288597.07 total 120.05 last 151.36) - open/close/unlink 18057 (time 1784288607.09 total 130.07 last 129.02) - open/close/unlink 19493 (time 1784288617.10 total 140.07 last 143.56) - open/close/unlink 20000 (time 1784288620.58 total 143.55 last 145.60) - open/close/unlink 21436 (time 1784288630.59 total 153.56 last 143.51) - open/close/unlink 22683 (time 1784288640.59 total 163.56 last 124.69) - open/close/unlink 24040 (time 1784288650.59 total 173.56 last 135.67) - open/close/unlink 25281 (time 1784288660.59 total 183.57 last 124.07) - open/close/unlink 26372 (time 1784288670.60 total 193.57 last 109.04) - open/close/unlink 27571 (time 1784288680.60 total 203.58 last 119.83) - open/close/unlink 28908 (time 1784288690.61 total 213.58 last 133.64) - open/close/unlink 30000 (time 1784288697.83 total 220.81 last 151.13) - open/close/unlink 31365 (time 1784288707.83 total 230.81 last 136.49) - open/close/unlink 32786 (time 1784288717.84 total 240.82 last 142.00) - open/close/unlink 34287 (time 1784288727.84 total 250.82 last 150.07) - open/close/unlink 35598 (time 1784288737.84 total 260.82 last 131.09) - open/close/unlink 36731 (time 1784288747.85 total 270.82 last 113.27) - open/close/unlink 38041 (time 1784288757.85 total 280.83 last 130.90) - open/close/unlink 39383 (time 1784288767.86 total 290.84 last 134.10) - open/close/unlink 40000 (time 1784288772.87 total 295.85 last 123.11) - open/close/unlink 41291 (time 1784288782.88 total 305.85 last 129.04) - open/close/unlink 42596 (time 1784288792.88 total 315.86 last 130.46) - open/close/unlink 43870 (time 1784288802.88 total 325.86 last 127.36) - open/close/unlink 45374 (time 1784288812.89 total 335.87 last 150.25) - open/close/unlink 46929 (time 1784288822.91 total 345.88 last 155.32) - open/close/unlink 48461 (time 1784288832.91 total 355.89 last 153.07) - open/close/unlink 49968 (time 1784288842.92 total 365.90 last 150.60) - open/close/unlink 51450 (time 1784288852.94 total 375.91 last 147.97) - open/close/unlink 52872 (time 1784288862.94 total 385.92 last 142.10) - open/close/unlink 54383 (time 1784288872.94 total 395.92 last 151.10) - open/close/unlink 55894 (time 1784288882.95 total 405.92 last 151.07) - open/close/unlink 57645 (time 1784288892.95 total 415.92 last 175.06) - open/close/unlink 59546 (time 1784288902.95 total 425.93 last 190.05) - open/close/unlink 60000 (time 1784288905.78 total 428.76 last 160.37) - open/close/unlink 61593 (time 1784288915.78 total 438.76 last 159.27) - open/close/unlink 63121 (time 1784288925.79 total 448.77 last 152.70) - open/close/unlink 64529 (time 1784288935.79 total 458.77 last 140.79) total: 64768 open/close/unlink in 460.12 seconds: 140.76 ops/second - open/close/unlink 1495 (time 1784288487.36 total 10.00 last 149.45) - open/close/unlink 2952 (time 1784288497.37 total 20.01 last 145.62) - open/close/unlink 4372 (time 1784288507.37 total 30.01 last 141.95) - open/close/unlink 5778 (time 1784288517.37 total 40.01 last 140.60) - open/close/unlink 7123 (time 1784288527.38 total 50.02 last 134.43) - open/close/unlink 8503 (time 1784288537.38 total 60.02 last 137.95) - open/close/unlink 9874 (time 1784288547.38 total 70.02 last 137.07) - open/close/unlink 11277 (time 1784288557.39 total 80.03 last 140.21) - open/close/unlink 12393 (time 1784288567.40 total 90.04 last 111.51) - open/close/unlink 13733 (time 1784288577.40 total 100.04 last 133.95) - open/close/unlink 15261 (time 1784288587.41 total 110.05 last 152.68) - open/close/unlink 16789 (time 1784288597.42 total 120.06 last 152.62) - open/close/unlink 18081 (time 1784288607.42 total 130.06 last 129.18) - open/close/unlink 19515 (time 1784288617.42 total 140.06 last 143.40) - open/close/unlink 20000 (time 1784288620.65 total 143.29 last 150.45) - open/close/unlink 21432 (time 1784288630.65 total 153.29 last 143.14) - open/close/unlink 22688 (time 1784288640.65 total 163.29 last 125.55) - open/close/unlink 24051 (time 1784288650.66 total 173.30 last 136.26) - open/close/unlink 25285 (time 1784288660.66 total 183.30 last 123.37) - open/close/unlink 26381 (time 1784288670.67 total 193.31 last 109.52) - open/close/unlink 27577 (time 1784288680.67 total 203.31 last 119.59) - open/close/unlink 28925 (time 1784288690.67 total 213.31 last 134.77) - open/close/unlink 30000 (time 1784288697.72 total 220.36 last 152.56) - open/close/unlink 31357 (time 1784288707.72 total 230.36 last 135.68) - open/close/unlink 32771 (time 1784288717.72 total 240.36 last 141.33) - open/close/unlink 34265 (time 1784288727.72 total 250.37 last 149.36) - open/close/unlink 35570 (time 1784288737.73 total 260.37 last 130.42) - open/close/unlink 36708 (time 1784288747.74 total 270.38 last 113.71) - open/close/unlink 38011 (time 1784288757.74 total 280.39 last 130.23) - open/close/unlink 39349 (time 1784288767.75 total 290.39 last 133.75) - open/close/unlink 40000 (time 1784288773.10 total 295.74 last 121.75) - open/close/unlink 41303 (time 1784288783.10 total 305.74 last 130.25) - open/close/unlink 42611 (time 1784288793.10 total 315.74 last 130.80) - open/close/unlink 43884 (time 1784288803.11 total 325.75 last 127.21) - open/close/unlink 45399 (time 1784288813.11 total 335.75 last 151.43) - open/close/unlink 46951 (time 1784288823.12 total 345.76 last 155.10) - open/close/unlink 48477 (time 1784288833.12 total 355.76 last 152.52) - open/close/unlink 49987 (time 1784288843.13 total 365.77 last 150.86) - open/close/unlink 51473 (time 1784288853.14 total 375.78 last 148.52) - open/close/unlink 52901 (time 1784288863.15 total 385.79 last 142.67) - open/close/unlink 54419 (time 1784288873.15 total 395.79 last 151.73) - open/close/unlink 55935 (time 1784288883.15 total 405.80 last 151.56) - open/close/unlink 57684 (time 1784288893.16 total 415.80 last 174.85) - open/close/unlink 59565 (time 1784288903.16 total 425.80 last 188.08) - open/close/unlink 60000 (time 1784288905.92 total 428.56 last 157.57) - open/close/unlink 61607 (time 1784288915.92 total 438.56 last 160.62) - open/close/unlink 63115 (time 1784288925.93 total 448.57 last 150.65) - open/close/unlink 64535 (time 1784288935.94 total 458.58 last 141.95) total: 64768 open/close/unlink in 460.21 seconds: 140.74 ops/second - open/close/unlink 1508 (time 1784288487.40 total 10.00 last 150.75) - open/close/unlink 2956 (time 1784288497.40 total 20.00 last 144.79) - open/close/unlink 4371 (time 1784288507.41 total 30.01 last 141.48) - open/close/unlink 5777 (time 1784288517.41 total 40.01 last 140.56) - open/close/unlink 7122 (time 1784288527.42 total 50.02 last 134.40) - open/close/unlink 8505 (time 1784288537.42 total 60.02 last 138.27) - open/close/unlink 9860 (time 1784288547.42 total 70.02 last 135.49) - open/close/unlink 11268 (time 1784288557.43 total 80.03 last 140.63) - open/close/unlink 12382 (time 1784288567.43 total 90.03 last 111.38) - open/close/unlink 13733 (time 1784288577.43 total 100.03 last 135.08) - open/close/unlink 15260 (time 1784288587.44 total 110.04 last 152.61) - open/close/unlink 16768 (time 1784288597.46 total 120.06 last 150.48) - open/close/unlink 18070 (time 1784288607.46 total 130.06 last 130.16) - open/close/unlink 19510 (time 1784288617.47 total 140.07 last 143.86) - open/close/unlink 20000 (time 1784288620.70 total 143.30 last 151.96) - open/close/unlink 21423 (time 1784288630.70 total 153.30 last 142.27) - open/close/unlink 22683 (time 1784288640.70 total 163.30 last 125.96) - open/close/unlink 24040 (time 1784288650.71 total 173.31 last 135.63) - open/close/unlink 25277 (time 1784288660.71 total 183.31 last 123.69) - open/close/unlink 26368 (time 1784288670.71 total 193.31 last 109.05) - open/close/unlink 27551 (time 1784288680.72 total 203.32 last 118.25) - open/close/unlink 28891 (time 1784288690.72 total 213.32 last 133.98) - open/close/unlink 30000 (time 1784288698.05 total 220.65 last 151.37) - open/close/unlink 31366 (time 1784288708.05 total 230.65 last 136.57) - open/close/unlink 32773 (time 1784288718.05 total 240.65 last 140.64) - open/close/unlink 34279 (time 1784288728.05 total 250.65 last 150.59) - open/close/unlink 35573 (time 1784288738.05 total 260.65 last 129.39) - open/close/unlink 36698 (time 1784288748.06 total 270.66 last 112.44) - open/close/unlink 38012 (time 1784288758.07 total 280.67 last 131.31) - open/close/unlink 39351 (time 1784288768.07 total 290.67 last 133.83) - open/close/unlink 40000 (time 1784288773.35 total 295.95 last 122.98) - open/close/unlink 41281 (time 1784288783.35 total 305.95 last 128.10) - open/close/unlink 42582 (time 1784288793.35 total 315.95 last 130.10) - open/close/unlink 43856 (time 1784288803.35 total 325.95 last 127.37) - open/close/unlink 45370 (time 1784288813.35 total 335.95 last 151.39) - open/close/unlink 46909 (time 1784288823.36 total 345.96 last 153.80) - open/close/unlink 48433 (time 1784288833.36 total 355.96 last 152.37) - open/close/unlink 49931 (time 1784288843.38 total 365.98 last 149.54) - open/close/unlink 51415 (time 1784288853.38 total 375.98 last 148.36) - open/close/unlink 52849 (time 1784288863.38 total 385.98 last 143.37) - open/close/unlink 54386 (time 1784288873.39 total 395.99 last 153.62) - open/close/unlink 55912 (time 1784288883.39 total 405.99 last 152.54) - open/close/unlink 57673 (time 1784288893.39 total 415.99 last 176.06) - open/close/unlink 59565 (time 1784288903.40 total 426.00 last 189.16) - open/close/unlink 60000 (time 1784288906.21 total 428.81 last 154.70) - open/close/unlink 61595 (time 1784288916.21 total 438.81 last 159.43) - open/close/unlink 63075 (time 1784288926.22 total 448.82 last 147.96) - open/close/unlink 64526 (time 1784288936.22 total 458.82 last 145.05) total: 64768 open/close/unlink in 461.39 seconds: 140.38 ops/second - open/close/unlink 1493 (time 1784288487.27 total 10.01 last 149.20) - open/close/unlink 2943 (time 1784288497.27 total 20.01 last 144.94) - open/close/unlink 4355 (time 1784288507.27 total 30.01 last 141.19) - open/close/unlink 5752 (time 1784288517.27 total 40.01 last 139.69) - open/close/unlink 7085 (time 1784288527.27 total 50.01 last 133.29) - open/close/unlink 8465 (time 1784288537.28 total 60.02 last 137.95) - open/close/unlink 9825 (time 1784288547.28 total 70.02 last 135.93) - open/close/unlink 11220 (time 1784288557.28 total 80.02 last 139.49) - open/close/unlink 12338 (time 1784288567.29 total 90.03 last 111.74) - open/close/unlink 13686 (time 1784288577.29 total 100.03 last 134.76) - open/close/unlink 15220 (time 1784288587.30 total 110.04 last 153.27) - open/close/unlink 16735 (time 1784288597.30 total 120.04 last 151.43) - open/close/unlink 18032 (time 1784288607.30 total 130.04 last 129.69) - open/close/unlink 19472 (time 1784288617.32 total 140.06 last 143.84) - open/close/unlink 20000 (time 1784288620.81 total 143.55 last 151.12) - open/close/unlink 21423 (time 1784288630.81 total 153.55 last 142.25) - open/close/unlink 22683 (time 1784288640.81 total 163.55 last 126.00) - open/close/unlink 24040 (time 1784288650.81 total 173.55 last 135.68) - open/close/unlink 25276 (time 1784288660.82 total 183.56 last 123.52) - open/close/unlink 26376 (time 1784288670.83 total 193.57 last 109.95) - open/close/unlink 27570 (time 1784288680.83 total 203.57 last 119.31) - open/close/unlink 28915 (time 1784288690.84 total 213.58 last 134.42) - open/close/unlink 30000 (time 1784288698.14 total 220.88 last 148.63) - open/close/unlink 31374 (time 1784288708.15 total 230.88 last 137.32) - open/close/unlink 32758 (time 1784288718.15 total 240.89 last 138.32) - open/close/unlink 34258 (time 1784288728.15 total 250.89 last 149.96) - open/close/unlink 35559 (time 1784288738.16 total 260.90 last 130.03) - open/close/unlink 36689 (time 1784288748.16 total 270.90 last 112.97) - open/close/unlink 37996 (time 1784288758.16 total 280.90 last 130.68) - open/close/unlink 39316 (time 1784288768.17 total 290.91 last 131.95) - open/close/unlink 40000 (time 1784288773.65 total 296.39 last 124.80) - open/close/unlink 41290 (time 1784288783.65 total 306.39 last 128.92) - open/close/unlink 42612 (time 1784288793.66 total 316.40 last 132.15) - open/close/unlink 43868 (time 1784288803.66 total 326.40 last 125.57) - open/close/unlink 45382 (time 1784288813.66 total 336.40 last 151.32) - open/close/unlink 46909 (time 1784288823.67 total 346.41 last 152.66) - open/close/unlink 48447 (time 1784288833.67 total 356.41 last 153.78) - open/close/unlink 49920 (time 1784288843.68 total 366.42 last 147.18) - open/close/unlink 51409 (time 1784288853.69 total 376.43 last 148.67) - open/close/unlink 52863 (time 1784288863.70 total 386.44 last 145.31) - open/close/unlink 54386 (time 1784288873.71 total 396.45 last 152.06) - open/close/unlink 55924 (time 1784288883.73 total 406.47 last 153.61) - open/close/unlink 57722 (time 1784288893.73 total 416.47 last 179.75) - open/close/unlink 59627 (time 1784288903.73 total 426.47 last 190.48) - open/close/unlink 60000 (time 1784288906.26 total 429.00 last 147.16) - open/close/unlink 61597 (time 1784288916.27 total 439.01 last 159.63) - open/close/unlink 63077 (time 1784288926.27 total 449.01 last 147.99) - open/close/unlink 64523 (time 1784288936.27 total 459.01 last 144.59) total: 64768 open/close/unlink in 461.80 seconds: 140.25 ops/second - open/close/unlink 1499 (time 1784288487.29 total 10.00 last 149.89) - open/close/unlink 2946 (time 1784288497.29 total 20.00 last 144.69) - open/close/unlink 4364 (time 1784288507.30 total 30.00 last 141.77) - open/close/unlink 5770 (time 1784288517.30 total 40.01 last 140.54) - open/close/unlink 7099 (time 1784288527.31 total 50.01 last 132.81) - open/close/unlink 8484 (time 1784288537.31 total 60.02 last 138.46) - open/close/unlink 9844 (time 1784288547.31 total 70.02 last 135.94) - open/close/unlink 11244 (time 1784288557.32 total 80.03 last 139.91) - open/close/unlink 12369 (time 1784288567.33 total 90.03 last 112.45) - open/close/unlink 13717 (time 1784288577.33 total 100.03 last 134.78) - open/close/unlink 15251 (time 1784288587.34 total 110.05 last 153.21) - open/close/unlink 16781 (time 1784288597.34 total 120.05 last 152.98) - open/close/unlink 18075 (time 1784288607.35 total 130.06 last 129.27) - open/close/unlink 19516 (time 1784288617.37 total 140.08 last 143.76) - open/close/unlink 20000 (time 1784288620.61 total 143.32 last 149.65) - open/close/unlink 21419 (time 1784288630.61 total 153.32 last 141.84) - open/close/unlink 22673 (time 1784288640.61 total 163.32 last 125.40) - open/close/unlink 24034 (time 1784288650.61 total 173.32 last 136.10) - open/close/unlink 25270 (time 1784288660.62 total 183.32 last 123.56) - open/close/unlink 26359 (time 1784288670.62 total 193.32 last 108.89) - open/close/unlink 27539 (time 1784288680.62 total 203.33 last 117.93) - open/close/unlink 28862 (time 1784288690.63 total 213.33 last 132.24) - open/close/unlink 30000 (time 1784288698.15 total 220.86 last 151.24) - open/close/unlink 31373 (time 1784288708.16 total 230.86 last 137.23) - open/close/unlink 32769 (time 1784288718.16 total 240.87 last 139.57) - open/close/unlink 34280 (time 1784288728.16 total 250.87 last 151.08) - open/close/unlink 35574 (time 1784288738.16 total 260.87 last 129.34) - open/close/unlink 36692 (time 1784288748.17 total 270.88 last 111.69) - open/close/unlink 38003 (time 1784288758.18 total 280.89 last 131.02) - open/close/unlink 39334 (time 1784288768.18 total 290.89 last 133.09) - open/close/unlink 40000 (time 1784288773.51 total 296.21 last 125.10) - open/close/unlink 41292 (time 1784288783.51 total 306.22 last 129.09) - open/close/unlink 42615 (time 1784288793.52 total 316.23 last 132.23) - open/close/unlink 43880 (time 1784288803.53 total 326.23 last 126.42) - open/close/unlink 45399 (time 1784288813.53 total 336.23 last 151.89) - open/close/unlink 46922 (time 1784288823.53 total 346.23 last 152.28) - open/close/unlink 48443 (time 1784288833.53 total 356.24 last 152.07) - open/close/unlink 49920 (time 1784288843.55 total 366.25 last 147.42) - open/close/unlink 51414 (time 1784288853.55 total 376.26 last 149.32) - open/close/unlink 52841 (time 1784288863.56 total 386.26 last 142.65) - open/close/unlink 54380 (time 1784288873.57 total 396.28 last 153.72) - open/close/unlink 55914 (time 1784288883.57 total 406.28 last 153.37) - open/close/unlink 57689 (time 1784288893.57 total 416.28 last 177.46) - open/close/unlink 59585 (time 1784288903.58 total 426.28 last 189.54) - open/close/unlink 60000 (time 1784288906.36 total 429.07 last 148.98) - open/close/unlink 61613 (time 1784288916.36 total 439.07 last 161.24) - open/close/unlink 63082 (time 1784288926.37 total 449.07 last 146.89) - open/close/unlink 64526 (time 1784288936.37 total 459.07 last 144.39) total: 64768 open/close/unlink in 462.02 seconds: 140.19 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete oleg457-server: cannot open 'lustre-ost2': no such pool pdsh@oleg457-client: oleg457-server: ssh exited with exit code 255 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg457-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg457-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg457-server stop mds service on oleg457-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg457-server unloading modules via unload_modules_local on: 'oleg457-server' oleg457-server: oleg457-server.virtnet: executing unload_modules_local oleg457-server: modules unloaded. pdsh@oleg457-client: oleg457-client: ssh exited with exit code 2 pdsh@oleg457-client: oleg457-server: ssh exited with exit code 2 pdsh@oleg457-client: oleg457-client: ssh exited with exit code 2 pdsh@oleg457-client: oleg457-server: ssh exited with exit code 2