== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 09:23:13 (1783516993) start mds service on oleg338-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg338-server: oleg338-server.virtnet: executing set_default_debug -1 all pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg338-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg338-server: oleg338-server.virtnet: executing set_default_debug -1 all pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1 Started lustre-OST0000 oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg338-client.virtnet: -o user_xattr,flock 192.168.203.138@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 92553 329 92224 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 12297 393 11904 4% /mnt/lustre[OST:0] filesystem_summary: 12233 329 11904 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close/unlink 1861 (time 1783517024.14 total 10.00 last 186.08) - open/close/unlink 3709 (time 1783517034.14 total 20.00 last 184.78) - open/close/unlink 5383 (time 1783517044.15 total 30.01 last 167.31) - open/close/unlink 6365 (time 1783517054.15 total 40.01 last 98.16) - open/close/unlink 7148 (time 1783517064.16 total 50.02 last 78.24) - open/close/unlink 8007 (time 1783517074.16 total 60.02 last 85.90) - open/close/unlink 9304 (time 1783517084.16 total 70.02 last 129.70) - open/close/unlink 10000 (time 1783517089.05 total 74.91 last 142.31) - open/close/unlink 11424 (time 1783517099.05 total 84.91 last 142.38) - open/close/unlink 12964 (time 1783517109.06 total 94.92 last 153.89) - open/close/unlink 14283 (time 1783517119.06 total 104.92 last 131.90) - open/close/unlink 15490 (time 1783517129.06 total 114.92 last 120.68) - open/close/unlink 16954 (time 1783517139.07 total 124.93 last 146.30) - open/close/unlink 18464 (time 1783517149.07 total 134.93 last 150.92) - open/close/unlink 19767 (time 1783517159.08 total 144.94 last 130.27) - open/close/unlink 20000 (time 1783517161.37 total 147.23 last 101.63) - open/close/unlink 21308 (time 1783517171.37 total 157.23 last 130.78) - open/close/unlink 22634 (time 1783517181.37 total 167.23 last 132.57) - open/close/unlink 24204 (time 1783517191.38 total 177.24 last 156.92) - open/close/unlink 25624 (time 1783517201.38 total 187.24 last 141.97) - open/close/unlink 26926 (time 1783517211.38 total 197.24 last 130.16) - open/close/unlink 27846 (time 1783517221.40 total 207.26 last 91.88) - open/close/unlink 28663 (time 1783517231.41 total 217.27 last 81.57) - open/close/unlink 29506 (time 1783517241.43 total 227.28 last 84.19) - open/close/unlink 30000 (time 1783517246.84 total 232.70 last 91.29) - open/close/unlink 31029 (time 1783517256.84 total 242.70 last 102.85) - open/close/unlink 32208 (time 1783517266.85 total 252.71 last 117.84) - open/close/unlink 33527 (time 1783517276.85 total 262.71 last 131.80) - open/close/unlink 34608 (time 1783517286.86 total 272.72 last 108.04) - open/close/unlink 35707 (time 1783517296.86 total 282.72 last 109.88) - open/close/unlink 37036 (time 1783517306.87 total 292.73 last 132.81) - open/close/unlink 38339 (time 1783517316.87 total 302.73 last 130.26) - open/close/unlink 39825 (time 1783517326.88 total 312.74 last 148.48) - open/close/unlink 41412 (time 1783517336.88 total 322.74 last 158.66) - open/close/unlink 42946 (time 1783517346.88 total 332.74 last 153.35) - open/close/unlink 44505 (time 1783517356.89 total 342.75 last 155.84) - open/close/unlink 46162 (time 1783517366.89 total 352.75 last 165.66) - open/close/unlink 47729 (time 1783517376.89 total 362.75 last 156.67) - open/close/unlink 49323 (time 1783517386.89 total 372.75 last 159.35) - open/close/unlink 50000 (time 1783517390.84 total 376.70 last 171.50) - open/close/unlink 51700 (time 1783517400.85 total 386.71 last 169.86) - open/close/unlink 53323 (time 1783517410.87 total 396.72 last 162.07) - open/close/unlink 54976 (time 1783517420.87 total 406.73 last 165.21) - open/close/unlink 56634 (time 1783517430.89 total 416.75 last 165.43) - open/close/unlink 58261 (time 1783517440.89 total 426.75 last 162.69) - open/close/unlink 59975 (time 1783517450.90 total 436.76 last 171.35) - open/close/unlink 61664 (time 1783517460.90 total 446.76 last 168.89) - open/close/unlink 63447 (time 1783517470.90 total 456.76 last 178.22) total: 64768 open/close/unlink in 465.13 seconds: 139.25 ops/second - open/close/unlink 1868 (time 1783517024.33 total 10.01 last 186.69) - open/close/unlink 3707 (time 1783517034.33 total 20.01 last 183.87) - open/close/unlink 5388 (time 1783517044.33 total 30.01 last 168.03) - open/close/unlink 6347 (time 1783517054.34 total 40.02 last 95.82) - open/close/unlink 7121 (time 1783517064.35 total 50.03 last 77.30) - open/close/unlink 7980 (time 1783517074.35 total 60.03 last 85.90) - open/close/unlink 9290 (time 1783517084.36 total 70.03 last 130.99) - open/close/unlink 10000 (time 1783517089.33 total 75.00 last 142.85) - open/close/unlink 11429 (time 1783517099.33 total 85.01 last 142.84) - open/close/unlink 12953 (time 1783517109.33 total 95.01 last 152.38) - open/close/unlink 14260 (time 1783517119.33 total 105.01 last 130.67) - open/close/unlink 15453 (time 1783517129.33 total 115.01 last 119.29) - open/close/unlink 16917 (time 1783517139.34 total 125.02 last 146.32) - open/close/unlink 18406 (time 1783517149.34 total 135.02 last 148.89) - open/close/unlink 19698 (time 1783517159.34 total 145.02 last 129.20) - open/close/unlink 20000 (time 1783517162.01 total 147.69 last 113.09) - open/close/unlink 21288 (time 1783517172.01 total 157.69 last 128.77) - open/close/unlink 22631 (time 1783517182.02 total 167.69 last 134.28) - open/close/unlink 24189 (time 1783517192.02 total 177.69 last 155.79) - open/close/unlink 25616 (time 1783517202.02 total 187.70 last 142.68) - open/close/unlink 26895 (time 1783517212.03 total 197.71 last 127.78) - open/close/unlink 27783 (time 1783517222.04 total 207.72 last 88.69) - open/close/unlink 28598 (time 1783517232.04 total 217.72 last 81.48) - open/close/unlink 29445 (time 1783517242.06 total 227.73 last 84.58) - open/close/unlink 30000 (time 1783517248.10 total 233.78 last 91.77) - open/close/unlink 31041 (time 1783517258.11 total 243.78 last 104.09) - open/close/unlink 32228 (time 1783517268.11 total 253.79 last 118.63) - open/close/unlink 33527 (time 1783517278.12 total 263.80 last 129.79) - open/close/unlink 34592 (time 1783517288.13 total 273.80 last 106.43) - open/close/unlink 35728 (time 1783517298.14 total 283.81 last 113.49) - open/close/unlink 37101 (time 1783517308.14 total 293.82 last 137.19) - open/close/unlink 38398 (time 1783517318.15 total 303.83 last 129.57) - open/close/unlink 39960 (time 1783517328.15 total 313.83 last 156.19) - open/close/unlink 41543 (time 1783517338.15 total 323.83 last 158.29) - open/close/unlink 43084 (time 1783517348.16 total 333.83 last 154.07) - open/close/unlink 44659 (time 1783517358.16 total 343.84 last 157.42) - open/close/unlink 46287 (time 1783517368.16 total 353.84 last 162.79) - open/close/unlink 47862 (time 1783517378.17 total 363.84 last 157.43) - open/close/unlink 49498 (time 1783517388.17 total 373.85 last 163.55) - open/close/unlink 50000 (time 1783517391.33 total 377.01 last 158.74) - open/close/unlink 51684 (time 1783517401.34 total 387.01 last 168.35) - open/close/unlink 53305 (time 1783517411.34 total 397.02 last 161.99) - open/close/unlink 54940 (time 1783517421.34 total 407.02 last 163.47) - open/close/unlink 56589 (time 1783517431.36 total 417.04 last 164.60) - open/close/unlink 58221 (time 1783517441.37 total 427.05 last 163.12) - open/close/unlink 59929 (time 1783517451.37 total 437.05 last 170.73) - open/close/unlink 61647 (time 1783517461.39 total 447.07 last 171.48) - open/close/unlink 63436 (time 1783517471.40 total 457.07 last 178.81) total: 64768 open/close/unlink in 465.97 seconds: 139.00 ops/second - open/close/unlink 1887 (time 1783517024.49 total 10.00 last 188.65) - open/close/unlink 3729 (time 1783517034.49 total 20.01 last 184.15) - open/close/unlink 5395 (time 1783517044.49 total 30.01 last 166.54) - open/close/unlink 6329 (time 1783517054.51 total 40.02 last 93.27) - open/close/unlink 7105 (time 1783517064.51 total 50.03 last 77.57) - open/close/unlink 7964 (time 1783517074.52 total 60.04 last 85.81) - open/close/unlink 9282 (time 1783517084.52 total 70.04 last 131.79) - open/close/unlink 10000 (time 1783517089.48 total 75.00 last 144.76) - open/close/unlink 11422 (time 1783517099.49 total 85.00 last 142.13) - open/close/unlink 12965 (time 1783517109.49 total 95.01 last 154.23) - open/close/unlink 14255 (time 1783517119.50 total 105.02 last 128.90) - open/close/unlink 15441 (time 1783517129.50 total 115.02 last 118.58) - open/close/unlink 16905 (time 1783517139.50 total 125.02 last 146.38) - open/close/unlink 18395 (time 1783517149.51 total 135.02 last 148.96) - open/close/unlink 19697 (time 1783517159.53 total 145.04 last 129.94) - open/close/unlink 20000 (time 1783517162.10 total 147.61 last 117.79) - open/close/unlink 21296 (time 1783517172.10 total 157.62 last 129.53) - open/close/unlink 22641 (time 1783517182.11 total 167.62 last 134.44) - open/close/unlink 24201 (time 1783517192.11 total 177.63 last 155.95) - open/close/unlink 25620 (time 1783517202.11 total 187.63 last 141.89) - open/close/unlink 26895 (time 1783517212.12 total 197.63 last 127.40) - open/close/unlink 27787 (time 1783517222.12 total 207.64 last 89.16) - open/close/unlink 28603 (time 1783517232.13 total 217.65 last 81.53) - open/close/unlink 29445 (time 1783517242.14 total 227.66 last 84.11) - open/close/unlink 30000 (time 1783517248.10 total 233.61 last 93.23) - open/close/unlink 31024 (time 1783517258.10 total 243.61 last 102.37) - open/close/unlink 32211 (time 1783517268.11 total 253.62 last 118.64) - open/close/unlink 33508 (time 1783517278.11 total 263.62 last 129.65) - open/close/unlink 34575 (time 1783517288.12 total 273.63 last 106.61) - open/close/unlink 35719 (time 1783517298.13 total 283.64 last 114.26) - open/close/unlink 37081 (time 1783517308.14 total 293.65 last 136.08) - open/close/unlink 38373 (time 1783517318.14 total 303.66 last 129.15) - open/close/unlink 39931 (time 1783517328.15 total 313.66 last 155.69) - open/close/unlink 41509 (time 1783517338.15 total 323.67 last 157.74) - open/close/unlink 43059 (time 1783517348.16 total 333.67 last 154.94) - open/close/unlink 44627 (time 1783517358.16 total 343.67 last 156.75) - open/close/unlink 46263 (time 1783517368.16 total 353.68 last 163.53) - open/close/unlink 47826 (time 1783517378.17 total 363.68 last 156.27) - open/close/unlink 49460 (time 1783517388.17 total 373.68 last 163.36) - open/close/unlink 50000 (time 1783517391.47 total 376.98 last 163.68) - open/close/unlink 51665 (time 1783517401.47 total 386.99 last 166.44) - open/close/unlink 53281 (time 1783517411.47 total 396.99 last 161.56) - open/close/unlink 54928 (time 1783517421.48 total 406.99 last 164.65) - open/close/unlink 56576 (time 1783517431.48 total 416.99 last 164.79) - open/close/unlink 58188 (time 1783517441.49 total 427.00 last 161.00) - open/close/unlink 59893 (time 1783517451.49 total 437.00 last 170.50) - open/close/unlink 61599 (time 1783517461.50 total 447.01 last 170.52) - open/close/unlink 63409 (time 1783517471.50 total 457.01 last 180.92) total: 64768 open/close/unlink in 466.89 seconds: 138.72 ops/second - open/close/unlink 1874 (time 1783517024.45 total 10.00 last 187.40) - open/close/unlink 3726 (time 1783517034.45 total 20.00 last 185.17) - open/close/unlink 5389 (time 1783517044.46 total 30.01 last 166.23) - open/close/unlink 6334 (time 1783517054.46 total 40.01 last 94.44) - open/close/unlink 7113 (time 1783517064.47 total 50.02 last 77.87) - open/close/unlink 7975 (time 1783517074.48 total 60.02 last 86.13) - open/close/unlink 9308 (time 1783517084.48 total 70.03 last 133.27) - open/close/unlink 10000 (time 1783517089.32 total 74.87 last 142.82) - open/close/unlink 11422 (time 1783517099.32 total 84.87 last 142.17) - open/close/unlink 12940 (time 1783517109.33 total 94.88 last 151.76) - open/close/unlink 14245 (time 1783517119.33 total 104.88 last 130.41) - open/close/unlink 15440 (time 1783517129.34 total 114.89 last 119.43) - open/close/unlink 16905 (time 1783517139.34 total 124.89 last 146.49) - open/close/unlink 18394 (time 1783517149.35 total 134.90 last 148.82) - open/close/unlink 19696 (time 1783517159.35 total 144.90 last 130.09) - open/close/unlink 20000 (time 1783517162.02 total 147.57 last 114.10) - open/close/unlink 21310 (time 1783517172.02 total 157.57 last 130.98) - open/close/unlink 22638 (time 1783517182.03 total 167.57 last 132.73) - open/close/unlink 24190 (time 1783517192.03 total 177.58 last 155.15) - open/close/unlink 25619 (time 1783517202.03 total 187.58 last 142.82) - open/close/unlink 26888 (time 1783517212.04 total 197.58 last 126.87) - open/close/unlink 27778 (time 1783517222.05 total 207.59 last 88.92) - open/close/unlink 28594 (time 1783517232.06 total 217.61 last 81.45) - open/close/unlink 29443 (time 1783517242.08 total 227.63 last 84.76) - open/close/unlink 30000 (time 1783517248.09 total 233.64 last 92.65) - open/close/unlink 31029 (time 1783517258.10 total 243.65 last 102.86) - open/close/unlink 32211 (time 1783517268.10 total 253.65 last 118.17) - open/close/unlink 33498 (time 1783517278.11 total 263.66 last 128.51) - open/close/unlink 34559 (time 1783517288.12 total 273.67 last 106.07) - open/close/unlink 35702 (time 1783517298.12 total 283.67 last 114.24) - open/close/unlink 37069 (time 1783517308.14 total 293.69 last 136.37) - open/close/unlink 38360 (time 1783517318.15 total 303.70 last 129.06) - open/close/unlink 39920 (time 1783517328.15 total 313.70 last 155.99) - open/close/unlink 41493 (time 1783517338.15 total 323.70 last 157.28) - open/close/unlink 43042 (time 1783517348.15 total 333.70 last 154.88) - open/close/unlink 44606 (time 1783517358.16 total 343.71 last 156.29) - open/close/unlink 46245 (time 1783517368.16 total 353.71 last 163.79) - open/close/unlink 47804 (time 1783517378.16 total 363.71 last 155.90) - open/close/unlink 49442 (time 1783517388.17 total 373.72 last 163.76) - open/close/unlink 50000 (time 1783517391.61 total 377.16 last 162.23) - open/close/unlink 51684 (time 1783517401.61 total 387.16 last 168.35) - open/close/unlink 53293 (time 1783517411.62 total 397.17 last 160.78) - open/close/unlink 54932 (time 1783517421.62 total 407.17 last 163.82) - open/close/unlink 56583 (time 1783517431.63 total 417.18 last 165.02) - open/close/unlink 58200 (time 1783517441.63 total 427.18 last 161.60) - open/close/unlink 59917 (time 1783517451.64 total 437.19 last 171.61) - open/close/unlink 61628 (time 1783517461.66 total 447.21 last 170.79) - open/close/unlink 63439 (time 1783517471.66 total 457.21 last 181.06) total: 64768 open/close/unlink in 467.04 seconds: 138.68 ops/second - open/close/unlink 1879 (time 1783517024.41 total 10.00 last 187.85) - open/close/unlink 3732 (time 1783517034.41 total 20.01 last 185.21) - open/close/unlink 5401 (time 1783517044.42 total 30.01 last 166.82) - open/close/unlink 6349 (time 1783517054.43 total 40.02 last 94.72) - open/close/unlink 7122 (time 1783517064.43 total 50.02 last 77.27) - open/close/unlink 7986 (time 1783517074.44 total 60.03 last 86.32) - open/close/unlink 9311 (time 1783517084.44 total 70.03 last 132.48) - open/close/unlink 10000 (time 1783517089.26 total 74.85 last 143.06) - open/close/unlink 11432 (time 1783517099.26 total 84.86 last 143.13) - open/close/unlink 12953 (time 1783517109.27 total 94.86 last 152.02) - open/close/unlink 14247 (time 1783517119.27 total 104.86 last 129.36) - open/close/unlink 15456 (time 1783517129.29 total 114.88 last 120.72) - open/close/unlink 16926 (time 1783517139.29 total 124.89 last 146.88) - open/close/unlink 18414 (time 1783517149.31 total 134.90 last 148.59) - open/close/unlink 19706 (time 1783517159.31 total 144.90 last 129.19) - open/close/unlink 20000 (time 1783517161.90 total 147.50 last 113.39) - open/close/unlink 21307 (time 1783517171.91 total 157.50 last 130.59) - open/close/unlink 22641 (time 1783517181.91 total 167.50 last 133.38) - open/close/unlink 24193 (time 1783517191.92 total 177.51 last 155.13) - open/close/unlink 25612 (time 1783517201.92 total 187.52 last 141.80) - open/close/unlink 26886 (time 1783517211.93 total 197.53 last 127.28) - open/close/unlink 27775 (time 1783517221.94 total 207.54 last 88.79) - open/close/unlink 28592 (time 1783517231.96 total 217.55 last 81.61) - open/close/unlink 29430 (time 1783517241.97 total 227.56 last 83.71) - open/close/unlink 30000 (time 1783517248.13 total 233.73 last 92.46) - open/close/unlink 31030 (time 1783517258.13 total 243.73 last 102.96) - open/close/unlink 32232 (time 1783517268.14 total 253.74 last 120.11) - open/close/unlink 33532 (time 1783517278.14 total 263.74 last 129.98) - open/close/unlink 34600 (time 1783517288.14 total 273.74 last 106.80) - open/close/unlink 35742 (time 1783517298.15 total 283.75 last 114.10) - open/close/unlink 37103 (time 1783517308.16 total 293.75 last 136.05) - open/close/unlink 38400 (time 1783517318.16 total 303.76 last 129.61) - open/close/unlink 39960 (time 1783517328.17 total 313.77 last 155.87) - open/close/unlink 41528 (time 1783517338.18 total 323.77 last 156.74) - open/close/unlink 43073 (time 1783517348.18 total 333.77 last 154.46) - open/close/unlink 44654 (time 1783517358.18 total 343.78 last 158.03) - open/close/unlink 46283 (time 1783517368.19 total 353.78 last 162.86) - open/close/unlink 47851 (time 1783517378.19 total 363.78 last 156.78) - open/close/unlink 49479 (time 1783517388.19 total 373.78 last 162.76) - open/close/unlink 50000 (time 1783517391.40 total 377.00 last 162.09) - open/close/unlink 51668 (time 1783517401.40 total 387.00 last 166.78) - open/close/unlink 53280 (time 1783517411.40 total 397.00 last 161.20) - open/close/unlink 54925 (time 1783517421.41 total 407.00 last 164.49) - open/close/unlink 56569 (time 1783517431.42 total 417.01 last 164.18) - open/close/unlink 58190 (time 1783517441.42 total 427.02 last 162.04) - open/close/unlink 59892 (time 1783517451.43 total 437.02 last 170.13) - open/close/unlink 61591 (time 1783517461.44 total 447.03 last 169.72) - open/close/unlink 63383 (time 1783517471.44 total 457.04 last 179.11) - open/close/unlink 64757 (time 1783517481.46 total 467.05 last 137.23) total: 64768 open/close/unlink in 467.20 seconds: 138.63 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete oleg338-server: cannot open 'lustre-ost2': no such pool pdsh@oleg338-client: oleg338-server: ssh exited with exit code 255 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg338-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg338-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg338-server stop mds service on oleg338-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg338-server unloading modules via unload_modules_local on: 'oleg338-server' oleg338-server: oleg338-server.virtnet: executing unload_modules_local oleg338-server: modules unloaded. pdsh@oleg338-client: oleg338-client: ssh exited with exit code 2 pdsh@oleg338-client: oleg338-server: ssh exited with exit code 2 pdsh@oleg338-client: oleg338-server: ssh exited with exit code 2 pdsh@oleg338-client: oleg338-client: ssh exited with exit code 2