== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 15:58:57 (1782849537) start mds service on oleg217-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg217-server: oleg217-server.virtnet: executing set_default_debug -1 all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg217-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg217-server: oleg217-server.virtnet: executing set_default_debug -1 all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Started lustre-OST0000 oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 92553 329 92224 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 12329 425 11904 4% /mnt/lustre[OST:0] filesystem_summary: 12233 329 11904 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close/unlink 1895 (time 1782849563.41 total 10.00 last 189.48) - open/close/unlink 3589 (time 1782849573.41 total 20.00 last 169.35) - open/close/unlink 5463 (time 1782849583.42 total 30.01 last 187.30) - open/close/unlink 7270 (time 1782849593.42 total 40.01 last 180.61) - open/close/unlink 9025 (time 1782849603.42 total 50.02 last 175.47) - open/close/unlink 10000 (time 1782849608.67 total 55.26 last 185.89) - open/close/unlink 11803 (time 1782849618.67 total 65.26 last 180.22) - open/close/unlink 13698 (time 1782849628.68 total 75.27 last 189.42) - open/close/unlink 15552 (time 1782849638.68 total 85.27 last 185.36) - open/close/unlink 17417 (time 1782849648.68 total 95.27 last 186.45) - open/close/unlink 19141 (time 1782849658.71 total 105.30 last 171.87) - open/close/unlink 20000 (time 1782849665.18 total 111.77 last 132.84) - open/close/unlink 21494 (time 1782849675.18 total 121.77 last 149.38) - open/close/unlink 22865 (time 1782849685.18 total 131.77 last 137.07) - open/close/unlink 24356 (time 1782849695.18 total 141.78 last 149.07) - open/close/unlink 25963 (time 1782849705.19 total 151.78 last 160.65) - open/close/unlink 27249 (time 1782849715.19 total 161.78 last 128.60) - open/close/unlink 28508 (time 1782849725.19 total 171.78 last 125.84) - open/close/unlink 30000 (time 1782849734.22 total 180.82 last 165.19) - open/close/unlink 31181 (time 1782849744.23 total 190.82 last 118.03) - open/close/unlink 32423 (time 1782849754.23 total 200.83 last 124.16) - open/close/unlink 33644 (time 1782849764.24 total 210.83 last 122.05) - open/close/unlink 34868 (time 1782849774.24 total 220.83 last 122.35) - open/close/unlink 36054 (time 1782849784.24 total 230.84 last 118.57) - open/close/unlink 37286 (time 1782849794.25 total 240.84 last 123.19) - open/close/unlink 38491 (time 1782849804.25 total 250.84 last 120.47) - open/close/unlink 39727 (time 1782849814.25 total 260.85 last 123.53) - open/close/unlink 40000 (time 1782849816.54 total 263.13 last 119.28) - open/close/unlink 41249 (time 1782849826.56 total 273.15 last 124.65) - open/close/unlink 42476 (time 1782849836.60 total 283.19 last 122.25) - open/close/unlink 43684 (time 1782849846.60 total 293.19 last 120.75) - open/close/unlink 44921 (time 1782849856.61 total 303.20 last 123.61) - open/close/unlink 46148 (time 1782849866.61 total 313.21 last 122.65) - open/close/unlink 47335 (time 1782849876.62 total 323.21 last 118.60) - open/close/unlink 48524 (time 1782849886.63 total 333.22 last 118.86) - open/close/unlink 49691 (time 1782849896.63 total 343.22 last 116.68) - open/close/unlink 50000 (time 1782849900.10 total 346.69 last 89.02) - open/close/unlink 51118 (time 1782849910.10 total 356.70 last 111.74) - open/close/unlink 52400 (time 1782849920.11 total 366.70 last 128.16) - open/close/unlink 54103 (time 1782849930.11 total 376.70 last 170.26) - open/close/unlink 55398 (time 1782849940.11 total 386.71 last 129.44) - open/close/unlink 56645 (time 1782849950.13 total 396.72 last 124.51) - open/close/unlink 57901 (time 1782849960.13 total 406.73 last 125.54) - open/close/unlink 59009 (time 1782849970.14 total 416.73 last 110.76) - open/close/unlink 60000 (time 1782849978.27 total 424.86 last 121.83) - open/close/unlink 61232 (time 1782849988.28 total 434.87 last 123.13) - open/close/unlink 62480 (time 1782849998.28 total 444.87 last 124.73) - open/close/unlink 63634 (time 1782850008.29 total 454.88 last 115.36) total: 64768 open/close/unlink in 464.81 seconds: 139.34 ops/second - open/close/unlink 1899 (time 1782849563.57 total 10.00 last 189.87) - open/close/unlink 3608 (time 1782849573.57 total 20.01 last 170.84) - open/close/unlink 5468 (time 1782849583.58 total 30.01 last 185.86) - open/close/unlink 7246 (time 1782849593.58 total 40.01 last 177.76) - open/close/unlink 9010 (time 1782849603.58 total 50.02 last 176.39) - open/close/unlink 10000 (time 1782849608.86 total 55.29 last 187.56) - open/close/unlink 11796 (time 1782849618.86 total 65.30 last 179.51) - open/close/unlink 13651 (time 1782849628.86 total 75.30 last 185.46) - open/close/unlink 15533 (time 1782849638.87 total 85.30 last 188.18) - open/close/unlink 17388 (time 1782849648.87 total 95.31 last 185.45) - open/close/unlink 19091 (time 1782849658.88 total 105.32 last 170.03) - open/close/unlink 20000 (time 1782849665.54 total 111.98 last 136.48) - open/close/unlink 21465 (time 1782849675.55 total 121.99 last 146.44) - open/close/unlink 22868 (time 1782849685.55 total 131.99 last 140.28) - open/close/unlink 24353 (time 1782849695.56 total 141.99 last 148.42) - open/close/unlink 25947 (time 1782849705.56 total 151.99 last 159.39) - open/close/unlink 27228 (time 1782849715.56 total 162.00 last 128.02) - open/close/unlink 28539 (time 1782849725.57 total 172.00 last 131.03) - open/close/unlink 30000 (time 1782849735.26 total 181.70 last 150.75) - open/close/unlink 31194 (time 1782849745.27 total 191.70 last 119.30) - open/close/unlink 32452 (time 1782849755.28 total 201.72 last 125.60) - open/close/unlink 33688 (time 1782849765.29 total 211.72 last 123.55) - open/close/unlink 34938 (time 1782849775.29 total 221.73 last 124.96) - open/close/unlink 36139 (time 1782849785.30 total 231.73 last 120.04) - open/close/unlink 37352 (time 1782849795.31 total 241.75 last 121.11) - open/close/unlink 38558 (time 1782849805.31 total 251.75 last 120.56) - open/close/unlink 39772 (time 1782849815.32 total 261.75 last 121.38) - open/close/unlink 40000 (time 1782849817.78 total 264.22 last 92.50) - open/close/unlink 41209 (time 1782849827.82 total 274.26 last 120.40) - open/close/unlink 42434 (time 1782849837.83 total 284.26 last 122.42) - open/close/unlink 43638 (time 1782849847.83 total 294.27 last 120.38) - open/close/unlink 44902 (time 1782849857.83 total 304.27 last 126.39) - open/close/unlink 46125 (time 1782849867.85 total 314.28 last 122.09) - open/close/unlink 47313 (time 1782849877.86 total 324.29 last 118.71) - open/close/unlink 48543 (time 1782849887.86 total 334.29 last 123.00) - open/close/unlink 49739 (time 1782849897.87 total 344.30 last 119.47) - open/close/unlink 50000 (time 1782849900.80 total 347.23 last 89.05) - open/close/unlink 51096 (time 1782849910.80 total 357.24 last 109.54) - open/close/unlink 52433 (time 1782849920.81 total 367.24 last 133.67) - open/close/unlink 54136 (time 1782849930.81 total 377.25 last 170.20) - open/close/unlink 55411 (time 1782849940.82 total 387.25 last 127.42) - open/close/unlink 56617 (time 1782849950.82 total 397.25 last 120.59) - open/close/unlink 57857 (time 1782849960.82 total 407.26 last 123.99) - open/close/unlink 58952 (time 1782849970.82 total 417.26 last 109.44) - open/close/unlink 60000 (time 1782849979.15 total 425.59 last 125.87) - open/close/unlink 61232 (time 1782849989.16 total 435.59 last 123.14) - open/close/unlink 62483 (time 1782849999.16 total 445.59 last 125.09) - open/close/unlink 63638 (time 1782850009.16 total 455.60 last 115.43) - open/close/unlink 64698 (time 1782850019.19 total 465.62 last 105.75) total: 64768 open/close/unlink in 468.34 seconds: 138.29 ops/second - open/close/unlink 1882 (time 1782849563.44 total 10.00 last 188.15) - open/close/unlink 3587 (time 1782849573.44 total 20.01 last 170.44) - open/close/unlink 5460 (time 1782849583.44 total 30.01 last 187.25) - open/close/unlink 7256 (time 1782849593.45 total 40.01 last 179.50) - open/close/unlink 9014 (time 1782849603.45 total 50.02 last 175.72) - open/close/unlink 10000 (time 1782849608.79 total 55.35 last 184.89) - open/close/unlink 11797 (time 1782849618.79 total 65.35 last 179.65) - open/close/unlink 13667 (time 1782849628.79 total 75.36 last 186.94) - open/close/unlink 15534 (time 1782849638.79 total 85.36 last 186.64) - open/close/unlink 17372 (time 1782849648.79 total 95.36 last 183.80) - open/close/unlink 19080 (time 1782849658.82 total 105.38 last 170.43) - open/close/unlink 20000 (time 1782849665.46 total 112.03 last 138.49) - open/close/unlink 21467 (time 1782849675.46 total 122.03 last 146.66) - open/close/unlink 22858 (time 1782849685.46 total 132.03 last 139.07) - open/close/unlink 24343 (time 1782849695.47 total 142.03 last 148.47) - open/close/unlink 25932 (time 1782849705.47 total 152.04 last 158.80) - open/close/unlink 27231 (time 1782849715.47 total 162.04 last 129.87) - open/close/unlink 28521 (time 1782849725.48 total 172.04 last 128.98) - open/close/unlink 30000 (time 1782849734.99 total 181.56 last 155.45) - open/close/unlink 31184 (time 1782849745.01 total 191.57 last 118.21) - open/close/unlink 32432 (time 1782849755.02 total 201.58 last 124.67) - open/close/unlink 33676 (time 1782849765.03 total 211.60 last 124.25) - open/close/unlink 34928 (time 1782849775.04 total 221.61 last 125.07) - open/close/unlink 36124 (time 1782849785.04 total 231.61 last 119.57) - open/close/unlink 37352 (time 1782849795.05 total 241.61 last 122.75) - open/close/unlink 38545 (time 1782849805.05 total 251.62 last 119.26) - open/close/unlink 39764 (time 1782849815.05 total 261.62 last 121.87) - open/close/unlink 40000 (time 1782849817.49 total 264.06 last 96.62) - open/close/unlink 41227 (time 1782849827.50 total 274.07 last 122.58) - open/close/unlink 42451 (time 1782849837.51 total 284.08 last 122.35) - open/close/unlink 43636 (time 1782849847.51 total 294.08 last 118.44) - open/close/unlink 44899 (time 1782849857.52 total 304.08 last 126.25) - open/close/unlink 46138 (time 1782849867.53 total 314.09 last 123.80) - open/close/unlink 47331 (time 1782849877.54 total 324.10 last 119.18) - open/close/unlink 48548 (time 1782849887.55 total 334.12 last 121.48) - open/close/unlink 49729 (time 1782849897.56 total 344.13 last 117.98) - open/close/unlink 50000 (time 1782849900.60 total 347.17 last 89.26) - open/close/unlink 51108 (time 1782849910.60 total 357.17 last 110.77) - open/close/unlink 52421 (time 1782849920.61 total 367.17 last 131.25) - open/close/unlink 54129 (time 1782849930.61 total 377.18 last 170.76) - open/close/unlink 55401 (time 1782849940.61 total 387.18 last 127.15) - open/close/unlink 56620 (time 1782849950.62 total 397.18 last 121.86) - open/close/unlink 57861 (time 1782849960.63 total 407.20 last 123.91) - open/close/unlink 58956 (time 1782849970.63 total 417.20 last 109.46) - open/close/unlink 60000 (time 1782849978.92 total 425.49 last 125.97) - open/close/unlink 61233 (time 1782849988.93 total 435.50 last 123.16) - open/close/unlink 62454 (time 1782849998.94 total 445.51 last 122.01) - open/close/unlink 63614 (time 1782850008.94 total 455.51 last 115.95) - open/close/unlink 64683 (time 1782850018.95 total 465.51 last 106.87) total: 64768 open/close/unlink in 468.53 seconds: 138.24 ops/second - open/close/unlink 1879 (time 1782849563.41 total 10.00 last 187.87) - open/close/unlink 3576 (time 1782849573.41 total 20.00 last 169.69) - open/close/unlink 5449 (time 1782849583.42 total 30.01 last 187.24) - open/close/unlink 7261 (time 1782849593.42 total 40.01 last 181.19) - open/close/unlink 9023 (time 1782849603.42 total 50.01 last 176.12) - open/close/unlink 10000 (time 1782849608.72 total 55.31 last 184.49) - open/close/unlink 11794 (time 1782849618.72 total 65.31 last 179.37) - open/close/unlink 13659 (time 1782849628.73 total 75.31 last 186.40) - open/close/unlink 15526 (time 1782849638.73 total 85.31 last 186.68) - open/close/unlink 17380 (time 1782849648.73 total 95.31 last 185.39) - open/close/unlink 19087 (time 1782849658.73 total 105.32 last 170.59) - open/close/unlink 20000 (time 1782849665.47 total 112.05 last 135.62) - open/close/unlink 21471 (time 1782849675.47 total 122.06 last 147.06) - open/close/unlink 22866 (time 1782849685.47 total 132.06 last 139.48) - open/close/unlink 24360 (time 1782849695.47 total 142.06 last 149.33) - open/close/unlink 25959 (time 1782849705.48 total 152.07 last 159.80) - open/close/unlink 27245 (time 1782849715.49 total 162.07 last 128.52) - open/close/unlink 28539 (time 1782849725.49 total 172.08 last 129.34) - open/close/unlink 30000 (time 1782849734.83 total 181.42 last 156.50) - open/close/unlink 31182 (time 1782849744.83 total 191.42 last 118.12) - open/close/unlink 32423 (time 1782849754.84 total 201.43 last 124.02) - open/close/unlink 33671 (time 1782849764.85 total 211.44 last 124.68) - open/close/unlink 34914 (time 1782849774.85 total 221.44 last 124.26) - open/close/unlink 36105 (time 1782849784.86 total 231.45 last 119.05) - open/close/unlink 37309 (time 1782849794.86 total 241.45 last 120.34) - open/close/unlink 38496 (time 1782849804.87 total 251.45 last 118.66) - open/close/unlink 39718 (time 1782849814.87 total 261.46 last 122.13) - open/close/unlink 40000 (time 1782849817.67 total 264.26 last 100.69) - open/close/unlink 41213 (time 1782849827.68 total 274.27 last 121.21) - open/close/unlink 42423 (time 1782849837.68 total 284.27 last 120.96) - open/close/unlink 43629 (time 1782849847.70 total 294.29 last 120.42) - open/close/unlink 44889 (time 1782849857.71 total 304.29 last 125.91) - open/close/unlink 46124 (time 1782849867.71 total 314.30 last 123.44) - open/close/unlink 47314 (time 1782849877.71 total 324.30 last 119.00) - open/close/unlink 48522 (time 1782849887.72 total 334.31 last 120.64) - open/close/unlink 49702 (time 1782849897.73 total 344.32 last 117.95) - open/close/unlink 50000 (time 1782849900.99 total 347.57 last 91.48) - open/close/unlink 51114 (time 1782849910.99 total 357.58 last 111.38) - open/close/unlink 52465 (time 1782849920.99 total 367.58 last 135.08) - open/close/unlink 54154 (time 1782849930.99 total 377.58 last 168.85) - open/close/unlink 55410 (time 1782849940.99 total 387.58 last 125.60) - open/close/unlink 56639 (time 1782849951.00 total 397.58 last 122.87) - open/close/unlink 57881 (time 1782849961.00 total 407.59 last 124.14) - open/close/unlink 58959 (time 1782849971.00 total 417.59 last 107.78) - open/close/unlink 60000 (time 1782849979.17 total 425.76 last 127.40) - open/close/unlink 61230 (time 1782849989.18 total 435.76 last 122.97) - open/close/unlink 62476 (time 1782849999.18 total 445.77 last 124.57) - open/close/unlink 63632 (time 1782850009.19 total 455.77 last 115.50) - open/close/unlink 64689 (time 1782850019.21 total 465.79 last 105.49) total: 64768 open/close/unlink in 468.69 seconds: 138.19 ops/second - open/close/unlink 1902 (time 1782849563.54 total 10.00 last 190.11) - open/close/unlink 3608 (time 1782849573.54 total 20.01 last 170.57) - open/close/unlink 5482 (time 1782849583.55 total 30.01 last 187.34) - open/close/unlink 7274 (time 1782849593.55 total 40.01 last 179.16) - open/close/unlink 9045 (time 1782849603.55 total 50.01 last 177.08) - open/close/unlink 10000 (time 1782849608.77 total 55.23 last 183.02) - open/close/unlink 11805 (time 1782849618.77 total 65.23 last 180.48) - open/close/unlink 13680 (time 1782849628.77 total 75.24 last 187.42) - open/close/unlink 15549 (time 1782849638.78 total 85.24 last 186.84) - open/close/unlink 17385 (time 1782849648.78 total 95.24 last 183.54) - open/close/unlink 19085 (time 1782849658.79 total 105.25 last 169.86) - open/close/unlink 20000 (time 1782849665.50 total 111.97 last 136.26) - open/close/unlink 21462 (time 1782849675.51 total 121.97 last 146.12) - open/close/unlink 22857 (time 1782849685.51 total 131.97 last 139.46) - open/close/unlink 24331 (time 1782849695.51 total 141.98 last 147.37) - open/close/unlink 25925 (time 1782849705.52 total 151.98 last 159.33) - open/close/unlink 27195 (time 1782849715.52 total 161.98 last 126.96) - open/close/unlink 28495 (time 1782849725.52 total 171.99 last 129.98) - open/close/unlink 29993 (time 1782849735.60 total 182.06 last 148.65) - open/close/unlink 31199 (time 1782849745.60 total 192.07 last 120.56) - open/close/unlink 32451 (time 1782849755.61 total 202.07 last 125.09) - open/close/unlink 33702 (time 1782849765.62 total 212.09 last 124.93) - open/close/unlink 34938 (time 1782849775.64 total 222.10 last 123.43) - open/close/unlink 36132 (time 1782849785.65 total 232.12 last 119.25) - open/close/unlink 37352 (time 1782849795.65 total 242.12 last 121.98) - open/close/unlink 38567 (time 1782849805.66 total 252.12 last 121.44) - open/close/unlink 39798 (time 1782849815.66 total 262.12 last 123.10) - open/close/unlink 40000 (time 1782849817.97 total 264.44 last 87.25) - open/close/unlink 41198 (time 1782849827.98 total 274.45 last 119.69) - open/close/unlink 42430 (time 1782849837.98 total 284.45 last 123.19) - open/close/unlink 43612 (time 1782849848.00 total 294.46 last 118.04) - open/close/unlink 44873 (time 1782849858.00 total 304.47 last 126.01) - open/close/unlink 46109 (time 1782849868.01 total 314.48 last 123.50) - open/close/unlink 47298 (time 1782849878.02 total 324.48 last 118.82) - open/close/unlink 48533 (time 1782849888.03 total 334.49 last 123.42) - open/close/unlink 49720 (time 1782849898.08 total 344.54 last 118.08) - open/close/unlink 50000 (time 1782849901.03 total 347.49 last 94.87) - open/close/unlink 51111 (time 1782849911.03 total 357.50 last 111.08) - open/close/unlink 52458 (time 1782849921.04 total 367.50 last 134.64) - open/close/unlink 54138 (time 1782849931.04 total 377.51 last 167.90) - open/close/unlink 55388 (time 1782849941.05 total 387.52 last 124.88) - open/close/unlink 56606 (time 1782849951.06 total 397.52 last 121.75) - open/close/unlink 57829 (time 1782849961.06 total 407.52 last 122.25) - open/close/unlink 58920 (time 1782849971.06 total 417.52 last 109.10) - open/close/unlink 60000 (time 1782849979.56 total 426.02 last 127.08) - open/close/unlink 61238 (time 1782849989.56 total 436.02 last 123.79) - open/close/unlink 62471 (time 1782849999.56 total 446.03 last 123.24) - open/close/unlink 63607 (time 1782850009.57 total 456.04 last 113.53) - open/close/unlink 64630 (time 1782850019.59 total 466.05 last 102.13) total: 64768 open/close/unlink in 470.80 seconds: 137.57 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete oleg217-server: cannot open 'lustre-ost2': no such pool pdsh@oleg217-client: oleg217-server: ssh exited with exit code 255 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg217-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg217-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg217-server stop mds service on oleg217-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg217-server unloading modules via unload_modules_local on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing unload_modules_local oleg217-server: modules unloaded. pdsh@oleg217-client: oleg217-client: ssh exited with exit code 2 pdsh@oleg217-client: oleg217-server: ssh exited with exit code 2 pdsh@oleg217-client: oleg217-server: ssh exited with exit code 2 pdsh@oleg217-client: oleg217-client: ssh exited with exit code 2