[ 1865.935237] LustreError: lustre-OST0000-osc-ffffa0fe457bc000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1865.949985] Lustre: 2357:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.119@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1865.963347] Lustre: lustre-OST0000-osc-ffffa0fe457bc000: Connection restored to 192.168.201.119@tcp (at 192.168.201.119@tcp)
[ 1865.967680] Lustre: Skipped 14 previous similar messages
[ 1871.023386] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 04:10:57 (1787731857)
[ 1872.761773] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1874.169149] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 04:11:00 (1787731860)
[ 1875.436866] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1876.826688] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 04:11:02 (1787731862)
[ 1880.392564] LustreError: lustre-MDT0000-mdc-ffffa0fe457bc000: operation mds_reint to node 192.168.201.119@tcp failed: rc = -19
[ 1880.402675] LustreError: Skipped 1 previous similar message
[ 1897.439598] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 192.168.201.119@tcp) was lost; in progress operations using this service will fail
[ 1907.648485] Lustre: Evicted from MGS (at 192.168.201.119@tcp) after server handle changed from 0x2f871d822482e23e to 0x2f871d822482fab7
[ 1999.122921] LustreError: lustre-MDT0000-mdc-ffffa0fe457bc000: operation mds_reint to node 192.168.201.119@tcp failed: rc = -19
[ 2020.768530] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 192.168.201.119@tcp) was lost; in progress operations using this service will fail
[ 2020.831321] Lustre: Evicted from MGS (at 192.168.201.119@tcp) after server handle changed from 0x2f871d822482fab7 to 0x2f871d8224854e3c
[ 2022.717099] Lustre: 15940:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.119@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2044.009724] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 04:13:50 (1787732030)
[ 2044.281034] Lustre: *** cfs_fail_loc=305, val=0***
[ 2044.282800] Lustre: Skipped 4 previous similar messages
[ 2063.560619] LustreError: lustre-OST0000-osc-ffffa0fe457bc000: operation ost_connect to node 192.168.201.119@tcp failed: rc = -75
[ 2063.569638] LustreError: Skipped 1 previous similar message
[ 2090.463474] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 192.168.201.119@tcp) was lost; in progress operations using this service will fail
[ 2101.551161] Lustre: Evicted from MGS (at 192.168.201.119@tcp) after server handle changed from 0x2f871d8224854e3c to 0x2f871d82248557f8
[ 2105.192818] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2106.289146] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2113.405566] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 04:14:59 (1787732099)
[ 2127.337372] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 192.168.201.119@tcp) was lost; in progress operations using this service will fail
[ 2127.359540] Lustre: Evicted from MGS (at 192.168.201.119@tcp) after server handle changed from 0x2f871d82248557f8 to 0x2f871d8224855a98
[ 2127.387271] LustreError: lustre-MDT0000-mdc-ffffa0fe457bc000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2143.146811] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 04:15:29 (1787732129)
[ 2168.820063] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 04:15:55 (1787732155)
[ 2200.162226] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 192.168.201.119@tcp) was lost; in progress operations using this service will fail
[ 2200.199974] Lustre: Evicted from MGS (at 192.168.201.119@tcp) after server handle changed from 0x2f871d8224855a98 to 0x2f871d822485c60e
[ 2224.377804] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2226.258862] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2290.374976] LustreError: lustre-MDT0000-mdc-ffffa0fe457bc000: operation mds_close to node 192.168.201.119@tcp failed: rc = -19
[ 2290.387623] LustreError: Skipped 1 previous similar message
[ 2310.128844] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 192.168.201.119@tcp) was lost; in progress operations using this service will fail
[ 2310.159523] Lustre: Evicted from MGS (at 192.168.201.119@tcp) after server handle changed from 0x2f871d822485c60e to 0x2f871d822487f868
[ 2311.465133] Lustre: 15940:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.119@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2337.069491] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2339.093549] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2422.766259] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 192.168.201.119@tcp) was lost; in progress operations using this service will fail
[ 2422.789920] Lustre: Evicted from MGS (at 192.168.201.119@tcp) after server handle changed from 0x2f871d822487f868 to 0x2f871d82248a0a8c
[ 2437.648090] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2439.808453] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2467.787751] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 04:20:53 (1787732453)
[ 2471.847298] Lustre: lustre-MDT0000-mdc-ffffa0fe457bc000: Connection to lustre-MDT0000 (at 192.168.201.119@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2471.862539] Lustre: Skipped 9 previous similar messages
[ 2490.335300] Lustre: 2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787732461/real 1787732461]  req@ffffa0fe47ad1880 x1874570820893952/t0(0) o400->MGC192.168.201.119@tcp@192.168.201.119@tcp:26/25 lens 224/224 e 0 to 1 dl 1787732477 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2490.387274] Lustre: 2357:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 13 previous similar messages
[ 2490.403183] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 192.168.201.119@tcp) was lost; in progress operations using this service will fail
[ 2499.009465] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2500.583488] Lustre: Evicted from MGS (at 192.168.201.119@tcp) after server handle changed from 0x2f871d82248a0a8c to 0x2f871d82248b4489
[ 2500.598390] Lustre: MGC192.168.201.119@tcp: Connection restored to 192.168.201.119@tcp (at 192.168.201.119@tcp)
[ 2500.613538] Lustre: Skipped 14 previous similar messages
[ 2526.129150] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2560.604345] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2574.106357] LustreError: lustre-MDT0000-mdc-ffffa0fe457bc000: operation mds_close to node 192.168.201.119@tcp failed: rc = -19
[ 2574.111229] LustreError: Skipped 6 previous similar messages
[ 2604.229171] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2646.002263] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 192.168.201.119@tcp) was lost; in progress operations using this service will fail
[ 2646.019386] LustreError: Skipped 3 previous similar messages
[ 2646.038852] Lustre: Evicted from MGS (at 192.168.201.119@tcp) after server handle changed from 0x2f871d82248be295 to 0x2f871d82248c8310
[ 2646.050184] Lustre: Skipped 3 previous similar messages
[ 2653.816821] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2707.975090] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2787.139861] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 04:26:13 (1787732773)
[ 2839.085858] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 2841.235116] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3131.365814] LustreError: lustre-OST0000-osc-ffffa0fe457bc000: operation ldlm_enqueue to node 192.168.201.119@tcp failed: rc = -107
[ 3131.377166] LustreError: Skipped 7 previous similar messages
[ 3131.378506] Lustre: lustre-OST0000-osc-ffffa0fe457bc000: Connection to lustre-OST0000 (at 192.168.201.119@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3131.399095] Lustre: Skipped 6 previous similar messages
[ 3165.749122] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3168.353976] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3496.907473] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3498.647059] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3756.615372] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 04:42:22 (1787733742)
[ 3772.895380] Lustre: 50631:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787733744/real 1787733744]  req@ffffa0fe47341500 x1874570846603136/t0(0) o101->lustre-MDT0000-mdc-ffffa0fe457bc000@192.168.201.119@tcp:12/10 lens 576/1152 e 0 to 1 dl 1787733760 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3772.919907] Lustre: 50631:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 3772.926512] Lustre: lustre-MDT0000-mdc-ffffa0fe457bc000: Connection to lustre-MDT0000 (at 192.168.201.119@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3772.948569] Lustre: Skipped 1 previous similar message
[ 3772.969425] Lustre: lustre-MDT0000-mdc-ffffa0fe457bc000: Connection restored to 192.168.201.119@tcp (at 192.168.201.119@tcp)
[ 3772.977855] Lustre: Skipped 12 previous similar messages
[ 3778.905663] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 04:42:45 (1787733765)
[ 3803.173190] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 04:43:09 (1787733789)
[ 3829.515711] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 04:43:34 (1787733814)
[ 3830.073844] Lustre: Mounted lustre-client - version 2.17.57_80_g62ff528
[ 3860.959992] Lustre: 2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787733832/real 1787733832]  req@ffffa0fe4678c700 x1874570846621056/t0(0) o400->MGC192.168.201.119@tcp@192.168.201.119@tcp:26/25 lens 224/224 e 0 to 1 dl 1787733848 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3860.981418] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 192.168.201.119@tcp) was lost; in progress operations using this service will fail
[ 3861.003872] Lustre: 2357:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
[ 3861.024883] LustreError: Skipped 2 previous similar messages
[ 3861.049879] Lustre: Evicted from MGS (at 192.168.201.119@tcp) after server handle changed from 0x2f871d82248eb6ba to 0x2f871d8224a97768
[ 3861.059726] Lustre: Skipped 2 previous similar messages
[ 3861.067328] Lustre: MGC192.168.201.119@tcp: Connection restored to 192.168.201.119@tcp (at 192.168.201.119@tcp)
[ 3861.080401] Lustre: Skipped 2 previous similar messages
[ 3872.481960] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 3874.129833] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3876.338763] Lustre: Unmounted lustre-client
[ 3882.780965] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 04:44:28 (1787733868)
[ 4018.672645] Lustre: 2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787733989/real 1787733989]  req@ffffa0fe466b6680 x1874570846652928/t0(0) o4->lustre-OST0000-osc-ffffa0fe457bc000@192.168.201.119@tcp:6/4 lens 488/448 e 0 to 1 dl 1787734005 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4018.689099] Lustre: lustre-OST0000-osc-ffffa0fe457bc000: Connection restored to 192.168.201.119@tcp (at 192.168.201.119@tcp)
[ 4018.691454] Lustre: 2355:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 65 previous similar messages
[ 4018.718554] Lustre: Skipped 8 previous similar messages
[ 4175.710243] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 04:49:21 (1787734161)
[ 4224.380769] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 04:50:10 (1787734210)
[ 4228.256228] Lustre: Unmounted lustre-client
[ 4254.116202] Lustre: Mounted lustre-client - version 2.17.57_80_g62ff528
[ 4261.775951] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 04:50:47 (1787734247)
[ 4262.212897] LustreError: 56552:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4263.271137] LustreError: 56552:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4263.578783] Lustre: *** cfs_fail_loc=305, val=0***
[ 4286.674719] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 04:51:12 (1787734272)
[ 4287.043915] Lustre: Mounted lustre-client - version 2.17.57_80_g62ff528
[ 4289.127421] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4299.546447] Lustre: Unmounted lustre-client
[ 4309.040475] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 04:51:34 (1787734294)
[ 4387.389211] LustreError: lustre-MDT0000-mdc-ffffa0fe5255d000: operation mds_reint to node 192.168.201.119@tcp failed: rc = -19
[ 4387.403755] LustreError: Skipped 1 previous similar message
[ 4387.406872] Lustre: lustre-MDT0000-mdc-ffffa0fe5255d000: Connection to lustre-MDT0000 (at 192.168.201.119@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4387.418210] Lustre: Skipped 23 previous similar messages
[ 4405.727391] Lustre: 2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787734376/real 1787734376]  req@ffffa0fe45c04700 x1874570849337728/t0(0) o400->MGC192.168.201.119@tcp@192.168.201.119@tcp:26/25 lens 224/224 e 0 to 1 dl 1787734392 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4405.765561] Lustre: 2355:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 83 previous similar messages
[ 4405.783442] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 192.168.201.119@tcp) was lost; in progress operations using this service will fail
[ 4414.946674] Lustre: Evicted from MGS (at 192.168.201.119@tcp) after server handle changed from 0x2f871d8224a98fd3 to 0x2f871d8224ac4742
[ 4414.960690] Lustre: MGC192.168.201.119@tcp: Connection restored to 192.168.201.119@tcp (at 192.168.201.119@tcp)
[ 4414.974389] Lustre: Skipped 11 previous similar messages
[ 4557.496539] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 04:55:43 (1787734543)
[ 4562.888970] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4577.776019] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 192.168.201.119@tcp) was lost; in progress operations using this service will fail
[ 4577.796745] Lustre: Evicted from MGS (at 192.168.201.119@tcp) after server handle changed from 0x2f871d8224ac4742 to 0x2f871d8224b10b10
[ 4577.846360] LustreError: lustre-MDT0000-mdc-ffffa0fe5255d000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4589.904320] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 04:56:15 (1787734575)
[ 4590.245332] Lustre: Mounted lustre-client - version 2.17.57_80_g62ff528
[ 4597.494398] LustreError: lustre-OST0000-osc-ffffa0fe5255d000: operation ldlm_enqueue to node 192.168.201.119@tcp failed: rc = -107
[ 4597.528830] LustreError: lustre-OST0000-osc-ffffa0fe5255d000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4609.739311] Lustre: Unmounted lustre-client
[ 4616.788298] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 04:56:42 (1787734602)
[ 4623.563673] LustreError: lustre-MDT0000-mdc-ffffa0fe5255d000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4623.584755] LustreError: 60939:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4630.647869] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 04:56:56 (1787734616)
[ 4630.825286] LustreError: 61639:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4636.128061] LustreError: 61639:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=0
[ 4636.139847] LustreError: 61639:0:(import.c:711:ptlrpc_connect_import_locked()) already connected
[ 4636.626697] LustreError: lustre-MDT0000-mdc-ffffa0fe5255d000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4636.639266] LustreError: 61660:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4636.720969] LustreError: 61665:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4637.811196] LustreError: 61672:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4637.824252] LustreError: 61672:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4640.164461] LustreError: 61694:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4640.173773] LustreError: 61694:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4644.863280] LustreError: 61739:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4644.871355] LustreError: 61739:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4654.146508] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 04:57:19 (1787734639)
[ 4693.476634] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4694.752788] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4703.053977] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 04:58:09 (1787734689)
[ 4739.836171] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4741.324532] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4750.923654] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 04:58:56 (1787734736)
[ 4811.880601] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4813.126318] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4819.934406] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 05:00:05 (1787734805)
[ 4858.391734] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4860.481102] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4866.659495] Lustre: Unmounted lustre-client
[ 4918.024813] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4920.508929] Lustre: Mounted lustre-client - version 2.17.57_80_g62ff528
[ 4926.925380] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 05:01:52 (1787734912)
[ 4929.026695] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 4930.888924] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 05:01:56 (1787734916)
[ 4958.262806] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 05:02:24 (1787734944)
[ 4959.782587] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 4961.528388] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 05:02:27 (1787734947)
[ 4962.756704] Lustre: *** cfs_fail_loc=805, val=0***
[ 4962.809604] Lustre: Mounted lustre-client - version 2.17.57_80_g62ff528
[ 4966.627982] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4988.392524] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 192.168.201.119@tcp) was lost; in progress operations using this service will fail
[ 4988.428333] Lustre: Evicted from MGS (at 192.168.201.119@tcp) after server handle changed from 0x2f871d8224b11519 to 0x2f871d8224b118b5
[ 4991.814590] Lustre: Unmounted lustre-client
[ 4997.431493] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 05:03:03 (1787734983)
[ 5013.983215] Lustre: 71471:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787734985/real 1787734985]  req@ffffa0fe526d1880 x1874570851385088/t0(0) o36->lustre-MDT0000-mdc-ffffa0fe471b3000@192.168.201.119@tcp:12/10 lens 504/576 e 0 to 1 dl 1787735001 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'mkdir.0' uid:0 gid:0 projid:4294967295
[ 5014.030556] Lustre: 71471:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5014.037663] Lustre: lustre-MDT0000-mdc-ffffa0fe471b3000: Connection to lustre-MDT0000 (at 192.168.201.119@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5014.066858] Lustre: Skipped 15 previous similar messages
[ 5018.088894] Lustre: MGC192.168.201.119@tcp: Connection restored to 192.168.201.119@tcp (at 192.168.201.119@tcp)
[ 5018.092924] Lustre: Skipped 16 previous similar messages
[ 5024.269582] LustreError: 2353:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffffa0fe51300000 x1874570851384576/t90194313220(90194313220) o101->lustre-MDT0000-mdc-ffffa0fe471b3000@192.168.201.119@tcp:12/10 lens 664/608 e 0 to 0 dl 1787735027 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5028.258607] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5029.395401] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5035.405560] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 05:03:41 (1787735021)
[ 5036.558966] LustreError: lustre-OST0000-osc-ffffa0fe471b3000: operation ost_write to node 192.168.201.119@tcp failed: rc = -107
[ 5036.569941] LustreError: Skipped 2 previous similar messages
[ 5036.595563] LustreError: lustre-OST0000-osc-ffffa0fe471b3000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5036.602924] Lustre: 2354:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.119@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5036.638366] LustreError: 72849:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffffa0fe471b3000: namespace resource [0x240000400:0x4d42:0x0].0x0 (ffffa0fe45c17c00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5044.256443] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 05:03:50 (1787735030)
[ 5045.889416] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5047.474633] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 05:03:53 (1787735033)
[ 5048.797869] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5050.408625] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 05:03:56 (1787735036)
[ 5051.842319] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5053.350739] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 05:03:59 (1787735039)
[ 5054.913842] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5056.975112] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 05:04:02 (1787735042)
[ 5058.614357] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5060.113232] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 05:04:06 (1787735046)
[ 5061.616238] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5063.049571] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 05:04:09 (1787735049)
[ 5064.567257] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5066.096096] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 05:04:12 (1787735052)
[ 5067.644415] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5069.105586] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 05:04:15 (1787735055)
[ 5070.332241] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5071.796109] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 05:04:17 (1787735057)
[ 5073.082561] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5074.594275] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 05:04:20 (1787735060)
[ 5075.992963] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5077.781972] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 05:04:23 (1787735063)
[ 5080.330793] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5082.094695] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 05:04:27 (1787735067)
[ 5111.704981] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 05:04:57 (1787735097)
[ 5140.366565] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 05:05:26 (1787735126)
[ 5140.863773] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5150.469527] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 05:05:36 (1787735136)
[ 5150.791989] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5160.763653] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 05:05:46 (1787735146)
[ 5161.081742] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5171.294494] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 05:05:56 (1787735156)
[ 5171.877796] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5180.505787] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 05:06:06 (1787735166)
[ 5180.875454] Lustre: *** cfs_fail_loc=510, val=3***
[ 5188.644757] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 05:06:14 (1787735174)
[ 5188.879788] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5199.606326] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 05:06:25 (1787735185)
[ 5199.946869] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5266.525190] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 05:07:32 (1787735252)
[ 5267.033798] LustreError: 83055:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5269.745924] LustreError: lustre-MDT0000-mdc-ffffa0fe471b3000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5269.772421] LustreError: 83069:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffffa0fe471b3000: inode [0x20000a042:0xf:0x0] mdc close failed: rc = -108
[ 5269.791586] LustreError: 83069:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffffa0fe471b3000: namespace resource [0x20000a042:0x11:0x0].0xc (ffffa0fe45c17000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5271.135158] LustreError: 83055:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5273.231488] LustreError: 83086:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5276.109311] LustreError: lustre-MDT0000-mdc-ffffa0fe471b3000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5276.127249] LustreError: 83100:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffffa0fe471b3000: namespace resource [0x20000a042:0x11:0x0].0xc (ffffa0fe47aca200) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5276.146796] LustreError: 83100:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5277.295129] LustreError: 83086:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5284.413480] LustreError: lustre-MDT0000-mdc-ffffa0fe471b3000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5284.951074] LustreError: 83142:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5287.844846] LustreError: lustre-MDT0000-mdc-ffffa0fe471b3000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5287.921494] LustreError: 83162:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5287.927574] LustreError: 83162:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5289.018770] LustreError: 83168:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5289.029517] LustreError: 83168:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5289.043057] LustreError: 83142:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5289.048891] LustreError: 83142:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffffa0fe471b3000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5291.315669] LustreError: 83190:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5291.330263] LustreError: 83190:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5294.976756] LustreError: 83217:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5298.088260] LustreError: lustre-MDT0000-mdc-ffffa0fe471b3000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5298.109172] LustreError: 83232:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffffa0fe471b3000: namespace resource [0x20000a042:0x11:0x0].0xc (ffffa0fe5208d200) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5298.138755] LustreError: 83232:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5299.063206] LustreError: 83217:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5306.023721] LustreError: lustre-MDT0000-mdc-ffffa0fe471b3000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5306.408553] LustreError: 83273:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5309.272343] LustreError: lustre-MDT0000-mdc-ffffa0fe471b3000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5309.295990] LustreError: 83288:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffffa0fe471b3000: namespace resource [0x20000a042:0x11:0x0].0xc (ffffa0fe5208d700) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5309.322844] LustreError: 83288:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5310.479133] LustreError: 83273:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5313.353052] LustreError: lustre-MDT0000-mdc-ffffa0fe471b3000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5313.426627] LustreError: 83321:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5313.438756] LustreError: 83321:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5314.695899] LustreError: 83301:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffffa0fe471b3000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5322.264159] LustreError: 83373:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5322.272523] LustreError: 83373:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5325.459579] LustreError: lustre-MDT0000-mdc-ffffa0fe471b3000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5325.516068] LustreError: 83395:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5325.519833] LustreError: 83395:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5326.337531] LustreError: 83373:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5326.348224] LustreError: 83373:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5326.352981] LustreError: 83373:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffffa0fe471b3000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5335.693517] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 05:08:41 (1787735321)
[ 5355.465769] LustreError: 79806:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5355.472102] LustreError: 79806:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5359.536334] LustreError: 79806:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5359.548371] LustreError: 79806:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5366.691763] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 05:09:12 (1787735352)
[ 5415.596309] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 05:10:01 (1787735401)
[ 5477.968844] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 05:11:03 (1787735463)
[ 5509.747764] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 05:11:35 (1787735495)
[ 5510.123243] Lustre: Mounted lustre-client - version 2.17.57_80_g62ff528
[ 5633.447512] Lustre: Unmounted lustre-client
[ 5639.864562] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 05:13:45 (1787735625)
[ 5640.541077] LustreError: 87698:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5642.994049] LustreError: lustre-OST0000-osc-ffffa0fe471b3000: operation ost_statfs to node 192.168.201.119@tcp failed: rc = -107
[ 5643.001059] LustreError: Skipped 9 previous similar messages
[ 5643.007679] Lustre: lustre-OST0000-osc-ffffa0fe471b3000: Connection to lustre-OST0000 (at 192.168.201.119@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5643.018255] Lustre: Skipped 15 previous similar messages
[ 5643.041464] LustreError: lustre-OST0000-osc-ffffa0fe471b3000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5643.058269] LustreError: 87785:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffffa0fe471b3000: namespace resource [0x240000400:0x4d6d:0x0].0x0 (ffffa0fe45c17900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5643.071169] LustreError: 87785:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5643.271122] LustreError: 87698:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5643.274924] Lustre: 2354:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.119@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5643.291245] Lustre: lustre-OST0000-osc-ffffa0fe471b3000: Connection restored to 192.168.201.119@tcp (at 192.168.201.119@tcp)
[ 5643.298029] Lustre: Skipped 15 previous similar messages
[ 5649.214699] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 05:13:55 (1787735635)
[ 5707.743176] Lustre: 88364:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787735639/real 1787735639]  req@ffffa0fe526d2d80 x1874570851552256/t0(0) o101->lustre-MDT0000-mdc-ffffa0fe471b3000@192.168.201.119@tcp:12/10 lens 328/344 e 0 to 1 dl 1787735694 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 5707.764744] Lustre: 88364:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 10 previous similar messages
[ 5713.593987] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 05:14:59 (1787735699)
[ 5714.800548] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 5716.036184] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 05:15:02 (1787735702)
[ 5778.059826] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 5779.412877] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 05:16:05 (1787735765)
[ 5838.455452] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 05:17:04 (1787735824)
[ 5840.289856] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 5842.626995] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 05:17:07 (1787735827)
[ 5843.866558] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 5845.217976] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 05:17:11 (1787735831)
[ 5875.569320] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 05:17:41 (1787735861)
[ 5885.627553] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5909.471313] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 192.168.201.119@tcp) was lost; in progress operations using this service will fail
[ 5909.481038] LustreError: Skipped 2 previous similar messages
[ 5919.791489] Lustre: Evicted from MGS (at 192.168.201.119@tcp) after server handle changed from 0x2f871d8224b1235f to 0x2f871d8224b13e6a
[ 5919.813096] Lustre: Skipped 2 previous similar messages
[ 5923.560406] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5925.435044] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5934.805076] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 05:18:41 (1787735921)
[ 5936.971403] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 5938.553759] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 05:18:44 (1787735924)
[ 5950.436499] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 192.168.201.119@tcp) was lost; in progress operations using this service will fail
[ 5962.026961] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 05:19:08 (1787735948)
[ 5991.416993] Lustre: Evicted from MGS (at 192.168.201.119@tcp) after server handle changed from 0x2f871d8224b13ff2 to 0x2f871d8224b14284
[ 5991.426528] Lustre: Skipped 1 previous similar message
[ 6007.758266] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 05:19:53 (1787735993)
[ 6059.728777] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6062.492206] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6143.968268] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 192.168.201.119@tcp) was lost; in progress operations using this service will fail
[ 6143.989372] LustreError: Skipped 1 previous similar message
[ 6154.241136] Lustre: Evicted from MGS (at 192.168.201.119@tcp) after server handle changed from 0x2f871d8224b14284 to 0x2f871d8224b150d1
[ 6171.524770] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6173.839647] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6210.434070] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6212.161257] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6325.384822] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 05:25:11 (1787736311)
[ 6334.209726] LustreError: lustre-OST0000-osc-ffffa0fe471b3000: operation ost_setattr to node 192.168.201.119@tcp failed: rc = -107
[ 6334.224348] LustreError: Skipped 47 previous similar messages
[ 6334.235708] Lustre: lustre-OST0000-osc-ffffa0fe471b3000: Connection to lustre-OST0000 (at 192.168.201.119@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6334.255682] Lustre: Skipped 9 previous similar messages
[ 6380.911588] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6383.927451] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6614.610986] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 05:30:00 (1787736600)
[ 6737.896543] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 192.168.201.119@tcp) was lost; in progress operations using this service will fail
[ 6737.905358] LustreError: Skipped 1 previous similar message
[ 6737.922319] Lustre: Evicted from MGS (at 192.168.201.119@tcp) after server handle changed from 0x2f871d8224b15833 to 0x2f871d8224c0f8f2
[ 6737.932174] Lustre: Skipped 1 previous similar message
[ 6737.935608] Lustre: MGC192.168.201.119@tcp: Connection restored to 192.168.201.119@tcp (at 192.168.201.119@tcp)
[ 6737.941834] Lustre: Skipped 13 previous similar messages
[ 6773.860065] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 05:32:39 (1787736759)
[ 6775.331211] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 6776.900611] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 05:32:42 (1787736762)
[ 6778.660475] LustreError: lustre-MDT0000-mdc-ffffa0fe471b3000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6784.619649] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 05:32:50 (1787736770)
[ 6951.526256] LustreError: lustre-OST0000-osc-ffffa0fe471b3000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6954.985930] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 05:35:39 (1787736939)
[ 6982.623159] Lustre: 2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787736949/real 1787736949]  req@ffffa0fe64145180 x1874570869251328/t0(0) o4->lustre-OST0000-osc-ffffa0fe471b3000@192.168.201.119@tcp:6/4 lens 4584/448 e 0 to 1 dl 1787736969 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 6982.659769] Lustre: 2355:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 26 previous similar messages
[ 6982.671250] Lustre: lustre-OST0000-osc-ffffa0fe471b3000: Connection to lustre-OST0000 (at 192.168.201.119@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6982.689896] Lustre: Skipped 3 previous similar messages
[ 7001.745886] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 05:36:27 (1787736987)
[ 7003.973805] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7005.848887] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 05:36:31 (1787736991)
[ 7008.073475] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7010.391988] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 05:36:35 (1787736995)
[ 7046.481555] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 05:37:12 (1787737032)
[ 7085.030815] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 192.168.201.119@tcp) was lost; in progress operations using this service will fail
[ 7085.050220] Lustre: Evicted from MGS (at 192.168.201.119@tcp) after server handle changed from 0x2f871d8224c0f8f2 to 0x2f871d8224c121a2
[ 7085.067299] Lustre: Skipped 1 previous similar message
[ 7103.785745] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 05:38:09 (1787737089)
[ 7105.517220] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7107.252143] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 05:38:13 (1787737093)
[ 7108.978517] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7110.784514] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 05:38:16 (1787737096)
[ 7122.818260] Lustre: Unmounted lustre-client
[ 7124.816181] Lustre: Mounted lustre-client - version 2.17.57_80_g62ff528
[ 7127.422531] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7146.464412] Lustre: 2356:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787737117/real 1787737117]  req@ffffa0fe51469c00 x1874570869438080/t0(0) o400->MGC192.168.201.119@tcp@192.168.201.119@tcp:26/25 lens 224/224 e 0 to 1 dl 1787737133 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7156.689404] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2