[ 1745.566922] LustreError: lustre-OST0000-osc-ffff8c6e081c5800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1745.578168] Lustre: 2362:0:(llite_lib.c:4149:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.107@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1749.476819] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 15:11:45 (1775848305)
[ 1750.724485] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1751.619708] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 15:11:48 (1775848308)
[ 1752.758151] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1753.835510] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 15:11:50 (1775848310)
[ 1756.580277] LustreError: lustre-MDT0000-mdc-ffff8c6e081c5800: operation ldlm_enqueue to node 192.168.202.107@tcp failed: rc = -19
[ 1756.590553] LustreError: Skipped 1 previous similar message
[ 1775.087819] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 192.168.202.107@tcp) was lost; in progress operations using this service will fail
[ 1775.099801] Lustre: Evicted from MGS (at 192.168.202.107@tcp) after server handle changed from 0x9dea60c351ae5263 to 0x9dea60c351ae74b4
[ 1870.987917] LustreError: lustre-MDT0000-mdc-ffff8c6e081c5800: operation mds_reint to node 192.168.202.107@tcp failed: rc = -19
[ 1888.544945] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 192.168.202.107@tcp) was lost; in progress operations using this service will fail
[ 1898.803383] Lustre: Evicted from MGS (at 192.168.202.107@tcp) after server handle changed from 0x9dea60c351ae74b4 to 0x9dea60c351b03c2b
[ 1900.024770] Lustre: 15829:0:(mgc_request.c:1917:mgc_process_log()) MGC192.168.202.107@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 1913.625437] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 15:14:29 (1775848469)
[ 1914.191095] Lustre: *** cfs_fail_loc=305, val=0***
[ 1914.192287] Lustre: Skipped 4 previous similar messages
[ 1954.272241] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 192.168.202.107@tcp) was lost; in progress operations using this service will fail
[ 1954.298340] Lustre: Evicted from MGS (at 192.168.202.107@tcp) after server handle changed from 0x9dea60c351b03c2b to 0x9dea60c351b0445f
[ 1956.080049] Lustre: 15829:0:(mgc_request.c:1917:mgc_process_log()) MGC192.168.202.107@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 1977.718502] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1980.152164] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1991.317090] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 15:15:46 (1775848546)
[ 2009.594559] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 192.168.202.107@tcp) was lost; in progress operations using this service will fail
[ 2009.655086] Lustre: Evicted from MGS (at 192.168.202.107@tcp) after server handle changed from 0x9dea60c351b0445f to 0x9dea60c351b04e6f
[ 2014.904364] LustreError: lustre-MDT0000-mdc-ffff8c6e081c5800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2031.457552] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 15:16:27 (1775848587)
[ 2064.606409] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 15:17:00 (1775848620)
[ 2078.513704] LustreError: lustre-MDT0000-mdc-ffff8c6e081c5800: operation mds_close to node 192.168.202.107@tcp failed: rc = -19
[ 2078.524905] LustreError: Skipped 1 previous similar message
[ 2097.119287] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 192.168.202.107@tcp) was lost; in progress operations using this service will fail
[ 2097.147192] Lustre: Evicted from MGS (at 192.168.202.107@tcp) after server handle changed from 0x9dea60c351b04e6f to 0x9dea60c351b0a18f
[ 2099.199483] Lustre: 15829:0:(mgc_request.c:1917:mgc_process_log()) MGC192.168.202.107@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2126.697616] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2129.677351] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2196.215467] Lustre: lustre-MDT0000-mdc-ffff8c6e081c5800: Connection to lustre-MDT0000 (at 192.168.202.107@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2196.234649] Lustre: Skipped 13 previous similar messages
[ 2213.903218] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 192.168.202.107@tcp) was lost; in progress operations using this service will fail
[ 2213.918331] Lustre: Evicted from MGS (at 192.168.202.107@tcp) after server handle changed from 0x9dea60c351b0a18f to 0x9dea60c351b272c2
[ 2213.932899] Lustre: MGC192.168.202.107@tcp: Connection restored to 192.168.202.107@tcp (at 192.168.202.107@tcp)
[ 2213.941984] Lustre: Skipped 17 previous similar messages
[ 2230.925051] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2233.597726] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2299.918661] LustreError: lustre-MDT0000-mdc-ffff8c6e081c5800: operation mds_reint to node 192.168.202.107@tcp failed: rc = -19
[ 2299.931241] LustreError: Skipped 2 previous similar messages
[ 2317.280518] Lustre: 2363:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775848858/real 1775848858]  req@ffff8c6e070cd500 x1862110185469568/t0(0) o400->MGC192.168.202.107@tcp@192.168.202.107@tcp:26/25 lens 224/224 e 0 to 1 dl 1775848874 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2317.326897] Lustre: 2363:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 17 previous similar messages
[ 2317.340888] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 192.168.202.107@tcp) was lost; in progress operations using this service will fail
[ 2327.551326] Lustre: Evicted from MGS (at 192.168.202.107@tcp) after server handle changed from 0x9dea60c351b272c2 to 0x9dea60c351b44791
[ 2342.687224] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2345.044415] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2373.953561] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 15:22:09 (1775848929)
[ 2398.879262] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 192.168.202.107@tcp) was lost; in progress operations using this service will fail
[ 2398.911338] Lustre: Evicted from MGS (at 192.168.202.107@tcp) after server handle changed from 0x9dea60c351b44791 to 0x9dea60c351b52266
[ 2406.301945] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2449.524765] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2483.082945] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2524.283985] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2567.157176] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 192.168.202.107@tcp) was lost; in progress operations using this service will fail
[ 2567.173812] LustreError: Skipped 3 previous similar messages
[ 2567.203419] Lustre: Evicted from MGS (at 192.168.202.107@tcp) after server handle changed from 0x9dea60c351b5a1fc to 0x9dea60c351b5f6e3
[ 2567.225044] Lustre: Skipped 3 previous similar messages
[ 2576.846229] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2605.773401] LustreError: lustre-MDT0000-mdc-ffff8c6e081c5800: operation ldlm_enqueue to node 192.168.202.107@tcp failed: rc = -19
[ 2605.782876] LustreError: Skipped 7 previous similar messages
[ 2633.689634] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2731.376985] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 15:28:06 (1775849286)
[ 2781.161435] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2783.388334] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3076.438863] Lustre: lustre-OST0000-osc-ffff8c6e081c5800: Connection to lustre-OST0000 (at 192.168.202.107@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3076.461170] Lustre: Skipped 8 previous similar messages
[ 3096.908747] Lustre: lustre-OST0000-osc-ffff8c6e081c5800: Connection restored to 192.168.202.107@tcp (at 192.168.202.107@tcp)
[ 3096.916551] Lustre: Skipped 16 previous similar messages
[ 3113.462510] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3115.779394] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3407.266981] LustreError: lustre-OST0000-osc-ffff8c6e081c5800: operation ldlm_enqueue to node 192.168.202.107@tcp failed: rc = -19
[ 3407.276030] LustreError: Skipped 4 previous similar messages
[ 3442.404283] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3444.804476] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3702.643966] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 15:44:18 (1775850258)
[ 3720.671171] Lustre: 50224:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775850261/real 1775850261]  req@ffff8c6e082fb100 x1862110208537856/t0(0) o101->lustre-MDT0000-mdc-ffff8c6e081c5800@192.168.202.107@tcp:12/10 lens 576/1152 e 0 to 1 dl 1775850277 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3720.682567] Lustre: 50224:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
[ 3720.686345] Lustre: lustre-MDT0000-mdc-ffff8c6e081c5800: Connection to lustre-MDT0000 (at 192.168.202.107@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3720.692839] Lustre: Skipped 1 previous similar message
[ 3720.726977] Lustre: lustre-MDT0000-mdc-ffff8c6e081c5800: Connection restored to 192.168.202.107@tcp (at 192.168.202.107@tcp)
[ 3720.732746] Lustre: Skipped 1 previous similar message
[ 3728.688738] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 15:44:44 (1775850284)
[ 3755.661072] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 15:45:11 (1775850311)
[ 3782.140823] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 15:45:37 (1775850337)
[ 3782.708453] Lustre: Mounted lustre-client
[ 3814.367164] Lustre: 2363:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775850355/real 1775850355]  req@ffff8c6e07e8b480 x1862110208556672/t0(0) o400->MGC192.168.202.107@tcp@192.168.202.107@tcp:26/25 lens 224/224 e 0 to 1 dl 1775850371 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3814.401643] Lustre: 2363:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 3814.420263] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 192.168.202.107@tcp) was lost; in progress operations using this service will fail
[ 3814.436241] LustreError: Skipped 2 previous similar messages
[ 3824.633360] Lustre: Evicted from MGS (at 192.168.202.107@tcp) after server handle changed from 0x9dea60c351b7da61 to 0x9dea60c351cf4317
[ 3824.641799] Lustre: Skipped 2 previous similar messages
[ 3829.387389] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3831.301516] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3833.588190] LustreError: 52992:0:(lov_obd.c:786:lov_cleanup()) lustre-clilov-ffff8c6e089e9000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 3833.596222] LustreError: 52992:0:(lov_obd.c:786:lov_cleanup()) Skipped 1 previous similar message
[ 3833.648939] Lustre: Unmounted lustre-client
[ 3840.924856] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 15:46:36 (1775850396)
[ 3980.255315] Lustre: 2365:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775850521/real 1775850521]  req@ffff8c6e07008a80 x1862110208590976/t0(0) o4->lustre-OST0000-osc-ffff8c6e081c5800@192.168.202.107@tcp:6/4 lens 488/448 e 0 to 1 dl 1775850537 ref 2 fl Rpc:XQr/602/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 3980.331412] Lustre: 2365:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 76 previous similar messages
[ 4141.394572] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 15:51:37 (1775850697)
[ 4192.335080] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 15:52:27 (1775850747)
[ 4196.136980] LustreError: 55157:0:(lov_obd.c:786:lov_cleanup()) lustre-clilov-ffff8c6e081c5800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 4196.148247] LustreError: 55157:0:(lov_obd.c:786:lov_cleanup()) Skipped 1 previous similar message
[ 4196.229090] Lustre: Unmounted lustre-client
[ 4216.407365] Lustre: Mounted lustre-client
[ 4222.687599] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 15:52:58 (1775850778)
[ 4223.089309] LustreError: 56143:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4224.137579] LustreError: 56143:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4224.356545] Lustre: *** cfs_fail_loc=305, val=0***
[ 4248.683805] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 15:53:24 (1775850804)
[ 4249.166759] Lustre: Mounted lustre-client
[ 4251.950848] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4262.311198] Lustre: Unmounted lustre-client
[ 4270.256299] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 15:53:45 (1775850825)
[ 4357.643807] LustreError: lustre-MDT0000-mdc-ffff8c6e1072d800: operation mds_reint to node 192.168.202.107@tcp failed: rc = -19
[ 4357.651529] Lustre: lustre-MDT0000-mdc-ffff8c6e1072d800: Connection to lustre-MDT0000 (at 192.168.202.107@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4357.663304] Lustre: Skipped 23 previous similar messages
[ 4377.887843] Lustre: 2363:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775850918/real 1775850918]  req@ffff8c6e043e2300 x1862110211268864/t0(0) o400->MGC192.168.202.107@tcp@192.168.202.107@tcp:26/25 lens 224/224 e 0 to 1 dl 1775850934 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4377.926579] Lustre: 2363:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 78 previous similar messages
[ 4377.932465] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 192.168.202.107@tcp) was lost; in progress operations using this service will fail
[ 4377.963057] Lustre: Evicted from MGS (at 192.168.202.107@tcp) after server handle changed from 0x9dea60c351cf638c to 0x9dea60c351d216ef
[ 4377.983122] Lustre: MGC192.168.202.107@tcp: Connection restored to 192.168.202.107@tcp (at 192.168.202.107@tcp)
[ 4377.995647] Lustre: Skipped 24 previous similar messages
[ 4379.633820] Lustre: 55535:0:(mgc_request.c:1917:mgc_process_log()) MGC192.168.202.107@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 4535.994427] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 15:58:11 (1775851091)
[ 4542.592380] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4555.252476] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 192.168.202.107@tcp) was lost; in progress operations using this service will fail
[ 4555.300385] Lustre: Evicted from MGS (at 192.168.202.107@tcp) after server handle changed from 0x9dea60c351d216ef to 0x9dea60c351d6e089
[ 4557.501067] LustreError: lustre-MDT0000-mdc-ffff8c6e1072d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4574.673293] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 15:58:50 (1775851130)
[ 4575.162968] Lustre: Mounted lustre-client
[ 4582.461369] LustreError: lustre-OST0000-osc-ffff8c6e1072d800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4593.899199] LustreError: 59895:0:(lov_obd.c:786:lov_cleanup()) lustre-clilov-ffff8c6e11da0800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 4593.905957] LustreError: 59895:0:(lov_obd.c:786:lov_cleanup()) Skipped 1 previous similar message
[ 4594.009733] Lustre: Unmounted lustre-client
[ 4601.564122] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 15:59:17 (1775851157)
[ 4608.304777] LustreError: lustre-MDT0000-mdc-ffff8c6e1072d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4608.329703] LustreError: 60537:0:(file.c:6097:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4616.958564] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 15:59:32 (1775851172)
[ 4617.315865] LustreError: 61241:0:(recover.c:330:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4622.795773] LustreError: lustre-MDT0000-mdc-ffff8c6e1072d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4622.807480] LustreError: 61258:0:(import.c:291:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4622.825368] LustreError: 61241:0:(recover.c:330:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=1
[ 4622.837785] LustreError: 61241:0:(import.c:716:ptlrpc_connect_import_locked()) already connecting
[ 4622.894891] LustreError: 61263:0:(file.c:6097:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4624.004713] LustreError: 61269:0:(file.c:6097:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4624.015452] LustreError: 61269:0:(file.c:6097:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4626.356046] LustreError: 61291:0:(file.c:6097:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4626.366848] LustreError: 61291:0:(file.c:6097:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4631.011583] LustreError: 61336:0:(file.c:6097:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4631.027337] LustreError: 61336:0:(file.c:6097:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4640.133947] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 15:59:55 (1775851195)
[ 4678.538562] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4680.214856] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4692.920247] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 16:00:48 (1775851248)
[ 4734.069877] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4736.349786] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4751.574266] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 16:01:46 (1775851306)
[ 4815.788965] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4817.825968] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4827.680569] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 16:03:03 (1775851383)
[ 4866.221912] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4867.730526] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4875.149730] LustreError: 66922:0:(lov_obd.c:786:lov_cleanup()) lustre-clilov-ffff8c6e1072d800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 4875.163265] LustreError: 66922:0:(lov_obd.c:786:lov_cleanup()) Skipped 1 previous similar message
[ 4875.219947] Lustre: Unmounted lustre-client
[ 4931.939650] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4935.250595] Lustre: Mounted lustre-client
[ 4943.272605] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 16:04:58 (1775851498)
[ 4946.171837] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 4947.842470] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 16:05:03 (1775851503)
[ 4980.313809] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 16:05:35 (1775851535)
[ 4982.214903] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 4984.337233] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 16:05:39 (1775851539)
[ 4986.081217] Lustre: *** cfs_fail_loc=805, val=0***
[ 4986.115151] Lustre: Mounted lustre-client
[ 4992.357196] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5012.959096] Lustre: 2362:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775851553/real 1775851553]  req@ffff8c6e104e3100 x1862110213319552/t0(0) o400->MGC192.168.202.107@tcp@192.168.202.107@tcp:26/25 lens 224/224 e 0 to 1 dl 1775851569 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5013.018598] Lustre: 2362:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5013.035103] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 192.168.202.107@tcp) was lost; in progress operations using this service will fail
[ 5013.059479] Lustre: lustre-MDT0000-mdc-ffff8c6e089d4000: Connection to lustre-MDT0000 (at 192.168.202.107@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5013.093140] Lustre: Skipped 10 previous similar messages
[ 5023.721978] Lustre: Evicted from MGS (at 192.168.202.107@tcp) after server handle changed from 0x9dea60c351d6ed0f to 0x9dea60c351d6f0b9
[ 5023.738669] Lustre: MGC192.168.202.107@tcp: Connection restored to 192.168.202.107@tcp (at 192.168.202.107@tcp)
[ 5023.743846] LustreError: 2361:0:(ldlm_resource.c:1170:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8c6e089d4000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff8c6e042ded00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5023.758176] Lustre: Skipped 10 previous similar messages
[ 5026.760914] LustreError: 70286:0:(lov_obd.c:786:lov_cleanup()) lustre-clilov-ffff8c6e089d4000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 5026.763482] LustreError: 70286:0:(lov_obd.c:786:lov_cleanup()) Skipped 1 previous similar message
[ 5026.791195] Lustre: Unmounted lustre-client
[ 5036.465274] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 16:06:32 (1775851592)
[ 5079.947056] LustreError: 2361:0:(client.c:3419:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff8c6e049ddf80 x1862110213330944/t90194313220(90194313220) o101->lustre-MDT0000-mdc-ffff8c6e043ca800@192.168.202.107@tcp:12/10 lens 664/608 e 0 to 0 dl 1775851653 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5086.518893] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5088.426749] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5097.409850] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 16:07:32 (1775851652)
[ 5098.999667] LustreError: lustre-OST0000-osc-ffff8c6e043ca800: operation ost_write to node 192.168.202.107@tcp failed: rc = -107
[ 5099.014971] LustreError: Skipped 3 previous similar messages
[ 5099.031343] LustreError: lustre-OST0000-osc-ffff8c6e043ca800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5099.052791] Lustre: 2362:0:(llite_lib.c:4149:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.107@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5099.086745] LustreError: 72286:0:(ldlm_resource.c:1170:ldlm_resource_complain()) lustre-OST0000-osc-ffff8c6e043ca800: namespace resource [0x240000400:0x45c2:0x0].0x0 (ffff8c6e1021fa00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5099.100290] LustreError: 72286:0:(ldlm_resource.c:1170:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5108.158301] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 16:07:43 (1775851663)
[ 5109.747387] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5111.297552] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 16:07:47 (1775851667)
[ 5112.892780] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5114.627330] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 16:07:50 (1775851670)
[ 5116.332036] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5118.091368] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 16:07:53 (1775851673)
[ 5119.914532] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5121.973900] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 16:07:57 (1775851677)
[ 5123.817821] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5125.341432] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 16:08:01 (1775851681)
[ 5127.022698] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5129.371797] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 16:08:04 (1775851684)
[ 5131.291434] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5133.096592] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 16:08:08 (1775851688)
[ 5135.307733] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5137.671423] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 16:08:13 (1775851693)
[ 5139.931608] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5142.913490] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 16:08:17 (1775851697)
[ 5144.867705] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5146.950312] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 16:08:22 (1775851702)
[ 5148.915405] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5150.908604] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 16:08:26 (1775851706)
[ 5153.846514] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5155.304942] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 16:08:31 (1775851711)
[ 5188.437392] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 16:09:03 (1775851743)
[ 5218.902946] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 16:09:34 (1775851774)
[ 5219.277438] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5230.696355] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 16:09:46 (1775851786)
[ 5231.390391] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5243.512448] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 16:09:59 (1775851799)
[ 5244.596802] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5256.204933] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 16:10:11 (1775851811)
[ 5256.816740] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5266.110415] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 16:10:21 (1775851821)
[ 5266.669273] Lustre: *** cfs_fail_loc=510, val=3***
[ 5276.311638] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 16:10:31 (1775851831)
[ 5276.654641] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5287.803378] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 16:10:43 (1775851843)
[ 5288.324592] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5357.079593] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 16:11:52 (1775851912)
[ 5357.475413] LustreError: 82485:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5360.259880] LustreError: lustre-MDT0000-mdc-ffff8c6e043ca800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5360.293556] LustreError: 82499:0:(ldlm_resource.c:1170:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8c6e043ca800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8c6e1021f800) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5361.575141] LustreError: 82485:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5363.746617] LustreError: 82515:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5366.829488] LustreError: lustre-MDT0000-mdc-ffff8c6e043ca800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5366.888623] LustreError: 82529:0:(ldlm_resource.c:1170:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8c6e043ca800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8c6e1021f600) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5366.914261] LustreError: 82529:0:(ldlm_resource.c:1170:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5367.772135] LustreError: 82515:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5374.908497] LustreError: lustre-MDT0000-mdc-ffff8c6e043ca800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5375.349277] LustreError: 82570:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5378.448614] LustreError: lustre-MDT0000-mdc-ffff8c6e043ca800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5378.509516] LustreError: 82590:0:(file.c:6097:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5378.512763] LustreError: 82590:0:(file.c:6097:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5379.423106] LustreError: 82570:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5379.426254] LustreError: 82570:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8c6e043ca800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5379.583984] LustreError: 82596:0:(file.c:6097:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5379.591208] LustreError: 82596:0:(file.c:6097:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5381.930742] LustreError: 82618:0:(file.c:6097:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5381.944916] LustreError: 82618:0:(file.c:6097:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5385.733838] LustreError: 82645:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5388.862087] LustreError: lustre-MDT0000-mdc-ffff8c6e043ca800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5388.900920] LustreError: 82660:0:(ldlm_resource.c:1170:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8c6e043ca800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8c6e042de800) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5388.933398] LustreError: 82660:0:(ldlm_resource.c:1170:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5389.818124] LustreError: 82645:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5396.848330] LustreError: lustre-MDT0000-mdc-ffff8c6e043ca800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5397.452151] LustreError: 82701:0:(ldlm_flock.c:856:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5400.490983] LustreError: lustre-MDT0000-mdc-ffff8c6e043ca800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5400.513900] LustreError: 82715:0:(ldlm_resource.c:1170:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8c6e043ca800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8c6e20dbe200) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5400.553231] LustreError: 82715:0:(ldlm_resource.c:1170:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5400.568224] LustreError: 82720:0:(file.c:6097:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5400.585091] LustreError: 82720:0:(file.c:6097:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5401.543128] LustreError: 82701:0:(ldlm_flock.c:856:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5404.799415] LustreError: lustre-MDT0000-mdc-ffff8c6e043ca800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5405.761546] LustreError: 82729:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8c6e043ca800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5413.869145] LustreError: 82800:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5413.903144] LustreError: 82800:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5417.661847] LustreError: lustre-MDT0000-mdc-ffff8c6e043ca800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5417.765492] LustreError: 82822:0:(file.c:6097:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5417.772414] LustreError: 82822:0:(file.c:6097:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 5417.887491] LustreError: 82800:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5417.898155] LustreError: 82800:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5417.907373] LustreError: 82800:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8c6e043ca800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5429.295795] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 16:13:05 (1775851985)
[ 5449.707789] LustreError: 67757:0:(ldlm_lockd.c:2883:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5449.714385] LustreError: 67757:0:(ldlm_lockd.c:2883:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5453.799141] LustreError: 67757:0:(ldlm_lockd.c:2883:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5453.809952] LustreError: 67757:0:(ldlm_lockd.c:2883:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5462.017471] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 16:13:37 (1775852017)
[ 5516.002088] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 16:14:31 (1775852071)
[ 5582.297104] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 16:15:37 (1775852137)
[ 5617.160765] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 16:16:13 (1775852173)
[ 5617.853908] Lustre: Mounted lustre-client
[ 5741.449968] LustreError: 86515:0:(lov_obd.c:786:lov_cleanup()) lustre-clilov-ffff8c6e081c1000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 5741.467837] LustreError: 86515:0:(lov_obd.c:786:lov_cleanup()) Skipped 1 previous similar message
[ 5741.516362] Lustre: Unmounted lustre-client
[ 5749.164525] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 16:18:24 (1775852304)
[ 5749.902076] LustreError: 87130:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5752.852296] LustreError: lustre-OST0000-osc-ffff8c6e043ca800: operation ldlm_enqueue to node 192.168.202.107@tcp failed: rc = -107
[ 5752.861483] LustreError: Skipped 9 previous similar messages
[ 5752.864857] Lustre: lustre-OST0000-osc-ffff8c6e043ca800: Connection to lustre-OST0000 (at 192.168.202.107@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5752.888928] Lustre: Skipped 17 previous similar messages
[ 5752.909582] LustreError: lustre-OST0000-osc-ffff8c6e043ca800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5752.917444] LustreError: 87212:0:(ldlm_resource.c:1170:ldlm_resource_complain()) lustre-OST0000-osc-ffff8c6e043ca800: namespace resource [0x240000400:0x45ed:0x0].0x0 (ffff8c6e04071d00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5752.934351] LustreError: 87212:0:(ldlm_resource.c:1170:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5753.319179] LustreError: 87130:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5753.332193] Lustre: 2362:0:(llite_lib.c:4149:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.107@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5760.715657] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 16:18:36 (1775852316)
[ 5818.335570] Lustre: 87797:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775852320/real 1775852320]  req@ffff8c6e04a52a00 x1862110213507072/t0(0) o101->lustre-MDT0000-mdc-ffff8c6e043ca800@192.168.202.107@tcp:12/10 lens 328/344 e 0 to 1 dl 1775852375 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 5818.375535] Lustre: 87797:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 21 previous similar messages
[ 5818.396989] Lustre: lustre-MDT0000-mdc-ffff8c6e043ca800: Connection restored to 192.168.202.107@tcp (at 192.168.202.107@tcp)
[ 5818.411736] Lustre: Skipped 18 previous similar messages
[ 5826.824743] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 16:19:42 (1775852382)
[ 5828.640982] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 5830.620374] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 16:19:46 (1775852386)
[ 5896.078691] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 5898.396903] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 16:20:53 (1775852453)
[ 5957.827211] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 16:21:52 (1775852512)
[ 5961.722242] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 5964.193739] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 16:21:59 (1775852519)
[ 5966.065828] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 5967.357453] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 16:22:03 (1775852523)
[ 6008.668761] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 16:22:44 (1775852564)
[ 6021.795343] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6048.735780] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 192.168.202.107@tcp) was lost; in progress operations using this service will fail
[ 6048.749417] LustreError: Skipped 2 previous similar messages
[ 6057.964060] Lustre: Evicted from MGS (at 192.168.202.107@tcp) after server handle changed from 0x9dea60c351d6fbda to 0x9dea60c351d713ce
[ 6057.974845] Lustre: Skipped 2 previous similar messages
[ 6059.246144] Lustre: 67759:0:(mgc_request.c:1917:mgc_process_log()) MGC192.168.202.107@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6073.267293] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6075.194727] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6086.987260] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 16:24:02 (1775852642)
[ 6089.432254] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6091.218807] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 16:24:06 (1775852646)
[ 6106.106480] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 192.168.202.107@tcp) was lost; in progress operations using this service will fail
[ 6106.151690] Lustre: Evicted from MGS (at 192.168.202.107@tcp) after server handle changed from 0x9dea60c351d713ce to 0x9dea60c351d71804
[ 6124.739149] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 16:24:40 (1775852680)
[ 6177.331311] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 16:25:33 (1775852733)
[ 6233.450280] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 6237.236884] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6325.217507] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 192.168.202.107@tcp) was lost; in progress operations using this service will fail
[ 6325.233447] LustreError: Skipped 1 previous similar message
[ 6335.409666] Lustre: Evicted from MGS (at 192.168.202.107@tcp) after server handle changed from 0x9dea60c351d71a96 to 0x9dea60c351d729e6
[ 6335.422311] Lustre: Skipped 1 previous similar message
[ 6350.188641] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6351.987649] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6386.672080] Lustre: lustre-MDT0000-mdc-ffff8c6e043ca800: Connection to lustre-MDT0000 (at 192.168.202.107@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6386.681474] Lustre: Skipped 8 previous similar messages
[ 6402.175988] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6405.157129] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6420.448638] Lustre: 2363:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775852917/real 1775852917]  req@ffff8c6e06c03480 x1862110220056576/t0(0) o400->lustre-MDT0000-mdc-ffff8c6e043ca800@192.168.202.107@tcp:12/10 lens 224/224 e 0 to 1 dl 1775852977 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6420.474615] Lustre: 2363:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 18 previous similar messages
[ 6561.583885] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 16:31:56 (1775853116)
[ 6572.596951] LustreError: lustre-OST0000-osc-ffff8c6e043ca800: operation ost_setattr to node 192.168.202.107@tcp failed: rc = -107
[ 6572.623484] LustreError: Skipped 165 previous similar messages
[ 6633.247694] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 6640.782162] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6909.277513] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 16:37:44 (1775853464)
[ 7087.020008] Lustre: lustre-MDT0000-mdc-ffff8c6e043ca800: Connection to lustre-MDT0000 (at 192.168.202.107@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7087.043144] Lustre: Skipped 1 previous similar message
[ 7102.453617] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 192.168.202.107@tcp) was lost; in progress operations using this service will fail
[ 7102.476867] LustreError: Skipped 1 previous similar message
[ 7102.513281] Lustre: Evicted from MGS (at 192.168.202.107@tcp) after server handle changed from 0x9dea60c351d72d90 to 0x9dea60c351e62aa1
[ 7102.530953] Lustre: Skipped 1 previous similar message
[ 7102.549105] Lustre: MGC192.168.202.107@tcp: Connection restored to 192.168.202.107@tcp (at 192.168.202.107@tcp)
[ 7102.567459] Lustre: Skipped 12 previous similar messages
[ 7144.446893] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 16:41:39 (1775853699)
[ 7146.601383] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7148.980867] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 16:41:44 (1775853704)
[ 7151.068577] LustreError: lustre-MDT0000-mdc-ffff8c6e043ca800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7151.080534] LustreError: 101363:0:(ldlm_resource.c:1170:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8c6e043ca800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff8c6e06e51300) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 7151.097116] LustreError: 101363:0:(ldlm_resource.c:1170:ldlm_resource_complain()) Skipped 1 previous similar message
[ 7158.292430] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 16:41:54 (1775853714)
[ 7326.893628] Lustre: Evicted from lustre-OST0000_UUID (at 192.168.202.107@tcp) after server handle changed from 0x9dea60c351e63ecf to 0x9dea60c351e63fd9
[ 7326.902807] LustreError: lustre-OST0000-osc-ffff8c6e043ca800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7328.974562] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 16:44:44 (1775853884)
[ 7356.767213] Lustre: 2363:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775853893/real 1775853893]  req@ffff8c6e07c99c00 x1862110230465408/t0(0) o4->lustre-OST0000-osc-ffff8c6e043ca800@192.168.202.107@tcp:6/4 lens 4584/448 e 0 to 1 dl 1775853913 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7356.802549] Lustre: 2363:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 7356.831056] Lustre: lustre-OST0000-osc-ffff8c6e043ca800: Connection restored to 192.168.202.107@tcp (at 192.168.202.107@tcp)
[ 7356.854783] Lustre: Skipped 2 previous similar messages
[ 7375.304328] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 16:45:30 (1775853930)
[ 7376.851653] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7379.340633] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 16:45:34 (1775853934)
[ 7381.253235] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7383.279925] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 16:45:38 (1775853938)
[ 7423.765799] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 16:46:18 (1775853978)
[ 7464.427959] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 192.168.202.107@tcp) was lost; in progress operations using this service will fail
[ 7464.454797] Lustre: Evicted from MGS (at 192.168.202.107@tcp) after server handle changed from 0x9dea60c351e62aa1 to 0x9dea60c351e662c4
[ 7485.168258] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 16:47:21 (1775854041)
[ 7486.627339] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7488.202401] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 16:47:24 (1775854044)
[ 7490.435578] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7491.757984] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 16:47:27 (1775854047)
[ 7504.702884] LustreError: 106107:0:(lov_obd.c:786:lov_cleanup()) lustre-clilov-ffff8c6e043ca800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 7504.722671] LustreError: 106107:0:(lov_obd.c:786:lov_cleanup()) Skipped 1 previous similar message
[ 7504.839865] Lustre: Unmounted lustre-client
[ 7507.896931] Lustre: Mounted lustre-client
[ 7511.142659] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7533.548495] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 192.168.202.107@tcp) was lost; in progress operations using this service will fail
[ 7533.579338] Lustre: Evicted from MGS (at 192.168.202.107@tcp) after server handle changed from 0x9dea60c351e66cfe to 0x9dea60c351e66d52
[ 7533.627623] Lustre: MGC192.168.202.107@tcp: Connection restored to 192.168.202.107@tcp (at 192.168.202.107@tcp)
[ 7533.639996] Lustre: Skipped 2 previous similar messages
[ 7534.962398] Lustre: 106142:0:(mgc_request.c:1917:mgc_process_log()) MGC192.168.202.107@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 7552.178551] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2