[ 1842.400299] LustreError: lustre-OST0000-osc-ffff98dc08d17000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1842.410928] Lustre: 2413:0:(llite_lib.c:4151:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.103@tcp:/lustre/fid: [0x240000402:0xa:0x0]// may get corrupted (rc -5)
[ 1842.421736] Lustre: 2414:0:(llite_lib.c:4151:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.103@tcp:/lustre/fid: [0x240000402:0xb:0x0]// may get corrupted (rc -108)
[ 1846.618508] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 10:05:55 (1773669955)
[ 1847.805919] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1848.831043] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 10:05:58 (1773669958)
[ 1849.821481] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1850.797846] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 10:06:00 (1773669960)
[ 1869.792330] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 1869.806159] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0x441c89a012507ad5 to 0x441c89a01250966c
[ 1963.753637] LustreError: lustre-MDT0000-mdc-ffff98dc08d17000: operation ldlm_enqueue to node 192.168.201.103@tcp failed: rc = -19
[ 1963.758499] LustreError: Skipped 3 previous similar messages
[ 1982.438960] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 1982.486919] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0x441c89a01250966c to 0x441c89a012548a8d
[ 1985.644631] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 10:08:15 (1773670095)
[ 1985.811675] Lustre: *** cfs_fail_loc=305, val=0***
[ 1985.812910] Lustre: Skipped 4 previous similar messages
[ 2023.396255] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 2023.403559] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0x441c89a012548a8d to 0x441c89a012548ec3
[ 2027.021848] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2027.797808] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2032.426293] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 10:09:01 (1773670141)
[ 2043.875620] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 2043.885285] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0x441c89a012548ec3 to 0x441c89a0125492d6
[ 2044.253668] LustreError: lustre-MDT0000-mdc-ffff98dc08d17000: operation ldlm_enqueue to node 192.168.201.103@tcp failed: rc = -107
[ 2044.259873] LustreError: Skipped 1 previous similar message
[ 2048.998653] LustreError: lustre-MDT0000-mdc-ffff98dc08d17000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2063.769804] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 10:09:33 (1773670173)
[ 2074.601670] LustreError: lustre-OST0000-osc-ffff98dc08d17000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2087.146717] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 10:09:56 (1773670196)
[ 2115.558215] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 2115.570865] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0x441c89a0125492d6 to 0x441c89a0125510dd
[ 2120.542596] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2121.593832] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2184.743298] LustreError: lustre-MDT0000-mdc-ffff98dc08d17000: operation mds_reint to node 192.168.201.103@tcp failed: rc = -19
[ 2184.748716] LustreError: Skipped 1 previous similar message
[ 2202.596587] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 2202.606050] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0x441c89a0125510dd to 0x441c89a01257e4d8
[ 2206.263916] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2207.215922] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2284.533800] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 2284.632645] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0x441c89a01257e4d8 to 0x441c89a0125abca7
[ 2291.369741] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2292.357554] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2317.320121] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 10:13:46 (1773670426)
[ 2337.416260] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2356.196733] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 2356.203967] LustreError: Skipped 1 previous similar message
[ 2356.210766] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0x441c89a0125c1536 to 0x441c89a0125c1b33
[ 2356.214898] Lustre: Skipped 1 previous similar message
[ 2357.955471] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2382.178548] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2412.163592] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2412.522789] Lustre: MGC192.168.201.103@tcp: Connection restored to 192.168.201.103@tcp (at 192.168.201.103@tcp)
[ 2412.527127] Lustre: Skipped 24 previous similar messages
[ 2433.419118] Lustre: lustre-MDT0000-mdc-ffff98dc08d17000: Connection to lustre-MDT0000 (at 192.168.201.103@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2433.426214] Lustre: Skipped 15 previous similar messages
[ 2451.942212] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2478.309048] LustreError: lustre-MDT0000-mdc-ffff98dc08d17000: operation mds_reint to node 192.168.201.103@tcp failed: rc = -19
[ 2478.313600] LustreError: Skipped 8 previous similar messages
[ 2494.435529] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 2494.441566] LustreError: Skipped 3 previous similar messages
[ 2494.474571] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0x441c89a0125daf9d to 0x441c89a0125eb727
[ 2494.481257] Lustre: Skipped 3 previous similar messages
[ 2496.340672] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2568.709317] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 10:17:58 (1773670678)
[ 2602.468932] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2603.462287] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 2936.082490] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2937.063552] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3243.130452] LustreError: lustre-OST0000-osc-ffff98dc08d17000: operation ldlm_enqueue to node 192.168.201.103@tcp failed: rc = -107
[ 3243.134672] LustreError: Skipped 4 previous similar messages
[ 3243.136548] Lustre: lustre-OST0000-osc-ffff98dc08d17000: Connection to lustre-OST0000 (at 192.168.201.103@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3243.143698] Lustre: Skipped 4 previous similar messages
[ 3259.409612] Lustre: lustre-OST0000-osc-ffff98dc08d17000: Connection restored to 192.168.201.103@tcp (at 192.168.201.103@tcp)
[ 3259.414338] Lustre: Skipped 9 previous similar messages
[ 3264.698017] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3265.734588] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3535.422643] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 10:34:04 (1773671644)
[ 3552.224197] Lustre: 51997:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773671646/real 1773671646]  req@ffff98dc10badf80 x1859825948723584/t0(0) o101->lustre-MDT0000-mdc-ffff98dc08d17000@192.168.201.103@tcp:12/10 lens 576/1152 e 0 to 1 dl 1773671662 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3552.239388] Lustre: 51997:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 3555.459155] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 10:34:25 (1773671665)
[ 3575.836622] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 10:34:45 (1773671685)
[ 3596.105184] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 10:35:05 (1773671705)
[ 3596.289339] Lustre: Mounted lustre-client
[ 3621.859767] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 3621.865213] LustreError: Skipped 1 previous similar message
[ 3621.870744] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0x441c89a0126000ba to 0x441c89a01286aacf
[ 3621.874312] Lustre: Skipped 1 previous similar message
[ 3627.770637] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3628.488199] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3629.911133] LustreError: 54801:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff98dc088ce000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 3629.916216] LustreError: 54801:0:(lov_obd.c:783:lov_cleanup()) Skipped 1 previous similar message
[ 3629.958058] Lustre: Unmounted lustre-client
[ 3632.555243] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 10:35:42 (1773671742)
[ 3652.576150] Lustre: 2413:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773671746/real 1773671746]  req@ffff98dc07cdf480 x1859825948764800/t0(0) o4->lustre-OST0000-osc-ffff98dc08d17000@192.168.201.103@tcp:6/4 lens 488/448 e 0 to 1 dl 1773671762 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 3652.588925] Lustre: 2413:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 9 previous similar messages
[ 3721.067589] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 10:37:10 (1773671830)
[ 3764.313532] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 10:37:54 (1773671874)
[ 3765.766706] LustreError: 56746:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff98dc08d17000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 3765.770438] LustreError: 56746:0:(lov_obd.c:783:lov_cleanup()) Skipped 1 previous similar message
[ 3765.806165] Lustre: Unmounted lustre-client
[ 3781.498986] Lustre: Mounted lustre-client
[ 3784.154703] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 10:38:13 (1773671893)
[ 3784.257827] LustreError: 57853:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 3785.304110] LustreError: 57853:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 3785.342727] Lustre: *** cfs_fail_loc=305, val=0***
[ 3803.829300] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 10:38:33 (1773671913)
[ 3803.997617] Lustre: Mounted lustre-client
[ 3804.509768] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 3814.783203] Lustre: Unmounted lustre-client
[ 3817.504383] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 10:38:47 (1773671927)
[ 3860.455385] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 3860.481394] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0x441c89a01286b7f6 to 0x441c89a01289a272
[ 3860.493038] Lustre: MGC192.168.201.103@tcp: Connection restored to 192.168.201.103@tcp (at 192.168.201.103@tcp)
[ 3860.505276] Lustre: Skipped 13 previous similar messages
[ 4035.163996] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 10:42:22 (1773672142)
[ 4051.112106] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4060.147197] Lustre: lustre-MDT0000-mdc-ffff98dc08620800: Connection to lustre-MDT0000 (at 192.168.201.103@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4060.179535] Lustre: Skipped 13 previous similar messages
[ 4076.512156] Lustre: 2412:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773672170/real 1773672170]  req@ffff98dc12fb9180 x1859825953034368/t0(0) o400->MGC192.168.201.103@tcp@192.168.201.103@tcp:26/25 lens 224/224 e 0 to 1 dl 1773672186 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4076.578135] Lustre: 2412:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 34 previous similar messages
[ 4076.614691] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 4076.649548] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0x441c89a01289a272 to 0x441c89a0128d1949
[ 4078.779366] LustreError: lustre-MDT0000-mdc-ffff98dc08620800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4110.690933] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 10:43:39 (1773672219)
[ 4112.086507] Lustre: Mounted lustre-client
[ 4119.642691] LustreError: lustre-OST0000-osc-ffff98dc08620800: operation ldlm_enqueue to node 192.168.201.103@tcp failed: rc = -107
[ 4119.659168] LustreError: Skipped 1 previous similar message
[ 4119.680237] LustreError: lustre-OST0000-osc-ffff98dc08620800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4132.949570] LustreError: 61929:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff98dc0850f800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 4132.965396] LustreError: 61929:0:(lov_obd.c:783:lov_cleanup()) Skipped 1 previous similar message
[ 4133.154249] Lustre: Unmounted lustre-client
[ 4142.630457] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 10:44:11 (1773672251)
[ 4150.658240] LustreError: lustre-MDT0000-mdc-ffff98dc08620800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4150.682588] LustreError: 62582:0:(file.c:6080:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4162.254610] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 10:44:30 (1773672270)
[ 4162.524357] LustreError: 63291:0:(recover.c:330:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4167.648335] LustreError: 63291:0:(recover.c:330:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=0
[ 4167.655629] LustreError: 63291:0:(import.c:710:ptlrpc_connect_import_locked()) already connected
[ 4169.306645] LustreError: lustre-MDT0000-mdc-ffff98dc08620800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4169.332603] LustreError: 63313:0:(import.c:291:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4169.396571] LustreError: 63318:0:(file.c:6080:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4170.531655] LustreError: 63324:0:(file.c:6080:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4170.556575] LustreError: 63324:0:(file.c:6080:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4172.992503] LustreError: 63346:0:(file.c:6080:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4173.004124] LustreError: 63346:0:(file.c:6080:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4177.052678] LustreError: 63388:0:(file.c:6080:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4177.062328] LustreError: 63388:0:(file.c:6080:ll_inode_revalidate_fini()) Skipped 24 previous similar messages
[ 4189.924883] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 10:44:57 (1773672297)
[ 4236.406365] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4238.558361] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4252.138229] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 10:46:00 (1773672360)
[ 4301.811341] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4305.145437] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4320.380095] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 10:47:08 (1773672428)
[ 4399.703184] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4402.348606] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4416.354577] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 10:48:44 (1773672524)
[ 4470.108318] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4473.049444] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4482.546646] LustreError: 69025:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff98dc08620800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 4482.564451] LustreError: 69025:0:(lov_obd.c:783:lov_cleanup()) Skipped 1 previous similar message
[ 4482.719113] Lustre: Unmounted lustre-client
[ 4568.442747] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4573.109151] Lustre: Mounted lustre-client
[ 4585.566748] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 10:51:32 (1773672692)
[ 4590.255741] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 4593.952382] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 10:51:41 (1773672701)
[ 4650.711420] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 10:52:38 (1773672758)
[ 4653.246954] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 4657.371294] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 10:52:44 (1773672764)
[ 4659.592455] Lustre: *** cfs_fail_loc=805, val=0***
[ 4659.762283] Lustre: Mounted lustre-client
[ 4674.747497] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4681.209319] Lustre: lustre-MDT0000-mdc-ffff98dc063d1000: Connection to lustre-MDT0000 (at 192.168.201.103@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4681.232628] Lustre: Skipped 7 previous similar messages
[ 4697.568543] Lustre: 2414:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773672791/real 1773672791]  req@ffff98dc04288a80 x1859825953145088/t0(0) o400->MGC192.168.201.103@tcp@192.168.201.103@tcp:26/25 lens 224/224 e 0 to 1 dl 1773672807 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4697.627284] Lustre: 2414:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 4697.662438] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 4722.666311] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0x441c89a0128d2726 to 0x441c89a0128d2aa6
[ 4722.708314] Lustre: MGC192.168.201.103@tcp: Connection restored to 192.168.201.103@tcp (at 192.168.201.103@tcp)
[ 4722.731423] Lustre: Skipped 10 previous similar messages
[ 4738.441334] LustreError: 72515:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff98dc0877e800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 4738.457086] LustreError: 72515:0:(lov_obd.c:783:lov_cleanup()) Skipped 1 previous similar message
[ 4738.593732] Lustre: Unmounted lustre-client
[ 4755.201493] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 10:54:22 (1773672862)
[ 4791.568880] LustreError: 2410:0:(client.c:3418:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff98dc07a00a80 x1859825953162368/t94489280516(94489280516) o101->lustre-MDT0000-mdc-ffff98dc063d1000@192.168.201.103@tcp:12/10 lens 520/664 e 0 to 0 dl 1773672917 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 4808.868881] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4811.895672] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4828.016585] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 10:55:35 (1773672935)
[ 4832.288602] LustreError: 2410:0:(osc_request.c:1020:osc_init_grant()) lustre-OST0000-osc-ffff98dc063d1000: granted 8437760 but already consumed 12632064
[ 4832.304146] LustreError: lustre-OST0000-osc-ffff98dc063d1000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4832.342291] Lustre: 2414:0:(llite_lib.c:4151:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.103@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 4832.626944] LustreError: 74547:0:(ldlm_resource.c:1170:ldlm_resource_complain()) lustre-OST0000-osc-ffff98dc063d1000: namespace resource [0x280000401:0x2382:0x0].0x0 (ffff98dc043b7c00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4845.381141] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 10:55:52 (1773672952)
[ 4881.457953] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 10:56:29 (1773672989)
[ 4912.815436] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 10:57:00 (1773673020)
[ 4947.853942] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 10:57:34 (1773673054)
[ 5019.006652] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 10:58:47 (1773673127)
[ 5092.282433] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 10:59:59 (1773673199)
[ 5126.873860] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 11:00:33 (1773673233)
[ 5196.335613] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 11:01:44 (1773673304)
[ 5230.027650] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 11:02:17 (1773673337)
[ 5261.962176] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 11:02:49 (1773673369)
[ 5292.663820] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 11:03:20 (1773673400)
[ 5293.139037] LustreError: 81301:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff98dc063d1000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 5293.152128] LustreError: 81301:0:(lov_obd.c:783:lov_cleanup()) Skipped 1 previous similar message
[ 5293.423158] Lustre: Unmounted lustre-client
[ 5357.620562] Lustre: Mounted lustre-client
[ 5380.810344] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 11:04:48 (1773673488)
[ 5404.199193] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 11:05:12 (1773673512)
[ 5408.783144] Lustre: lustre-MDT0000-mdc-ffff98dc07f29800: Connection to lustre-MDT0000 (at 192.168.201.103@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5408.807260] Lustre: Skipped 8 previous similar messages
[ 5430.240191] Lustre: 2414:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773673524/real 1773673524]  req@ffff98dc07ff4380 x1859825953301504/t0(0) o400->MGC192.168.201.103@tcp@192.168.201.103@tcp:26/25 lens 224/224 e 0 to 1 dl 1773673540 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5430.314763] Lustre: 2414:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 5430.345420] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 192.168.201.103@tcp) was lost; in progress operations using this service will fail
[ 5430.371239] LustreError: Skipped 1 previous similar message
[ 5439.997957] Lustre: Evicted from MGS (at 192.168.201.103@tcp) after server handle changed from 0x441c89a0128d544b to 0x441c89a0128d59a0
[ 5440.027367] Lustre: Skipped 1 previous similar message
[ 5440.041553] Lustre: MGC192.168.201.103@tcp: Connection restored to 192.168.201.103@tcp (at 192.168.201.103@tcp)
[ 5440.058679] Lustre: Skipped 8 previous similar messages
[ 5457.670360] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 11:06:05 (1773673565)
[ 5495.865558] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 11:06:43 (1773673603)
[ 5496.704354] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5512.235553] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 11:06:59 (1773673619)
[ 5512.855870] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5526.482894] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 11:07:14 (1773673634)
[ 5527.097593] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5543.444735] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 11:07:30 (1773673650)
[ 5543.876414] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5557.439837] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 11:07:45 (1773673665)
[ 5558.076496] Lustre: *** cfs_fail_loc=510, val=3***
[ 5572.511405] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 11:08:00 (1773673680)
[ 5573.062756] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5644.830390] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 11:09:12 (1773673752)
[ 5645.471526] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5714.303850] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 11:10:22 (1773673822)
[ 5714.772960] LustreError: 89409:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5718.020654] LustreError: lustre-MDT0000-mdc-ffff98dc07f29800: operation ldlm_enqueue to node 192.168.201.103@tcp failed: rc = -107
[ 5718.044342] LustreError: Skipped 2 previous similar messages
[ 5718.066594] LustreError: lustre-MDT0000-mdc-ffff98dc07f29800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5718.124967] LustreError: 89424:0:(ldlm_resource.c:1170:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff98dc07f29800: namespace resource [0x20000afe2:0xc:0x0].0xc (ffff98dc10a91b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5718.189964] LustreError: 89429:0:(file.c:6080:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5718.218849] LustreError: 89429:0:(file.c:6080:ll_inode_revalidate_fini()) Skipped 10 previous similar messages
[ 5718.818521] LustreError: 89409:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5721.025520] LustreError: 89439:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5724.430204] LustreError: lustre-MDT0000-mdc-ffff98dc07f29800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5724.482377] LustreError: 89454:0:(ldlm_resource.c:1170:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff98dc07f29800: namespace resource [0x20000afe2:0xc:0x0].0xc (ffff98dc043b7000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5724.532932] LustreError: 89454:0:(ldlm_resource.c:1170:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5725.076393] LustreError: 89439:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5732.588632] LustreError: lustre-MDT0000-mdc-ffff98dc07f29800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5733.212569] LustreError: 89496:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5736.650794] LustreError: lustre-MDT0000-mdc-ffff98dc07f29800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5736.808573] LustreError: 89515:0:(file.c:6080:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5737.225992] LustreError: 89496:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5737.241750] LustreError: 89496:0:(file.c:249:ll_close_inode_openhandle()) lustre-clilmv-ffff98dc07f29800: inode [0x20000afe2:0xc:0x0] mdc close failed: rc = -108
[ 5739.349490] LustreError: 89533:0:(file.c:6080:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5739.367823] LustreError: 89533:0:(file.c:6080:ll_inode_revalidate_fini()) Skipped 12 previous similar messages
[ 5744.645791] LustreError: 89571:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5748.712312] LustreError: 89571:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5749.040259] LustreError: lustre-MDT0000-mdc-ffff98dc07f29800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5756.433372] LustreError: lustre-MDT0000-mdc-ffff98dc07f29800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5757.211033] LustreError: 89627:0:(ldlm_flock.c:856:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5760.448880] LustreError: lustre-MDT0000-mdc-ffff98dc07f29800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5760.491494] LustreError: 89642:0:(ldlm_resource.c:1170:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff98dc07f29800: namespace resource [0x20000afe2:0xc:0x0].0xc (ffff98dc02dabe00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5760.554262] LustreError: 89642:0:(ldlm_resource.c:1170:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5760.600519] LustreError: 89647:0:(file.c:6080:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5760.618233] LustreError: 89647:0:(file.c:6080:ll_inode_revalidate_fini()) Skipped 9 previous similar messages
[ 5761.240814] LustreError: 89627:0:(ldlm_flock.c:856:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5764.739761] LustreError: lustre-MDT0000-mdc-ffff98dc07f29800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5765.490531] LustreError: 89655:0:(file.c:249:ll_close_inode_openhandle()) lustre-clilmv-ffff98dc07f29800: inode [0x20000afe2:0xc:0x0] mdc close failed: rc = -108
[ 5768.669665] LustreError: 89709:0:(file.c:6080:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5768.685917] LustreError: 89709:0:(file.c:6080:ll_inode_revalidate_fini()) Skipped 21 previous similar messages
[ 5774.656356] LustreError: 89727:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5774.676138] LustreError: 89727:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5778.672248] LustreError: 89727:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5778.688279] LustreError: 89727:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5778.979642] LustreError: lustre-MDT0000-mdc-ffff98dc07f29800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5788.296860] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 11:11:36 (1773673896)
[ 5824.379878] LustreError: 86103:0:(ldlm_lockd.c:2883:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5828.458709] LustreError: 86103:0:(ldlm_lockd.c:2883:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5837.596125] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 11:12:25 (1773673945)
[ 5880.331474] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 11:13:08 (1773673988)
[ 5926.244544] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 11:13:54 (1773674034)
[ 5968.930514] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 11:14:36 (1773674076)
[ 5969.802062] Lustre: Mounted lustre-client
[ 6096.173247] LustreError: 93472:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff98dc088eb800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 6096.207197] LustreError: 93472:0:(lov_obd.c:783:lov_cleanup()) Skipped 1 previous similar message
[ 6096.405145] Lustre: Unmounted lustre-client
[ 6108.477238] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 11:16:56 (1773674216)
[ 6109.579798] LustreError: 94095:0:(osc_request.c:2983:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 4000ms
[ 6113.672214] LustreError: 94095:0:(osc_request.c:2983:osc_build_rpc()) cfs_fail_timeout id 414 awake
[ 6114.881162] LustreError: lustre-OST0000-osc-ffff98dc07f29800: operation ost_statfs to node 192.168.201.103@tcp failed: rc = -107
[ 6114.898211] LustreError: Skipped 7 previous similar messages
[ 6114.915226] Lustre: lustre-OST0000-osc-ffff98dc07f29800: Connection to lustre-OST0000 (at 192.168.201.103@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6114.955092] Lustre: Skipped 14 previous similar messages
[ 6114.976880] LustreError: lustre-OST0000-osc-ffff98dc07f29800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6115.007413] Lustre: lustre-OST0000-osc-ffff98dc07f29800: Connection restored to 192.168.201.103@tcp (at 192.168.201.103@tcp)
[ 6115.042078] Lustre: Skipped 13 previous similar messages
[ 6117.990731] Lustre: DEBUG MARKER: recovery-small test_131: @@@@@@ FAIL: dd succeeded