[ 1881.856525] LustreError: lustre-OST0000-osc-ffff8b690bcda000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1881.875503] Lustre: 2360:0:(llite_lib.c:4203:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.143@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1886.967871] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 09:59:06 (1782482346)
[ 1888.816698] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1890.241846] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 09:59:09 (1782482349)
[ 1891.699188] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1893.001173] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 09:59:12 (1782482352)
[ 1895.802182] LustreError: lustre-MDT0000-mdc-ffff8b690bcda000: operation mds_reint to node 192.168.204.143@tcp failed: rc = -19
[ 1895.814475] LustreError: Skipped 2 previous similar messages
[ 1913.831203] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 192.168.204.143@tcp) was lost; in progress operations using this service will fail
[ 1913.852528] Lustre: Evicted from MGS (at 192.168.204.143@tcp) after server handle changed from 0xcdbfd0446c8a9db6 to 0xcdbfd0446c8aba88
[ 2011.873954] LustreError: lustre-MDT0000-mdc-ffff8b690bcda000: operation ldlm_enqueue to node 192.168.204.143@tcp failed: rc = -19
[ 2031.592153] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 192.168.204.143@tcp) was lost; in progress operations using this service will fail
[ 2031.620398] Lustre: Evicted from MGS (at 192.168.204.143@tcp) after server handle changed from 0xcdbfd0446c8aba88 to 0xcdbfd0446c8dd16c
[ 2035.959881] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 10:01:35 (1782482495)
[ 2036.313923] Lustre: *** cfs_fail_loc=305, val=0***
[ 2036.318538] Lustre: Skipped 2 previous similar messages
[ 2072.544684] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 192.168.204.143@tcp) was lost; in progress operations using this service will fail
[ 2072.568494] Lustre: Evicted from MGS (at 192.168.204.143@tcp) after server handle changed from 0xcdbfd0446c8dd16c to 0xcdbfd0446c8dd9d1
[ 2080.078523] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2080.996805] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2087.042613] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 10:02:26 (1782482546)
[ 2098.156190] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 192.168.204.143@tcp) was lost; in progress operations using this service will fail
[ 2098.177216] Lustre: Evicted from MGS (at 192.168.204.143@tcp) after server handle changed from 0xcdbfd0446c8dd9d1 to 0xcdbfd0446c8ddff8
[ 2098.203348] LustreError: lustre-MDT0000-mdc-ffff8b690bcda000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2110.478878] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 10:02:50 (1782482570)
[ 2119.614093] LustreError: lustre-OST0000-osc-ffff8b690bcda000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2132.915598] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 10:03:12 (1782482592)
[ 2145.338218] LustreError: lustre-MDT0000-mdc-ffff8b690bcda000: operation ldlm_enqueue to node 192.168.204.143@tcp failed: rc = -19
[ 2164.731244] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 192.168.204.143@tcp) was lost; in progress operations using this service will fail
[ 2164.766285] Lustre: Evicted from MGS (at 192.168.204.143@tcp) after server handle changed from 0xcdbfd0446c8ddff8 to 0xcdbfd0446c8e4aaa
[ 2173.045875] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2174.658714] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2238.978841] Lustre: lustre-MDT0000-mdc-ffff8b690bcda000: Connection to lustre-MDT0000 (at 192.168.204.143@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2238.997016] Lustre: Skipped 14 previous similar messages
[ 2256.879923] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 192.168.204.143@tcp) was lost; in progress operations using this service will fail
[ 2256.912106] Lustre: Evicted from MGS (at 192.168.204.143@tcp) after server handle changed from 0xcdbfd0446c8e4aaa to 0xcdbfd0446c90b360
[ 2268.565786] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2270.340233] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2334.415931] LustreError: lustre-MDT0000-mdc-ffff8b690bcda000: operation ldlm_enqueue to node 192.168.204.143@tcp failed: rc = -19
[ 2334.419975] LustreError: Skipped 2 previous similar messages
[ 2354.152450] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 192.168.204.143@tcp) was lost; in progress operations using this service will fail
[ 2354.172398] Lustre: Evicted from MGS (at 192.168.204.143@tcp) after server handle changed from 0xcdbfd0446c90b360 to 0xcdbfd0446c933b8f
[ 2354.183848] Lustre: MGC192.168.204.143@tcp: Connection restored to 192.168.204.143@tcp (at 192.168.204.143@tcp)
[ 2354.191859] Lustre: Skipped 20 previous similar messages
[ 2359.994544] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2361.685381] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2387.589498] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 10:07:27 (1782482847)
[ 2412.210791] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2435.046494] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 192.168.204.143@tcp) was lost; in progress operations using this service will fail
[ 2435.052310] LustreError: Skipped 1 previous similar message
[ 2435.058971] Lustre: Evicted from MGS (at 192.168.204.143@tcp) after server handle changed from 0xcdbfd0446c94552c to 0xcdbfd0446c9486a3
[ 2435.069196] Lustre: Skipped 1 previous similar message
[ 2435.383947] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2463.286695] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2495.926420] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2539.695148] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2586.600581] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 192.168.204.143@tcp) was lost; in progress operations using this service will fail
[ 2586.607879] LustreError: Skipped 3 previous similar messages
[ 2586.617483] Lustre: Evicted from MGS (at 192.168.204.143@tcp) after server handle changed from 0xcdbfd0446c96274d to 0xcdbfd0446c97456d
[ 2586.627356] Lustre: Skipped 3 previous similar messages
[ 2588.443877] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2620.357306] LustreError: lustre-MDT0000-mdc-ffff8b690bcda000: operation mds_reint to node 192.168.204.143@tcp failed: rc = -19
[ 2620.365677] LustreError: Skipped 9 previous similar messages
[ 2666.453617] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 10:12:06 (1782483126)
[ 2706.724824] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2708.074751] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3009.702349] Lustre: lustre-OST0000-osc-ffff8b690bcda000: Connection to lustre-OST0000 (at 192.168.204.143@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3009.710786] Lustre: Skipped 9 previous similar messages
[ 3026.590198] Lustre: lustre-OST0000-osc-ffff8b690bcda000: Connection restored to 192.168.204.143@tcp (at 192.168.204.143@tcp)
[ 3026.594726] Lustre: Skipped 16 previous similar messages
[ 3034.593728] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3035.928491] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3339.956465] LustreError: lustre-OST0000-osc-ffff8b690bcda000: operation ldlm_enqueue to node 192.168.204.143@tcp failed: rc = -107
[ 3339.962158] LustreError: Skipped 2 previous similar messages
[ 3362.229085] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3363.158635] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3632.207861] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 10:28:12 (1782484092)
[ 3648.992168] Lustre: 50460:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782484093/real 1782484093]  req@ffff8b6909129180 x1869066402567296/t0(0) o101->lustre-MDT0000-mdc-ffff8b690bcda000@192.168.204.143@tcp:12/10 lens 576/1152 e 0 to 1 dl 1782484109 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3649.001068] Lustre: 50460:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 12 previous similar messages
[ 3649.003601] Lustre: lustre-MDT0000-mdc-ffff8b690bcda000: Connection to lustre-MDT0000 (at 192.168.204.143@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3649.009424] Lustre: Skipped 1 previous similar message
[ 3649.022200] Lustre: lustre-MDT0000-mdc-ffff8b690bcda000: Connection restored to 192.168.204.143@tcp (at 192.168.204.143@tcp)
[ 3649.025802] Lustre: Skipped 1 previous similar message
[ 3651.612942] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 10:28:31 (1782484111)
[ 3670.277800] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 10:28:50 (1782484130)
[ 3689.294132] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 10:29:09 (1782484149)
[ 3689.443798] Lustre: Mounted lustre-client
[ 3720.168584] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 192.168.204.143@tcp) was lost; in progress operations using this service will fail
[ 3720.189336] LustreError: Skipped 1 previous similar message
[ 3720.202068] Lustre: Evicted from MGS (at 192.168.204.143@tcp) after server handle changed from 0xcdbfd0446c986420 to 0xcdbfd0446cbff47f
[ 3720.210439] Lustre: Skipped 1 previous similar message
[ 3725.344163] Lustre: 2360:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782484170/real 1782484170]  req@ffff8b6904583480 x1869066402585344/t0(0) o400->lustre-MDT0000-mdc-ffff8b690bcda000@192.168.204.143@tcp:12/10 lens 224/224 e 0 to 1 dl 1782484186 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3725.370490] Lustre: 2360:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
[ 3731.911818] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3733.132641] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3735.210412] Lustre: Unmounted lustre-client
[ 3741.197668] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 10:30:00 (1782484200)
[ 3875.808729] Lustre: 2359:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782484320/real 1782484320]  req@ffff8b6904581880 x1869066402616448/t0(0) o4->lustre-OST0000-osc-ffff8b690bcda000@192.168.204.143@tcp:6/4 lens 488/448 e 0 to 1 dl 1782484336 ref 2 fl Rpc:XQr/602/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 3875.826973] Lustre: 2359:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 66 previous similar messages
[ 4031.341771] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 10:34:51 (1782484491)
[ 4080.858144] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 10:35:40 (1782484540)
[ 4084.131747] Lustre: Unmounted lustre-client
[ 4102.881433] Lustre: Mounted lustre-client
[ 4108.752598] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 10:36:08 (1782484568)
[ 4109.262214] LustreError: 56362:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4110.312116] LustreError: 56362:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4110.417591] Lustre: *** cfs_fail_loc=305, val=0***
[ 4132.252706] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 10:36:31 (1782484591)
[ 4132.625696] Lustre: Mounted lustre-client
[ 4134.344862] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4144.650311] Lustre: Unmounted lustre-client
[ 4149.835515] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 10:36:49 (1782484609)
[ 4233.889736] LustreError: lustre-MDT0000-mdc-ffff8b6907825000: operation mds_reint to node 192.168.204.143@tcp failed: rc = -19
[ 4251.620526] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 192.168.204.143@tcp) was lost; in progress operations using this service will fail
[ 4251.639088] Lustre: Evicted from MGS (at 192.168.204.143@tcp) after server handle changed from 0xcdbfd0446cc00f8a to 0xcdbfd0446cc2c387
[ 4251.658072] Lustre: MGC192.168.204.143@tcp: Connection restored to 192.168.204.143@tcp (at 192.168.204.143@tcp)
[ 4251.682141] Lustre: Skipped 24 previous similar messages
[ 4340.134567] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 10:40:00 (1782484800)
[ 4343.129316] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4354.024797] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 192.168.204.143@tcp) was lost; in progress operations using this service will fail
[ 4354.036075] Lustre: lustre-MDT0000-mdc-ffff8b6907825000: Connection to lustre-MDT0000 (at 192.168.204.143@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4354.040044] Lustre: Evicted from MGS (at 192.168.204.143@tcp) after server handle changed from 0xcdbfd0446cc2c387 to 0xcdbfd0446cc7904d
[ 4354.047698] Lustre: Skipped 24 previous similar messages
[ 4354.099611] LustreError: lustre-MDT0000-mdc-ffff8b6907825000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4361.045273] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 10:40:20 (1782484820)
[ 4361.243281] Lustre: Mounted lustre-client
[ 4367.996977] LustreError: lustre-OST0000-osc-ffff8b6907825000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4368.009577] LustreError: 60046:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8b6907825000: namespace resource [0x240000400:0x6683:0x0].0x0 (ffff8b6907137200) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4377.568161] Lustre: 2358:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782484822/real 1782484822]  req@ffff8b690479b100 x1869066407268992/t0(0) o103->lustre-OST0000-osc-ffff8b690bd19800@192.168.204.143@tcp:17/18 lens 328/224 e 0 to 1 dl 1782484838 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295
[ 4377.590492] Lustre: 2358:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 74 previous similar messages
[ 4378.459169] Lustre: Unmounted lustre-client
[ 4381.308852] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 10:40:41 (1782484841)
[ 4386.380801] LustreError: lustre-MDT0000-mdc-ffff8b6907825000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4386.389121] LustreError: 60715:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4386.389872] LustreError: 60799:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8b6907825000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff8b6907137200) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4386.404980] LustreError: 60799:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4389.667080] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 10:40:49 (1782484849)
[ 4389.751218] LustreError: 61417:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4394.396473] LustreError: lustre-MDT0000-mdc-ffff8b6907825000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4394.404161] LustreError: 61434:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4394.408186] LustreError: 61417:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=348
[ 4394.412076] LustreError: 61417:0:(import.c:719:ptlrpc_connect_import_locked()) already connecting
[ 4394.425219] LustreError: 61439:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4395.465900] LustreError: 61446:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4395.472293] LustreError: 61446:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4397.586625] LustreError: 61468:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4397.591812] LustreError: 61468:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4401.852124] LustreError: 61513:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4401.862043] LustreError: 61513:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4408.606259] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 10:41:08 (1782484868)
[ 4438.625244] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4440.577817] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4453.747553] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 10:41:51 (1782484911)
[ 4499.849877] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4501.380936] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4511.844918] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 10:42:51 (1782484971)
[ 4567.984287] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4569.218477] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4575.430307] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 10:43:55 (1782485035)
[ 4603.796716] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4604.997924] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4609.107639] Lustre: Unmounted lustre-client
[ 4646.763437] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4648.863775] Lustre: Mounted lustre-client
[ 4653.586292] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 10:45:13 (1782485113)
[ 4655.097439] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 4656.732436] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 10:45:16 (1782485116)
[ 4675.987261] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 10:45:35 (1782485135)
[ 4677.002217] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 4678.059464] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 10:45:37 (1782485137)
[ 4678.889844] Lustre: *** cfs_fail_loc=805, val=0***
[ 4678.923037] Lustre: Mounted lustre-client
[ 4682.519498] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4699.629710] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 192.168.204.143@tcp) was lost; in progress operations using this service will fail
[ 4699.652373] Lustre: Evicted from MGS (at 192.168.204.143@tcp) after server handle changed from 0xcdbfd0446cc79a4f to 0xcdbfd0446cc79d66
[ 4703.860857] Lustre: Unmounted lustre-client
[ 4708.322284] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 10:46:08 (1782485168)
[ 4735.498744] LustreError: 2356:0:(client.c:3447:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff8b690479aa00 x1869066407345536/t94489280516(94489280516) o101->lustre-MDT0000-mdc-ffff8b690bd1e000@192.168.204.143@tcp:12/10 lens 664/608 e 0 to 0 dl 1782485212 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 4738.851765] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4740.028657] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4745.992842] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 10:46:45 (1782485205)
[ 4746.939795] LustreError: lustre-OST0000-osc-ffff8b690bd1e000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4746.954558] Lustre: 2357:0:(llite_lib.c:4203:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.143@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 4746.978079] LustreError: 72536:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8b690bd1e000: namespace resource [0x240000400:0x66e2:0x0].0x0 (ffff8b6907137900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4752.886820] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 10:46:52 (1782485212)
[ 4753.819134] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 4754.968984] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 10:46:54 (1782485214)
[ 4755.840745] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 4756.850773] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 10:46:56 (1782485216)
[ 4757.741236] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 4758.886607] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 10:46:58 (1782485218)
[ 4759.934077] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 4761.061424] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 10:47:00 (1782485220)
[ 4762.099419] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 4763.367665] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 10:47:03 (1782485223)
[ 4764.442774] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 4765.868563] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 10:47:05 (1782485225)
[ 4766.965264] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 4768.419249] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 10:47:07 (1782485227)
[ 4769.481403] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 4770.796455] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 10:47:10 (1782485230)
[ 4771.758148] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 4772.853548] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 10:47:12 (1782485232)
[ 4773.825495] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 4775.141232] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 10:47:14 (1782485234)
[ 4776.324321] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 4777.533938] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 10:47:17 (1782485237)
[ 4779.114149] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 4780.477469] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 10:47:20 (1782485240)
[ 4800.174332] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 10:47:40 (1782485260)
[ 4826.155238] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 10:48:06 (1782485286)
[ 4826.438693] Lustre: *** cfs_fail_loc=51b, val=3***
[ 4834.931202] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 10:48:14 (1782485294)
[ 4835.226749] Lustre: *** cfs_fail_loc=51b, val=4***
[ 4843.461255] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 10:48:23 (1782485303)
[ 4843.674508] Lustre: *** cfs_fail_loc=50f, val=3***
[ 4849.843790] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 10:48:29 (1782485309)
[ 4850.010437] Lustre: *** cfs_fail_loc=50f, val=4***
[ 4855.650423] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 10:48:35 (1782485315)
[ 4855.870946] Lustre: *** cfs_fail_loc=510, val=3***
[ 4861.509912] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 10:48:41 (1782485321)
[ 4861.731371] Lustre: *** cfs_fail_loc=51b, val=3***
[ 4869.506546] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 10:48:49 (1782485329)
[ 4932.794361] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 10:49:52 (1782485392)
[ 4932.970808] LustreError: 82721:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4935.538169] LustreError: lustre-MDT0000-mdc-ffff8b690bd1e000: operation ldlm_enqueue to node 192.168.204.143@tcp failed: rc = -107
[ 4935.543967] LustreError: Skipped 4 previous similar messages
[ 4935.550693] LustreError: lustre-MDT0000-mdc-ffff8b690bd1e000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4935.558376] LustreError: 82735:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8b690bd1e000: inode [0x20000a042:0x8:0x0] mdc close failed: rc = -108
[ 4935.573168] LustreError: 82735:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8b690bd1e000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8b6907243300) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4935.584833] Lustre: lustre-MDT0000-mdc-ffff8b690bd1e000: Connection restored to 192.168.204.143@tcp (at 192.168.204.143@tcp)
[ 4935.590533] Lustre: Skipped 20 previous similar messages
[ 4937.040231] LustreError: 82721:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4939.100698] LustreError: 82751:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4941.587282] LustreError: lustre-MDT0000-mdc-ffff8b690bd1e000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4941.604889] LustreError: 82765:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8b690bd1e000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8b6906fb6d00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4941.617309] LustreError: 82765:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4943.176135] LustreError: 82751:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4949.702396] LustreError: lustre-MDT0000-mdc-ffff8b690bd1e000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4949.722498] LustreError: 82793:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8b690bd1e000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff8b6905d69400) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4949.739857] LustreError: 82793:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4949.949774] LustreError: 82806:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 4952.530330] LustreError: lustre-MDT0000-mdc-ffff8b690bd1e000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4952.589670] LustreError: 82826:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4952.597676] LustreError: 82826:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 20 previous similar messages
[ 4954.008324] LustreError: 82806:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 4954.020826] LustreError: 82806:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8b690bd1e000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 4954.031450] LustreError: 82806:0:(file.c:251:ll_close_inode_openhandle()) Skipped 4 previous similar messages
[ 4959.216723] LustreError: 82881:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 4961.765250] Lustre: lustre-MDT0000-mdc-ffff8b690bd1e000: Connection to lustre-MDT0000 (at 192.168.204.143@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4961.776621] Lustre: Skipped 21 previous similar messages
[ 4961.787601] LustreError: lustre-MDT0000-mdc-ffff8b690bd1e000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4963.288138] LustreError: 82881:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 4969.792484] LustreError: lustre-MDT0000-mdc-ffff8b690bd1e000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4970.030794] LustreError: 82936:0:(ldlm_flock.c:858:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 4972.504596] LustreError: lustre-MDT0000-mdc-ffff8b690bd1e000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4972.527692] LustreError: 82951:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8b690bd1e000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8b6907243100) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4972.540374] LustreError: 82951:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 2 previous similar messages
[ 4974.096119] LustreError: 82936:0:(ldlm_flock.c:858:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 4976.679084] LustreError: lustre-MDT0000-mdc-ffff8b690bd1e000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4976.717086] LustreError: 82983:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4976.723282] LustreError: 82983:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 4978.210819] LustreError: 82964:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8b690bd1e000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 4985.306832] LustreError: 83035:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4985.310057] LustreError: 83035:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 4988.234594] LustreError: lustre-MDT0000-mdc-ffff8b690bd1e000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4989.368121] LustreError: 83035:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4989.373455] LustreError: 83035:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 4989.380941] LustreError: 83035:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8b690bd1e000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 4996.308225] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 10:50:56 (1782485456)
[ 5012.448183] Lustre: 83743:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782485457/real 1782485457]  req@ffff8b6906ca2a00 x1869066407449728/t0(0) o101->lustre-MDT0000-mdc-ffff8b690bd1e000@192.168.204.143@tcp:12/10 lens 576/1152 e 0 to 1 dl 1782485473 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'stat.0' uid:0 gid:0 projid:0
[ 5012.473800] Lustre: 83743:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 13 previous similar messages
[ 5013.349689] LustreError: 68029:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5013.359086] LustreError: 68029:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5017.424128] LustreError: 68029:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5017.429923] LustreError: 68029:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5021.370839] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 10:51:21 (1782485481)
[ 5067.690384] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 10:52:07 (1782485527)
[ 5128.480839] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 10:53:08 (1782485588)
[ 5156.375410] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 10:53:36 (1782485616)
[ 5156.604836] Lustre: Mounted lustre-client
[ 5280.787214] Lustre: Unmounted lustre-client
[ 5289.013217] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 10:55:48 (1782485748)
[ 5289.850892] LustreError: 87359:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5293.178506] LustreError: lustre-OST0000-osc-ffff8b690bd1e000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5293.193531] LustreError: 87441:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8b690bd1e000: namespace resource [0x240000400:0x670d:0x0].0x0 (ffff8b69081d8b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5293.220139] LustreError: 87441:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5293.568201] LustreError: 87359:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5293.571797] Lustre: 2358:0:(llite_lib.c:4203:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.143@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5300.141751] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 10:55:59 (1782485759)
[ 5362.924033] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 10:57:02 (1782485822)
[ 5364.188198] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 5365.480721] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 10:57:05 (1782485825)
[ 5426.981920] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 5428.138760] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 10:58:07 (1782485887)
[ 5485.773134] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 10:59:05 (1782485945)
[ 5487.594390] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 5488.926827] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 10:59:08 (1782485948)
[ 5490.347231] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 5492.076983] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 10:59:11 (1782485951)
[ 5524.412740] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 10:59:43 (1782485983)
[ 5535.232939] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5561.312241] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 192.168.204.143@tcp) was lost; in progress operations using this service will fail
[ 5561.326267] LustreError: Skipped 2 previous similar messages
[ 5561.347217] Lustre: Evicted from MGS (at 192.168.204.143@tcp) after server handle changed from 0xcdbfd0446cc7a8a3 to 0xcdbfd0446cc7c107
[ 5561.357863] Lustre: Skipped 2 previous similar messages
[ 5561.368934] Lustre: MGC192.168.204.143@tcp: Connection restored to 192.168.204.143@tcp (at 192.168.204.143@tcp)
[ 5561.383754] Lustre: Skipped 14 previous similar messages
[ 5562.609855] Lustre: 68031:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.204.143@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 5571.562770] Lustre: lustre-MDT0000-mdc-ffff8b690bd1e000: Connection to lustre-MDT0000 (at 192.168.204.143@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5571.580232] Lustre: Skipped 11 previous similar messages
[ 5584.259081] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5585.833650] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5595.768661] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 11:00:55 (1782486055)
[ 5597.930489] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 5599.587238] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 11:00:59 (1782486059)
[ 5616.608363] Lustre: 2359:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782486022/real 1782486022]  req@ffff8b6906f90000 x1869066407548160/t0(0) o400->lustre-MDT0000-mdc-ffff8b690bd1e000@192.168.204.143@tcp:12/10 lens 224/224 e 0 to 1 dl 1782486077 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5616.630112] Lustre: 2359:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 9 previous similar messages
[ 5619.230734] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 11:01:19 (1782486079)
[ 5647.041121] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 11:01:46 (1782486106)
[ 5651.618318] LustreError: lustre-OST0000-osc-ffff8b690bd1e000: operation ost_setattr to node 192.168.204.143@tcp failed: rc = -107
[ 5651.634748] LustreError: Skipped 9 previous similar messages
[ 5683.842859] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5685.005795] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5770.492930] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5771.350421] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5798.031577] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5799.046746] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5881.261221] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 11:05:41 (1782486341)
[ 5916.714564] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5918.788528] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6112.642286] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 11:09:32 (1782486572)
[ 6201.318568] Lustre: lustre-MDT0000-mdc-ffff8b690bd1e000: Connection to lustre-MDT0000 (at 192.168.204.143@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6201.344358] Lustre: Skipped 6 previous similar messages
[ 6201.452384] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 192.168.204.143@tcp) was lost; in progress operations using this service will fail
[ 6201.509059] LustreError: Skipped 4 previous similar messages
[ 6201.543361] Lustre: Evicted from MGS (at 192.168.204.143@tcp) after server handle changed from 0xcdbfd0446cc7dac2 to 0xcdbfd0446cd77d33
[ 6201.557137] Lustre: Skipped 4 previous similar messages
[ 6201.563958] Lustre: MGC192.168.204.143@tcp: Connection restored to 192.168.204.143@tcp (at 192.168.204.143@tcp)
[ 6201.570230] Lustre: Skipped 9 previous similar messages
[ 6235.899936] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 11:11:34 (1782486694)
[ 6238.582733] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 6241.628764] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 11:11:40 (1782486700)
[ 6244.179675] LustreError: lustre-MDT0000-mdc-ffff8b690bd1e000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6256.353488] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 11:11:55 (1782486715)
[ 6259.557212] LustreError: lustre-OST0000-osc-ffff8b690bd1e000: operation ost_statfs to node 192.168.204.143@tcp failed: rc = -107
[ 6259.576818] LustreError: Skipped 367 previous similar messages
[ 6424.378112] LustreError: lustre-OST0000-osc-ffff8b690bd1e000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6425.762800] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 11:14:45 (1782486885)
[ 6450.144553] Lustre: 2357:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782486890/real 1782486890]  req@ffff8b6908f11500 x1869066425246848/t0(0) o4->lustre-OST0000-osc-ffff8b690bd1e000@192.168.204.143@tcp:6/4 lens 4584/448 e 0 to 1 dl 1782486910 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 6450.170753] Lustre: 2357:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 12 previous similar messages
[ 6465.849751] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 11:15:25 (1782486925)
[ 6467.058718] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 6468.850410] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 11:15:28 (1782486928)
[ 6470.120740] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 6471.534381] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 11:15:31 (1782486931)
[ 6503.891705] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 11:16:03 (1782486963)
[ 6549.724400] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 11:16:49 (1782487009)
[ 6550.731486] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 6552.335736] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 11:16:51 (1782487011)
[ 6553.543285] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 6554.969986] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 11:16:54 (1782487014)
[ 6565.126846] Lustre: Unmounted lustre-client
[ 6566.612158] Lustre: Mounted lustre-client
[ 6568.656289] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6592.804763] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2