[ 1884.661355] LustreError: lustre-OST0000-osc-ffff9300a0e90000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1884.680635] Lustre: 2381:0:(llite_lib.c:4187:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.140@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1890.856433] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 03:36:15 (1789112175)
[ 1892.854780] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1894.323709] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 03:36:19 (1789112179)
[ 1895.911696] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1897.579654] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 03:36:22 (1789112182)
[ 1900.761859] LustreError: lustre-MDT0000-mdc-ffff9300a0e90000: operation ldlm_enqueue to node 192.168.206.140@tcp failed: rc = -19
[ 1900.777044] LustreError: Skipped 1 previous similar message
[ 1916.959228] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 192.168.206.140@tcp) was lost; in progress operations using this service will fail
[ 1927.147651] Lustre: Evicted from MGS (at 192.168.206.140@tcp) after server handle changed from 0x6e427bd79342c2c to 0x6e427bd79344458
[ 2019.819473] LustreError: lustre-MDT0000-mdc-ffff9300a0e90000: operation mds_reint to node 192.168.206.140@tcp failed: rc = -19
[ 2019.825942] LustreError: Skipped 1 previous similar message
[ 2038.773298] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 192.168.206.140@tcp) was lost; in progress operations using this service will fail
[ 2038.803743] Lustre: Evicted from MGS (at 192.168.206.140@tcp) after server handle changed from 0x6e427bd79344458 to 0x6e427bd79367164
[ 2049.005479] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 03:38:53 (1789112333)
[ 2049.334892] Lustre: *** cfs_fail_loc=305, val=0***
[ 2049.338147] Lustre: Skipped 2 previous similar messages
[ 2089.951705] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 192.168.206.140@tcp) was lost; in progress operations using this service will fail
[ 2089.986185] Lustre: Evicted from MGS (at 192.168.206.140@tcp) after server handle changed from 0x6e427bd79367164 to 0x6e427bd79369583
[ 2096.062433] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2097.470588] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2104.727132] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 03:39:49 (1789112389)
[ 2115.570281] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 192.168.206.140@tcp) was lost; in progress operations using this service will fail
[ 2115.605250] Lustre: Evicted from MGS (at 192.168.206.140@tcp) after server handle changed from 0x6e427bd79369583 to 0x6e427bd79369a29
[ 2115.690982] LustreError: lustre-MDT0000-mdc-ffff9300a0e90000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2137.445430] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 03:40:22 (1789112422)
[ 2163.056080] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 03:40:48 (1789112448)
[ 2176.051965] LustreError: lustre-MDT0000-mdc-ffff9300a0e90000: operation mds_close to node 192.168.206.140@tcp failed: rc = -19
[ 2193.375455] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 192.168.206.140@tcp) was lost; in progress operations using this service will fail
[ 2197.588890] Lustre: lustre-MDT0000-mdc-ffff9300a0e90000: Connection restored to 192.168.206.140@tcp (at 192.168.206.140@tcp)
[ 2197.619324] Lustre: Skipped 16 previous similar messages
[ 2203.776471] Lustre: Evicted from MGS (at 192.168.206.140@tcp) after server handle changed from 0x6e427bd79369a29 to 0x6e427bd79371d5b
[ 2207.235465] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2209.330459] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2274.452983] Lustre: lustre-MDT0000-mdc-ffff9300a0e90000: Connection to lustre-MDT0000 (at 192.168.206.140@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2274.468379] Lustre: Skipped 14 previous similar messages
[ 2291.167318] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 192.168.206.140@tcp) was lost; in progress operations using this service will fail
[ 2301.366673] Lustre: Evicted from MGS (at 192.168.206.140@tcp) after server handle changed from 0x6e427bd79371d5b to 0x6e427bd7938d40f
[ 2314.681762] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2316.877306] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2381.674754] LustreError: lustre-MDT0000-mdc-ffff9300a0e90000: operation ldlm_enqueue to node 192.168.206.140@tcp failed: rc = -19
[ 2381.686578] LustreError: Skipped 4 previous similar messages
[ 2398.431104] Lustre: 2380:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789112668/real 1789112668]  req@ffff930086188000 x1876018216029440/t0(0) o400->MGC192.168.206.140@tcp@192.168.206.140@tcp:26/25 lens 224/224 e 0 to 1 dl 1789112684 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2398.475415] Lustre: 2380:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 13 previous similar messages
[ 2398.484124] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 192.168.206.140@tcp) was lost; in progress operations using this service will fail
[ 2408.682212] Lustre: Evicted from MGS (at 192.168.206.140@tcp) after server handle changed from 0x6e427bd7938d40f to 0x6e427bd793ac571
[ 2422.163748] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2424.710697] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2452.782268] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 03:45:37 (1789112737)
[ 2473.461501] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 192.168.206.140@tcp) was lost; in progress operations using this service will fail
[ 2473.485681] Lustre: Evicted from MGS (at 192.168.206.140@tcp) after server handle changed from 0x6e427bd793ac571 to 0x6e427bd793b9d83
[ 2482.435370] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2510.823751] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2547.545681] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2583.190763] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2622.952803] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 192.168.206.140@tcp) was lost; in progress operations using this service will fail
[ 2622.972659] LustreError: Skipped 3 previous similar messages
[ 2622.992918] Lustre: Evicted from MGS (at 192.168.206.140@tcp) after server handle changed from 0x6e427bd793cbde8 to 0x6e427bd793d8b96
[ 2623.006728] Lustre: Skipped 3 previous similar messages
[ 2629.772867] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2657.651666] LustreError: lustre-MDT0000-mdc-ffff9300a0e90000: operation ldlm_enqueue to node 192.168.206.140@tcp failed: rc = -19
[ 2657.671546] LustreError: Skipped 8 previous similar messages
[ 2691.878509] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2776.435257] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 03:51:01 (1789113061)
[ 2824.089619] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2826.229658] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3122.620733] Lustre: lustre-OST0000-osc-ffff9300a0e90000: Connection to lustre-OST0000 (at 192.168.206.140@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3122.633670] Lustre: Skipped 9 previous similar messages
[ 3156.735249] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3159.921336] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3453.949171] LustreError: lustre-OST0000-osc-ffff9300a0e90000: operation ldlm_enqueue to node 192.168.206.140@tcp failed: rc = -19
[ 3453.970183] LustreError: Skipped 3 previous similar messages
[ 3474.334369] Lustre: lustre-OST0000-osc-ffff9300a0e90000: Connection restored to 192.168.206.140@tcp (at 192.168.206.140@tcp)
[ 3474.349268] Lustre: Skipped 19 previous similar messages
[ 3487.727660] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3489.970831] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3751.551889] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 04:07:16 (1789114036)
[ 3769.311688] Lustre: 50293:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789114039/real 1789114039]  req@ffff930086299500 x1876018239771392/t0(0) o101->lustre-MDT0000-mdc-ffff9300a0e90000@192.168.206.140@tcp:12/10 lens 576/1152 e 0 to 1 dl 1789114055 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3769.344822] Lustre: 50293:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 3769.362748] Lustre: lustre-MDT0000-mdc-ffff9300a0e90000: Connection to lustre-MDT0000 (at 192.168.206.140@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3769.382747] Lustre: Skipped 1 previous similar message
[ 3769.436526] Lustre: lustre-MDT0000-mdc-ffff9300a0e90000: Connection restored to 192.168.206.140@tcp (at 192.168.206.140@tcp)
[ 3776.249366] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 04:07:41 (1789114061)
[ 3800.412970] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 04:08:05 (1789114085)
[ 3826.027850] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 04:08:30 (1789114110)
[ 3826.418533] Lustre: Mounted lustre-client
[ 3857.400554] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 192.168.206.140@tcp) was lost; in progress operations using this service will fail
[ 3857.414909] LustreError: Skipped 2 previous similar messages
[ 3857.429269] Lustre: Evicted from MGS (at 192.168.206.140@tcp) after server handle changed from 0x6e427bd793f2d0b to 0x6e427bd79567912
[ 3857.436799] Lustre: Skipped 2 previous similar messages
[ 3858.405832] Lustre: 2381:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789114128/real 1789114128]  req@ffff930086299880 x1876018239788800/t0(0) o400->lustre-MDT0000-mdc-ffff9300a0e90000@192.168.206.140@tcp:12/10 lens 224/224 e 0 to 1 dl 1789114144 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3858.458872] Lustre: 2381:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 3868.590341] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3870.370830] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3872.396372] LustreError: 53044:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff930098391800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 3872.417278] LustreError: 53044:0:(lov_obd.c:792:lov_cleanup()) Skipped 1 previous similar message
[ 3872.462518] Lustre: Unmounted lustre-client
[ 3878.902525] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 04:09:23 (1789114163)
[ 3919.862267] Lustre: lustre-OST0000-osc-ffff9300a0e90000: Connection restored to 192.168.206.140@tcp (at 192.168.206.140@tcp)
[ 3919.873102] Lustre: Skipped 5 previous similar messages
[ 4016.095149] Lustre: 2381:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789114286/real 1789114286]  req@ffff93008629b800 x1876018239820928/t0(0) o4->lustre-OST0000-osc-ffff9300a0e90000@192.168.206.140@tcp:6/4 lens 488/448 e 0 to 1 dl 1789114302 ref 2 fl Rpc:XQr/602/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4016.135595] Lustre: 2381:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 75 previous similar messages
[ 4170.433913] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 04:14:15 (1789114455)
[ 4219.416509] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 04:15:04 (1789114504)
[ 4222.798701] LustreError: 55197:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff9300a0e90000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 4222.813097] LustreError: 55197:0:(lov_obd.c:792:lov_cleanup()) Skipped 1 previous similar message
[ 4222.896232] Lustre: Unmounted lustre-client
[ 4245.101704] Lustre: Mounted lustre-client
[ 4251.609941] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 04:15:36 (1789114536)
[ 4251.818415] LustreError: 56179:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4252.863208] LustreError: 56179:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4253.140787] Lustre: *** cfs_fail_loc=305, val=0***
[ 4275.445778] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 04:16:00 (1789114560)
[ 4275.861557] Lustre: Mounted lustre-client
[ 4278.336764] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4288.666292] Lustre: Unmounted lustre-client
[ 4294.714963] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 04:16:19 (1789114579)
[ 4375.390614] LustreError: lustre-MDT0000-mdc-ffff930098392000: operation mds_reint to node 192.168.206.140@tcp failed: rc = -19
[ 4375.404250] Lustre: lustre-MDT0000-mdc-ffff930098392000: Connection to lustre-MDT0000 (at 192.168.206.140@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4375.426136] Lustre: Skipped 23 previous similar messages
[ 4393.951372] Lustre: 2381:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789114664/real 1789114664]  req@ffff9300855ec380 x1876018242504064/t0(0) o400->MGC192.168.206.140@tcp@192.168.206.140@tcp:26/25 lens 224/224 e 0 to 1 dl 1789114680 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4393.984046] Lustre: 2381:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 77 previous similar messages
[ 4393.987528] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 192.168.206.140@tcp) was lost; in progress operations using this service will fail
[ 4394.005940] Lustre: Evicted from MGS (at 192.168.206.140@tcp) after server handle changed from 0x6e427bd79568fee to 0x6e427bd79594709
[ 4394.022273] Lustre: MGC192.168.206.140@tcp: Connection restored to 192.168.206.140@tcp (at 192.168.206.140@tcp)
[ 4394.027632] Lustre: Skipped 17 previous similar messages
[ 4395.243876] Lustre: 55577:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.206.140@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 4552.454753] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 04:20:37 (1789114837)
[ 4559.058228] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4572.147437] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 192.168.206.140@tcp) was lost; in progress operations using this service will fail
[ 4572.194799] Lustre: Evicted from MGS (at 192.168.206.140@tcp) after server handle changed from 0x6e427bd79594709 to 0x6e427bd795e0941
[ 4572.266858] LustreError: lustre-MDT0000-mdc-ffff930098392000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4588.934876] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 04:21:13 (1789114873)
[ 4589.292729] Lustre: Mounted lustre-client
[ 4596.344683] LustreError: lustre-OST0000-osc-ffff930098392000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4608.367091] LustreError: 59911:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff9300983dc000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 4608.377761] LustreError: 59911:0:(lov_obd.c:792:lov_cleanup()) Skipped 1 previous similar message
[ 4608.440675] Lustre: Unmounted lustre-client
[ 4614.535705] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 04:21:39 (1789114899)
[ 4621.940838] LustreError: lustre-MDT0000-mdc-ffff930098392000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4621.967854] LustreError: 60553:0:(file.c:6098:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4629.723846] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 04:21:54 (1789114914)
[ 4629.876133] LustreError: 61252:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4635.103147] LustreError: 61252:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=0
[ 4635.119933] LustreError: 61252:0:(import.c:713:ptlrpc_connect_import_locked()) already connected
[ 4636.630056] LustreError: lustre-MDT0000-mdc-ffff930098392000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4636.638912] LustreError: 61275:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4636.706949] LustreError: 61281:0:(file.c:6098:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4637.771336] LustreError: 61287:0:(file.c:6098:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4637.794886] LustreError: 61287:0:(file.c:6098:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4640.173027] LustreError: 61309:0:(file.c:6098:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4640.203713] LustreError: 61309:0:(file.c:6098:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4644.788995] LustreError: 61354:0:(file.c:6098:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4644.803757] LustreError: 61354:0:(file.c:6098:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4654.972764] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 04:22:19 (1789114939)
[ 4693.631623] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4695.617510] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4709.156788] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 04:23:13 (1789114993)
[ 4745.160855] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4746.872345] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4757.281192] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 04:24:01 (1789115041)
[ 4817.866806] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4820.845717] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4831.745507] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 04:25:16 (1789115116)
[ 4865.005835] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4866.432626] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4871.885923] LustreError: 66922:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff930098392000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 4871.897632] LustreError: 66922:0:(lov_obd.c:792:lov_cleanup()) Skipped 1 previous similar message
[ 4871.963911] Lustre: Unmounted lustre-client
[ 4919.785479] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4922.453478] Lustre: Mounted lustre-client
[ 4928.221727] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 04:26:53 (1789115213)
[ 4930.773833] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 4932.381389] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 04:26:57 (1789115217)
[ 4957.212457] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 04:27:22 (1789115242)
[ 4958.492532] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 4960.029414] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 04:27:24 (1789115244)
[ 4961.454413] Lustre: *** cfs_fail_loc=805, val=0***
[ 4961.513216] Lustre: Mounted lustre-client
[ 4966.230901] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4987.371388] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 192.168.206.140@tcp) was lost; in progress operations using this service will fail
[ 4987.383758] Lustre: lustre-MDT0000-mdc-ffff930098294000: Connection to lustre-MDT0000 (at 192.168.206.140@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4987.404829] Lustre: Skipped 12 previous similar messages
[ 4987.407930] Lustre: Evicted from MGS (at 192.168.206.140@tcp) after server handle changed from 0x6e427bd795e15b9 to 0x6e427bd795e194e
[ 4994.029058] Lustre: 2381:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789115263/real 1789115263]  req@ffff9300842f9f80 x1876018244541952/t0(0) o400->lustre-MDT0000-mdc-ffff930098294000@192.168.206.140@tcp:12/10 lens 224/224 e 0 to 1 dl 1789115279 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4994.064426] Lustre: 2381:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
[ 4994.207959] LustreError: 70253:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff930098294000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 4994.222928] LustreError: 70253:0:(lov_obd.c:792:lov_cleanup()) Skipped 1 previous similar message
[ 4994.265481] Lustre: Unmounted lustre-client
[ 5001.740287] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 04:28:06 (1789115286)
[ 5024.753307] Lustre: MGC192.168.206.140@tcp: Connection restored to 192.168.206.140@tcp (at 192.168.206.140@tcp)
[ 5024.759786] Lustre: Skipped 15 previous similar messages
[ 5028.893304] LustreError: 2377:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff93008553ea00 x1876018244552064/t94489280516(94489280516) o101->lustre-MDT0000-mdc-ffff930088802800@192.168.206.140@tcp:12/10 lens 648/608 e 0 to 0 dl 1789115331 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5035.304533] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5036.655945] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5045.170208] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 04:28:49 (1789115329)
[ 5046.604810] LustreError: lustre-OST0000-osc-ffff930088802800: operation ost_write to node 192.168.206.140@tcp failed: rc = -107
[ 5046.626506] LustreError: Skipped 3 previous similar messages
[ 5046.640929] LustreError: lustre-OST0000-osc-ffff930088802800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5046.659469] Lustre: 2380:0:(llite_lib.c:4187:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.140@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5054.873296] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 04:28:59 (1789115339)
[ 5056.198870] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5057.688785] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 04:29:02 (1789115342)
[ 5059.142904] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5060.840863] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 04:29:05 (1789115345)
[ 5062.050392] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5063.868306] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 04:29:08 (1789115348)
[ 5066.008730] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5067.697805] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 04:29:12 (1789115352)
[ 5069.093718] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5070.760764] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 04:29:15 (1789115355)
[ 5072.188399] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5073.977780] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 04:29:18 (1789115358)
[ 5075.170136] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5076.838294] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 04:29:21 (1789115361)
[ 5078.036576] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5079.701746] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 04:29:24 (1789115364)
[ 5081.104754] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5082.727397] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 04:29:27 (1789115367)
[ 5084.124551] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5085.810477] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 04:29:30 (1789115370)
[ 5087.374045] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5089.388271] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 04:29:34 (1789115374)
[ 5091.802886] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5093.217251] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 04:29:38 (1789115378)
[ 5124.106561] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 04:30:08 (1789115408)
[ 5155.131544] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 04:30:40 (1789115440)
[ 5155.804990] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5171.613936] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 04:30:55 (1789115455)
[ 5171.892244] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5182.643382] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 04:31:07 (1789115467)
[ 5183.013464] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5191.952246] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 04:31:16 (1789115476)
[ 5192.615948] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5200.944752] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 04:31:25 (1789115485)
[ 5201.282774] Lustre: *** cfs_fail_loc=510, val=3***
[ 5210.141410] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 04:31:34 (1789115494)
[ 5210.556605] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5220.243356] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 04:31:45 (1789115505)
[ 5220.645621] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5285.444896] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 04:32:50 (1789115570)
[ 5285.929236] LustreError: 82444:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5288.860369] LustreError: lustre-MDT0000-mdc-ffff930088802800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5288.893741] LustreError: 82458:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff930088802800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9300854dfc00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5289.999697] LustreError: 82444:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5292.112075] LustreError: 82474:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5294.815481] LustreError: lustre-MDT0000-mdc-ffff930088802800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5294.831822] LustreError: 82488:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff930088802800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9300855e6600) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5294.856842] LustreError: 82488:0:(ldlm_resource.c:1180:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5296.183125] LustreError: 82474:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5303.096524] LustreError: lustre-MDT0000-mdc-ffff930088802800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5303.106918] LustreError: 82517:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff930088802800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff9300854df100) refcount nonzero (2) after lock cleanup; forcing cleanup.
[ 5303.132101] LustreError: 82517:0:(ldlm_resource.c:1180:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5303.465736] LustreError: 82530:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5306.265924] LustreError: lustre-MDT0000-mdc-ffff930088802800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5306.312887] LustreError: 82549:0:(file.c:6098:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5306.319254] LustreError: 82549:0:(file.c:6098:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5307.442727] LustreError: 82556:0:(file.c:6098:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5307.459130] LustreError: 82556:0:(file.c:6098:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5307.540853] LustreError: 82530:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5307.562683] LustreError: 82530:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff930088802800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5309.908682] LustreError: 82578:0:(file.c:6098:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5309.920827] LustreError: 82578:0:(file.c:6098:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5313.853654] LustreError: 82605:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5317.188205] LustreError: lustre-MDT0000-mdc-ffff930088802800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5317.226960] LustreError: 82619:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff930088802800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9300854dfe00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5317.935124] LustreError: 82605:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5324.779472] LustreError: lustre-MDT0000-mdc-ffff930088802800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5325.199201] LustreError: 82661:0:(ldlm_flock.c:858:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5328.074824] LustreError: lustre-MDT0000-mdc-ffff930088802800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5328.101402] LustreError: 82676:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff930088802800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9300854df800) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5328.149832] LustreError: 82676:0:(ldlm_resource.c:1180:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5328.164490] LustreError: 82681:0:(file.c:6098:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5328.187121] LustreError: 82681:0:(file.c:6098:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5329.287155] LustreError: 82661:0:(ldlm_flock.c:858:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5332.116393] LustreError: lustre-MDT0000-mdc-ffff930088802800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5333.423735] LustreError: 82689:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff930088802800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5341.169654] LustreError: 82760:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5341.182805] LustreError: 82760:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5344.869720] LustreError: lustre-MDT0000-mdc-ffff930088802800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5344.985187] LustreError: 82783:0:(file.c:6098:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5345.011071] LustreError: 82783:0:(file.c:6098:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 5345.255178] LustreError: 82760:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5345.269614] LustreError: 82760:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5345.282185] LustreError: 82760:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff930088802800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5355.722746] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 04:34:00 (1789115640)
[ 5374.836404] LustreError: 79188:0:(ldlm_lockd.c:2883:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5374.849196] LustreError: 79188:0:(ldlm_lockd.c:2883:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5378.911797] LustreError: 79188:0:(ldlm_lockd.c:2883:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5378.917022] LustreError: 79188:0:(ldlm_lockd.c:2883:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5387.732460] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 04:34:32 (1789115672)
[ 5438.853653] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 04:35:23 (1789115723)
[ 5502.192334] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 04:36:26 (1789115786)
[ 5536.347501] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 04:37:01 (1789115821)
[ 5536.817981] Lustre: Mounted lustre-client
[ 5660.717637] LustreError: 86470:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff9300983bc000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 5660.727233] LustreError: 86470:0:(lov_obd.c:792:lov_cleanup()) Skipped 1 previous similar message
[ 5660.768962] Lustre: Unmounted lustre-client
[ 5668.197757] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 04:39:12 (1789115952)
[ 5668.873739] LustreError: 87088:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5672.086446] LustreError: lustre-OST0000-osc-ffff930088802800: operation ost_statfs to node 192.168.206.140@tcp failed: rc = -107
[ 5672.097752] LustreError: Skipped 9 previous similar messages
[ 5672.100696] Lustre: lustre-OST0000-osc-ffff930088802800: Connection to lustre-OST0000 (at 192.168.206.140@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5672.113333] Lustre: Skipped 17 previous similar messages
[ 5672.135303] LustreError: lustre-OST0000-osc-ffff930088802800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5672.164200] LustreError: 87175:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-OST0000-osc-ffff930088802800: namespace resource [0x240000400:0x45cd:0x0].0x0 (ffff9300854dfe00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5672.183058] LustreError: 87175:0:(ldlm_resource.c:1180:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5672.576278] LustreError: 87088:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5672.584359] Lustre: 2381:0:(llite_lib.c:4187:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.140@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5672.615204] Lustre: lustre-OST0000-osc-ffff930088802800: Connection restored to 192.168.206.140@tcp (at 192.168.206.140@tcp)
[ 5672.638325] Lustre: Skipped 15 previous similar messages
[ 5679.290878] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 04:39:24 (1789115964)
[ 5737.439135] Lustre: 87754:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789115968/real 1789115968]  req@ffff930084146680 x1876018244721664/t0(0) o101->lustre-MDT0000-mdc-ffff930088802800@192.168.206.140@tcp:12/10 lens 328/344 e 0 to 1 dl 1789116023 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 5737.462957] Lustre: 87754:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 13 previous similar messages
[ 5745.146765] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 04:40:29 (1789116029)
[ 5746.828920] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 5749.056178] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 04:40:33 (1789116033)
[ 5813.948635] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 5816.281271] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 04:41:40 (1789116100)
[ 5874.444727] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 04:42:39 (1789116159)
[ 5877.092455] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 5878.532951] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 04:42:43 (1789116163)
[ 5879.855836] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 5881.616511] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 04:42:46 (1789116166)
[ 5912.569450] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 04:43:17 (1789116197)
[ 5922.389454] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5949.418382] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 192.168.206.140@tcp) was lost; in progress operations using this service will fail
[ 5949.436698] LustreError: Skipped 2 previous similar messages
[ 5949.488224] Lustre: Evicted from MGS (at 192.168.206.140@tcp) after server handle changed from 0x6e427bd795e23f8 to 0x6e427bd795e3dd6
[ 5949.507255] Lustre: Skipped 2 previous similar messages
[ 5963.661157] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5966.113259] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5977.763728] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 04:44:22 (1789116262)
[ 5980.206740] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 5982.041404] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 04:44:26 (1789116266)
[ 5995.499400] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 192.168.206.140@tcp) was lost; in progress operations using this service will fail
[ 5995.524738] Lustre: Evicted from MGS (at 192.168.206.140@tcp) after server handle changed from 0x6e427bd795e3dd6 to 0x6e427bd795e4110
[ 6006.282516] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 04:44:51 (1789116291)
[ 6045.419797] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 04:45:30 (1789116330)
[ 6099.791084] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 6105.096280] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6191.071536] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 192.168.206.140@tcp) was lost; in progress operations using this service will fail
[ 6191.086826] LustreError: Skipped 1 previous similar message
[ 6191.113939] Lustre: Evicted from MGS (at 192.168.206.140@tcp) after server handle changed from 0x6e427bd795e466c to 0x6e427bd795e5178
[ 6191.124486] Lustre: Skipped 1 previous similar message
[ 6204.595558] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6206.142493] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6250.166909] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6251.677964] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6379.759261] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 04:51:04 (1789116664)
[ 6391.233680] LustreError: lustre-OST0000-osc-ffff930088802800: operation ost_setattr to node 192.168.206.140@tcp failed: rc = -107
[ 6391.234483] Lustre: lustre-OST0000-osc-ffff930088802800: Connection to lustre-OST0000 (at 192.168.206.140@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6391.245188] LustreError: Skipped 25 previous similar messages
[ 6391.269413] Lustre: Skipped 9 previous similar messages
[ 6437.208756] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 6442.929038] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6694.704760] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 04:56:19 (1789116979)
[ 6830.055447] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 192.168.206.140@tcp) was lost; in progress operations using this service will fail
[ 6830.067103] LustreError: Skipped 1 previous similar message
[ 6830.097674] Lustre: Evicted from MGS (at 192.168.206.140@tcp) after server handle changed from 0x6e427bd795e5656 to 0x6e427bd796df9ca
[ 6830.106941] Lustre: Skipped 1 previous similar message
[ 6830.113296] Lustre: MGC192.168.206.140@tcp: Connection restored to 192.168.206.140@tcp (at 192.168.206.140@tcp)
[ 6830.124685] Lustre: Skipped 13 previous similar messages
[ 6874.309857] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 04:59:18 (1789117158)
[ 6876.364579] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 6878.238808] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 04:59:22 (1789117162)
[ 6880.802369] LustreError: lustre-MDT0000-mdc-ffff930088802800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6890.368678] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 04:59:35 (1789117175)
[ 7056.890993] LustreError: lustre-OST0000-osc-ffff930088802800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7059.600238] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 05:02:23 (1789117343)
[ 7086.495192] Lustre: 2380:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789117352/real 1789117352]  req@ffff93009e25f800 x1876018262422016/t0(0) o4->lustre-OST0000-osc-ffff930088802800@192.168.206.140@tcp:6/4 lens 4584/448 e 0 to 1 dl 1789117372 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7086.523307] Lustre: 2380:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 22 previous similar messages
[ 7086.531836] Lustre: lustre-OST0000-osc-ffff930088802800: Connection to lustre-OST0000 (at 192.168.206.140@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7086.547926] Lustre: Skipped 3 previous similar messages
[ 7102.916895] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 05:03:07 (1789117387)
[ 7104.167718] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7105.517292] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 05:03:10 (1789117390)
[ 7106.876704] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7108.580388] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 05:03:13 (1789117393)
[ 7141.537485] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 05:03:46 (1789117426)
[ 7179.758128] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 192.168.206.140@tcp) was lost; in progress operations using this service will fail
[ 7179.779660] Lustre: Evicted from MGS (at 192.168.206.140@tcp) after server handle changed from 0x6e427bd796df9ca to 0x6e427bd796e21a8
[ 7179.788938] Lustre: Skipped 1 previous similar message
[ 7197.348881] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 05:04:42 (1789117482)
[ 7198.757390] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7200.705551] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 05:04:45 (1789117485)
[ 7202.086622] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7203.941679] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 05:04:48 (1789117488)
[ 7217.191407] LustreError: 105999:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff930088802800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 7217.205368] LustreError: 105999:0:(lov_obd.c:792:lov_cleanup()) Skipped 1 previous similar message
[ 7217.286554] Lustre: Unmounted lustre-client
[ 7218.842861] Lustre: Mounted lustre-client
[ 7221.274102] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7240.671526] Lustre: 2381:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789117510/real 1789117510]  req@ffff930086310700 x1876018262608896/t0(0) o400->MGC192.168.206.140@tcp@192.168.206.140@tcp:26/25 lens 224/224 e 0 to 1 dl 1789117526 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7260.585829] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2