[ 2030.194051] LustreError: lustre-OST0000-osc-ffff8ed091ed1800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2030.218720] Lustre: 2366:0:(llite_lib.c:4339:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.140@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 2037.936074] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 03:38:24 (1785397104)
[ 2040.510224] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 2042.631654] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 03:38:29 (1785397109)
[ 2044.327510] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 2046.809049] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 03:38:33 (1785397113)
[ 2050.705506] LustreError: lustre-MDT0000-mdc-ffff8ed091ed1800: operation ldlm_enqueue to node 192.168.201.140@tcp failed: rc = -19
[ 2050.722980] LustreError: Skipped 1 previous similar message
[ 2069.472474] LustreError: MGC192.168.201.140@tcp: Connection to MGS (at 192.168.201.140@tcp) was lost; in progress operations using this service will fail
[ 2078.731483] Lustre: Evicted from MGS (at 192.168.201.140@tcp) after server handle changed from 0xc146da9e024f2b2c to 0xc146da9e024f466f
[ 2182.524072] LustreError: lustre-MDT0000-mdc-ffff8ed091ed1800: operation mds_reint to node 192.168.201.140@tcp failed: rc = -19
[ 2201.570063] LustreError: MGC192.168.201.140@tcp: Connection to MGS (at 192.168.201.140@tcp) was lost; in progress operations using this service will fail
[ 2201.602360] Lustre: Evicted from MGS (at 192.168.201.140@tcp) after server handle changed from 0xc146da9e024f466f to 0xc146da9e02510c8f
[ 2202.733811] Lustre: 15949:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.140@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2220.162889] Lustre: lustre-MDT0000-mdc-ffff8ed091ed1800: Connection restored to 192.168.201.140@tcp (at 192.168.201.140@tcp)
[ 2220.180507] Lustre: Skipped 13 previous similar messages
[ 2227.528503] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 03:41:34 (1785397294)
[ 2227.941452] Lustre: *** cfs_fail_loc=305, val=0***
[ 2227.942910] Lustre: Skipped 2 previous similar messages
[ 2271.201810] LustreError: MGC192.168.201.140@tcp: Connection to MGS (at 192.168.201.140@tcp) was lost; in progress operations using this service will fail
[ 2281.460412] Lustre: Evicted from MGS (at 192.168.201.140@tcp) after server handle changed from 0xc146da9e02510c8f to 0xc146da9e0251117b
[ 2282.738750] Lustre: 15949:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.140@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2286.580123] Lustre: lustre-MDT0000-mdc-ffff8ed091ed1800: Connection to lustre-MDT0000 (at 192.168.201.140@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2286.611437] Lustre: Skipped 12 previous similar messages
[ 2304.003272] Lustre: DEBUG MARKER: oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2305.927926] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2315.873670] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 03:43:02 (1785397382)
[ 2332.691479] LustreError: MGC192.168.201.140@tcp: Connection to MGS (at 192.168.201.140@tcp) was lost; in progress operations using this service will fail
[ 2332.712976] Lustre: Evicted from MGS (at 192.168.201.140@tcp) after server handle changed from 0xc146da9e0251117b to 0xc146da9e0251186d
[ 2332.770973] LustreError: lustre-MDT0000-mdc-ffff8ed091ed1800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2358.453391] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 03:43:45 (1785397425)
[ 2373.912382] LustreError: lustre-OST0000-osc-ffff8ed091ed1800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2395.790865] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 03:44:22 (1785397462)
[ 2410.071063] LustreError: lustre-MDT0000-mdc-ffff8ed091ed1800: operation ldlm_enqueue to node 192.168.201.140@tcp failed: rc = -19
[ 2429.856219] LustreError: MGC192.168.201.140@tcp: Connection to MGS (at 192.168.201.140@tcp) was lost; in progress operations using this service will fail
[ 2429.893208] Lustre: Evicted from MGS (at 192.168.201.140@tcp) after server handle changed from 0xc146da9e0251186d to 0xc146da9e02516a1a
[ 2459.214796] Lustre: DEBUG MARKER: oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2462.210243] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2547.680220] Lustre: 2369:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785397599/real 1785397599]  req@ffff8ed089c1fb80 x1872122475907072/t0(0) o400->MGC192.168.201.140@tcp@192.168.201.140@tcp:26/25 lens 224/224 e 0 to 1 dl 1785397615 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2547.731908] Lustre: 2369:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 19 previous similar messages
[ 2547.743127] LustreError: MGC192.168.201.140@tcp: Connection to MGS (at 192.168.201.140@tcp) was lost; in progress operations using this service will fail
[ 2557.951453] Lustre: Evicted from MGS (at 192.168.201.140@tcp) after server handle changed from 0xc146da9e02516a1a to 0xc146da9e0252f6b2
[ 2578.667701] Lustre: DEBUG MARKER: oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2582.219063] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2648.881287] LustreError: lustre-MDT0000-mdc-ffff8ed091ed1800: operation ldlm_enqueue to node 192.168.201.140@tcp failed: rc = -19
[ 2648.892606] LustreError: Skipped 2 previous similar messages
[ 2669.536358] LustreError: MGC192.168.201.140@tcp: Connection to MGS (at 192.168.201.140@tcp) was lost; in progress operations using this service will fail
[ 2678.704446] Lustre: Evicted from MGS (at 192.168.201.140@tcp) after server handle changed from 0xc146da9e0252f6b2 to 0xc146da9e02546a6f
[ 2705.001580] Lustre: DEBUG MARKER: oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2707.624220] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2742.286456] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 03:50:08 (1785397808)
[ 2764.768824] LustreError: MGC192.168.201.140@tcp: Connection to MGS (at 192.168.201.140@tcp) was lost; in progress operations using this service will fail
[ 2774.969613] Lustre: Evicted from MGS (at 192.168.201.140@tcp) after server handle changed from 0xc146da9e02546a6f to 0xc146da9e02553473
[ 2778.959546] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2811.768506] Lustre: 15949:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.140@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2826.211557] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2829.459581] Lustre: lustre-MDT0000-mdc-ffff8ed091ed1800: Connection restored to 192.168.201.140@tcp (at 192.168.201.140@tcp)
[ 2829.473173] Lustre: Skipped 13 previous similar messages
[ 2877.935636] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2891.896223] Lustre: lustre-MDT0000-mdc-ffff8ed091ed1800: Connection to lustre-MDT0000 (at 192.168.201.140@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2891.928183] Lustre: Skipped 7 previous similar messages
[ 2911.202374] LustreError: MGC192.168.201.140@tcp: Connection to MGS (at 192.168.201.140@tcp) was lost; in progress operations using this service will fail
[ 2911.220303] LustreError: Skipped 2 previous similar messages
[ 2911.266524] Lustre: Evicted from MGS (at 192.168.201.140@tcp) after server handle changed from 0xc146da9e02554eeb to 0xc146da9e0255afe1
[ 2911.278276] Lustre: Skipped 2 previous similar messages
[ 2922.902908] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2946.883391] LustreError: lustre-MDT0000-mdc-ffff8ed091ed1800: operation mds_reint to node 192.168.201.140@tcp failed: rc = -19
[ 2946.890355] LustreError: Skipped 3 previous similar messages
[ 2986.933462] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 3056.635588] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 3151.478422] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 03:56:58 (1785398218)
[ 3214.074483] Lustre: DEBUG MARKER: oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3216.847564] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3497.140324] LustreError: lustre-OST0000-osc-ffff8ed091ed1800: operation ldlm_enqueue to node 192.168.201.140@tcp failed: rc = -19
[ 3497.149253] LustreError: Skipped 5 previous similar messages
[ 3497.152336] Lustre: lustre-OST0000-osc-ffff8ed091ed1800: Connection to lustre-OST0000 (at 192.168.201.140@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3497.157522] Lustre: Skipped 4 previous similar messages
[ 3519.586301] Lustre: lustre-OST0000-osc-ffff8ed091ed1800: Connection restored to 192.168.201.140@tcp (at 192.168.201.140@tcp)
[ 3519.593981] Lustre: Skipped 10 previous similar messages
[ 3542.575455] Lustre: DEBUG MARKER: oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3545.249036] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3878.772718] Lustre: DEBUG MARKER: oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3881.526333] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4124.759367] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 04:13:11 (1785399191)
[ 4142.560151] Lustre: 50451:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785399194/real 1785399194]  req@ffff8ed0901f6680 x1872122495062272/t0(0) o101->lustre-MDT0000-mdc-ffff8ed091ed1800@192.168.201.140@tcp:12/10 lens 576/1152 e 0 to 1 dl 1785399210 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 4142.592599] Lustre: 50451:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
[ 4142.598264] Lustre: lustre-MDT0000-mdc-ffff8ed091ed1800: Connection to lustre-MDT0000 (at 192.168.201.140@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4142.613667] Lustre: Skipped 1 previous similar message
[ 4142.649280] Lustre: lustre-MDT0000-mdc-ffff8ed091ed1800: Connection restored to 192.168.201.140@tcp (at 192.168.201.140@tcp)
[ 4142.677410] Lustre: Skipped 1 previous similar message
[ 4154.321283] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 04:13:41 (1785399221)
[ 4182.470207] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 04:14:08 (1785399248)
[ 4209.944540] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 04:14:36 (1785399276)
[ 4210.671538] Lustre: Mounted lustre-client
[ 4242.400266] Lustre: 2368:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785399294/real 1785399294]  req@ffff8ed0901f6d80 x1872122495082624/t0(0) o400->lustre-MDT0000-mdc-ffff8ed08a3a5000@192.168.201.140@tcp:12/10 lens 224/224 e 0 to 1 dl 1785399310 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4242.428320] Lustre: 2368:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 4242.445914] LustreError: MGC192.168.201.140@tcp: Connection to MGS (at 192.168.201.140@tcp) was lost; in progress operations using this service will fail
[ 4242.458202] LustreError: Skipped 3 previous similar messages
[ 4252.660420] Lustre: Evicted from MGS (at 192.168.201.140@tcp) after server handle changed from 0xc146da9e02573667 to 0xc146da9e02695cdf
[ 4252.669294] Lustre: Skipped 3 previous similar messages
[ 4254.190642] Lustre: 15949:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.140@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 4276.021768] Lustre: DEBUG MARKER: oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4278.392885] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4280.564117] Lustre: Unmounted lustre-client
[ 4288.183767] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 04:15:54 (1785399354)
[ 4399.073951] Lustre: 2368:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785399451/real 1785399451]  req@ffff8ed0901f6d80 x1872122495118848/t0(0) o4->lustre-OST0000-osc-ffff8ed091ed1800@192.168.201.140@tcp:6/4 lens 488/448 e 0 to 1 dl 1785399467 ref 2 fl Rpc:XQr/602/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4399.108131] Lustre: 2368:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 57 previous similar messages
[ 4590.723497] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 04:20:57 (1785399657)
[ 4641.226714] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 04:21:48 (1785399708)
[ 4645.288962] Lustre: Unmounted lustre-client
[ 4673.077456] Lustre: Mounted lustre-client
[ 4682.001869] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 04:22:28 (1785399748)
[ 4682.316670] LustreError: 56369:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4683.356114] LustreError: 56369:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4683.608817] Lustre: *** cfs_fail_loc=305, val=0***
[ 4708.287894] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 04:22:54 (1785399774)
[ 4708.759451] Lustre: Mounted lustre-client
[ 4710.753711] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4721.119457] Lustre: Unmounted lustre-client
[ 4729.739673] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 04:23:16 (1785399796)
[ 4811.288489] LustreError: lustre-MDT0000-mdc-ffff8ed08a3a2800: operation mds_reint to node 192.168.201.140@tcp failed: rc = -19
[ 4811.296938] LustreError: Skipped 1 previous similar message
[ 4811.302409] Lustre: lustre-MDT0000-mdc-ffff8ed08a3a2800: Connection to lustre-MDT0000 (at 192.168.201.140@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4811.318561] Lustre: Skipped 23 previous similar messages
[ 4828.641066] Lustre: 2367:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785399880/real 1785399880]  req@ffff8ed08587d180 x1872122497788032/t0(0) o400->MGC192.168.201.140@tcp@192.168.201.140@tcp:26/25 lens 224/224 e 0 to 1 dl 1785399896 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4828.685211] Lustre: 2367:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 85 previous similar messages
[ 4828.695464] LustreError: MGC192.168.201.140@tcp: Connection to MGS (at 192.168.201.140@tcp) was lost; in progress operations using this service will fail
[ 4838.951269] Lustre: Evicted from MGS (at 192.168.201.140@tcp) after server handle changed from 0xc146da9e0269778f to 0xc146da9e026c27aa
[ 4838.961346] Lustre: MGC192.168.201.140@tcp: Connection restored to 192.168.201.140@tcp (at 192.168.201.140@tcp)
[ 4838.964019] Lustre: Skipped 24 previous similar messages
[ 4840.485168] Lustre: 55769:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.140@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 5010.147296] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 04:27:56 (1785400076)
[ 5015.298648] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5031.428513] LustreError: MGC192.168.201.140@tcp: Connection to MGS (at 192.168.201.140@tcp) was lost; in progress operations using this service will fail
[ 5031.485071] Lustre: Evicted from MGS (at 192.168.201.140@tcp) after server handle changed from 0xc146da9e026c27aa to 0xc146da9e0270f955
[ 5031.672732] LustreError: lustre-MDT0000-mdc-ffff8ed08a3a2800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5054.412602] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 04:28:40 (1785400120)
[ 5055.245522] Lustre: Mounted lustre-client
[ 5063.209645] LustreError: lustre-OST0000-osc-ffff8ed08a3a2800: operation ldlm_enqueue to node 192.168.201.140@tcp failed: rc = -107
[ 5063.227984] LustreError: lustre-OST0000-osc-ffff8ed08a3a2800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5063.238798] LustreError: 60070:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8ed08a3a2800: namespace resource [0x240000400:0x37e3:0x0].0x0 (ffff8ed085961b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5076.515605] Lustre: Unmounted lustre-client
[ 5085.087237] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 04:29:11 (1785400151)
[ 5091.489209] LustreError: lustre-MDT0000-mdc-ffff8ed08a3a2800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5091.518117] LustreError: 60742:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 5100.932681] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 04:29:27 (1785400167)
[ 5101.277357] LustreError: 61446:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 5105.345324] LustreError: lustre-MDT0000-mdc-ffff8ed08a3a2800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5105.367315] LustreError: 61462:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 5105.381354] LustreError: 61446:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=912
[ 5105.394235] LustreError: 61446:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting
[ 5105.505882] LustreError: 61467:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5106.578381] LustreError: 61474:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5106.586521] LustreError: 61474:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5109.035589] LustreError: 61496:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5109.055436] LustreError: 61496:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5113.071845] LustreError: 61529:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5113.090213] LustreError: 61529:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 20 previous similar messages
[ 5115.881923] LustreError: 61462:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8ed08a3a2800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff8ed085b28e00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5115.907637] LustreError: 61462:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5126.530960] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 04:29:53 (1785400193)
[ 5174.396941] Lustre: DEBUG MARKER: oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5177.282558] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5189.525470] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 04:30:56 (1785400256)
[ 5237.156266] Lustre: DEBUG MARKER: oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5239.055232] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5250.996087] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 04:31:57 (1785400317)
[ 5318.782196] Lustre: DEBUG MARKER: oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5321.069879] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5330.984333] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 04:33:16 (1785400396)
[ 5375.602845] Lustre: DEBUG MARKER: oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5377.634824] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5384.497855] Lustre: Unmounted lustre-client
[ 5445.501642] Lustre: DEBUG MARKER: oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5448.269110] Lustre: Mounted lustre-client
[ 5455.655913] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 04:35:22 (1785400522)
[ 5459.312477] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5461.286493] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 04:35:27 (1785400527)
[ 5498.171889] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 04:36:04 (1785400564)
[ 5500.093927] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5502.026873] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 04:36:08 (1785400568)
[ 5504.026336] Lustre: *** cfs_fail_loc=805, val=0***
[ 5504.058766] Lustre: Mounted lustre-client
[ 5513.465815] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5534.688319] Lustre: 2369:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785400587/real 1785400587]  req@ffff8ed0870e5f80 x1872122499857024/t0(0) o400->lustre-MDT0000-mdc-ffff8ed0905ec000@192.168.201.140@tcp:12/10 lens 224/224 e 0 to 1 dl 1785400603 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5534.769770] Lustre: 2369:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5534.797866] Lustre: lustre-MDT0000-mdc-ffff8ed0905ec000: Connection to lustre-MDT0000 (at 192.168.201.140@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5534.832200] Lustre: Skipped 12 previous similar messages
[ 5534.843515] LustreError: MGC192.168.201.140@tcp: Connection to MGS (at 192.168.201.140@tcp) was lost; in progress operations using this service will fail
[ 5545.466661] Lustre: Evicted from MGS (at 192.168.201.140@tcp) after server handle changed from 0xc146da9e02710453 to 0xc146da9e0271066e
[ 5545.483352] Lustre: MGC192.168.201.140@tcp: Connection restored to 192.168.201.140@tcp (at 192.168.201.140@tcp)
[ 5545.493634] Lustre: Skipped 12 previous similar messages
[ 5559.810856] LustreError: 2365:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8ed090ffc800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff8ed085bb5e00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5564.136209] Lustre: Unmounted lustre-client
[ 5573.062326] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 04:37:19 (1785400639)
[ 5616.206673] LustreError: 2365:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff8ed08732b480 x1872122499868032/t90194313220(90194313220) o101->lustre-MDT0000-mdc-ffff8ed0905ec000@192.168.201.140@tcp:12/10 lens 664/608 e 0 to 0 dl 1785400700 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5625.543190] Lustre: DEBUG MARKER: oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5627.376525] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5637.201643] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 04:38:23 (1785400703)
[ 5639.108756] LustreError: lustre-OST0000-osc-ffff8ed0905ec000: operation ost_write to node 192.168.201.140@tcp failed: rc = -107
[ 5639.118620] LustreError: Skipped 2 previous similar messages
[ 5639.135075] LustreError: lustre-OST0000-osc-ffff8ed0905ec000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5639.151473] Lustre: 2369:0:(llite_lib.c:4339:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.140@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5639.216379] LustreError: 72591:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8ed0905ec000: namespace resource [0x240000400:0x3842:0x0].0x0 (ffff8ed085b28600) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5639.237868] LustreError: 72591:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5650.736304] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 04:38:36 (1785400716)
[ 5652.886420] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5655.377899] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 04:38:41 (1785400721)
[ 5657.779722] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5659.998771] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 04:38:46 (1785400726)
[ 5662.968689] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5665.547574] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 04:38:51 (1785400731)
[ 5667.274910] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5669.851640] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 04:38:56 (1785400736)
[ 5671.511823] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5673.620599] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 04:39:00 (1785400740)
[ 5675.702100] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5677.983658] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 04:39:04 (1785400744)
[ 5680.131809] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5682.164089] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 04:39:08 (1785400748)
[ 5684.447559] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5686.722508] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 04:39:13 (1785400753)
[ 5688.822014] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5691.326625] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 04:39:17 (1785400757)
[ 5693.119859] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5694.427862] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 04:39:21 (1785400761)
[ 5696.637470] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5698.695157] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 04:39:25 (1785400765)
[ 5700.976858] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5702.950774] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 04:39:29 (1785400769)
[ 5733.882901] Lustre: 68069:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.140@tcp: IR log lustre-cliir failed, not fatal: rc = -5
[ 5746.627838] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 04:40:13 (1785400813)
[ 5782.285644] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 04:40:48 (1785400848)
[ 5782.731929] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5794.154773] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 04:41:00 (1785400860)
[ 5794.521748] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5805.105262] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 04:41:11 (1785400871)
[ 5805.500347] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5814.919772] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 04:41:21 (1785400881)
[ 5815.170581] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5824.253155] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 04:41:31 (1785400891)
[ 5824.776557] Lustre: *** cfs_fail_loc=510, val=3***
[ 5833.029247] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 04:41:40 (1785400900)
[ 5833.645379] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5846.295792] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 04:41:53 (1785400913)
[ 5846.615108] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5911.563803] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 04:42:58 (1785400978)
[ 5912.077416] LustreError: 82787:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5915.143583] LustreError: lustre-MDT0000-mdc-ffff8ed0905ec000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5915.172822] LustreError: 82802:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8ed0905ec000: inode [0x20000a042:0xf:0x0] mdc close failed: rc = -108
[ 5915.210768] LustreError: 82802:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8ed0905ec000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8ed085b28600) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5916.152333] LustreError: 82787:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5918.422952] LustreError: 82817:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5921.319569] LustreError: lustre-MDT0000-mdc-ffff8ed0905ec000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5922.497988] LustreError: 82817:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5929.436363] LustreError: lustre-MDT0000-mdc-ffff8ed0905ec000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5929.792298] LustreError: 82873:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5933.025511] LustreError: lustre-MDT0000-mdc-ffff8ed0905ec000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5933.121659] LustreError: 82893:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5933.130939] LustreError: 82893:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 16 previous similar messages
[ 5933.888144] LustreError: 82873:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5933.901585] LustreError: 82873:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8ed0905ec000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5933.945338] LustreError: 82873:0:(file.c:251:ll_close_inode_openhandle()) Skipped 1 previous similar message
[ 5934.253160] LustreError: 82899:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5934.270811] LustreError: 82899:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5937.062793] LustreError: 82921:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5937.075734] LustreError: 82921:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5939.623500] LustreError: 82937:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5942.686541] LustreError: lustre-MDT0000-mdc-ffff8ed0905ec000: operation ldlm_enqueue to node 192.168.201.140@tcp failed: rc = -107
[ 5942.697162] LustreError: Skipped 4 previous similar messages
[ 5942.716869] LustreError: lustre-MDT0000-mdc-ffff8ed0905ec000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5942.754483] LustreError: 82951:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8ed0905ec000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8ed08c17ab00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5942.799969] LustreError: 82951:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 3 previous similar messages
[ 5943.712333] LustreError: 82937:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5951.354867] LustreError: lustre-MDT0000-mdc-ffff8ed0905ec000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5951.704314] LustreError: 82993:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5955.280797] LustreError: lustre-MDT0000-mdc-ffff8ed0905ec000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5955.766108] LustreError: 82993:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5960.024525] LustreError: lustre-MDT0000-mdc-ffff8ed0905ec000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5960.041207] LustreError: 83021:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8ed0905ec000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -5
[ 5964.515992] LustreError: 83048:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5964.524373] LustreError: 83048:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5967.671917] LustreError: lustre-MDT0000-mdc-ffff8ed0905ec000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5967.783664] LustreError: 83070:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5968.528165] LustreError: 83048:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5968.534215] LustreError: 83048:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5968.552976] LustreError: 83048:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8ed0905ec000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5980.339926] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 04:44:07 (1785401047)
[ 6001.179418] LustreError: 68066:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 6001.201348] LustreError: 68066:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 6005.264125] LustreError: 68066:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 6005.284471] LustreError: 68066:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 6015.462864] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 04:44:41 (1785401081)
[ 6067.217640] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 04:45:34 (1785401134)
[ 6133.906917] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 04:46:40 (1785401200)
[ 6170.618839] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 04:47:17 (1785401237)
[ 6171.445690] Lustre: Mounted lustre-client
[ 6297.027866] Lustre: Unmounted lustre-client
[ 6305.131815] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 04:49:31 (1785401371)
[ 6306.082789] LustreError: 87381:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 6309.541737] Lustre: lustre-OST0000-osc-ffff8ed0905ec000: Connection to lustre-OST0000 (at 192.168.201.140@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6309.556031] Lustre: Skipped 17 previous similar messages
[ 6309.596059] LustreError: lustre-OST0000-osc-ffff8ed0905ec000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6309.614755] LustreError: 87463:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8ed0905ec000: namespace resource [0x240000400:0x386d:0x0].0x0 (ffff8ed085bb5b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6309.643577] LustreError: 87463:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 4 previous similar messages
[ 6309.968206] LustreError: 87381:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout interrupted
[ 6309.973567] Lustre: 2369:0:(llite_lib.c:4339:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.140@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 6317.770941] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 04:49:44 (1785401384)
[ 6376.416158] Lustre: 88048:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785401389/real 1785401389]  req@ffff8ed08732a300 x1872122500039040/t0(0) o101->lustre-MDT0000-mdc-ffff8ed0905ec000@192.168.201.140@tcp:12/10 lens 328/344 e 0 to 1 dl 1785401444 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 6376.468536] Lustre: 88048:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 20 previous similar messages
[ 6376.533094] Lustre: lustre-MDT0000-mdc-ffff8ed0905ec000: Connection restored to 192.168.201.140@tcp (at 192.168.201.140@tcp)
[ 6376.551852] Lustre: Skipped 19 previous similar messages
[ 6389.331670] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 04:50:55 (1785401455)
[ 6391.824754] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 6394.109959] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 04:51:00 (1785401460)
[ 6460.768575] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6462.791984] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 04:52:09 (1785401529)
[ 6521.836317] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 04:53:08 (1785401588)
[ 6526.888115] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6529.460580] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 04:53:15 (1785401595)
[ 6532.027161] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6534.961053] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 04:53:21 (1785401601)
[ 6580.585883] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 04:54:07 (1785401647)
[ 6594.838721] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6623.200392] LustreError: MGC192.168.201.140@tcp: Connection to MGS (at 192.168.201.140@tcp) was lost; in progress operations using this service will fail
[ 6623.220816] LustreError: Skipped 3 previous similar messages
[ 6633.473891] Lustre: Evicted from MGS (at 192.168.201.140@tcp) after server handle changed from 0xc146da9e02711333 to 0xc146da9e02712aef
[ 6633.481907] Lustre: Skipped 3 previous similar messages
[ 6652.113347] Lustre: DEBUG MARKER: oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6655.037776] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6670.913217] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 04:55:36 (1785401736)
[ 6674.410752] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6677.630646] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 04:55:43 (1785401743)
[ 6695.919446] LustreError: MGC192.168.201.140@tcp: Connection to MGS (at 192.168.201.140@tcp) was lost; in progress operations using this service will fail
[ 6696.002695] Lustre: Evicted from MGS (at 192.168.201.140@tcp) after server handle changed from 0xc146da9e02712aef to 0xc146da9e02712f8e
[ 6712.739637] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 04:56:18 (1785401778)
[ 6768.546729] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 04:57:15 (1785401835)
[ 6774.934152] LustreError: lustre-OST0000-osc-ffff8ed0905ec000: operation ost_setattr to node 192.168.201.140@tcp failed: rc = -19
[ 6774.957442] LustreError: Skipped 6 previous similar messages
[ 6842.565884] Lustre: DEBUG MARKER: oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6847.192074] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6915.080030] Lustre: lustre-MDT0000-mdc-ffff8ed0905ec000: Connection to lustre-MDT0000 (at 192.168.201.140@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6915.132302] Lustre: Skipped 7 previous similar messages
[ 6936.544548] LustreError: MGC192.168.201.140@tcp: Connection to MGS (at 192.168.201.140@tcp) was lost; in progress operations using this service will fail
[ 6936.557269] LustreError: Skipped 1 previous similar message
[ 6945.779615] Lustre: Evicted from MGS (at 192.168.201.140@tcp) after server handle changed from 0xc146da9e02713297 to 0xc146da9e02714004
[ 6945.801283] Lustre: Skipped 1 previous similar message
[ 6947.522973] Lustre: 68069:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.140@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6964.690523] Lustre: DEBUG MARKER: oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6966.285899] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6990.818572] Lustre: 2369:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785402043/real 1785402043]  req@ffff8ed089791c00 x1872122506578560/t0(0) o400->MGC192.168.201.140@tcp@192.168.201.140@tcp:26/25 lens 224/224 e 0 to 1 dl 1785402059 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6990.843724] Lustre: 2369:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 13 previous similar messages
[ 6990.868544] Lustre: MGC192.168.201.140@tcp: Connection restored to 192.168.201.140@tcp (at 192.168.201.140@tcp)
[ 6990.882897] Lustre: Skipped 10 previous similar messages
[ 6992.306991] Lustre: 68069:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.140@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 7020.156741] Lustre: DEBUG MARKER: oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 7022.653602] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 7195.137536] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 05:04:21 (1785402261)
[ 7260.475280] Lustre: DEBUG MARKER: oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 7268.407626] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 7555.120182] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 05:10:21 (1785402621)
[ 7708.135904] Lustre: lustre-MDT0000-mdc-ffff8ed0905ec000: Connection to lustre-MDT0000 (at 192.168.201.140@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7708.147357] Lustre: Skipped 2 previous similar messages
[ 7708.157626] LustreError: MGC192.168.201.140@tcp: Connection to MGS (at 192.168.201.140@tcp) was lost; in progress operations using this service will fail
[ 7708.167091] LustreError: Skipped 1 previous similar message
[ 7708.212530] Lustre: Evicted from MGS (at 192.168.201.140@tcp) after server handle changed from 0xc146da9e02714337 to 0xc146da9e0280e737
[ 7708.225296] Lustre: Skipped 1 previous similar message
[ 7708.232911] Lustre: MGC192.168.201.140@tcp: Connection restored to 192.168.201.140@tcp (at 192.168.201.140@tcp)
[ 7708.244196] Lustre: Skipped 1 previous similar message
[ 7731.680660] Lustre: 2368:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785402761/real 1785402761]  req@ffff8ed084760a80 x1872122517744896/t0(0) o400->lustre-MDT0000-mdc-ffff8ed0905ec000@192.168.201.140@tcp:12/10 lens 224/224 e 0 to 1 dl 1785402799 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7731.709500] Lustre: 2368:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
[ 7746.019791] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 05:13:32 (1785402812)
[ 7748.111604] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7749.859378] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 05:13:36 (1785402816)
[ 7752.445403] LustreError: lustre-MDT0000-mdc-ffff8ed0905ec000: operation ldlm_enqueue to node 192.168.201.140@tcp failed: rc = -107
[ 7752.452979] LustreError: Skipped 109 previous similar messages
[ 7752.461834] LustreError: lustre-MDT0000-mdc-ffff8ed0905ec000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7761.813263] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 05:13:48 (1785402828)
[ 7928.371489] Lustre: Evicted from lustre-OST0000_UUID (at 192.168.201.140@tcp) after server handle changed from 0xc146da9e0280ea5c to 0xc146da9e0280eb66
[ 7928.381188] LustreError: lustre-OST0000-osc-ffff8ed0905ec000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7931.000816] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 05:16:36 (1785402996)
[ 7977.018929] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 05:17:23 (1785403043)
[ 7978.967294] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7980.808413] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 05:17:27 (1785403047)
[ 7983.101994] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7985.687348] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 05:17:32 (1785403052)
[ 8024.344669] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 05:18:10 (1785403090)
[ 8071.659580] LustreError: MGC192.168.201.140@tcp: Connection to MGS (at 192.168.201.140@tcp) was lost; in progress operations using this service will fail
[ 8071.698713] Lustre: Evicted from MGS (at 192.168.201.140@tcp) after server handle changed from 0xc146da9e0280e737 to 0xc146da9e02810eb3
[ 8090.754597] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 05:19:17 (1785403157)
[ 8093.017663] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 8095.311289] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 05:19:21 (1785403161)
[ 8096.836577] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 8099.059880] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 05:19:25 (1785403165)
[ 8117.986026] Lustre: Unmounted lustre-client
[ 8120.613614] Lustre: Mounted lustre-client
[ 8125.397437] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 8147.427373] LustreError: MGC192.168.201.140@tcp: Connection to MGS (at 192.168.201.140@tcp) was lost; in progress operations using this service will fail
[ 8157.672644] Lustre: Evicted from MGS (at 192.168.201.140@tcp) after server handle changed from 0xc146da9e028117ab to 0xc146da9e02811979
[ 8169.367237] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2