[ 1964.539303] LustreError: lustre-OST0000-osc-ffff8cdd0cfea800: operation ost_sync to node 192.168.203.151@tcp failed: rc = -107
[ 1964.570856] LustreError: lustre-OST0000-osc-ffff8cdd0cfea800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1966.028760] Lustre: DEBUG MARKER: recovery-small test_24b: @@@@@@ IGNORE (bz5494): multiop didn't fail fsync: 5 or close: 0
[ 1968.977128] Lustre: DEBUG MARKER: recovery-small test_24b: @@@@@@ FAIL: no discarded dirty page found!
[ 1985.521706] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 06:33:44 (1786185224)
[ 1987.912117] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1989.581807] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 06:33:48 (1786185228)
[ 1991.063600] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1992.740682] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 06:33:51 (1786185231)
[ 1996.647691] LustreError: lustre-MDT0000-mdc-ffff8cdd0cfea800: operation mds_reint to node 192.168.203.151@tcp failed: rc = -19
[ 2014.176332] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 192.168.203.151@tcp) was lost; in progress operations using this service will fail
[ 2024.375387] Lustre: Evicted from MGS (at 192.168.203.151@tcp) after server handle changed from 0x44cdc2145c23cb7f to 0x44cdc2145c23e151
[ 2117.393221] LustreError: lustre-MDT0000-mdc-ffff8cdd0cfea800: operation ldlm_enqueue to node 192.168.203.151@tcp failed: rc = -19
[ 2137.137703] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 192.168.203.151@tcp) was lost; in progress operations using this service will fail
[ 2137.153900] Lustre: Evicted from MGS (at 192.168.203.151@tcp) after server handle changed from 0x44cdc2145c23e151 to 0x44cdc2145c24f450
[ 2138.420300] Lustre: 15937:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.151@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2163.143263] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 06:36:41 (1786185401)
[ 2163.642823] Lustre: *** cfs_fail_loc=305, val=0***
[ 2163.655288] Lustre: Skipped 4 previous similar messages
[ 2183.066517] LustreError: lustre-OST0001-osc-ffff8cdd0cfea800: operation ost_connect to node 192.168.203.151@tcp failed: rc = -75
[ 2207.711294] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 192.168.203.151@tcp) was lost; in progress operations using this service will fail
[ 2207.734511] Lustre: Evicted from MGS (at 192.168.203.151@tcp) after server handle changed from 0x44cdc2145c24f450 to 0x44cdc2145c24fa31
[ 2209.193774] Lustre: 15937:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.151@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2228.346847] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2229.570368] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2236.797495] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 06:37:56 (1786185476)
[ 2248.776470] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 192.168.203.151@tcp) was lost; in progress operations using this service will fail
[ 2248.845056] Lustre: Evicted from MGS (at 192.168.203.151@tcp) after server handle changed from 0x44cdc2145c24fa31 to 0x44cdc2145c250035
[ 2248.982186] LustreError: lustre-MDT0000-mdc-ffff8cdd0cfea800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2268.376884] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 06:38:27 (1786185507)
[ 2280.381546] LustreError: lustre-OST0000-osc-ffff8cdd0cfea800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2297.232532] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 06:38:56 (1786185536)
[ 2311.618194] LustreError: lustre-MDT0000-mdc-ffff8cdd0cfea800: operation mds_close to node 192.168.203.151@tcp failed: rc = -19
[ 2311.634228] LustreError: Skipped 1 previous similar message
[ 2330.621715] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 192.168.203.151@tcp) was lost; in progress operations using this service will fail
[ 2330.655237] Lustre: Evicted from MGS (at 192.168.203.151@tcp) after server handle changed from 0x44cdc2145c250035 to 0x44cdc2145c25372b
[ 2352.825889] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2356.496802] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2443.241181] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 192.168.203.151@tcp) was lost; in progress operations using this service will fail
[ 2443.408516] Lustre: Evicted from MGS (at 192.168.203.151@tcp) after server handle changed from 0x44cdc2145c25372b to 0x44cdc2145c267891
[ 2464.601798] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2468.584645] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2537.151226] LustreError: lustre-MDT0000-mdc-ffff8cdd0cfea800: operation ldlm_enqueue to node 192.168.203.151@tcp failed: rc = -19
[ 2537.160845] LustreError: Skipped 1 previous similar message
[ 2556.394954] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 192.168.203.151@tcp) was lost; in progress operations using this service will fail
[ 2556.421378] Lustre: Evicted from MGS (at 192.168.203.151@tcp) after server handle changed from 0x44cdc2145c267891 to 0x44cdc2145c27c208
[ 2556.429732] Lustre: MGC192.168.203.151@tcp: Connection restored to 192.168.203.151@tcp (at 192.168.203.151@tcp)
[ 2556.434822] Lustre: Skipped 13 previous similar messages
[ 2578.671704] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2582.099984] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2612.441801] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 06:44:11 (1786185851)
[ 2617.268638] Lustre: lustre-MDT0000-mdc-ffff8cdd0cfea800: Connection to lustre-MDT0000 (at 192.168.203.151@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2617.283445] Lustre: Skipped 9 previous similar messages
[ 2634.207159] Lustre: 2364:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786185858/real 1786185858]  req@ffff8cdd0897b480 x1872948990406144/t0(0) o400->MGC192.168.203.151@tcp@192.168.203.151@tcp:26/25 lens 224/224 e 0 to 1 dl 1786185874 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2634.237660] Lustre: 2364:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 13 previous similar messages
[ 2634.245782] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 192.168.203.151@tcp) was lost; in progress operations using this service will fail
[ 2644.543415] Lustre: Evicted from MGS (at 192.168.203.151@tcp) after server handle changed from 0x44cdc2145c27c208 to 0x44cdc2145c2860ae
[ 2653.646399] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2687.954835] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2727.666652] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2763.308544] Lustre: 15937:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.151@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2778.027491] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2802.625875] LustreError: lustre-MDT0000-mdc-ffff8cdd0cfea800: operation mds_reint to node 192.168.203.151@tcp failed: rc = -19
[ 2802.631726] LustreError: Skipped 7 previous similar messages
[ 2819.560772] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 192.168.203.151@tcp) was lost; in progress operations using this service will fail
[ 2819.581505] LustreError: Skipped 3 previous similar messages
[ 2829.872763] Lustre: Evicted from MGS (at 192.168.203.151@tcp) after server handle changed from 0x44cdc2145c28eb3b to 0x44cdc2145c293c86
[ 2829.894426] Lustre: Skipped 3 previous similar messages
[ 2831.592162] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2893.033696] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2982.533624] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 06:50:21 (1786186221)
[ 3047.708612] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3052.139135] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3329.465552] LustreError: lustre-OST0000-osc-ffff8cdd0cfea800: operation ldlm_enqueue to node 192.168.203.151@tcp failed: rc = -19
[ 3329.495425] LustreError: Skipped 6 previous similar messages
[ 3329.499830] Lustre: lustre-OST0000-osc-ffff8cdd0cfea800: Connection to lustre-OST0000 (at 192.168.203.151@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3329.512259] Lustre: Skipped 6 previous similar messages
[ 3353.363933] Lustre: lustre-OST0000-osc-ffff8cdd0cfea800: Connection restored to 192.168.203.151@tcp (at 192.168.203.151@tcp)
[ 3353.376288] Lustre: Skipped 15 previous similar messages
[ 3379.425742] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3381.973762] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3715.176578] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3719.014901] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3955.581157] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 07:06:34 (1786187194)
[ 3974.113145] Lustre: 50989:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786187198/real 1786187198]  req@ffff8cdd07262d80 x1872949002966912/t0(0) o101->lustre-MDT0000-mdc-ffff8cdd0cfea800@192.168.203.151@tcp:12/10 lens 576/1152 e 0 to 1 dl 1786187214 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3974.126109] Lustre: 50989:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 3974.129823] Lustre: lustre-MDT0000-mdc-ffff8cdd0cfea800: Connection to lustre-MDT0000 (at 192.168.203.151@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3974.136903] Lustre: Skipped 1 previous similar message
[ 3974.158417] Lustre: lustre-MDT0000-mdc-ffff8cdd0cfea800: Connection restored to 192.168.203.151@tcp (at 192.168.203.151@tcp)
[ 3982.201926] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 07:07:01 (1786187221)
[ 4009.771709] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 07:07:28 (1786187248)
[ 4035.576564] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 07:07:54 (1786187274)
[ 4036.417415] Lustre: Mounted lustre-client - version 2.17.56_50_g80e3cdc
[ 4068.319147] Lustre: 2362:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786187292/real 1786187292]  req@ffff8cdd0607e680 x1872949002984576/t0(0) o400->MGC192.168.203.151@tcp@192.168.203.151@tcp:26/25 lens 224/224 e 0 to 1 dl 1786187308 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4068.334360] Lustre: 2362:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
[ 4068.339855] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 192.168.203.151@tcp) was lost; in progress operations using this service will fail
[ 4068.344937] LustreError: Skipped 2 previous similar messages
[ 4068.359151] Lustre: Evicted from MGS (at 192.168.203.151@tcp) after server handle changed from 0x44cdc2145c2a282c to 0x44cdc2145c370ad7
[ 4068.366511] Lustre: Skipped 2 previous similar messages
[ 4070.380344] Lustre: 15937:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.151@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 4093.230823] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4094.814988] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4097.015203] Lustre: Unmounted lustre-client
[ 4103.110854] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 07:09:02 (1786187342)
[ 4226.015265] Lustre: 2364:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786187450/real 1786187450]  req@ffff8cdd09269180 x1872949003019136/t0(0) o4->lustre-OST0000-osc-ffff8cdd0cfea800@192.168.203.151@tcp:6/4 lens 488/448 e 0 to 1 dl 1786187466 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4226.047644] Lustre: 2364:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 63 previous similar messages
[ 4402.454880] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 07:14:01 (1786187641)
[ 4452.545996] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 07:14:51 (1786187691)
[ 4455.541187] Lustre: Unmounted lustre-client
[ 4479.287648] Lustre: Mounted lustre-client - version 2.17.56_50_g80e3cdc
[ 4487.298204] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 07:15:26 (1786187726)
[ 4488.203425] LustreError: 56923:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4489.273702] LustreError: 56923:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4489.456585] Lustre: *** cfs_fail_loc=305, val=0***
[ 4516.190660] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 07:15:55 (1786187755)
[ 4517.023488] LustreError: 2360:0:(lmv_obd.c:211:lmv_notify()) activation of lustre-MDT0000_UUID failed: -22
[ 4517.077556] Lustre: Mounted lustre-client - version 2.17.56_50_g80e3cdc
[ 4518.812918] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4529.091100] Lustre: Unmounted lustre-client
[ 4537.073883] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 07:16:15 (1786187775)
[ 4635.482872] LustreError: lustre-MDT0000-mdc-ffff8cdd091b3800: operation mds_reint to node 192.168.203.151@tcp failed: rc = -19
[ 4635.493094] LustreError: Skipped 2 previous similar messages
[ 4635.495437] Lustre: lustre-MDT0000-mdc-ffff8cdd091b3800: Connection to lustre-MDT0000 (at 192.168.203.151@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4635.502616] Lustre: Skipped 23 previous similar messages
[ 4656.480313] Lustre: 2361:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786187880/real 1786187880]  req@ffff8cdd10a89f80 x1872949005693312/t0(0) o400->MGC192.168.203.151@tcp@192.168.203.151@tcp:26/25 lens 224/224 e 0 to 1 dl 1786187896 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4656.524985] Lustre: 2361:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 84 previous similar messages
[ 4656.532302] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 192.168.203.151@tcp) was lost; in progress operations using this service will fail
[ 4665.712524] Lustre: Evicted from MGS (at 192.168.203.151@tcp) after server handle changed from 0x44cdc2145c3725b8 to 0x44cdc2145c39d9f4
[ 4665.728114] Lustre: MGC192.168.203.151@tcp: Connection restored to 192.168.203.151@tcp (at 192.168.203.151@tcp)
[ 4665.736809] Lustre: Skipped 23 previous similar messages
[ 4867.533343] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 07:21:46 (1786188106)
[ 4873.143942] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4890.126222] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 192.168.203.151@tcp) was lost; in progress operations using this service will fail
[ 4890.144356] Lustre: Evicted from MGS (at 192.168.203.151@tcp) after server handle changed from 0x44cdc2145c39d9f4 to 0x44cdc2145c3ea3e9
[ 4891.647311] LustreError: lustre-MDT0000-mdc-ffff8cdd091b3800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4911.228524] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 07:22:30 (1786188150)
[ 4911.850170] Lustre: Mounted lustre-client - version 2.17.56_50_g80e3cdc
[ 4919.377092] LustreError: lustre-OST0000-osc-ffff8cdd091b3800: operation ldlm_enqueue to node 192.168.203.151@tcp failed: rc = -107
[ 4919.389662] LustreError: lustre-OST0000-osc-ffff8cdd091b3800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4931.526705] Lustre: Unmounted lustre-client
[ 4938.993984] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 07:22:57 (1786188177)
[ 4945.998523] LustreError: lustre-MDT0000-mdc-ffff8cdd091b3800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4946.005934] LustreError: 61321:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4946.016639] LustreError: 61405:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8cdd091b3800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff8cdd035e0800) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4954.674481] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 07:23:13 (1786188193)
[ 4955.505413] LustreError: 62025:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4959.671492] LustreError: lustre-MDT0000-mdc-ffff8cdd091b3800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4959.678024] LustreError: 62042:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4959.681804] LustreError: 62025:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=832
[ 4959.689221] LustreError: 62025:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting
[ 4959.702493] LustreError: 62047:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4960.827420] LustreError: 62053:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4960.841109] LustreError: 62053:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4963.088494] LustreError: 62076:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4963.094127] LustreError: 62076:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4967.692463] LustreError: 62121:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4967.701327] LustreError: 62121:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4979.253919] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 07:23:37 (1786188217)
[ 5019.763942] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5023.309188] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5038.144454] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 07:24:37 (1786188277)
[ 5079.088426] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5080.844165] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5092.043246] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 07:25:31 (1786188331)
[ 5163.897612] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5165.976341] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5176.159351] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 07:26:54 (1786188414)
[ 5219.423797] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5221.485961] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5226.529028] Lustre: Unmounted lustre-client
[ 5285.934309] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5289.936304] Lustre: Mounted lustre-client - version 2.17.56_50_g80e3cdc
[ 5297.204527] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 07:28:56 (1786188536)
[ 5300.128916] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5303.207893] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 07:29:01 (1786188541)
[ 5337.754954] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 07:29:36 (1786188576)
[ 5339.855498] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5342.852715] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 07:29:41 (1786188581)
[ 5344.381793] Lustre: *** cfs_fail_loc=805, val=0***
[ 5344.431213] Lustre: Mounted lustre-client - version 2.17.56_50_g80e3cdc
[ 5351.431408] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5371.407559] Lustre: 2362:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786188595/real 1786188595]  req@ffff8cdd10b87b80 x1872949007755264/t0(0) o400->lustre-MDT0000-mdc-ffff8cdd0cfed000@192.168.203.151@tcp:12/10 lens 224/224 e 0 to 1 dl 1786188611 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5371.445638] Lustre: 2362:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5371.451485] Lustre: lustre-MDT0000-mdc-ffff8cdd0cfed000: Connection to lustre-MDT0000 (at 192.168.203.151@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5371.460586] Lustre: Skipped 10 previous similar messages
[ 5371.463512] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 192.168.203.151@tcp) was lost; in progress operations using this service will fail
[ 5381.626100] Lustre: Evicted from MGS (at 192.168.203.151@tcp) after server handle changed from 0x44cdc2145c3eb022 to 0x44cdc2145c3eb339
[ 5381.682853] Lustre: MGC192.168.203.151@tcp: Connection restored to 192.168.203.151@tcp (at 192.168.203.151@tcp)
[ 5381.705056] Lustre: Skipped 11 previous similar messages
[ 5391.594398] Lustre: Unmounted lustre-client
[ 5400.776291] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 07:30:40 (1786188640)
[ 5443.705947] LustreError: 2360:0:(client.c:3435:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff8cdd04115180 x1872949007767296/t90194313220(90194313220) o101->lustre-MDT0000-mdc-ffff8cdd0cfed000@192.168.203.151@tcp:12/10 lens 664/608 e 0 to 0 dl 1786188700 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5450.724805] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5452.630529] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5462.633902] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 07:31:41 (1786188701)
[ 5465.101810] LustreError: lustre-OST0000-osc-ffff8cdd0cfed000: operation ost_write to node 192.168.203.151@tcp failed: rc = -107
[ 5465.115456] LustreError: Skipped 2 previous similar messages
[ 5465.130258] LustreError: lustre-OST0000-osc-ffff8cdd0cfed000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5465.149224] Lustre: 2364:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.151@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5465.216467] LustreError: 73240:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8cdd0cfed000: namespace resource [0x240000400:0x2d42:0x0].0x0 (ffff8cdd040d5f00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5473.898714] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 07:31:52 (1786188712)
[ 5475.529550] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5477.188339] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 07:31:56 (1786188716)
[ 5478.847214] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5480.746991] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 07:31:59 (1786188719)
[ 5482.780149] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5484.123339] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 07:32:03 (1786188723)
[ 5486.298512] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5488.434403] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 07:32:07 (1786188727)
[ 5489.990237] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5492.481148] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 07:32:11 (1786188731)
[ 5494.371298] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5496.844436] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 07:32:15 (1786188735)
[ 5498.265741] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5499.705360] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 07:32:19 (1786188739)
[ 5501.988308] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5503.373954] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 07:32:22 (1786188742)
[ 5505.304503] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5507.222670] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 07:32:26 (1786188746)
[ 5509.243308] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5512.780410] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 07:32:30 (1786188750)
[ 5514.759277] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5516.817724] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 07:32:35 (1786188755)
[ 5519.403563] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5521.159617] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 07:32:40 (1786188760)
[ 5545.066831] Lustre: 68689:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.151@tcp: IR log lustre-cliir failed, not fatal: rc = -5
[ 5558.636957] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 07:33:17 (1786188797)
[ 5589.190543] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 07:33:48 (1786188828)
[ 5589.930103] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5601.755110] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 07:34:00 (1786188840)
[ 5602.175337] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5613.433312] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 07:34:12 (1786188852)
[ 5614.160975] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5625.700534] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 07:34:24 (1786188864)
[ 5626.480624] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5637.167369] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 07:34:36 (1786188876)
[ 5637.702836] Lustre: *** cfs_fail_loc=510, val=3***
[ 5649.072452] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 07:34:47 (1786188887)
[ 5650.002921] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5661.958771] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 07:35:00 (1786188900)
[ 5662.812904] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5731.580878] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 07:36:10 (1786188970)
[ 5732.224508] LustreError: 83454:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5735.643394] LustreError: lustre-MDT0000-mdc-ffff8cdd0cfed000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5735.660754] LustreError: 83469:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8cdd0cfed000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8cdd040d5900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5736.319549] LustreError: 83454:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5738.450806] LustreError: 83484:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5741.417447] LustreError: lustre-MDT0000-mdc-ffff8cdd0cfed000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5741.445814] LustreError: 83499:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8cdd0cfed000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8cdd041dc100) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5741.542055] LustreError: 83499:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5742.487147] LustreError: 83484:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5749.396929] LustreError: lustre-MDT0000-mdc-ffff8cdd0cfed000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5749.914780] LustreError: 83540:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5752.895140] LustreError: lustre-MDT0000-mdc-ffff8cdd0cfed000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5752.941797] LustreError: 83560:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5752.968201] LustreError: 83560:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5753.927095] LustreError: 83540:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5753.938176] LustreError: 83540:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8cdd0cfed000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5754.106818] LustreError: 83566:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5754.113432] LustreError: 83566:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5756.497018] LustreError: 83589:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5756.522691] LustreError: 83589:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5760.054516] LustreError: 83615:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5762.971758] LustreError: lustre-MDT0000-mdc-ffff8cdd0cfed000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5762.981530] LustreError: 83630:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8cdd0cfed000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8cdd040d5000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5762.997315] LustreError: 83630:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5764.135154] LustreError: 83615:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5771.325470] LustreError: lustre-MDT0000-mdc-ffff8cdd0cfed000: operation ldlm_enqueue to node 192.168.203.151@tcp failed: rc = -107
[ 5771.330162] LustreError: Skipped 5 previous similar messages
[ 5771.365981] LustreError: lustre-MDT0000-mdc-ffff8cdd0cfed000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5771.378671] LustreError: 83659:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8cdd0cfed000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff8cdd041dc300) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5771.389864] LustreError: 83659:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5772.004608] LustreError: 83672:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5775.088704] LustreError: lustre-MDT0000-mdc-ffff8cdd0cfed000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5776.079149] LustreError: 83672:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5779.280159] LustreError: lustre-MDT0000-mdc-ffff8cdd0cfed000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5779.326400] LustreError: 83719:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5779.331986] LustreError: 83719:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5780.207685] LustreError: 83699:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8cdd0cfed000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5788.157851] LustreError: 83771:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5788.162344] LustreError: 83771:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5792.083294] LustreError: lustre-MDT0000-mdc-ffff8cdd0cfed000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5792.258304] LustreError: 83771:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5792.286328] LustreError: 83771:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5792.337154] LustreError: 83771:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8cdd0cfed000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5792.348091] LustreError: 83794:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5792.413530] LustreError: 83794:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5804.723712] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 07:37:23 (1786189043)
[ 5826.412301] LustreError: 68686:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5826.424358] LustreError: 68686:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5830.423192] LustreError: 68686:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5830.427277] LustreError: 68686:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5838.012748] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 07:37:57 (1786189077)
[ 5891.903872] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 07:38:50 (1786189130)
[ 5957.205640] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 07:39:56 (1786189196)
[ 5993.838349] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 07:40:32 (1786189232)
[ 5994.883337] Lustre: Mounted lustre-client - version 2.17.56_50_g80e3cdc
[ 6118.590717] Lustre: Unmounted lustre-client
[ 6129.681456] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 07:42:47 (1786189367)
[ 6131.207711] LustreError: 88102:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 6133.864544] Lustre: lustre-OST0000-osc-ffff8cdd0cfed000: Connection to lustre-OST0000 (at 192.168.203.151@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6133.873918] Lustre: Skipped 17 previous similar messages
[ 6133.882471] LustreError: lustre-OST0000-osc-ffff8cdd0cfed000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6133.900960] LustreError: 88184:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8cdd0cfed000: namespace resource [0x240000400:0x2d6d:0x0].0x0 (ffff8cdd040d5300) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6133.910075] LustreError: 88184:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 4 previous similar messages
[ 6134.191101] LustreError: 88102:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout interrupted
[ 6134.197955] Lustre: 2364:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.151@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 6141.030763] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 07:43:00 (1786189380)
[ 6199.308409] Lustre: 88769:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786189384/real 1786189384]  req@ffff8cdd064fbb80 x1872949007945472/t0(0) o101->lustre-MDT0000-mdc-ffff8cdd0cfed000@192.168.203.151@tcp:12/10 lens 328/344 e 0 to 1 dl 1786189439 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 6199.328508] Lustre: 88769:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 19 previous similar messages
[ 6199.361687] Lustre: lustre-MDT0000-mdc-ffff8cdd0cfed000: Connection restored to 192.168.203.151@tcp (at 192.168.203.151@tcp)
[ 6199.367190] Lustre: Skipped 18 previous similar messages
[ 6208.875378] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 07:44:07 (1786189447)
[ 6210.889897] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 6213.508879] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 07:44:12 (1786189452)
[ 6279.634971] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6281.797531] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 07:45:20 (1786189520)
[ 6341.506520] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 07:46:20 (1786189580)
[ 6344.126904] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6346.369659] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 07:46:25 (1786189585)
[ 6347.848078] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6349.870983] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 07:46:28 (1786189588)
[ 6389.236779] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 07:47:08 (1786189628)
[ 6404.769478] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6432.774723] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 192.168.203.151@tcp) was lost; in progress operations using this service will fail
[ 6432.830818] LustreError: Skipped 3 previous similar messages
[ 6443.547596] Lustre: Evicted from MGS (at 192.168.203.151@tcp) after server handle changed from 0x44cdc2145c3ebea0 to 0x44cdc2145c3ed6b7
[ 6443.567973] Lustre: Skipped 3 previous similar messages
[ 6445.286412] Lustre: 68689:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.151@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6462.607087] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6464.464656] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6476.257418] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 07:48:35 (1786189715)
[ 6478.613182] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6480.458843] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 07:48:39 (1786189719)
[ 6493.155420] LustreError: lustre-MDT0000-mdc-ffff8cdd0cfed000: operation ost_set_info to node 192.168.203.151@tcp failed: rc = -107
[ 6493.163356] LustreError: Skipped 4 previous similar messages
[ 6494.721632] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 192.168.203.151@tcp) was lost; in progress operations using this service will fail
[ 6494.759207] Lustre: Evicted from MGS (at 192.168.203.151@tcp) after server handle changed from 0x44cdc2145c3ed6b7 to 0x44cdc2145c3edd24
[ 6507.421523] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 07:49:06 (1786189746)
[ 6559.971778] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 07:49:59 (1786189799)
[ 6649.005809] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6656.760603] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6747.615294] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 192.168.203.151@tcp) was lost; in progress operations using this service will fail
[ 6747.634237] LustreError: Skipped 1 previous similar message
[ 6757.937980] Lustre: Evicted from MGS (at 192.168.203.151@tcp) after server handle changed from 0x44cdc2145c3edef2 to 0x44cdc2145c3eebc5
[ 6757.946820] Lustre: Skipped 1 previous similar message
[ 6760.233786] Lustre: lustre-MDT0000-mdc-ffff8cdd0cfed000: Connection to lustre-MDT0000 (at 192.168.203.151@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6760.241667] Lustre: Skipped 7 previous similar messages
[ 6776.108889] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6777.983142] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6802.911800] Lustre: 2362:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786189988/real 1786189988]  req@ffff8cdd382e2300 x1872949014469888/t0(0) o400->lustre-MDT0000-mdc-ffff8cdd0cfed000@192.168.203.151@tcp:12/10 lens 224/224 e 0 to 1 dl 1786190043 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6802.959673] Lustre: 2362:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 24 previous similar messages
[ 6814.326281] Lustre: lustre-MDT0000-mdc-ffff8cdd0cfed000: Connection restored to 192.168.203.151@tcp (at 192.168.203.151@tcp)
[ 6814.331854] Lustre: Skipped 10 previous similar messages
[ 6828.108810] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6830.271646] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 7028.807102] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 07:57:47 (1786190267)
[ 7143.149864] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 7151.146346] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 7451.363871] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 08:04:50 (1786190690)
[ 7619.048326] Lustre: lustre-MDT0000-mdc-ffff8cdd0cfed000: Connection to lustre-MDT0000 (at 192.168.203.151@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7619.063688] Lustre: Skipped 2 previous similar messages
[ 7634.440528] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 192.168.203.151@tcp) was lost; in progress operations using this service will fail
[ 7634.458532] LustreError: Skipped 1 previous similar message
[ 7634.471772] Lustre: Evicted from MGS (at 192.168.203.151@tcp) after server handle changed from 0x44cdc2145c3eefdf to 0x44cdc2145c4e93df
[ 7634.477384] Lustre: Skipped 1 previous similar message
[ 7634.481516] Lustre: MGC192.168.203.151@tcp: Connection restored to 192.168.203.151@tcp (at 192.168.203.151@tcp)
[ 7634.485855] Lustre: Skipped 1 previous similar message
[ 7670.080420] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 08:08:29 (1786190909)
[ 7671.940581] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7674.515975] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 08:08:32 (1786190912)
[ 7676.776676] LustreError: lustre-MDT0000-mdc-ffff8cdd0cfed000: operation ldlm_enqueue to node 192.168.203.151@tcp failed: rc = -107
[ 7676.782248] LustreError: Skipped 304 previous similar messages
[ 7676.798960] LustreError: lustre-MDT0000-mdc-ffff8cdd0cfed000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7676.807407] LustreError: 102503:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8cdd0cfed000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff8cdd0437c900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 7676.820547] LustreError: 102503:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 7685.199775] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 08:08:44 (1786190924)
[ 7853.479815] Lustre: Evicted from lustre-OST0000_UUID (at 192.168.203.151@tcp) after server handle changed from 0x44cdc2145c4e98c4 to 0x44cdc2145c4e99ce
[ 7853.488930] LustreError: lustre-OST0000-osc-ffff8cdd0cfed000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7855.039634] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 08:11:34 (1786191094)
[ 7882.215806] Lustre: 2362:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786191102/real 1786191102]  req@ffff8cdd07ae4380 x1872949025699456/t0(0) o4->lustre-OST0000-osc-ffff8cdd0cfed000@192.168.203.151@tcp:6/4 lens 4584/448 e 0 to 1 dl 1786191122 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7882.239519] Lustre: 2362:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 9 previous similar messages
[ 7901.084259] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 08:12:19 (1786191139)
[ 7903.876936] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7905.311210] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 08:12:24 (1786191144)
[ 7906.972333] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7908.713476] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 08:12:27 (1786191147)
[ 7944.265765] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 08:13:02 (1786191182)
[ 7984.624178] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 192.168.203.151@tcp) was lost; in progress operations using this service will fail
[ 7984.645871] Lustre: Evicted from MGS (at 192.168.203.151@tcp) after server handle changed from 0x44cdc2145c4e93df to 0x44cdc2145c4ebd37
[ 8000.672260] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 08:13:59 (1786191239)
[ 8002.656103] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 8004.506536] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 08:14:03 (1786191243)
[ 8006.689520] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 8008.579398] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 08:14:07 (1786191247)
[ 8018.529149] Lustre: Unmounted lustre-client
[ 8020.683305] Lustre: Mounted lustre-client - version 2.17.56_50_g80e3cdc
[ 8023.487589] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 8042.463512] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 192.168.203.151@tcp) was lost; in progress operations using this service will fail
[ 8051.689283] Lustre: Evicted from MGS (at 192.168.203.151@tcp) after server handle changed from 0x44cdc2145c4ec70f to 0x44cdc2145c4ec763
[ 8061.496979] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2