[ 1963.506854] LustreError: lustre-OST0000-osc-ffff8e9951114000: operation ost_sync to node 192.168.202.111@tcp failed: rc = -107
[ 1963.525483] LustreError: lustre-OST0000-osc-ffff8e9951114000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1965.698190] Lustre: DEBUG MARKER: recovery-small test_24b: @@@@@@ IGNORE (bz5494): multiop didn't fail fsync: 5 or close: 0
[ 1968.660361] Lustre: DEBUG MARKER: recovery-small test_24b: @@@@@@ FAIL: no discarded dirty page found!
[ 1987.347531] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 18:08:55 (1786831735)
[ 1990.321351] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1992.584393] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 18:09:00 (1786831740)
[ 1994.306817] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1996.126950] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 18:09:04 (1786831744)
[ 1999.567864] LustreError: lustre-MDT0000-mdc-ffff8e9951114000: operation ldlm_enqueue to node 192.168.202.111@tcp failed: rc = -19
[ 2018.272495] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 192.168.202.111@tcp) was lost; in progress operations using this service will fail
[ 2028.519267] Lustre: Evicted from MGS (at 192.168.202.111@tcp) after server handle changed from 0x38c96fa7eb56e500 to 0x38c96fa7eb56fc6f
[ 2136.988212] LustreError: lustre-MDT0000-mdc-ffff8e9951114000: operation ldlm_enqueue to node 192.168.202.111@tcp failed: rc = -19
[ 2137.008396] LustreError: Skipped 1 previous similar message
[ 2155.488503] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 192.168.202.111@tcp) was lost; in progress operations using this service will fail
[ 2165.686590] Lustre: Evicted from MGS (at 192.168.202.111@tcp) after server handle changed from 0x38c96fa7eb56fc6f to 0x38c96fa7eb590524
[ 2180.686259] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 18:12:08 (1786831928)
[ 2181.489238] Lustre: *** cfs_fail_loc=305, val=0***
[ 2181.492530] Lustre: Skipped 2 previous similar messages
[ 2220.003156] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 192.168.202.111@tcp) was lost; in progress operations using this service will fail
[ 2230.263025] Lustre: lustre-MDT0000-mdc-ffff8e9951114000: Connection to lustre-MDT0000 (at 192.168.202.111@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2230.277770] Lustre: Skipped 12 previous similar messages
[ 2230.288791] Lustre: Evicted from MGS (at 192.168.202.111@tcp) after server handle changed from 0x38c96fa7eb590524 to 0x38c96fa7eb590968
[ 2230.312942] Lustre: MGC192.168.202.111@tcp: Connection restored to 192.168.202.111@tcp (at 192.168.202.111@tcp)
[ 2230.324646] Lustre: Skipped 13 previous similar messages
[ 2245.826691] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2247.485647] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2258.674655] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 18:13:25 (1786832005)
[ 2276.339081] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 192.168.202.111@tcp) was lost; in progress operations using this service will fail
[ 2276.386464] Lustre: Evicted from MGS (at 192.168.202.111@tcp) after server handle changed from 0x38c96fa7eb590968 to 0x38c96fa7eb590e70
[ 2276.549234] LustreError: lustre-MDT0000-mdc-ffff8e9951114000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2302.359532] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 18:14:09 (1786832049)
[ 2315.745463] LustreError: lustre-OST0000-osc-ffff8e9951114000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2334.150758] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 18:14:42 (1786832082)
[ 2348.220769] LustreError: lustre-MDT0000-mdc-ffff8e9951114000: operation ldlm_enqueue to node 192.168.202.111@tcp failed: rc = -19
[ 2368.499428] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 192.168.202.111@tcp) was lost; in progress operations using this service will fail
[ 2368.515312] Lustre: Evicted from MGS (at 192.168.202.111@tcp) after server handle changed from 0x38c96fa7eb590e70 to 0x38c96fa7eb596326
[ 2370.284367] Lustre: 15922:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.202.111@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2394.549579] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2396.472176] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2481.120172] Lustre: 2349:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786832214/real 1786832214]  req@ffff8e99442f2d80 x1873626843025408/t0(0) o400->MGC192.168.202.111@tcp@192.168.202.111@tcp:26/25 lens 224/224 e 0 to 1 dl 1786832230 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2481.144208] Lustre: 2349:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 15 previous similar messages
[ 2481.150273] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 192.168.202.111@tcp) was lost; in progress operations using this service will fail
[ 2491.380421] Lustre: Evicted from MGS (at 192.168.202.111@tcp) after server handle changed from 0x38c96fa7eb596326 to 0x38c96fa7eb5b050b
[ 2508.643128] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2510.609263] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2575.922329] LustreError: lustre-MDT0000-mdc-ffff8e9951114000: operation ldlm_enqueue to node 192.168.202.111@tcp failed: rc = -19
[ 2575.935176] LustreError: Skipped 3 previous similar messages
[ 2592.741730] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 192.168.202.111@tcp) was lost; in progress operations using this service will fail
[ 2602.943372] Lustre: Evicted from MGS (at 192.168.202.111@tcp) after server handle changed from 0x38c96fa7eb5b050b to 0x38c96fa7eb5cebae
[ 2604.523296] Lustre: 15922:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.202.111@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2624.494392] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2626.530699] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2655.263413] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 18:20:03 (1786832403)
[ 2678.752363] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 192.168.202.111@tcp) was lost; in progress operations using this service will fail
[ 2688.938367] Lustre: Evicted from MGS (at 192.168.202.111@tcp) after server handle changed from 0x38c96fa7eb5cebae to 0x38c96fa7eb5dccf7
[ 2699.887154] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2732.060797] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2768.177860] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2820.604585] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2843.706696] LustreError: lustre-MDT0000-mdc-ffff8e9951114000: operation ldlm_enqueue to node 192.168.202.111@tcp failed: rc = -19
[ 2843.713175] Lustre: lustre-MDT0000-mdc-ffff8e9951114000: Connection to lustre-MDT0000 (at 192.168.202.111@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2843.719375] LustreError: Skipped 6 previous similar messages
[ 2843.730435] Lustre: Skipped 9 previous similar messages
[ 2860.000227] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 192.168.202.111@tcp) was lost; in progress operations using this service will fail
[ 2860.013802] LustreError: Skipped 3 previous similar messages
[ 2870.199439] Lustre: Evicted from MGS (at 192.168.202.111@tcp) after server handle changed from 0x38c96fa7eb5ecb3c to 0x38c96fa7eb5f68d1
[ 2870.217770] Lustre: Skipped 3 previous similar messages
[ 2870.231542] Lustre: MGC192.168.202.111@tcp: Connection restored to 192.168.202.111@tcp (at 192.168.202.111@tcp)
[ 2870.244717] Lustre: Skipped 17 previous similar messages
[ 2872.552285] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2940.233218] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 3042.476106] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 18:26:30 (1786832790)
[ 3105.451183] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3108.103687] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3388.722910] LustreError: lustre-OST0000-osc-ffff8e9951114000: operation ldlm_enqueue to node 192.168.202.111@tcp failed: rc = -19
[ 3388.738475] LustreError: Skipped 3 previous similar messages
[ 3430.567604] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3432.445669] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3719.534586] Lustre: lustre-OST0000-osc-ffff8e9951114000: Connection to lustre-OST0000 (at 192.168.202.111@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3719.549173] Lustre: Skipped 4 previous similar messages
[ 3757.535097] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3759.367972] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4014.286891] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 18:42:42 (1786833762)
[ 4030.944150] Lustre: 51009:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786833764/real 1786833764]  req@ffff8e99489d0e00 x1873626866517120/t0(0) o101->lustre-MDT0000-mdc-ffff8e9951114000@192.168.202.111@tcp:12/10 lens 576/1152 e 0 to 1 dl 1786833780 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 4030.969180] Lustre: 51009:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 4031.024184] Lustre: lustre-MDT0000-mdc-ffff8e9951114000: Connection restored to 192.168.202.111@tcp (at 192.168.202.111@tcp)
[ 4031.038453] Lustre: Skipped 6 previous similar messages
[ 4037.480895] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 18:43:05 (1786833785)
[ 4062.876225] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 18:43:31 (1786833811)
[ 4088.240574] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 18:43:56 (1786833836)
[ 4088.825823] Lustre: Mounted lustre-client - version 2.17.57_2_gb7122c1
[ 4120.480255] Lustre: 2349:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786833853/real 1786833853]  req@ffff8e9953360a80 x1873626866535424/t0(0) o400->lustre-MDT0000-mdc-ffff8e9951114000@192.168.202.111@tcp:12/10 lens 224/224 e 0 to 1 dl 1786833869 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4120.509276] Lustre: 2349:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 4120.515541] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 192.168.202.111@tcp) was lost; in progress operations using this service will fail
[ 4120.528551] LustreError: Skipped 2 previous similar messages
[ 4129.786793] Lustre: Evicted from MGS (at 192.168.202.111@tcp) after server handle changed from 0x38c96fa7eb60aaca to 0x38c96fa7eb7670c2
[ 4129.804983] Lustre: Skipped 2 previous similar messages
[ 4150.376271] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4151.776576] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4153.960179] Lustre: Unmounted lustre-client
[ 4159.698315] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 18:45:08 (1786833908)
[ 4280.288486] Lustre: 2351:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786834013/real 1786834013]  req@ffff8e99489d1500 x1873626866569728/t0(0) o4->lustre-OST0000-osc-ffff8e9951114000@192.168.202.111@tcp:6/4 lens 488/448 e 0 to 1 dl 1786834029 ref 2 fl Rpc:XQr/602/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4280.321655] Lustre: 2351:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 62 previous similar messages
[ 4328.416731] Lustre: lustre-OST0000-osc-ffff8e9951114000: Connection to lustre-OST0000 (at 192.168.202.111@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4328.436403] Lustre: Skipped 14 previous similar messages
[ 4451.944831] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 18:50:00 (1786834200)
[ 4500.194386] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 18:50:48 (1786834248)
[ 4503.415578] Lustre: Unmounted lustre-client
[ 4525.968832] Lustre: Mounted lustre-client - version 2.17.57_2_gb7122c1
[ 4533.319249] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 18:51:20 (1786834280)
[ 4533.986224] LustreError: 56931:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4535.042306] LustreError: 56931:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4535.341256] Lustre: *** cfs_fail_loc=305, val=0***
[ 4558.152902] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 18:51:46 (1786834306)
[ 4558.971640] Lustre: Mounted lustre-client - version 2.17.57_2_gb7122c1
[ 4561.641484] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4571.979747] Lustre: Unmounted lustre-client
[ 4578.378091] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 18:52:06 (1786834326)
[ 4651.917287] LustreError: lustre-MDT0000-mdc-ffff8e994891a000: operation mds_reint to node 192.168.202.111@tcp failed: rc = -19
[ 4651.922849] LustreError: Skipped 1 previous similar message
[ 4670.952844] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 192.168.202.111@tcp) was lost; in progress operations using this service will fail
[ 4670.971091] Lustre: Evicted from MGS (at 192.168.202.111@tcp) after server handle changed from 0x38c96fa7eb768d94 to 0x38c96fa7eb7941d7
[ 4670.987603] Lustre: MGC192.168.202.111@tcp: Connection restored to 192.168.202.111@tcp (at 192.168.202.111@tcp)
[ 4670.993899] Lustre: Skipped 23 previous similar messages
[ 4673.015220] Lustre: 56330:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.202.111@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 4801.932574] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 18:55:50 (1786834550)
[ 4805.889831] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4819.432612] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 192.168.202.111@tcp) was lost; in progress operations using this service will fail
[ 4819.450694] Lustre: Evicted from MGS (at 192.168.202.111@tcp) after server handle changed from 0x38c96fa7eb7941d7 to 0x38c96fa7eb7e0bf6
[ 4819.519188] LustreError: lustre-MDT0000-mdc-ffff8e994891a000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4828.655672] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 18:56:16 (1786834576)
[ 4829.039445] Lustre: Mounted lustre-client - version 2.17.57_2_gb7122c1
[ 4836.034420] LustreError: lustre-OST0000-osc-ffff8e994891a000: operation ldlm_enqueue to node 192.168.202.111@tcp failed: rc = -107
[ 4836.053747] LustreError: lustre-OST0000-osc-ffff8e994891a000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4846.560207] Lustre: 2350:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786834579/real 1786834579]  req@ffff8e994377f100 x1873626871222528/t0(0) o103->lustre-OST0000-osc-ffff8e99492ef800@192.168.202.111@tcp:17/18 lens 328/224 e 0 to 1 dl 1786834595 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295
[ 4846.577786] Lustre: 2350:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 85 previous similar messages
[ 4848.406680] Lustre: Unmounted lustre-client
[ 4853.668246] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 18:56:41 (1786834601)
[ 4859.823227] LustreError: lustre-MDT0000-mdc-ffff8e994891a000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4859.841627] LustreError: 61396:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8e994891a000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff8e9946c87e00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4859.872210] LustreError: 61312:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4866.224664] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 18:56:54 (1786834614)
[ 4866.452498] LustreError: 62015:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4870.464800] LustreError: lustre-MDT0000-mdc-ffff8e994891a000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4870.469075] LustreError: 62032:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4870.474465] LustreError: 62015:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=988
[ 4870.477050] LustreError: 62015:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting
[ 4870.521332] LustreError: 62037:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4871.615542] LustreError: 62043:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4871.636159] LustreError: 62043:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4873.929958] LustreError: 62066:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4873.968516] LustreError: 62066:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4878.683255] LustreError: 62111:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4878.692905] LustreError: 62111:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4887.060287] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 18:57:15 (1786834635)
[ 4921.565715] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4923.120817] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4932.162393] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 18:58:00 (1786834680)
[ 4937.206931] Lustre: lustre-OST0000-osc-ffff8e994891a000: Connection to lustre-OST0000 (at 192.168.202.111@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4937.219564] Lustre: Skipped 19 previous similar messages
[ 4969.258345] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4970.833613] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4980.032986] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 18:58:48 (1786834728)
[ 5041.679632] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5043.083201] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5050.885757] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 18:59:59 (1786834799)
[ 5086.069131] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5087.273331] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5091.109320] Lustre: Unmounted lustre-client
[ 5136.151131] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5138.601281] Lustre: Mounted lustre-client - version 2.17.57_2_gb7122c1
[ 5143.846690] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 19:01:31 (1786834891)
[ 5146.344559] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5147.869700] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 19:01:35 (1786834895)
[ 5171.762814] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 19:01:59 (1786834919)
[ 5173.094613] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5174.481228] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 19:02:02 (1786834922)
[ 5175.591959] Lustre: *** cfs_fail_loc=805, val=0***
[ 5175.622676] Lustre: Mounted lustre-client - version 2.17.57_2_gb7122c1
[ 5180.210364] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5201.393784] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 192.168.202.111@tcp) was lost; in progress operations using this service will fail
[ 5201.454381] Lustre: Evicted from MGS (at 192.168.202.111@tcp) after server handle changed from 0x38c96fa7eb7e15ea to 0x38c96fa7eb7e197f
[ 5207.663691] Lustre: Unmounted lustre-client
[ 5213.943421] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 19:02:42 (1786834962)
[ 5237.420936] Lustre: 68687:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.202.111@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 5256.746961] LustreError: 2348:0:(client.c:3435:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff8e994377ed80 x1873626871302528/t94489280516(94489280516) o101->lustre-MDT0000-mdc-ffff8e99456b0800@192.168.202.111@tcp:12/10 lens 648/608 e 0 to 0 dl 1786835022 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5262.140889] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5263.393436] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5272.157981] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 19:03:40 (1786835020)
[ 5273.863405] LustreError: lustre-OST0000-osc-ffff8e99456b0800: operation ost_write to node 192.168.202.111@tcp failed: rc = -107
[ 5273.873239] LustreError: Skipped 2 previous similar messages
[ 5273.889906] LustreError: lustre-OST0000-osc-ffff8e99456b0800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5273.907257] Lustre: 2351:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.111@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5273.933967] LustreError: 73228:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8e99456b0800: namespace resource [0x240000400:0x4162:0x0].0x0 (ffff8e9946c87e00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5281.673585] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 19:03:49 (1786835029)
[ 5283.269300] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5284.898388] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 19:03:52 (1786835032)
[ 5286.466870] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5288.514839] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 19:03:56 (1786835036)
[ 5290.130305] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5292.027547] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 19:03:59 (1786835039)
[ 5293.631918] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5295.487701] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 19:04:03 (1786835043)
[ 5297.360972] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5299.472273] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 19:04:07 (1786835047)
[ 5300.980563] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5302.924573] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 19:04:11 (1786835051)
[ 5304.965727] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5306.759773] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 19:04:14 (1786835054)
[ 5308.480080] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5310.587485] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 19:04:18 (1786835058)
[ 5312.528056] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5314.281564] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 19:04:22 (1786835062)
[ 5315.860833] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5317.885444] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 19:04:25 (1786835065)
[ 5319.703323] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5321.489130] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 19:04:29 (1786835069)
[ 5323.986474] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5325.417595] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 19:04:33 (1786835073)
[ 5343.721945] Lustre: MGC192.168.202.111@tcp: Connection restored to 192.168.202.111@tcp (at 192.168.202.111@tcp)
[ 5343.732646] Lustre: Skipped 18 previous similar messages
[ 5353.544190] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 19:05:01 (1786835101)
[ 5382.329761] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 19:05:30 (1786835130)
[ 5382.693620] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5391.506942] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 19:05:39 (1786835139)
[ 5391.816837] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5402.165704] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 19:05:50 (1786835150)
[ 5402.589981] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5410.462431] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 19:05:58 (1786835158)
[ 5410.840175] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5420.637128] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 19:06:08 (1786835168)
[ 5420.847991] Lustre: *** cfs_fail_loc=510, val=3***
[ 5428.793291] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 19:06:16 (1786835176)
[ 5429.063137] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5488.097436] Lustre: 2350:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786835182/real 1786835182]  req@ffff8e9945ed7800 x1873626871344512/t0(0) o3->lustre-OST0000-osc-ffff8e99456b0800@192.168.202.111@tcp:6/4 lens 488/4536 e 0 to 1 dl 1786835237 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 5488.124879] Lustre: 2350:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
[ 5493.431716] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 19:07:21 (1786835241)
[ 5493.742378] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5553.122129] Lustre: lustre-OST0001-osc-ffff8e99456b0800: Connection to lustre-OST0001 (at 192.168.202.111@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5553.137328] Lustre: Skipped 9 previous similar messages
[ 5558.462718] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 19:08:26 (1786835306)
[ 5558.773598] LustreError: 83441:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5561.570890] LustreError: lustre-MDT0000-mdc-ffff8e99456b0800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5561.587961] LustreError: 83455:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8e99456b0800: inode [0x20000a042:0xe:0x0] mdc close failed: rc = -108
[ 5561.614400] LustreError: 83455:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8e99456b0800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8e9947cf5c00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5562.848916] LustreError: 83441:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5564.938160] LustreError: 83471:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5567.610158] LustreError: lustre-MDT0000-mdc-ffff8e99456b0800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5567.622247] LustreError: 83485:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8e99456b0800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8e9946e20d00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5567.632476] LustreError: 83485:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5569.016153] LustreError: 83471:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5575.788720] LustreError: lustre-MDT0000-mdc-ffff8e99456b0800: operation ldlm_enqueue to node 192.168.202.111@tcp failed: rc = -107
[ 5575.797292] LustreError: Skipped 2 previous similar messages
[ 5575.812502] LustreError: lustre-MDT0000-mdc-ffff8e99456b0800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5576.169539] LustreError: 83527:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5578.869992] LustreError: lustre-MDT0000-mdc-ffff8e99456b0800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5578.937856] LustreError: 83546:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5578.944759] LustreError: 83546:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5580.029860] LustreError: 83552:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5580.037153] LustreError: 83552:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5580.257449] LustreError: 83527:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5580.263789] LustreError: 83527:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8e99456b0800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5580.274418] LustreError: 83527:0:(file.c:251:ll_close_inode_openhandle()) Skipped 1 previous similar message
[ 5582.253943] LustreError: 83575:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5582.259356] LustreError: 83575:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5585.804702] LustreError: 83601:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5588.660101] LustreError: lustre-MDT0000-mdc-ffff8e99456b0800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5588.681907] LustreError: 83616:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8e99456b0800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff8e9946c87a00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5588.695179] LustreError: 83616:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5589.888136] LustreError: 83601:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5596.609792] LustreError: lustre-MDT0000-mdc-ffff8e99456b0800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5596.874202] LustreError: 83658:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5599.567869] LustreError: lustre-MDT0000-mdc-ffff8e99456b0800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5599.588946] LustreError: 83672:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8e99456b0800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8e9946c87000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5599.633849] LustreError: 83672:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 2 previous similar messages
[ 5600.952263] LustreError: 83658:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5603.786602] LustreError: lustre-MDT0000-mdc-ffff8e99456b0800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5603.854227] LustreError: 83705:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5603.869501] LustreError: 83705:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5605.146242] LustreError: 83685:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8e99456b0800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5612.737472] LustreError: 83757:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5612.747890] LustreError: 83757:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5616.376750] LustreError: lustre-MDT0000-mdc-ffff8e99456b0800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5616.463496] LustreError: 83779:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5616.475533] LustreError: 83779:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5616.816860] LustreError: 83757:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5616.825212] LustreError: 83757:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5616.836383] LustreError: 83757:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8e99456b0800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5625.599361] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 19:09:33 (1786835373)
[ 5644.612818] LustreError: 68685:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5644.617638] LustreError: 68685:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5648.680405] LustreError: 68685:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5648.688437] LustreError: 68685:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5653.715947] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 19:10:02 (1786835402)
[ 5701.494220] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 19:10:49 (1786835449)
[ 5762.330323] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 19:11:50 (1786835510)
[ 5792.158131] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 19:12:20 (1786835540)
[ 5792.563933] Lustre: Mounted lustre-client - version 2.17.57_2_gb7122c1
[ 5914.636969] Lustre: Unmounted lustre-client
[ 5918.410647] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 19:14:26 (1786835666)
[ 5918.730419] LustreError: 88083:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5920.451318] LustreError: lustre-OST0000-osc-ffff8e99456b0800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5920.468886] LustreError: 88165:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8e99456b0800: namespace resource [0x240000400:0x418d:0x0].0x0 (ffff8e99435c0300) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5920.481388] LustreError: 88165:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5920.720300] LustreError: 88083:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5920.725594] Lustre: 2350:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.111@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5923.994405] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 19:14:32 (1786835672)
[ 5981.163972] Lustre: lustre-MDT0000-mdc-ffff8e99456b0800: Connection restored to 192.168.202.111@tcp (at 192.168.202.111@tcp)
[ 5981.170102] Lustre: Skipped 13 previous similar messages
[ 5985.615891] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 19:15:34 (1786835734)
[ 5986.765547] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 5987.710572] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 19:15:36 (1786835736)
[ 6048.205781] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6049.086956] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 19:16:37 (1786835797)
[ 6105.057199] Lustre: 90478:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786835799/real 1786835799]  req@ffff8e9945ed4a80 x1873626871495680/t0(0) o36->lustre-MDT0000-mdc-ffff8e99456b0800@192.168.202.111@tcp:12/10 lens 496/440 e 0 to 1 dl 1786835854 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ln.0' uid:0 gid:0 projid:4294967295
[ 6105.076852] Lustre: 90478:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 6106.178585] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 19:17:34 (1786835854)
[ 6107.685471] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6108.907302] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 19:17:37 (1786835857)
[ 6109.776284] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6110.918939] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 19:17:39 (1786835859)
[ 6130.483612] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 19:17:58 (1786835878)
[ 6137.034347] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6161.379145] Lustre: lustre-MDT0000-mdc-ffff8e99456b0800: Connection to lustre-MDT0000 (at 192.168.202.111@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6161.384171] Lustre: Skipped 16 previous similar messages
[ 6161.386634] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 192.168.202.111@tcp) was lost; in progress operations using this service will fail
[ 6161.391864] LustreError: Skipped 2 previous similar messages
[ 6161.398669] Lustre: Evicted from MGS (at 192.168.202.111@tcp) after server handle changed from 0x38c96fa7eb7e2429 to 0x38c96fa7eb7e3e0e
[ 6161.403087] Lustre: Skipped 2 previous similar messages
[ 6164.849675] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6165.612185] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6171.143689] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 19:18:39 (1786835919)
[ 6172.418894] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6173.520344] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 19:18:41 (1786835921)
[ 6187.541918] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 19:18:56 (1786835936)
[ 6202.346950] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 192.168.202.111@tcp) was lost; in progress operations using this service will fail
[ 6202.354382] LustreError: Skipped 1 previous similar message
[ 6202.374831] Lustre: Evicted from MGS (at 192.168.202.111@tcp) after server handle changed from 0x38c96fa7eb7e40ca to 0x38c96fa7eb7e44cf
[ 6202.381836] Lustre: Skipped 1 previous similar message
[ 6209.774741] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 19:19:18 (1786835958)
[ 6213.621786] LustreError: lustre-OST0000-osc-ffff8e99456b0800: operation ost_setattr to node 192.168.202.111@tcp failed: rc = -19
[ 6213.626825] LustreError: Skipped 7 previous similar messages
[ 6241.918919] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6244.053747] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6325.230956] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 192.168.202.111@tcp) was lost; in progress operations using this service will fail
[ 6325.253899] Lustre: Evicted from MGS (at 192.168.202.111@tcp) after server handle changed from 0x38c96fa7eb7e44cf to 0x38c96fa7eb7e531c
[ 6330.832475] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6331.584882] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6358.953523] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6359.950979] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6435.823985] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 19:23:04 (1786836184)
[ 6472.201758] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6473.720500] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6659.728225] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 19:26:48 (1786836408)
[ 6724.580410] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 192.168.202.111@tcp) was lost; in progress operations using this service will fail
[ 6724.585634] LustreError: Skipped 1 previous similar message
[ 6724.596111] Lustre: Evicted from MGS (at 192.168.202.111@tcp) after server handle changed from 0x38c96fa7eb7e56aa to 0x38c96fa7eb8d8abf
[ 6724.600612] Lustre: Skipped 1 previous similar message
[ 6724.603869] Lustre: MGC192.168.202.111@tcp: Connection restored to 192.168.202.111@tcp (at 192.168.202.111@tcp)
[ 6724.607321] Lustre: Skipped 12 previous similar messages
[ 6744.895243] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 19:28:13 (1786836493)
[ 6745.637834] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 6746.591567] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 19:28:15 (1786836495)
[ 6747.821539] LustreError: lustre-MDT0000-mdc-ffff8e99456b0800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6751.409951] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 19:28:20 (1786836500)
[ 6917.100530] LustreError: lustre-OST0000-osc-ffff8e99456b0800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6918.042393] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 19:31:06 (1786836666)
[ 6940.640149] Lustre: 2349:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786836669/real 1786836669]  req@ffff8e9948b7b480 x1873626888646272/t0(0) o4->lustre-OST0000-osc-ffff8e99456b0800@192.168.202.111@tcp:6/4 lens 4584/448 e 0 to 1 dl 1786836689 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 6940.649877] Lustre: 2349:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 13 previous similar messages
[ 6940.652839] Lustre: lustre-OST0000-osc-ffff8e99456b0800: Connection to lustre-OST0000 (at 192.168.202.111@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6940.657618] Lustre: Skipped 9 previous similar messages
[ 6953.520798] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 19:31:42 (1786836702)
[ 6954.277376] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 6955.128474] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 19:31:43 (1786836703)
[ 6955.821219] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 6956.583883] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 19:31:45 (1786836705)
[ 6982.707819] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 19:32:11 (1786836731)
[ 7014.371412] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 192.168.202.111@tcp) was lost; in progress operations using this service will fail
[ 7014.380666] Lustre: Evicted from MGS (at 192.168.202.111@tcp) after server handle changed from 0x38c96fa7eb8d8abf to 0x38c96fa7eb8db9c7
[ 7014.385569] Lustre: Skipped 1 previous similar message
[ 7018.388459] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 19:32:46 (1786836766)
[ 7019.235883] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7020.172284] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 19:32:48 (1786836768)
[ 7021.258862] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7022.275581] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 19:32:50 (1786836770)
[ 7029.428140] Lustre: Unmounted lustre-client
[ 7030.048378] Lustre: Mounted lustre-client - version 2.17.57_2_gb7122c1
[ 7031.428315] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7049.944483] LustreError: lustre-MDT0000-mdc-ffff8e99492a5800: operation ldlm_enqueue to node 192.168.202.111@tcp failed: rc = -107
[ 7049.950186] LustreError: Skipped 70 previous similar messages
[ 7051.541142] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2