[ 1996.422541] LustreError: lustre-OST0000-osc-ffff908bd920d800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1996.436348] Lustre: 2358:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.125@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 2003.480327] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 17:41:47 (1787694107)
[ 2006.251308] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 2008.338499] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 17:41:52 (1787694112)
[ 2010.130021] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 2011.604677] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 17:41:55 (1787694115)
[ 2031.647331] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 192.168.202.125@tcp) was lost; in progress operations using this service will fail
[ 2041.839224] Lustre: Evicted from MGS (at 192.168.202.125@tcp) after server handle changed from 0x2c22264a0adf8285 to 0x2c22264a0adfa0e6
[ 2137.391659] LustreError: lustre-MDT0000-mdc-ffff908bd920d800: operation mds_reint to node 192.168.202.125@tcp failed: rc = -19
[ 2137.405726] LustreError: Skipped 2 previous similar messages
[ 2154.463337] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 192.168.202.125@tcp) was lost; in progress operations using this service will fail
[ 2163.626725] Lustre: Evicted from MGS (at 192.168.202.125@tcp) after server handle changed from 0x2c22264a0adfa0e6 to 0x2c22264a0ae19264
[ 2165.547269] Lustre: 15951:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.125@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2182.286802] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 17:44:46 (1787694286)
[ 2182.692485] Lustre: *** cfs_fail_loc=305, val=0***
[ 2182.694717] Lustre: Skipped 2 previous similar messages
[ 2225.119405] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 192.168.202.125@tcp) was lost; in progress operations using this service will fail
[ 2225.140964] Lustre: Evicted from MGS (at 192.168.202.125@tcp) after server handle changed from 0x2c22264a0ae19264 to 0x2c22264a0ae1a35f
[ 2227.181416] Lustre: 15951:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.125@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2234.343305] Lustre: lustre-MDT0000-mdc-ffff908bd920d800: Connection to lustre-MDT0000 (at 192.168.202.125@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2234.363871] Lustre: Skipped 11 previous similar messages
[ 2246.206597] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2247.690768] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2255.018095] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 17:45:59 (1787694359)
[ 2270.190575] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 192.168.202.125@tcp) was lost; in progress operations using this service will fail
[ 2270.229101] Lustre: Evicted from MGS (at 192.168.202.125@tcp) after server handle changed from 0x2c22264a0ae1a35f to 0x2c22264a0ae1aa35
[ 2270.271499] Lustre: MGC192.168.202.125@tcp: Connection restored to 192.168.202.125@tcp (at 192.168.202.125@tcp)
[ 2270.281657] LustreError: lustre-MDT0000-mdc-ffff908bd920d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2270.298870] Lustre: Skipped 14 previous similar messages
[ 2298.296528] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 17:46:42 (1787694402)
[ 2325.903973] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 17:47:10 (1787694430)
[ 2339.474826] LustreError: lustre-MDT0000-mdc-ffff908bd920d800: operation mds_reint to node 192.168.202.125@tcp failed: rc = -19
[ 2358.239267] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 192.168.202.125@tcp) was lost; in progress operations using this service will fail
[ 2368.489721] Lustre: Evicted from MGS (at 192.168.202.125@tcp) after server handle changed from 0x2c22264a0ae1aa35 to 0x2c22264a0ae20cb3
[ 2387.486853] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2390.095118] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2474.980662] Lustre: 2360:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787694564/real 1787694564]  req@ffff908be0171180 x1874531161351936/t0(0) o400->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 224/224 e 0 to 1 dl 1787694580 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2475.031483] Lustre: 2360:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 13 previous similar messages
[ 2475.043669] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 192.168.202.125@tcp) was lost; in progress operations using this service will fail
[ 2485.238203] Lustre: Evicted from MGS (at 192.168.202.125@tcp) after server handle changed from 0x2c22264a0ae20cb3 to 0x2c22264a0ae3e269
[ 2501.856951] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2503.934367] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2569.416984] LustreError: lustre-MDT0000-mdc-ffff908bd920d800: operation mds_reint to node 192.168.202.125@tcp failed: rc = -19
[ 2569.426605] LustreError: Skipped 2 previous similar messages
[ 2586.592332] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 192.168.202.125@tcp) was lost; in progress operations using this service will fail
[ 2596.793413] Lustre: Evicted from MGS (at 192.168.202.125@tcp) after server handle changed from 0x2c22264a0ae3e269 to 0x2c22264a0ae5b731
[ 2613.741550] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2616.554393] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2645.559747] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 17:52:30 (1787694750)
[ 2668.512614] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 192.168.202.125@tcp) was lost; in progress operations using this service will fail
[ 2677.748221] Lustre: Evicted from MGS (at 192.168.202.125@tcp) after server handle changed from 0x2c22264a0ae5b731 to 0x2c22264a0ae67ba8
[ 2679.061477] Lustre: 15951:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.125@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2696.028717] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2737.213702] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2772.668494] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2803.679211] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 192.168.202.125@tcp) was lost; in progress operations using this service will fail
[ 2803.688409] LustreError: Skipped 2 previous similar messages
[ 2813.866272] Lustre: Evicted from MGS (at 192.168.202.125@tcp) after server handle changed from 0x2c22264a0ae68190 to 0x2c22264a0ae693b8
[ 2813.881023] Lustre: Skipped 2 previous similar messages
[ 2814.345987] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2839.672231] LustreError: lustre-MDT0000-mdc-ffff908bd920d800: operation ldlm_enqueue to node 192.168.202.125@tcp failed: rc = -19
[ 2839.696696] LustreError: Skipped 4 previous similar messages
[ 2839.711280] Lustre: lustre-MDT0000-mdc-ffff908bd920d800: Connection to lustre-MDT0000 (at 192.168.202.125@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2839.742050] Lustre: Skipped 6 previous similar messages
[ 2862.069894] Lustre: 15951:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.125@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2876.478927] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2879.027846] Lustre: lustre-MDT0000-mdc-ffff908bd920d800: Connection restored to 192.168.202.125@tcp (at 192.168.202.125@tcp)
[ 2879.040447] Lustre: Skipped 14 previous similar messages
[ 2927.216478] Lustre: 15951:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.125@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2945.212361] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 3008.059976] Lustre: 15951:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.125@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 3050.475240] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 17:59:14 (1787695154)
[ 3110.928647] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3113.522089] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3398.661678] LustreError: lustre-OST0000-osc-ffff908bd920d800: operation ldlm_enqueue to node 192.168.202.125@tcp failed: rc = -19
[ 3398.679520] LustreError: Skipped 6 previous similar messages
[ 3451.224285] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3453.882713] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3726.982613] Lustre: lustre-OST0000-osc-ffff908bd920d800: Connection to lustre-OST0000 (at 192.168.202.125@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3726.997294] Lustre: Skipped 4 previous similar messages
[ 3749.371598] Lustre: lustre-OST0000-osc-ffff908bd920d800: Connection restored to 192.168.202.125@tcp (at 192.168.202.125@tcp)
[ 3749.379751] Lustre: Skipped 4 previous similar messages
[ 3770.811821] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3773.149320] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4024.889273] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 18:15:28 (1787696128)
[ 4042.720430] Lustre: 50602:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787696132/real 1787696132]  req@ffff908be0161880 x1874531182327552/t0(0) o101->lustre-MDT0000-mdc-ffff908bd920d800@192.168.202.125@tcp:12/10 lens 576/1152 e 0 to 1 dl 1787696148 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 4042.748534] Lustre: 50602:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
[ 4052.771345] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 18:15:56 (1787696156)
[ 4082.089558] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 18:16:25 (1787696185)
[ 4108.432097] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 18:16:52 (1787696212)
[ 4109.070479] Lustre: Mounted lustre-client - version 2.17.57_80_g4b309f5
[ 4141.024415] Lustre: 2360:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787696230/real 1787696230]  req@ffff908bc8a49500 x1874531182346624/t0(0) o400->lustre-MDT0000-mdc-ffff908bd920d800@192.168.202.125@tcp:12/10 lens 224/224 e 0 to 1 dl 1787696246 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4141.030257] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 192.168.202.125@tcp) was lost; in progress operations using this service will fail
[ 4141.052613] Lustre: 2360:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 4141.093877] LustreError: Skipped 3 previous similar messages
[ 4141.132720] Lustre: Evicted from MGS (at 192.168.202.125@tcp) after server handle changed from 0x2c22264a0ae83415 to 0x2c22264a0afc85e7
[ 4141.159400] Lustre: Skipped 3 previous similar messages
[ 4157.837070] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4160.231560] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4162.670516] Lustre: Unmounted lustre-client
[ 4172.220173] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 18:17:55 (1787696275)
[ 4294.623166] Lustre: 2360:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787696384/real 1787696384]  req@ffff908be1b39880 x1874531182380160/t0(0) o4->lustre-OST0000-osc-ffff908bd920d800@192.168.202.125@tcp:6/4 lens 488/448 e 0 to 1 dl 1787696400 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4294.693541] Lustre: 2360:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 50 previous similar messages
[ 4327.391332] Lustre: lustre-OST0000-osc-ffff908bd920d800: Connection to lustre-OST0000 (at 192.168.202.125@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4327.422250] Lustre: Skipped 13 previous similar messages
[ 4359.174900] Lustre: lustre-OST0000-osc-ffff908bd920d800: Connection restored to 192.168.202.125@tcp (at 192.168.202.125@tcp)
[ 4359.184119] Lustre: Skipped 15 previous similar messages
[ 4472.939903] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 18:22:56 (1787696576)
[ 4524.214879] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 18:23:47 (1787696627)
[ 4528.987951] Lustre: Unmounted lustre-client
[ 4554.389758] Lustre: Mounted lustre-client - version 2.17.57_80_g4b309f5
[ 4562.529443] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 18:24:26 (1787696666)
[ 4562.925149] LustreError: 56535:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4563.981410] LustreError: 56535:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4564.248086] Lustre: *** cfs_fail_loc=305, val=0***
[ 4588.198588] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 18:24:52 (1787696692)
[ 4588.574353] Lustre: Mounted lustre-client - version 2.17.57_80_g4b309f5
[ 4590.438586] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4600.738699] Lustre: Unmounted lustre-client
[ 4613.226300] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 18:25:16 (1787696716)
[ 4704.693282] LustreError: lustre-MDT0000-mdc-ffff908bc8bd4800: operation mds_reint to node 192.168.202.125@tcp failed: rc = -107
[ 4704.707351] LustreError: Skipped 2 previous similar messages
[ 4722.655350] Lustre: 2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787696812/real 1787696812]  req@ffff908bc4a54e00 x1874531185044352/t0(0) o400->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 224/224 e 0 to 1 dl 1787696828 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4722.697587] Lustre: 2359:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 89 previous similar messages
[ 4722.706873] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 192.168.202.125@tcp) was lost; in progress operations using this service will fail
[ 4732.900823] Lustre: Evicted from MGS (at 192.168.202.125@tcp) after server handle changed from 0x2c22264a0afca0b3 to 0x2c22264a0aff4fbd
[ 4900.609820] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 18:30:03 (1787697003)
[ 4907.450435] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4931.551624] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 192.168.202.125@tcp) was lost; in progress operations using this service will fail
[ 4931.593705] Lustre: Evicted from MGS (at 192.168.202.125@tcp) after server handle changed from 0x2c22264a0aff4fbd to 0x2c22264a0b0425ba
[ 4931.599211] Lustre: lustre-MDT0000-mdc-ffff908bc8bd4800: Connection to lustre-MDT0000 (at 192.168.202.125@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4931.652480] Lustre: Skipped 11 previous similar messages
[ 4931.849437] LustreError: lustre-MDT0000-mdc-ffff908bc8bd4800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4950.080121] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 18:30:53 (1787697053)
[ 4950.832877] Lustre: Mounted lustre-client - version 2.17.57_80_g4b309f5
[ 4958.757235] LustreError: lustre-OST0000-osc-ffff908bc8bd4800: operation ldlm_enqueue to node 192.168.202.125@tcp failed: rc = -107
[ 4958.777679] LustreError: lustre-OST0000-osc-ffff908bc8bd4800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4969.490179] Lustre: lustre-OST0000-osc-ffff908bd8da1800: Connection restored to 192.168.202.125@tcp (at 192.168.202.125@tcp)
[ 4969.499492] Lustre: Skipped 12 previous similar messages
[ 4972.086248] Lustre: Unmounted lustre-client
[ 4980.153929] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 18:31:23 (1787697083)
[ 4986.619660] LustreError: lustre-MDT0000-mdc-ffff908bc8bd4800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4986.632848] LustreError: 60941:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4995.730517] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 18:31:39 (1787697099)
[ 4996.283600] LustreError: 61645:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 5001.695126] LustreError: 61645:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=0
[ 5001.707694] LustreError: 61645:0:(import.c:711:ptlrpc_connect_import_locked()) already connected
[ 5002.766887] LustreError: lustre-MDT0000-mdc-ffff908bc8bd4800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5002.806457] LustreError: 61662:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 5002.885481] LustreError: 61667:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5003.980885] LustreError: 61673:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5004.002015] LustreError: 61673:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5006.562972] LustreError: 61696:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5006.587336] LustreError: 61696:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5011.533383] LustreError: 61742:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5011.557986] LustreError: 61742:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 5022.026588] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 18:32:05 (1787697125)
[ 5068.471665] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5070.746678] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5082.887622] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 18:33:06 (1787697186)
[ 5125.943346] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5127.510035] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5137.386497] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 18:34:01 (1787697241)
[ 5204.679633] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5206.325614] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5215.802961] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 18:35:19 (1787697319)
[ 5260.612709] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5262.344703] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5267.864545] Lustre: Unmounted lustre-client
[ 5323.115590] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5325.955550] Lustre: Mounted lustre-client - version 2.17.57_80_g4b309f5
[ 5335.415363] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 18:37:18 (1787697438)
[ 5339.225860] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5341.671971] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 18:37:25 (1787697445)
[ 5383.207112] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 18:38:07 (1787697487)
[ 5384.961485] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5386.946862] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 18:38:10 (1787697490)
[ 5388.263268] Lustre: *** cfs_fail_loc=805, val=0***
[ 5388.297200] Lustre: Mounted lustre-client - version 2.17.57_80_g4b309f5
[ 5393.817580] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5413.856370] Lustre: 2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787697503/real 1787697503]  req@ffff908bc4a78380 x1874531187122816/t0(0) o400->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 224/224 e 0 to 1 dl 1787697519 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5413.892276] Lustre: 2359:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 5413.906379] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 192.168.202.125@tcp) was lost; in progress operations using this service will fail
[ 5424.169254] Lustre: Evicted from MGS (at 192.168.202.125@tcp) after server handle changed from 0x2c22264a0b042fca to 0x2c22264a0b0431e5
[ 5425.530621] Lustre: 68314:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.125@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 5439.541991] LustreError: 2356:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff908bd0042800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff908bc7bdd300) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5443.169322] Lustre: Unmounted lustre-client
[ 5450.321647] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 18:39:14 (1787697554)
[ 5477.423577] LustreError: 2356:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff908bd953fb80 x1874531187135616/t85899345924(85899345924) o101->lustre-MDT0000-mdc-ffff908bc6506800@192.168.202.125@tcp:12/10 lens 648/608 e 0 to 0 dl 1787697598 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5488.290846] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5490.225384] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5500.173517] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 18:40:03 (1787697603)
[ 5502.014038] LustreError: lustre-OST0000-osc-ffff908bc6506800: operation ost_write to node 192.168.202.125@tcp failed: rc = -107
[ 5502.026416] LustreError: Skipped 2 previous similar messages
[ 5502.042435] LustreError: lustre-OST0000-osc-ffff908bc6506800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5502.058376] Lustre: 2358:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.125@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5502.145759] LustreError: 72854:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff908bc6506800: namespace resource [0x240000400:0x3d42:0x0].0x0 (ffff908bd94a9900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5502.177585] LustreError: 72854:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5511.092309] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 18:40:15 (1787697615)
[ 5512.740438] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5514.423864] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 18:40:18 (1787697618)
[ 5516.726396] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5518.997562] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 18:40:22 (1787697622)
[ 5520.852238] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5523.582123] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 18:40:26 (1787697626)
[ 5525.182158] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5526.856449] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 18:40:31 (1787697631)
[ 5528.637876] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5531.017905] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 18:40:34 (1787697634)
[ 5532.477334] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5534.445591] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 18:40:38 (1787697638)
[ 5536.058508] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5538.056932] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 18:40:42 (1787697642)
[ 5539.506671] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5541.366610] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 18:40:45 (1787697645)
[ 5543.095107] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5544.730407] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 18:40:48 (1787697648)
[ 5546.619354] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5548.293839] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 18:40:52 (1787697652)
[ 5549.783249] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5551.734321] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 18:40:55 (1787697655)
[ 5554.382622] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5556.363195] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 18:41:00 (1787697660)
[ 5579.168807] Lustre: lustre-MDT0000-mdc-ffff908bc6506800: Connection to lustre-MDT0000 (at 192.168.202.125@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5579.197237] Lustre: Skipped 15 previous similar messages
[ 5579.213109] Lustre: MGC192.168.202.125@tcp: Connection restored to 192.168.202.125@tcp (at 192.168.202.125@tcp)
[ 5579.224722] Lustre: Skipped 13 previous similar messages
[ 5596.188728] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 18:41:39 (1787697699)
[ 5626.844961] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 18:42:10 (1787697730)
[ 5627.333577] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5637.399473] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 18:42:21 (1787697741)
[ 5637.744608] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5647.997259] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 18:42:32 (1787697752)
[ 5648.399915] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5658.479931] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 18:42:42 (1787697762)
[ 5658.928266] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5669.640760] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 18:42:53 (1787697773)
[ 5670.434229] Lustre: *** cfs_fail_loc=510, val=3***
[ 5682.847801] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 18:43:06 (1787697786)
[ 5683.536885] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5694.934573] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 18:43:18 (1787697798)
[ 5695.458879] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5761.749619] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 18:44:25 (1787697865)
[ 5762.233503] LustreError: 83065:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5765.345686] LustreError: lustre-MDT0000-mdc-ffff908bc6506800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5765.364641] LustreError: 83080:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff908bc6506800: inode [0x20000a042:0xb:0x0] mdc close failed: rc = -108
[ 5765.385503] LustreError: 83080:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff908bc6506800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff908bd94a9700) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5766.239216] LustreError: 83065:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5768.389633] LustreError: 83095:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5771.496436] LustreError: lustre-MDT0000-mdc-ffff908bc6506800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5771.529850] LustreError: 83110:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff908bc6506800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff908bc7320300) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5771.556638] LustreError: 83110:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5772.399121] LustreError: 83095:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5779.966740] LustreError: lustre-MDT0000-mdc-ffff908bc6506800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5780.413168] LustreError: 83151:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5783.697903] LustreError: lustre-MDT0000-mdc-ffff908bc6506800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5783.826731] LustreError: 83170:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5783.835883] LustreError: 83170:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 10 previous similar messages
[ 5784.507304] LustreError: 83151:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5784.524068] LustreError: 83151:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff908bc6506800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5784.972638] LustreError: 83176:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5784.988596] LustreError: 83176:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5787.702951] LustreError: 83199:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5787.713821] LustreError: 83199:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5790.240239] LustreError: 83214:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5793.646273] LustreError: lustre-MDT0000-mdc-ffff908bc6506800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5793.681814] LustreError: 83229:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff908bc6506800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff908bd94a9500) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5793.742893] LustreError: 83229:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5794.263105] LustreError: 83214:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5801.589495] LustreError: lustre-MDT0000-mdc-ffff908bc6506800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5802.062330] LustreError: 83270:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5805.064137] LustreError: lustre-MDT0000-mdc-ffff908bc6506800: operation ldlm_enqueue to node 192.168.202.125@tcp failed: rc = -107
[ 5805.086290] LustreError: Skipped 6 previous similar messages
[ 5805.107957] LustreError: lustre-MDT0000-mdc-ffff908bc6506800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5805.121027] LustreError: 83285:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff908bc6506800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff908bd94a9700) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5805.134655] LustreError: 83285:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5806.135189] LustreError: 83270:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5809.462668] LustreError: lustre-MDT0000-mdc-ffff908bc6506800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5809.540750] LustreError: 83317:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5810.376283] LustreError: 83298:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff908bc6506800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5818.637872] LustreError: 83369:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5818.647803] LustreError: 83369:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5822.655267] LustreError: 83369:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5822.663582] LustreError: 83369:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5823.179352] LustreError: lustre-MDT0000-mdc-ffff908bc6506800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5832.888622] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 18:45:36 (1787697936)
[ 5852.731043] LustreError: 79816:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5856.791579] LustreError: 79816:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5864.027069] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 18:46:07 (1787697967)
[ 5916.514218] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 18:47:00 (1787698020)
[ 5980.941586] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 18:48:04 (1787698084)
[ 6015.432809] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 18:48:39 (1787698119)
[ 6015.908291] Lustre: Mounted lustre-client - version 2.17.57_80_g4b309f5
[ 6140.188853] Lustre: Unmounted lustre-client
[ 6147.661875] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 18:50:52 (1787698252)
[ 6148.267191] LustreError: 87651:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 6151.008540] LustreError: lustre-OST0000-osc-ffff908bc6506800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6151.035373] LustreError: 87733:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff908bc6506800: namespace resource [0x240000400:0x3d6d:0x0].0x0 (ffff908bc7a2f200) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6151.060784] LustreError: 87733:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 2 previous similar messages
[ 6151.503201] LustreError: 87651:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout interrupted
[ 6151.510621] Lustre: 2360:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.125@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 6158.464450] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 18:51:02 (1787698262)
[ 6215.647211] Lustre: 88318:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787698266/real 1787698266]  req@ffff908bc8a4a300 x1874531187305728/t0(0) o101->lustre-MDT0000-mdc-ffff908bc6506800@192.168.202.125@tcp:12/10 lens 328/344 e 0 to 1 dl 1787698321 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 6215.696311] Lustre: 88318:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 15 previous similar messages
[ 6215.713698] Lustre: lustre-MDT0000-mdc-ffff908bc6506800: Connection to lustre-MDT0000 (at 192.168.202.125@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6215.723593] Lustre: Skipped 14 previous similar messages
[ 6215.739714] Lustre: lustre-MDT0000-mdc-ffff908bc6506800: Connection restored to 192.168.202.125@tcp (at 192.168.202.125@tcp)
[ 6215.755993] Lustre: Skipped 13 previous similar messages
[ 6224.860161] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 18:52:08 (1787698328)
[ 6226.269321] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 6228.593604] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 18:52:12 (1787698332)
[ 6294.555178] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6296.654394] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 18:53:20 (1787698400)
[ 6355.686160] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 18:54:19 (1787698459)
[ 6358.187790] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6359.648503] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 18:54:23 (1787698463)
[ 6360.944051] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6362.400342] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 18:54:26 (1787698466)
[ 6403.893889] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 18:55:07 (1787698507)
[ 6417.890187] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6446.051696] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 192.168.202.125@tcp) was lost; in progress operations using this service will fail
[ 6446.070909] LustreError: Skipped 2 previous similar messages
[ 6446.120564] Lustre: Evicted from MGS (at 192.168.202.125@tcp) after server handle changed from 0x2c22264a0b043d92 to 0x2c22264a0b04565f
[ 6446.141026] Lustre: Skipped 2 previous similar messages
[ 6461.892344] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6463.345300] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6473.937380] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 18:56:17 (1787698577)
[ 6476.194431] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6477.954903] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 18:56:22 (1787698582)
[ 6492.134941] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 192.168.202.125@tcp) was lost; in progress operations using this service will fail
[ 6492.171950] Lustre: Evicted from MGS (at 192.168.202.125@tcp) after server handle changed from 0x2c22264a0b04565f to 0x2c22264a0b045aaa
[ 6504.697656] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 18:56:48 (1787698608)
[ 6557.162842] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 18:57:41 (1787698661)
[ 6565.378768] LustreError: lustre-OST0000-osc-ffff908bc6506800: operation ost_setattr to node 192.168.202.125@tcp failed: rc = -107
[ 6565.390281] LustreError: Skipped 12 previous similar messages
[ 6617.751407] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6621.188324] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6708.191321] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 192.168.202.125@tcp) was lost; in progress operations using this service will fail
[ 6708.217322] LustreError: Skipped 1 previous similar message
[ 6718.501899] Lustre: Evicted from MGS (at 192.168.202.125@tcp) after server handle changed from 0x2c22264a0b045db3 to 0x2c22264a0b046ebc
[ 6718.516093] Lustre: Skipped 1 previous similar message
[ 6724.027933] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6725.679605] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6766.692784] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6768.341390] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6921.698869] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 19:03:45 (1787699025)
[ 6933.635259] Lustre: lustre-OST0000-osc-ffff908bc6506800: Connection to lustre-OST0000 (at 192.168.202.125@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6933.665685] Lustre: Skipped 8 previous similar messages
[ 6988.546417] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6992.621861] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 7241.365556] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 19:09:05 (1787699345)
[ 7394.276686] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 192.168.202.125@tcp) was lost; in progress operations using this service will fail
[ 7394.286681] LustreError: Skipped 1 previous similar message
[ 7394.307618] Lustre: Evicted from MGS (at 192.168.202.125@tcp) after server handle changed from 0x2c22264a0b04700c to 0x2c22264a0b13da99
[ 7394.321221] Lustre: Skipped 1 previous similar message
[ 7394.325268] Lustre: MGC192.168.202.125@tcp: Connection restored to 192.168.202.125@tcp (at 192.168.202.125@tcp)
[ 7394.346262] Lustre: Skipped 12 previous similar messages
[ 7423.967183] Lustre: 2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787699489/real 1787699489]  req@ffff908bc6184a80 x1874531204768768/t0(0) o400->lustre-MDT0000-mdc-ffff908bc6506800@192.168.202.125@tcp:12/10 lens 224/224 e 0 to 1 dl 1787699529 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7424.053285] Lustre: 2359:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 19 previous similar messages
[ 7435.068878] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 19:12:18 (1787699538)
[ 7437.510779] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7440.028412] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 19:12:23 (1787699543)
[ 7442.291918] LustreError: lustre-MDT0000-mdc-ffff908bc6506800: operation ldlm_enqueue to node 192.168.202.125@tcp failed: rc = -107
[ 7442.307952] LustreError: Skipped 306 previous similar messages
[ 7442.336628] LustreError: lustre-MDT0000-mdc-ffff908bc6506800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7451.203112] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 19:12:35 (1787699555)
[ 7619.640748] LustreError: lustre-OST0000-osc-ffff908bc6506800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7621.544310] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 19:15:25 (1787699725)
[ 7647.199906] Lustre: lustre-OST0000-osc-ffff908bc6506800: Connection to lustre-OST0000 (at 192.168.202.125@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7647.234189] Lustre: Skipped 3 previous similar messages
[ 7665.398792] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 19:16:09 (1787699769)
[ 7666.907120] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7668.462537] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 19:16:12 (1787699772)
[ 7670.091823] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7671.863324] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 19:16:15 (1787699775)
[ 7704.709891] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 19:16:48 (1787699808)
[ 7742.461297] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 192.168.202.125@tcp) was lost; in progress operations using this service will fail
[ 7742.490889] Lustre: Evicted from MGS (at 192.168.202.125@tcp) after server handle changed from 0x2c22264a0b13da99 to 0x2c22264a0b140820
[ 7742.498252] Lustre: Skipped 1 previous similar message
[ 7759.278393] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 19:17:43 (1787699863)
[ 7761.142811] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7762.611843] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 19:17:46 (1787699866)
[ 7764.137453] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7765.702520] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 19:17:49 (1787699869)
[ 7777.995640] Lustre: Unmounted lustre-client
[ 7780.349631] Lustre: Mounted lustre-client - version 2.17.57_80_g4b309f5
[ 7783.593623] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7808.378599] Lustre: 106784:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.125@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 7823.722577] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2