[ 2007.835279] LustreError: lustre-OST0000-osc-ffff88ee47fa2000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2007.857386] Lustre: 2375:0:(llite_lib.c:4153:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.112@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 2014.644733] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 15:22:14 (1777576934)
[ 2016.541259] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 2018.346502] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 15:22:18 (1777576938)
[ 2020.088798] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 2022.236703] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 15:22:22 (1777576942)
[ 2046.432125] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 192.168.202.112@tcp) was lost; in progress operations using this service will fail
[ 2046.470132] Lustre: Evicted from MGS (at 192.168.202.112@tcp) after server handle changed from 0x8879571a1b6e1a to 0x8879571a1b8589
[ 2048.183406] Lustre: 15869:0:(mgc_request.c:1897:mgc_process_log()) MGC192.168.202.112@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2158.742234] LustreError: lustre-MDT0000-mdc-ffff88ee47fa2000: operation mds_reint to node 192.168.202.112@tcp failed: rc = -19
[ 2158.774387] LustreError: Skipped 3 previous similar messages
[ 2179.423401] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 192.168.202.112@tcp) was lost; in progress operations using this service will fail
[ 2179.464599] Lustre: Evicted from MGS (at 192.168.202.112@tcp) after server handle changed from 0x8879571a1b8589 to 0x8879571a1d48a0
[ 2207.366463] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 15:25:27 (1777577127)
[ 2207.948058] Lustre: *** cfs_fail_loc=305, val=0***
[ 2207.954727] Lustre: Skipped 2 previous similar messages
[ 2227.990462] LustreError: lustre-OST0001-osc-ffff88ee47fa2000: operation ost_connect to node 192.168.202.112@tcp failed: rc = -75
[ 2250.207291] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 192.168.202.112@tcp) was lost; in progress operations using this service will fail
[ 2254.260129] Lustre: lustre-MDT0000-mdc-ffff88ee47fa2000: Connection to lustre-MDT0000 (at 192.168.202.112@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2254.269657] Lustre: Skipped 11 previous similar messages
[ 2254.413449] Lustre: lustre-MDT0000-mdc-ffff88ee47fa2000: Connection restored to 192.168.202.112@tcp (at 192.168.202.112@tcp)
[ 2254.427416] Lustre: Skipped 13 previous similar messages
[ 2259.513901] Lustre: Evicted from MGS (at 192.168.202.112@tcp) after server handle changed from 0x8879571a1d48a0 to 0x8879571a1d4fbc
[ 2266.017394] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2267.655655] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2276.521549] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 15:26:36 (1777577196)
[ 2290.150206] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 192.168.202.112@tcp) was lost; in progress operations using this service will fail
[ 2290.180930] Lustre: Evicted from MGS (at 192.168.202.112@tcp) after server handle changed from 0x8879571a1d4fbc to 0x8879571a1d526a
[ 2299.500901] LustreError: lustre-MDT0000-mdc-ffff88ee47fa2000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2314.965814] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 15:27:15 (1777577235)
[ 2328.265607] LustreError: lustre-OST0000-osc-ffff88ee47fa2000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2348.283930] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 15:27:48 (1777577268)
[ 2361.953146] LustreError: lustre-MDT0000-mdc-ffff88ee47fa2000: operation mds_reint to node 192.168.202.112@tcp failed: rc = -19
[ 2381.311904] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 192.168.202.112@tcp) was lost; in progress operations using this service will fail
[ 2381.336682] Lustre: Evicted from MGS (at 192.168.202.112@tcp) after server handle changed from 0x8879571a1d526a to 0x8879571a1d92d6
[ 2400.980230] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2403.966350] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2489.823531] Lustre: 2375:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777577395/real 1777577395]  req@ffff88ee48669c00 x1863922523564544/t0(0) o400->MGC192.168.202.112@tcp@192.168.202.112@tcp:26/25 lens 224/224 e 0 to 1 dl 1777577411 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2489.857824] Lustre: 2375:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 14 previous similar messages
[ 2489.863568] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 192.168.202.112@tcp) was lost; in progress operations using this service will fail
[ 2500.089366] Lustre: Evicted from MGS (at 192.168.202.112@tcp) after server handle changed from 0x8879571a1d92d6 to 0x8879571a1f4b82
[ 2515.950793] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2519.025929] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2601.449383] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 192.168.202.112@tcp) was lost; in progress operations using this service will fail
[ 2611.634945] Lustre: Evicted from MGS (at 192.168.202.112@tcp) after server handle changed from 0x8879571a1f4b82 to 0x8879571a20ccf2
[ 2628.005321] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2631.018730] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2662.173679] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 15:33:02 (1777577582)
[ 2667.581678] LustreError: lustre-MDT0000-mdc-ffff88ee47fa2000: operation ldlm_enqueue to node 192.168.202.112@tcp failed: rc = -19
[ 2667.589280] LustreError: Skipped 2 previous similar messages
[ 2687.455786] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 192.168.202.112@tcp) was lost; in progress operations using this service will fail
[ 2687.490603] Lustre: Evicted from MGS (at 192.168.202.112@tcp) after server handle changed from 0x8879571a20ccf2 to 0x8879571a218411
[ 2689.395625] Lustre: 15869:0:(mgc_request.c:1897:mgc_process_log()) MGC192.168.202.112@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2705.779697] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2738.797098] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2780.654794] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2825.142429] Lustre: Evicted from MGS (at 192.168.202.112@tcp) after server handle changed from 0x8879571a219c13 to 0x8879571a21fe60
[ 2825.161116] Lustre: Skipped 2 previous similar messages
[ 2831.969241] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2855.684631] Lustre: lustre-MDT0000-mdc-ffff88ee47fa2000: Connection to lustre-MDT0000 (at 192.168.202.112@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2855.695791] Lustre: Skipped 8 previous similar messages
[ 2875.359533] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 192.168.202.112@tcp) was lost; in progress operations using this service will fail
[ 2875.380512] LustreError: Skipped 3 previous similar messages
[ 2885.659173] Lustre: MGC192.168.202.112@tcp: Connection restored to 192.168.202.112@tcp (at 192.168.202.112@tcp)
[ 2885.669942] Lustre: Skipped 17 previous similar messages
[ 2896.830051] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2955.124127] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 3047.191357] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 15:39:27 (1777577967)
[ 3101.939459] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3104.820772] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3394.798938] LustreError: lustre-OST0000-osc-ffff88ee47fa2000: operation ldlm_enqueue to node 192.168.202.112@tcp failed: rc = -19
[ 3394.816462] LustreError: Skipped 8 previous similar messages
[ 3441.672293] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3444.920550] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3726.380709] Lustre: lustre-OST0000-osc-ffff88ee47fa2000: Connection to lustre-OST0000 (at 192.168.202.112@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3726.422251] Lustre: Skipped 4 previous similar messages
[ 3769.361589] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3772.032970] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4023.476192] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 15:55:43 (1777578943)
[ 4042.207310] Lustre: 50184:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777578947/real 1777578947]  req@ffff88ee48669180 x1863922541647232/t0(0) o101->lustre-MDT0000-mdc-ffff88ee47fa2000@192.168.202.112@tcp:12/10 lens 576/1152 e 0 to 1 dl 1777578963 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 4042.240835] Lustre: 50184:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
[ 4042.282596] Lustre: lustre-MDT0000-mdc-ffff88ee47fa2000: Connection restored to 192.168.202.112@tcp (at 192.168.202.112@tcp)
[ 4042.305331] Lustre: Skipped 6 previous similar messages
[ 4052.794280] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 15:56:12 (1777578972)
[ 4080.608282] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 15:56:40 (1777579000)
[ 4106.413336] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 15:57:06 (1777579026)
[ 4107.319172] Lustre: Mounted lustre-client
[ 4138.719531] Lustre: 2374:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777579044/real 1777579044]  req@ffff88ee486bb100 x1863922541665152/t0(0) o400->lustre-MDT0000-mdc-ffff88ee47fa2000@192.168.202.112@tcp:12/10 lens 224/224 e 0 to 1 dl 1777579060 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4138.744061] Lustre: 2374:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 4138.975233] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 192.168.202.112@tcp) was lost; in progress operations using this service will fail
[ 4138.991047] LustreError: Skipped 2 previous similar messages
[ 4149.260976] Lustre: Evicted from MGS (at 192.168.202.112@tcp) after server handle changed from 0x8879571a23ad81 to 0x8879571a346242
[ 4149.287958] Lustre: Skipped 3 previous similar messages
[ 4164.389856] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4166.569320] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4168.841345] LustreError: 52938:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff88ee47fd4800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 4168.846426] LustreError: 52938:0:(lov_obd.c:792:lov_cleanup()) Skipped 1 previous similar message
[ 4168.928202] Lustre: Unmounted lustre-client
[ 4176.628874] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 15:58:16 (1777579096)
[ 4299.551894] Lustre: 2375:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777579205/real 1777579205]  req@ffff88ee51253480 x1863922541699712/t0(0) o4->lustre-OST0000-osc-ffff88ee47fa2000@192.168.202.112@tcp:6/4 lens 488/448 e 0 to 1 dl 1777579221 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4299.596905] Lustre: 2375:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 44 previous similar messages
[ 4331.488791] Lustre: lustre-OST0000-osc-ffff88ee47fa2000: Connection to lustre-OST0000 (at 192.168.202.112@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4331.516922] Lustre: Skipped 13 previous similar messages
[ 4476.158386] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 16:03:16 (1777579396)
[ 4529.605649] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 16:04:09 (1777579449)
[ 4533.956142] LustreError: 55107:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff88ee47fa2000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 4533.973285] LustreError: 55107:0:(lov_obd.c:792:lov_cleanup()) Skipped 1 previous similar message
[ 4534.047325] Lustre: Unmounted lustre-client
[ 4561.879760] Lustre: Mounted lustre-client
[ 4569.671702] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 16:04:49 (1777579489)
[ 4570.184095] LustreError: 56091:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4571.239905] LustreError: 56091:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4571.515380] Lustre: *** cfs_fail_loc=305, val=0***
[ 4597.625338] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 16:05:17 (1777579517)
[ 4598.094172] Lustre: Mounted lustre-client
[ 4600.388542] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4610.727842] Lustre: Unmounted lustre-client
[ 4618.546973] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 16:05:38 (1777579538)
[ 4729.048394] LustreError: lustre-MDT0000-mdc-ffff88ee450b4800: operation mds_reint to node 192.168.202.112@tcp failed: rc = -19
[ 4729.058099] LustreError: Skipped 1 previous similar message
[ 4747.743624] Lustre: 2373:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777579653/real 1777579653]  req@ffff88ee4836ce00 x1863922544369024/t0(0) o400->MGC192.168.202.112@tcp@192.168.202.112@tcp:26/25 lens 224/224 e 0 to 1 dl 1777579669 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4747.765053] Lustre: 2373:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 77 previous similar messages
[ 4747.786342] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 192.168.202.112@tcp) was lost; in progress operations using this service will fail
[ 4758.018110] Lustre: Evicted from MGS (at 192.168.202.112@tcp) after server handle changed from 0x8879571a347ece to 0x8879571a372ee9
[ 4758.035343] Lustre: MGC192.168.202.112@tcp: Connection restored to 192.168.202.112@tcp (at 192.168.202.112@tcp)
[ 4758.047488] Lustre: Skipped 23 previous similar messages
[ 4969.536129] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 16:11:29 (1777579889)
[ 4976.286325] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4998.121182] Lustre: lustre-MDT0000-mdc-ffff88ee450b4800: Connection to lustre-MDT0000 (at 192.168.202.112@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4998.136629] Lustre: Skipped 11 previous similar messages
[ 4998.146339] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 192.168.202.112@tcp) was lost; in progress operations using this service will fail
[ 4998.183131] Lustre: Evicted from MGS (at 192.168.202.112@tcp) after server handle changed from 0x8879571a372ee9 to 0x8879571a3c015f
[ 4998.310885] LustreError: lustre-MDT0000-mdc-ffff88ee450b4800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5014.696496] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 16:12:14 (1777579934)
[ 5015.403329] Lustre: Mounted lustre-client
[ 5023.247632] LustreError: lustre-OST0000-osc-ffff88ee450b4800: operation ldlm_enqueue to node 192.168.202.112@tcp failed: rc = -107
[ 5023.276385] LustreError: lustre-OST0000-osc-ffff88ee450b4800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5023.292158] LustreError: 59810:0:(ldlm_resource.c:1172:ldlm_resource_complain()) lustre-OST0000-osc-ffff88ee450b4800: namespace resource [0x240000400:0x36c3:0x0].0x0 (ffff88ee49068100) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5035.365968] LustreError: 59840:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff88ee47fa1000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 5035.380125] LustreError: 59840:0:(lov_obd.c:792:lov_cleanup()) Skipped 1 previous similar message
[ 5035.475742] Lustre: Unmounted lustre-client
[ 5043.643396] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 16:12:43 (1777579963)
[ 5050.794534] LustreError: lustre-MDT0000-mdc-ffff88ee450b4800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5050.820464] LustreError: 60482:0:(file.c:6097:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 5059.841318] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 16:12:59 (1777579979)
[ 5060.065737] LustreError: 61180:0:(recover.c:330:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 5064.366762] LustreError: lustre-MDT0000-mdc-ffff88ee450b4800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5064.385526] LustreError: 61203:0:(import.c:291:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 5064.403779] LustreError: 61180:0:(recover.c:330:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=668
[ 5064.409402] LustreError: 61180:0:(import.c:716:ptlrpc_connect_import_locked()) already connecting
[ 5064.466349] LustreError: 61208:0:(file.c:6097:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5065.552084] LustreError: 61214:0:(file.c:6097:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5065.563733] LustreError: 61214:0:(file.c:6097:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5068.057260] LustreError: 61236:0:(file.c:6097:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5068.063703] LustreError: 61236:0:(file.c:6097:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5072.069129] LustreError: 61279:0:(file.c:6097:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5072.085122] LustreError: 61279:0:(file.c:6097:ll_inode_revalidate_fini()) Skipped 25 previous similar messages
[ 5083.738831] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 16:13:23 (1777580003)
[ 5127.823986] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5130.599395] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5144.941489] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 16:14:24 (1777580064)
[ 5188.966816] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5191.579554] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5204.500112] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 16:15:24 (1777580124)
[ 5274.741213] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5276.720264] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5288.530474] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 16:16:47 (1777580207)
[ 5331.524992] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5334.192457] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5340.875542] LustreError: 66859:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff88ee450b4800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 5340.887350] LustreError: 66859:0:(lov_obd.c:792:lov_cleanup()) Skipped 1 previous similar message
[ 5340.942086] Lustre: Unmounted lustre-client
[ 5399.864874] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5403.587297] Lustre: Mounted lustre-client
[ 5412.410534] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 16:18:52 (1777580332)
[ 5416.038965] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5418.483728] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 16:18:58 (1777580338)
[ 5460.093465] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 16:19:40 (1777580380)
[ 5462.158035] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5464.482555] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 16:19:44 (1777580384)
[ 5466.734634] Lustre: *** cfs_fail_loc=805, val=0***
[ 5466.824757] Lustre: Mounted lustre-client
[ 5474.478331] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5497.834458] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 192.168.202.112@tcp) was lost; in progress operations using this service will fail
[ 5497.861753] Lustre: Evicted from MGS (at 192.168.202.112@tcp) after server handle changed from 0x8879571a3c0b92 to 0x8879571a3c0dad
[ 5497.884949] Lustre: MGC192.168.202.112@tcp: Connection restored to 192.168.202.112@tcp (at 192.168.202.112@tcp)
[ 5497.893492] Lustre: Skipped 12 previous similar messages
[ 5498.847279] Lustre: 2373:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777580404/real 1777580404]  req@ffff88ee486f0a80 x1863922546440448/t0(0) o400->lustre-MDT0000-mdc-ffff88ee47fd4000@192.168.202.112@tcp:12/10 lens 224/224 e 0 to 1 dl 1777580420 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5498.899220] Lustre: 2373:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5499.913267] Lustre: 67695:0:(mgc_request.c:1897:mgc_process_log()) MGC192.168.202.112@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 5519.208066] LustreError: 70198:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff88ee47fd4000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 5519.219727] LustreError: 70198:0:(lov_obd.c:792:lov_cleanup()) Skipped 1 previous similar message
[ 5519.270729] Lustre: Unmounted lustre-client
[ 5527.965234] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 16:20:48 (1777580448)
[ 5556.262859] LustreError: 2372:0:(client.c:3465:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff88ee47df2d80 x1863922546452480/t90194313221(90194313221) o101->lustre-MDT0000-mdc-ffff88ee4ac4b800@192.168.202.112@tcp:12/10 lens 648/608 e 0 to 0 dl 1777580493 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5567.983956] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5569.587947] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5581.381691] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 16:21:41 (1777580501)
[ 5583.658591] LustreError: lustre-OST0000-osc-ffff88ee4ac4b800: operation ost_write to node 192.168.202.112@tcp failed: rc = -107
[ 5583.668955] LustreError: Skipped 2 previous similar messages
[ 5583.723559] LustreError: lustre-OST0000-osc-ffff88ee4ac4b800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5583.741437] Lustre: 2373:0:(llite_lib.c:4153:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.112@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5583.828454] LustreError: 72192:0:(ldlm_resource.c:1172:ldlm_resource_complain()) lustre-OST0000-osc-ffff88ee4ac4b800: namespace resource [0x240000400:0x3722:0x0].0x0 (ffff88ee4911b000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5583.858290] LustreError: 72192:0:(ldlm_resource.c:1172:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5596.138233] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 16:21:55 (1777580515)
[ 5598.469915] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5601.386879] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 16:22:00 (1777580520)
[ 5603.598680] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5605.968277] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 16:22:05 (1777580525)
[ 5609.030879] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5611.507931] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 16:22:11 (1777580531)
[ 5613.952692] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5616.369352] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 16:22:16 (1777580536)
[ 5618.221508] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5620.278681] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 16:22:20 (1777580540)
[ 5622.626123] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5624.778985] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 16:22:24 (1777580544)
[ 5626.583592] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5628.478705] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 16:22:28 (1777580548)
[ 5630.636241] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5632.902434] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 16:22:32 (1777580552)
[ 5634.771724] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5637.049490] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 16:22:37 (1777580557)
[ 5639.081498] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5641.023594] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 16:22:41 (1777580561)
[ 5642.958256] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5645.174763] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 16:22:45 (1777580565)
[ 5648.804723] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5651.231128] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 16:22:50 (1777580570)
[ 5673.439294] Lustre: lustre-MDT0000-mdc-ffff88ee4ac4b800: Connection to lustre-MDT0000 (at 192.168.202.112@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5673.464020] Lustre: Skipped 15 previous similar messages
[ 5690.549635] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 16:23:30 (1777580610)
[ 5722.487570] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 16:24:02 (1777580642)
[ 5723.194640] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5735.783637] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 16:24:15 (1777580655)
[ 5736.318395] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5747.481309] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 16:24:27 (1777580667)
[ 5747.770058] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5758.382762] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 16:24:38 (1777580678)
[ 5758.758685] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5769.391479] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 16:24:49 (1777580689)
[ 5769.914389] Lustre: *** cfs_fail_loc=510, val=3***
[ 5780.175821] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 16:25:00 (1777580700)
[ 5780.587189] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5790.999490] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 16:25:11 (1777580711)
[ 5791.543581] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5861.261343] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 16:26:20 (1777580780)
[ 5862.075088] LustreError: 82393:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5865.453808] LustreError: lustre-MDT0000-mdc-ffff88ee4ac4b800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5865.484736] LustreError: 82407:0:(ldlm_resource.c:1172:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff88ee4ac4b800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff88ee4911b600) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5866.177052] LustreError: 82393:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5868.371780] LustreError: 82423:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5871.871388] LustreError: lustre-MDT0000-mdc-ffff88ee4ac4b800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5871.916529] LustreError: 82437:0:(ldlm_resource.c:1172:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff88ee4ac4b800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff88ee49068500) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5871.949475] LustreError: 82437:0:(ldlm_resource.c:1172:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5872.378972] LustreError: 82423:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5879.760736] LustreError: lustre-MDT0000-mdc-ffff88ee4ac4b800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5880.414403] LustreError: 82479:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5883.793861] LustreError: lustre-MDT0000-mdc-ffff88ee4ac4b800: operation ldlm_enqueue to node 192.168.202.112@tcp failed: rc = -107
[ 5883.797470] LustreError: Skipped 3 previous similar messages
[ 5883.826969] LustreError: lustre-MDT0000-mdc-ffff88ee4ac4b800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5883.982930] LustreError: 82499:0:(file.c:6097:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5883.994431] LustreError: 82499:0:(file.c:6097:ll_inode_revalidate_fini()) Skipped 15 previous similar messages
[ 5884.415242] LustreError: 82479:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5884.426856] LustreError: 82479:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff88ee4ac4b800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5885.163879] LustreError: 82505:0:(file.c:6097:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5885.178316] LustreError: 82505:0:(file.c:6097:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5887.718824] LustreError: 82527:0:(file.c:6097:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5887.740797] LustreError: 82527:0:(file.c:6097:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5891.722281] LustreError: 82554:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5895.794924] LustreError: 82554:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5896.471058] LustreError: lustre-MDT0000-mdc-ffff88ee4ac4b800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5903.238287] LustreError: lustre-MDT0000-mdc-ffff88ee4ac4b800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5903.833518] LustreError: 82610:0:(ldlm_flock.c:856:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5906.846643] LustreError: lustre-MDT0000-mdc-ffff88ee4ac4b800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5906.861405] LustreError: 82625:0:(ldlm_resource.c:1172:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff88ee4ac4b800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff88ee49068c00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5906.870470] LustreError: 82625:0:(ldlm_resource.c:1172:ldlm_resource_complain()) Skipped 2 previous similar messages
[ 5907.912097] LustreError: 82610:0:(ldlm_flock.c:856:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5911.585308] LustreError: lustre-MDT0000-mdc-ffff88ee4ac4b800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5911.739440] LustreError: 82658:0:(file.c:6097:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5911.744856] LustreError: 82658:0:(file.c:6097:ll_inode_revalidate_fini()) Skipped 3 previous similar messages
[ 5912.215825] LustreError: 82638:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff88ee4ac4b800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5920.786823] LustreError: 82710:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5920.804303] LustreError: 82710:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5924.807151] LustreError: 82710:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5924.817723] LustreError: 82710:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5925.379223] LustreError: lustre-MDT0000-mdc-ffff88ee4ac4b800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5934.266371] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 16:27:33 (1777580853)
[ 5953.479687] LustreError: 79141:0:(ldlm_lockd.c:2883:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5957.559261] LustreError: 79141:0:(ldlm_lockd.c:2883:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5964.375556] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 16:28:04 (1777580884)
[ 6017.283289] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 16:28:56 (1777580936)
[ 6082.260937] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 16:30:02 (1777581002)
[ 6119.507911] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 16:30:39 (1777581039)
[ 6120.037440] Lustre: Mounted lustre-client
[ 6244.746533] LustreError: 86381:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff88ee47fd7000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 6244.769452] LustreError: 86381:0:(lov_obd.c:792:lov_cleanup()) Skipped 1 previous similar message
[ 6244.853079] Lustre: Unmounted lustre-client
[ 6254.047212] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 16:32:53 (1777581173)
[ 6254.882079] LustreError: 86995:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 6258.201950] LustreError: lustre-OST0000-osc-ffff88ee4ac4b800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6258.232641] LustreError: 87077:0:(ldlm_resource.c:1172:ldlm_resource_complain()) lustre-OST0000-osc-ffff88ee4ac4b800: namespace resource [0x240000400:0x374d:0x0].0x0 (ffff88ee49068e00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6258.276079] LustreError: 87077:0:(ldlm_resource.c:1172:ldlm_resource_complain()) Skipped 1 previous similar message
[ 6258.655182] LustreError: 86995:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout interrupted
[ 6258.663848] Lustre: 2375:0:(llite_lib.c:4153:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.112@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 6258.699207] Lustre: lustre-OST0000-osc-ffff88ee4ac4b800: Connection restored to 192.168.202.112@tcp (at 192.168.202.112@tcp)
[ 6258.713639] Lustre: Skipped 18 previous similar messages
[ 6266.002104] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 16:33:05 (1777581185)
[ 6324.191242] Lustre: 87663:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777581190/real 1777581190]  req@ffff88ee4866b800 x1863922546620416/t0(0) o101->lustre-MDT0000-mdc-ffff88ee4ac4b800@192.168.202.112@tcp:12/10 lens 328/344 e 0 to 1 dl 1777581245 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 6324.229559] Lustre: 87663:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 20 previous similar messages
[ 6324.241136] Lustre: lustre-MDT0000-mdc-ffff88ee4ac4b800: Connection to lustre-MDT0000 (at 192.168.202.112@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6324.272287] Lustre: Skipped 14 previous similar messages
[ 6334.778925] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 16:34:14 (1777581254)
[ 6337.116262] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 6339.747747] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 16:34:19 (1777581259)
[ 6406.365106] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6409.325532] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 16:35:28 (1777581328)
[ 6468.660302] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 16:36:28 (1777581388)
[ 6472.258824] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6474.389605] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 16:36:34 (1777581394)
[ 6476.390102] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6478.767481] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 16:36:38 (1777581398)
[ 6520.583437] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 16:37:20 (1777581440)
[ 6536.283662] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6564.834053] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 192.168.202.112@tcp) was lost; in progress operations using this service will fail
[ 6564.861567] LustreError: Skipped 2 previous similar messages
[ 6575.162464] Lustre: Evicted from MGS (at 192.168.202.112@tcp) after server handle changed from 0x8879571a3c19df to 0x8879571a3c3220
[ 6575.182580] Lustre: Skipped 2 previous similar messages
[ 6594.614993] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6596.598854] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6610.187415] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 16:38:49 (1777581529)
[ 6612.867750] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6614.818628] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 16:38:54 (1777581534)
[ 6627.814044] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 192.168.202.112@tcp) was lost; in progress operations using this service will fail
[ 6627.859555] Lustre: Evicted from MGS (at 192.168.202.112@tcp) after server handle changed from 0x8879571a3c3220 to 0x8879571a3c35c3
[ 6644.384784] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 16:39:24 (1777581564)
[ 6675.377934] Lustre: 67695:0:(mgc_request.c:1897:mgc_process_log()) MGC192.168.202.112@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6698.227925] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 16:40:18 (1777581618)
[ 6706.849395] LustreError: lustre-OST0000-osc-ffff88ee4ac4b800: operation ost_setattr to node 192.168.202.112@tcp failed: rc = -107
[ 6706.859229] LustreError: Skipped 5 previous similar messages
[ 6761.692711] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 6765.843332] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6857.199061] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 192.168.202.112@tcp) was lost; in progress operations using this service will fail
[ 6857.223448] LustreError: Skipped 1 previous similar message
[ 6857.244761] Lustre: Evicted from MGS (at 192.168.202.112@tcp) after server handle changed from 0x8879571a3c38e1 to 0x8879571a3c48af
[ 6857.255511] Lustre: Skipped 1 previous similar message
[ 6869.455204] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6871.102297] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6904.308169] Lustre: MGC192.168.202.112@tcp: Connection restored to 192.168.202.112@tcp (at 192.168.202.112@tcp)
[ 6904.327615] Lustre: Skipped 11 previous similar messages
[ 6916.890916] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6918.495507] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6938.079252] Lustre: 2376:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777581799/real 1777581799]  req@ffff88ee7075dc00 x1863922553157504/t0(0) o400->lustre-MDT0000-mdc-ffff88ee4ac4b800@192.168.202.112@tcp:12/10 lens 224/224 e 0 to 1 dl 1777581859 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6938.112639] Lustre: 2376:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 16 previous similar messages
[ 7074.582490] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 16:46:34 (1777581994)
[ 7088.408628] Lustre: lustre-OST0000-osc-ffff88ee4ac4b800: Connection to lustre-OST0000 (at 192.168.202.112@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7088.435986] Lustre: Skipped 8 previous similar messages
[ 7146.543225] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 7152.916161] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 7437.909460] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 16:52:38 (1777582358)
[ 7646.687312] Lustre: 2373:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777582552/real 1777582552]  req@ffff88ee4a51a680 x1863922564324352/t0(0) o400->MGC192.168.202.112@tcp@192.168.202.112@tcp:26/25 lens 224/224 e 0 to 1 dl 1777582568 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7646.712221] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 192.168.202.112@tcp) was lost; in progress operations using this service will fail
[ 7646.735792] Lustre: 2373:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 7646.749166] LustreError: Skipped 1 previous similar message
[ 7646.758871] Lustre: Evicted from MGS (at 192.168.202.112@tcp) after server handle changed from 0x8879571a3c4cbb to 0x8879571a4bef41
[ 7646.783911] Lustre: Skipped 1 previous similar message
[ 7646.798482] Lustre: MGC192.168.202.112@tcp: Connection restored to 192.168.202.112@tcp (at 192.168.202.112@tcp)
[ 7646.817496] Lustre: Skipped 1 previous similar message
[ 7683.534031] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 16:56:43 (1777582603)
[ 7685.512782] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7687.299949] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 16:56:47 (1777582607)
[ 7689.914148] LustreError: lustre-MDT0000-mdc-ffff88ee4ac4b800: operation ldlm_enqueue to node 192.168.202.112@tcp failed: rc = -107
[ 7689.922384] LustreError: Skipped 130 previous similar messages
[ 7689.928246] Lustre: lustre-MDT0000-mdc-ffff88ee4ac4b800: Connection to lustre-MDT0000 (at 192.168.202.112@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7689.941067] Lustre: Skipped 1 previous similar message
[ 7689.952362] LustreError: lustre-MDT0000-mdc-ffff88ee4ac4b800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7699.731636] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 16:56:59 (1777582619)
[ 7869.582142] Lustre: Evicted from lustre-OST0000_UUID (at 192.168.202.112@tcp) after server handle changed from 0x8879571a4bf346 to 0x8879571a4bf450
[ 7869.614736] LustreError: lustre-OST0000-osc-ffff88ee4ac4b800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7872.107473] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 16:59:51 (1777582791)
[ 7920.563510] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 17:00:40 (1777582840)
[ 7922.670826] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7924.762728] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 17:00:44 (1777582844)
[ 7926.693950] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7928.365385] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 17:00:48 (1777582848)
[ 7967.733183] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 17:01:27 (1777582887)
[ 8008.173158] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 192.168.202.112@tcp) was lost; in progress operations using this service will fail
[ 8008.233700] Lustre: Evicted from MGS (at 192.168.202.112@tcp) after server handle changed from 0x8879571a4bef41 to 0x8879571a4c1718
[ 8029.720023] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 17:02:29 (1777582949)
[ 8032.060350] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 8034.194045] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 17:02:34 (1777582954)
[ 8036.295594] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 8038.384282] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 17:02:38 (1777582958)
[ 8056.216196] LustreError: 105940:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff88ee4ac4b800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 8056.224255] LustreError: 105940:0:(lov_obd.c:792:lov_cleanup()) Skipped 1 previous similar message
[ 8056.312773] Lustre: Unmounted lustre-client
[ 8058.401346] Lustre: Mounted lustre-client
[ 8061.610887] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 8084.959735] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 192.168.202.112@tcp) was lost; in progress operations using this service will fail
[ 8095.529599] Lustre: Evicted from MGS (at 192.168.202.112@tcp) after server handle changed from 0x8879571a4c2095 to 0x8879571a4c2263
[ 8104.753280] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2