[ 1819.403228] LustreError: lustre-OST0000-osc-ffff9e0018822800: operation ost_sync to node 192.168.203.145@tcp failed: rc = -107
[ 1819.422431] LustreError: lustre-OST0000-osc-ffff9e0018822800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1819.441821] Lustre: 2348:0:(llite_lib.c:4339:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.145@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1826.230840] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 01:17:39 (1787116659)
[ 1828.408592] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1830.211746] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 01:17:42 (1787116662)
[ 1832.056639] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1833.944667] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 01:17:46 (1787116666)
[ 1837.828514] LustreError: lustre-MDT0000-mdc-ffff9e0018822800: operation mds_reint to node 192.168.203.145@tcp failed: rc = -19
[ 1858.545464] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 192.168.203.145@tcp) was lost; in progress operations using this service will fail
[ 1858.582602] Lustre: Evicted from MGS (at 192.168.203.145@tcp) after server handle changed from 0xb35bf54c637a7b43 to 0xb35bf54c637a93bc
[ 1860.201733] Lustre: 15893:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.145@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 1966.843215] LustreError: lustre-MDT0000-mdc-ffff9e0018822800: operation mds_reint to node 192.168.203.145@tcp failed: rc = -19
[ 1966.858641] LustreError: Skipped 1 previous similar message
[ 1985.546180] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 192.168.203.145@tcp) was lost; in progress operations using this service will fail
[ 1985.582263] Lustre: Evicted from MGS (at 192.168.203.145@tcp) after server handle changed from 0xb35bf54c637a93bc to 0xb35bf54c637c3ec3
[ 1998.151926] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 01:20:31 (1787116831)
[ 1998.529815] Lustre: *** cfs_fail_loc=305, val=0***
[ 1998.534456] Lustre: Skipped 2 previous similar messages
[ 2037.727766] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 192.168.203.145@tcp) was lost; in progress operations using this service will fail
[ 2047.029441] Lustre: Evicted from MGS (at 192.168.203.145@tcp) after server handle changed from 0xb35bf54c637c3ec3 to 0xb35bf54c637c6a1a
[ 2052.336963] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2053.609805] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2061.766945] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 01:21:34 (1787116894)
[ 2077.680824] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 192.168.203.145@tcp) was lost; in progress operations using this service will fail
[ 2077.705740] Lustre: Evicted from MGS (at 192.168.203.145@tcp) after server handle changed from 0xb35bf54c637c6a1a to 0xb35bf54c637c6cba
[ 2077.760980] LustreError: lustre-MDT0000-mdc-ffff9e0018822800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2103.813156] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 01:22:16 (1787116936)
[ 2136.050995] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 01:22:48 (1787116968)
[ 2152.487104] LustreError: lustre-MDT0000-mdc-ffff9e0018822800: operation mds_reint to node 192.168.203.145@tcp failed: rc = -19
[ 2152.515634] Lustre: lustre-MDT0000-mdc-ffff9e0018822800: Connection to lustre-MDT0000 (at 192.168.203.145@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2152.541801] Lustre: Skipped 13 previous similar messages
[ 2170.849873] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 192.168.203.145@tcp) was lost; in progress operations using this service will fail
[ 2181.106972] Lustre: Evicted from MGS (at 192.168.203.145@tcp) after server handle changed from 0xb35bf54c637c6cba to 0xb35bf54c637cac7e
[ 2181.136791] Lustre: MGC192.168.203.145@tcp: Connection restored to 192.168.203.145@tcp (at 192.168.203.145@tcp)
[ 2181.152482] Lustre: Skipped 16 previous similar messages
[ 2196.973666] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2199.103812] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2282.018104] Lustre: 2350:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787117100/real 1787117100]  req@ffff9e001895b480 x1873925777237120/t0(0) o400->MGC192.168.203.145@tcp@192.168.203.145@tcp:26/25 lens 224/224 e 0 to 1 dl 1787117116 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2282.082908] Lustre: 2350:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
[ 2282.105922] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 192.168.203.145@tcp) was lost; in progress operations using this service will fail
[ 2292.357691] Lustre: Evicted from MGS (at 192.168.203.145@tcp) after server handle changed from 0xb35bf54c637cac7e to 0xb35bf54c637e7cd1
[ 2309.674898] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2312.028492] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2376.946923] LustreError: lustre-MDT0000-mdc-ffff9e0018822800: operation ldlm_enqueue to node 192.168.203.145@tcp failed: rc = -19
[ 2376.958460] LustreError: Skipped 1 previous similar message
[ 2394.079430] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 192.168.203.145@tcp) was lost; in progress operations using this service will fail
[ 2404.273281] Lustre: Evicted from MGS (at 192.168.203.145@tcp) after server handle changed from 0xb35bf54c637e7cd1 to 0xb35bf54c63803d02
[ 2420.450232] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2423.659438] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2451.992417] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 01:28:05 (1787117285)
[ 2474.975547] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 192.168.203.145@tcp) was lost; in progress operations using this service will fail
[ 2485.015731] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2485.188553] Lustre: Evicted from MGS (at 192.168.203.145@tcp) after server handle changed from 0xb35bf54c63803d02 to 0xb35bf54c6380fff1
[ 2508.464800] Lustre: 15893:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.145@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2524.438471] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2560.514575] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2596.529091] Lustre: 15893:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.145@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2610.297228] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2633.272593] LustreError: lustre-MDT0000-mdc-ffff9e0018822800: operation mds_reint to node 192.168.203.145@tcp failed: rc = -19
[ 2633.287673] LustreError: Skipped 4 previous similar messages
[ 2651.632628] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 192.168.203.145@tcp) was lost; in progress operations using this service will fail
[ 2651.647970] LustreError: Skipped 3 previous similar messages
[ 2651.680666] Lustre: Evicted from MGS (at 192.168.203.145@tcp) after server handle changed from 0xb35bf54c638161ff to 0xb35bf54c6381cbfb
[ 2651.690844] Lustre: Skipped 3 previous similar messages
[ 2661.103504] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2727.910036] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2761.105457] Lustre: lustre-MDT0000-mdc-ffff9e0018822800: Connection to lustre-MDT0000 (at 192.168.203.145@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2761.135970] Lustre: Skipped 7 previous similar messages
[ 2789.829877] Lustre: MGC192.168.203.145@tcp: Connection restored to 192.168.203.145@tcp (at 192.168.203.145@tcp)
[ 2789.838320] Lustre: Skipped 16 previous similar messages
[ 2815.170776] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 01:34:07 (1787117647)
[ 2873.688082] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 2875.262098] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3160.193349] LustreError: lustre-OST0000-osc-ffff9e0018822800: operation ldlm_enqueue to node 192.168.203.145@tcp failed: rc = -19
[ 3160.203117] LustreError: Skipped 4 previous similar messages
[ 3199.274615] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3201.439853] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3490.969025] Lustre: lustre-OST0000-osc-ffff9e0018822800: Connection to lustre-OST0000 (at 192.168.203.145@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3490.986855] Lustre: Skipped 2 previous similar messages
[ 3533.607160] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3536.571819] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3786.403688] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 01:50:18 (1787118618)
[ 3803.104285] Lustre: 50404:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787118621/real 1787118621]  req@ffff9e0004855500 x1873925798433792/t0(0) o101->lustre-MDT0000-mdc-ffff9e0018822800@192.168.203.145@tcp:12/10 lens 576/1152 e 0 to 1 dl 1787118637 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3803.144487] Lustre: 50404:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 3803.209503] Lustre: lustre-MDT0000-mdc-ffff9e0018822800: Connection restored to 192.168.203.145@tcp (at 192.168.203.145@tcp)
[ 3803.221797] Lustre: Skipped 1 previous similar message
[ 3811.631609] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 01:50:44 (1787118644)
[ 3838.712579] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 01:51:11 (1787118671)
[ 3864.809813] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 01:51:37 (1787118697)
[ 3865.301817] Lustre: Mounted lustre-client
[ 3897.311258] Lustre: 2347:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787118715/real 1787118715]  req@ffff9e0008254700 x1873925798452352/t0(0) o400->MGC192.168.203.145@tcp@192.168.203.145@tcp:26/25 lens 224/224 e 0 to 1 dl 1787118731 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3897.354784] Lustre: 2347:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 3897.366712] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 192.168.203.145@tcp) was lost; in progress operations using this service will fail
[ 3897.393105] LustreError: Skipped 2 previous similar messages
[ 3897.422640] Lustre: Evicted from MGS (at 192.168.203.145@tcp) after server handle changed from 0xb35bf54c63836256 to 0xb35bf54c63974498
[ 3897.436666] Lustre: Skipped 2 previous similar messages
[ 3912.215235] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 3913.814924] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3915.751667] Lustre: Unmounted lustre-client
[ 3922.529781] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 01:52:35 (1787118755)
[ 4061.151130] Lustre: 2350:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787118879/real 1787118879]  req@ffff9e0004855180 x1873925798485760/t0(0) o4->lustre-OST0000-osc-ffff9e0018822800@192.168.203.145@tcp:6/4 lens 488/448 e 0 to 1 dl 1787118895 ref 2 fl Rpc:XQr/602/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4061.208091] Lustre: 2350:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 65 previous similar messages
[ 4092.895264] Lustre: lustre-OST0000-osc-ffff9e0018822800: Connection to lustre-OST0000 (at 192.168.203.145@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4092.914916] Lustre: Skipped 14 previous similar messages
[ 4221.694048] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 01:57:34 (1787119054)
[ 4271.127348] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 01:58:23 (1787119103)
[ 4274.394974] Lustre: Unmounted lustre-client
[ 4298.360140] Lustre: Mounted lustre-client
[ 4304.470542] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 01:58:57 (1787119137)
[ 4304.820964] LustreError: 56312:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4305.880185] LustreError: 56312:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4306.170956] Lustre: *** cfs_fail_loc=305, val=0***
[ 4329.348975] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 01:59:21 (1787119161)
[ 4329.793946] Lustre: Mounted lustre-client
[ 4332.989266] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4343.327533] Lustre: Unmounted lustre-client
[ 4350.081944] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 01:59:43 (1787119183)
[ 4440.330844] LustreError: lustre-MDT0000-mdc-ffff9e00102bc800: operation mds_reint to node 192.168.203.145@tcp failed: rc = -19
[ 4440.349087] LustreError: Skipped 1 previous similar message
[ 4457.959631] Lustre: 2350:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787119276/real 1787119276]  req@ffff9e0005b3df80 x1873925801163648/t0(0) o400->MGC192.168.203.145@tcp@192.168.203.145@tcp:26/25 lens 224/224 e 0 to 1 dl 1787119292 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4458.002633] Lustre: 2350:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 75 previous similar messages
[ 4458.021309] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 192.168.203.145@tcp) was lost; in progress operations using this service will fail
[ 4468.200084] Lustre: Evicted from MGS (at 192.168.203.145@tcp) after server handle changed from 0xb35bf54c6397657d to 0xb35bf54c639a1c3d
[ 4468.230527] Lustre: MGC192.168.203.145@tcp: Connection restored to 192.168.203.145@tcp (at 192.168.203.145@tcp)
[ 4468.252455] Lustre: Skipped 23 previous similar messages
[ 4633.580546] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 02:04:26 (1787119466)
[ 4638.918870] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4651.508938] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 192.168.203.145@tcp) was lost; in progress operations using this service will fail
[ 4651.537759] Lustre: Evicted from MGS (at 192.168.203.145@tcp) after server handle changed from 0xb35bf54c639a1c3d to 0xb35bf54c639ee226
[ 4652.856487] LustreError: lustre-MDT0000-mdc-ffff9e00102bc800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4668.874479] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 02:05:01 (1787119501)
[ 4669.485386] Lustre: Mounted lustre-client
[ 4676.847244] LustreError: lustre-OST0000-osc-ffff9e00102bc800: operation ldlm_enqueue to node 192.168.203.145@tcp failed: rc = -107
[ 4676.872982] LustreError: lustre-OST0000-osc-ffff9e00102bc800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4689.844144] Lustre: Unmounted lustre-client
[ 4697.456260] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 02:05:30 (1787119530)
[ 4701.042900] Lustre: lustre-MDT0000-mdc-ffff9e00102bc800: Connection to lustre-MDT0000 (at 192.168.203.145@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4701.054162] Lustre: Skipped 14 previous similar messages
[ 4704.116932] LustreError: lustre-MDT0000-mdc-ffff9e00102bc800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4704.152218] LustreError: 60697:0:(file.c:6230:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4711.895915] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 02:05:44 (1787119544)
[ 4712.309629] LustreError: 61403:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4717.444274] LustreError: lustre-MDT0000-mdc-ffff9e00102bc800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4717.471069] LustreError: 61419:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4717.483171] LustreError: 61403:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=1
[ 4717.493614] LustreError: 61403:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting
[ 4717.544982] LustreError: 61424:0:(file.c:6230:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4718.666975] LustreError: 61431:0:(file.c:6230:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4718.682410] LustreError: 61431:0:(file.c:6230:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4720.958294] LustreError: 61453:0:(file.c:6230:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4720.966512] LustreError: 61453:0:(file.c:6230:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4725.497903] LustreError: 61498:0:(file.c:6230:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4725.522136] LustreError: 61498:0:(file.c:6230:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4736.708521] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 02:06:09 (1787119569)
[ 4777.842577] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4779.460082] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4789.659983] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 02:07:02 (1787119622)
[ 4827.807376] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4829.469784] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4840.261248] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 02:07:52 (1787119672)
[ 4902.378617] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4903.713430] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4910.672060] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 02:09:03 (1787119743)
[ 4947.748444] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4949.123634] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4955.819369] Lustre: Unmounted lustre-client
[ 5003.660249] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5006.158995] Lustre: Mounted lustre-client
[ 5012.589236] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 02:10:45 (1787119845)
[ 5015.082811] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5016.871593] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 02:10:49 (1787119849)
[ 5045.453343] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 02:11:18 (1787119878)
[ 5047.050777] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5048.573150] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 02:11:21 (1787119881)
[ 5050.156427] Lustre: *** cfs_fail_loc=805, val=0***
[ 5050.204276] Lustre: Mounted lustre-client
[ 5055.571925] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5075.935274] Lustre: 2349:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787119894/real 1787119894]  req@ffff9e00068a1880 x1873925803214592/t0(0) o400->lustre-MDT0000-mdc-ffff9e0018824800@192.168.203.145@tcp:12/10 lens 224/224 e 0 to 1 dl 1787119910 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5075.942434] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 192.168.203.145@tcp) was lost; in progress operations using this service will fail
[ 5075.980084] Lustre: 2349:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5076.004880] Lustre: Evicted from MGS (at 192.168.203.145@tcp) after server handle changed from 0xb35bf54c639eee82 to 0xb35bf54c639ef09d
[ 5076.032601] Lustre: MGC192.168.203.145@tcp: Connection restored to 192.168.203.145@tcp (at 192.168.203.145@tcp)
[ 5076.045917] Lustre: Skipped 13 previous similar messages
[ 5077.807455] Lustre: 68009:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.145@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 5095.574290] Lustre: Unmounted lustre-client
[ 5103.186649] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 02:12:16 (1787119936)
[ 5130.713128] LustreError: 2346:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff9e00068a1500 x1873925803227008/t90194313220(90194313220) o101->lustre-MDT0000-mdc-ffff9e0018824800@192.168.203.145@tcp:12/10 lens 664/608 e 0 to 0 dl 1787119980 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5138.833666] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5140.535784] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5148.530429] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 02:13:01 (1787119981)
[ 5149.965977] LustreError: lustre-OST0000-osc-ffff9e0018824800: operation ost_write to node 192.168.203.145@tcp failed: rc = -107
[ 5149.973677] LustreError: Skipped 2 previous similar messages
[ 5149.989278] LustreError: lustre-OST0000-osc-ffff9e0018824800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5150.007728] Lustre: 2349:0:(llite_lib.c:4339:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.145@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5158.375633] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 02:13:11 (1787119991)
[ 5159.958982] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5161.791468] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 02:13:14 (1787119994)
[ 5163.417928] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5165.142902] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 02:13:17 (1787119997)
[ 5166.792976] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5168.809308] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 02:13:21 (1787120001)
[ 5170.499291] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5172.387598] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 02:13:24 (1787120004)
[ 5174.084501] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5176.053425] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 02:13:28 (1787120008)
[ 5177.689444] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5179.578996] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 02:13:32 (1787120012)
[ 5181.617559] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5183.667938] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 02:13:36 (1787120016)
[ 5185.345216] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5186.952695] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 02:13:39 (1787120019)
[ 5188.651939] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5190.409100] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 02:13:43 (1787120023)
[ 5191.935163] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5193.785123] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 02:13:46 (1787120026)
[ 5195.496185] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5197.254735] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 02:13:49 (1787120029)
[ 5200.161090] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5202.053861] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 02:13:54 (1787120034)
[ 5236.450972] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 02:14:29 (1787120069)
[ 5266.599391] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 02:14:59 (1787120099)
[ 5267.163617] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5277.682868] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 02:15:10 (1787120110)
[ 5277.921487] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5289.376280] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 02:15:21 (1787120121)
[ 5289.723626] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5299.639898] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 02:15:32 (1787120132)
[ 5299.940029] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5308.896750] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 02:15:41 (1787120141)
[ 5309.228536] Lustre: *** cfs_fail_loc=510, val=3***
[ 5317.380519] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 02:15:50 (1787120150)
[ 5317.756655] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5327.696348] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 02:16:00 (1787120160)
[ 5328.167422] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5387.615242] Lustre: lustre-OST0000-osc-ffff9e0018824800: Connection to lustre-OST0000 (at 192.168.203.145@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5387.635100] Lustre: Skipped 13 previous similar messages
[ 5393.971216] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 02:17:06 (1787120226)
[ 5394.328893] LustreError: 82718:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5397.327422] LustreError: lustre-MDT0000-mdc-ffff9e0018824800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5397.352830] LustreError: 82733:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9e0018824800: inode [0x20000a042:0x8:0x0] mdc close failed: rc = -108
[ 5397.397505] LustreError: 82733:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9e0018824800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9e0004860c00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5398.399197] LustreError: 82718:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5400.529950] LustreError: 82748:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5403.423717] LustreError: lustre-MDT0000-mdc-ffff9e0018824800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5403.454807] LustreError: 82762:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9e0018824800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9e000832d700) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5403.479359] LustreError: 82762:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5404.624451] LustreError: 82748:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5411.697232] LustreError: lustre-MDT0000-mdc-ffff9e0018824800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5412.172160] LustreError: 82804:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5415.044663] LustreError: lustre-MDT0000-mdc-ffff9e0018824800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5415.102682] LustreError: 82823:0:(file.c:6230:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5415.116614] LustreError: 82823:0:(file.c:6230:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5416.238858] LustreError: 82830:0:(file.c:6230:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5416.248697] LustreError: 82830:0:(file.c:6230:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5416.259672] LustreError: 82804:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5416.266622] LustreError: 82804:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9e0018824800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5418.539796] LustreError: 82852:0:(file.c:6230:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5418.558853] LustreError: 82852:0:(file.c:6230:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5422.200546] LustreError: 82879:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5424.956804] LustreError: lustre-MDT0000-mdc-ffff9e0018824800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5424.978984] LustreError: 82893:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9e0018824800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9e000832d000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5425.006135] LustreError: 82893:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5426.280064] LustreError: 82879:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5433.264533] LustreError: lustre-MDT0000-mdc-ffff9e0018824800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5433.672776] LustreError: 82935:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5436.403306] LustreError: lustre-MDT0000-mdc-ffff9e0018824800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5436.419438] LustreError: 82950:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9e0018824800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9e0004860500) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5436.451545] LustreError: 82950:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5436.457208] LustreError: 82955:0:(file.c:6230:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5436.472106] LustreError: 82955:0:(file.c:6230:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5437.738035] LustreError: 82935:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5440.504402] LustreError: lustre-MDT0000-mdc-ffff9e0018824800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5441.904109] LustreError: 82963:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9e0018824800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5449.386598] LustreError: 83034:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5449.401215] LustreError: 83034:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5452.817623] LustreError: lustre-MDT0000-mdc-ffff9e0018824800: operation ldlm_enqueue to node 192.168.203.145@tcp failed: rc = -107
[ 5452.824787] LustreError: Skipped 8 previous similar messages
[ 5452.847845] LustreError: lustre-MDT0000-mdc-ffff9e0018824800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5452.930506] LustreError: 83057:0:(file.c:6230:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5452.943435] LustreError: 83057:0:(file.c:6230:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 5453.399908] LustreError: 83034:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5453.413190] LustreError: 83034:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5453.419293] LustreError: 83034:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9e0018824800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5463.719869] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 02:18:16 (1787120296)
[ 5483.361653] LustreError: 68007:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5483.364606] LustreError: 68007:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5487.440212] LustreError: 68007:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5487.452485] LustreError: 68007:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5493.361810] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 02:18:46 (1787120326)
[ 5543.423793] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 02:19:36 (1787120376)
[ 5606.758231] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 02:20:39 (1787120439)
[ 5640.005452] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 02:21:12 (1787120472)
[ 5640.434857] Lustre: Mounted lustre-client
[ 5763.978118] Lustre: Unmounted lustre-client
[ 5770.946598] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 02:23:23 (1787120603)
[ 5771.874216] LustreError: 87362:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5774.910218] LustreError: lustre-OST0000-osc-ffff9e0018824800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5774.926867] LustreError: 87450:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff9e0018824800: namespace resource [0x240000400:0x3ead:0x0].0x0 (ffff9e0004860b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5774.954953] LustreError: 87450:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5775.247428] LustreError: 87362:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5775.250921] Lustre: 2349:0:(llite_lib.c:4339:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.145@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5775.260584] Lustre: lustre-OST0000-osc-ffff9e0018824800: Connection restored to 192.168.203.145@tcp (at 192.168.203.145@tcp)
[ 5775.267660] Lustre: Skipped 18 previous similar messages
[ 5782.647725] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 02:23:35 (1787120615)
[ 5839.840334] Lustre: 88029:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787120619/real 1787120619]  req@ffff9e000480c000 x1873925803396736/t0(0) o101->lustre-MDT0000-mdc-ffff9e0018824800@192.168.203.145@tcp:12/10 lens 328/344 e 0 to 1 dl 1787120674 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 5839.878912] Lustre: 88029:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 20 previous similar messages
[ 5846.076755] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 02:24:38 (1787120678)
[ 5847.386501] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 5849.273334] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 02:24:41 (1787120681)
[ 5913.501204] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 5915.327511] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 02:25:48 (1787120748)
[ 5973.314933] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 02:26:46 (1787120806)
[ 5975.859681] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 5977.658284] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 02:26:50 (1787120810)
[ 5978.960468] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 5980.731691] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 02:26:53 (1787120813)
[ 6013.598835] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 02:27:26 (1787120846)
[ 6024.603924] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6049.759234] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 192.168.203.145@tcp) was lost; in progress operations using this service will fail
[ 6049.769129] LustreError: Skipped 2 previous similar messages
[ 6049.784230] Lustre: Evicted from MGS (at 192.168.203.145@tcp) after server handle changed from 0xb35bf54c639efcc8 to 0xb35bf54c639f152c
[ 6049.796814] Lustre: Skipped 2 previous similar messages
[ 6051.565814] Lustre: 68009:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.145@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6054.879297] Lustre: lustre-MDT0000-mdc-ffff9e0018824800: Connection to lustre-MDT0000 (at 192.168.203.145@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6054.919336] Lustre: Skipped 16 previous similar messages
[ 6077.701179] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6079.035841] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6088.165853] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 02:28:41 (1787120921)
[ 6090.145333] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6091.606849] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 02:28:44 (1787120924)
[ 6106.086609] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 192.168.203.145@tcp) was lost; in progress operations using this service will fail
[ 6106.102621] Lustre: Evicted from MGS (at 192.168.203.145@tcp) after server handle changed from 0xb35bf54c639f152c to 0xb35bf54c639f19e7
[ 6114.485678] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 02:29:07 (1787120947)
[ 6138.419821] Lustre: 68009:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.145@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6160.233884] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 02:29:53 (1787120993)
[ 6166.008459] LustreError: lustre-OST0000-osc-ffff9e0018824800: operation ost_setattr to node 192.168.203.145@tcp failed: rc = -19
[ 6166.038708] LustreError: Skipped 1 previous similar message
[ 6216.312490] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6219.576049] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6305.260538] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 192.168.203.145@tcp) was lost; in progress operations using this service will fail
[ 6305.284151] LustreError: Skipped 1 previous similar message
[ 6305.298507] Lustre: Evicted from MGS (at 192.168.203.145@tcp) after server handle changed from 0xb35bf54c639f1bf4 to 0xb35bf54c639f2b44
[ 6305.308039] Lustre: Skipped 1 previous similar message
[ 6319.700160] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6321.378951] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6352.693209] Lustre: 68009:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.145@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6368.604538] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6369.991456] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6501.632707] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 02:35:34 (1787121334)
[ 6555.182822] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6557.480110] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6805.788886] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 02:40:38 (1787121638)
[ 6950.373354] Lustre: lustre-MDT0000-mdc-ffff9e0018824800: Connection to lustre-MDT0000 (at 192.168.203.145@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6950.394869] Lustre: Skipped 6 previous similar messages
[ 6965.734438] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 192.168.203.145@tcp) was lost; in progress operations using this service will fail
[ 6965.746556] LustreError: Skipped 1 previous similar message
[ 6965.766504] Lustre: Evicted from MGS (at 192.168.203.145@tcp) after server handle changed from 0xb35bf54c639f2d74 to 0xb35bf54c63ae9a3f
[ 6965.776791] Lustre: Skipped 1 previous similar message
[ 6965.785945] Lustre: MGC192.168.203.145@tcp: Connection restored to 192.168.203.145@tcp (at 192.168.203.145@tcp)
[ 6965.790027] Lustre: Skipped 13 previous similar messages
[ 6997.151421] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 02:43:50 (1787121830)
[ 6998.676588] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7000.374788] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 02:43:53 (1787121833)
[ 7002.203133] LustreError: lustre-MDT0000-mdc-ffff9e0018824800: operation ldlm_enqueue to node 192.168.203.145@tcp failed: rc = -107
[ 7002.219921] LustreError: Skipped 591 previous similar messages
[ 7002.229940] LustreError: lustre-MDT0000-mdc-ffff9e0018824800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7008.475933] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 02:44:01 (1787121841)
[ 7175.664441] LustreError: lustre-OST0000-osc-ffff9e0018824800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7177.105438] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 02:46:50 (1787122010)
[ 7201.247211] Lustre: 2350:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787122015/real 1787122015]  req@ffff9e002a7a9c00 x1873925820910336/t0(0) o4->lustre-OST0000-osc-ffff9e0018824800@192.168.203.145@tcp:6/4 lens 4584/448 e 0 to 1 dl 1787122035 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7201.285017] Lustre: 2350:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 24 previous similar messages
[ 7217.245886] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 02:47:30 (1787122050)
[ 7218.674217] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7220.033431] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 02:47:33 (1787122053)
[ 7221.140143] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7222.572590] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 02:47:35 (1787122055)
[ 7255.525879] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 02:48:08 (1787122088)
[ 7293.412208] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 192.168.203.145@tcp) was lost; in progress operations using this service will fail
[ 7293.436737] Lustre: Evicted from MGS (at 192.168.203.145@tcp) after server handle changed from 0xb35bf54c63ae9a3f to 0xb35bf54c63aec955
[ 7293.447669] Lustre: Skipped 1 previous similar message
[ 7304.062883] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 02:48:57 (1787122137)
[ 7305.245162] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7306.484187] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 02:48:59 (1787122139)
[ 7307.815680] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7309.303165] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 02:49:02 (1787122142)
[ 7320.912382] Lustre: Unmounted lustre-client
[ 7322.295057] Lustre: Mounted lustre-client
[ 7324.404673] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7343.071756] Lustre: 2349:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787122161/real 1787122161]  req@ffff9e0008a58700 x1873925821096192/t0(0) o400->MGC192.168.203.145@tcp@192.168.203.145@tcp:26/25 lens 224/224 e 0 to 1 dl 1787122177 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7350.917232] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2