[ 1933.148658] LustreError: lustre-OST0000-osc-ffff890910b21800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1933.184358] Lustre: 2363:0:(llite_lib.c:4202:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.136@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1939.167435] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 04:57:56 (1787389076)
[ 1941.285582] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1942.628686] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 04:57:59 (1787389079)
[ 1944.475796] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1946.209723] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 04:58:03 (1787389083)
[ 1969.138615] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 192.168.203.136@tcp) was lost; in progress operations using this service will fail
[ 1969.166237] Lustre: Evicted from MGS (at 192.168.203.136@tcp) after server handle changed from 0xe66c0a81b86db760 to 0xe66c0a81b86dcfd2
[ 2069.548571] LustreError: lustre-MDT0000-mdc-ffff890910b21800: operation mds_reint to node 192.168.203.136@tcp failed: rc = -19
[ 2069.560373] LustreError: Skipped 3 previous similar messages
[ 2086.895907] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 192.168.203.136@tcp) was lost; in progress operations using this service will fail
[ 2086.908848] Lustre: Evicted from MGS (at 192.168.203.136@tcp) after server handle changed from 0xe66c0a81b86dcfd2 to 0xe66c0a81b8703118
[ 2112.762171] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 05:00:49 (1787389249)
[ 2113.110881] Lustre: *** cfs_fail_loc=305, val=0***
[ 2113.118321] Lustre: Skipped 2 previous similar messages
[ 2153.445302] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 192.168.203.136@tcp) was lost; in progress operations using this service will fail
[ 2153.467173] Lustre: Evicted from MGS (at 192.168.203.136@tcp) after server handle changed from 0xe66c0a81b8703118 to 0xe66c0a81b870375b
[ 2155.001695] Lustre: 15903:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.136@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2171.230403] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2172.356308] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2179.288795] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 05:01:56 (1787389316)
[ 2189.294675] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 192.168.203.136@tcp) was lost; in progress operations using this service will fail
[ 2189.318225] Lustre: Evicted from MGS (at 192.168.203.136@tcp) after server handle changed from 0xe66c0a81b870375b to 0xe66c0a81b8703bfa
[ 2190.681982] LustreError: lustre-MDT0000-mdc-ffff890910b21800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2209.909978] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 05:02:27 (1787389347)
[ 2214.894211] Lustre: lustre-OST0000-osc-ffff890910b21800: Connection to lustre-OST0000 (at 192.168.203.136@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2214.909813] Lustre: Skipped 13 previous similar messages
[ 2220.355163] LustreError: lustre-OST0000-osc-ffff890910b21800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2237.858226] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 05:02:54 (1787389374)
[ 2250.200224] LustreError: lustre-MDT0000-mdc-ffff890910b21800: operation mds_reint to node 192.168.203.136@tcp failed: rc = -19
[ 2250.214764] LustreError: Skipped 1 previous similar message
[ 2268.139955] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 192.168.203.136@tcp) was lost; in progress operations using this service will fail
[ 2268.157194] Lustre: Evicted from MGS (at 192.168.203.136@tcp) after server handle changed from 0xe66c0a81b8703bfa to 0xe66c0a81b8709b1b
[ 2268.173599] Lustre: MGC192.168.203.136@tcp: Connection restored to 192.168.203.136@tcp (at 192.168.203.136@tcp)
[ 2268.178878] Lustre: Skipped 17 previous similar messages
[ 2281.599859] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2283.579820] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2366.431279] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 192.168.203.136@tcp) was lost; in progress operations using this service will fail
[ 2366.480505] Lustre: Evicted from MGS (at 192.168.203.136@tcp) after server handle changed from 0xe66c0a81b8709b1b to 0xe66c0a81b872bbf5
[ 2393.640025] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2396.725470] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2461.973222] LustreError: lustre-MDT0000-mdc-ffff890910b21800: operation mds_reint to node 192.168.203.136@tcp failed: rc = -19
[ 2461.984689] LustreError: Skipped 1 previous similar message
[ 2481.127623] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 192.168.203.136@tcp) was lost; in progress operations using this service will fail
[ 2481.185604] Lustre: Evicted from MGS (at 192.168.203.136@tcp) after server handle changed from 0xe66c0a81b872bbf5 to 0xe66c0a81b8749873
[ 2495.357802] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2497.203140] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2525.673685] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 05:07:42 (1787389662)
[ 2547.688591] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 192.168.203.136@tcp) was lost; in progress operations using this service will fail
[ 2547.707778] Lustre: Evicted from MGS (at 192.168.203.136@tcp) after server handle changed from 0xe66c0a81b8749873 to 0xe66c0a81b875a6b0
[ 2555.663738] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2584.831557] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2615.424161] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2652.030552] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2693.094538] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 192.168.203.136@tcp) was lost; in progress operations using this service will fail
[ 2693.105963] LustreError: Skipped 3 previous similar messages
[ 2693.124451] Lustre: Evicted from MGS (at 192.168.203.136@tcp) after server handle changed from 0xe66c0a81b876cf65 to 0xe66c0a81b877a2f4
[ 2693.134449] Lustre: Skipped 3 previous similar messages
[ 2699.164725] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2727.041995] LustreError: lustre-MDT0000-mdc-ffff890910b21800: operation mds_reint to node 192.168.203.136@tcp failed: rc = -19
[ 2727.055633] LustreError: Skipped 8 previous similar messages
[ 2751.286149] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2801.631505] Lustre: 2363:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787389923/real 1787389923]  req@ffff8909083c9180 x1874211388921984/t0(0) o400->MGC192.168.203.136@tcp@192.168.203.136@tcp:26/25 lens 224/224 e 0 to 1 dl 1787389939 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2801.668978] Lustre: 2363:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 10 previous similar messages
[ 2803.500607] Lustre: 15903:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.136@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2843.615677] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 05:13:00 (1787389980)
[ 2857.890764] Lustre: lustre-OST0000-osc-ffff890910b21800: Connection to lustre-OST0000 (at 192.168.203.136@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2857.907895] Lustre: Skipped 10 previous similar messages
[ 2877.908831] Lustre: lustre-OST0000-osc-ffff890910b21800: Connection restored to 192.168.203.136@tcp (at 192.168.203.136@tcp)
[ 2877.923750] Lustre: Skipped 19 previous similar messages
[ 2891.235739] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2893.099698] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3219.190623] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3220.848399] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3519.747645] LustreError: lustre-OST0000-osc-ffff890910b21800: operation ldlm_enqueue to node 192.168.203.136@tcp failed: rc = -107
[ 3519.754689] LustreError: Skipped 3 previous similar messages
[ 3519.758151] Lustre: lustre-OST0000-osc-ffff890910b21800: Connection to lustre-OST0000 (at 192.168.203.136@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3519.775384] Lustre: Skipped 1 previous similar message
[ 3540.803259] Lustre: lustre-OST0000-osc-ffff890910b21800: Connection restored to 192.168.203.136@tcp (at 192.168.203.136@tcp)
[ 3540.820196] Lustre: Skipped 1 previous similar message
[ 3559.425105] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3561.966098] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3814.435727] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 05:29:11 (1787390951)
[ 3832.287289] Lustre: 50413:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787390954/real 1787390954]  req@ffff89090459f800 x1874211410305664/t0(0) o101->lustre-MDT0000-mdc-ffff890910b21800@192.168.203.136@tcp:12/10 lens 576/1152 e 0 to 1 dl 1787390970 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3839.667478] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 05:29:36 (1787390976)
[ 3865.464197] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 05:30:02 (1787391002)
[ 3891.571421] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 05:30:28 (1787391028)
[ 3892.040067] Lustre: Mounted lustre-client
[ 3922.916041] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 192.168.203.136@tcp) was lost; in progress operations using this service will fail
[ 3922.925231] LustreError: Skipped 2 previous similar messages
[ 3933.168760] Lustre: Evicted from MGS (at 192.168.203.136@tcp) after server handle changed from 0xe66c0a81b879b61b to 0xe66c0a81b892aa6d
[ 3933.176391] Lustre: Skipped 2 previous similar messages
[ 3940.033465] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3941.896579] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3944.347401] Lustre: Unmounted lustre-client
[ 3953.361304] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 05:31:29 (1787391089)
[ 4124.643221] Lustre: lustre-OST0000-osc-ffff890910b21800: Connection to lustre-OST0000 (at 192.168.203.136@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4124.675072] Lustre: Skipped 14 previous similar messages
[ 4156.354334] Lustre: lustre-OST0000-osc-ffff890910b21800: Connection restored to 192.168.203.136@tcp (at 192.168.203.136@tcp)
[ 4156.375232] Lustre: Skipped 17 previous similar messages
[ 4247.367554] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 05:36:24 (1787391384)
[ 4296.221702] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 05:37:12 (1787391432)
[ 4300.231138] Lustre: Unmounted lustre-client
[ 4323.429350] Lustre: Mounted lustre-client
[ 4330.125572] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 05:37:46 (1787391466)
[ 4330.443749] LustreError: 56322:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4331.487333] LustreError: 56322:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4331.675293] Lustre: *** cfs_fail_loc=305, val=0***
[ 4353.571624] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 05:38:10 (1787391490)
[ 4353.982484] Lustre: Mounted lustre-client
[ 4355.763811] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4366.139889] Lustre: Unmounted lustre-client
[ 4374.167551] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 05:38:30 (1787391510)
[ 4459.729215] LustreError: lustre-MDT0000-mdc-ffff890908a0e800: operation mds_reint to node 192.168.203.136@tcp failed: rc = -19
[ 4476.895141] Lustre: 2365:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787391599/real 1787391599]  req@ffff890910124000 x1874211413038720/t0(0) o400->MGC192.168.203.136@tcp@192.168.203.136@tcp:26/25 lens 224/224 e 0 to 1 dl 1787391615 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4476.935764] Lustre: 2365:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 142 previous similar messages
[ 4476.942611] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 192.168.203.136@tcp) was lost; in progress operations using this service will fail
[ 4487.204423] Lustre: Evicted from MGS (at 192.168.203.136@tcp) after server handle changed from 0xe66c0a81b892bff9 to 0xe66c0a81b89574e4
[ 4668.595251] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 05:43:25 (1787391805)
[ 4674.118713] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4690.940029] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 192.168.203.136@tcp) was lost; in progress operations using this service will fail
[ 4690.977682] Lustre: Evicted from MGS (at 192.168.203.136@tcp) after server handle changed from 0xe66c0a81b89574e4 to 0xe66c0a81b89a3db3
[ 4691.073422] LustreError: lustre-MDT0000-mdc-ffff890908a0e800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4712.260801] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 05:44:08 (1787391848)
[ 4712.726430] Lustre: Mounted lustre-client
[ 4720.396241] LustreError: lustre-OST0000-osc-ffff890908a0e800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4720.421192] LustreError: 60031:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff890908a0e800: namespace resource [0x240000400:0x4a23:0x0].0x0 (ffff890906a00500) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4730.335359] Lustre: lustre-OST0000-osc-ffff89090a193800: Connection to lustre-OST0000 (at 192.168.203.136@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4730.345055] Lustre: Skipped 12 previous similar messages
[ 4732.094875] Lustre: Unmounted lustre-client
[ 4739.720587] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 05:44:35 (1787391875)
[ 4746.575207] LustreError: lustre-MDT0000-mdc-ffff890908a0e800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4746.606034] LustreError: 60703:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4754.800709] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 05:44:51 (1787391891)
[ 4754.978884] LustreError: 61403:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4758.804424] LustreError: lustre-MDT0000-mdc-ffff890908a0e800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4758.825274] LustreError: 61425:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4758.837197] LustreError: 61403:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=1168
[ 4758.859959] LustreError: 61403:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting
[ 4758.860590] LustreError: 61430:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4759.945437] LustreError: 61436:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4759.958405] LustreError: 61436:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4762.214256] LustreError: 61459:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4762.255242] LustreError: 61459:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4766.824427] LustreError: 61504:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4766.838868] LustreError: 61504:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4769.261738] Lustre: lustre-MDT0000-mdc-ffff890908a0e800: Connection restored to 192.168.203.136@tcp (at 192.168.203.136@tcp)
[ 4769.278895] Lustre: Skipped 15 previous similar messages
[ 4777.152119] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 05:45:13 (1787391913)
[ 4816.511745] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4818.129510] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4830.328975] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 05:46:06 (1787391966)
[ 4870.413624] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4871.911910] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4882.379522] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 05:46:59 (1787392019)
[ 4944.738253] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4946.511499] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4955.505323] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 05:48:12 (1787392092)
[ 4997.804342] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4999.632663] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5006.252887] Lustre: Unmounted lustre-client
[ 5052.488841] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5054.769157] Lustre: Mounted lustre-client
[ 5059.591048] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 05:49:56 (1787392196)
[ 5061.388752] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5062.515637] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 05:49:59 (1787392199)
[ 5088.979651] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 05:50:25 (1787392225)
[ 5090.360729] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5091.637599] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 05:50:28 (1787392228)
[ 5092.824983] Lustre: *** cfs_fail_loc=805, val=0***
[ 5092.861466] Lustre: Mounted lustre-client
[ 5097.327977] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5118.457296] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 192.168.203.136@tcp) was lost; in progress operations using this service will fail
[ 5118.523749] Lustre: Evicted from MGS (at 192.168.203.136@tcp) after server handle changed from 0xe66c0a81b89a47d8 to 0xe66c0a81b89a4b74
[ 5119.455241] Lustre: 2362:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787392241/real 1787392241]  req@ffff8909087d6300 x1874211415089024/t0(0) o400->lustre-MDT0000-mdc-ffff8909082a2800@192.168.203.136@tcp:12/10 lens 224/224 e 0 to 1 dl 1787392257 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5119.497242] Lustre: 2362:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5127.531131] Lustre: Unmounted lustre-client
[ 5136.063784] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 05:51:13 (1787392273)
[ 5163.599496] LustreError: 2361:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff8909087d6680 x1874211415100672/t94489280517(94489280517) o101->lustre-MDT0000-mdc-ffff8909082a2800@192.168.203.136@tcp:12/10 lens 648/608 e 0 to 0 dl 1787392317 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5173.622643] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5175.399995] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5182.755111] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 05:51:59 (1787392319)
[ 5184.206368] LustreError: lustre-OST0000-osc-ffff8909082a2800: operation ost_write to node 192.168.203.136@tcp failed: rc = -107
[ 5184.224263] LustreError: Skipped 3 previous similar messages
[ 5184.237177] LustreError: lustre-OST0000-osc-ffff8909082a2800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5184.256230] Lustre: 2365:0:(llite_lib.c:4202:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.136@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5184.297880] LustreError: 72537:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8909082a2800: namespace resource [0x240000400:0x4a82:0x0].0x0 (ffff890906917000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5191.361339] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 05:52:08 (1787392328)
[ 5192.589396] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5193.822193] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 05:52:10 (1787392330)
[ 5195.021485] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5196.521875] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 05:52:13 (1787392333)
[ 5197.928661] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5199.310444] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 05:52:16 (1787392336)
[ 5200.929812] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5202.472958] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 05:52:19 (1787392339)
[ 5204.020391] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5205.505452] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 05:52:22 (1787392342)
[ 5206.874738] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5208.524376] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 05:52:25 (1787392345)
[ 5209.994925] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5211.545617] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 05:52:28 (1787392348)
[ 5213.022462] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5214.893150] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 05:52:31 (1787392351)
[ 5216.106742] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5217.730176] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 05:52:34 (1787392354)
[ 5219.020300] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5220.651727] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 05:52:37 (1787392357)
[ 5222.003546] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5223.458809] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 05:52:40 (1787392360)
[ 5225.917800] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5227.740993] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 05:52:44 (1787392364)
[ 5255.311552] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 05:53:12 (1787392392)
[ 5284.327660] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 05:53:41 (1787392421)
[ 5284.857686] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5294.566430] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 05:53:51 (1787392431)
[ 5294.912157] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5305.541568] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 05:54:02 (1787392442)
[ 5305.937851] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5314.712921] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 05:54:11 (1787392451)
[ 5315.010491] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5323.016317] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 05:54:19 (1787392459)
[ 5323.317090] Lustre: *** cfs_fail_loc=510, val=3***
[ 5331.406140] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 05:54:28 (1787392468)
[ 5331.902515] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5342.788148] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 05:54:39 (1787392479)
[ 5343.152649] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5402.591413] Lustre: lustre-OST0001-osc-ffff8909082a2800: Connection to lustre-OST0001 (at 192.168.203.136@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5402.616490] Lustre: Skipped 14 previous similar messages
[ 5408.944028] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 05:55:45 (1787392545)
[ 5409.361621] LustreError: 82725:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5412.066175] LustreError: lustre-MDT0000-mdc-ffff8909082a2800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5412.089261] LustreError: 82739:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8909082a2800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff890906a00a00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5412.110511] Lustre: lustre-MDT0000-mdc-ffff8909082a2800: Connection restored to 192.168.203.136@tcp (at 192.168.203.136@tcp)
[ 5412.117853] Lustre: Skipped 11 previous similar messages
[ 5413.423146] LustreError: 82725:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5415.541224] LustreError: 82755:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5418.434976] LustreError: lustre-MDT0000-mdc-ffff8909082a2800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5418.444836] LustreError: 82769:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8909082a2800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff890906917e00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5418.464456] LustreError: 82769:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5418.488924] LustreError: 82774:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5418.509152] LustreError: 82774:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 17 previous similar messages
[ 5419.631171] LustreError: 82755:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5426.727197] LustreError: lustre-MDT0000-mdc-ffff8909082a2800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5427.099744] LustreError: 82811:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5429.856286] LustreError: lustre-MDT0000-mdc-ffff8909082a2800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5429.931307] LustreError: 82830:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5431.160257] LustreError: 82811:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5431.186779] LustreError: 82811:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8909082a2800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5432.110492] LustreError: 82848:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5432.132993] LustreError: 82848:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 12 previous similar messages
[ 5437.022901] LustreError: 82886:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5439.937905] LustreError: lustre-MDT0000-mdc-ffff8909082a2800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5439.964958] LustreError: 82900:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8909082a2800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff890906917200) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5440.017183] LustreError: 82900:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5440.042079] LustreError: 82905:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5440.051175] LustreError: 82905:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5441.103297] LustreError: 82886:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5448.119789] LustreError: lustre-MDT0000-mdc-ffff8909082a2800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5448.616081] LustreError: 82942:0:(ldlm_flock.c:858:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5451.505062] LustreError: lustre-MDT0000-mdc-ffff8909082a2800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5451.528313] LustreError: 82957:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8909082a2800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff890906a00b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5451.546977] LustreError: 82957:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 2 previous similar messages
[ 5452.703811] LustreError: 82942:0:(ldlm_flock.c:858:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5455.651543] LustreError: lustre-MDT0000-mdc-ffff8909082a2800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5455.742568] LustreError: 82989:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5456.943134] LustreError: 82970:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8909082a2800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5464.996415] LustreError: 83041:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5465.008649] LustreError: 83041:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5468.680537] LustreError: lustre-MDT0000-mdc-ffff8909082a2800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5469.079132] LustreError: 83041:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5469.097814] LustreError: 83041:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5469.117937] LustreError: 83041:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8909082a2800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5472.313945] LustreError: 83092:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5472.333980] LustreError: 83092:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 46 previous similar messages
[ 5480.198967] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 05:56:56 (1787392616)
[ 5499.725815] LustreError: 68023:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5499.734740] LustreError: 68023:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5503.807454] LustreError: 68023:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5503.817974] LustreError: 68023:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5510.209297] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 05:57:27 (1787392647)
[ 5559.926678] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 05:58:16 (1787392696)
[ 5624.414615] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 05:59:21 (1787392761)
[ 5658.145127] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 05:59:55 (1787392795)
[ 5658.607622] Lustre: Mounted lustre-client
[ 5782.568060] Lustre: Unmounted lustre-client
[ 5789.443691] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 06:02:06 (1787392926)
[ 5790.101043] LustreError: 87382:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5792.684574] LustreError: lustre-OST0000-osc-ffff8909082a2800: operation ldlm_enqueue to node 192.168.203.136@tcp failed: rc = -107
[ 5792.697424] LustreError: Skipped 9 previous similar messages
[ 5792.713485] LustreError: lustre-OST0000-osc-ffff8909082a2800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5792.729133] LustreError: 87464:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8909082a2800: namespace resource [0x240000400:0x4aad:0x0].0x0 (ffff890909adfd00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5792.777839] LustreError: 87464:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5793.024119] LustreError: 87382:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5793.036930] Lustre: 2363:0:(llite_lib.c:4202:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.136@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5798.768326] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 06:02:15 (1787392935)
[ 5856.223154] Lustre: 88049:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787392939/real 1787392939]  req@ffff890910127100 x1874211415272192/t0(0) o101->lustre-MDT0000-mdc-ffff8909082a2800@192.168.203.136@tcp:12/10 lens 328/344 e 0 to 1 dl 1787392994 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 5856.258980] Lustre: 88049:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 14 previous similar messages
[ 5862.753935] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 06:03:19 (1787392999)
[ 5864.349741] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 5865.844339] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 06:03:22 (1787393002)
[ 5929.048426] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 5930.691229] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 06:04:27 (1787393067)
[ 5988.692731] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 06:05:25 (1787393125)
[ 5990.714965] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 5992.424213] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 06:05:29 (1787393129)
[ 5993.685210] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 5995.555329] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 06:05:32 (1787393132)
[ 6029.749980] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 06:06:06 (1787393166)
[ 6040.151214] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6064.095231] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 192.168.203.136@tcp) was lost; in progress operations using this service will fail
[ 6064.112036] LustreError: Skipped 2 previous similar messages
[ 6070.303467] Lustre: lustre-MDT0000-mdc-ffff8909082a2800: Connection to lustre-MDT0000 (at 192.168.203.136@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6070.335634] Lustre: Skipped 16 previous similar messages
[ 6074.476959] Lustre: Evicted from MGS (at 192.168.203.136@tcp) after server handle changed from 0xe66c0a81b89a5633 to 0xe66c0a81b89a6e58
[ 6074.490128] Lustre: Skipped 2 previous similar messages
[ 6074.500752] Lustre: MGC192.168.203.136@tcp: Connection restored to 192.168.203.136@tcp (at 192.168.203.136@tcp)
[ 6074.512370] Lustre: Skipped 14 previous similar messages
[ 6088.464933] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6089.900838] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6099.131753] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 06:07:16 (1787393236)
[ 6101.464338] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6102.755309] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 06:07:19 (1787393239)
[ 6116.883326] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 192.168.203.136@tcp) was lost; in progress operations using this service will fail
[ 6116.908292] Lustre: Evicted from MGS (at 192.168.203.136@tcp) after server handle changed from 0xe66c0a81b89a6e58 to 0xe66c0a81b89a72fe
[ 6129.337714] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 06:07:45 (1787393265)
[ 6168.865438] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 06:08:25 (1787393305)
[ 6221.748898] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 6225.173263] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6312.421128] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 192.168.203.136@tcp) was lost; in progress operations using this service will fail
[ 6312.437205] LustreError: Skipped 1 previous similar message
[ 6312.462590] Lustre: Evicted from MGS (at 192.168.203.136@tcp) after server handle changed from 0xe66c0a81b89a7853 to 0xe66c0a81b89a84cb
[ 6312.473080] Lustre: Skipped 1 previous similar message
[ 6323.521972] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6325.385389] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6349.869933] Lustre: 68025:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.136@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6369.653194] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6371.243548] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6502.627519] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 06:13:59 (1787393639)
[ 6511.368043] LustreError: lustre-OST0000-osc-ffff8909082a2800: operation ost_setattr to node 192.168.203.136@tcp failed: rc = -19
[ 6511.392693] LustreError: Skipped 425 previous similar messages
[ 6555.343150] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 6558.806341] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6784.664870] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 06:18:41 (1787393921)
[ 6937.068464] Lustre: lustre-MDT0000-mdc-ffff8909082a2800: Connection to lustre-MDT0000 (at 192.168.203.136@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6937.087067] Lustre: Skipped 6 previous similar messages
[ 6947.302848] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 192.168.203.136@tcp) was lost; in progress operations using this service will fail
[ 6947.314269] LustreError: Skipped 1 previous similar message
[ 6947.332189] Lustre: Evicted from MGS (at 192.168.203.136@tcp) after server handle changed from 0xe66c0a81b89a867d to 0xe66c0a81b8a93186
[ 6947.347986] Lustre: Skipped 1 previous similar message
[ 6947.357613] Lustre: MGC192.168.203.136@tcp: Connection restored to 192.168.203.136@tcp (at 192.168.203.136@tcp)
[ 6947.375379] Lustre: Skipped 9 previous similar messages
[ 6985.930844] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 06:22:02 (1787394122)
[ 6987.641659] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 6989.437408] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 06:22:06 (1787394126)
[ 6991.577424] LustreError: lustre-MDT0000-mdc-ffff8909082a2800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6998.893850] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 06:22:15 (1787394135)
[ 7165.926236] LustreError: lustre-OST0000-osc-ffff8909082a2800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7167.577804] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 06:25:04 (1787394304)
[ 7192.543156] Lustre: 2362:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787394310/real 1787394310]  req@ffff890906574e00 x1874211431872128/t0(0) o4->lustre-OST0000-osc-ffff8909082a2800@192.168.203.136@tcp:6/4 lens 4584/448 e 0 to 1 dl 1787394330 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7192.570281] Lustre: 2362:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 21 previous similar messages
[ 7209.567724] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 06:25:46 (1787394346)
[ 7211.070776] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7212.671472] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 06:25:49 (1787394349)
[ 7214.003822] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7215.946538] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 06:25:52 (1787394352)
[ 7250.121104] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 06:26:26 (1787394386)
[ 7289.838118] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 192.168.203.136@tcp) was lost; in progress operations using this service will fail
[ 7289.863869] Lustre: Evicted from MGS (at 192.168.203.136@tcp) after server handle changed from 0xe66c0a81b8a93186 to 0xe66c0a81b8a9639e
[ 7289.872212] Lustre: Skipped 1 previous similar message
[ 7303.741975] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 06:27:20 (1787394440)
[ 7305.219700] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7306.679354] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 06:27:23 (1787394443)
[ 7308.086602] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7310.041829] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 06:27:26 (1787394446)
[ 7325.201679] Lustre: Unmounted lustre-client
[ 7327.022650] Lustre: Mounted lustre-client
[ 7329.658563] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7348.703177] Lustre: 2364:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787394470/real 1787394470]  req@ffff8909319b4e00 x1874211432058496/t0(0) o400->MGC192.168.203.136@tcp@192.168.203.136@tcp:26/25 lens 224/224 e 0 to 1 dl 1787394486 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7368.933303] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2