[ 1933.207755] LustreError: lustre-OST0000-osc-ffff8ac786353000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1933.233684] Lustre: 2361:0:(llite_lib.c:4187:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.134@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1941.421909] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 02:07:46 (1779689266)
[ 1944.131605] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1946.343926] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 02:07:51 (1779689271)
[ 1948.585666] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1951.492385] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 02:07:56 (1779689276)
[ 1955.993445] LustreError: lustre-MDT0000-mdc-ffff8ac786353000: operation mds_reint to node 192.168.204.134@tcp failed: rc = -19
[ 1956.007110] LustreError: Skipped 1 previous similar message
[ 1975.263296] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 192.168.204.134@tcp) was lost; in progress operations using this service will fail
[ 1975.311912] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0x42ce1623a52bb0e0 to 0x42ce1623a52bc99f
[ 2078.378507] LustreError: lustre-MDT0000-mdc-ffff8ac786353000: operation ldlm_enqueue to node 192.168.204.134@tcp failed: rc = -19
[ 2078.384717] LustreError: Skipped 1 previous similar message
[ 2097.631221] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 192.168.204.134@tcp) was lost; in progress operations using this service will fail
[ 2107.819736] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0x42ce1623a52bc99f to 0x42ce1623a52dbb01
[ 2124.348217] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 02:10:50 (1779689450)
[ 2124.938807] Lustre: *** cfs_fail_loc=305, val=0***
[ 2124.942995] Lustre: Skipped 2 previous similar messages
[ 2145.420887] LustreError: lustre-OST0001-osc-ffff8ac786353000: operation ost_connect to node 192.168.204.134@tcp failed: rc = -75
[ 2167.263615] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 192.168.204.134@tcp) was lost; in progress operations using this service will fail
[ 2178.225180] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0x42ce1623a52dbb01 to 0x42ce1623a52dc11a
[ 2181.308848] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2183.307172] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2192.774496] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 02:11:58 (1779689518)
[ 2209.267104] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 192.168.204.134@tcp) was lost; in progress operations using this service will fail
[ 2209.272092] Lustre: lustre-MDT0000-mdc-ffff8ac786353000: Connection to lustre-MDT0000 (at 192.168.204.134@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2209.294691] Lustre: Skipped 12 previous similar messages
[ 2209.299907] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0x42ce1623a52dc11a to 0x42ce1623a52dc3b3
[ 2209.318680] Lustre: MGC192.168.204.134@tcp: Connection restored to 192.168.204.134@tcp (at 192.168.204.134@tcp)
[ 2209.328202] Lustre: Skipped 14 previous similar messages
[ 2209.396213] LustreError: lustre-MDT0000-mdc-ffff8ac786353000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2228.778359] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 02:12:34 (1779689554)
[ 2245.101095] LustreError: lustre-OST0000-osc-ffff8ac786353000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2260.196662] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 02:13:05 (1779689585)
[ 2291.167902] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 192.168.204.134@tcp) was lost; in progress operations using this service will fail
[ 2301.368976] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0x42ce1623a52dc3b3 to 0x42ce1623a52e15c9
[ 2315.533550] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2317.710731] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2382.460579] LustreError: lustre-MDT0000-mdc-ffff8ac786353000: operation ldlm_enqueue to node 192.168.204.134@tcp failed: rc = -19
[ 2382.466272] LustreError: Skipped 2 previous similar messages
[ 2402.795176] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 192.168.204.134@tcp) was lost; in progress operations using this service will fail
[ 2402.858476] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0x42ce1623a52e15c9 to 0x42ce1623a52feabb
[ 2414.695649] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2417.005450] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2500.064118] Lustre: 2361:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779689811/real 1779689811]  req@ffff8ac79025fb80 x1866137496588416/t0(0) o400->MGC192.168.204.134@tcp@192.168.204.134@tcp:26/25 lens 224/224 e 0 to 1 dl 1779689827 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2500.097174] Lustre: 2361:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 12 previous similar messages
[ 2500.103498] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 192.168.204.134@tcp) was lost; in progress operations using this service will fail
[ 2510.311787] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0x42ce1623a52feabb to 0x42ce1623a531d968
[ 2524.801667] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2527.516256] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2554.793840] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 02:18:00 (1779689880)
[ 2575.844687] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 192.168.204.134@tcp) was lost; in progress operations using this service will fail
[ 2575.860225] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0x42ce1623a531d968 to 0x42ce1623a532ad28
[ 2583.690608] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2612.940932] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2647.655681] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2660.994678] LustreError: lustre-MDT0000-mdc-ffff8ac786353000: operation mds_reint to node 192.168.204.134@tcp failed: rc = -19
[ 2661.002512] LustreError: Skipped 5 previous similar messages
[ 2689.277626] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2732.031499] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 192.168.204.134@tcp) was lost; in progress operations using this service will fail
[ 2732.048420] LustreError: Skipped 3 previous similar messages
[ 2732.084482] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0x42ce1623a5337bb6 to 0x42ce1623a53412b4
[ 2732.096864] Lustre: Skipped 3 previous similar messages
[ 2740.121860] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2793.492847] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2826.274435] Lustre: lustre-MDT0000-mdc-ffff8ac786353000: Connection to lustre-MDT0000 (at 192.168.204.134@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2826.289488] Lustre: Skipped 10 previous similar messages
[ 2844.691638] Lustre: MGC192.168.204.134@tcp: Connection restored to 192.168.204.134@tcp (at 192.168.204.134@tcp)
[ 2844.707877] Lustre: Skipped 20 previous similar messages
[ 2876.538206] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 02:23:22 (1779690202)
[ 2921.938958] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2923.798758] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3220.761965] LustreError: lustre-OST0000-osc-ffff8ac786353000: operation ldlm_cancel to node 192.168.204.134@tcp failed: rc = -19
[ 3220.772478] LustreError: Skipped 7 previous similar messages
[ 3256.092774] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3257.711380] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3551.783142] Lustre: lustre-OST0000-osc-ffff8ac786353000: Connection to lustre-OST0000 (at 192.168.204.134@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3551.792046] Lustre: Skipped 2 previous similar messages
[ 3571.448267] Lustre: lustre-OST0000-osc-ffff8ac786353000: Connection restored to 192.168.204.134@tcp (at 192.168.204.134@tcp)
[ 3571.455747] Lustre: Skipped 3 previous similar messages
[ 3579.092582] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3580.366911] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3848.061276] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 02:39:33 (1779691173)
[ 3865.056620] Lustre: 50228:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779691176/real 1779691176]  req@ffff8ac787ef8700 x1866137523946112/t0(0) o101->lustre-MDT0000-mdc-ffff8ac786353000@192.168.204.134@tcp:12/10 lens 576/1152 e 0 to 1 dl 1779691192 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3865.093023] Lustre: 50228:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 3872.595777] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 02:39:58 (1779691198)
[ 3896.090528] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 02:40:21 (1779691221)
[ 3921.343641] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 02:40:46 (1779691246)
[ 3921.807683] Lustre: Mounted lustre-client
[ 3953.631295] Lustre: 2362:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779691264/real 1779691264]  req@ffff8ac790225c00 x1866137523964928/t0(0) o400->lustre-MDT0000-mdc-ffff8ac786353000@192.168.204.134@tcp:12/10 lens 224/224 e 0 to 1 dl 1779691280 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3953.631608] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 192.168.204.134@tcp) was lost; in progress operations using this service will fail
[ 3953.650873] Lustre: 2362:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 3953.666590] LustreError: Skipped 2 previous similar messages
[ 3953.699614] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0x42ce1623a5361b5b to 0x42ce1623a551cc4e
[ 3953.715543] Lustre: Skipped 2 previous similar messages
[ 3964.987178] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3966.544201] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3968.532355] LustreError: 52980:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff8ac786490800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 3968.543328] LustreError: 52980:0:(lov_obd.c:792:lov_cleanup()) Skipped 1 previous similar message
[ 3968.580233] Lustre: Unmounted lustre-client
[ 3974.103744] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 02:41:39 (1779691299)
[ 4109.215159] Lustre: 2362:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779691420/real 1779691420]  req@ffff8ac790225500 x1866137523997952/t0(0) o4->lustre-OST0000-osc-ffff8ac786353000@192.168.204.134@tcp:6/4 lens 488/448 e 0 to 1 dl 1779691436 ref 2 fl Rpc:XQr/602/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4109.229300] Lustre: 2362:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 69 previous similar messages
[ 4157.407622] Lustre: lustre-OST0000-osc-ffff8ac786353000: Connection to lustre-OST0000 (at 192.168.204.134@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4157.422540] Lustre: Skipped 15 previous similar messages
[ 4173.603692] Lustre: lustre-OST0000-osc-ffff8ac786353000: Connection restored to 192.168.204.134@tcp (at 192.168.204.134@tcp)
[ 4173.609504] Lustre: Skipped 17 previous similar messages
[ 4260.955547] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 02:46:27 (1779691587)
[ 4305.854245] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 02:47:12 (1779691632)
[ 4307.766827] LustreError: 55139:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff8ac786353000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 4307.775678] LustreError: 55139:0:(lov_obd.c:792:lov_cleanup()) Skipped 1 previous similar message
[ 4307.810346] Lustre: Unmounted lustre-client
[ 4319.434646] Lustre: Mounted lustre-client
[ 4322.566678] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 02:47:28 (1779691648)
[ 4322.724935] LustreError: 56115:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4323.776667] LustreError: 56115:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4323.827679] Lustre: *** cfs_fail_loc=305, val=0***
[ 4343.412838] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 02:47:49 (1779691669)
[ 4343.618968] Lustre: Mounted lustre-client
[ 4344.528721] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4354.808460] Lustre: Unmounted lustre-client
[ 4357.911318] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 02:48:04 (1779691684)
[ 4404.874840] LustreError: lustre-MDT0000-mdc-ffff8ac790b8f800: operation mds_reint to node 192.168.204.134@tcp failed: rc = -19
[ 4404.879163] LustreError: Skipped 1 previous similar message
[ 4420.578971] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 192.168.204.134@tcp) was lost; in progress operations using this service will fail
[ 4420.591127] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0x42ce1623a551ea38 to 0x42ce1623a554ba51
[ 4503.564380] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 02:50:29 (1779691829)
[ 4506.015960] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4512.742503] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 192.168.204.134@tcp) was lost; in progress operations using this service will fail
[ 4512.753807] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0x42ce1623a554ba51 to 0x42ce1623a559546c
[ 4512.764600] LustreError: lustre-MDT0000-mdc-ffff8ac790b8f800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4520.029886] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 02:50:46 (1779691846)
[ 4520.176414] Lustre: Mounted lustre-client
[ 4525.558235] LustreError: lustre-OST0000-osc-ffff8ac790b8f800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4525.565172] LustreError: 59789:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-OST0000-osc-ffff8ac790b8f800: namespace resource [0x240000400:0x4d23:0x0].0x0 (ffff8ac790648c00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4536.799798] Lustre: 2364:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779691847/real 1779691847]  req@ffff8ac79aa9ed80 x1866137528619776/t0(0) o103->lustre-OST0000-osc-ffff8ac7912ec800@192.168.204.134@tcp:17/18 lens 328/224 e 0 to 1 dl 1779691863 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295
[ 4536.814910] Lustre: 2364:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 77 previous similar messages
[ 4537.403050] LustreError: 59819:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff8ac7912ec800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 4537.409028] LustreError: 59819:0:(lov_obd.c:792:lov_cleanup()) Skipped 1 previous similar message
[ 4537.437501] Lustre: Unmounted lustre-client
[ 4540.743543] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 02:51:07 (1779691867)
[ 4545.956935] LustreError: lustre-MDT0000-mdc-ffff8ac790b8f800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4545.970461] LustreError: 60459:0:(file.c:6098:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4549.417350] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 02:51:15 (1779691875)
[ 4549.529952] LustreError: 61162:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4553.611276] LustreError: lustre-MDT0000-mdc-ffff8ac790b8f800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4553.619389] LustreError: 61179:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4553.627221] LustreError: 61162:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=907
[ 4553.630265] LustreError: 61162:0:(import.c:719:ptlrpc_connect_import_locked()) already connecting
[ 4553.659660] LustreError: 61184:0:(file.c:6098:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4554.736747] LustreError: 61190:0:(file.c:6098:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4554.748389] LustreError: 61190:0:(file.c:6098:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4556.916927] LustreError: 61213:0:(file.c:6098:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4556.923325] LustreError: 61213:0:(file.c:6098:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4561.200510] LustreError: 61257:0:(file.c:6098:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4561.212075] LustreError: 61257:0:(file.c:6098:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4567.940851] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 02:51:34 (1779691894)
[ 4590.981373] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4591.762727] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4597.316853] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 02:52:03 (1779691923)
[ 4620.811505] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4621.610784] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4626.964682] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 02:52:33 (1779691953)
[ 4674.802568] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4675.791322] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4680.972605] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 02:53:27 (1779692007)
[ 4702.512241] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4703.176752] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4705.800368] LustreError: 66825:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff8ac790b8f800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 4705.805713] LustreError: 66825:0:(lov_obd.c:792:lov_cleanup()) Skipped 1 previous similar message
[ 4705.832979] Lustre: Unmounted lustre-client
[ 4733.227571] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4734.980709] Lustre: Mounted lustre-client
[ 4738.176095] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 02:54:24 (1779692064)
[ 4739.340881] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 4740.213108] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 02:54:26 (1779692066)
[ 4753.534421] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 02:54:40 (1779692080)
[ 4754.168829] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 4754.883713] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 02:54:41 (1779692081)
[ 4755.428476] Lustre: *** cfs_fail_loc=805, val=0***
[ 4755.455618] Lustre: Mounted lustre-client
[ 4757.598243] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4774.587601] Lustre: lustre-MDT0000-mdc-ffff8ac791250800: Connection to lustre-MDT0000 (at 192.168.204.134@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4774.596314] Lustre: Skipped 21 previous similar messages
[ 4774.603913] LustreError: 2360:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8ac791250800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff8ac790648b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4774.613537] LustreError: 2360:0:(ldlm_resource.c:1180:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4774.624292] Lustre: lustre-MDT0000-mdc-ffff8ac791250800: Connection restored to 192.168.204.134@tcp (at 192.168.204.134@tcp)
[ 4774.629866] Lustre: Skipped 21 previous similar messages
[ 4775.143181] LustreError: 70140:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff8ac791250800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 4775.148100] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 192.168.204.134@tcp) was lost; in progress operations using this service will fail
[ 4775.148930] LustreError: 70140:0:(lov_obd.c:792:lov_cleanup()) Skipped 1 previous similar message
[ 4775.161710] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0x42ce1623a5595ede to 0x42ce1623a55964cd
[ 4775.182806] Lustre: Unmounted lustre-client
[ 4778.196221] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 02:55:04 (1779692104)
[ 4794.869978] LustreError: 2360:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff8ac787e9e680 x1866137528686080/t94489280515(94489280515) o101->lustre-MDT0000-mdc-ffff8ac790be1000@192.168.204.134@tcp:12/10 lens 664/608 e 0 to 0 dl 1779692137 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 4799.242304] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4800.009677] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4804.276869] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 02:55:30 (1779692130)
[ 4804.934271] LustreError: lustre-OST0000-osc-ffff8ac790be1000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4804.943622] Lustre: 2364:0:(llite_lib.c:4187:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.134@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 4804.968171] LustreError: 72132:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-OST0000-osc-ffff8ac790be1000: namespace resource [0x240000400:0x4d82:0x0].0x0 (ffff8ac7859cd000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4804.979255] LustreError: 72132:0:(ldlm_resource.c:1180:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4809.436242] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 02:55:35 (1779692135)
[ 4810.151435] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 4810.923943] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 02:55:37 (1779692137)
[ 4811.592734] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 4812.418301] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 02:55:38 (1779692138)
[ 4813.264771] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 4814.139792] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 02:55:40 (1779692140)
[ 4814.930293] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 4815.768350] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 02:55:42 (1779692142)
[ 4816.594162] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 4817.343835] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 02:55:43 (1779692143)
[ 4818.144304] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 4819.079428] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 02:55:45 (1779692145)
[ 4819.840410] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 4820.583784] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 02:55:47 (1779692147)
[ 4821.314518] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 4822.168433] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 02:55:48 (1779692148)
[ 4822.938889] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 4823.835627] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 02:55:50 (1779692150)
[ 4824.489686] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 4825.227192] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 02:55:51 (1779692151)
[ 4825.905992] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 4826.617942] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 02:55:53 (1779692153)
[ 4827.683334] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 4828.413245] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 02:55:54 (1779692154)
[ 4843.117507] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 02:56:09 (1779692169)
[ 4875.299277] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 02:56:40 (1779692200)
[ 4875.754099] Lustre: *** cfs_fail_loc=51b, val=3***
[ 4886.597589] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 02:56:52 (1779692212)
[ 4887.010172] Lustre: *** cfs_fail_loc=51b, val=4***
[ 4897.025413] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 02:57:02 (1779692222)
[ 4897.325505] Lustre: *** cfs_fail_loc=50f, val=3***
[ 4906.210839] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 02:57:12 (1779692232)
[ 4906.533478] Lustre: *** cfs_fail_loc=50f, val=4***
[ 4915.345421] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 02:57:21 (1779692241)
[ 4916.138808] Lustre: *** cfs_fail_loc=510, val=3***
[ 4925.375894] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 02:57:30 (1779692250)
[ 4925.717826] Lustre: *** cfs_fail_loc=51b, val=3***
[ 4934.533705] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 02:57:40 (1779692260)
[ 4934.877930] Lustre: *** cfs_fail_loc=51c, val=3***
[ 4997.504346] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 02:58:43 (1779692323)
[ 4997.690809] LustreError: 82310:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5000.384458] LustreError: lustre-MDT0000-mdc-ffff8ac790be1000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5000.392797] LustreError: 82324:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff8ac790be1000: inode [0x20000a042:0x8:0x0] mdc close failed: rc = -108
[ 5000.405365] LustreError: 82324:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8ac790be1000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8ac790648200) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5001.775146] LustreError: 82310:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5003.825640] LustreError: 82340:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5006.496496] LustreError: lustre-MDT0000-mdc-ffff8ac790be1000: operation ldlm_enqueue to node 192.168.204.134@tcp failed: rc = -107
[ 5006.507357] LustreError: Skipped 6 previous similar messages
[ 5006.523704] LustreError: lustre-MDT0000-mdc-ffff8ac790be1000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5006.539713] LustreError: 82354:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8ac790be1000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8ac790648100) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5006.550226] LustreError: 82354:0:(ldlm_resource.c:1180:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5007.895159] LustreError: 82340:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5014.484293] LustreError: lustre-MDT0000-mdc-ffff8ac790be1000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5014.815748] LustreError: 82395:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5017.407923] LustreError: lustre-MDT0000-mdc-ffff8ac790be1000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5017.459760] LustreError: 82414:0:(file.c:6098:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5017.466748] LustreError: 82414:0:(file.c:6098:ll_inode_revalidate_fini()) Skipped 20 previous similar messages
[ 5018.887134] LustreError: 82395:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5018.894659] LustreError: 82395:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff8ac790be1000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5024.174443] LustreError: 82469:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5026.837945] LustreError: lustre-MDT0000-mdc-ffff8ac790be1000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5026.850557] LustreError: 82484:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8ac790be1000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8ac79abddf00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5026.867319] LustreError: 82484:0:(ldlm_resource.c:1180:ldlm_resource_complain()) Skipped 2 previous similar messages
[ 5028.247128] LustreError: 82469:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5034.738321] LustreError: lustre-MDT0000-mdc-ffff8ac790be1000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5035.008782] LustreError: 82525:0:(ldlm_flock.c:858:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5037.589296] LustreError: lustre-MDT0000-mdc-ffff8ac790be1000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5039.079579] LustreError: 82525:0:(ldlm_flock.c:858:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5041.801366] LustreError: lustre-MDT0000-mdc-ffff8ac790be1000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5041.846633] LustreError: 82572:0:(file.c:6098:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5041.859278] LustreError: 82572:0:(file.c:6098:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5043.233502] LustreError: 82553:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff8ac790be1000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5050.509000] LustreError: 82624:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5050.516731] LustreError: 82624:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5053.568990] LustreError: lustre-MDT0000-mdc-ffff8ac790be1000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5054.583754] LustreError: 82624:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5054.590576] LustreError: 82624:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5054.609844] LustreError: 82624:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff8ac790be1000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5062.500299] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 02:59:48 (1779692388)
[ 5079.925453] LustreError: 79065:0:(ldlm_lockd.c:2883:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5079.931343] LustreError: 79065:0:(ldlm_lockd.c:2883:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5083.999141] LustreError: 79065:0:(ldlm_lockd.c:2883:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5084.005829] LustreError: 79065:0:(ldlm_lockd.c:2883:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5088.299920] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 03:00:14 (1779692414)
[ 5135.267946] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 03:01:01 (1779692461)
[ 5191.647158] Lustre: 84811:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779692463/real 1779692463]  req@ffff8ac79aa9ed80 x1866137528804992/t0(0) o101->lustre-MDT0000-mdc-ffff8ac790be1000@192.168.204.134@tcp:12/10 lens 576/1152 e 0 to 1 dl 1779692518 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'stat.0' uid:0 gid:0 projid:0
[ 5191.663779] Lustre: 84811:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 10 previous similar messages
[ 5195.491682] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 03:02:01 (1779692521)
[ 5224.768658] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 03:02:30 (1779692550)
[ 5225.060329] Lustre: Mounted lustre-client
[ 5347.575027] LustreError: 86344:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff8ac783556800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 5347.588421] LustreError: 86344:0:(lov_obd.c:792:lov_cleanup()) Skipped 1 previous similar message
[ 5347.628867] Lustre: Unmounted lustre-client
[ 5352.254663] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 03:04:38 (1779692678)
[ 5352.770970] LustreError: 86958:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5354.631825] LustreError: lustre-OST0000-osc-ffff8ac790be1000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5354.647793] LustreError: 87046:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-OST0000-osc-ffff8ac790be1000: namespace resource [0x240000400:0x4dad:0x0].0x0 (ffff8ac790648000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5354.660573] LustreError: 87046:0:(ldlm_resource.c:1180:ldlm_resource_complain()) Skipped 3 previous similar messages
[ 5354.874430] LustreError: 86958:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5354.883133] Lustre: 2362:0:(llite_lib.c:4187:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.134@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5359.267175] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 03:04:45 (1779692685)
[ 5416.927817] Lustre: lustre-MDT0000-mdc-ffff8ac790be1000: Connection to lustre-MDT0000 (at 192.168.204.134@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5416.935110] Lustre: Skipped 18 previous similar messages
[ 5416.949213] Lustre: lustre-MDT0000-mdc-ffff8ac790be1000: Connection restored to 192.168.204.134@tcp (at 192.168.204.134@tcp)
[ 5416.955448] Lustre: Skipped 19 previous similar messages
[ 5420.700153] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 03:05:46 (1779692746)
[ 5421.541375] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 5422.516234] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 03:05:48 (1779692748)
[ 5482.002152] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 5482.935452] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 03:06:49 (1779692809)
[ 5539.602521] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 03:07:45 (1779692865)
[ 5540.802436] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 5541.698647] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 03:07:47 (1779692867)
[ 5542.569174] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 5543.403632] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 03:07:49 (1779692869)
[ 5561.056394] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 03:08:07 (1779692887)
[ 5566.986770] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5589.985864] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 192.168.204.134@tcp) was lost; in progress operations using this service will fail
[ 5589.996181] LustreError: Skipped 2 previous similar messages
[ 5590.045371] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0x42ce1623a5596d01 to 0x42ce1623a55986fb
[ 5590.066293] Lustre: Skipped 2 previous similar messages
[ 5607.383546] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5609.621920] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5624.277313] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 03:09:09 (1779692949)
[ 5628.351907] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 5630.644129] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 03:09:15 (1779692955)
[ 5664.350541] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 03:09:50 (1779692990)
[ 5717.157387] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 03:10:42 (1779693042)
[ 5723.028115] LustreError: lustre-OST0000-osc-ffff8ac790be1000: operation ost_setattr to node 192.168.204.134@tcp failed: rc = -19
[ 5723.038075] LustreError: Skipped 8 previous similar messages
[ 5765.851668] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5769.181418] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5855.199758] Lustre: 2362:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779693166/real 1779693166]  req@ffff8ac7afaf7480 x1866137535376768/t0(0) o400->MGC192.168.204.134@tcp@192.168.204.134@tcp:26/25 lens 224/224 e 0 to 1 dl 1779693182 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5855.222494] Lustre: 2362:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 13 previous similar messages
[ 5866.604875] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5868.412774] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5904.734614] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5906.526394] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6037.587522] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 03:16:02 (1779693362)
[ 6045.570320] Lustre: lustre-OST0000-osc-ffff8ac790be1000: Connection to lustre-OST0000 (at 192.168.204.134@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6045.599111] Lustre: Skipped 8 previous similar messages
[ 6091.158534] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 6095.048281] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6320.273514] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 03:20:46 (1779693646)
[ 6506.469027] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 192.168.204.134@tcp) was lost; in progress operations using this service will fail
[ 6506.492612] LustreError: Skipped 4 previous similar messages
[ 6506.518943] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0x42ce1623a559a1f8 to 0x42ce1623a5682ea7
[ 6506.533426] Lustre: Skipped 4 previous similar messages
[ 6506.547071] Lustre: MGC192.168.204.134@tcp: Connection restored to 192.168.204.134@tcp (at 192.168.204.134@tcp)
[ 6506.556648] Lustre: Skipped 12 previous similar messages
[ 6542.275920] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 03:24:27 (1779693867)
[ 6543.695832] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 6545.335215] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 03:24:31 (1779693871)
[ 6547.218890] LustreError: lustre-MDT0000-mdc-ffff8ac790be1000: operation ldlm_enqueue to node 192.168.204.134@tcp failed: rc = -107
[ 6547.225927] LustreError: Skipped 448 previous similar messages
[ 6547.235489] LustreError: lustre-MDT0000-mdc-ffff8ac790be1000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6547.262187] LustreError: 101112:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8ac790be1000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff8ac782cea800) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6547.277847] LustreError: 101112:0:(ldlm_resource.c:1180:ldlm_resource_complain()) Skipped 1 previous similar message
[ 6555.903105] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 03:24:41 (1779693881)
[ 6723.187522] LustreError: lustre-OST0000-osc-ffff8ac790be1000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6725.261767] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 03:27:30 (1779694050)
[ 6751.199424] Lustre: 2364:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779694057/real 1779694057]  req@ffff8ac791279500 x1866137545306624/t0(0) o4->lustre-OST0000-osc-ffff8ac790be1000@192.168.204.134@tcp:6/4 lens 4584/448 e 0 to 1 dl 1779694077 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 6751.253115] Lustre: 2364:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
[ 6751.258912] Lustre: lustre-OST0000-osc-ffff8ac790be1000: Connection to lustre-OST0000 (at 192.168.204.134@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6751.276331] Lustre: Skipped 3 previous similar messages
[ 6768.871821] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 03:28:14 (1779694094)
[ 6770.864768] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 6772.912997] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 03:28:18 (1779694098)
[ 6774.582368] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 6776.771384] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 03:28:22 (1779694102)
[ 6813.221564] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 03:28:58 (1779694138)
[ 6871.900512] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 03:29:57 (1779694197)
[ 6873.402803] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 6875.151737] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 03:30:00 (1779694200)
[ 6876.774218] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 6879.069062] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 03:30:04 (1779694204)
[ 6895.740721] LustreError: 105852:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff8ac790be1000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 6895.749677] LustreError: 105852:0:(lov_obd.c:792:lov_cleanup()) Skipped 1 previous similar message
[ 6895.926190] Lustre: Unmounted lustre-client
[ 6897.956582] Lustre: Mounted lustre-client
[ 6900.948627] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6940.775329] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2