[ 1983.673703] LustreError: lustre-OST0000-osc-ffffa09b0ab14000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1983.713229] Lustre: 2374:0:(llite_lib.c:4187:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.125@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1994.656054] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 03:41:43 (1779694903)
[ 1997.682678] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 2000.155967] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 03:41:49 (1779694909)
[ 2001.738390] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 2004.285722] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 03:41:53 (1779694913)
[ 2008.521599] LustreError: lustre-MDT0000-mdc-ffffa09b0ab14000: operation ldlm_enqueue to node 192.168.204.125@tcp failed: rc = -19
[ 2008.530956] LustreError: Skipped 1 previous similar message
[ 2028.511580] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 192.168.204.125@tcp) was lost; in progress operations using this service will fail
[ 2028.541883] Lustre: Evicted from MGS (at 192.168.204.125@tcp) after server handle changed from 0x799e3d1c008b66e3 to 0x799e3d1c008b7e8a
[ 2133.435533] LustreError: lustre-MDT0000-mdc-ffffa09b0ab14000: operation mds_reint to node 192.168.204.125@tcp failed: rc = -19
[ 2133.443966] LustreError: Skipped 1 previous similar message
[ 2150.369952] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 192.168.204.125@tcp) was lost; in progress operations using this service will fail
[ 2160.693751] Lustre: Evicted from MGS (at 192.168.204.125@tcp) after server handle changed from 0x799e3d1c008b7e8a to 0x799e3d1c008d4f70
[ 2167.791036] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 03:44:37 (1779695077)
[ 2168.193742] Lustre: *** cfs_fail_loc=305, val=0***
[ 2168.197132] Lustre: Skipped 2 previous similar messages
[ 2212.639440] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 192.168.204.125@tcp) was lost; in progress operations using this service will fail
[ 2212.663697] Lustre: Evicted from MGS (at 192.168.204.125@tcp) after server handle changed from 0x799e3d1c008d4f70 to 0x799e3d1c008d528e
[ 2223.933621] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2225.568159] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2233.796372] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 03:45:43 (1779695143)
[ 2247.676555] Lustre: lustre-MDT0000-mdc-ffffa09b0ab14000: Connection to lustre-MDT0000 (at 192.168.204.125@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2247.683551] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 192.168.204.125@tcp) was lost; in progress operations using this service will fail
[ 2247.710782] Lustre: Skipped 12 previous similar messages
[ 2247.761750] Lustre: Evicted from MGS (at 192.168.204.125@tcp) after server handle changed from 0x799e3d1c008d528e to 0x799e3d1c008d591e
[ 2247.798321] Lustre: MGC192.168.204.125@tcp: Connection restored to 192.168.204.125@tcp (at 192.168.204.125@tcp)
[ 2247.804609] Lustre: Skipped 15 previous similar messages
[ 2247.823635] LustreError: lustre-MDT0000-mdc-ffffa09b0ab14000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2274.304843] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 03:46:23 (1779695183)
[ 2289.380487] LustreError: lustre-OST0000-osc-ffffa09b0ab14000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2310.936987] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 03:47:00 (1779695220)
[ 2325.199889] LustreError: lustre-MDT0000-mdc-ffffa09b0ab14000: operation ldlm_enqueue to node 192.168.204.125@tcp failed: rc = -19
[ 2325.222030] LustreError: Skipped 3 previous similar messages
[ 2344.927918] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 192.168.204.125@tcp) was lost; in progress operations using this service will fail
[ 2344.950618] Lustre: Evicted from MGS (at 192.168.204.125@tcp) after server handle changed from 0x799e3d1c008d591e to 0x799e3d1c008dac22
[ 2370.024137] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2372.145312] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2457.631157] Lustre: 2373:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779695352/real 1779695352]  req@ffffa09b0770ca80 x1866143418101632/t0(0) o400->MGC192.168.204.125@tcp@192.168.204.125@tcp:26/25 lens 224/224 e 0 to 1 dl 1779695368 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2457.664116] Lustre: 2373:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 14 previous similar messages
[ 2457.673219] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 192.168.204.125@tcp) was lost; in progress operations using this service will fail
[ 2466.865508] Lustre: Evicted from MGS (at 192.168.204.125@tcp) after server handle changed from 0x799e3d1c008dac22 to 0x799e3d1c008f5d50
[ 2480.962900] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2483.244392] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2568.695340] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 192.168.204.125@tcp) was lost; in progress operations using this service will fail
[ 2568.779695] Lustre: Evicted from MGS (at 192.168.204.125@tcp) after server handle changed from 0x799e3d1c008f5d50 to 0x799e3d1c0091064a
[ 2570.228701] Lustre: 15900:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.125@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2595.559445] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2599.040845] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2628.846072] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 03:52:18 (1779695538)
[ 2632.921614] LustreError: lustre-MDT0000-mdc-ffffa09b0ab14000: operation mds_reint to node 192.168.204.125@tcp failed: rc = -19
[ 2632.931194] LustreError: Skipped 2 previous similar messages
[ 2650.592263] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 192.168.204.125@tcp) was lost; in progress operations using this service will fail
[ 2659.827208] Lustre: Evicted from MGS (at 192.168.204.125@tcp) after server handle changed from 0x799e3d1c0091064a to 0x799e3d1c0091cd6f
[ 2662.363553] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2693.378532] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2722.152961] Lustre: 15900:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.125@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2738.695588] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2780.959866] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2823.648343] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 192.168.204.125@tcp) was lost; in progress operations using this service will fail
[ 2823.667895] LustreError: Skipped 3 previous similar messages
[ 2823.685668] Lustre: Evicted from MGS (at 192.168.204.125@tcp) after server handle changed from 0x799e3d1c00923692 to 0x799e3d1c0092f583
[ 2823.698160] Lustre: Skipped 3 previous similar messages
[ 2824.953486] Lustre: 15900:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.125@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2843.314320] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2871.566041] Lustre: lustre-MDT0000-mdc-ffffa09b0ab14000: Connection to lustre-MDT0000 (at 192.168.204.125@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2871.575367] Lustre: Skipped 8 previous similar messages
[ 2899.459761] Lustre: MGC192.168.204.125@tcp: Connection restored to 192.168.204.125@tcp (at 192.168.204.125@tcp)
[ 2899.481054] Lustre: Skipped 17 previous similar messages
[ 2909.202028] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2995.101410] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 03:58:24 (1779695904)
[ 3048.039245] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3050.502748] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3339.788225] LustreError: lustre-OST0000-osc-ffffa09b0ab14000: operation ost_write to node 192.168.204.125@tcp failed: rc = -19
[ 3339.800525] LustreError: Skipped 11 previous similar messages
[ 3381.496192] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3384.246964] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3671.147975] Lustre: lustre-OST0000-osc-ffffa09b0ab14000: Connection to lustre-OST0000 (at 192.168.204.125@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3671.161989] Lustre: Skipped 3 previous similar messages
[ 3695.719565] Lustre: lustre-OST0000-osc-ffffa09b0ab14000: Connection restored to 192.168.204.125@tcp (at 192.168.204.125@tcp)
[ 3695.735769] Lustre: Skipped 5 previous similar messages
[ 3710.856035] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3713.053858] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3967.366886] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 04:14:37 (1779696877)
[ 3985.378870] Lustre: 50194:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779696880/real 1779696880]  req@ffffa09b06a54000 x1866143441104640/t0(0) o101->lustre-MDT0000-mdc-ffffa09b0ab14000@192.168.204.125@tcp:12/10 lens 576/1152 e 0 to 1 dl 1779696896 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3985.430960] Lustre: 50194:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
[ 3994.597747] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 04:15:04 (1779696904)
[ 4021.526721] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 04:15:31 (1779696931)
[ 4048.875699] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 04:15:58 (1779696958)
[ 4049.321722] Lustre: Mounted lustre-client
[ 4080.863169] Lustre: 2373:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779696976/real 1779696976]  req@ffffa09b06a54e00 x1866143441124480/t0(0) o400->lustre-MDT0000-mdc-ffffa09b0ab14000@192.168.204.125@tcp:12/10 lens 224/224 e 0 to 1 dl 1779696992 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4080.887293] Lustre: 2373:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 4080.896324] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 192.168.204.125@tcp) was lost; in progress operations using this service will fail
[ 4080.910639] LustreError: Skipped 2 previous similar messages
[ 4080.932206] Lustre: Evicted from MGS (at 192.168.204.125@tcp) after server handle changed from 0x799e3d1c00948c9b to 0x799e3d1c00aa3543
[ 4080.939375] Lustre: Skipped 2 previous similar messages
[ 4092.820563] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4094.522303] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4096.741975] LustreError: 52953:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffffa09b07228000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 4096.754080] LustreError: 52953:0:(lov_obd.c:792:lov_cleanup()) Skipped 1 previous similar message
[ 4096.836191] Lustre: Unmounted lustre-client
[ 4102.959620] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 04:16:53 (1779697013)
[ 4240.736280] Lustre: 2373:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779697136/real 1779697136]  req@ffffa09b06a54700 x1866143441158400/t0(0) o4->lustre-OST0000-osc-ffffa09b0ab14000@192.168.204.125@tcp:6/4 lens 488/448 e 0 to 1 dl 1779697152 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4240.755043] Lustre: 2373:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 55 previous similar messages
[ 4273.631253] Lustre: lustre-OST0000-osc-ffffa09b0ab14000: Connection to lustre-OST0000 (at 192.168.204.125@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4273.652246] Lustre: Skipped 14 previous similar messages
[ 4305.215293] Lustre: lustre-OST0000-osc-ffffa09b0ab14000: Connection restored to 192.168.204.125@tcp (at 192.168.204.125@tcp)
[ 4305.230899] Lustre: Skipped 17 previous similar messages
[ 4399.004899] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 04:21:48 (1779697308)
[ 4447.872048] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 04:22:37 (1779697357)
[ 4451.047918] LustreError: 55109:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffffa09b0ab14000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 4451.056928] LustreError: 55109:0:(lov_obd.c:792:lov_cleanup()) Skipped 1 previous similar message
[ 4451.121245] Lustre: Unmounted lustre-client
[ 4473.392606] Lustre: Mounted lustre-client
[ 4480.184712] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 04:23:09 (1779697389)
[ 4480.519065] LustreError: 56090:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4481.567080] LustreError: 56090:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4481.730273] Lustre: *** cfs_fail_loc=305, val=0***
[ 4505.974710] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 04:23:35 (1779697415)
[ 4506.765548] Lustre: Mounted lustre-client
[ 4509.958409] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4520.325764] Lustre: Unmounted lustre-client
[ 4528.820992] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 04:23:58 (1779697438)
[ 4634.060615] LustreError: lustre-MDT0000-mdc-ffffa09b18049000: operation mds_reint to node 192.168.204.125@tcp failed: rc = -19
[ 4634.082154] LustreError: Skipped 2 previous similar messages
[ 4651.487669] Lustre: 2375:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779697546/real 1779697546]  req@ffffa09b1110c380 x1866143443835008/t0(0) o400->MGC192.168.204.125@tcp@192.168.204.125@tcp:26/25 lens 224/224 e 0 to 1 dl 1779697562 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4651.518765] Lustre: 2375:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 73 previous similar messages
[ 4651.534826] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 192.168.204.125@tcp) was lost; in progress operations using this service will fail
[ 4661.732625] Lustre: Evicted from MGS (at 192.168.204.125@tcp) after server handle changed from 0x799e3d1c00aa5190 to 0x799e3d1c00ad052b
[ 4838.322981] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 04:29:07 (1779697747)
[ 4845.101701] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4859.375336] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 192.168.204.125@tcp) was lost; in progress operations using this service will fail
[ 4859.425325] Lustre: Evicted from MGS (at 192.168.204.125@tcp) after server handle changed from 0x799e3d1c00ad052b to 0x799e3d1c00b1ce78
[ 4860.826445] LustreError: lustre-MDT0000-mdc-ffffa09b18049000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4878.613842] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 04:29:48 (1779697788)
[ 4879.296424] Lustre: Mounted lustre-client
[ 4887.482608] LustreError: lustre-OST0000-osc-ffffa09b18049000: operation ldlm_enqueue to node 192.168.204.125@tcp failed: rc = -107
[ 4887.496395] Lustre: lustre-OST0000-osc-ffffa09b18049000: Connection to lustre-OST0000 (at 192.168.204.125@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4887.510993] Lustre: Skipped 11 previous similar messages
[ 4887.541127] LustreError: lustre-OST0000-osc-ffffa09b18049000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4899.619212] LustreError: 59832:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffffa09b0ab17800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 4899.649334] LustreError: 59832:0:(lov_obd.c:792:lov_cleanup()) Skipped 1 previous similar message
[ 4899.734417] Lustre: Unmounted lustre-client
[ 4908.829351] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 04:30:18 (1779697818)
[ 4913.128327] Lustre: lustre-MDT0000-mdc-ffffa09b18049000: Connection restored to 192.168.204.125@tcp (at 192.168.204.125@tcp)
[ 4913.141412] Lustre: Skipped 13 previous similar messages
[ 4916.085878] LustreError: lustre-MDT0000-mdc-ffffa09b18049000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4916.118714] LustreError: 60474:0:(file.c:6098:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4924.173473] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 04:30:33 (1779697833)
[ 4924.381540] LustreError: 61173:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4928.460427] LustreError: lustre-MDT0000-mdc-ffffa09b18049000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4928.483843] LustreError: 61195:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4928.507461] LustreError: 61173:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=880
[ 4928.521286] LustreError: 61173:0:(import.c:719:ptlrpc_connect_import_locked()) already connecting
[ 4928.582634] LustreError: 61200:0:(file.c:6098:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4929.659764] LustreError: 61206:0:(file.c:6098:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4929.673736] LustreError: 61206:0:(file.c:6098:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4932.002360] LustreError: 61229:0:(file.c:6098:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4932.016682] LustreError: 61229:0:(file.c:6098:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4936.695561] LustreError: 61274:0:(file.c:6098:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4936.723036] LustreError: 61274:0:(file.c:6098:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4948.208780] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 04:30:57 (1779697857)
[ 4990.411111] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4992.249906] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5004.706743] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 04:31:54 (1779697914)
[ 5046.254214] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5048.169832] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5060.681418] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 04:32:49 (1779697969)
[ 5127.843186] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5130.069519] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5139.596547] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 04:34:09 (1779698049)
[ 5182.636155] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5184.493436] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5192.339855] LustreError: 66850:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffffa09b18049000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 5192.365165] LustreError: 66850:0:(lov_obd.c:792:lov_cleanup()) Skipped 1 previous similar message
[ 5192.443512] Lustre: Unmounted lustre-client
[ 5243.395390] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5246.371060] Lustre: Mounted lustre-client
[ 5254.639965] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 04:36:04 (1779698164)
[ 5258.090064] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5260.944875] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 04:36:09 (1779698169)
[ 5292.725991] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 04:36:41 (1779698201)
[ 5294.806689] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5297.511993] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 04:36:46 (1779698206)
[ 5299.363776] Lustre: *** cfs_fail_loc=805, val=0***
[ 5299.441985] Lustre: Mounted lustre-client
[ 5306.120645] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5326.303309] Lustre: 2374:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779698221/real 1779698221]  req@ffffa09b0859b100 x1866143445888512/t0(0) o400->lustre-MDT0000-mdc-ffffa09b0ab46800@192.168.204.125@tcp:12/10 lens 224/224 e 0 to 1 dl 1779698237 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5326.379691] Lustre: 2374:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5326.388426] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 192.168.204.125@tcp) was lost; in progress operations using this service will fail
[ 5336.556843] Lustre: Evicted from MGS (at 192.168.204.125@tcp) after server handle changed from 0x799e3d1c00b1dadb to 0x799e3d1c00b1dcf6
[ 5350.896955] LustreError: 2371:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-MDT0000-mdc-ffffa09b21760800: namespace resource [0x200000007:0x1:0x0].0x0 (ffffa09b105c4500) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5354.588060] LustreError: 70186:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffffa09b21760800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 5354.600320] LustreError: 70186:0:(lov_obd.c:792:lov_cleanup()) Skipped 1 previous similar message
[ 5354.705357] Lustre: Unmounted lustre-client
[ 5364.578579] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 04:37:54 (1779698274)
[ 5391.931899] LustreError: 2371:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffffa09b104fb800 x1866143445900416/t90194313220(90194313220) o101->lustre-MDT0000-mdc-ffffa09b0ab46800@192.168.204.125@tcp:12/10 lens 664/608 e 0 to 0 dl 1779698319 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5401.188936] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5402.641617] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5410.251938] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 04:38:40 (1779698320)
[ 5411.550593] LustreError: lustre-OST0000-osc-ffffa09b0ab46800: operation ost_write to node 192.168.204.125@tcp failed: rc = -107
[ 5411.556582] LustreError: Skipped 2 previous similar messages
[ 5411.573925] LustreError: lustre-OST0000-osc-ffffa09b0ab46800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5411.592918] Lustre: 2372:0:(llite_lib.c:4187:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.125@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5411.623895] LustreError: 72180:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-OST0000-osc-ffffa09b0ab46800: namespace resource [0x240000400:0x4062:0x0].0x0 (ffffa09b104f3d00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5411.637331] LustreError: 72180:0:(ldlm_resource.c:1180:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5420.289447] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 04:38:50 (1779698330)
[ 5421.728601] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5424.035892] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 04:38:53 (1779698333)
[ 5425.540142] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5427.187598] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 04:38:57 (1779698337)
[ 5428.670045] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5430.838541] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 04:39:00 (1779698340)
[ 5432.332334] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5434.435354] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 04:39:04 (1779698344)
[ 5436.268769] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5439.082466] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 04:39:08 (1779698348)
[ 5441.142208] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5443.182357] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 04:39:12 (1779698352)
[ 5444.813410] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5446.759519] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 04:39:16 (1779698356)
[ 5448.581268] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5451.142596] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 04:39:20 (1779698360)
[ 5452.558346] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5454.132762] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 04:39:24 (1779698364)
[ 5455.379719] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5457.138289] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 04:39:26 (1779698366)
[ 5458.584849] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5460.400320] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 04:39:30 (1779698370)
[ 5463.187789] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5465.566558] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 04:39:34 (1779698374)
[ 5497.730170] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 04:40:07 (1779698407)
[ 5527.004602] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 04:40:37 (1779698437)
[ 5527.468871] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5538.148447] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 04:40:47 (1779698447)
[ 5538.496323] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5548.892790] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 04:40:58 (1779698458)
[ 5549.213194] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5557.018814] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 04:41:07 (1779698467)
[ 5557.313234] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5564.984438] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 04:41:15 (1779698475)
[ 5565.279872] Lustre: *** cfs_fail_loc=510, val=3***
[ 5574.693892] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 04:41:24 (1779698484)
[ 5575.227707] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5635.039248] Lustre: lustre-OST0000-osc-ffffa09b0ab46800: Connection to lustre-OST0000 (at 192.168.204.125@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5635.054651] Lustre: Skipped 15 previous similar messages
[ 5642.027144] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 04:42:32 (1779698552)
[ 5642.484212] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5709.318322] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 04:43:39 (1779698619)
[ 5709.711356] LustreError: 82381:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5712.860645] LustreError: lustre-MDT0000-mdc-ffffa09b0ab46800: operation ldlm_enqueue to node 192.168.204.125@tcp failed: rc = -107
[ 5712.886441] LustreError: lustre-MDT0000-mdc-ffffa09b0ab46800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5712.920395] LustreError: 82396:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-MDT0000-mdc-ffffa09b0ab46800: namespace resource [0x20000a042:0x11:0x0].0xc (ffffa09b104f3d00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5712.961415] Lustre: lustre-MDT0000-mdc-ffffa09b0ab46800: Connection restored to 192.168.204.125@tcp (at 192.168.204.125@tcp)
[ 5712.977872] Lustre: Skipped 13 previous similar messages
[ 5713.775158] LustreError: 82381:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5715.928778] LustreError: 82412:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5719.071264] LustreError: lustre-MDT0000-mdc-ffffa09b0ab46800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5719.092041] LustreError: 82426:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-MDT0000-mdc-ffffa09b0ab46800: namespace resource [0x20000a042:0x11:0x0].0xc (ffffa09b105c4c00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5719.132934] LustreError: 82426:0:(ldlm_resource.c:1180:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5719.999118] LustreError: 82412:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5727.058188] LustreError: lustre-MDT0000-mdc-ffffa09b0ab46800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5727.562900] LustreError: 82468:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5730.427753] LustreError: lustre-MDT0000-mdc-ffffa09b0ab46800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5730.538470] LustreError: 82488:0:(file.c:6098:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5730.550784] LustreError: 82488:0:(file.c:6098:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5731.647147] LustreError: 82468:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5731.653708] LustreError: 82494:0:(file.c:6098:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5731.666951] LustreError: 82468:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffffa09b0ab46800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5731.705067] LustreError: 82494:0:(file.c:6098:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5734.075645] LustreError: 82516:0:(file.c:6098:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5734.098809] LustreError: 82516:0:(file.c:6098:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5737.890141] LustreError: 82543:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5740.700836] LustreError: lustre-MDT0000-mdc-ffffa09b0ab46800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5740.719115] LustreError: 82558:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-MDT0000-mdc-ffffa09b0ab46800: namespace resource [0x20000a042:0x11:0x0].0xc (ffffa09b104f3000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5740.764810] LustreError: 82558:0:(ldlm_resource.c:1180:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5741.959672] LustreError: 82543:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5748.992775] LustreError: lustre-MDT0000-mdc-ffffa09b0ab46800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5749.403168] LustreError: 82599:0:(ldlm_flock.c:858:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5752.584455] LustreError: lustre-MDT0000-mdc-ffffa09b0ab46800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5752.610642] LustreError: 82614:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-MDT0000-mdc-ffffa09b0ab46800: namespace resource [0x20000a042:0x11:0x0].0xc (ffffa09b0a943200) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5752.647931] LustreError: 82614:0:(ldlm_resource.c:1180:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5753.479134] LustreError: 82599:0:(ldlm_flock.c:858:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5756.496508] LustreError: lustre-MDT0000-mdc-ffffa09b0ab46800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5756.573198] LustreError: 82647:0:(file.c:6098:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5756.592880] LustreError: 82647:0:(file.c:6098:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5757.655675] LustreError: 82627:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffffa09b0ab46800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5765.598804] LustreError: 82699:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5765.606381] LustreError: 82699:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5769.671522] LustreError: lustre-MDT0000-mdc-ffffa09b0ab46800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5769.681259] LustreError: 82699:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5769.705978] LustreError: 82699:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5769.716236] LustreError: 82699:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffffa09b0ab46800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5769.796514] LustreError: 82721:0:(file.c:6098:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5769.816328] LustreError: 82721:0:(file.c:6098:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5781.489914] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 04:44:51 (1779698691)
[ 5801.687527] LustreError: 67682:0:(ldlm_lockd.c:2883:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5801.701306] LustreError: 67682:0:(ldlm_lockd.c:2883:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5805.776811] LustreError: 67682:0:(ldlm_lockd.c:2883:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5805.791288] LustreError: 67682:0:(ldlm_lockd.c:2883:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5814.338299] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 04:45:24 (1779698724)
[ 5865.712148] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 04:46:15 (1779698775)
[ 5931.449098] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 04:47:21 (1779698841)
[ 5967.001274] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 04:47:56 (1779698876)
[ 5967.579108] Lustre: Mounted lustre-client
[ 6093.248383] LustreError: 86413:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffffa09b0ab15000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 6093.263215] LustreError: 86413:0:(lov_obd.c:792:lov_cleanup()) Skipped 1 previous similar message
[ 6093.350603] Lustre: Unmounted lustre-client
[ 6100.568645] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 04:50:10 (1779699010)
[ 6101.246849] LustreError: 87028:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 6104.611933] LustreError: lustre-OST0000-osc-ffffa09b0ab46800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6104.630137] LustreError: 87111:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-OST0000-osc-ffffa09b0ab46800: namespace resource [0x240000400:0x408d:0x0].0x0 (ffffa09b0a943300) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6104.647208] LustreError: 87111:0:(ldlm_resource.c:1180:ldlm_resource_complain()) Skipped 1 previous similar message
[ 6105.007782] LustreError: 87028:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout interrupted
[ 6105.019149] Lustre: 2373:0:(llite_lib.c:4187:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.125@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 6112.506732] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 04:50:21 (1779699021)
[ 6170.591699] Lustre: 87695:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779699026/real 1779699026]  req@ffffa09b104f9500 x1866143446079104/t0(0) o101->lustre-MDT0000-mdc-ffffa09b0ab46800@192.168.204.125@tcp:12/10 lens 328/344 e 0 to 1 dl 1779699081 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 6170.608659] Lustre: 87695:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 19 previous similar messages
[ 6179.141682] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 04:51:28 (1779699088)
[ 6180.571946] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 6181.997961] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 04:51:32 (1779699092)
[ 6239.199859] Lustre: lustre-MDT0000-mdc-ffffa09b0ab46800: Connection to lustre-MDT0000 (at 192.168.204.125@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6239.219936] Lustre: Skipped 15 previous similar messages
[ 6246.774585] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6248.595989] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 04:52:38 (1779699158)
[ 6306.753335] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 04:53:36 (1779699216)
[ 6309.330892] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6311.300208] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 04:53:41 (1779699221)
[ 6313.000329] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6314.960191] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 04:53:44 (1779699224)
[ 6350.693750] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 04:54:20 (1779699260)
[ 6362.531136] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6390.780821] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 192.168.204.125@tcp) was lost; in progress operations using this service will fail
[ 6390.789045] LustreError: Skipped 2 previous similar messages
[ 6390.799563] Lustre: Evicted from MGS (at 192.168.204.125@tcp) after server handle changed from 0x799e3d1c00b1e913 to 0x799e3d1c00b20226
[ 6390.811607] Lustre: Skipped 2 previous similar messages
[ 6390.821426] Lustre: MGC192.168.204.125@tcp: Connection restored to 192.168.204.125@tcp (at 192.168.204.125@tcp)
[ 6390.827439] Lustre: Skipped 14 previous similar messages
[ 6405.538962] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6407.832523] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6419.941119] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 04:55:29 (1779699329)
[ 6422.177790] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6423.938625] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 04:55:33 (1779699333)
[ 6436.852515] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 192.168.204.125@tcp) was lost; in progress operations using this service will fail
[ 6436.888593] Lustre: Evicted from MGS (at 192.168.204.125@tcp) after server handle changed from 0x799e3d1c00b20226 to 0x799e3d1c00b205de
[ 6450.411570] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 04:55:59 (1779699359)
[ 6501.792724] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 04:56:51 (1779699411)
[ 6508.311236] LustreError: lustre-OST0000-osc-ffffa09b0ab46800: operation ost_setattr to node 192.168.204.125@tcp failed: rc = -19
[ 6508.326237] LustreError: Skipped 9 previous similar messages
[ 6560.788716] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 6565.674588] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6651.360087] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 192.168.204.125@tcp) was lost; in progress operations using this service will fail
[ 6651.376857] LustreError: Skipped 1 previous similar message
[ 6662.382504] Lustre: Evicted from MGS (at 192.168.204.125@tcp) after server handle changed from 0x799e3d1c00b208c4 to 0x799e3d1c00b21925
[ 6662.405259] Lustre: Skipped 1 previous similar message
[ 6664.633296] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6666.284783] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6690.484660] Lustre: 67686:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.125@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6711.438300] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6713.495493] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6855.190506] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 05:02:45 (1779699765)
[ 6865.975510] Lustre: lustre-OST0000-osc-ffffa09b0ab46800: Connection to lustre-OST0000 (at 192.168.204.125@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6865.992304] Lustre: Skipped 7 previous similar messages
[ 6913.194795] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 6917.886144] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 7167.461600] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 05:07:57 (1779700077)
[ 7323.619809] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 192.168.204.125@tcp) was lost; in progress operations using this service will fail
[ 7323.634380] LustreError: Skipped 1 previous similar message
[ 7323.648478] Lustre: Evicted from MGS (at 192.168.204.125@tcp) after server handle changed from 0x799e3d1c00b21964 to 0x799e3d1c00c1bd64
[ 7323.661667] Lustre: Skipped 1 previous similar message
[ 7323.667631] Lustre: MGC192.168.204.125@tcp: Connection restored to 192.168.204.125@tcp (at 192.168.204.125@tcp)
[ 7323.681378] Lustre: Skipped 9 previous similar messages
[ 7358.122222] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 05:11:08 (1779700268)
[ 7359.716513] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7361.239493] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 05:11:11 (1779700271)
[ 7362.945429] LustreError: lustre-MDT0000-mdc-ffffa09b0ab46800: operation ldlm_enqueue to node 192.168.204.125@tcp failed: rc = -107
[ 7362.955030] LustreError: Skipped 37 previous similar messages
[ 7362.976405] LustreError: lustre-MDT0000-mdc-ffffa09b0ab46800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7362.985237] LustreError: 101205:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-MDT0000-mdc-ffffa09b0ab46800: namespace resource [0x200000007:0x1:0x0].0x0 (ffffa09b217e6f00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 7363.000785] LustreError: 101205:0:(ldlm_resource.c:1180:ldlm_resource_complain()) Skipped 1 previous similar message
[ 7370.461244] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 05:11:20 (1779700280)
[ 7537.127257] LustreError: lustre-OST0000-osc-ffffa09b0ab46800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7539.979994] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 05:14:09 (1779700449)
[ 7567.839378] Lustre: 2373:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779700458/real 1779700458]  req@ffffa09b061db480 x1866143463784448/t0(0) o4->lustre-OST0000-osc-ffffa09b0ab46800@192.168.204.125@tcp:6/4 lens 4584/448 e 0 to 1 dl 1779700478 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7567.868498] Lustre: 2373:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 24 previous similar messages
[ 7567.876738] Lustre: lustre-OST0000-osc-ffffa09b0ab46800: Connection to lustre-OST0000 (at 192.168.204.125@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7567.895444] Lustre: Skipped 3 previous similar messages
[ 7585.490640] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 05:14:55 (1779700495)
[ 7586.930438] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7588.932451] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 05:14:58 (1779700498)
[ 7590.624996] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7592.203464] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 05:15:02 (1779700502)
[ 7625.625652] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 05:15:35 (1779700535)
[ 7665.127260] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 192.168.204.125@tcp) was lost; in progress operations using this service will fail
[ 7665.160645] Lustre: Evicted from MGS (at 192.168.204.125@tcp) after server handle changed from 0x799e3d1c00c1bd64 to 0x799e3d1c00c1e5d5
[ 7665.170409] Lustre: Skipped 1 previous similar message
[ 7680.554531] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 05:16:30 (1779700590)
[ 7682.079883] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7683.756231] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 05:16:33 (1779700593)
[ 7685.225525] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7687.180377] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 05:16:37 (1779700597)
[ 7691.743123] Lustre: 2375:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779700566/real 1779700566]  req@ffffa09b10885c00 x1866143463939840/t0(0) o400->lustre-MDT0000-mdc-ffffa09b0ab46800@192.168.204.125@tcp:12/10 lens 224/224 e 0 to 1 dl 1779700602 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7700.233200] LustreError: 105941:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffffa09b0ab46800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 7700.248756] LustreError: 105941:0:(lov_obd.c:792:lov_cleanup()) Skipped 1 previous similar message
[ 7700.434832] Lustre: Unmounted lustre-client
[ 7702.230224] Lustre: Mounted lustre-client
[ 7704.927705] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7731.835939] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2