[ 1963.720119] LustreError: lustre-OST0000-osc-ffff8e8e0b4e0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1963.766037] Lustre: 2360:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.138@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1969.798964] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 18:44:09 (1789685049)
[ 1971.582677] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1972.996663] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 18:44:12 (1789685052)
[ 1974.258366] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1975.592343] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 18:44:15 (1789685055)
[ 1978.941045] LustreError: lustre-MDT0000-mdc-ffff8e8e0b4e0000: operation mds_reint to node 192.168.201.138@tcp failed: rc = -19
[ 1978.947629] LustreError: Skipped 2 previous similar messages
[ 1998.307573] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 192.168.201.138@tcp) was lost; in progress operations using this service will fail
[ 1998.342207] Lustre: Evicted from MGS (at 192.168.201.138@tcp) after server handle changed from 0x6b16df555c7eedc7 to 0x6b16df555c7f08af
[ 2097.777329] LustreError: lustre-MDT0000-mdc-ffff8e8e0b4e0000: operation mds_reint to node 192.168.201.138@tcp failed: rc = -19
[ 2097.798121] LustreError: Skipped 1 previous similar message
[ 2116.063700] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 192.168.201.138@tcp) was lost; in progress operations using this service will fail
[ 2116.100742] Lustre: Evicted from MGS (at 192.168.201.138@tcp) after server handle changed from 0x6b16df555c7f08af to 0x6b16df555c817540
[ 2143.054624] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 18:47:02 (1789685222)
[ 2144.247460] Lustre: *** cfs_fail_loc=305, val=0***
[ 2144.250656] Lustre: Skipped 2 previous similar messages
[ 2187.743807] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 192.168.201.138@tcp) was lost; in progress operations using this service will fail
[ 2196.968084] Lustre: Evicted from MGS (at 192.168.201.138@tcp) after server handle changed from 0x6b16df555c817540 to 0x6b16df555c817b28
[ 2212.894730] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2214.857776] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2225.401613] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 18:48:25 (1789685305)
[ 2243.062146] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 192.168.201.138@tcp) was lost; in progress operations using this service will fail
[ 2243.064125] Lustre: lustre-MDT0000-mdc-ffff8e8e0b4e0000: Connection to lustre-MDT0000 (at 192.168.201.138@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2243.109147] Lustre: Skipped 12 previous similar messages
[ 2243.119048] Lustre: Evicted from MGS (at 192.168.201.138@tcp) after server handle changed from 0x6b16df555c817b28 to 0x6b16df555c817fa4
[ 2243.140768] Lustre: MGC192.168.201.138@tcp: Connection restored to 192.168.201.138@tcp (at 192.168.201.138@tcp)
[ 2243.149900] Lustre: Skipped 15 previous similar messages
[ 2243.221664] LustreError: lustre-MDT0000-mdc-ffff8e8e0b4e0000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2266.777182] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 18:49:06 (1789685346)
[ 2284.016162] LustreError: lustre-OST0000-osc-ffff8e8e0b4e0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2297.968474] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 18:49:37 (1789685377)
[ 2312.005436] LustreError: lustre-MDT0000-mdc-ffff8e8e0b4e0000: operation mds_reint to node 192.168.201.138@tcp failed: rc = -19
[ 2330.079369] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 192.168.201.138@tcp) was lost; in progress operations using this service will fail
[ 2340.268970] Lustre: Evicted from MGS (at 192.168.201.138@tcp) after server handle changed from 0x6b16df555c817fa4 to 0x6b16df555c81db14
[ 2357.805962] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2359.785508] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2424.879394] LustreError: lustre-MDT0000-mdc-ffff8e8e0b4e0000: operation mds_close to node 192.168.201.138@tcp failed: rc = -19
[ 2442.719233] Lustre: 2359:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789685507/real 1789685507]  req@ffff8e8e1096c700 x1876618837595264/t0(0) o400->MGC192.168.201.138@tcp@192.168.201.138@tcp:26/25 lens 224/224 e 0 to 1 dl 1789685523 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2442.745743] Lustre: 2359:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 17 previous similar messages
[ 2442.752983] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 192.168.201.138@tcp) was lost; in progress operations using this service will fail
[ 2452.967069] Lustre: Evicted from MGS (at 192.168.201.138@tcp) after server handle changed from 0x6b16df555c81db14 to 0x6b16df555c838e02
[ 2468.740571] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2470.572796] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2559.455312] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 192.168.201.138@tcp) was lost; in progress operations using this service will fail
[ 2569.643750] Lustre: Evicted from MGS (at 192.168.201.138@tcp) after server handle changed from 0x6b16df555c838e02 to 0x6b16df555c85461b
[ 2590.543546] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2592.792841] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2621.722313] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 18:55:01 (1789685701)
[ 2626.999162] LustreError: lustre-MDT0000-mdc-ffff8e8e0b4e0000: operation mds_reint to node 192.168.201.138@tcp failed: rc = -19
[ 2627.011551] LustreError: Skipped 2 previous similar messages
[ 2646.495762] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 192.168.201.138@tcp) was lost; in progress operations using this service will fail
[ 2656.790819] Lustre: Evicted from MGS (at 192.168.201.138@tcp) after server handle changed from 0x6b16df555c85461b to 0x6b16df555c861aad
[ 2669.983256] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2714.253634] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2752.376176] Lustre: 15939:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.138@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2765.476358] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2797.535330] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 192.168.201.138@tcp) was lost; in progress operations using this service will fail
[ 2797.553033] LustreError: Skipped 2 previous similar messages
[ 2807.725750] Lustre: Evicted from MGS (at 192.168.201.138@tcp) after server handle changed from 0x6b16df555c86409a to 0x6b16df555c8669cf
[ 2807.735344] Lustre: Skipped 2 previous similar messages
[ 2820.270841] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2844.833353] Lustre: lustre-MDT0000-mdc-ffff8e8e0b4e0000: Connection to lustre-MDT0000 (at 192.168.201.138@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2844.844326] Lustre: Skipped 7 previous similar messages
[ 2872.833300] Lustre: MGC192.168.201.138@tcp: Connection restored to 192.168.201.138@tcp (at 192.168.201.138@tcp)
[ 2872.836730] Lustre: Skipped 15 previous similar messages
[ 2887.140814] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2915.049540] LustreError: lustre-MDT0000-mdc-ffff8e8e0b4e0000: operation mds_reint to node 192.168.201.138@tcp failed: rc = -19
[ 2915.057981] LustreError: Skipped 3 previous similar messages
[ 2944.178810] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 3050.284736] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 19:02:08 (1789686128)
[ 3123.318981] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3127.232334] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3441.991281] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3444.604774] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3726.806134] LustreError: lustre-OST0000-osc-ffff8e8e0b4e0000: operation ldlm_enqueue to node 192.168.201.138@tcp failed: rc = -107
[ 3726.812828] LustreError: Skipped 3 previous similar messages
[ 3726.815238] Lustre: lustre-OST0000-osc-ffff8e8e0b4e0000: Connection to lustre-OST0000 (at 192.168.201.138@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3726.827708] Lustre: Skipped 4 previous similar messages
[ 3746.251277] Lustre: lustre-OST0000-osc-ffff8e8e0b4e0000: Connection restored to 192.168.201.138@tcp (at 192.168.201.138@tcp)
[ 3746.257683] Lustre: Skipped 7 previous similar messages
[ 3759.147437] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3760.528984] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4019.232545] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 19:18:19 (1789687099)
[ 4036.576654] Lustre: 50609:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789687101/real 1789687101]  req@ffff8e8e0c192680 x1876618863269504/t0(0) o101->lustre-MDT0000-mdc-ffff8e8e0b4e0000@192.168.201.138@tcp:12/10 lens 576/1152 e 0 to 1 dl 1789687117 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 4036.593058] Lustre: 50609:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
[ 4040.510252] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 19:18:40 (1789687120)
[ 4061.626739] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 19:19:02 (1789687142)
[ 4082.872852] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 19:19:23 (1789687163)
[ 4083.100840] Lustre: Mounted lustre-client - version 2.17.58_39_g12a437a
[ 4114.847899] Lustre: 2357:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789687179/real 1789687179]  req@ffff8e8e085b7100 x1876618863286784/t0(0) o400->lustre-MDT0000-mdc-ffff8e8e0b4e0000@192.168.201.138@tcp:12/10 lens 224/224 e 0 to 1 dl 1789687195 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4114.868457] Lustre: 2357:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 4114.911243] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 192.168.201.138@tcp) was lost; in progress operations using this service will fail
[ 4114.917662] LustreError: Skipped 3 previous similar messages
[ 4124.173146] Lustre: Evicted from MGS (at 192.168.201.138@tcp) after server handle changed from 0x6b16df555c884e65 to 0x6b16df555ca1a910
[ 4124.200449] Lustre: Skipped 3 previous similar messages
[ 4125.951688] Lustre: 15939:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.138@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 4157.241593] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4159.206767] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4161.829713] Lustre: Unmounted lustre-client
[ 4168.540831] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 19:20:48 (1789687248)
[ 4273.124109] Lustre: 2360:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789687338/real 1789687338]  req@ffff8e8e0c190700 x1876618863324544/t0(0) o4->lustre-OST0000-osc-ffff8e8e0b4e0000@192.168.201.138@tcp:6/4 lens 488/448 e 0 to 1 dl 1789687354 ref 2 fl Rpc:XQr/602/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4273.165546] Lustre: 2360:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 58 previous similar messages
[ 4337.631455] Lustre: lustre-OST0000-osc-ffff8e8e0b4e0000: Connection to lustre-OST0000 (at 192.168.201.138@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4337.647593] Lustre: Skipped 14 previous similar messages
[ 4353.031218] Lustre: lustre-OST0000-osc-ffff8e8e0b4e0000: Connection restored to 192.168.201.138@tcp (at 192.168.201.138@tcp)
[ 4353.048597] Lustre: Skipped 16 previous similar messages
[ 4463.612573] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 19:25:43 (1789687543)
[ 4516.229920] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 19:26:35 (1789687595)
[ 4519.718160] Lustre: Unmounted lustre-client
[ 4544.337362] Lustre: Mounted lustre-client - version 2.17.58_39_g12a437a
[ 4551.356869] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 19:27:11 (1789687631)
[ 4551.999933] LustreError: 56537:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4553.047523] LustreError: 56537:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4553.271679] Lustre: *** cfs_fail_loc=305, val=0***
[ 4577.393930] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 19:27:36 (1789687656)
[ 4577.985337] Lustre: Mounted lustre-client - version 2.17.58_39_g12a437a
[ 4580.464609] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4590.799113] Lustre: Unmounted lustre-client
[ 4598.557815] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 19:27:58 (1789687678)
[ 4690.494175] LustreError: lustre-MDT0000-mdc-ffff8e8e090ed000: operation mds_reint to node 192.168.201.138@tcp failed: rc = -19
[ 4711.402388] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 192.168.201.138@tcp) was lost; in progress operations using this service will fail
[ 4711.450426] Lustre: Evicted from MGS (at 192.168.201.138@tcp) after server handle changed from 0x6b16df555ca1c812 to 0x6b16df555ca47c2b
[ 4877.411651] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 19:32:36 (1789687956)
[ 4883.374571] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4900.916207] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 192.168.201.138@tcp) was lost; in progress operations using this service will fail
[ 4900.938404] Lustre: Evicted from MGS (at 192.168.201.138@tcp) after server handle changed from 0x6b16df555ca47c2b to 0x6b16df555ca94643
[ 4901.013115] LustreError: lustre-MDT0000-mdc-ffff8e8e090ed000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4921.239248] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 19:33:20 (1789688000)
[ 4921.936166] Lustre: Mounted lustre-client - version 2.17.58_39_g12a437a
[ 4929.454040] LustreError: lustre-OST0000-osc-ffff8e8e090ed000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4939.241997] Lustre: 2360:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789688004/real 1789688004]  req@ffff8e8e043fc380 x1876618867983872/t0(0) o103->lustre-OST0000-osc-ffff8e8e091a2800@192.168.201.138@tcp:17/18 lens 328/224 e 0 to 1 dl 1789688020 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295
[ 4939.285386] Lustre: 2360:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 115 previous similar messages
[ 4939.303320] Lustre: lustre-OST0000-osc-ffff8e8e091a2800: Connection to lustre-OST0000 (at 192.168.201.138@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4939.318273] Lustre: Skipped 12 previous similar messages
[ 4942.314852] Lustre: Unmounted lustre-client
[ 4951.812104] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 19:33:50 (1789688030)
[ 4955.701209] Lustre: lustre-MDT0000-mdc-ffff8e8e090ed000: Connection restored to 192.168.201.138@tcp (at 192.168.201.138@tcp)
[ 4955.712676] Lustre: Skipped 14 previous similar messages
[ 4958.878323] LustreError: lustre-MDT0000-mdc-ffff8e8e090ed000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4958.907237] LustreError: 60995:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4967.940452] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 19:34:07 (1789688047)
[ 4968.230873] LustreError: 61636:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4972.548175] LustreError: lustre-MDT0000-mdc-ffff8e8e090ed000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4972.587125] LustreError: 61653:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4972.601700] LustreError: 61636:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=634
[ 4972.605044] LustreError: 61636:0:(import.c:762:ptlrpc_connect_import_locked()) already connecting
[ 4972.678986] LustreError: 61658:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4973.831771] LustreError: 61664:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4973.837100] LustreError: 61664:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4976.244034] LustreError: 61687:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4976.256074] LustreError: 61687:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4981.045338] LustreError: 61732:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4981.074074] LustreError: 61732:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4992.044724] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 19:34:31 (1789688071)
[ 5036.135046] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5038.362454] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5049.705803] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 19:35:29 (1789688129)
[ 5091.407944] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5093.440477] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5106.797923] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 19:36:26 (1789688186)
[ 5174.841140] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5176.546540] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5185.870647] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 19:37:45 (1789688265)
[ 5228.723762] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5230.711492] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5238.001725] Lustre: Unmounted lustre-client
[ 5297.059756] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5299.686624] Lustre: Mounted lustre-client - version 2.17.58_39_g12a437a
[ 5306.345462] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 19:39:46 (1789688386)
[ 5309.521538] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5311.504890] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 19:39:50 (1789688390)
[ 5346.250302] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 19:40:25 (1789688425)
[ 5347.582656] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5349.642599] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 19:40:29 (1789688429)
[ 5351.221521] Lustre: *** cfs_fail_loc=805, val=0***
[ 5351.296987] Lustre: Mounted lustre-client - version 2.17.58_39_g12a437a
[ 5357.458540] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5376.992460] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 192.168.201.138@tcp) was lost; in progress operations using this service will fail
[ 5387.239630] Lustre: Evicted from MGS (at 192.168.201.138@tcp) after server handle changed from 0x6b16df555ca950ca to 0x6b16df555ca952e5
[ 5402.608746] LustreError: 2356:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8e8e10217800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff8e8e04164500) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5404.464461] Lustre: Unmounted lustre-client
[ 5411.935037] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 19:41:31 (1789688491)
[ 5454.903718] LustreError: 2356:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff8e8e041b9180 x1876618868066304/t90194313221(90194313221) o101->lustre-MDT0000-mdc-ffff8e8e04d82800@192.168.201.138@tcp:12/10 lens 648/608 e 0 to 0 dl 1789688551 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5460.299884] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5461.797913] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5471.478958] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 19:42:30 (1789688550)
[ 5473.108050] LustreError: lustre-OST0000-osc-ffff8e8e04d82800: operation ost_write to node 192.168.201.138@tcp failed: rc = -107
[ 5473.117956] LustreError: Skipped 3 previous similar messages
[ 5473.126130] LustreError: lustre-OST0000-osc-ffff8e8e04d82800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5473.149785] Lustre: 2357:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.138@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5473.172873] LustreError: 72852:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8e8e04d82800: namespace resource [0x240000400:0x4762:0x0].0x0 (ffff8e8e04295400) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5473.185733] LustreError: 72852:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5483.154564] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 19:42:42 (1789688562)
[ 5485.028984] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5487.484234] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 19:42:46 (1789688566)
[ 5489.376853] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5491.420809] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 19:42:50 (1789688570)
[ 5493.154759] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5495.117234] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 19:42:54 (1789688574)
[ 5496.660300] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5498.771309] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 19:42:58 (1789688578)
[ 5500.600135] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5502.590447] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 19:43:02 (1789688582)
[ 5504.517525] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5506.549774] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 19:43:06 (1789688586)
[ 5507.880619] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5509.449325] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 19:43:09 (1789688589)
[ 5510.829925] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5512.619869] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 19:43:12 (1789688592)
[ 5514.001651] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5515.740118] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 19:43:15 (1789688595)
[ 5517.114909] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5519.019352] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 19:43:18 (1789688598)
[ 5520.716924] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5522.077622] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 19:43:22 (1789688602)
[ 5524.546660] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5526.413871] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 19:43:26 (1789688606)
[ 5546.975281] Lustre: 2359:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789688612/real 1789688612]  req@ffff8e8e041b8000 x1876618868081536/t0(0) o400->lustre-MDT0000-mdc-ffff8e8e04d82800@192.168.201.138@tcp:12/10 lens 224/224 e 0 to 1 dl 1789688628 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5547.012045] Lustre: 2359:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 13 previous similar messages
[ 5547.023533] Lustre: lustre-MDT0000-mdc-ffff8e8e04d82800: Connection to lustre-MDT0000 (at 192.168.201.138@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5547.046815] Lustre: Skipped 11 previous similar messages
[ 5559.735738] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 19:43:59 (1789688639)
[ 5590.650178] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 19:44:30 (1789688670)
[ 5591.145971] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5602.099587] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 19:44:41 (1789688681)
[ 5602.499607] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5613.758606] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 19:44:52 (1789688692)
[ 5614.192171] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5623.835903] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 19:45:03 (1789688703)
[ 5624.266394] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5634.502189] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 19:45:13 (1789688713)
[ 5634.845592] Lustre: *** cfs_fail_loc=510, val=3***
[ 5643.835909] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 19:45:23 (1789688723)
[ 5644.451638] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5713.386865] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 19:46:32 (1789688792)
[ 5713.842828] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5781.341451] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 19:47:40 (1789688860)
[ 5781.707959] LustreError: 83068:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5784.586721] LustreError: lustre-MDT0000-mdc-ffff8e8e04d82800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5784.614330] LustreError: 83083:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8e8e04d82800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8e8e06996400) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5784.655477] Lustre: lustre-MDT0000-mdc-ffff8e8e04d82800: Connection restored to 192.168.201.138@tcp (at 192.168.201.138@tcp)
[ 5784.664525] Lustre: Skipped 12 previous similar messages
[ 5785.767216] LustreError: 83068:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5787.886219] LustreError: 83098:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5790.646366] LustreError: lustre-MDT0000-mdc-ffff8e8e04d82800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5790.681928] LustreError: 83112:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8e8e04d82800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8e8e04164900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5790.714393] LustreError: 83112:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5790.723533] LustreError: 83117:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5790.732297] LustreError: 83117:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5791.959825] LustreError: 83098:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5798.968051] LustreError: lustre-MDT0000-mdc-ffff8e8e04d82800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5799.483542] LustreError: 83154:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5802.447656] LustreError: lustre-MDT0000-mdc-ffff8e8e04d82800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5802.505645] LustreError: 83174:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5803.559575] LustreError: 83154:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5803.573480] LustreError: 83154:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8e8e04d82800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5804.756895] LustreError: 83191:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5804.777208] LustreError: 83191:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 12 previous similar messages
[ 5809.789291] LustreError: 83229:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5812.764070] LustreError: lustre-MDT0000-mdc-ffff8e8e04d82800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5812.788979] LustreError: 83244:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8e8e04d82800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff8e8e04164500) refcount nonzero (2) after lock cleanup; forcing cleanup.
[ 5812.829139] LustreError: 83244:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5812.861990] LustreError: 83249:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5812.880503] LustreError: 83249:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5813.889602] LustreError: 83229:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5820.822750] LustreError: lustre-MDT0000-mdc-ffff8e8e04d82800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5821.230582] LustreError: 83285:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5824.341793] LustreError: lustre-MDT0000-mdc-ffff8e8e04d82800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5824.367953] LustreError: 83300:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8e8e04d82800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8e8e06996000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5824.396775] LustreError: 83300:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 2 previous similar messages
[ 5825.311181] LustreError: 83285:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5828.552610] LustreError: lustre-MDT0000-mdc-ffff8e8e04d82800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5828.662420] LustreError: 83333:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5829.596127] LustreError: 83313:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8e8e04d82800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5837.716196] LustreError: 83385:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5837.731468] LustreError: 83385:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5841.379873] LustreError: lustre-MDT0000-mdc-ffff8e8e04d82800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5841.727179] LustreError: 83385:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5841.736745] LustreError: 83385:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5841.745590] LustreError: 83385:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8e8e04d82800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5845.082781] LustreError: 83436:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5845.098412] LustreError: 83436:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 46 previous similar messages
[ 5853.609660] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 19:48:53 (1789688933)
[ 5873.591854] LustreError: 68305:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5873.606024] LustreError: 68305:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5877.671281] LustreError: 68305:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5877.683307] LustreError: 68305:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5884.317742] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 19:49:24 (1789688964)
[ 5935.106656] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 19:50:15 (1789689015)
[ 6001.214895] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 19:51:20 (1789689080)
[ 6038.439184] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 19:51:57 (1789689117)
[ 6039.245354] Lustre: Mounted lustre-client - version 2.17.58_39_g12a437a
[ 6163.176522] Lustre: Unmounted lustre-client
[ 6171.635595] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 19:54:11 (1789689251)
[ 6172.534605] LustreError: 87714:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 6175.206078] LustreError: lustre-OST0000-osc-ffff8e8e04d82800: operation ldlm_enqueue to node 192.168.201.138@tcp failed: rc = -107
[ 6175.220774] LustreError: Skipped 8 previous similar messages
[ 6175.223979] Lustre: lustre-OST0000-osc-ffff8e8e04d82800: Connection to lustre-OST0000 (at 192.168.201.138@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6175.234864] Lustre: Skipped 14 previous similar messages
[ 6175.255601] LustreError: lustre-OST0000-osc-ffff8e8e04d82800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6175.268865] LustreError: 87796:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8e8e04d82800: namespace resource [0x240000400:0x478d:0x0].0x0 (ffff8e8e04267400) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6175.293882] LustreError: 87796:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 6175.584058] LustreError: 87714:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout interrupted
[ 6175.604414] Lustre: 2357:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.138@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 6183.288803] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 19:54:22 (1789689262)
[ 6242.271928] Lustre: 88382:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789689267/real 1789689267]  req@ffff8e8e04237b80 x1876618868243712/t0(0) o101->lustre-MDT0000-mdc-ffff8e8e04d82800@192.168.201.138@tcp:12/10 lens 328/344 e 0 to 1 dl 1789689323 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 6242.288413] Lustre: 88382:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
[ 6252.129813] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 19:55:31 (1789689331)
[ 6254.511066] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 6257.286812] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 19:55:36 (1789689336)
[ 6322.275563] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6323.753127] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 19:56:43 (1789689403)
[ 6385.266835] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 19:57:43 (1789689463)
[ 6388.085150] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6389.960524] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 19:57:49 (1789689469)
[ 6391.564708] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6393.191098] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 19:57:53 (1789689473)
[ 6434.010994] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 19:58:33 (1789689513)
[ 6449.052871] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6480.735291] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 192.168.201.138@tcp) was lost; in progress operations using this service will fail
[ 6480.748127] LustreError: Skipped 2 previous similar messages
[ 6480.783308] Lustre: Evicted from MGS (at 192.168.201.138@tcp) after server handle changed from 0x6b16df555ca95efb to 0x6b16df555ca97823
[ 6480.802881] Lustre: Skipped 2 previous similar messages
[ 6480.824208] Lustre: MGC192.168.201.138@tcp: Connection restored to 192.168.201.138@tcp (at 192.168.201.138@tcp)
[ 6480.845294] Lustre: Skipped 14 previous similar messages
[ 6497.052872] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6498.226209] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6509.316714] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 19:59:49 (1789689589)
[ 6512.312106] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6514.342185] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 19:59:53 (1789689593)
[ 6526.959727] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 192.168.201.138@tcp) was lost; in progress operations using this service will fail
[ 6527.001297] Lustre: Evicted from MGS (at 192.168.201.138@tcp) after server handle changed from 0x6b16df555ca97823 to 0x6b16df555ca97a76
[ 6541.569826] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 20:00:21 (1789689621)
[ 6592.959971] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 20:01:12 (1789689672)
[ 6662.751703] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6666.300703] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6752.161075] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 192.168.201.138@tcp) was lost; in progress operations using this service will fail
[ 6752.178443] LustreError: Skipped 1 previous similar message
[ 6762.474639] Lustre: Evicted from MGS (at 192.168.201.138@tcp) after server handle changed from 0x6b16df555ca97e0b to 0x6b16df555ca98ca5
[ 6762.485488] Lustre: Skipped 1 previous similar message
[ 6779.394636] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6781.107720] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6816.128150] Lustre: lustre-MDT0000-mdc-ffff8e8e04d82800: Connection to lustre-MDT0000 (at 192.168.201.138@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6816.173143] Lustre: Skipped 8 previous similar messages
[ 6833.261799] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6834.954468] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6848.482176] Lustre: 2358:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789689869/real 1789689869]  req@ffff8e8e08975180 x1876618874778368/t0(0) o400->lustre-MDT0000-mdc-ffff8e8e04d82800@192.168.201.138@tcp:12/10 lens 224/224 e 0 to 1 dl 1789689929 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6848.508101] Lustre: 2358:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 22 previous similar messages
[ 6983.153238] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 20:07:42 (1789690062)
[ 6993.407366] LustreError: lustre-OST0000-osc-ffff8e8e04d82800: operation ost_setattr to node 192.168.201.138@tcp failed: rc = -19
[ 6993.420262] LustreError: Skipped 65 previous similar messages
[ 7056.363069] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 7061.502985] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 7345.222199] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 20:13:45 (1789690425)
[ 7521.257902] Lustre: lustre-MDT0000-mdc-ffff8e8e04d82800: Connection to lustre-MDT0000 (at 192.168.201.138@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7521.271797] Lustre: Skipped 1 previous similar message
[ 7536.621234] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 192.168.201.138@tcp) was lost; in progress operations using this service will fail
[ 7536.638632] LustreError: Skipped 1 previous similar message
[ 7536.670422] Lustre: Evicted from MGS (at 192.168.201.138@tcp) after server handle changed from 0x6b16df555ca99048 to 0x6b16df555cb9345d
[ 7536.677818] Lustre: Skipped 1 previous similar message
[ 7536.690911] Lustre: MGC192.168.201.138@tcp: Connection restored to 192.168.201.138@tcp (at 192.168.201.138@tcp)
[ 7536.700799] Lustre: Skipped 9 previous similar messages
[ 7574.272629] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 20:17:34 (1789690654)
[ 7577.264365] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7579.166173] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 20:17:38 (1789690658)
[ 7583.318329] LustreError: lustre-MDT0000-mdc-ffff8e8e04d82800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7595.837362] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 20:17:54 (1789690674)
[ 7598.385535] LustreError: lustre-OST0000-osc-ffff8e8e04d82800: operation ost_statfs to node 192.168.201.138@tcp failed: rc = -107
[ 7598.395214] LustreError: Skipped 145 previous similar messages
[ 7763.110776] Lustre: Evicted from lustre-OST0000_UUID (at 192.168.201.138@tcp) after server handle changed from 0x6b16df555cb9393b to 0x6b16df555cb93a45
[ 7763.120725] LustreError: lustre-OST0000-osc-ffff8e8e04d82800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7765.278537] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 20:20:44 (1789690844)
[ 7790.687230] Lustre: 2358:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789690851/real 1789690851]  req@ffff8e8e0b7c0000 x1876618885979264/t0(0) o4->lustre-OST0000-osc-ffff8e8e04d82800@192.168.201.138@tcp:6/4 lens 4584/448 e 0 to 1 dl 1789690871 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7790.712043] Lustre: 2358:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 7811.687840] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 20:21:30 (1789690890)
[ 7813.698348] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7815.755697] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 20:21:35 (1789690895)
[ 7817.108261] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7819.613184] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 20:21:38 (1789690898)
[ 7855.326641] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 20:22:14 (1789690934)
[ 7897.611541] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 192.168.201.138@tcp) was lost; in progress operations using this service will fail
[ 7897.639280] Lustre: Evicted from MGS (at 192.168.201.138@tcp) after server handle changed from 0x6b16df555cb9345d to 0x6b16df555cb95d3e
[ 7918.502812] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 20:23:17 (1789690997)
[ 7920.335507] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7922.778558] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 20:23:21 (1789691001)
[ 7924.544414] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7926.890435] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 20:23:26 (1789691006)
[ 7938.045449] Lustre: Unmounted lustre-client
[ 7940.716244] Lustre: Mounted lustre-client - version 2.17.58_39_g12a437a
[ 7944.100481] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7966.176399] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 192.168.201.138@tcp) was lost; in progress operations using this service will fail
[ 7976.430915] Lustre: Evicted from MGS (at 192.168.201.138@tcp) after server handle changed from 0x6b16df555cb9676a to 0x6b16df555cb96938
[ 7985.184387] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2