[ 1994.916940] LustreError: lustre-OST0000-osc-ffff8d32c2e59000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1994.938621] Lustre: 2351:0:(llite_lib.c:4341:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.118@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 2003.288976] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 15:11:49 (1788462709)
[ 2005.895104] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 2008.204961] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 15:11:54 (1788462714)
[ 2010.959719] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 2013.270501] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 15:11:58 (1788462718)
[ 2019.821385] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 192.168.202.118@tcp) was lost; in progress operations using this service will fail
[ 2019.843224] Lustre: Evicted from MGS (at 192.168.202.118@tcp) after server handle changed from 0x76d62900af8583fc to 0x76d62900af859a6f
[ 2037.350043] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 192.168.202.118@tcp) was lost; in progress operations using this service will fail
[ 2037.363884] Lustre: 15940:0:(mgc_request.c:2378:mgc_process_log()) MGC192.168.202.118@tcp: IR log lustre-cliir failed, not fatal: rc = -107
[ 2037.372796] Lustre: Evicted from MGS (at 192.168.202.118@tcp) after server handle changed from 0x76d62900af859a6f to 0x76d62900af859ba3
[ 2144.623760] LustreError: lustre-MDT0000-mdc-ffff8d32c2e59000: operation mds_close to node 192.168.202.118@tcp failed: rc = -19
[ 2144.645089] LustreError: Skipped 4 previous similar messages
[ 2147.307575] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 192.168.202.118@tcp) was lost; in progress operations using this service will fail
[ 2147.367923] Lustre: Evicted from MGS (at 192.168.202.118@tcp) after server handle changed from 0x76d62900af859ba3 to 0x76d62900af872818
[ 2171.764650] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 192.168.202.118@tcp) was lost; in progress operations using this service will fail
[ 2171.796696] Lustre: 15940:0:(mgc_request.c:2378:mgc_process_log()) MGC192.168.202.118@tcp: IR log lustre-cliir failed, not fatal: rc = -107
[ 2171.806322] Lustre: Evicted from MGS (at 192.168.202.118@tcp) after server handle changed from 0x76d62900af872818 to 0x76d62900af872a48
[ 2189.546677] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 15:14:55 (1788462895)
[ 2189.904496] Lustre: *** cfs_fail_loc=305, val=0***
[ 2189.907057] Lustre: Skipped 4 previous similar messages
[ 2217.970079] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 192.168.202.118@tcp) was lost; in progress operations using this service will fail
[ 2218.002199] Lustre: Evicted from MGS (at 192.168.202.118@tcp) after server handle changed from 0x76d62900af872a48 to 0x76d62900af872d4a
[ 2232.744896] Lustre: 15940:0:(mgc_request.c:2378:mgc_process_log()) MGC192.168.202.118@tcp: IR log lustre-cliir failed, not fatal: rc = -107
[ 2249.635288] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2251.480795] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2268.322255] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 15:16:12 (1788462972)
[ 2279.400927] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 192.168.202.118@tcp) was lost; in progress operations using this service will fail
[ 2279.407881] LustreError: Skipped 1 previous similar message
[ 2279.422710] Lustre: Evicted from MGS (at 192.168.202.118@tcp) after server handle changed from 0x76d62900af872df2 to 0x76d62900af8731b8
[ 2279.431033] Lustre: Skipped 1 previous similar message
[ 2292.023482] LustreError: MGC192.168.202.118@tcp: operation ldlm_enqueue to node 192.168.202.118@tcp failed: rc = -107
[ 2292.035633] LustreError: Skipped 2 previous similar messages
[ 2292.046165] Lustre: 15940:0:(mgc_request.c:2378:mgc_process_log()) MGC192.168.202.118@tcp: IR log lustre-cliir failed, not fatal: rc = -107
[ 2293.430282] LustreError: lustre-MDT0000-mdc-ffff8d32c2e59000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2320.035471] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 15:17:06 (1788463026)
[ 2337.707417] LustreError: lustre-OST0000-osc-ffff8d32c2e59000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2360.959419] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 15:17:46 (1788463066)
[ 2376.705511] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 192.168.202.118@tcp) was lost; in progress operations using this service will fail
[ 2376.724383] LustreError: Skipped 1 previous similar message
[ 2376.757265] Lustre: Evicted from MGS (at 192.168.202.118@tcp) after server handle changed from 0x76d62900af873497 to 0x76d62900af87768b
[ 2376.767197] Lustre: Skipped 1 previous similar message
[ 2396.792572] Lustre: 15940:0:(mgc_request.c:2378:mgc_process_log()) MGC192.168.202.118@tcp: IR log lustre-cliir failed, not fatal: rc = -107
[ 2422.950659] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2425.853071] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2512.676728] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 192.168.202.118@tcp) was lost; in progress operations using this service will fail
[ 2512.693521] LustreError: Skipped 2 previous similar messages
[ 2512.710760] Lustre: 15940:0:(mgc_request.c:2378:mgc_process_log()) MGC192.168.202.118@tcp: IR log lustre-cliir failed, not fatal: rc = -107
[ 2512.720531] Lustre: Evicted from MGS (at 192.168.202.118@tcp) after server handle changed from 0x76d62900af88e46e to 0x76d62900af88e674
[ 2512.745554] Lustre: Skipped 2 previous similar messages
[ 2512.753900] Lustre: MGC192.168.202.118@tcp: Connection restored to 192.168.202.118@tcp (at 192.168.202.118@tcp)
[ 2512.764855] Lustre: Skipped 21 previous similar messages
[ 2539.520844] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2542.132492] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2609.668931] LustreError: lustre-MDT0000-mdc-ffff8d32c2e59000: operation ldlm_enqueue to node 192.168.202.118@tcp failed: rc = -19
[ 2609.679829] LustreError: Skipped 4 previous similar messages
[ 2609.686432] Lustre: lustre-MDT0000-mdc-ffff8d32c2e59000: Connection to lustre-MDT0000 (at 192.168.202.118@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2609.697991] Lustre: Skipped 10 previous similar messages
[ 2631.693819] Lustre: 15940:0:(mgc_request.c:2378:mgc_process_log()) MGC192.168.202.118@tcp: IR log lustre-cliir failed, not fatal: rc = -5
[ 2657.664491] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2660.473311] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2691.735905] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 15:23:17 (1788463397)
[ 2719.146725] Lustre: 15940:0:(mgc_request.c:2378:mgc_process_log()) MGC192.168.202.118@tcp: IR log lustre-cliir failed, not fatal: rc = -107
[ 2730.413646] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2766.531901] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2778.087941] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 192.168.202.118@tcp) was lost; in progress operations using this service will fail
[ 2778.109206] LustreError: Skipped 6 previous similar messages
[ 2778.133552] Lustre: Evicted from MGS (at 192.168.202.118@tcp) after server handle changed from 0x76d62900af8b311d to 0x76d62900af8b3705
[ 2778.146272] Lustre: Skipped 6 previous similar messages
[ 2806.152904] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2852.526908] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2897.409640] Lustre: 15940:0:(mgc_request.c:2378:mgc_process_log()) MGC192.168.202.118@tcp: IR log lustre-cliir failed, not fatal: rc = -107
[ 2897.432979] Lustre: 15940:0:(mgc_request.c:2378:mgc_process_log()) Skipped 3 previous similar messages
[ 2909.316039] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2968.853605] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 3060.396407] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 15:29:25 (1788463765)
[ 3119.656133] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3122.977747] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3407.106120] LustreError: lustre-OST0000-osc-ffff8d32c2e59000: operation ldlm_enqueue to node 192.168.202.118@tcp failed: rc = -19
[ 3407.123357] LustreError: Skipped 18 previous similar messages
[ 3407.131324] Lustre: lustre-OST0000-osc-ffff8d32c2e59000: Connection to lustre-OST0000 (at 192.168.202.118@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3407.159203] Lustre: Skipped 8 previous similar messages
[ 3431.993781] Lustre: lustre-OST0000-osc-ffff8d32c2e59000: Connection restored to 192.168.202.118@tcp (at 192.168.202.118@tcp)
[ 3432.002976] Lustre: Skipped 26 previous similar messages
[ 3452.699036] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3454.904192] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3787.786025] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3790.583265] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4033.933637] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 15:45:39 (1788464739)
[ 4051.424039] Lustre: 50627:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788464743/real 1788464743]  req@ffff8d32c81ef800 x1875337046436864/t0(0) o101->lustre-MDT0000-mdc-ffff8d32c2e59000@192.168.202.118@tcp:12/10 lens 576/1152 e 0 to 1 dl 1788464759 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 4051.459717] Lustre: 50627:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 12 previous similar messages
[ 4051.466930] Lustre: lustre-MDT0000-mdc-ffff8d32c2e59000: Connection to lustre-MDT0000 (at 192.168.202.118@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4051.480269] Lustre: Skipped 1 previous similar message
[ 4051.510944] Lustre: lustre-MDT0000-mdc-ffff8d32c2e59000: Connection restored to 192.168.202.118@tcp (at 192.168.202.118@tcp)
[ 4051.524791] Lustre: Skipped 1 previous similar message
[ 4060.885593] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 15:46:06 (1788464766)
[ 4087.475927] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 15:46:33 (1788464793)
[ 4113.116991] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 15:46:59 (1788464819)
[ 4113.727481] Lustre: Mounted lustre-client - version 2.17.58_7_g065ac5e
[ 4129.283630] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 192.168.202.118@tcp) was lost; in progress operations using this service will fail
[ 4129.298038] LustreError: Skipped 9 previous similar messages
[ 4129.329163] Lustre: Evicted from MGS (at 192.168.202.118@tcp) after server handle changed from 0x76d62900af8ca5c8 to 0x76d62900af9eb708
[ 4129.339867] Lustre: Skipped 9 previous similar messages
[ 4145.698203] Lustre: 2350:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788464836/real 1788464836]  req@ffff8d32c81ee680 x1875337046456320/t0(0) o400->lustre-MDT0000-mdc-ffff8d32c2e59000@192.168.202.118@tcp:12/10 lens 224/224 e 0 to 1 dl 1788464852 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4145.742470] Lustre: 2350:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 4149.100169] LustreError: MGC192.168.202.118@tcp: operation ldlm_enqueue to node 192.168.202.118@tcp failed: rc = -107
[ 4149.111060] LustreError: Skipped 1 previous similar message
[ 4149.114841] Lustre: 15940:0:(mgc_request.c:2378:mgc_process_log()) MGC192.168.202.118@tcp: IR log lustre-cliir failed, not fatal: rc = -107
[ 4149.137588] Lustre: 15940:0:(mgc_request.c:2378:mgc_process_log()) Skipped 2 previous similar messages
[ 4167.413408] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4169.600358] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4171.746302] Lustre: Unmounted lustre-client
[ 4179.318866] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 15:48:05 (1788464885)
[ 4300.703168] Lustre: 2351:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788464992/real 1788464992]  req@ffff8d32d0f4d880 x1875337046501376/t0(0) o4->lustre-OST0000-osc-ffff8d32c2e59000@192.168.202.118@tcp:6/4 lens 488/448 e 0 to 1 dl 1788465008 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4300.724694] Lustre: 2351:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 39 previous similar messages
[ 4478.850882] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 15:53:04 (1788465184)
[ 4530.550843] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 15:53:56 (1788465236)
[ 4534.459717] Lustre: Unmounted lustre-client
[ 4563.314056] Lustre: Mounted lustre-client - version 2.17.58_7_g065ac5e
[ 4572.469434] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 15:54:38 (1788465278)
[ 4572.840976] LustreError: 56562:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4573.899179] LustreError: 56562:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4574.320120] Lustre: *** cfs_fail_loc=305, val=0***
[ 4598.632477] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 15:55:05 (1788465305)
[ 4599.341292] Lustre: Mounted lustre-client - version 2.17.58_7_g065ac5e
[ 4602.246094] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4612.575719] Lustre: Unmounted lustre-client
[ 4621.005875] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 15:55:27 (1788465327)
[ 4722.133913] Lustre: lustre-MDT0000-mdc-ffff8d32c7fc2000: Connection to lustre-MDT0000 (at 192.168.202.118@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4722.154381] Lustre: Skipped 23 previous similar messages
[ 4728.301421] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 192.168.202.118@tcp) was lost; in progress operations using this service will fail
[ 4728.314553] LustreError: Skipped 1 previous similar message
[ 4728.325746] Lustre: Evicted from MGS (at 192.168.202.118@tcp) after server handle changed from 0x76d62900af9ed3ef to 0x76d62900afa187e5
[ 4728.339181] Lustre: Skipped 1 previous similar message
[ 4728.349587] Lustre: MGC192.168.202.118@tcp: Connection restored to 192.168.202.118@tcp (at 192.168.202.118@tcp)
[ 4728.364709] Lustre: Skipped 25 previous similar messages
[ 4742.900063] Lustre: 55959:0:(mgc_request.c:2378:mgc_process_log()) MGC192.168.202.118@tcp: IR log lustre-cliir failed, not fatal: rc = -107
[ 4906.284402] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 16:00:12 (1788465612)
[ 4911.891817] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4917.233488] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 192.168.202.118@tcp) was lost; in progress operations using this service will fail
[ 4917.241966] LustreError: Skipped 1 previous similar message
[ 4917.251865] Lustre: Evicted from MGS (at 192.168.202.118@tcp) after server handle changed from 0x76d62900afa18878 to 0x76d62900afa65028
[ 4917.260671] Lustre: Skipped 1 previous similar message
[ 4923.239372] LustreError: MGC192.168.202.118@tcp: operation ldlm_enqueue to node 192.168.202.118@tcp failed: rc = -107
[ 4923.257649] LustreError: Skipped 2 previous similar messages
[ 4923.274817] Lustre: 55959:0:(mgc_request.c:2378:mgc_process_log()) MGC192.168.202.118@tcp: IR log lustre-cliir failed, not fatal: rc = -107
[ 4925.293952] LustreError: lustre-MDT0000-mdc-ffff8d32c7fc2000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4939.049805] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 16:00:45 (1788465645)
[ 4939.689321] Lustre: Mounted lustre-client - version 2.17.58_7_g065ac5e
[ 4947.116832] LustreError: lustre-OST0000-osc-ffff8d32c7fc2000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4956.640191] Lustre: 2349:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788465648/real 1788465648]  req@ffff8d32c405c380 x1875337051168768/t0(0) o103->lustre-OST0000-osc-ffff8d32c2e59000@192.168.202.118@tcp:17/18 lens 328/224 e 0 to 1 dl 1788465664 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295
[ 4956.682066] Lustre: 2349:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 65 previous similar messages
[ 4959.770701] Lustre: Unmounted lustre-client
[ 4968.238982] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 16:01:14 (1788465674)
[ 4975.468874] LustreError: lustre-MDT0000-mdc-ffff8d32c7fc2000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4975.505783] LustreError: 60970:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4982.836793] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 16:01:29 (1788465689)
[ 4983.071486] LustreError: 61668:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4987.890257] LustreError: lustre-MDT0000-mdc-ffff8d32c7fc2000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4987.906921] LustreError: 61691:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4987.925429] LustreError: 61668:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=163
[ 4987.946529] LustreError: 61668:0:(import.c:761:ptlrpc_connect_import_locked()) already connecting
[ 4987.982623] LustreError: 61696:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4989.094646] LustreError: 61702:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4989.106847] LustreError: 61702:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4991.395779] LustreError: 61724:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4991.399074] LustreError: 61724:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4995.897320] LustreError: 61769:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4995.904530] LustreError: 61769:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 5005.697307] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 16:01:51 (1788465711)
[ 5053.284210] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5055.653509] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5069.495213] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 16:02:54 (1788465774)
[ 5102.508235] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5103.920961] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5113.805320] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 16:03:39 (1788465819)
[ 5178.133880] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5179.267795] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5188.185673] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 16:04:54 (1788465894)
[ 5230.537182] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5232.031303] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5238.787517] Lustre: Unmounted lustre-client
[ 5299.694411] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5302.183857] Lustre: Mounted lustre-client - version 2.17.58_7_g065ac5e
[ 5309.819964] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 16:06:56 (1788466016)
[ 5312.697676] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5314.565299] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 16:07:00 (1788466020)
[ 5351.057357] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 16:07:36 (1788466056)
[ 5352.737765] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5354.827268] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 16:07:40 (1788466060)
[ 5356.334767] Lustre: *** cfs_fail_loc=805, val=0***
[ 5356.373693] Lustre: Mounted lustre-client - version 2.17.58_7_g065ac5e
[ 5363.193256] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5371.890176] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 192.168.202.118@tcp) was lost; in progress operations using this service will fail
[ 5371.899681] LustreError: Skipped 1 previous similar message
[ 5371.915587] Lustre: Evicted from MGS (at 192.168.202.118@tcp) after server handle changed from 0x76d62900afa65f9b to 0x76d62900afa661d9
[ 5371.926333] Lustre: Skipped 1 previous similar message
[ 5371.930819] Lustre: MGC192.168.202.118@tcp: Connection restored to 192.168.202.118@tcp (at 192.168.202.118@tcp)
[ 5371.940090] Lustre: Skipped 14 previous similar messages
[ 5383.071929] Lustre: lustre-MDT0000-mdc-ffff8d32c2e5d000: Connection to lustre-MDT0000 (at 192.168.202.118@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5383.113190] Lustre: Skipped 12 previous similar messages
[ 5385.578569] Lustre: 68611:0:(mgc_request.c:2378:mgc_process_log()) MGC192.168.202.118@tcp: IR log lustre-cliir failed, not fatal: rc = -107
[ 5397.815859] Lustre: Unmounted lustre-client
[ 5407.912861] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 16:08:34 (1788466114)
[ 5450.344826] LustreError: 2348:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff8d32c405ca80 x1875337051259776/t94489280516(94489280516) o101->lustre-MDT0000-mdc-ffff8d32c2e5d000@192.168.202.118@tcp:12/10 lens 664/608 e 0 to 0 dl 1788466173 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5457.456797] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5459.764646] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5472.004861] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 16:09:37 (1788466177)
[ 5473.436700] LustreError: lustre-OST0000-osc-ffff8d32c2e5d000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5473.456898] Lustre: 2351:0:(llite_lib.c:4341:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.118@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5473.495115] LustreError: 73211:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8d32c2e5d000: namespace resource [0x240000400:0x37c2:0x0].0x0 (ffff8d32c43bd900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5486.195400] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 16:09:51 (1788466191)
[ 5488.325981] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5491.965893] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 16:09:56 (1788466196)
[ 5494.318115] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5496.204942] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 16:10:02 (1788466202)
[ 5497.908541] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5500.163869] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 16:10:06 (1788466206)
[ 5501.898499] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5504.435785] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 16:10:10 (1788466210)
[ 5506.341343] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5508.439863] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 16:10:14 (1788466214)
[ 5509.955667] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5512.140878] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 16:10:18 (1788466218)
[ 5513.433626] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5515.859052] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 16:10:21 (1788466221)
[ 5517.632979] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5519.366108] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 16:10:25 (1788466225)
[ 5521.412088] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5523.995857] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 16:10:29 (1788466229)
[ 5526.643941] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5528.976784] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 16:10:34 (1788466234)
[ 5531.537899] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5533.608423] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 16:10:39 (1788466239)
[ 5537.226180] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5539.423903] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 16:10:45 (1788466245)
[ 5555.691050] LustreError: MGC192.168.202.118@tcp: operation ldlm_enqueue to node 192.168.202.118@tcp failed: rc = -107
[ 5555.699443] LustreError: Skipped 6 previous similar messages
[ 5564.383262] Lustre: 2350:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788466255/real 1788466255]  req@ffff8d32c411b100 x1875337051282432/t0(0) o400->lustre-MDT0000-mdc-ffff8d32c2e5d000@192.168.202.118@tcp:12/10 lens 224/224 e 0 to 1 dl 1788466271 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5564.421895] Lustre: 2350:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 13 previous similar messages
[ 5575.932429] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 16:11:22 (1788466282)
[ 5609.881676] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 16:11:55 (1788466315)
[ 5610.461684] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5621.210631] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 16:12:07 (1788466327)
[ 5621.627982] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5632.022358] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 16:12:18 (1788466338)
[ 5632.316209] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5641.523174] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 16:12:27 (1788466347)
[ 5641.883198] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5650.842543] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 16:12:37 (1788466357)
[ 5651.117250] Lustre: *** cfs_fail_loc=510, val=3***
[ 5659.138525] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 16:12:45 (1788466365)
[ 5659.500979] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5669.255803] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 16:12:55 (1788466375)
[ 5669.855371] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5736.461935] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 16:14:02 (1788466442)
[ 5737.262670] LustreError: 83427:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5740.626721] LustreError: lustre-MDT0000-mdc-ffff8d32c2e5d000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5740.673300] LustreError: 83441:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8d32c2e5d000: inode [0x20000a042:0xe:0x0] mdc close failed: rc = -108
[ 5740.729951] LustreError: 83441:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8d32c2e5d000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8d32c4178c00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5741.346281] LustreError: 83427:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5743.490615] LustreError: 83457:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5746.545606] LustreError: lustre-MDT0000-mdc-ffff8d32c2e5d000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5746.559802] LustreError: 83471:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8d32c2e5d000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8d32d1672900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5746.613376] LustreError: 83471:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5747.583251] LustreError: 83457:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5754.487251] LustreError: lustre-MDT0000-mdc-ffff8d32c2e5d000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5755.153279] LustreError: 83512:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5757.989428] LustreError: lustre-MDT0000-mdc-ffff8d32c2e5d000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5758.048266] LustreError: 83532:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5758.053536] LustreError: 83532:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5759.181577] LustreError: 83538:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5759.208103] LustreError: 83538:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5759.232212] LustreError: 83512:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5759.242778] LustreError: 83512:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8d32c2e5d000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5761.545480] LustreError: 83560:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5761.554767] LustreError: 83560:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5765.242161] LustreError: 83587:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5768.386372] LustreError: lustre-MDT0000-mdc-ffff8d32c2e5d000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5768.400703] LustreError: 83601:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8d32c2e5d000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff8d32c4badb00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5768.414133] LustreError: 83601:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5769.313583] LustreError: 83587:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5776.241555] LustreError: lustre-MDT0000-mdc-ffff8d32c2e5d000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5776.751085] LustreError: 83644:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5780.107308] LustreError: lustre-MDT0000-mdc-ffff8d32c2e5d000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5780.127490] LustreError: 83658:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8d32c2e5d000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8d32d1672e00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5780.143994] LustreError: 83658:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 3 previous similar messages
[ 5780.759185] LustreError: 83644:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5783.780913] LustreError: lustre-MDT0000-mdc-ffff8d32c2e5d000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5783.857980] LustreError: 83691:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5783.868642] LustreError: 83691:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5785.050396] LustreError: 83671:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8d32c2e5d000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5793.192573] LustreError: 83742:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5793.214212] LustreError: 83742:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5796.962053] LustreError: lustre-MDT0000-mdc-ffff8d32c2e5d000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5797.064676] LustreError: 83764:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5797.070965] LustreError: 83764:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5797.215188] LustreError: 83742:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5797.227317] LustreError: 83742:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5797.245393] LustreError: 83742:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8d32c2e5d000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5810.097885] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 16:15:16 (1788466516)
[ 5830.358868] LustreError: 68608:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5830.366460] LustreError: 68608:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5834.447204] LustreError: 68608:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5834.467087] LustreError: 68608:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5842.274925] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 16:15:48 (1788466548)
[ 5896.687856] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 16:16:42 (1788466602)
[ 5960.650576] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 16:17:46 (1788466666)
[ 5994.657445] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 16:18:20 (1788466700)
[ 5995.055802] Lustre: Mounted lustre-client - version 2.17.58_7_g065ac5e
[ 6118.461351] Lustre: Unmounted lustre-client
[ 6130.293829] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 16:20:35 (1788466835)
[ 6131.145109] LustreError: 88070:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 6134.520571] Lustre: lustre-OST0000-osc-ffff8d32c2e5d000: Connection to lustre-OST0000 (at 192.168.202.118@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6134.542165] Lustre: Skipped 17 previous similar messages
[ 6134.557390] LustreError: lustre-OST0000-osc-ffff8d32c2e5d000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6134.574779] LustreError: 88153:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8d32c2e5d000: namespace resource [0x240000400:0x37ed:0x0].0x0 (ffff8d32c4badb00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6134.603105] LustreError: 88153:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 6134.775059] LustreError: 88070:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout interrupted
[ 6134.779864] Lustre: 2349:0:(llite_lib.c:4341:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.118@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 6142.780851] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 16:20:48 (1788466848)
[ 6201.311236] Lustre: 88738:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788466853/real 1788466853]  req@ffff8d32c4fb5880 x1875337051443840/t0(0) o101->lustre-MDT0000-mdc-ffff8d32c2e5d000@192.168.202.118@tcp:12/10 lens 328/344 e 0 to 1 dl 1788466908 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 6201.338308] Lustre: 88738:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
[ 6201.382300] Lustre: lustre-MDT0000-mdc-ffff8d32c2e5d000: Connection restored to 192.168.202.118@tcp (at 192.168.202.118@tcp)
[ 6201.395756] Lustre: Skipped 22 previous similar messages
[ 6208.504387] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 16:21:54 (1788466914)
[ 6209.947462] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 6211.951380] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 16:21:58 (1788466918)
[ 6276.524630] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6278.323204] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 16:23:04 (1788466984)
[ 6335.900970] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 16:24:02 (1788467042)
[ 6338.155871] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6340.066678] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 16:24:06 (1788467046)
[ 6341.385140] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6343.170135] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 16:24:09 (1788467049)
[ 6374.783299] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 16:24:41 (1788467081)
[ 6386.048130] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6395.892051] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 192.168.202.118@tcp) was lost; in progress operations using this service will fail
[ 6395.914112] LustreError: Skipped 6 previous similar messages
[ 6395.928923] Lustre: Evicted from MGS (at 192.168.202.118@tcp) after server handle changed from 0x76d62900afa67088 to 0x76d62900afa68940
[ 6395.953236] Lustre: Skipped 6 previous similar messages
[ 6424.137369] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6425.465858] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6435.538308] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 16:25:41 (1788467141)
[ 6437.832947] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6439.644075] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 16:25:45 (1788467145)
[ 6450.278228] LustreError: MGC192.168.202.118@tcp: operation ldlm_enqueue to node 192.168.202.118@tcp failed: rc = -107
[ 6450.281987] LustreError: Skipped 10 previous similar messages
[ 6450.290984] Lustre: 68611:0:(mgc_request.c:2378:mgc_process_log()) MGC192.168.202.118@tcp: IR log lustre-cliir failed, not fatal: rc = -107
[ 6450.310141] Lustre: 68611:0:(mgc_request.c:2378:mgc_process_log()) Skipped 3 previous similar messages
[ 6466.836801] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 16:26:12 (1788467172)
[ 6491.374651] Lustre: 68611:0:(mgc_request.c:2378:mgc_process_log()) MGC192.168.202.118@tcp: IR log lustre-cliir failed, not fatal: rc = -107
[ 6508.152073] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 16:26:53 (1788467213)
[ 6560.589553] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6563.842289] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6648.678889] Lustre: 68611:0:(mgc_request.c:2378:mgc_process_log()) MGC192.168.202.118@tcp: IR log lustre-cliir failed, not fatal: rc = -107
[ 6662.396847] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6663.899662] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6699.609991] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6701.019442] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6807.915225] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 16:31:54 (1788467514)
[ 6814.983027] Lustre: lustre-OST0000-osc-ffff8d32c2e5d000: Connection to lustre-OST0000 (at 192.168.202.118@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6814.990723] Lustre: Skipped 9 previous similar messages
[ 6854.313313] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6856.338246] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 7094.353533] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 16:36:40 (1788467800)
[ 7220.209037] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 192.168.202.118@tcp) was lost; in progress operations using this service will fail
[ 7220.222026] LustreError: Skipped 9 previous similar messages
[ 7220.232855] Lustre: Evicted from MGS (at 192.168.202.118@tcp) after server handle changed from 0x76d62900afa6a682 to 0x76d62900afb64891
[ 7220.242289] Lustre: Skipped 9 previous similar messages
[ 7220.248202] Lustre: MGC192.168.202.118@tcp: Connection restored to 192.168.202.118@tcp (at 192.168.202.118@tcp)
[ 7220.257556] Lustre: Skipped 17 previous similar messages
[ 7228.907294] LustreError: MGC192.168.202.118@tcp: operation ldlm_enqueue to node 192.168.202.118@tcp failed: rc = -107
[ 7228.915825] LustreError: Skipped 361 previous similar messages
[ 7228.928322] Lustre: 68611:0:(mgc_request.c:2378:mgc_process_log()) MGC192.168.202.118@tcp: IR log lustre-cliir failed, not fatal: rc = -107
[ 7228.944073] Lustre: 68611:0:(mgc_request.c:2378:mgc_process_log()) Skipped 1 previous similar message
[ 7260.599868] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 16:39:27 (1788467967)
[ 7261.855388] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7263.423462] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 16:39:29 (1788467969)
[ 7265.511275] LustreError: lustre-MDT0000-mdc-ffff8d32c2e5d000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7272.200905] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 16:39:38 (1788467978)
[ 7439.020639] LustreError: lustre-OST0000-osc-ffff8d32c2e5d000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7441.004905] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 16:42:27 (1788468147)
[ 7465.951340] Lustre: 2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788468153/real 1788468153]  req@ffff8d32f1203480 x1875337069206400/t0(0) o4->lustre-OST0000-osc-ffff8d32c2e5d000@192.168.202.118@tcp:6/4 lens 4584/448 e 0 to 1 dl 1788468173 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7465.987070] Lustre: 2352:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 17 previous similar messages
[ 7465.995582] Lustre: lustre-OST0000-osc-ffff8d32c2e5d000: Connection to lustre-OST0000 (at 192.168.202.118@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7466.016694] Lustre: Skipped 3 previous similar messages
[ 7486.605234] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 16:43:12 (1788468192)
[ 7488.577181] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7490.835659] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 16:43:16 (1788468196)
[ 7492.430933] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7494.077934] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 16:43:20 (1788468200)
[ 7528.970485] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 16:43:54 (1788468234)
[ 7582.062501] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 16:44:48 (1788468288)
[ 7583.219395] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7585.052610] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 16:44:51 (1788468291)
[ 7587.075810] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7588.869716] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 16:44:55 (1788468295)
[ 7598.963238] Lustre: Unmounted lustre-client
[ 7601.514885] Lustre: Mounted lustre-client - version 2.17.58_7_g065ac5e
[ 7604.446392] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7625.538760] Lustre: 107192:0:(mgc_request.c:2378:mgc_process_log()) MGC192.168.202.118@tcp: IR log lustre-cliir failed, not fatal: rc = -107
[ 7635.091091] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2