[ 1869.836259] LustreError: lustre-OST0000-osc-ffff9ea34913f800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1869.852110] Lustre: 2351:0:(llite_lib.c:4366:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.139@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1875.563764] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 15:55:05 (1787860505)
[ 1877.932907] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1879.666555] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 15:55:09 (1787860509)
[ 1881.382763] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1883.349623] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 15:55:13 (1787860513)
[ 1887.308648] LustreError: lustre-MDT0000-mdc-ffff9ea34913f800: operation ldlm_enqueue to node 192.168.203.139@tcp failed: rc = -19
[ 1887.314154] LustreError: Skipped 1 previous similar message
[ 1909.748315] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 192.168.203.139@tcp) was lost; in progress operations using this service will fail
[ 1909.766134] Lustre: Evicted from MGS (at 192.168.203.139@tcp) after server handle changed from 0xd2400d36a2a90b91 to 0xd2400d36a2a923cb
[ 1911.416215] Lustre: 15980:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.139@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2018.847208] LustreError: lustre-MDT0000-mdc-ffff9ea34913f800: operation ldlm_enqueue to node 192.168.203.139@tcp failed: rc = -19
[ 2037.599953] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 192.168.203.139@tcp) was lost; in progress operations using this service will fail
[ 2047.813151] Lustre: Evicted from MGS (at 192.168.203.139@tcp) after server handle changed from 0xd2400d36a2a923cb to 0xd2400d36a2aaefda
[ 2064.369303] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 15:58:13 (1787860693)
[ 2064.921840] Lustre: *** cfs_fail_loc=305, val=0***
[ 2064.925016] Lustre: Skipped 2 previous similar messages
[ 2103.071369] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 192.168.203.139@tcp) was lost; in progress operations using this service will fail
[ 2103.095517] Lustre: Evicted from MGS (at 192.168.203.139@tcp) after server handle changed from 0xd2400d36a2aaefda to 0xd2400d36a2aaf26c
[ 2104.556254] Lustre: 15980:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.139@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2113.513585] Lustre: lustre-MDT0000-mdc-ffff9ea34913f800: Connection to lustre-MDT0000 (at 192.168.203.139@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2113.524452] Lustre: Skipped 11 previous similar messages
[ 2115.073099] Lustre: lustre-MDT0000-mdc-ffff9ea34913f800: Connection restored to 192.168.203.139@tcp (at 192.168.203.139@tcp)
[ 2115.083648] Lustre: Skipped 14 previous similar messages
[ 2124.880518] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2126.420600] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2134.713236] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 15:59:24 (1787860764)
[ 2149.354527] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 192.168.203.139@tcp) was lost; in progress operations using this service will fail
[ 2149.378483] Lustre: Evicted from MGS (at 192.168.203.139@tcp) after server handle changed from 0xd2400d36a2aaf26c to 0xd2400d36a2aaf957
[ 2149.445439] LustreError: lustre-MDT0000-mdc-ffff9ea34913f800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2171.132312] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 16:00:00 (1787860800)
[ 2187.909658] LustreError: lustre-OST0000-osc-ffff9ea34913f800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2206.412993] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 16:00:36 (1787860836)
[ 2221.387691] LustreError: lustre-MDT0000-mdc-ffff9ea34913f800: operation ldlm_enqueue to node 192.168.203.139@tcp failed: rc = -19
[ 2241.515522] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 192.168.203.139@tcp) was lost; in progress operations using this service will fail
[ 2241.538401] Lustre: Evicted from MGS (at 192.168.203.139@tcp) after server handle changed from 0xd2400d36a2aaf957 to 0xd2400d36a2ab445f
[ 2242.621536] Lustre: 15980:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.139@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2268.464510] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2270.741243] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2354.144479] Lustre: 2352:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787860969/real 1787860969]  req@ffff9ea346334f00 x1874705735086336/t0(0) o400->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 224/224 e 0 to 1 dl 1787860985 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2354.180396] Lustre: 2352:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
[ 2354.203776] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 192.168.203.139@tcp) was lost; in progress operations using this service will fail
[ 2364.593658] Lustre: Evicted from MGS (at 192.168.203.139@tcp) after server handle changed from 0xd2400d36a2ab445f to 0xd2400d36a2ace6bb
[ 2373.907343] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2376.459338] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2442.364915] LustreError: lustre-MDT0000-mdc-ffff9ea34913f800: operation mds_close to node 192.168.203.139@tcp failed: rc = -19
[ 2442.373487] LustreError: Skipped 4 previous similar messages
[ 2463.199402] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 192.168.203.139@tcp) was lost; in progress operations using this service will fail
[ 2463.225379] Lustre: Evicted from MGS (at 192.168.203.139@tcp) after server handle changed from 0xd2400d36a2ace6bb to 0xd2400d36a2ae5642
[ 2464.894364] Lustre: 15980:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.139@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2491.299346] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2493.624428] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2522.916982] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 16:05:52 (1787861152)
[ 2544.095313] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 192.168.203.139@tcp) was lost; in progress operations using this service will fail
[ 2554.287572] Lustre: Evicted from MGS (at 192.168.203.139@tcp) after server handle changed from 0xd2400d36a2ae5642 to 0xd2400d36a2af0fec
[ 2563.473621] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2595.628294] Lustre: 15980:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.139@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2605.351134] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2648.270130] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2680.279405] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 192.168.203.139@tcp) was lost; in progress operations using this service will fail
[ 2680.289098] LustreError: Skipped 2 previous similar messages
[ 2687.625219] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2711.331811] LustreError: lustre-MDT0000-mdc-ffff9ea34913f800: operation mds_close to node 192.168.203.139@tcp failed: rc = -19
[ 2711.347026] LustreError: Skipped 4 previous similar messages
[ 2730.525283] Lustre: Evicted from MGS (at 192.168.203.139@tcp) after server handle changed from 0xd2400d36a2afb0fa to 0xd2400d36a2aff621
[ 2730.534123] Lustre: Skipped 3 previous similar messages
[ 2730.540733] Lustre: MGC192.168.203.139@tcp: Connection restored to 192.168.203.139@tcp (at 192.168.203.139@tcp)
[ 2730.548627] Lustre: Skipped 16 previous similar messages
[ 2741.860175] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2769.796124] Lustre: lustre-MDT0000-mdc-ffff9ea34913f800: Connection to lustre-MDT0000 (at 192.168.203.139@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2769.813090] Lustre: Skipped 10 previous similar messages
[ 2807.268614] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2897.661908] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 16:12:07 (1787861527)
[ 2964.469836] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 2967.514925] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3244.050978] LustreError: lustre-OST0000-osc-ffff9ea34913f800: operation ldlm_enqueue to node 192.168.203.139@tcp failed: rc = -107
[ 3244.070885] LustreError: Skipped 7 previous similar messages
[ 3288.334027] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3290.659183] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3573.915346] Lustre: lustre-OST0000-osc-ffff9ea34913f800: Connection to lustre-OST0000 (at 192.168.203.139@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3573.931869] Lustre: Skipped 3 previous similar messages
[ 3594.865399] Lustre: lustre-OST0000-osc-ffff9ea34913f800: Connection restored to 192.168.203.139@tcp (at 192.168.203.139@tcp)
[ 3594.881931] Lustre: Skipped 7 previous similar messages
[ 3612.001090] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3613.900960] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3869.894730] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 16:28:19 (1787862499)
[ 3887.071208] Lustre: 50650:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787862502/real 1787862502]  req@ffff9ea3462ff0c0 x1874705754801408/t0(0) o101->lustre-MDT0000-mdc-ffff9ea34913f800@192.168.203.139@tcp:12/10 lens 576/1152 e 0 to 1 dl 1787862518 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3887.099192] Lustre: 50650:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 3897.455438] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 16:28:47 (1787862527)
[ 3923.374981] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 16:29:13 (1787862553)
[ 3949.065745] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 16:29:38 (1787862578)
[ 3949.580601] Lustre: Mounted lustre-client - version 2.17.57_124_g5f91490
[ 3981.279194] Lustre: 2351:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787862596/real 1787862596]  req@ffff9ea3462fcf00 x1874705754820608/t0(0) o400->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 224/224 e 0 to 1 dl 1787862612 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3981.279194] Lustre: 2352:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787862596/real 1787862596]  req@ffff9ea3462fc000 x1874705754820736/t0(0) o400->lustre-MDT0000-mdc-ffff9ea34913f800@192.168.203.139@tcp:12/10 lens 224/224 e 0 to 1 dl 1787862612 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3981.279221] Lustre: 2351:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 3981.279327] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 192.168.203.139@tcp) was lost; in progress operations using this service will fail
[ 3981.372485] LustreError: Skipped 3 previous similar messages
[ 3981.387157] Lustre: Evicted from MGS (at 192.168.203.139@tcp) after server handle changed from 0xd2400d36a2b19c89 to 0xd2400d36a2c3e8fc
[ 3981.400119] Lustre: Skipped 2 previous similar messages
[ 3994.920612] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 3996.603313] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3998.708381] Lustre: Unmounted lustre-client
[ 4004.224825] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 16:30:34 (1787862634)
[ 4140.831155] Lustre: 2354:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787862756/real 1787862756]  req@ffff9ea3463f52c0 x1874705754854656/t0(0) o4->lustre-OST0000-osc-ffff9ea34913f800@192.168.203.139@tcp:6/4 lens 488/448 e 0 to 1 dl 1787862772 ref 2 fl Rpc:XQr/602/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4140.880912] Lustre: 2354:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 70 previous similar messages
[ 4189.151244] Lustre: lustre-OST0000-osc-ffff9ea34913f800: Connection to lustre-OST0000 (at 192.168.203.139@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4189.169395] Lustre: Skipped 15 previous similar messages
[ 4205.545908] Lustre: lustre-OST0000-osc-ffff9ea34913f800: Connection restored to 192.168.203.139@tcp (at 192.168.203.139@tcp)
[ 4205.560093] Lustre: Skipped 17 previous similar messages
[ 4299.289446] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 16:35:29 (1787862929)
[ 4347.021647] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 16:36:17 (1787862977)
[ 4350.246343] Lustre: Unmounted lustre-client
[ 4369.391352] Lustre: Mounted lustre-client - version 2.17.57_124_g5f91490
[ 4374.966929] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 16:36:45 (1787863005)
[ 4375.450843] LustreError: 56572:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4376.496360] LustreError: 56572:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4376.671226] Lustre: *** cfs_fail_loc=305, val=0***
[ 4400.335259] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 16:37:10 (1787863030)
[ 4400.782613] Lustre: Mounted lustre-client - version 2.17.57_124_g5f91490
[ 4403.161117] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4413.446254] Lustre: Unmounted lustre-client
[ 4420.058422] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 16:37:30 (1787863050)
[ 4525.491789] LustreError: lustre-MDT0000-mdc-ffff9ea344321000: operation mds_reint to node 192.168.203.139@tcp failed: rc = -19
[ 4525.501904] LustreError: Skipped 1 previous similar message
[ 4545.504715] Lustre: 2354:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787863160/real 1787863160]  req@ffff9ea359a23c00 x1874705757537280/t0(0) o400->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 224/224 e 0 to 1 dl 1787863176 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4545.528335] Lustre: 2354:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 89 previous similar messages
[ 4545.537280] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 192.168.203.139@tcp) was lost; in progress operations using this service will fail
[ 4545.573490] Lustre: Evicted from MGS (at 192.168.203.139@tcp) after server handle changed from 0xd2400d36a2c403ac to 0xd2400d36a2c6bbfb
[ 4735.561832] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 16:42:45 (1787863365)
[ 4741.224785] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4758.514390] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 192.168.203.139@tcp) was lost; in progress operations using this service will fail
[ 4758.544052] Lustre: Evicted from MGS (at 192.168.203.139@tcp) after server handle changed from 0xd2400d36a2c6bbfb to 0xd2400d36a2cb8016
[ 4758.589122] LustreError: lustre-MDT0000-mdc-ffff9ea344321000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4772.453669] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 16:43:22 (1787863402)
[ 4773.334189] Lustre: Mounted lustre-client - version 2.17.57_124_g5f91490
[ 4781.016920] LustreError: lustre-OST0000-osc-ffff9ea344321000: operation ldlm_enqueue to node 192.168.203.139@tcp failed: rc = -107
[ 4781.036847] LustreError: lustre-OST0000-osc-ffff9ea344321000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4790.815883] Lustre: lustre-OST0000-osc-ffff9ea34887c000: Connection to lustre-OST0000 (at 192.168.203.139@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4790.844987] Lustre: Skipped 11 previous similar messages
[ 4793.258428] Lustre: Unmounted lustre-client
[ 4800.791651] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 16:43:50 (1787863430)
[ 4807.686715] LustreError: lustre-MDT0000-mdc-ffff9ea344321000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4807.706913] LustreError: 60980:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4807.718644] Lustre: lustre-MDT0000-mdc-ffff9ea344321000: Connection restored to 192.168.203.139@tcp (at 192.168.203.139@tcp)
[ 4807.739411] Lustre: Skipped 14 previous similar messages
[ 4816.135792] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 16:44:05 (1787863445)
[ 4816.656599] LustreError: 61684:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4821.983234] LustreError: 61684:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=0
[ 4822.004903] LustreError: 61684:0:(import.c:711:ptlrpc_connect_import_locked()) already connected
[ 4823.732060] LustreError: lustre-MDT0000-mdc-ffff9ea344321000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4823.770535] LustreError: 61702:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4823.843410] LustreError: 61708:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4824.949777] LustreError: 61714:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4824.956904] LustreError: 61714:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4827.364868] LustreError: 61736:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4827.395903] LustreError: 61736:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4832.049761] LustreError: 61781:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4832.056294] LustreError: 61781:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4844.961723] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 16:44:34 (1787863474)
[ 4886.655245] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4888.362164] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4899.705143] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 16:45:29 (1787863529)
[ 4943.668683] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4945.414271] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4958.782511] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 16:46:28 (1787863588)
[ 5031.247181] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5033.019424] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5042.770926] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 16:47:52 (1787863672)
[ 5082.644185] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5084.329878] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5089.474162] Lustre: Unmounted lustre-client
[ 5143.681745] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5146.633564] Lustre: Mounted lustre-client - version 2.17.57_124_g5f91490
[ 5154.595766] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 16:49:43 (1787863783)
[ 5157.377592] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5159.379702] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 16:49:48 (1787863788)
[ 5191.513585] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 16:50:21 (1787863821)
[ 5193.161980] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5194.594950] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 16:50:24 (1787863824)
[ 5195.740610] Lustre: *** cfs_fail_loc=805, val=0***
[ 5195.782132] Lustre: Mounted lustre-client - version 2.17.57_124_g5f91490
[ 5200.372159] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5221.358516] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 192.168.203.139@tcp) was lost; in progress operations using this service will fail
[ 5221.382950] Lustre: Evicted from MGS (at 192.168.203.139@tcp) after server handle changed from 0xd2400d36a2cb8a03 to 0xd2400d36a2cb8c1e
[ 5222.370459] Lustre: 2353:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787863837/real 1787863837]  req@ffff9ea349bc12c0 x1874705759582976/t0(0) o400->lustre-MDT0000-mdc-ffff9ea358789000@192.168.203.139@tcp:12/10 lens 224/224 e 0 to 1 dl 1787863853 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5222.410530] Lustre: 2353:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 5222.960593] Lustre: 68349:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.139@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 5242.579609] Lustre: Unmounted lustre-client
[ 5251.498636] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 16:51:21 (1787863881)
[ 5278.493258] LustreError: 2350:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff9ea34632f840 x1874705759595520/t94489280516(94489280516) o101->lustre-MDT0000-mdc-ffff9ea358789000@192.168.203.139@tcp:12/10 lens 664/608 e 0 to 0 dl 1787863925 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5286.869244] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5288.383776] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5297.479583] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 16:52:06 (1787863926)
[ 5298.976473] LustreError: lustre-OST0000-osc-ffff9ea358789000: operation ost_write to node 192.168.203.139@tcp failed: rc = -107
[ 5298.981337] LustreError: Skipped 2 previous similar messages
[ 5299.000405] LustreError: lustre-OST0000-osc-ffff9ea358789000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5299.017327] Lustre: 2353:0:(llite_lib.c:4366:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.139@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5309.320244] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 16:52:18 (1787863938)
[ 5311.403378] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5313.662179] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 16:52:23 (1787863943)
[ 5315.543444] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5317.971522] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 16:52:27 (1787863947)
[ 5319.775216] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5321.364060] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 16:52:31 (1787863951)
[ 5323.027934] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5325.303202] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 16:52:34 (1787863954)
[ 5326.873847] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5328.507610] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 16:52:38 (1787863958)
[ 5330.190704] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5331.873442] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 16:52:41 (1787863961)
[ 5333.569146] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5335.688792] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 16:52:45 (1787863965)
[ 5337.223737] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5339.064443] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 16:52:48 (1787863968)
[ 5340.698420] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5342.474803] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 16:52:52 (1787863972)
[ 5343.887716] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5345.573826] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 16:52:55 (1787863975)
[ 5347.312442] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5349.159918] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 16:52:58 (1787863978)
[ 5351.901258] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5354.110795] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 16:53:03 (1787863983)
[ 5389.857531] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 16:53:39 (1787864019)
[ 5419.910784] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 16:54:09 (1787864049)
[ 5420.388317] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5430.626404] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 16:54:20 (1787864060)
[ 5431.228413] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5441.284379] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 16:54:31 (1787864071)
[ 5441.748402] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5450.347526] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 16:54:40 (1787864080)
[ 5450.652522] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5459.360345] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 16:54:49 (1787864089)
[ 5459.622410] Lustre: *** cfs_fail_loc=510, val=3***
[ 5468.678161] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 16:54:58 (1787864098)
[ 5468.985307] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5480.154484] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 16:55:09 (1787864109)
[ 5480.650146] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5538.784541] Lustre: lustre-OST0000-osc-ffff9ea358789000: Connection to lustre-OST0000 (at 192.168.203.139@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5538.811494] Lustre: Skipped 13 previous similar messages
[ 5545.665790] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 16:56:15 (1787864175)
[ 5545.967626] LustreError: 83099:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5548.658521] LustreError: lustre-MDT0000-mdc-ffff9ea358789000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5548.680539] LustreError: 83113:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9ea358789000: inode [0x20000a042:0xe:0x0] mdc close failed: rc = -108
[ 5548.715502] LustreError: 83113:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9ea358789000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9ea359b89b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5548.746523] Lustre: lustre-MDT0000-mdc-ffff9ea358789000: Connection restored to 192.168.203.139@tcp (at 192.168.203.139@tcp)
[ 5548.751800] Lustre: Skipped 11 previous similar messages
[ 5550.041588] LustreError: 83099:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5552.140058] LustreError: 83129:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5555.070441] LustreError: lustre-MDT0000-mdc-ffff9ea358789000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5555.087404] LustreError: 83143:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9ea358789000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9ea3611d8300) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5555.096269] LustreError: 83143:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5556.231189] LustreError: 83129:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5562.965653] LustreError: lustre-MDT0000-mdc-ffff9ea358789000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5563.314839] LustreError: 83184:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5565.974672] LustreError: lustre-MDT0000-mdc-ffff9ea358789000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5566.030294] LustreError: 83204:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5566.047479] LustreError: 83204:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5567.118669] LustreError: 83210:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5567.124952] LustreError: 83210:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5567.399661] LustreError: 83184:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5567.406978] LustreError: 83184:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9ea358789000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5569.389664] LustreError: 83232:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5569.403797] LustreError: 83232:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5572.948845] LustreError: 83259:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5575.907787] LustreError: lustre-MDT0000-mdc-ffff9ea358789000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5575.958251] LustreError: 83273:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9ea358789000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9ea3611d8b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5575.980092] LustreError: 83273:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5577.047181] LustreError: 83259:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5583.943573] LustreError: lustre-MDT0000-mdc-ffff9ea358789000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5584.324514] LustreError: 83315:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5587.272517] LustreError: lustre-MDT0000-mdc-ffff9ea358789000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5587.295666] LustreError: 83329:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9ea358789000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9ea349b64200) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5587.319445] LustreError: 83329:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5588.392040] LustreError: 83315:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5591.479575] LustreError: lustre-MDT0000-mdc-ffff9ea358789000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5591.569204] LustreError: 83362:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5591.587802] LustreError: 83362:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5592.601713] LustreError: 83342:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9ea358789000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5600.630560] LustreError: 83414:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5600.641840] LustreError: 83414:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5604.151490] LustreError: lustre-MDT0000-mdc-ffff9ea358789000: operation ldlm_enqueue to node 192.168.203.139@tcp failed: rc = -107
[ 5604.167746] LustreError: Skipped 8 previous similar messages
[ 5604.179508] LustreError: lustre-MDT0000-mdc-ffff9ea358789000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5604.267145] LustreError: 83436:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5604.271790] LustreError: 83436:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5604.727245] LustreError: 83414:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5604.734147] LustreError: 83414:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5604.757226] LustreError: 83414:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9ea358789000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5615.831648] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 16:57:25 (1787864245)
[ 5636.030252] LustreError: 70870:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5636.038928] LustreError: 70870:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5640.112084] LustreError: 70870:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5640.129474] LustreError: 70870:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5646.346361] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 16:57:56 (1787864276)
[ 5695.854902] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 16:58:46 (1787864326)
[ 5759.348768] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 16:59:49 (1787864389)
[ 5793.555446] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 17:00:23 (1787864423)
[ 5793.992989] Lustre: Mounted lustre-client - version 2.17.57_124_g5f91490
[ 5917.649076] Lustre: Unmounted lustre-client
[ 5925.726721] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 17:02:35 (1787864555)
[ 5926.217065] LustreError: 87752:0:(osc_request.c:3192:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5929.153809] LustreError: lustre-OST0000-osc-ffff9ea358789000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5929.187367] LustreError: 87840:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff9ea358789000: namespace resource [0x240000400:0x3a0d:0x0].0x0 (ffff9ea3610ca100) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5929.198659] LustreError: 87840:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5929.479209] LustreError: 87752:0:(osc_request.c:3192:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5929.485100] Lustre: 2351:0:(llite_lib.c:4366:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.139@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5936.738197] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 17:02:46 (1787864566)
[ 5994.463177] Lustre: 88419:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787864570/real 1787864570]  req@ffff9ea349bc1680 x1874705759766144/t0(0) o101->lustre-MDT0000-mdc-ffff9ea358789000@192.168.203.139@tcp:12/10 lens 328/344 e 0 to 1 dl 1787864625 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 5994.504890] Lustre: 88419:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 21 previous similar messages
[ 6000.214995] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 17:03:50 (1787864630)
[ 6001.737326] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 6003.608099] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 17:03:53 (1787864633)
[ 6066.979153] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6068.584254] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 17:04:58 (1787864698)
[ 6127.398799] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 17:05:57 (1787864757)
[ 6129.642246] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6131.368770] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 17:06:01 (1787864761)
[ 6132.653506] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6134.399480] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 17:06:04 (1787864764)
[ 6168.558944] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 17:06:38 (1787864798)
[ 6181.014322] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6207.474287] Lustre: lustre-MDT0000-mdc-ffff9ea358789000: Connection to lustre-MDT0000 (at 192.168.203.139@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6207.474571] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 192.168.203.139@tcp) was lost; in progress operations using this service will fail
[ 6207.487957] Lustre: Skipped 16 previous similar messages
[ 6207.507969] LustreError: Skipped 2 previous similar messages
[ 6207.529138] Lustre: Evicted from MGS (at 192.168.203.139@tcp) after server handle changed from 0xd2400d36a2cb97b6 to 0xd2400d36a2cbb0a6
[ 6207.544880] Lustre: Skipped 2 previous similar messages
[ 6207.557263] Lustre: MGC192.168.203.139@tcp: Connection restored to 192.168.203.139@tcp (at 192.168.203.139@tcp)
[ 6207.566205] Lustre: Skipped 15 previous similar messages
[ 6222.114243] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6223.836300] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6234.544922] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 17:07:43 (1787864863)
[ 6237.443350] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6239.408422] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 17:07:49 (1787864869)
[ 6253.568383] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 192.168.203.139@tcp) was lost; in progress operations using this service will fail
[ 6253.602629] Lustre: Evicted from MGS (at 192.168.203.139@tcp) after server handle changed from 0xd2400d36a2cbb0a6 to 0xd2400d36a2cbb51b
[ 6267.304437] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 17:08:17 (1787864897)
[ 6317.267825] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 17:09:06 (1787864946)
[ 6323.942574] LustreError: lustre-OST0000-osc-ffff9ea358789000: operation ost_setattr to node 192.168.203.139@tcp failed: rc = -107
[ 6323.948727] LustreError: Skipped 1 previous similar message
[ 6381.131770] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6384.810067] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6469.343412] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 192.168.203.139@tcp) was lost; in progress operations using this service will fail
[ 6469.347366] LustreError: Skipped 1 previous similar message
[ 6478.827587] Lustre: Evicted from MGS (at 192.168.203.139@tcp) after server handle changed from 0xd2400d36a2cbb8cc to 0xd2400d36a2cbc59f
[ 6478.837619] Lustre: Skipped 1 previous similar message
[ 6479.929558] Lustre: 68349:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.139@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6494.482098] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6496.372028] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6545.313566] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6546.878377] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6687.183860] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 17:15:16 (1787865316)
[ 6755.805260] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6760.726778] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 7019.628255] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 17:20:49 (1787865649)
[ 7195.623147] Lustre: lustre-MDT0000-mdc-ffff9ea358789000: Connection to lustre-MDT0000 (at 192.168.203.139@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7195.651901] Lustre: Skipped 6 previous similar messages
[ 7195.664971] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 192.168.203.139@tcp) was lost; in progress operations using this service will fail
[ 7195.683901] LustreError: Skipped 1 previous similar message
[ 7195.704325] Lustre: Evicted from MGS (at 192.168.203.139@tcp) after server handle changed from 0xd2400d36a2cbcaae to 0xd2400d36a2db6dce
[ 7195.719556] Lustre: Skipped 1 previous similar message
[ 7195.734067] Lustre: MGC192.168.203.139@tcp: Connection restored to 192.168.203.139@tcp (at 192.168.203.139@tcp)
[ 7195.755404] Lustre: Skipped 9 previous similar messages
[ 7221.215507] Lustre: 2354:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787865811/real 1787865811]  req@ffff9ea372692940 x1874705777513472/t0(0) o400->lustre-MDT0000-mdc-ffff9ea358789000@192.168.203.139@tcp:12/10 lens 224/224 e 0 to 1 dl 1787865852 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7221.243034] Lustre: 2354:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 29 previous similar messages
[ 7225.702978] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 17:24:15 (1787865855)
[ 7227.117750] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7228.852914] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 17:24:18 (1787865858)
[ 7231.152565] LustreError: lustre-MDT0000-mdc-ffff9ea358789000: operation ldlm_enqueue to node 192.168.203.139@tcp failed: rc = -107
[ 7231.167137] LustreError: Skipped 517 previous similar messages
[ 7231.191408] LustreError: lustre-MDT0000-mdc-ffff9ea358789000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7239.785688] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 17:24:29 (1787865869)
[ 7407.081203] LustreError: lustre-OST0000-osc-ffff9ea358789000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7408.594772] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 17:27:18 (1787866038)
[ 7433.695155] Lustre: 2354:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787866044/real 1787866044]  req@ffff9ea372697c00 x1874705777534464/t0(0) o4->lustre-OST0000-osc-ffff9ea358789000@192.168.203.139@tcp:6/4 lens 4584/448 e 0 to 1 dl 1787866064 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7433.733147] Lustre: 2354:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 7452.355053] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 17:28:02 (1787866082)
[ 7453.641277] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7455.452547] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 17:28:05 (1787866085)
[ 7457.115307] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7458.728601] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 17:28:08 (1787866088)
[ 7492.875065] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 17:28:42 (1787866122)
[ 7529.955154] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 192.168.203.139@tcp) was lost; in progress operations using this service will fail
[ 7529.978404] Lustre: Evicted from MGS (at 192.168.203.139@tcp) after server handle changed from 0xd2400d36a2db6dce to 0xd2400d36a2db96d9
[ 7530.006245] Lustre: Skipped 1 previous similar message
[ 7548.670832] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 17:29:38 (1787866178)
[ 7550.394209] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7552.287227] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 17:29:41 (1787866181)
[ 7553.889966] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7555.856531] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 17:29:45 (1787866185)
[ 7567.601107] Lustre: Unmounted lustre-client
[ 7569.564182] Lustre: Mounted lustre-client - version 2.17.57_124_g5f91490
[ 7572.484241] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7591.391814] Lustre: 2352:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787866206/real 1787866206]  req@ffff9ea36b423c00 x1874705777720704/t0(0) o400->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 224/224 e 0 to 1 dl 1787866222 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7613.598870] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2