[ 1741.684390] LustreError: lustre-OST0000-osc-ffff946f42cee000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1741.694669] Lustre: 2356:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.152@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1745.072829] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 19:32:16 (1788391936)
[ 1746.128246] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1746.941112] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 19:32:18 (1788391938)
[ 1747.749334] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1748.562927] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 19:32:19 (1788391939)
[ 1765.858077] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 192.168.203.152@tcp) was lost; in progress operations using this service will fail
[ 1765.869476] Lustre: Evicted from MGS (at 192.168.203.152@tcp) after server handle changed from 0xd727712ca7fb48 to 0xd727712ca81a74
[ 1861.542906] LustreError: lustre-MDT0000-mdc-ffff946f42cee000: operation mds_reint to node 192.168.203.152@tcp failed: rc = -19
[ 1861.548457] LustreError: Skipped 2 previous similar messages
[ 1876.457320] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 192.168.203.152@tcp) was lost; in progress operations using this service will fail
[ 1876.475632] Lustre: Evicted from MGS (at 192.168.203.152@tcp) after server handle changed from 0xd727712ca81a74 to 0xd727712cac840d
[ 1881.708766] Lustre: lustre-MDT0000-mdc-ffff946f42cee000: Connection restored to 192.168.203.152@tcp (at 192.168.203.152@tcp)
[ 1881.714524] Lustre: Skipped 14 previous similar messages
[ 1884.792449] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 19:34:36 (1788392076)
[ 1885.018487] Lustre: *** cfs_fail_loc=305, val=0***
[ 1885.020847] Lustre: Skipped 4 previous similar messages
[ 1922.169337] Lustre: lustre-MDT0000-mdc-ffff946f42cee000: Connection to lustre-MDT0000 (at 192.168.203.152@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 1922.177074] Lustre: Skipped 13 previous similar messages
[ 1922.530188] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 192.168.203.152@tcp) was lost; in progress operations using this service will fail
[ 1922.544074] Lustre: Evicted from MGS (at 192.168.203.152@tcp) after server handle changed from 0xd727712cac840d to 0xd727712cac8e47
[ 1924.927653] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 1925.702389] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1930.004540] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 19:35:21 (1788392121)
[ 1937.893035] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 192.168.203.152@tcp) was lost; in progress operations using this service will fail
[ 1937.903718] Lustre: Evicted from MGS (at 192.168.203.152@tcp) after server handle changed from 0xd727712cac8e47 to 0xd727712cac91dc
[ 1937.910026] LustreError: lustre-MDT0000-mdc-ffff946f42cee000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 1947.343090] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 19:35:38 (1788392138)
[ 1954.358593] LustreError: lustre-OST0000-osc-ffff946f42cee000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1963.727241] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 19:35:55 (1788392155)
[ 1975.481952] LustreError: lustre-MDT0000-mdc-ffff946f42cee000: operation mds_close to node 192.168.203.152@tcp failed: rc = -19
[ 1975.487920] LustreError: Skipped 1 previous similar message
[ 1993.190517] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 192.168.203.152@tcp) was lost; in progress operations using this service will fail
[ 1993.208100] Lustre: Evicted from MGS (at 192.168.203.152@tcp) after server handle changed from 0xd727712cac91dc to 0xd727712cad15ee
[ 1996.368599] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 1997.303504] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2075.108199] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 192.168.203.152@tcp) was lost; in progress operations using this service will fail
[ 2075.121761] Lustre: Evicted from MGS (at 192.168.203.152@tcp) after server handle changed from 0xd727712cad15ee to 0xd727712cb076fa
[ 2082.660799] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2083.570172] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2146.506761] LustreError: lustre-MDT0000-mdc-ffff946f42cee000: operation ldlm_enqueue to node 192.168.203.152@tcp failed: rc = -19
[ 2146.516153] LustreError: Skipped 3 previous similar messages
[ 2161.635151] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 192.168.203.152@tcp) was lost; in progress operations using this service will fail
[ 2161.644916] Lustre: Evicted from MGS (at 192.168.203.152@tcp) after server handle changed from 0xd727712cb076fa to 0xd727712cb382c4
[ 2169.528283] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2170.438415] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2195.850081] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 19:39:47 (1788392387)
[ 2215.981320] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2235.710614] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2236.387038] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 192.168.203.152@tcp) was lost; in progress operations using this service will fail
[ 2236.392330] LustreError: Skipped 1 previous similar message
[ 2236.410306] Lustre: Evicted from MGS (at 192.168.203.152@tcp) after server handle changed from 0xd727712cb4ceb1 to 0xd727712cb4ed74
[ 2236.414676] Lustre: Skipped 1 previous similar message
[ 2259.440684] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2288.190450] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2326.633890] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2368.997729] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 192.168.203.152@tcp) was lost; in progress operations using this service will fail
[ 2369.004323] LustreError: Skipped 3 previous similar messages
[ 2369.029138] Lustre: Evicted from MGS (at 192.168.203.152@tcp) after server handle changed from 0xd727712cb6ce36 to 0xd727712cb8171a
[ 2369.034375] Lustre: Skipped 3 previous similar messages
[ 2370.206933] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2440.529138] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 19:43:52 (1788392632)
[ 2451.726040] LustreError: lustre-OST0000-osc-ffff946f42cee000: operation ldlm_enqueue to node 192.168.203.152@tcp failed: rc = -19
[ 2451.730765] LustreError: Skipped 11 previous similar messages
[ 2475.185640] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 2476.037965] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 2782.940969] Lustre: lustre-OST0000-osc-ffff946f42cee000: Connection to lustre-OST0000 (at 192.168.203.152@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2782.955590] Lustre: Skipped 13 previous similar messages
[ 2799.619777] Lustre: lustre-OST0000-osc-ffff946f42cee000: Connection restored to 192.168.203.152@tcp (at 192.168.203.152@tcp)
[ 2799.630181] Lustre: Skipped 26 previous similar messages
[ 2809.435556] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 2810.724870] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3116.655643] LustreError: lustre-OST0000-osc-ffff946f42cee000: operation ldlm_enqueue to node 192.168.203.152@tcp failed: rc = -19
[ 3116.667315] LustreError: Skipped 1 previous similar message
[ 3159.638721] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3162.172584] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3414.015876] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 20:00:03 (1788393603)
[ 3431.903183] Lustre: 50649:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788393607/real 1788393607]  req@ffff946f4582b480 x1875263167679360/t0(0) o101->lustre-MDT0000-mdc-ffff946f42cee000@192.168.203.152@tcp:12/10 lens 576/1152 e 0 to 1 dl 1788393623 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3431.932339] Lustre: 50649:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 3431.948455] Lustre: lustre-MDT0000-mdc-ffff946f42cee000: Connection to lustre-MDT0000 (at 192.168.203.152@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3431.974996] Lustre: Skipped 1 previous similar message
[ 3432.010783] Lustre: lustre-MDT0000-mdc-ffff946f42cee000: Connection restored to 192.168.203.152@tcp (at 192.168.203.152@tcp)
[ 3432.022273] Lustre: Skipped 1 previous similar message
[ 3439.076292] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 20:00:29 (1788393629)
[ 3464.470439] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 20:00:55 (1788393655)
[ 3488.498803] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 20:01:19 (1788393679)
[ 3489.428134] Lustre: Mounted lustre-client - version 2.17.57_103_g4d1530a
[ 3521.510438] Lustre: 2356:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788393697/real 1788393697]  req@ffff946f4582a680 x1875263167698176/t0(0) o400->lustre-MDT0000-mdc-ffff946f42cee000@192.168.203.152@tcp:12/10 lens 224/224 e 0 to 1 dl 1788393713 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3521.551487] Lustre: 2356:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 3521.560829] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 192.168.203.152@tcp) was lost; in progress operations using this service will fail
[ 3521.586648] LustreError: Skipped 1 previous similar message
[ 3521.614063] Lustre: Evicted from MGS (at 192.168.203.152@tcp) after server handle changed from 0xd727712cb9bc9b to 0xd727712cd7468c
[ 3521.623996] Lustre: Skipped 1 previous similar message
[ 3522.866293] Lustre: 15922:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.152@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 3543.745129] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 3544.973404] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3547.092105] Lustre: Unmounted lustre-client
[ 3552.519762] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 20:02:23 (1788393743)
[ 3672.543177] Lustre: 2353:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788393848/real 1788393848]  req@ffff946f436d5880 x1875263167733632/t0(0) o4->lustre-OST0000-osc-ffff946f42cee000@192.168.203.152@tcp:6/4 lens 488/448 e 0 to 1 dl 1788393864 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 3672.543177] Lustre: 2354:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788393848/real 1788393848]  req@ffff946f436d6d80 x1875263167733504/t0(0) o4->lustre-OST0000-osc-ffff946f42cee000@192.168.203.152@tcp:6/4 lens 488/448 e 0 to 1 dl 1788393864 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 3672.543196] Lustre: 2353:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 51 previous similar messages
[ 3672.574298] Lustre: 2354:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 54 previous similar messages
[ 3847.739559] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 20:07:18 (1788394038)
[ 3898.372734] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 20:08:08 (1788394088)
[ 3901.072882] Lustre: Unmounted lustre-client
[ 3924.071706] Lustre: Mounted lustre-client - version 2.17.57_103_g4d1530a
[ 3930.615921] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 20:08:41 (1788394121)
[ 3931.071157] LustreError: 56588:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 3932.140073] LustreError: 56588:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 3932.308183] Lustre: *** cfs_fail_loc=305, val=0***
[ 3956.954755] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 20:09:07 (1788394147)
[ 3957.684654] Lustre: Mounted lustre-client - version 2.17.57_103_g4d1530a
[ 3960.201425] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 3970.511347] Lustre: Unmounted lustre-client
[ 3978.074469] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 20:09:28 (1788394168)
[ 4064.610620] LustreError: lustre-MDT0000-mdc-ffff946f4afa2800: operation mds_reint to node 192.168.203.152@tcp failed: rc = -19
[ 4064.616861] Lustre: lustre-MDT0000-mdc-ffff946f4afa2800: Connection to lustre-MDT0000 (at 192.168.203.152@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4064.625202] Lustre: Skipped 23 previous similar messages
[ 4082.661414] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 192.168.203.152@tcp) was lost; in progress operations using this service will fail
[ 4092.902524] Lustre: Evicted from MGS (at 192.168.203.152@tcp) after server handle changed from 0xd727712cd764c3 to 0xd727712cda1b4b
[ 4092.925558] Lustre: MGC192.168.203.152@tcp: Connection restored to 192.168.203.152@tcp (at 192.168.203.152@tcp)
[ 4092.935385] Lustre: Skipped 24 previous similar messages
[ 4248.574560] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 20:13:59 (1788394439)
[ 4253.790322] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4270.069312] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 192.168.203.152@tcp) was lost; in progress operations using this service will fail
[ 4270.106641] Lustre: Evicted from MGS (at 192.168.203.152@tcp) after server handle changed from 0xd727712cda1b4b to 0xd727712cdee2ed
[ 4270.142684] LustreError: lustre-MDT0000-mdc-ffff946f4afa2800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4284.040192] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 20:14:34 (1788394474)
[ 4284.602295] Lustre: Mounted lustre-client - version 2.17.57_103_g4d1530a
[ 4292.105186] LustreError: lustre-OST0000-osc-ffff946f4afa2800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4302.304659] Lustre: 2353:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788394478/real 1788394478]  req@ffff946f434e9f80 x1875263172389760/t0(0) o103->lustre-OST0000-osc-ffff946f4ade4800@192.168.203.152@tcp:17/18 lens 328/224 e 0 to 1 dl 1788394494 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295
[ 4302.343149] Lustre: 2353:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 64 previous similar messages
[ 4305.232870] Lustre: Unmounted lustre-client
[ 4312.877076] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 20:15:03 (1788394503)
[ 4319.665107] LustreError: lustre-MDT0000-mdc-ffff946f4afa2800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4319.688184] LustreError: 60991:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4328.376975] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 20:15:18 (1788394518)
[ 4328.631969] LustreError: 61692:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4333.071794] LustreError: lustre-MDT0000-mdc-ffff946f4afa2800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4333.095815] LustreError: 61711:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4333.114927] LustreError: 61692:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=526
[ 4333.128302] LustreError: 61692:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting
[ 4333.178452] LustreError: 61716:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4334.261545] LustreError: 61722:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4334.281881] LustreError: 61722:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4336.634363] LustreError: 61745:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4336.660058] LustreError: 61745:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4341.291084] LustreError: 61789:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4341.303718] LustreError: 61789:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4351.102325] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 20:15:41 (1788394541)
[ 4387.504104] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4388.816773] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4398.880953] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 20:16:29 (1788394589)
[ 4438.080281] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4440.294892] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4450.052305] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 20:17:20 (1788394640)
[ 4515.325911] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4516.971827] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4525.182182] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 20:18:35 (1788394715)
[ 4564.927515] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4566.838399] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4573.218082] Lustre: Unmounted lustre-client
[ 4637.996778] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4640.736753] Lustre: Mounted lustre-client - version 2.17.57_103_g4d1530a
[ 4648.284756] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 20:20:38 (1788394838)
[ 4651.091759] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 4653.033814] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 20:20:43 (1788394843)
[ 4684.745449] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 20:21:15 (1788394875)
[ 4686.380291] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 4687.987234] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 20:21:18 (1788394878)
[ 4689.490077] Lustre: *** cfs_fail_loc=805, val=0***
[ 4689.529481] Lustre: Mounted lustre-client - version 2.17.57_103_g4d1530a
[ 4694.859718] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4715.497911] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 192.168.203.152@tcp) was lost; in progress operations using this service will fail
[ 4715.525272] Lustre: Evicted from MGS (at 192.168.203.152@tcp) after server handle changed from 0xd727712cdeed0b to 0xd727712cdeef26
[ 4715.535574] Lustre: MGC192.168.203.152@tcp: Connection restored to 192.168.203.152@tcp (at 192.168.203.152@tcp)
[ 4715.542295] Lustre: Skipped 12 previous similar messages
[ 4716.255464] Lustre: lustre-MDT0000-mdc-ffff946f451f8800: Connection to lustre-MDT0000 (at 192.168.203.152@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4716.274945] Lustre: Skipped 12 previous similar messages
[ 4717.161917] Lustre: 68353:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.152@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 4732.413023] Lustre: Unmounted lustre-client
[ 4739.570577] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 20:22:10 (1788394930)
[ 4766.605402] LustreError: 2352:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff946f434e9180 x1875263172471680/t94489280516(94489280516) o101->lustre-MDT0000-mdc-ffff946f48375000@192.168.203.152@tcp:12/10 lens 664/608 e 0 to 0 dl 1788394974 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 4776.384491] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4777.952279] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4785.883244] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 20:22:56 (1788394976)
[ 4787.266042] LustreError: lustre-OST0000-osc-ffff946f48375000: operation ost_write to node 192.168.203.152@tcp failed: rc = -107
[ 4787.275523] LustreError: Skipped 3 previous similar messages
[ 4787.285064] LustreError: lustre-OST0000-osc-ffff946f48375000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4787.299642] Lustre: 2353:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.152@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 4787.325945] LustreError: 72893:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff946f48375000: namespace resource [0x240000400:0x5ca2:0x0].0x0 (ffff946f46abf000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4795.481922] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 20:23:06 (1788394986)
[ 4797.370923] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 4799.332889] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 20:23:09 (1788394989)
[ 4801.295346] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 4803.318157] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 20:23:13 (1788394993)
[ 4804.931672] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 4806.265356] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 20:23:17 (1788394997)
[ 4807.342836] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 4809.143484] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 20:23:19 (1788394999)
[ 4811.124921] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 4812.988806] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 20:23:23 (1788395003)
[ 4814.938896] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 4816.413829] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 20:23:27 (1788395007)
[ 4818.114593] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 4819.583378] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 20:23:30 (1788395010)
[ 4821.082315] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 4822.651845] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 20:23:33 (1788395013)
[ 4824.205547] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 4826.356237] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 20:23:36 (1788395016)
[ 4828.045843] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 4830.218742] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 20:23:40 (1788395020)
[ 4832.287629] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 4834.410406] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 20:23:44 (1788395024)
[ 4836.847432] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 4838.601660] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 20:23:49 (1788395029)
[ 4858.861504] Lustre: 68353:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.152@tcp: IR log lustre-cliir failed, not fatal: rc = -5
[ 4868.048180] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 20:24:18 (1788395058)
[ 4896.869299] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 20:24:47 (1788395087)
[ 4897.400500] Lustre: *** cfs_fail_loc=51b, val=3***
[ 4907.316622] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 20:24:58 (1788395098)
[ 4907.642168] Lustre: *** cfs_fail_loc=51b, val=4***
[ 4918.206121] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 20:25:08 (1788395108)
[ 4918.480930] Lustre: *** cfs_fail_loc=50f, val=3***
[ 4927.957251] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 20:25:18 (1788395118)
[ 4928.433174] Lustre: *** cfs_fail_loc=50f, val=4***
[ 4937.408685] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 20:25:28 (1788395128)
[ 4937.765859] Lustre: *** cfs_fail_loc=510, val=3***
[ 4947.535843] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 20:25:37 (1788395137)
[ 4947.988119] Lustre: *** cfs_fail_loc=51b, val=3***
[ 4958.827417] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 20:25:49 (1788395149)
[ 4959.087933] Lustre: *** cfs_fail_loc=51c, val=3***
[ 4994.975628] Lustre: 2353:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788395132/real 1788395132]  req@ffff946f434e8e00 x1875263172512128/t0(0) o400->lustre-OST0000-osc-ffff946f48375000@192.168.203.152@tcp:28/4 lens 224/224 e 0 to 1 dl 1788395187 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4995.016024] Lustre: 2353:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 16 previous similar messages
[ 5024.879967] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 20:26:55 (1788395215)
[ 5025.302913] LustreError: 83099:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5028.119386] LustreError: lustre-MDT0000-mdc-ffff946f48375000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5028.134318] LustreError: 83114:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff946f48375000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff946f43787f00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5029.375622] LustreError: 83099:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5031.478906] LustreError: 83130:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5034.356885] LustreError: lustre-MDT0000-mdc-ffff946f48375000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5034.375553] LustreError: 83144:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff946f48375000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff946f4545cb00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5034.404795] LustreError: 83144:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5034.410131] LustreError: 83149:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5034.419210] LustreError: 83149:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5035.559740] LustreError: 83130:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5042.497903] LustreError: lustre-MDT0000-mdc-ffff946f48375000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5042.902703] LustreError: 83185:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5045.781269] LustreError: lustre-MDT0000-mdc-ffff946f48375000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5045.868380] LustreError: 83205:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5046.975189] LustreError: 83185:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5046.983480] LustreError: 83185:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff946f48375000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5048.100213] LustreError: 83222:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5048.120090] LustreError: 83222:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 12 previous similar messages
[ 5052.824530] LustreError: 83260:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5055.793493] LustreError: lustre-MDT0000-mdc-ffff946f48375000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5055.803109] LustreError: 83274:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff946f48375000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff946f43787c00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5055.833220] LustreError: 83274:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5056.895177] LustreError: 83260:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5063.825198] LustreError: lustre-MDT0000-mdc-ffff946f48375000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5064.165426] LustreError: 83316:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5067.024617] LustreError: lustre-MDT0000-mdc-ffff946f48375000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5067.038608] LustreError: 83330:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff946f48375000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff946f43787400) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5067.078845] LustreError: 83335:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5067.078855] LustreError: 83335:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5067.117917] LustreError: 83330:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5068.240494] LustreError: 83316:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5071.582994] LustreError: lustre-MDT0000-mdc-ffff946f48375000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5072.751591] LustreError: 83354:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff946f48375000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5080.477199] LustreError: 83425:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5080.486119] LustreError: 83425:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5083.836644] LustreError: lustre-MDT0000-mdc-ffff946f48375000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5083.928866] LustreError: 83447:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5083.936087] LustreError: 83447:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 29 previous similar messages
[ 5084.567143] LustreError: 83425:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5084.577988] LustreError: 83425:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5084.595182] LustreError: 83425:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff946f48375000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5095.040722] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 20:28:05 (1788395285)
[ 5114.566418] LustreError: 79852:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5114.574857] LustreError: 79852:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5118.639137] LustreError: 79852:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5118.642791] LustreError: 79852:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5125.563994] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 20:28:36 (1788395316)
[ 5175.883073] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 20:29:26 (1788395366)
[ 5238.581734] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 20:30:29 (1788395429)
[ 5271.198626] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 20:31:01 (1788395461)
[ 5271.845234] Lustre: Mounted lustre-client - version 2.17.57_103_g4d1530a
[ 5395.004918] Lustre: Unmounted lustre-client
[ 5400.503732] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 20:33:11 (1788395591)
[ 5401.169060] LustreError: 87754:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5403.319435] LustreError: lustre-OST0000-osc-ffff946f48375000: operation ldlm_enqueue to node 192.168.203.152@tcp failed: rc = -107
[ 5403.333156] LustreError: Skipped 9 previous similar messages
[ 5403.347624] Lustre: lustre-OST0000-osc-ffff946f48375000: Connection to lustre-OST0000 (at 192.168.203.152@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5403.373893] Lustre: Skipped 17 previous similar messages
[ 5403.386601] LustreError: lustre-OST0000-osc-ffff946f48375000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5403.399292] LustreError: 87836:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff946f48375000: namespace resource [0x240000400:0x5ccd:0x0].0x0 (ffff946f46abf600) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5403.431782] LustreError: 87836:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5403.695156] LustreError: 87754:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5403.703422] Lustre: 2354:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.152@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5409.953306] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 20:33:20 (1788395600)
[ 5467.122232] Lustre: lustre-MDT0000-mdc-ffff946f48375000: Connection restored to 192.168.203.152@tcp (at 192.168.203.152@tcp)
[ 5467.131491] Lustre: Skipped 19 previous similar messages
[ 5474.582563] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 20:34:25 (1788395665)
[ 5476.144830] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 5478.386173] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 20:34:28 (1788395668)
[ 5543.575286] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 5545.227784] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 20:35:35 (1788395735)
[ 5601.248853] Lustre: 90152:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788395738/real 1788395738]  req@ffff946f44772300 x1875263172667136/t0(0) o36->lustre-MDT0000-mdc-ffff946f48375000@192.168.203.152@tcp:12/10 lens 496/440 e 0 to 1 dl 1788395793 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ln.0' uid:0 gid:0 projid:4294967295
[ 5601.275816] Lustre: 90152:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 5602.927297] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 20:36:33 (1788395793)
[ 5605.066066] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 5607.276390] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 20:36:37 (1788395797)
[ 5608.682429] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 5610.244885] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 20:36:41 (1788395801)
[ 5641.874897] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 20:37:12 (1788395832)
[ 5653.025857] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5678.047188] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 192.168.203.152@tcp) was lost; in progress operations using this service will fail
[ 5678.056395] LustreError: Skipped 3 previous similar messages
[ 5688.313594] Lustre: Evicted from MGS (at 192.168.203.152@tcp) after server handle changed from 0xd727712cdefb74 to 0xd727712cdf1552
[ 5688.325590] Lustre: Skipped 3 previous similar messages
[ 5701.200569] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5702.919682] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5712.753917] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 20:38:23 (1788395903)
[ 5715.116661] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 5717.213656] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 20:38:27 (1788395907)
[ 5730.282277] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 192.168.203.152@tcp) was lost; in progress operations using this service will fail
[ 5730.310502] Lustre: Evicted from MGS (at 192.168.203.152@tcp) after server handle changed from 0xd727712cdf1552 to 0xd727712cdf180e
[ 5744.241363] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 20:38:54 (1788395934)
[ 5793.084574] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 20:39:43 (1788395983)
[ 5851.878225] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5855.784839] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5939.174604] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 192.168.203.152@tcp) was lost; in progress operations using this service will fail
[ 5939.197333] LustreError: Skipped 1 previous similar message
[ 5949.479410] Lustre: Evicted from MGS (at 192.168.203.152@tcp) after server handle changed from 0xd727712cdf1b48 to 0xd727712cdf2bcc
[ 5949.498179] Lustre: Skipped 1 previous similar message
[ 5955.725426] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5957.447088] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6004.955813] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6006.882778] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6155.858600] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 20:45:46 (1788396346)
[ 6165.942968] LustreError: lustre-OST0000-osc-ffff946f48375000: operation ost_setattr to node 192.168.203.152@tcp failed: rc = -19
[ 6165.958618] LustreError: Skipped 17 previous similar messages
[ 6165.966106] Lustre: lustre-OST0000-osc-ffff946f48375000: Connection to lustre-OST0000 (at 192.168.203.152@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6165.977935] Lustre: Skipped 9 previous similar messages
[ 6231.107837] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6236.010624] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6501.838020] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 20:51:32 (1788396692)
[ 6666.225143] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 192.168.203.152@tcp) was lost; in progress operations using this service will fail
[ 6666.244871] LustreError: Skipped 1 previous similar message
[ 6666.261810] Lustre: Evicted from MGS (at 192.168.203.152@tcp) after server handle changed from 0xd727712cdf2d0e to 0xd727712ceecfa9
[ 6666.284222] Lustre: Skipped 1 previous similar message
[ 6666.312386] Lustre: MGC192.168.203.152@tcp: Connection restored to 192.168.203.152@tcp (at 192.168.203.152@tcp)
[ 6666.331296] Lustre: Skipped 12 previous similar messages
[ 6698.975358] Lustre: 2354:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788396847/real 1788396848]  req@ffff946f6d514a80 x1875263190368768/t0(0) o400->lustre-MDT0000-mdc-ffff946f48375000@192.168.203.152@tcp:12/10 lens 224/224 e 0 to 1 dl 1788396890 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6699.009506] Lustre: 2354:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 23 previous similar messages
[ 6700.386676] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 20:54:50 (1788396890)
[ 6702.226532] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 6703.915910] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 20:54:54 (1788396894)
[ 6705.627288] LustreError: lustre-MDT0000-mdc-ffff946f48375000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6712.754489] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 20:55:03 (1788396903)
[ 6880.880787] LustreError: lustre-OST0000-osc-ffff946f48375000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6882.967443] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 20:57:53 (1788397073)
[ 6909.921160] Lustre: lustre-OST0000-osc-ffff946f48375000: Connection to lustre-OST0000 (at 192.168.203.152@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6909.937671] Lustre: Skipped 3 previous similar messages
[ 6926.725964] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 20:58:37 (1788397117)
[ 6928.170199] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 6929.798329] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 20:58:40 (1788397120)
[ 6931.190280] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 6932.772499] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 20:58:43 (1788397123)
[ 6966.981869] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 20:59:17 (1788397157)
[ 7004.151709] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 192.168.203.152@tcp) was lost; in progress operations using this service will fail
[ 7004.180358] Lustre: Evicted from MGS (at 192.168.203.152@tcp) after server handle changed from 0xd727712ceecfa9 to 0xd727712ceef73a
[ 7004.190219] Lustre: Skipped 1 previous similar message
[ 7018.630862] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 21:00:09 (1788397209)
[ 7020.206862] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7021.745710] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 21:00:12 (1788397212)
[ 7023.536325] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7025.368409] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 21:00:15 (1788397215)
[ 7036.050953] Lustre: Unmounted lustre-client
[ 7038.050263] Lustre: Mounted lustre-client - version 2.17.57_103_g4d1530a
[ 7040.757734] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7071.173058] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2