[ 1872.024687] LustreError: lustre-OST0000-osc-ffff9de547352800: operation ost_sync to node 192.168.203.114@tcp failed: rc = -107
[ 1872.046787] LustreError: lustre-OST0000-osc-ffff9de547352800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1872.067396] Lustre: 2360:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.114@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1878.258700] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 18:59:28 (1786748368)
[ 1880.371750] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1881.999950] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 18:59:31 (1786748371)
[ 1883.421777] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1884.766841] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 18:59:34 (1786748374)
[ 1908.206337] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 192.168.203.114@tcp) was lost; in progress operations using this service will fail
[ 1908.227712] Lustre: Evicted from MGS (at 192.168.203.114@tcp) after server handle changed from 0x95ff34474c6c86c to 0x95ff34474c6e3b6
[ 1909.548875] Lustre: 15928:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.114@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2018.070751] LustreError: lustre-MDT0000-mdc-ffff9de547352800: operation mds_reint to node 192.168.203.114@tcp failed: rc = -19
[ 2018.083435] LustreError: Skipped 1 previous similar message
[ 2037.215221] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 192.168.203.114@tcp) was lost; in progress operations using this service will fail
[ 2047.401806] Lustre: Evicted from MGS (at 192.168.203.114@tcp) after server handle changed from 0x95ff34474c6e3b6 to 0x95ff34474c8a34d
[ 2063.427445] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 19:02:33 (1786748553)
[ 2063.844247] Lustre: *** cfs_fail_loc=305, val=0***
[ 2063.848470] Lustre: Skipped 2 previous similar messages
[ 2084.704629] LustreError: lustre-OST0001-osc-ffff9de547352800: operation ost_connect to node 192.168.203.114@tcp failed: rc = -75
[ 2110.441349] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 192.168.203.114@tcp) was lost; in progress operations using this service will fail
[ 2110.465247] Lustre: Evicted from MGS (at 192.168.203.114@tcp) after server handle changed from 0x95ff34474c8a34d to 0x95ff34474c8a767
[ 2112.237381] Lustre: 15928:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.114@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2131.961208] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2133.082479] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2142.590623] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 19:03:52 (1786748632)
[ 2156.521622] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 192.168.203.114@tcp) was lost; in progress operations using this service will fail
[ 2156.537583] Lustre: lustre-MDT0000-mdc-ffff9de547352800: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2156.547464] Lustre: Evicted from MGS (at 192.168.203.114@tcp) after server handle changed from 0x95ff34474c8a767 to 0x95ff34474c8ad64
[ 2156.565295] Lustre: Skipped 12 previous similar messages
[ 2156.587553] Lustre: MGC192.168.203.114@tcp: Connection restored to 192.168.203.114@tcp (at 192.168.203.114@tcp)
[ 2156.602730] Lustre: Skipped 15 previous similar messages
[ 2156.662783] LustreError: lustre-MDT0000-mdc-ffff9de547352800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2176.435296] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 19:04:26 (1786748666)
[ 2189.098972] LustreError: lustre-OST0000-osc-ffff9de547352800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2206.730990] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 19:04:56 (1786748696)
[ 2220.504860] LustreError: lustre-MDT0000-mdc-ffff9de547352800: operation mds_reint to node 192.168.203.114@tcp failed: rc = -19
[ 2239.455242] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 192.168.203.114@tcp) was lost; in progress operations using this service will fail
[ 2239.494945] Lustre: Evicted from MGS (at 192.168.203.114@tcp) after server handle changed from 0x95ff34474c8ad64 to 0x95ff34474c900b5
[ 2269.699030] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2272.541332] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2356.191439] Lustre: 2359:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786748831/real 1786748831]  req@ffff9de544639180 x1873539555057152/t0(0) o400->MGC192.168.203.114@tcp@192.168.203.114@tcp:26/25 lens 224/224 e 0 to 1 dl 1786748847 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2356.212268] Lustre: 2359:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 12 previous similar messages
[ 2356.219409] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 192.168.203.114@tcp) was lost; in progress operations using this service will fail
[ 2356.243355] Lustre: Evicted from MGS (at 192.168.203.114@tcp) after server handle changed from 0x95ff34474c900b5 to 0x95ff34474ca949a
[ 2384.257995] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2386.191677] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2451.460101] LustreError: lustre-MDT0000-mdc-ffff9de547352800: operation mds_reint to node 192.168.203.114@tcp failed: rc = -19
[ 2451.479851] LustreError: Skipped 1 previous similar message
[ 2467.807194] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 192.168.203.114@tcp) was lost; in progress operations using this service will fail
[ 2477.992119] Lustre: Evicted from MGS (at 192.168.203.114@tcp) after server handle changed from 0x95ff34474ca949a to 0x95ff34474cc2d3a
[ 2479.421068] Lustre: 15928:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.114@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2503.228937] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2505.129934] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2535.324171] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 19:10:24 (1786749024)
[ 2557.934209] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 192.168.203.114@tcp) was lost; in progress operations using this service will fail
[ 2557.957696] Lustre: Evicted from MGS (at 192.168.203.114@tcp) after server handle changed from 0x95ff34474cc2d3a to 0x95ff34474cd0a8c
[ 2568.266626] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2608.710940] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2645.484063] Lustre: 15928:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.114@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2657.488800] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2689.567311] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 192.168.203.114@tcp) was lost; in progress operations using this service will fail
[ 2689.592809] LustreError: Skipped 2 previous similar messages
[ 2698.805706] Lustre: Evicted from MGS (at 192.168.203.114@tcp) after server handle changed from 0x95ff34474cd71da to 0x95ff34474cdd3fd
[ 2698.825455] Lustre: Skipped 2 previous similar messages
[ 2700.996803] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2727.677549] LustreError: lustre-MDT0000-mdc-ffff9de547352800: operation ldlm_enqueue to node 192.168.203.114@tcp failed: rc = -19
[ 2727.696015] LustreError: Skipped 7 previous similar messages
[ 2756.469855] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2764.880877] Lustre: lustre-MDT0000-mdc-ffff9de547352800: Connection restored to 192.168.203.114@tcp (at 192.168.203.114@tcp)
[ 2764.890607] Lustre: Skipped 17 previous similar messages
[ 2784.103537] Lustre: lustre-MDT0000-mdc-ffff9de547352800: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2784.133271] Lustre: Skipped 9 previous similar messages
[ 2820.441658] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2908.483130] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 19:16:38 (1786749398)
[ 2967.508412] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 2970.275958] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3253.381796] LustreError: lustre-OST0000-osc-ffff9de547352800: operation ldlm_enqueue to node 192.168.203.114@tcp failed: rc = -19
[ 3253.397348] LustreError: Skipped 5 previous similar messages
[ 3296.362190] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3298.496311] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3584.251613] Lustre: lustre-OST0000-osc-ffff9de547352800: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3584.275060] Lustre: Skipped 3 previous similar messages
[ 3605.318555] Lustre: lustre-OST0000-osc-ffff9de547352800: Connection restored to 192.168.203.114@tcp (at 192.168.203.114@tcp)
[ 3605.330283] Lustre: Skipped 6 previous similar messages
[ 3621.878381] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3623.798435] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3882.418604] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 19:32:51 (1786750371)
[ 3900.384131] Lustre: 50566:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786750375/real 1786750375]  req@ffff9de543e21f80 x1873539576938752/t0(0) o101->lustre-MDT0000-mdc-ffff9de547352800@192.168.203.114@tcp:12/10 lens 576/1152 e 0 to 1 dl 1786750391 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3900.428804] Lustre: 50566:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 3907.821151] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 19:33:17 (1786750397)
[ 3932.375547] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 19:33:42 (1786750422)
[ 3956.056790] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 19:34:06 (1786750446)
[ 3956.486221] Lustre: Mounted lustre-client - version 2.17.57_3_g6085841
[ 3988.191232] Lustre: 2359:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786750463/real 1786750463]  req@ffff9de543569500 x1873539576958336/t0(0) o400->lustre-MDT0000-mdc-ffff9de5435d2000@192.168.203.114@tcp:12/10 lens 224/224 e 0 to 1 dl 1786750479 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3988.237215] Lustre: 2359:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 3988.247844] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 192.168.203.114@tcp) was lost; in progress operations using this service will fail
[ 3988.257568] LustreError: Skipped 3 previous similar messages
[ 3988.271963] Lustre: Evicted from MGS (at 192.168.203.114@tcp) after server handle changed from 0x95ff34474cf6e64 to 0x95ff34474e420d8
[ 3988.283392] Lustre: Skipped 3 previous similar messages
[ 3990.005520] Lustre: 15928:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.114@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 4012.144579] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4013.493475] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4015.535898] Lustre: Unmounted lustre-client
[ 4021.699353] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 19:35:11 (1786750511)
[ 4143.007872] Lustre: 2360:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786750618/real 1786750618]  req@ffff9de543569500 x1873539576994944/t0(0) o4->lustre-OST0000-osc-ffff9de547352800@192.168.203.114@tcp:6/4 lens 488/448 e 0 to 1 dl 1786750634 ref 2 fl Rpc:XQr/602/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4143.041638] Lustre: 2360:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 63 previous similar messages
[ 4191.199851] Lustre: lustre-OST0000-osc-ffff9de547352800: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4191.229030] Lustre: Skipped 14 previous similar messages
[ 4207.557465] Lustre: lustre-OST0000-osc-ffff9de547352800: Connection restored to 192.168.203.114@tcp (at 192.168.203.114@tcp)
[ 4207.572459] Lustre: Skipped 16 previous similar messages
[ 4316.314237] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 19:40:06 (1786750806)
[ 4364.380845] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 19:40:54 (1786750854)
[ 4367.182616] Lustre: Unmounted lustre-client
[ 4387.501775] Lustre: Mounted lustre-client - version 2.17.57_3_g6085841
[ 4393.211761] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 19:41:23 (1786750883)
[ 4393.579790] LustreError: 56488:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4394.624161] LustreError: 56488:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4394.750549] Lustre: *** cfs_fail_loc=305, val=0***
[ 4418.864834] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 19:41:48 (1786750908)
[ 4419.669806] Lustre: Mounted lustre-client - version 2.17.57_3_g6085841
[ 4422.436589] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4432.816478] Lustre: Unmounted lustre-client
[ 4442.888161] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 19:42:12 (1786750932)
[ 4541.224440] LustreError: lustre-MDT0000-mdc-ffff9de5448e3800: operation mds_reint to node 192.168.203.114@tcp failed: rc = -19
[ 4541.240289] LustreError: Skipped 1 previous similar message
[ 4558.815465] Lustre: 2361:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786751034/real 1786751034]  req@ffff9de547827480 x1873539579678208/t0(0) o400->MGC192.168.203.114@tcp@192.168.203.114@tcp:26/25 lens 224/224 e 0 to 1 dl 1786751050 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4558.855508] Lustre: 2361:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 84 previous similar messages
[ 4558.873346] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 192.168.203.114@tcp) was lost; in progress operations using this service will fail
[ 4569.072775] Lustre: Evicted from MGS (at 192.168.203.114@tcp) after server handle changed from 0x95ff34474e43a69 to 0x95ff34474e6f494
[ 4736.040551] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 19:47:06 (1786751226)
[ 4741.248766] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4757.476670] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 192.168.203.114@tcp) was lost; in progress operations using this service will fail
[ 4757.514910] Lustre: Evicted from MGS (at 192.168.203.114@tcp) after server handle changed from 0x95ff34474e6f494 to 0x95ff34474ebb694
[ 4757.596751] LustreError: lustre-MDT0000-mdc-ffff9de5448e3800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4773.591375] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 19:47:43 (1786751263)
[ 4774.278568] Lustre: Mounted lustre-client - version 2.17.57_3_g6085841
[ 4781.588229] LustreError: lustre-OST0000-osc-ffff9de5448e3800: operation ldlm_enqueue to node 192.168.203.114@tcp failed: rc = -107
[ 4781.609549] LustreError: lustre-OST0000-osc-ffff9de5448e3800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4781.622852] LustreError: 60218:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff9de5448e3800: namespace resource [0x240000400:0x3d23:0x0].0x0 (ffff9de54463c000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4792.171204] Lustre: Unmounted lustre-client
[ 4798.631857] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 19:48:08 (1786751288)
[ 4802.388702] Lustre: lustre-MDT0000-mdc-ffff9de5448e3800: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4802.400690] Lustre: Skipped 13 previous similar messages
[ 4805.425775] LustreError: lustre-MDT0000-mdc-ffff9de5448e3800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4805.445115] LustreError: 60889:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4812.297102] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 19:48:22 (1786751302)
[ 4812.667246] LustreError: 61592:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4815.348274] Lustre: lustre-MDT0000-mdc-ffff9de5448e3800: Connection restored to 192.168.203.114@tcp (at 192.168.203.114@tcp)
[ 4815.359826] Lustre: Skipped 15 previous similar messages
[ 4817.259632] LustreError: lustre-MDT0000-mdc-ffff9de5448e3800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4817.273307] LustreError: 61609:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4817.291217] LustreError: 61592:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=383
[ 4817.310844] LustreError: 61592:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting
[ 4817.365670] LustreError: 61614:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4818.446938] LustreError: 61621:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4818.463540] LustreError: 61621:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4820.819752] LustreError: 61643:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4820.829951] LustreError: 61643:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4825.616306] LustreError: 61688:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4825.630253] LustreError: 61688:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4834.713513] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 19:48:44 (1786751324)
[ 4872.116647] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4873.695654] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4886.048763] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 19:49:35 (1786751375)
[ 4926.648499] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4928.057056] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4937.912248] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 19:50:27 (1786751427)
[ 5001.213083] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5002.684327] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5010.903947] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 19:51:40 (1786751500)
[ 5046.769268] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5048.212314] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5053.535111] Lustre: Unmounted lustre-client
[ 5107.664602] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5110.111290] Lustre: Mounted lustre-client - version 2.17.57_3_g6085841
[ 5116.491376] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 19:53:26 (1786751606)
[ 5118.974949] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5120.392366] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 19:53:30 (1786751610)
[ 5148.553574] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 19:53:58 (1786751638)
[ 5149.970796] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5151.645081] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 19:54:01 (1786751641)
[ 5152.908087] Lustre: *** cfs_fail_loc=805, val=0***
[ 5152.977954] Lustre: Mounted lustre-client - version 2.17.57_3_g6085841
[ 5157.725431] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5178.847205] Lustre: 2359:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786751654/real 1786751654]  req@ffff9de548984700 x1873539581718400/t0(0) o400->lustre-MDT0000-mdc-ffff9de5447d9800@192.168.203.114@tcp:12/10 lens 224/224 e 0 to 1 dl 1786751670 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5178.850257] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 192.168.203.114@tcp) was lost; in progress operations using this service will fail
[ 5178.885992] Lustre: 2359:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 5178.953876] Lustre: Evicted from MGS (at 192.168.203.114@tcp) after server handle changed from 0x95ff34474ebc0b2 to 0x95ff34474ebc398
[ 5188.566491] Lustre: Unmounted lustre-client
[ 5195.825205] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 19:54:45 (1786751685)
[ 5223.406814] LustreError: 2358:0:(client.c:3435:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff9de543455880 x1873539581729024/t94489280516(94489280516) o101->lustre-MDT0000-mdc-ffff9de558c73800@192.168.203.114@tcp:12/10 lens 664/608 e 0 to 0 dl 1786751730 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5232.189353] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5233.500443] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5241.053443] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 19:55:31 (1786751731)
[ 5242.256627] LustreError: lustre-OST0000-osc-ffff9de558c73800: operation ost_write to node 192.168.203.114@tcp failed: rc = -107
[ 5242.266506] LustreError: Skipped 2 previous similar messages
[ 5242.297896] LustreError: lustre-OST0000-osc-ffff9de558c73800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5242.312135] Lustre: 2360:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.114@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5242.341525] LustreError: 72793:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff9de558c73800: namespace resource [0x240000400:0x3d82:0x0].0x0 (ffff9de54463c200) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5249.526132] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 19:55:39 (1786751739)
[ 5250.977031] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5252.713287] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 19:55:42 (1786751742)
[ 5254.087981] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5256.206940] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 19:55:46 (1786751746)
[ 5257.565856] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5258.925451] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 19:55:49 (1786751749)
[ 5260.606257] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5262.560766] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 19:55:52 (1786751752)
[ 5264.094572] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5266.177384] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 19:55:55 (1786751755)
[ 5268.293615] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5270.619323] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 19:55:59 (1786751759)
[ 5272.746135] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5275.171954] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 19:56:04 (1786751764)
[ 5277.597603] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5280.474808] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 19:56:09 (1786751769)
[ 5282.829617] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5284.746232] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 19:56:14 (1786751774)
[ 5286.428869] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5288.340633] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 19:56:17 (1786751777)
[ 5290.388856] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5291.988463] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 19:56:21 (1786751781)
[ 5295.132405] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5297.432053] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 19:56:26 (1786751786)
[ 5332.004235] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 19:57:01 (1786751821)
[ 5360.776521] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 19:57:30 (1786751850)
[ 5361.409040] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5370.591805] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 19:57:40 (1786751860)
[ 5371.028907] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5380.023899] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 19:57:50 (1786751870)
[ 5380.295269] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5388.311131] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 19:57:58 (1786751878)
[ 5388.547657] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5395.733719] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 19:58:05 (1786751885)
[ 5396.061639] Lustre: *** cfs_fail_loc=510, val=3***
[ 5404.249289] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 19:58:14 (1786751894)
[ 5404.617435] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5413.826572] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 19:58:23 (1786751903)
[ 5414.176806] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5473.759752] Lustre: lustre-OST0001-osc-ffff9de558c73800: Connection to lustre-OST0001 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5473.774438] Lustre: Skipped 13 previous similar messages
[ 5480.819923] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 19:59:30 (1786751970)
[ 5481.360341] LustreError: 82999:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5484.257724] LustreError: lustre-MDT0000-mdc-ffff9de558c73800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5484.274107] LustreError: 83014:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9de558c73800: inode [0x20000a042:0xb:0x0] mdc close failed: rc = -108
[ 5484.295752] LustreError: 83014:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9de558c73800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9de5501fe800) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5484.309409] LustreError: 83019:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5484.318338] Lustre: lustre-MDT0000-mdc-ffff9de558c73800: Connection restored to 192.168.203.114@tcp (at 192.168.203.114@tcp)
[ 5484.322814] LustreError: 83019:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5484.329693] Lustre: Skipped 12 previous similar messages
[ 5485.447186] LustreError: 82999:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5487.582983] LustreError: 83029:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5490.558713] LustreError: lustre-MDT0000-mdc-ffff9de558c73800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5490.578341] LustreError: 83044:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9de558c73800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9de54463c000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5490.609919] LustreError: 83049:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5490.611610] LustreError: 83044:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5491.591990] LustreError: 83029:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5498.720835] LustreError: lustre-MDT0000-mdc-ffff9de558c73800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5499.157100] LustreError: 83096:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5502.015928] LustreError: lustre-MDT0000-mdc-ffff9de558c73800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5502.116575] LustreError: 83116:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5502.127835] LustreError: 83116:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 3 previous similar messages
[ 5503.239377] LustreError: 83096:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5503.250604] LustreError: 83096:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9de558c73800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5509.369722] LustreError: 83171:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5512.349386] LustreError: lustre-MDT0000-mdc-ffff9de558c73800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5512.380698] LustreError: 83186:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9de558c73800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9de5501fed00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5512.409337] LustreError: 83186:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 2 previous similar messages
[ 5512.427442] LustreError: 83191:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5512.439549] LustreError: 83191:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5513.471183] LustreError: 83171:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5520.474837] LustreError: lustre-MDT0000-mdc-ffff9de558c73800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5520.860917] LustreError: 83228:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5523.698920] LustreError: lustre-MDT0000-mdc-ffff9de558c73800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5523.721526] LustreError: 83242:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9de558c73800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9de54463c500) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5523.761263] LustreError: 83242:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5524.935113] LustreError: 83228:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5527.794184] LustreError: lustre-MDT0000-mdc-ffff9de558c73800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5527.850603] LustreError: 83275:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5529.111769] LustreError: 83255:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9de558c73800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5536.740856] LustreError: 83327:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5536.750725] LustreError: 83327:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5540.493940] LustreError: lustre-MDT0000-mdc-ffff9de558c73800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5540.839290] LustreError: 83327:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5540.851526] LustreError: 83327:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5540.875660] LustreError: 83327:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9de558c73800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5544.002461] LustreError: 83379:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5544.014036] LustreError: 83379:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 46 previous similar messages
[ 5551.162593] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 20:00:41 (1786752041)
[ 5570.010925] LustreError: 68252:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5570.021863] LustreError: 68252:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5574.103139] LustreError: 68252:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5574.109326] LustreError: 68252:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5580.218505] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 20:01:10 (1786752070)
[ 5633.052394] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 20:02:02 (1786752122)
[ 5698.024929] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 20:03:08 (1786752188)
[ 5730.746418] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 20:03:40 (1786752220)
[ 5731.147435] Lustre: Mounted lustre-client - version 2.17.57_3_g6085841
[ 5854.495560] Lustre: Unmounted lustre-client
[ 5864.145568] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 20:05:53 (1786752353)
[ 5865.274977] LustreError: 87670:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5867.788675] LustreError: lustre-OST0000-osc-ffff9de558c73800: operation ldlm_enqueue to node 192.168.203.114@tcp failed: rc = -107
[ 5867.806069] LustreError: Skipped 9 previous similar messages
[ 5867.830253] LustreError: lustre-OST0000-osc-ffff9de558c73800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5867.866588] LustreError: 87753:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff9de558c73800: namespace resource [0x240000400:0x3dad:0x0].0x0 (ffff9de5501e0100) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5867.904717] LustreError: 87753:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5868.439119] LustreError: 87670:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5868.457044] Lustre: 2361:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.114@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5876.739386] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 20:06:06 (1786752366)
[ 5936.095265] Lustre: 88338:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786752371/real 1786752371]  req@ffff9de547827b80 x1873539581903360/t0(0) o101->lustre-MDT0000-mdc-ffff9de558c73800@192.168.203.114@tcp:12/10 lens 328/344 e 0 to 1 dl 1786752427 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 5936.142046] Lustre: 88338:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 14 previous similar messages
[ 5944.803723] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 20:07:13 (1786752433)
[ 5946.517604] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 5948.635240] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 20:07:18 (1786752438)
[ 6016.891226] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6019.103392] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 20:08:28 (1786752508)
[ 6077.407375] Lustre: lustre-MDT0000-mdc-ffff9de558c73800: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6077.422606] Lustre: Skipped 15 previous similar messages
[ 6079.452678] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 20:09:28 (1786752568)
[ 6081.621863] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6083.738817] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 20:09:33 (1786752573)
[ 6085.513300] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6087.368630] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 20:09:37 (1786752577)
[ 6124.295565] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 20:10:14 (1786752614)
[ 6137.401360] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6165.471346] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 192.168.203.114@tcp) was lost; in progress operations using this service will fail
[ 6165.485542] LustreError: Skipped 2 previous similar messages
[ 6165.523538] Lustre: Evicted from MGS (at 192.168.203.114@tcp) after server handle changed from 0x95ff34474ebcf1b to 0x95ff34474ebe75c
[ 6165.534411] Lustre: Skipped 2 previous similar messages
[ 6165.548157] Lustre: MGC192.168.203.114@tcp: Connection restored to 192.168.203.114@tcp (at 192.168.203.114@tcp)
[ 6165.554203] Lustre: Skipped 14 previous similar messages
[ 6166.656840] Lustre: 68254:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.114@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6190.708890] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6192.711578] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6202.666820] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 20:11:32 (1786752692)
[ 6205.108411] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6206.979446] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 20:11:36 (1786752696)
[ 6221.815431] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 192.168.203.114@tcp) was lost; in progress operations using this service will fail
[ 6221.847216] Lustre: Evicted from MGS (at 192.168.203.114@tcp) after server handle changed from 0x95ff34474ebe75c to 0x95ff34474ebec17
[ 6235.541291] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 20:12:05 (1786752725)
[ 6285.112692] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 20:12:55 (1786752775)
[ 6342.185752] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6345.675639] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6431.725758] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 192.168.203.114@tcp) was lost; in progress operations using this service will fail
[ 6431.738024] LustreError: Skipped 1 previous similar message
[ 6431.754395] Lustre: Evicted from MGS (at 192.168.203.114@tcp) after server handle changed from 0x95ff34474ebef12 to 0x95ff34474ebfc71
[ 6431.761916] Lustre: Skipped 1 previous similar message
[ 6432.951622] Lustre: 68254:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.114@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6455.850247] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6457.213518] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6492.468311] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6493.706356] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6537.183672] Lustre: 2359:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786752968/real 1786752968]  req@ffff9de572215180 x1873539588463104/t0(0) o400->lustre-MDT0000-mdc-ffff9de558c73800@192.168.203.114@tcp:12/10 lens 224/224 e 0 to 1 dl 1786753028 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6537.213154] Lustre: 2359:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 21 previous similar messages
[ 6623.654789] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 20:18:33 (1786753113)
[ 6634.489154] LustreError: lustre-OST0000-osc-ffff9de558c73800: operation ost_setattr to node 192.168.203.114@tcp failed: rc = -19
[ 6634.504983] LustreError: Skipped 259 previous similar messages
[ 6682.164827] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6684.945604] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6935.411554] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 20:23:45 (1786753425)
[ 7096.299304] Lustre: lustre-MDT0000-mdc-ffff9de558c73800: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7096.341933] Lustre: Skipped 7 previous similar messages
[ 7096.351198] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 192.168.203.114@tcp) was lost; in progress operations using this service will fail
[ 7096.371383] LustreError: Skipped 1 previous similar message
[ 7096.478181] Lustre: Evicted from MGS (at 192.168.203.114@tcp) after server handle changed from 0x95ff34474ec03da to 0x95ff34474fba47d
[ 7096.498802] Lustre: Skipped 1 previous similar message
[ 7096.524159] Lustre: MGC192.168.203.114@tcp: Connection restored to 192.168.203.114@tcp (at 192.168.203.114@tcp)
[ 7096.548835] Lustre: Skipped 9 previous similar messages
[ 7127.768212] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 20:26:57 (1786753617)
[ 7129.241623] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7131.428594] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 20:27:01 (1786753621)
[ 7133.244419] LustreError: lustre-MDT0000-mdc-ffff9de558c73800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7140.265835] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 20:27:10 (1786753630)
[ 7307.051888] LustreError: lustre-OST0000-osc-ffff9de558c73800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7309.132490] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 20:29:58 (1786753798)
[ 7333.856327] Lustre: 2359:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786753804/real 1786753804]  req@ffff9de56a5a4000 x1873539599656832/t0(0) o4->lustre-OST0000-osc-ffff9de558c73800@192.168.203.114@tcp:6/4 lens 4584/448 e 0 to 1 dl 1786753824 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7349.763640] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 20:30:39 (1786753839)
[ 7350.926993] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7352.429669] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 20:30:42 (1786753842)
[ 7353.582614] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7355.464938] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 20:30:45 (1786753845)
[ 7388.152083] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 20:31:18 (1786753878)
[ 7425.004546] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 192.168.203.114@tcp) was lost; in progress operations using this service will fail
[ 7425.025162] Lustre: Evicted from MGS (at 192.168.203.114@tcp) after server handle changed from 0x95ff34474fba47d to 0x95ff34474fbcc69
[ 7425.047115] Lustre: Skipped 1 previous similar message
[ 7440.818522] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 20:32:10 (1786753930)
[ 7442.217840] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7444.277524] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 20:32:13 (1786753933)
[ 7445.846178] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7447.864868] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 20:32:17 (1786753937)
[ 7465.891530] Lustre: Unmounted lustre-client
[ 7467.868343] Lustre: Mounted lustre-client - version 2.17.57_3_g6085841
[ 7470.905341] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7503.240913] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2