[ 1878.660308] LustreError: lustre-OST0000-osc-ffff9cc786f2e000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1878.679464] Lustre: 2350:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.140@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1884.039832] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 18:05:42 (1788213942)
[ 1885.833116] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1887.420822] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 18:05:45 (1788213945)
[ 1889.234239] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1890.728381] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 18:05:48 (1788213948)
[ 1913.326636] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 192.168.206.140@tcp) was lost; in progress operations using this service will fail
[ 1913.356086] Lustre: Evicted from MGS (at 192.168.206.140@tcp) after server handle changed from 0x5b89ed621b3ab06d to 0x5b89ed621b3ac8ca
[ 2017.846511] LustreError: lustre-MDT0000-mdc-ffff9cc786f2e000: operation ldlm_enqueue to node 192.168.206.140@tcp failed: rc = -19
[ 2017.866256] LustreError: Skipped 3 previous similar messages
[ 2037.215354] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 192.168.206.140@tcp) was lost; in progress operations using this service will fail
[ 2047.463348] Lustre: Evicted from MGS (at 192.168.206.140@tcp) after server handle changed from 0x5b89ed621b3ac8ca to 0x5b89ed621b3d17b7
[ 2063.808654] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 18:08:41 (1788214121)
[ 2064.270907] Lustre: *** cfs_fail_loc=305, val=0***
[ 2064.273698] Lustre: Skipped 2 previous similar messages
[ 2102.751342] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 192.168.206.140@tcp) was lost; in progress operations using this service will fail
[ 2113.076559] Lustre: Evicted from MGS (at 192.168.206.140@tcp) after server handle changed from 0x5b89ed621b3d17b7 to 0x5b89ed621b3d212d
[ 2117.717862] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2119.323228] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2127.430619] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 18:09:45 (1788214185)
[ 2143.731347] Lustre: lustre-MDT0000-mdc-ffff9cc786f2e000: Connection to lustre-MDT0000 (at 192.168.206.140@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2143.754131] Lustre: Skipped 12 previous similar messages
[ 2143.760288] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 192.168.206.140@tcp) was lost; in progress operations using this service will fail
[ 2143.790751] Lustre: Evicted from MGS (at 192.168.206.140@tcp) after server handle changed from 0x5b89ed621b3d212d to 0x5b89ed621b3d23cd
[ 2143.806046] Lustre: MGC192.168.206.140@tcp: Connection restored to 192.168.206.140@tcp (at 192.168.206.140@tcp)
[ 2143.809987] Lustre: Skipped 15 previous similar messages
[ 2143.822077] LustreError: lustre-MDT0000-mdc-ffff9cc786f2e000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2163.583405] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 18:10:21 (1788214221)
[ 2176.792272] LustreError: lustre-OST0000-osc-ffff9cc786f2e000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2196.469081] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 18:10:54 (1788214254)
[ 2209.306367] LustreError: lustre-MDT0000-mdc-ffff9cc786f2e000: operation mds_close to node 192.168.206.140@tcp failed: rc = -19
[ 2209.320769] LustreError: Skipped 1 previous similar message
[ 2226.655356] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 192.168.206.140@tcp) was lost; in progress operations using this service will fail
[ 2235.826995] Lustre: Evicted from MGS (at 192.168.206.140@tcp) after server handle changed from 0x5b89ed621b3d23cd to 0x5b89ed621b3d72ef
[ 2250.878576] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2252.899776] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2339.233942] Lustre: 2350:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788214381/real 1788214381]  req@ffff9cc787b3dc00 x1875076317107456/t0(0) o400->MGC192.168.206.140@tcp@192.168.206.140@tcp:26/25 lens 224/224 e 0 to 1 dl 1788214397 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2339.274660] Lustre: 2350:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 12 previous similar messages
[ 2339.281851] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 192.168.206.140@tcp) was lost; in progress operations using this service will fail
[ 2339.322064] Lustre: Evicted from MGS (at 192.168.206.140@tcp) after server handle changed from 0x5b89ed621b3d72ef to 0x5b89ed621b3f1b8e
[ 2365.743198] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2367.869343] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2433.777430] LustreError: lustre-MDT0000-mdc-ffff9cc786f2e000: operation mds_reint to node 192.168.206.140@tcp failed: rc = -19
[ 2433.807330] LustreError: Skipped 2 previous similar messages
[ 2449.887371] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 192.168.206.140@tcp) was lost; in progress operations using this service will fail
[ 2460.074469] Lustre: Evicted from MGS (at 192.168.206.140@tcp) after server handle changed from 0x5b89ed621b3f1b8e to 0x5b89ed621b40d560
[ 2481.487546] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2483.420214] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2513.166741] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 18:16:11 (1788214571)
[ 2535.903217] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 192.168.206.140@tcp) was lost; in progress operations using this service will fail
[ 2535.928727] Lustre: Evicted from MGS (at 192.168.206.140@tcp) after server handle changed from 0x5b89ed621b40d560 to 0x5b89ed621b41a759
[ 2537.846172] Lustre: 15918:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.140@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2553.205736] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2581.770868] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2624.648974] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2665.006936] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2706.911311] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 192.168.206.140@tcp) was lost; in progress operations using this service will fail
[ 2706.941588] LustreError: Skipped 3 previous similar messages
[ 2715.215427] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2717.174730] Lustre: Evicted from MGS (at 192.168.206.140@tcp) after server handle changed from 0x5b89ed621b425e8d to 0x5b89ed621b42a256
[ 2717.187613] Lustre: Skipped 3 previous similar messages
[ 2744.378759] LustreError: lustre-MDT0000-mdc-ffff9cc786f2e000: operation mds_close to node 192.168.206.140@tcp failed: rc = -19
[ 2744.385366] Lustre: lustre-MDT0000-mdc-ffff9cc786f2e000: Connection to lustre-MDT0000 (at 192.168.206.140@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2744.392164] LustreError: Skipped 11 previous similar messages
[ 2744.426194] Lustre: Skipped 9 previous similar messages
[ 2773.488912] Lustre: MGC192.168.206.140@tcp: Connection restored to 192.168.206.140@tcp (at 192.168.206.140@tcp)
[ 2773.499199] Lustre: Skipped 18 previous similar messages
[ 2782.375088] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2867.296769] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 18:22:05 (1788214925)
[ 2920.384395] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 2922.519443] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3249.082685] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3251.301730] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3543.101133] LustreError: lustre-OST0000-osc-ffff9cc786f2e000: operation ldlm_enqueue to node 192.168.206.140@tcp failed: rc = -19
[ 3543.107473] LustreError: Skipped 3 previous similar messages
[ 3543.111842] Lustre: lustre-OST0000-osc-ffff9cc786f2e000: Connection to lustre-OST0000 (at 192.168.206.140@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3543.124093] Lustre: Skipped 3 previous similar messages
[ 3565.349383] Lustre: lustre-OST0000-osc-ffff9cc786f2e000: Connection restored to 192.168.206.140@tcp (at 192.168.206.140@tcp)
[ 3565.361782] Lustre: Skipped 5 previous similar messages
[ 3584.450850] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3586.535475] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3839.653044] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 18:38:17 (1788215897)
[ 3856.865850] Lustre: 50588:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788215900/real 1788215900]  req@ffff9cc788baed80 x1875076338862336/t0(0) o101->lustre-MDT0000-mdc-ffff9cc786f2e000@192.168.206.140@tcp:12/10 lens 576/1152 e 0 to 1 dl 1788215916 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3856.908904] Lustre: 50588:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 3866.680815] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 18:38:44 (1788215924)
[ 3893.403879] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 18:39:11 (1788215951)
[ 3918.955772] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 18:39:36 (1788215976)
[ 3919.423486] Lustre: Mounted lustre-client - version 2.17.57_85_gb71fc30
[ 3949.983153] Lustre: 2352:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788215993/real 1788215993]  req@ffff9cc7a0e78a80 x1875076338882048/t0(0) o400->MGC192.168.206.140@tcp@192.168.206.140@tcp:26/25 lens 224/224 e 0 to 1 dl 1788216009 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3950.011920] Lustre: 2352:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 3950.023148] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 192.168.206.140@tcp) was lost; in progress operations using this service will fail
[ 3950.036770] LustreError: Skipped 2 previous similar messages
[ 3960.303103] Lustre: Evicted from MGS (at 192.168.206.140@tcp) after server handle changed from 0x5b89ed621b43c298 to 0x5b89ed621b58a262
[ 3960.313714] Lustre: Skipped 2 previous similar messages
[ 3965.708715] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 3967.753512] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3969.951861] Lustre: Unmounted lustre-client
[ 3976.455669] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 18:40:34 (1788216034)
[ 4112.863149] Lustre: 2353:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788216156/real 1788216156]  req@ffff9cc7a0e79180 x1875076338915072/t0(0) o4->lustre-OST0000-osc-ffff9cc786f2e000@192.168.206.140@tcp:6/4 lens 488/448 e 0 to 1 dl 1788216172 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4112.900250] Lustre: 2353:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 69 previous similar messages
[ 4145.439605] Lustre: lustre-OST0000-osc-ffff9cc786f2e000: Connection to lustre-OST0000 (at 192.168.206.140@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4145.456177] Lustre: Skipped 14 previous similar messages
[ 4177.387779] Lustre: lustre-OST0000-osc-ffff9cc786f2e000: Connection restored to 192.168.206.140@tcp (at 192.168.206.140@tcp)
[ 4177.397454] Lustre: Skipped 17 previous similar messages
[ 4269.722596] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 18:45:27 (1788216327)
[ 4318.235632] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 18:46:16 (1788216376)
[ 4321.392454] Lustre: Unmounted lustre-client
[ 4341.952226] Lustre: Mounted lustre-client - version 2.17.57_85_gb71fc30
[ 4348.057325] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 18:46:46 (1788216406)
[ 4348.362829] LustreError: 56512:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4349.412121] LustreError: 56512:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4349.525816] Lustre: *** cfs_fail_loc=305, val=0***
[ 4373.131131] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 18:47:10 (1788216430)
[ 4373.572486] Lustre: Mounted lustre-client - version 2.17.57_85_gb71fc30
[ 4375.670377] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4386.000507] Lustre: Unmounted lustre-client
[ 4391.702545] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 18:47:29 (1788216449)
[ 4470.996636] LustreError: lustre-MDT0000-mdc-ffff9cc78927e000: operation mds_reint to node 192.168.206.140@tcp failed: rc = -19
[ 4488.162253] Lustre: 2350:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788216531/real 1788216531]  req@ffff9cc7a3b88a80 x1875076341593984/t0(0) o400->MGC192.168.206.140@tcp@192.168.206.140@tcp:26/25 lens 224/224 e 0 to 1 dl 1788216547 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4488.193436] Lustre: 2350:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 80 previous similar messages
[ 4488.204539] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 192.168.206.140@tcp) was lost; in progress operations using this service will fail
[ 4498.414348] Lustre: Evicted from MGS (at 192.168.206.140@tcp) after server handle changed from 0x5b89ed621b58bbad to 0x5b89ed621b5b7434
[ 4641.381105] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 18:51:39 (1788216699)
[ 4645.691561] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4656.109454] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 192.168.206.140@tcp) was lost; in progress operations using this service will fail
[ 4656.129691] Lustre: Evicted from MGS (at 192.168.206.140@tcp) after server handle changed from 0x5b89ed621b5b7434 to 0x5b89ed621b60365e
[ 4657.585671] LustreError: lustre-MDT0000-mdc-ffff9cc78927e000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4671.290361] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 18:52:09 (1788216729)
[ 4671.670527] Lustre: Mounted lustre-client - version 2.17.57_85_gb71fc30
[ 4679.011346] LustreError: lustre-OST0000-osc-ffff9cc78927e000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4692.249522] Lustre: Unmounted lustre-client
[ 4699.020121] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 18:52:37 (1788216757)
[ 4705.659132] LustreError: lustre-MDT0000-mdc-ffff9cc78927e000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4705.671192] LustreError: 60899:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4712.346181] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 18:52:50 (1788216770)
[ 4712.499338] LustreError: 61599:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4716.475661] LustreError: lustre-MDT0000-mdc-ffff9cc78927e000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4716.493016] LustreError: 61619:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4716.499974] LustreError: 61599:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=1015
[ 4716.510423] LustreError: 61599:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting
[ 4716.566635] LustreError: 61624:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4717.671466] LustreError: 61630:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4717.684194] LustreError: 61630:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4720.023568] LustreError: 61654:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4720.033394] LustreError: 61654:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4724.643738] LustreError: 61699:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4724.658576] LustreError: 61699:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4735.126573] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 18:53:12 (1788216792)
[ 4771.229271] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4772.845033] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4783.524768] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 18:54:01 (1788216841)
[ 4787.692578] Lustre: lustre-OST0000-osc-ffff9cc78927e000: Connection to lustre-OST0000 (at 192.168.206.140@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4787.700547] Lustre: Skipped 19 previous similar messages
[ 4807.376623] Lustre: lustre-OST0000-osc-ffff9cc78927e000: Connection restored to 192.168.206.140@tcp (at 192.168.206.140@tcp)
[ 4807.384494] Lustre: Skipped 17 previous similar messages
[ 4820.553886] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4822.170831] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4833.283062] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 18:54:51 (1788216891)
[ 4893.625850] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4895.329262] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4902.388295] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 18:56:00 (1788216960)
[ 4935.507738] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4936.746860] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4941.946741] Lustre: Unmounted lustre-client
[ 4991.282812] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4993.787134] Lustre: Mounted lustre-client - version 2.17.57_85_gb71fc30
[ 5000.005689] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 18:57:38 (1788217058)
[ 5002.425776] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5003.833971] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 18:57:42 (1788217062)
[ 5031.950903] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 18:58:10 (1788217090)
[ 5033.192302] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5034.424833] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 18:58:12 (1788217092)
[ 5035.433804] Lustre: *** cfs_fail_loc=805, val=0***
[ 5035.482562] Lustre: Mounted lustre-client - version 2.17.57_85_gb71fc30
[ 5040.122753] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5061.087175] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 192.168.206.140@tcp) was lost; in progress operations using this service will fail
[ 5061.131868] Lustre: Evicted from MGS (at 192.168.206.140@tcp) after server handle changed from 0x5b89ed621b60433f to 0x5b89ed621b604656
[ 5069.060546] Lustre: Unmounted lustre-client
[ 5077.421578] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 18:58:54 (1788217134)
[ 5093.855682] Lustre: 71425:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788217137/real 1788217137]  req@ffff9cc789306d80 x1875076343647232/t0(0) o36->lustre-MDT0000-mdc-ffff9cc786f25000@192.168.206.140@tcp:12/10 lens 504/576 e 0 to 1 dl 1788217153 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'mkdir.0' uid:0 gid:0 projid:4294967295
[ 5093.895026] Lustre: 71425:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
[ 5100.972854] LustreError: 2349:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff9cc7a3a24a80 x1875076343647488/t94489280517(94489280517) o101->lustre-MDT0000-mdc-ffff9cc786f25000@192.168.206.140@tcp:12/10 lens 648/608 e 0 to 0 dl 1788217176 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5113.313741] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5114.532845] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5122.353898] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 18:59:40 (1788217180)
[ 5123.569161] LustreError: lustre-OST0000-osc-ffff9cc786f25000: operation ost_write to node 192.168.206.140@tcp failed: rc = -107
[ 5123.586668] LustreError: Skipped 3 previous similar messages
[ 5123.612576] LustreError: lustre-OST0000-osc-ffff9cc786f25000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5123.626876] Lustre: 2350:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.140@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5123.659114] LustreError: 72804:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff9cc786f25000: namespace resource [0x240000400:0x3f42:0x0].0x0 (ffff9cc7a3b3eb00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5130.866196] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 18:59:49 (1788217189)
[ 5132.178034] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5133.406151] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 18:59:51 (1788217191)
[ 5134.735299] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5136.111705] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 18:59:54 (1788217194)
[ 5137.356218] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5138.722798] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 18:59:56 (1788217196)
[ 5139.969619] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5141.649972] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 18:59:59 (1788217199)
[ 5143.098555] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5145.007466] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 19:00:02 (1788217202)
[ 5146.357843] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5148.187438] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 19:00:06 (1788217206)
[ 5149.679383] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5151.165188] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 19:00:09 (1788217209)
[ 5152.547975] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5154.227734] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 19:00:12 (1788217212)
[ 5155.510754] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5156.837849] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 19:00:15 (1788217215)
[ 5158.153916] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5159.791153] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 19:00:17 (1788217217)
[ 5161.265534] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5162.810471] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 19:00:20 (1788217220)
[ 5165.220509] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5167.036851] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 19:00:24 (1788217224)
[ 5198.410530] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 19:00:56 (1788217256)
[ 5227.528759] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 19:01:25 (1788217285)
[ 5228.018620] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5237.855273] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 19:01:35 (1788217295)
[ 5238.107442] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5247.412246] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 19:01:45 (1788217305)
[ 5247.793413] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5256.818066] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 19:01:54 (1788217314)
[ 5257.099213] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5266.206035] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 19:02:03 (1788217323)
[ 5266.513374] Lustre: *** cfs_fail_loc=510, val=3***
[ 5274.487252] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 19:02:12 (1788217332)
[ 5274.801050] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5341.526903] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 19:03:19 (1788217399)
[ 5342.336562] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5401.567775] Lustre: lustre-OST0001-osc-ffff9cc786f25000: Connection to lustre-OST0001 (at 192.168.206.140@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5401.605485] Lustre: Skipped 9 previous similar messages
[ 5408.193457] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 19:04:26 (1788217466)
[ 5408.551951] LustreError: 83029:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5411.414095] LustreError: lustre-MDT0000-mdc-ffff9cc786f25000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5411.444058] LustreError: 83043:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9cc786f25000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9cc7a3b3b900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5411.470785] LustreError: 83043:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5411.486581] Lustre: lustre-MDT0000-mdc-ffff9cc786f25000: Connection restored to 192.168.206.140@tcp (at 192.168.206.140@tcp)
[ 5411.499581] Lustre: Skipped 10 previous similar messages
[ 5412.641894] LustreError: 83029:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5414.746395] LustreError: 83059:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5417.658767] LustreError: lustre-MDT0000-mdc-ffff9cc786f25000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5417.689818] LustreError: 83073:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9cc786f25000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9cc790cc8500) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5417.740679] LustreError: 83073:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5418.831968] LustreError: 83059:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5425.727197] LustreError: lustre-MDT0000-mdc-ffff9cc786f25000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5426.196135] LustreError: 83115:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5429.234261] LustreError: lustre-MDT0000-mdc-ffff9cc786f25000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5429.345755] LustreError: 83135:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5429.352616] LustreError: 83135:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5430.264828] LustreError: 83115:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5430.283112] LustreError: 83115:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9cc786f25000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5430.496843] LustreError: 83141:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5430.511019] LustreError: 83141:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5432.868526] LustreError: 83163:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5432.885398] LustreError: 83163:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5436.473855] LustreError: 83190:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5439.384535] LustreError: lustre-MDT0000-mdc-ffff9cc786f25000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5439.422775] LustreError: 83204:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9cc786f25000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9cc7a3b3eb00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5439.466519] LustreError: 83204:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5440.576809] LustreError: 83190:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5447.916586] LustreError: lustre-MDT0000-mdc-ffff9cc786f25000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5448.462133] LustreError: 83246:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5451.429985] LustreError: lustre-MDT0000-mdc-ffff9cc786f25000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5451.452392] LustreError: 83261:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9cc786f25000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9cc7a3b3ec00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5451.481992] LustreError: 83261:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5452.535128] LustreError: 83246:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5455.637797] LustreError: lustre-MDT0000-mdc-ffff9cc786f25000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5455.706882] LustreError: 83294:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5455.716156] LustreError: 83294:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5456.711632] LustreError: 83274:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9cc786f25000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5464.726162] LustreError: 83346:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5464.739482] LustreError: 83346:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5468.143170] LustreError: lustre-MDT0000-mdc-ffff9cc786f25000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5468.235219] LustreError: 83368:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5468.249772] LustreError: 83368:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5468.799199] LustreError: 83346:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5468.812973] LustreError: 83346:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5468.831924] LustreError: 83346:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9cc786f25000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5479.645220] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 19:05:37 (1788217537)
[ 5498.886989] LustreError: 68265:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5498.894740] LustreError: 68265:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5502.975105] LustreError: 68265:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5502.992518] LustreError: 68265:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5508.799957] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 19:06:06 (1788217566)
[ 5559.132287] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 19:06:56 (1788217616)
[ 5622.774660] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 19:08:00 (1788217680)
[ 5656.488727] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 19:08:34 (1788217714)
[ 5656.846284] Lustre: Mounted lustre-client - version 2.17.57_85_gb71fc30
[ 5779.919600] Lustre: Unmounted lustre-client
[ 5785.688798] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 19:10:43 (1788217843)
[ 5786.221050] LustreError: 87670:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5788.088339] LustreError: lustre-OST0000-osc-ffff9cc786f25000: operation ldlm_enqueue to node 192.168.206.140@tcp failed: rc = -107
[ 5788.102031] LustreError: Skipped 9 previous similar messages
[ 5788.112155] LustreError: lustre-OST0000-osc-ffff9cc786f25000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5788.131392] LustreError: 87753:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff9cc786f25000: namespace resource [0x240000400:0x3f6d:0x0].0x0 (ffff9cc7a3b3e600) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5788.151774] LustreError: 87753:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5788.416090] LustreError: 87670:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5788.424660] Lustre: 2351:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.140@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5793.978256] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 19:10:51 (1788217851)
[ 5852.128146] Lustre: 88338:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788217856/real 1788217856]  req@ffff9cc7a3a24000 x1875076343823232/t0(0) o101->lustre-MDT0000-mdc-ffff9cc786f25000@192.168.206.140@tcp:12/10 lens 328/344 e 0 to 1 dl 1788217911 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 5852.170658] Lustre: 88338:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 10 previous similar messages
[ 5857.928806] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 19:11:56 (1788217916)
[ 5859.048255] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 5860.499280] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 19:11:58 (1788217918)
[ 5924.047879] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 5925.639234] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 19:13:03 (1788217983)
[ 5983.841923] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 19:14:01 (1788218041)
[ 5986.080677] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 5988.435493] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 19:14:05 (1788218045)
[ 5989.887973] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 5991.413460] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 19:14:09 (1788218049)
[ 6028.599311] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 19:14:46 (1788218086)
[ 6042.205293] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6070.239738] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 192.168.206.140@tcp) was lost; in progress operations using this service will fail
[ 6070.256887] LustreError: Skipped 2 previous similar messages
[ 6080.498826] Lustre: lustre-MDT0000-mdc-ffff9cc786f25000: Connection to lustre-MDT0000 (at 192.168.206.140@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6080.521193] Lustre: Skipped 16 previous similar messages
[ 6080.534418] Lustre: Evicted from MGS (at 192.168.206.140@tcp) after server handle changed from 0x5b89ed621b605193 to 0x5b89ed621b606ade
[ 6080.546381] Lustre: Skipped 2 previous similar messages
[ 6080.560083] Lustre: MGC192.168.206.140@tcp: Connection restored to 192.168.206.140@tcp (at 192.168.206.140@tcp)
[ 6080.568232] Lustre: Skipped 14 previous similar messages
[ 6094.099487] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6095.508214] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6105.253317] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 19:16:02 (1788218162)
[ 6108.674909] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6110.636828] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 19:16:08 (1788218168)
[ 6126.574156] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 192.168.206.140@tcp) was lost; in progress operations using this service will fail
[ 6126.641320] Lustre: Evicted from MGS (at 192.168.206.140@tcp) after server handle changed from 0x5b89ed621b606ade to 0x5b89ed621b606e96
[ 6136.907722] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 19:16:34 (1788218194)
[ 6191.429323] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 19:17:29 (1788218249)
[ 6258.773532] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6262.891250] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6347.743459] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 192.168.206.140@tcp) was lost; in progress operations using this service will fail
[ 6347.752775] LustreError: Skipped 1 previous similar message
[ 6358.012516] Lustre: Evicted from MGS (at 192.168.206.140@tcp) after server handle changed from 0x5b89ed621b6070a3 to 0x5b89ed621b607ef7
[ 6358.044653] Lustre: Skipped 1 previous similar message
[ 6374.887547] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6376.352440] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6428.277818] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6430.217405] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6454.239621] Lustre: 2350:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788218458/real 1788218458]  req@ffff9cc7b88f1880 x1875076350391040/t0(0) o400->lustre-MDT0000-mdc-ffff9cc786f25000@192.168.206.140@tcp:12/10 lens 224/224 e 0 to 1 dl 1788218513 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6454.312890] Lustre: 2350:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 30 previous similar messages
[ 6571.072883] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 19:23:48 (1788218628)
[ 6583.238173] LustreError: lustre-OST0000-osc-ffff9cc786f25000: operation ost_setattr to node 192.168.206.140@tcp failed: rc = -107
[ 6583.248992] LustreError: Skipped 320 previous similar messages
[ 6638.166577] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6643.075655] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6882.937530] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 19:29:00 (1788218940)
[ 7008.230916] Lustre: lustre-MDT0000-mdc-ffff9cc786f25000: Connection to lustre-MDT0000 (at 192.168.206.140@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7008.243431] Lustre: Skipped 6 previous similar messages
[ 7023.592112] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 192.168.206.140@tcp) was lost; in progress operations using this service will fail
[ 7023.606392] LustreError: Skipped 1 previous similar message
[ 7023.620121] Lustre: Evicted from MGS (at 192.168.206.140@tcp) after server handle changed from 0x5b89ed621b608724 to 0x5b89ed621b6fd6e5
[ 7023.626689] Lustre: Skipped 1 previous similar message
[ 7023.633580] Lustre: MGC192.168.206.140@tcp: Connection restored to 192.168.206.140@tcp (at 192.168.206.140@tcp)
[ 7023.650087] Lustre: Skipped 9 previous similar messages
[ 7055.856918] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 19:31:53 (1788219113)
[ 7057.023621] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7058.822730] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 19:31:56 (1788219116)
[ 7060.515647] LustreError: lustre-MDT0000-mdc-ffff9cc786f25000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7067.085203] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 19:32:04 (1788219124)
[ 7234.048745] LustreError: lustre-OST0000-osc-ffff9cc786f25000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7235.860528] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 19:34:53 (1788219293)
[ 7261.663151] Lustre: 2352:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788219300/real 1788219300]  req@ffff9cc79ffc1500 x1875076361188096/t0(0) o4->lustre-OST0000-osc-ffff9cc786f25000@192.168.206.140@tcp:6/4 lens 4584/448 e 0 to 1 dl 1788219320 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7261.707505] Lustre: 2352:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 7278.629921] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 19:35:36 (1788219336)
[ 7280.035224] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7281.409636] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 19:35:39 (1788219339)
[ 7282.859935] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7284.408499] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 19:35:42 (1788219342)
[ 7320.009364] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 19:36:17 (1788219377)
[ 7358.963894] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 192.168.206.140@tcp) was lost; in progress operations using this service will fail
[ 7359.001833] Lustre: Evicted from MGS (at 192.168.206.140@tcp) after server handle changed from 0x5b89ed621b6fd6e5 to 0x5b89ed621b700ebb
[ 7359.021141] Lustre: Skipped 1 previous similar message
[ 7371.838357] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 19:37:09 (1788219429)
[ 7373.226703] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7374.594595] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 19:37:12 (1788219432)
[ 7375.790037] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7377.543646] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 19:37:15 (1788219435)
[ 7385.969497] Lustre: Unmounted lustre-client
[ 7387.152868] Lustre: Mounted lustre-client - version 2.17.57_85_gb71fc30
[ 7389.821981] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7421.326571] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2